"Fossies" - the Fresh Open Source Software Archive

Member "roundup-2.0.0/share/doc/roundup/html/_sources/design.txt" (16 May 2020, 66532 Bytes) of package /linux/www/roundup-2.0.0.tar.gz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. See also the latest Fossies "Diffs" side-by-side code changes report for "design.txt": 1.6.1_vs_2.0.0.

    1 ========================================================
    2 Roundup - An Issue-Tracking System for Knowledge Workers
    3 ========================================================
    4 
    5 :Authors: Ka-Ping Yee (original), Richard Jones (implementation)
    6 
    7 Contents
    8 
    9 .. contents::
   10    :local:
   11 
   12 Introduction
   13 ---------------
   14 
   15 This document presents a description of the components of the Roundup
   16 system and specifies their interfaces and behaviour in sufficient detail
   17 to guide an implementation. For the philosophy and rationale behind the
   18 Roundup design, see the first-round Software Carpentry `submission for
   19 Roundup`__. This document fleshes out that design as well as specifying
   20 interfaces so that the components can be developed separately.
   21 
   22 __ spec.html
   23 
   24 
   25 The Layer Cake
   26 -----------------
   27 
   28 Lots of software design documents come with a picture of a cake.
   29 Everybody seems to like them.  I also like cakes (I think they are
   30 tasty).  So I, too, shall include a picture of a cake here::
   31 
   32      ________________________________________________________________
   33     | E-mail Client |  Web Browser  |  Detector Scripts  |   Shell   |
   34     |---------------+---------------+--------------------+-----------|
   35     |  E-mail User  |   Web User    |     Detector       |  Command  | 
   36     |----------------------------------------------------------------|
   37     |                    Roundup Database Layer                      |
   38     |----------------------------------------------------------------|
   39     |                     Hyperdatabase Layer                        |
   40     |----------------------------------------------------------------|
   41     |                        Storage Layer                           |
   42      ----------------------------------------------------------------
   43 
   44 The colourful parts of the cake are part of our system; the faint grey
   45 parts of the cake are external components.
   46 
   47 I will now proceed to forgo all table manners and eat from the bottom of
   48 the cake to the top.  You may want to stand back a bit so you don't get
   49 covered in crumbs.
   50 
   51 
   52 Hyperdatabase
   53 -------------
   54 
   55 The lowest-level component to be implemented is the hyperdatabase. The
   56 hyperdatabase is a flexible data store that can hold configurable data
   57 in records which we call items.
   58 
   59 The hyperdatabase is implemented on top of the storage layer, an
   60 external module for storing its data. The "batteries-includes" distribution
   61 implements the hyperdatabase on the standard anydbm module.  The storage
   62 layer could be a third-party RDBMS; for a low-maintenance solution,
   63 implementing the hyperdatabase on the SQLite RDBMS is suggested.
   64 
   65 
   66 Dates and Date Arithmetic
   67 ~~~~~~~~~~~~~~~~~~~~~~~~~
   68 
   69 Before we get into the hyperdatabase itself, we need a way of handling
   70 dates.  The hyperdatabase module provides Timestamp objects for
   71 representing date-and-time stamps and Interval objects for representing
   72 date-and-time intervals.
   73 
   74 As strings, date-and-time stamps are specified with the date in ISO8601
   75 international standard format (``yyyy-mm-dd``) joined to the time
   76 (``hh:mm:ss``) by a period "``.``".  Dates in this form can be easily
   77 compared and are fairly readable when printed.  An example of a valid
   78 stamp is "``2000-06-24.13:03:59``". We'll call this the "full date
   79 format".  When Timestamp objects are printed as strings, they appear in
   80 the full date format with the time always given in GMT.  The full date
   81 format is always exactly 19 characters long.
   82 
   83 For user input, some partial forms are also permitted: the whole time or
   84 just the seconds may be omitted; and the whole date may be omitted or
   85 just the year may be omitted.  If the time is given, the time is
   86 interpreted in the user's local time zone. The Date constructor takes
   87 care of these conversions. In the following examples, suppose that
   88 ``yyyy`` is the current year, ``mm`` is the current month, and ``dd`` is
   89 the current day of the month; and suppose that the user is on Eastern
   90 Standard Time.
   91 
   92 -   "2000-04-17" means <Date 2000-04-17.00:00:00>
   93 -   "01-25" means <Date yyyy-01-25.00:00:00>
   94 -   "2000-04-17.03:45" means <Date 2000-04-17.08:45:00>
   95 -   "08-13.22:13" means <Date yyyy-08-14.03:13:00>
   96 -   "11-07.09:32:43" means <Date yyyy-11-07.14:32:43>
   97 -   "14:25" means <Date yyyy-mm-dd.19:25:00>
   98 -   "8:47:11" means <Date yyyy-mm-dd.13:47:11>
   99 -   the special date "." means "right now"
  100 
  101 
  102 Date intervals are specified using the suffixes "y", "m", and "d".  The
  103 suffix "w" (for "week") means 7 days. Time intervals are specified in
  104 hh:mm:ss format (the seconds may be omitted, but the hours and minutes
  105 may not).
  106 
  107 -   "3y" means three years
  108 -   "2y 1m" means two years and one month
  109 -   "1m 25d" means one month and 25 days
  110 -   "2w 3d" means two weeks and three days
  111 -   "1d 2:50" means one day, two hours, and 50 minutes
  112 -   "14:00" means 14 hours
  113 -   "0:04:33" means four minutes and 33 seconds
  114 
  115 
  116 The Date class should understand simple date expressions of the form
  117 *stamp* ``+`` *interval* and *stamp* ``-`` *interval*. When adding or
  118 subtracting intervals involving months or years, the components are
  119 handled separately.  For example, when evaluating "``2000-06-25 + 1m
  120 10d``", we first add one month to get 2000-07-25, then add 10 days to
  121 get 2000-08-04 (rather than trying to decide whether 1m 10d means 38 or
  122 40 or 41 days).
  123 
  124 Here is an outline of the Date and Interval classes::
  125 
  126     class Date:
  127         def __init__(self, spec, offset):
  128             """Construct a date given a specification and a time zone
  129             offset.
  130 
  131             'spec' is a full date or a partial form, with an optional
  132             added or subtracted interval.  'offset' is the local time
  133             zone offset from GMT in hours.
  134             """
  135 
  136         def __add__(self, interval):
  137             """Add an interval to this date to produce another date."""
  138 
  139         def __sub__(self, interval):
  140             """Subtract an interval from this date to produce another
  141             date.
  142             """
  143 
  144         def __cmp__(self, other):
  145             """Compare this date to another date."""
  146 
  147         def __str__(self):
  148             """Return this date as a string in the yyyy-mm-dd.hh:mm:ss
  149             format.
  150             """
  151 
  152         def local(self, offset):
  153             """Return this date as yyyy-mm-dd.hh:mm:ss in a local time
  154             zone.
  155             """
  156 
  157     class Interval:
  158         def __init__(self, spec):
  159             """Construct an interval given a specification."""
  160 
  161         def __cmp__(self, other):
  162             """Compare this interval to another interval."""
  163             
  164         def __str__(self):
  165             """Return this interval as a string."""
  166 
  167 
  168 
  169 Here are some examples of how these classes would behave in practice.
  170 For the following examples, assume that we are on Eastern Standard Time
  171 and the current local time is 19:34:02 on 25 June 2000::
  172 
  173     >>> Date(".")
  174     <Date 2000-06-26.00:34:02>
  175     >>> _.local(-5)
  176     "2000-06-25.19:34:02"
  177     >>> Date(". + 2d")
  178     <Date 2000-06-28.00:34:02>
  179     >>> Date("1997-04-17", -5)
  180     <Date 1997-04-17.00:00:00>
  181     >>> Date("01-25", -5)
  182     <Date 2000-01-25.00:00:00>
  183     >>> Date("08-13.22:13", -5)
  184     <Date 2000-08-14.03:13:00>
  185     >>> Date("14:25", -5)
  186     <Date 2000-06-25.19:25:00>
  187     >>> Interval("  3w  1  d  2:00")
  188     <Interval 22d 2:00>
  189     >>> Date(". + 2d") - Interval("3w")
  190     <Date 2000-06-07.00:34:02>
  191 
  192 
  193 Items and Classes
  194 ~~~~~~~~~~~~~~~~~
  195 
  196 Items contain data in properties.  To Python, these properties are
  197 presented as the key-value pairs of a dictionary. Each item belongs to a
  198 class which defines the names and types of its properties.  The database
  199 permits the creation and modification of classes as well as items.
  200 
  201 
  202 Identifiers and Designators
  203 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  204 
  205 Each item has a numeric identifier which is unique among items in its
  206 class.  The items are numbered sequentially within each class in order
  207 of creation, starting from 1. The designator for an item is a way to
  208 identify an item in the database, and consists of the name of the item's
  209 class concatenated with the item's numeric identifier.
  210 
  211 For example, if "spam" and "eggs" are classes, the first item created in
  212 class "spam" has id 1 and designator "spam1". The first item created in
  213 class "eggs" also has id 1 but has the distinct designator "eggs1". Item
  214 designators are conventionally enclosed in square brackets when
  215 mentioned in plain text.  This permits a casual mention of, say,
  216 "[patch37]" in an e-mail message to be turned into an active hyperlink.
  217 
  218 
  219 Property Names and Types
  220 ~~~~~~~~~~~~~~~~~~~~~~~~
  221 
  222 Property names must begin with a letter.
  223 
  224 A property may be one of five basic types:
  225 
  226 - String properties are for storing arbitrary-length strings.
  227 
  228 - Boolean properties are for storing true/false, or yes/no values.
  229 
  230 - Integer properties are for storing Integer (non real) numeric values.
  231 
  232 - Number properties are for storing numeric values.
  233 
  234 - Date properties store date-and-time stamps. Their values are Timestamp
  235   objects.
  236 
  237 - A Link property refers to a single other item selected from a
  238   specified class.  The class is part of the property; the value is an
  239   integer, the id of the chosen item.
  240 
  241 - A Multilink property refers to possibly many items in a specified
  242   class.  The value is a list of integers.
  243 
  244 *None* is also a permitted value for any of these property types.  An
  245 attempt to store None into a Multilink property stores an empty list.
  246 
  247 A property that is not specified will return as None from a *get*
  248 operation.
  249 
  250 
  251 Hyperdb Interface Specification
  252 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  253 
  254 TODO: replace the Interface Specifications with links to the pydoc
  255 
  256 The hyperdb module provides property objects to designate the different
  257 kinds of properties.
  258 
  259 All property objects support the following settings:
  260 
  261 quiet=False:
  262       if set to True, changes to the property will not be shown to the
  263       user. This can be used for administrative properties that are
  264       automatically updated when a user makes some other change. This
  265       reduces confusion by the user and clutter in the display.
  266       The property change will not be shown in:
  267 
  268         - the change confirmation message when a change is entered in the web interface
  269         - the property change section of the change note email ("nosy email")
  270         - the web history shown at the bottom of an item page
  271 
  272 required=False:
  273         if set to True, the property name is returned when calling
  274         get_required_props(self, propnames = []). Any additional props
  275         specified in propnames is merged with the required props.
  276 
  277 default_value=None or [] depending on object type:
  278         this sets the default value if none is specified. The default
  279         value can be retrieved by calling the get_default_value()
  280         method on the property object.
  281  
  282 E.G. assuming title is part of an Issue::
  283 
  284     title=String(required=True, default_value="not set",quiet=True)
  285 
  286 will create a property called ``title`` that will be included in the
  287 get_required_props() output. Calling
  288 db.issue.properties['title'].get_default_value() will return "not set".
  289 Changes to the property will not be displayed in:
  290 
  291    - emailed change notes,
  292    - the history at the end of the item pages in the web interface
  293    - in the confirmation notice (displayed as a green banner)
  294      shown on changes.
  295 
  296 These objects are used when specifying what properties belong in classes::
  297 
  298     class String:
  299         def __init__(self, indexme='no'):
  300             """An object designating a String property."""
  301 
  302     class Boolean:
  303         def __init__(self):
  304             """An object designating a Boolean property."""
  305 
  306     class Integer:
  307         def __init__(self):
  308             """An object designating an Integer property."""
  309 
  310     class Number:
  311         def __init__(self):
  312             """An object designating a Number property."""
  313 
  314     class Date:
  315         def __init__(self):
  316             """An object designating a Date property."""
  317 
  318     class Link:
  319         def __init__(self, classname, do_journal='yes'):
  320             """An object designating a Link property that links to
  321             items in a specified class.
  322 
  323             If the do_journal argument is not 'yes' then changes to
  324             the property are not journalled in the linked item.
  325             """
  326 
  327     class Multilink:
  328         def __init__(self, classname, do_journal='yes'):
  329             """An object designating a Multilink property that links
  330             to items in a specified class.
  331 
  332             If the do_journal argument is not 'yes' then changes to
  333             the property are not journalled in the linked item(s).
  334             """
  335 
  336 
  337 Here is the interface provided by the hyperdatabase::
  338 
  339     class Database:
  340         """A database for storing records containing flexible data
  341         types.
  342         """
  343 
  344         def __init__(self, config, journaltag=None):
  345             """Open a hyperdatabase given a specifier to some storage.
  346 
  347             The 'storagelocator' is obtained from config.DATABASE. The
  348             meaning of 'storagelocator' depends on the particular
  349             implementation of the hyperdatabase.  It could be a file
  350             name, a directory path, a socket descriptor for a connection
  351             to a database over the network, etc.
  352 
  353             The 'journaltag' is a token that will be attached to the
  354             journal entries for any edits done on the database.  If
  355             'journaltag' is None, the database is opened in read-only
  356             mode: the Class.create(), Class.set(), Class.retire(), and
  357             Class.restore() methods are disabled.
  358             """
  359 
  360         def __getattr__(self, classname):
  361             """A convenient way of calling self.getclass(classname)."""
  362 
  363         def getclasses(self):
  364             """Return a list of the names of all existing classes."""
  365 
  366         def getclass(self, classname):
  367             """Get the Class object representing a particular class.
  368 
  369             If 'classname' is not a valid class name, a KeyError is
  370             raised.
  371             """
  372 
  373     class Class:
  374         """The handle to a particular class of items in a hyperdatabase.
  375         """
  376 
  377         def __init__(self, db, classname, **properties):
  378             """Create a new class with a given name and property
  379             specification.
  380 
  381             'classname' must not collide with the name of an existing
  382             class, or a ValueError is raised.  The keyword arguments in
  383             'properties' must map names to property objects, or a
  384             TypeError is raised.
  385 
  386             A proxied reference to the database is available as the
  387             'db' attribute on instances. For example, in
  388             'IssueClass.send_message', the following is used to lookup
  389             users, messages and files::
  390 
  391                 users = self.db.user
  392                 messages = self.db.msg
  393                 files = self.db.file
  394             """
  395 
  396         # Editing items:
  397 
  398         def create(self, **propvalues):
  399             """Create a new item of this class and return its id.
  400 
  401             The keyword arguments in 'propvalues' map property names to
  402             values. The values of arguments must be acceptable for the
  403             types of their corresponding properties or a TypeError is
  404             raised.  If this class has a key property, it must be
  405             present and its value must not collide with other key
  406             strings or a ValueError is raised.  Any other properties on
  407             this class that are missing from the 'propvalues' dictionary
  408             are set to None.  If an id in a link or multilink property
  409             does not refer to a valid item, an IndexError is raised.
  410             """
  411 
  412         def get(self, itemid, propname):
  413             """Get the value of a property on an existing item of this
  414             class.
  415 
  416             'itemid' must be the id of an existing item of this class or
  417             an IndexError is raised.  'propname' must be the name of a
  418             property of this class or a KeyError is raised.
  419             """
  420 
  421         def set(self, itemid, **propvalues):
  422             """Modify a property on an existing item of this class.
  423             
  424             'itemid' must be the id of an existing item of this class or
  425             an IndexError is raised.  Each key in 'propvalues' must be
  426             the name of a property of this class or a KeyError is
  427             raised.  All values in 'propvalues' must be acceptable types
  428             for their corresponding properties or a TypeError is raised.
  429             If the value of the key property is set, it must not collide
  430             with other key strings or a ValueError is raised.  If the
  431             value of a Link or Multilink property contains an invalid
  432             item id, a ValueError is raised.
  433             """
  434 
  435         def retire(self, itemid):
  436             """Retire an item.
  437             
  438             The properties on the item remain available from the get()
  439             method, and the item's id is never reused.  Retired items
  440             are not returned by the find(), list(), or lookup() methods,
  441             and other items may reuse the values of their key
  442             properties.
  443             """
  444 
  445         def restore(self, nodeid):
  446         '''Restore a retired node.
  447 
  448         Make node available for all operations like it was before
  449         retirement.
  450         '''
  451 
  452         def history(self, itemid):
  453             """Retrieve the journal of edits on a particular item.
  454 
  455             'itemid' must be the id of an existing item of this class or
  456             an IndexError is raised.
  457 
  458             The returned list contains tuples of the form
  459 
  460                 (date, tag, action, params)
  461 
  462             'date' is a Timestamp object specifying the time of the
  463             change and 'tag' is the journaltag specified when the
  464             database was opened. 'action' may be:
  465 
  466                 'create' or 'set' -- 'params' is a dictionary of
  467                     property values
  468                 'link' or 'unlink' -- 'params' is (classname, itemid,
  469                     propname)
  470                 'retire' -- 'params' is None
  471             """
  472 
  473         # Locating items:
  474 
  475         def setkey(self, propname):
  476             """Select a String property of this class to be the key
  477             property.
  478 
  479             'propname' must be the name of a String property of this
  480             class or None, or a TypeError is raised.  The values of the
  481             key property on all existing items must be unique or a
  482             ValueError is raised.
  483             """
  484 
  485         def getkey(self):
  486             """Return the name of the key property for this class or
  487             None.
  488             """
  489 
  490         def lookup(self, keyvalue):
  491             """Locate a particular item by its key property and return
  492             its id.
  493 
  494             If this class has no key property, a TypeError is raised.
  495             If the 'keyvalue' matches one of the values for the key
  496             property among the items in this class, the matching item's
  497             id is returned; otherwise a KeyError is raised.
  498             """
  499 
  500         def find(self, **propspec):
  501             """Get the ids of items in this class which link to the
  502             given items.
  503 
  504             'propspec' consists of keyword args propname=itemid or
  505                        propname={<itemid 1>:1, <itemid 2>:1, ...}
  506             'propname' must be the name of a property in this class,
  507                        or a KeyError is raised.  That property must
  508                        be a Link or Multilink property, or a TypeError
  509                        is raised.
  510 
  511             Any item in this class whose 'propname' property links to
  512             any of the itemids will be returned. Examples::
  513 
  514                 db.issue.find(messages='1')
  515                 db.issue.find(messages={'1':1,'3':1}, files={'7':1})
  516             """
  517 
  518         def filter(self, search_matches, filterspec, sort, group,
  519                    retired, exact_match_spec, limit, offset):
  520             """Return a list of the ids of the active nodes in this class that
  521             match the 'filter' spec, sorted by the group spec and then the
  522             sort spec. The arguments sort, group, retired, and
  523             exact_match_spec are optional.
  524 
  525             "search_matches" is a container type which by default is
  526             None and optionally contains IDs of items to match. If
  527             non-empty only IDs of the initial set are returned.
  528 
  529             "filterspec" is {propname: value(s)}
  530             "exact_match_spec" is the same format as "filterspec" but
  531             specifies exact match for the given propnames. This only
  532             makes a difference for String properties, these specify case
  533             insensitive substring search when in "filterspec" and exact
  534             match when in exact_match_spec.
  535 
  536             "sort" and "group" are [(dir, prop), ...] where dir is '+', '-'
  537             or None and prop is a prop name or None. Note that for
  538             backward-compatibility reasons a single (dir, prop) tuple is
  539             also allowed.
  540 
  541             The parameter retired when set to False, returns only live
  542             (un-retired) results. When setting it to True, only retired
  543             items are returned. If None, both retired and unretired
  544             items are returned. The default is False, i.e. only live
  545             items are returned by default.
  546 
  547             The "limit" and "offset" parameters define a limit on the
  548             number of results returned and an offset before returning
  549             any results, respectively. These can be used when displaying
  550             a number of items in a pagination application or similar. A
  551             common use-case is returning the first item of a sorted
  552             search by specifying limit=1 (i.e. the maximum or minimum
  553             depending on sort order).
  554 
  555             The filter must match all properties specificed. If the property
  556             value to match is a list:
  557 
  558             1. String properties must match all elements in the list, and
  559             2. Other properties must match any of the elements in the list.
  560 
  561             This also means that for strings in exact_match_spec it
  562             doesn't make sense to specify multiple values because those
  563             cannot all be matched.
  564 
  565             The propname in filterspec and prop in a sort/group spec may be
  566             transitive, i.e., it may contain properties of the form
  567             link.link.link.name, e.g. you can search for all issues where
  568             a message was added by a certain user in the last week with a
  569             filterspec of
  570             {'messages.author' : '42', 'messages.creation' : '.-1w;'}
  571             """
  572 
  573         def list(self):
  574             """Return a list of the ids of the active items in this
  575             class.
  576             """
  577 
  578         def count(self):
  579             """Get the number of items in this class.
  580 
  581             If the returned integer is 'numitems', the ids of all the
  582             items in this class run from 1 to numitems, and numitems+1
  583             will be the id of the next item to be created in this class.
  584             """
  585 
  586         # Manipulating properties:
  587 
  588         def getprops(self):
  589             """Return a dictionary mapping property names to property
  590             objects.
  591             """
  592 
  593         def addprop(self, **properties):
  594             """Add properties to this class.
  595 
  596             The keyword arguments in 'properties' must map names to
  597             property objects, or a TypeError is raised.  None of the
  598             keys in 'properties' may collide with the names of existing
  599             properties, or a ValueError is raised before any properties
  600             have been added.
  601             """
  602 
  603         def getitem(self, itemid, cache=1):
  604             """ Return a Item convenience wrapper for the item.
  605 
  606             'itemid' must be the id of an existing item of this class or
  607             an IndexError is raised.
  608 
  609             'cache' indicates whether the transaction cache should be
  610             queried for the item. If the item has been modified and you
  611             need to determine what its values prior to modification are,
  612             you need to set cache=0.
  613             """
  614 
  615     class Item:
  616         """ A convenience wrapper for the given item. It provides a
  617         mapping interface to a single item's properties
  618         """
  619 
  620 Hyperdatabase Implementations
  621 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  622 
  623 Hyperdatabase implementations exist to create the interface described in
  624 the `hyperdb interface specification`_ over an existing storage
  625 mechanism. Examples are relational databases, \*dbm key-value databases,
  626 and so on.
  627 
  628 Several implementations are provided - they belong in the
  629 ``roundup.backends`` package.
  630 
  631 
  632 Application Example
  633 ~~~~~~~~~~~~~~~~~~~
  634 
  635 Here is an example of how the hyperdatabase module would work in
  636 practice::
  637 
  638     >>> import hyperdb
  639     >>> db = hyperdb.Database("foo.db", "ping")
  640     >>> db
  641     <hyperdb.Database "foo.db" opened by "ping">
  642     >>> hyperdb.Class(db, "status", name=hyperdb.String())
  643     <hyperdb.Class "status">
  644     >>> _.setkey("name")
  645     >>> db.status.create(name="unread")
  646     1
  647     >>> db.status.create(name="in-progress")
  648     2
  649     >>> db.status.create(name="testing")
  650     3
  651     >>> db.status.create(name="resolved")
  652     4
  653     >>> db.status.count()
  654     4
  655     >>> db.status.list()
  656     [1, 2, 3, 4]
  657     >>> db.status.lookup("in-progress")
  658     2
  659     >>> db.status.retire(3)
  660     >>> db.status.list()
  661     [1, 2, 4]
  662     >>> hyperdb.Class(db, "issue", title=hyperdb.String(), status=hyperdb.Link("status"))
  663     <hyperdb.Class "issue">
  664     >>> db.issue.create(title="spam", status=1)
  665     1
  666     >>> db.issue.create(title="eggs", status=2)
  667     2
  668     >>> db.issue.create(title="ham", status=4)
  669     3
  670     >>> db.issue.create(title="arguments", status=2)
  671     4
  672     >>> db.issue.create(title="abuse", status=1)
  673     5
  674     >>> hyperdb.Class(db, "user", username=hyperdb.String(),
  675     ... password=hyperdb.String())
  676     <hyperdb.Class "user">
  677     >>> db.issue.addprop(fixer=hyperdb.Link("user"))
  678     >>> db.issue.getprops()
  679     {"title": <hyperdb.String>, "status": <hyperdb.Link to "status">,
  680      "user": <hyperdb.Link to "user">}
  681     >>> db.issue.set(5, status=2)
  682     >>> db.issue.get(5, "status")
  683     2
  684     >>> db.status.get(2, "name")
  685     "in-progress"
  686     >>> db.issue.get(5, "title")
  687     "abuse"
  688     >>> db.issue.find("status", db.status.lookup("in-progress"))
  689     [2, 4, 5]
  690     >>> db.issue.history(5)
  691     [(<Date 2000-06-28.19:09:43>, "ping", "create", {"title": "abuse",
  692     "status": 1}),
  693      (<Date 2000-06-28.19:11:04>, "ping", "set", {"status": 2})]
  694     >>> db.status.history(1)
  695     [(<Date 2000-06-28.19:09:43>, "ping", "link", ("issue", 5, "status")),
  696      (<Date 2000-06-28.19:11:04>, "ping", "unlink", ("issue", 5, "status"))]
  697     >>> db.status.history(2)
  698     [(<Date 2000-06-28.19:11:04>, "ping", "link", ("issue", 5, "status"))]
  699 
  700 
  701 For the purposes of journalling, when a Multilink property is set to a
  702 new list of items, the hyperdatabase compares the old list to the new
  703 list. The journal records "unlink" events for all the items that appear
  704 in the old list but not the new list, and "link" events for all the
  705 items that appear in the new list but not in the old list.
  706 
  707 
  708 Roundup Database
  709 ----------------
  710 
  711 The Roundup database layer is implemented on top of the hyperdatabase
  712 and mediates calls to the database. Some of the classes in the Roundup
  713 database are considered issue classes. The Roundup database layer adds
  714 detectors and user items, and on issues it provides mail spools, nosy
  715 lists, and superseders.
  716 
  717 
  718 Reserved Classes
  719 ~~~~~~~~~~~~~~~~
  720 
  721 Internal to this layer we reserve three special classes of items that
  722 are not issues.
  723 
  724 Users
  725 """""
  726 
  727 Users are stored in the hyperdatabase as items of class "user".  The
  728 "user" class has the definition::
  729 
  730     hyperdb.Class(db, "user", username=hyperdb.String(),
  731                               password=hyperdb.String(),
  732                               address=hyperdb.String())
  733     db.user.setkey("username")
  734 
  735 Messages
  736 """"""""
  737 
  738 E-mail messages are represented by hyperdatabase items of class "msg".
  739 The actual text content of the messages is stored in separate files.
  740 (There's no advantage to be gained by stuffing them into the
  741 hyperdatabase, and if messages are stored in ordinary text files, they
  742 can be grepped from the command line.)  The text of a message is saved
  743 in a file named after the message item designator (e.g. "msg23") for the
  744 sake of the command interface (see below).  Attachments are stored
  745 separately and associated with "file" items. The "msg" class has the
  746 definition::
  747 
  748     hyperdb.Class(db, "msg", author=hyperdb.Link("user"),
  749                              recipients=hyperdb.Multilink("user"),
  750                              date=hyperdb.Date(),
  751                              summary=hyperdb.String(),
  752                              files=hyperdb.Multilink("file"))
  753 
  754 The "author" property indicates the author of the message (a "user" item
  755 must exist in the hyperdatabase for any messages that are stored in the
  756 system). The "summary" property contains a summary of the message for
  757 display in a message index.
  758 
  759 
  760 Files
  761 """""
  762 
  763 Submitted files are represented by hyperdatabase items of class "file".
  764 Like e-mail messages, the file content is stored in files outside the
  765 database, named after the file item designator (e.g. "file17"). The
  766 "file" class has the definition::
  767 
  768     hyperdb.Class(db, "file", user=hyperdb.Link("user"),
  769                               name=hyperdb.String(),
  770                               type=hyperdb.String())
  771 
  772 The "user" property indicates the user who submitted the file, the
  773 "name" property holds the original name of the file, and the "type"
  774 property holds the MIME type of the file as received.
  775 
  776 
  777 Issue Classes
  778 ~~~~~~~~~~~~~
  779 
  780 All issues have the following standard properties:
  781 
  782 =========== ==========================
  783 Property    Definition
  784 =========== ==========================
  785 title       hyperdb.String()
  786 messages    hyperdb.Multilink("msg")
  787 files       hyperdb.Multilink("file")
  788 nosy        hyperdb.Multilink("user")
  789 superseder  hyperdb.Multilink("issue")
  790 =========== ==========================
  791 
  792 Also, two Date properties named "creation" and "activity" are fabricated
  793 by the Roundup database layer. Two user Link properties, "creator" and
  794 "actor" are also fabricated. By "fabricated" we mean that no such
  795 properties are actually stored in the hyperdatabase, but when properties
  796 on issues are requested, the "creation"/"creator" and "activity"/"actor"
  797 properties are made available. The value of the "creation"/"creator"
  798 properties relate to issue creation, and the value of the "activity"/
  799 "actor" properties relate to the last editing of any property on the issue
  800 (equivalently, these are the dates on the first and last records in the
  801 issue's journal).
  802 
  803 
  804 Roundupdb Interface Specification
  805 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  806 
  807 The interface to a Roundup database delegates most method calls to the
  808 hyperdatabase, except for the following changes and additional methods::
  809 
  810     class Database:
  811         def getuid(self):
  812             """Return the id of the "user" item associated with the user
  813             that owns this connection to the hyperdatabase."""
  814 
  815     class Class:
  816         # Overridden methods:
  817 
  818         def create(self, **propvalues):
  819         def set(self, **propvalues):
  820         def retire(self, itemid):
  821             """These operations trigger detectors and can be vetoed.
  822             Attempts to modify the "creation", "creator", "activity"
  823             properties or "actor" cause a KeyError.
  824             """
  825 
  826     class IssueClass(Class):
  827         # Overridden methods:
  828 
  829         def __init__(self, db, classname, **properties):
  830             """The newly-created class automatically includes the
  831             "messages", "files", "nosy", and "superseder" properties.
  832             If the 'properties' dictionary attempts to specify any of
  833             these properties or a "creation", "creator", "activity" or
  834             "actor" property, a ValueError is raised."""
  835 
  836         def get(self, itemid, propname):
  837         def getprops(self):
  838             """In addition to the actual properties on the item, these
  839             methods provide the "creation", "creator", "activity" and
  840             "actor" properties."""
  841 
  842         # New methods:
  843 
  844         def addmessage(self, itemid, summary, text):
  845             """Add a message to an issue's mail spool.
  846 
  847             A new "msg" item is constructed using the current date, the
  848             user that owns the database connection as the author, and
  849             the specified summary text.  The "files" and "recipients"
  850             fields are left empty.  The given text is saved as the body
  851             of the message and the item is appended to the "messages"
  852             field of the specified issue.
  853             """
  854 
  855         def nosymessage(self, itemid, msgid):
  856             """Send a message to the members of an issue's nosy list.
  857 
  858             The message is sent only to users on the nosy list who are
  859             not already on the "recipients" list for the message.  These
  860             users are then added to the message's "recipients" list.
  861             """
  862 
  863 
  864 Default Schema
  865 ~~~~~~~~~~~~~~
  866 
  867 The default schema included with Roundup turns it into a typical
  868 software bug tracker.  The database is set up like this::
  869 
  870     pri = Class(db, "priority", name=hyperdb.String(),
  871                 order=hyperdb.String())
  872     pri.setkey("name")
  873     pri.create(name="critical", order="1")
  874     pri.create(name="urgent", order="2")
  875     pri.create(name="bug", order="3")
  876     pri.create(name="feature", order="4")
  877     pri.create(name="wish", order="5")
  878 
  879     stat = Class(db, "status", name=hyperdb.String(),
  880                  order=hyperdb.String())
  881     stat.setkey("name")
  882     stat.create(name="unread", order="1")
  883     stat.create(name="deferred", order="2")
  884     stat.create(name="chatting", order="3")
  885     stat.create(name="need-eg", order="4")
  886     stat.create(name="in-progress", order="5")
  887     stat.create(name="testing", order="6")
  888     stat.create(name="done-cbb", order="7")
  889     stat.create(name="resolved", order="8")
  890 
  891     Class(db, "keyword", name=hyperdb.String())
  892 
  893     Class(db, "issue", fixer=hyperdb.Multilink("user"),
  894                        keyword=hyperdb.Multilink("keyword"),
  895                        priority=hyperdb.Link("priority"),
  896                        status=hyperdb.Link("status"))
  897 
  898 (The "order" property hasn't been explained yet.  It gets used by the
  899 Web user interface for sorting.)
  900 
  901 The above isn't as pretty-looking as the schema specification in the
  902 first-stage submission, but it could be made just as easy with the
  903 addition of a convenience function like Choice for setting up the
  904 "priority" and "status" classes::
  905 
  906     def Choice(name, *options):
  907         cl = Class(db, name, name=hyperdb.String(),
  908                    order=hyperdb.String())
  909         for i in range(len(options)):
  910             cl.create(name=option[i], order=i)
  911         return hyperdb.Link(name)
  912 
  913 
  914 .. index:: schema; detectors design
  915    single: detectors; design of
  916    single: pair: detectors; auditors
  917    single: pair: detectors; reactors
  918 
  919 Detector Interface
  920 ------------------
  921 
  922 Detectors are Python functions that are triggered on certain kinds of
  923 events.  These functions are placed in a special directory which exists
  924 just for this purpose.  Importing the Roundup
  925 database module also imports all the modules in this directory, and the
  926 ``init()`` function of each module is called when a database is opened
  927 to provide it a chance to register its detectors.
  928 
  929 There are two kinds of detectors:
  930 
  931 1. an `auditor` is triggered just before modifying an item
  932 2. a `reactor` is triggered just after an item has been modified
  933 
  934 When the Roundup database is about to perform a ``create()``, ``set()``,
  935 ``retire()``, or ``restore`` operation, it first calls any *auditors*
  936 that have been registered for that operation on that class. Any auditor
  937 may raise a *Reject* exception to abort the operation.
  938 
  939 If none of the auditors raises an exception, the database proceeds to
  940 carry out the operation.  After it's done, it then calls all of the
  941 *reactors* that have been registered for the operation.
  942 
  943 
  944 .. index:: detectors; interface specification
  945    single: detectors; register
  946    single: auditors; class registration method
  947    single: reactors; class registration method
  948    pair: class methods; audit
  949    pair: class methods; react
  950 
  951 Detector Interface Specification
  952 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  953 
  954 The ``audit()`` and ``react()`` methods register detectors on a given
  955 class of items::
  956 
  957     class Class:
  958         def audit(self, event, detector, priority=100):
  959             """Register an auditor on this class.
  960 
  961             'event' should be one of "create", "set", "retire", or
  962             "restore". 'detector' should be a function accepting four
  963             arguments. Detectors are called in priority order, execution
  964             order is undefined for detectors with the same priority.
  965             """
  966 
  967         def react(self, event, detector, priority=100):
  968             """Register a reactor on this class.
  969 
  970             'event' should be one of "create", "set", "retire", or
  971             "restore". 'detector' should be a function accepting four
  972             arguments. Detectors are called in priority order, execution
  973             order is undefined for detectors with the same priority.
  974             """
  975 
  976 Auditors are called with the arguments::
  977 
  978     audit(db, cl, itemid, newdata)
  979 
  980 where ``db`` is the database, ``cl`` is an instance of Class or
  981 IssueClass within the database, and ``newdata`` is a dictionary mapping
  982 property names to values.
  983 
  984 For a ``create()`` operation, the ``itemid`` argument is None and
  985 newdata contains all of the initial property values with which the item
  986 is about to be created.
  987 
  988 For a ``set()`` operation, newdata contains only the names and values of
  989 properties that are about to be changed.
  990 
  991 For a ``retire()`` or ``restore()`` operation, newdata is None.
  992 
  993 Reactors are called with the arguments::
  994 
  995     react(db, cl, itemid, olddata)
  996 
  997 where ``db`` is the database, ``cl`` is an instance of Class or
  998 IssueClass within the database, and ``olddata`` is a dictionary mapping
  999 property names to values.
 1000 
 1001 For a ``create()`` operation, the ``itemid`` argument is the id of the
 1002 newly-created item and ``olddata`` is None.
 1003 
 1004 For a ``set()`` operation, ``olddata`` contains the names and previous
 1005 values of properties that were changed.
 1006 
 1007 For a ``retire()`` or ``restore()`` operation, ``itemid`` is the id of
 1008 the retired or restored item and ``olddata`` is None.
 1009 
 1010 
 1011 Detector Example
 1012 ~~~~~~~~~~~~~~~~
 1013 
 1014 Here is an example of detectors written for a hypothetical
 1015 project-management application, where users can signal approval of a
 1016 project by adding themselves to an "approvals" list, and a project
 1017 proceeds when it has three approvals::
 1018 
 1019     # Permit users only to add themselves to the "approvals" list.
 1020 
 1021     def check_approvals(db, cl, id, newdata):
 1022         if "approvals" in newdata:
 1023             if cl.get(id, "status") == db.status.lookup("approved"):
 1024                 raise Reject("You can't modify the approvals list "
 1025                     "for a project that has already been approved.")
 1026             old = cl.get(id, "approvals")
 1027             new = newdata["approvals"]
 1028             for uid in old:
 1029                 if uid not in new and uid != db.getuid():
 1030                     raise Reject("You can't remove other users from "
 1031                         "the approvals list; you can only remove "
 1032                         "yourself.")
 1033             for uid in new:
 1034                 if uid not in old and uid != db.getuid():
 1035                     raise Reject("You can't add other users to the "
 1036                         "approvals list; you can only add yourself.")
 1037 
 1038     # When three people have approved a project, change its status from
 1039     # "pending" to "approved".
 1040 
 1041     def approve_project(db, cl, id, olddata):
 1042         if ("approvals" in olddata and 
 1043             len(cl.get(id, "approvals")) == 3):
 1044             if cl.get(id, "status") == db.status.lookup("pending"):
 1045                 cl.set(id, status=db.status.lookup("approved"))
 1046 
 1047     def init(db):
 1048         db.project.audit("set", check_approval)
 1049         db.project.react("set", approve_project)
 1050 
 1051 Here is another example of a detector that can allow or prevent the
 1052 creation of new items.  In this scenario, patches for a software project
 1053 are submitted by sending in e-mail with an attached file, and we want to
 1054 ensure that there are text/plain attachments on the message.  The
 1055 maintainer of the package can then apply the patch by setting its status
 1056 to "applied"::
 1057 
 1058     # Only accept attempts to create new patches that come with patch
 1059     # files.
 1060 
 1061     def check_new_patch(db, cl, id, newdata):
 1062         if not newdata["files"]:
 1063             raise Reject("You can't submit a new patch without "
 1064                          "attaching a patch file.")
 1065         for fileid in newdata["files"]:
 1066             if db.file.get(fileid, "type") != "text/plain":
 1067                 raise Reject("Submitted patch files must be "
 1068                              "text/plain.")
 1069 
 1070     # When the status is changed from "approved" to "applied", apply the
 1071     # patch.
 1072 
 1073     def apply_patch(db, cl, id, olddata):
 1074         if (cl.get(id, "status") == db.status.lookup("applied") and 
 1075             olddata["status"] == db.status.lookup("approved")):
 1076             # ...apply the patch...
 1077 
 1078     def init(db):
 1079         db.patch.audit("create", check_new_patch)
 1080         db.patch.react("set", apply_patch)
 1081 
 1082 
 1083 Command Interface
 1084 -----------------
 1085 
 1086 The command interface is a very simple and minimal interface, intended
 1087 only for quick searches and checks from the shell prompt. (Anything more
 1088 interesting can simply be written in Python using the Roundup database
 1089 module.)
 1090 
 1091 
 1092 .. index:: roundup-admin; command line specification
 1093 
 1094 Command Interface Specification
 1095 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 1096 
 1097 A single command, ``roundup-admin``, provides basic access to the hyperdatabase
 1098 from the command line::
 1099 
 1100     roundup-admin help
 1101     roundup-admin get [-list] designator[, designator,...] propname
 1102     roundup-admin set designator[, designator,...] propname=value ...
 1103     roundup-admin find [-list] classname propname=value ...
 1104 
 1105 See ``roundup-admin help commands`` for a complete list of commands.
 1106 
 1107 Property values are represented as strings in command arguments and in
 1108 the printed results:
 1109 
 1110 - Strings are, well, strings.
 1111 
 1112 - Integers/Numbers are displayed the same as strings.
 1113 
 1114 - Booleans are displayed as 'Yes' or 'No'.
 1115 
 1116 - Date values are printed in the full date format in the local time
 1117   zone, and accepted in the full format or any of the partial formats
 1118   explained above.
 1119 
 1120 - Link values are printed as item designators.  When given as an
 1121   argument, item designators and key strings are both accepted.
 1122 
 1123 - Multilink values are printed as lists of item designators joined by
 1124   commas.  When given as an argument, item designators and key strings
 1125   are both accepted; an empty string, a single item, or a list of items
 1126   joined by commas is accepted.
 1127 
 1128 When multiple items are specified to the roundup-admin get or roundup-admin set
 1129 commands, the specified properties are retrieved or set on all the
 1130 listed items.
 1131 
 1132 When multiple results are returned by the roundup-admin get or
 1133 roundup-admin find
 1134 commands, they are printed one per line (default) or joined by commas
 1135 (with the -list) option.
 1136 
 1137 
 1138 .. index:: roundup-admin; usage in scripts
 1139 
 1140 Usage Example
 1141 ~~~~~~~~~~~~~
 1142 
 1143 To find all messages regarding in-progress issues that contain the word
 1144 "spam", for example, you could execute the following command from the
 1145 directory where the database dumps its files::
 1146 
 1147     shell% for issue in `roundup-admin find issue status=in-progress`; do
 1148     > grep -l spam `roundup-admin get $issue messages`
 1149     > done
 1150     msg23
 1151     msg49
 1152     msg50
 1153     msg61
 1154     shell%
 1155 
 1156 Or, using the -list option, this can be written as a single command::
 1157 
 1158     shell% grep -l spam `roundup-admin get \
 1159         \`roundup-admin find -list issue status=in-progress\` messages`
 1160     msg23
 1161     msg49
 1162     msg50
 1163     msg61
 1164     shell%
 1165     
 1166 
 1167 E-mail User Interface
 1168 ---------------------
 1169 
 1170 The Roundup system must be assigned an e-mail address at which to
 1171 receive mail.  Messages should be piped to the Roundup mail-handling
 1172 script by the mail delivery system (e.g. using an alias beginning with
 1173 "|" for sendmail).
 1174 
 1175 
 1176 Message Processing
 1177 ~~~~~~~~~~~~~~~~~~
 1178 
 1179 Incoming messages are examined for multiple parts. In a multipart/mixed
 1180 message or part, each subpart is extracted and examined.  In a
 1181 multipart/alternative message or part, we look for a text/plain subpart
 1182 and ignore the other parts.  The text/plain subparts are assembled to
 1183 form the textual body of the message, to be stored in the file
 1184 associated with a "msg" class item. Any parts of other types are each
 1185 stored in separate files and given "file" class items that are linked to
 1186 the "msg" item.
 1187 
 1188 The "summary" property on message items is taken from the first
 1189 non-quoting section in the message body. The message body is divided
 1190 into sections by blank lines. Sections where the second and all
 1191 subsequent lines begin with a ">" or "|" character are considered
 1192 "quoting sections".  The first line of the first non-quoting section
 1193 becomes the summary of the message.
 1194 
 1195 All of the addresses in the To: and Cc: headers of the incoming message
 1196 are looked up among the user items, and the corresponding users are
 1197 placed in the "recipients" property on the new "msg" item.  The address
 1198 in the From: header similarly determines the "author" property of the
 1199 new "msg" item. The default handling for addresses that don't have
 1200 corresponding users is to create new users with no passwords and a
 1201 username equal to the address.  (The web interface does not permit
 1202 logins for users with no passwords.)  If we prefer to reject mail from
 1203 outside sources, we can simply register an auditor on the "user" class
 1204 that prevents the creation of user items with no passwords.
 1205 
 1206 The subject line of the incoming message is examined to determine
 1207 whether the message is an attempt to create a new issue or to discuss an
 1208 existing issue.  A designator enclosed in square brackets is sought as
 1209 the first thing on the subject line (after skipping any "Fwd:" or "Re:"
 1210 prefixes).
 1211 
 1212 If an issue designator (class name and id number) is found there, the
 1213 newly created "msg" item is added to the "messages" property for that
 1214 issue, and any new "file" items are added to the "files" property for
 1215 the issue.
 1216 
 1217 If just an issue class name is found there, we attempt to create a new
 1218 issue of that class with its "messages" property initialized to contain
 1219 the new "msg" item and its "files" property initialized to contain any
 1220 new "file" items.
 1221 
 1222 Both cases may trigger detectors (in the first case we are calling the
 1223 set() method to add the message to the issue's spool; in the second case
 1224 we are calling the create() method to create a new item).  If an auditor
 1225 raises an exception, the original message is bounced back to the sender
 1226 with the explanatory message given in the exception.
 1227 
 1228 
 1229 Nosy Lists
 1230 ~~~~~~~~~~
 1231 
 1232 A standard detector is provided that watches for additions to the
 1233 "messages" property.  When a new message is added, the detector sends it
 1234 to all the users on the "nosy" list for the issue that are not already
 1235 on the "recipients" list of the message.  Those users are then appended
 1236 to the "recipients" property on the message, so multiple copies of a
 1237 message are never sent to the same user.  The journal recorded by the
 1238 hyperdatabase on the "recipients" property then provides a log of when
 1239 the message was sent to whom.
 1240 
 1241 
 1242 Setting Properties
 1243 ~~~~~~~~~~~~~~~~~~
 1244 
 1245 The e-mail interface also provides a simple way to set properties on
 1246 issues.  At the end of the subject line, ``propname=value`` pairs can be
 1247 specified in square brackets, using the same conventions as for the
 1248 roundup-admin ``set`` shell command.
 1249 
 1250 
 1251 Web User Interface
 1252 ------------------
 1253 
 1254 The web interface is provided by a CGI script that can be run under any
 1255 web server.  A simple web server can easily be built on the standard
 1256 CGIHTTPServer module, and should also be included in the distribution
 1257 for quick out-of-the-box deployment.
 1258 
 1259 The user interface is constructed from a number of template files
 1260 containing mostly HTML.  Among the HTML tags in templates are
 1261 interspersed some nonstandard tags, which we use as placeholders to be
 1262 replaced by properties and their values.
 1263 
 1264 
 1265 Views and View Specifiers
 1266 ~~~~~~~~~~~~~~~~~~~~~~~~~
 1267 
 1268 There are two main kinds of views: *index* views and *issue* views. An
 1269 index view displays a list of issues of a particular class, optionally
 1270 sorted and filtered as requested.  An issue view presents the properties
 1271 of a particular issue for editing and displays the message spool for the
 1272 issue.
 1273 
 1274 A view specifier is a string that specifies all the options needed to
 1275 construct a particular view. It goes after the URL to the Roundup CGI
 1276 script or the web server to form the complete URL to a view.  When the
 1277 result of selecting a link or submitting a form takes the user to a new
 1278 view, the Web browser should be redirected to a canonical location
 1279 containing a complete view specifier so that the view can be bookmarked.
 1280 
 1281 
 1282 Displaying Properties
 1283 ~~~~~~~~~~~~~~~~~~~~~
 1284 
 1285 Properties appear in the user interface in three contexts: in indices,
 1286 in editors, and as search filters.  For each type of property, there are
 1287 several display possibilities.  For example, in an index view, a string
 1288 property may just be printed as a plain string, but in an editor view,
 1289 that property should be displayed in an editable field.
 1290 
 1291 The display of a property is handled by functions in the
 1292 ``cgi.templating`` module.
 1293 
 1294 Displayer functions are triggered by ``tal:content`` or ``tal:replace``
 1295 tag attributes in templates.  The value of the attribute provides an
 1296 expression for calling the displayer function. For example, the
 1297 occurrence of::
 1298 
 1299     tal:content="context/status/plain"
 1300 
 1301 in a template triggers a call to::
 1302     
 1303     context['status'].plain()
 1304 
 1305 where the context would be an item of the "issue" class.  The displayer
 1306 functions can accept extra arguments to further specify details about
 1307 the widgets that should be generated.
 1308 
 1309 Some of the standard displayer functions include:
 1310 
 1311 ========= ==============================================================
 1312 Function  Description
 1313 ========= ==============================================================
 1314 plain     display a String property directly;
 1315           display a Date property in a specified time zone with an
 1316           option to omit the time from the date stamp; for a Link or
 1317           Multilink property, display the key strings of the linked
 1318           items (or the ids if the linked class has no key property)
 1319 field     display a property like the plain displayer above, but in a
 1320           text field to be edited
 1321 menu      for a Link property, display a menu of the available choices
 1322 ========= ==============================================================
 1323 
 1324 See the `customisation`_ documentation for the complete list.
 1325 
 1326 
 1327 Index Views
 1328 ~~~~~~~~~~~
 1329 
 1330 An index view contains two sections: a filter section and an index
 1331 section. The filter section provides some widgets for selecting which
 1332 issues appear in the index.  The index section is a table of issues.
 1333 
 1334 
 1335 Index View Specifiers
 1336 """""""""""""""""""""
 1337 
 1338 An index view specifier looks like this (whitespace has been added for
 1339 clarity)::
 1340 
 1341     /issue?status=unread,in-progress,resolved&
 1342         keyword=security,ui&
 1343         :group=priority,-status&
 1344         :sort=-activity&
 1345         :filters=status,keyword&
 1346         :columns=title,status,fixer
 1347 
 1348 
 1349 The index view is determined by two parts of the specifier: the layout
 1350 part and the filter part. The layout part consists of the query
 1351 parameters that begin with colons, and it determines the way that the
 1352 properties of selected items are displayed. The filter part consists of
 1353 all the other query parameters, and it determines the criteria by which
 1354 items are selected for display.
 1355 
 1356 The filter part is interactively manipulated with the form widgets
 1357 displayed in the filter section.  The layout part is interactively
 1358 manipulated by clicking on the column headings in the table.
 1359 
 1360 The filter part selects the union of the sets of issues with values
 1361 matching any specified Link properties and the intersection of the sets
 1362 of issues with values matching any specified Multilink properties.
 1363 
 1364 The example specifies an index of "issue" items. Only issues with a
 1365 "status" of either "unread" or "in-progres" or "resolved" are displayed,
 1366 and only issues with "keyword" values including both "security" and "ui"
 1367 are displayed.  The items are grouped by priority arranged in ascending
 1368 order and in descending order by status; and within groups, sorted by
 1369 activity, arranged in descending order. The filter section shows
 1370 filters for the "status" and "keyword" properties, and the table includes
 1371 columns for the "title", "status", and "fixer" properties.
 1372 
 1373 Associated with each issue class is a default layout specifier.  The
 1374 layout specifier in the above example is the default layout to be
 1375 provided with the default bug-tracker schema described above in section
 1376 4.4.
 1377 
 1378 Index Section
 1379 """""""""""""
 1380 
 1381 The template for an index section describes one row of the index table.
 1382 Fragments protected by a ``tal:condition="request/show/<property>"`` are
 1383 included or omitted depending on whether the view specifier requests a
 1384 column for a particular property. The table cells are filled by the
 1385 ``tal:content="context/<property>"`` directive, which displays the value
 1386 of the property.
 1387 
 1388 Here's a simple example of an index template::
 1389 
 1390     <tr>
 1391       <td tal:condition="request/show/title"
 1392           tal:content="contex/title"></td>
 1393       <td tal:condition="request/show/status"
 1394           tal:content="contex/status"></td>
 1395       <td tal:condition="request/show/fixer"
 1396           tal:content="contex/fixer"></td>
 1397     </tr>
 1398 
 1399 Sorting
 1400 """""""
 1401 
 1402 String and Date values are sorted in the natural way. Link properties
 1403 are sorted according to the value of the "order" property on the linked
 1404 items if it is present; or otherwise on the key string of the linked
 1405 items; or finally on the item ids.  Multilink properties are sorted
 1406 according to how many links are present.
 1407 
 1408 Issue Views
 1409 ~~~~~~~~~~~
 1410 
 1411 An issue view contains an editor section and a spool section. At the top
 1412 of an issue view, links to superseding and superseded issues are always
 1413 displayed.
 1414 
 1415 Issue View Specifiers
 1416 """""""""""""""""""""
 1417 
 1418 An issue view specifier is simply the issue's designator::
 1419 
 1420     /patch23
 1421 
 1422 
 1423 Editor Section
 1424 """"""""""""""
 1425 
 1426 The editor section is generated from a template containing
 1427 ``tal:content="context/<property>/<widget>"`` directives to insert the
 1428 appropriate widgets for editing properties.
 1429 
 1430 Here's an example of a basic editor template::
 1431 
 1432     <table>
 1433     <tr>
 1434         <td colspan=2
 1435             tal:content="python:context.title.field(size='60')"></td>
 1436     </tr>
 1437     <tr>
 1438         <td tal:content="context/fixer/field"></td>
 1439         <td tal:content="context/status/menu"></td>
 1440     </tr>
 1441     <tr>
 1442         <td tal:content="context/nosy/field"></td>
 1443         <td tal:content="context/priority/menu"></td>
 1444     </tr>
 1445     <tr>
 1446         <td colspan=2>
 1447           <textarea name=":note" rows=5 cols=60></textarea>
 1448         </td>
 1449     </tr>
 1450     </table>
 1451 
 1452 As shown in the example, the editor template can also include a ":note"
 1453 field, which is a text area for entering a note to go along with a
 1454 change.
 1455 
 1456 When a change is submitted, the system automatically generates a message
 1457 describing the changed properties. The message displays all of the
 1458 property values on the issue and indicates which ones have changed. An
 1459 example of such a message might be this::
 1460 
 1461     title: Polly Parrot is dead
 1462     priority: critical
 1463     status: unread -> in-progress
 1464     fixer: (none)
 1465     keywords: parrot,plumage,perch,nailed,dead
 1466 
 1467 If a note is given in the ":note" field, the note is appended to the
 1468 description.  The message is then added to the issue's message spool
 1469 (thus triggering the standard detector to react by sending out this
 1470 message to the nosy list).
 1471 
 1472 
 1473 Spool Section
 1474 """""""""""""
 1475 
 1476 The spool section lists messages in the issue's "messages" property.
 1477 The index of messages displays the "date", "author", and "summary"
 1478 properties on the message items, and selecting a message takes you to
 1479 its content.
 1480 
 1481 Access Control
 1482 --------------
 1483 
 1484 At each point that requires an action to be performed, the security
 1485 mechanisms are asked if the current user has permission. This permission
 1486 is defined as a Permission.
 1487 
 1488 Individual assignment of Permission to user is unwieldy. The concept of
 1489 a Role, which encompasses several Permissions and may be assigned to
 1490 many Users, is quite well developed in many projects. Roundup will take
 1491 this path, and allow the multiple assignment of Roles to Users, and
 1492 multiple Permissions to Roles. These definitions are not persistent -
 1493 they're defined when the application initialises.
 1494 
 1495 There will be three levels of Permission. The Class level permissions
 1496 define logical permissions associated with all items of a particular
 1497 class (or all classes). The Item level permissions define logical
 1498 permissions associated with specific items by way of their user-linked
 1499 properties. The Property level permissions define logical permissions
 1500 associated with a specific property of an item.
 1501 
 1502 
 1503 Access Control Interface Specification
 1504 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 1505 
 1506 The security module defines::
 1507 
 1508     class Permission:
 1509         ''' Defines a Permission with the attributes
 1510             - name
 1511             - description
 1512             - klass (optional)
 1513             - properties (optional)
 1514             - check function (optional)
 1515 
 1516             The klass may be unset, indicating that this permission is
 1517             not locked to a particular hyperdb class. There may be
 1518             multiple Permissions for the same name for different
 1519             classes.
 1520 
 1521             If property names are set, permission is restricted to those
 1522             properties only.
 1523 
 1524             If check function is set, permission is granted only when
 1525             the function returns value interpreted as boolean true.
 1526             The function is called with arguments db, userid, itemid.
 1527         '''
 1528 
 1529     class Role:
 1530         ''' Defines a Role with the attributes
 1531             - name
 1532             - description
 1533             - permissions
 1534         '''
 1535 
 1536     class Security:
 1537         def __init__(self, db):
 1538             ''' Initialise the permission and role stores, and add in
 1539                 the base roles (for admin user).
 1540             '''
 1541 
 1542         def getPermission(self, permission, classname=None, properties=None,
 1543                 check=None):
 1544             ''' Find the Permission exactly matching the name, class,
 1545                 properties list and check function.
 1546 
 1547                 Raise ValueError if there is no exact match.
 1548             '''
 1549 
 1550         def hasPermission(self, permission, userid, classname=None,
 1551                 property=None, itemid=None):
 1552             ''' Look through all the Roles, and hence Permissions, and
 1553                 see if "permission" exists given the constraints of
 1554                 classname, property and itemid.
 1555 
 1556                 If classname is specified (and only classname) then the
 1557                 search will match if there is *any* Permission for that
 1558                 classname, even if the Permission has additional
 1559                 constraints.
 1560 
 1561                 If property is specified, the Permission matched must have
 1562                 either no properties listed or the property must appear in
 1563                 the list.
 1564 
 1565                 If itemid is specified, the Permission matched must have
 1566                 either no check function defined or the check function,
 1567                 when invoked, must return a True value.
 1568 
 1569                 Note that this functionality is actually implemented by the
 1570                 Permission.test() method.
 1571             '''
 1572 
 1573         def addPermission(self, **propspec):
 1574             ''' Create a new Permission with the properties defined in
 1575                 'propspec'. See the Permission class for the possible
 1576                 keyword args.
 1577             '''
 1578 
 1579         def addRole(self, **propspec):
 1580             ''' Create a new Role with the properties defined in
 1581                 'propspec'
 1582             '''
 1583 
 1584         def addPermissionToRole(self, rolename, permission):
 1585             ''' Add the permission to the role's permission list.
 1586 
 1587                 'rolename' is the name of the role to add permission to.
 1588             '''
 1589 
 1590 Modules such as ``cgi/client.py`` and ``mailgw.py`` define their own
 1591 permissions like so (this example is ``cgi/client.py``)::
 1592 
 1593     def initialiseSecurity(security):
 1594         ''' Create some Permissions and Roles on the security object
 1595 
 1596             This function is directly invoked by
 1597             security.Security.__init__() as a part of the Security
 1598             object instantiation.
 1599         '''
 1600         p = security.addPermission(name="Web Registration",
 1601             description="Anonymous users may register through the web")
 1602         security.addToRole('Anonymous', p)
 1603 
 1604 Detectors may also define roles in their init() function::
 1605 
 1606     def init(db):
 1607         # register an auditor that checks that a user has the "May
 1608         # Resolve" Permission before allowing them to set an issue
 1609         # status to "resolved"
 1610         db.issue.audit('set', checkresolvedok)
 1611         p = db.security.addPermission(name="May Resolve", klass="issue")
 1612         security.addToRole('Manager', p)
 1613 
 1614 The tracker dbinit module then has in ``open()``::
 1615 
 1616     # open the database - it must be modified to init the Security class
 1617     # from security.py as db.security
 1618     db = Database(config, name)
 1619 
 1620     # add some extra permissions and associate them with roles
 1621     ei = db.security.addPermission(name="Edit", klass="issue",
 1622                     description="User is allowed to edit issues")
 1623     db.security.addPermissionToRole('User', ei)
 1624     ai = db.security.addPermission(name="View", klass="issue",
 1625                     description="User is allowed to access issues")
 1626     db.security.addPermissionToRole('User', ai)
 1627 
 1628 In the dbinit ``init()``::
 1629 
 1630     # create the two default users
 1631     user.create(username="admin", password=Password(adminpw),
 1632                 address=config.ADMIN_EMAIL, roles='Admin')
 1633     user.create(username="anonymous", roles='Anonymous')
 1634 
 1635 Then in the code that matters, calls to ``hasPermission`` and
 1636 ``hasItemPermission`` are made to determine if the user has permission
 1637 to perform some action::
 1638 
 1639     if db.security.hasPermission('issue', 'Edit', userid):
 1640         # all ok
 1641 
 1642     if db.security.hasItemPermission('issue', itemid,
 1643                                      assignedto=userid):
 1644         # all ok
 1645 
 1646 Code in the core will make use of these methods, as should code in
 1647 auditors in custom templates. The HTML templating may access the access
 1648 controls through the *user* attribute of the *request* variable. It
 1649 exposes a ``hasPermission()`` method::
 1650 
 1651   tal:condition="python:request.user.hasPermission('Edit', 'issue')"
 1652 
 1653 or, if the *context* is *issue*, then the following is the same::
 1654 
 1655   tal:condition="python:request.user.hasPermission('Edit')"
 1656 
 1657 
 1658 Authentication of Users
 1659 ~~~~~~~~~~~~~~~~~~~~~~~
 1660 
 1661 Users must be authenticated correctly for the above controls to work.
 1662 This is not done in the current mail gateway at all. Use of digital
 1663 signing of messages could alleviate this problem.
 1664 
 1665 The exact mechanism of registering the digital signature should be
 1666 flexible, with perhaps a level of trust. Users who supply their
 1667 signature through their first message into the tracker should be at a
 1668 lower level of trust to those who supply their signature to an admin for
 1669 submission to their user details.
 1670 
 1671 
 1672 Anonymous Users
 1673 ~~~~~~~~~~~~~~~
 1674 
 1675 The "anonymous" user must always exist, and defines the access
 1676 permissions for anonymous users. Unknown users accessing Roundup through
 1677 the web or email interfaces will be logged in as the "anonymous" user.
 1678 
 1679 
 1680 Use Cases
 1681 ~~~~~~~~~
 1682 
 1683 public - end users can submit bugs, request new features, request
 1684     support
 1685     The Users would be given the default "User" Role which gives "View"
 1686     and "Edit" Permission to the "issue" class.
 1687 developer - developers can fix bugs, implement new features, provide
 1688     support
 1689     A new Role "Developer" is created with the Permission "Fixer" which
 1690     is checked for in custom auditors that see whether the issue is
 1691     being resolved with a particular resolution ("fixed", "implemented",
 1692     "supported") and allows that resolution only if the permission is
 1693     available.
 1694 manager - approvers/managers can approve new features and signoff bug
 1695     fixes
 1696     A new Role "Manager" is created with the Permission "Signoff" which
 1697     is checked for in custom auditors that see whether the issue status
 1698     is being changed similar to the developer example. admin -
 1699     administrators can add users and set user's roles The existing Role
 1700     "Admin" has the Permissions "Edit" for all classes (including
 1701     "user") and "Web Roles" which allow the desired actions.
 1702 system - automated request handlers running various report/escalation
 1703     scripts
 1704     A combination of existing and new Roles, Permissions and auditors
 1705     could be used here.
 1706 privacy - issues that are only visible to some users
 1707     A new property is added to the issue which marks the user or group
 1708     of users who are allowed to view and edit the issue. An auditor will
 1709     check for edit access, and the template user object can check for
 1710     view access.
 1711 
 1712 
 1713 Deployment Scenarios
 1714 --------------------
 1715 
 1716 The design described above should be general enough to permit the use of
 1717 Roundup for bug tracking, managing projects, managing patches, or
 1718 holding discussions.  By using items of multiple types, one could deploy
 1719 a system that maintains requirement specifications, catalogs bugs, and
 1720 manages submitted patches, where patches could be linked to the bugs and
 1721 requirements they address.
 1722 
 1723 
 1724 Acknowledgements
 1725 ----------------
 1726 
 1727 My thanks are due to Christy Heyl for reviewing and contributing
 1728 suggestions to this paper and motivating me to get it done, and to Jesse
 1729 Vincent, Mark Miller, Christopher Simons, Jeff Dunmall, Wayne Gramlich,
 1730 and Dean Tribble for their assistance with the first-round submission.
 1731 
 1732 
 1733 Changes to this document
 1734 ------------------------
 1735 
 1736 - Added docs for quiet, default_value and required arguments for properties.
 1737 - Added Boolean, Integer and Number types
 1738 - Added section Hyperdatabase Implementations
 1739 - "Item" has been renamed to "Issue" to account for the more specific
 1740   nature of the Class.
 1741 - New Templating
 1742 - Access Controls
 1743 - Added "actor" property
 1744 
 1745 .. _customisation: customizing.html
 1746