"Fossies" - the Fresh Open Source Software Archive

Member "roundup-2.0.0/doc/customizing.txt" (6 Jul 2020, 227461 Bytes) of package /linux/www/roundup-2.0.0.tar.gz:


The requested HTML page contains a <FORM> tag that is unusable on "Fossies" in "automatic" (rendered) mode so that page is shown as HTML source code (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. See also the latest Fossies "Diffs" side-by-side code changes report for "customizing.txt": 1.6.1_vs_2.0.0.

    1 :tocdepth: 2
    2 
    3 ===================
    4 Customising Roundup
    5 ===================
    6 
    7 .. This document borrows from the ZopeBook section on ZPT. The original is at:
    8    http://www.zope.org/Documentation/Books/ZopeBook/current/ZPT.stx
    9 
   10 .. contents::
   11    :depth: 2
   12    :local:
   13 
   14 What You Can Do
   15 ===============
   16 
   17 Before you get too far, it's probably worth having a quick read of the Roundup
   18 `design documentation`_.
   19 
   20 Customisation of Roundup can take one of six forms:
   21 
   22 1. `tracker configuration`_ changes
   23 2. database, or `tracker schema`_ changes
   24 3. "definition" class `database content`_ changes
   25 4. behavioural changes through detectors_, extensions_ and interfaces.py_
   26 5. `security / access controls`_
   27 6. change the `web interface`_
   28 
   29 The third case is special because it takes two distinctly different forms
   30 depending upon whether the tracker has been initialised or not. The other two
   31 may be done at any time, before or after tracker initialisation. Yes, this
   32 includes adding or removing properties from classes.
   33 
   34 
   35 Trackers in a Nutshell
   36 ======================
   37 
   38 Trackers have the following structure:
   39 
   40 .. index::
   41    single: tracker; structure db directory
   42    single: tracker; structure detectors directory
   43    single: tracker; structure extensions directory
   44    single: tracker; structure html directory
   45    single: tracker; structure html directory
   46    single: tracker; structure lib directory
   47 
   48 =================== ========================================================
   49 Tracker File        Description
   50 =================== ========================================================
   51 config.ini          Holds the basic `tracker configuration`_                 
   52 schema.py           Holds the `tracker schema`_                              
   53 initial_data.py     Holds any data to be entered into the database when the
   54                     tracker is initialised.
   55 db/                 Holds the tracker's database                             
   56 db/files/           Holds the tracker's upload files and messages            
   57 db/backend_name     Names the database back-end for the tracker (obsolete).
   58                     Current way uses the ``backend`` setting in the rdbms
   59 		    section of config.ini.
   60 detectors/          Auditors and reactors for this tracker                   
   61 extensions/         Additional actions and `templating utilities`_
   62 html/               Web interface templates, images and style sheets         
   63 lib/                optional common imports for detectors and extensions
   64 =================== ======================================================== 
   65 
   66 
   67 .. index:: config.ini
   68 .. index:: configuration; see config.ini 
   69 
   70 Tracker Configuration
   71 =====================
   72 
   73 The ``config.ini`` located in your tracker home contains the basic
   74 configuration for the web and e-mail components of roundup's interfaces.
   75 
   76 Changes to the data captured by your tracker is controlled by the `tracker
   77 schema`_.  Some configuration is also performed using permissions - see the 
   78 `security / access controls`_ section. For example, to allow users to
   79 automatically register through the email interface, you must grant the
   80 "Anonymous" Role the "Email Access" Permission.
   81 
   82 .. index::
   83    single: config.ini; sections
   84    see: configuration; config.ini
   85 
   86 The following is taken from the `Python Library Reference`__ (July 18, 2018)
   87 section "ConfigParser -- Configuration file parser":
   88 
   89  The configuration file consists of sections, led by a [section] header
   90  and followed by name: value entries, with continuations in the style
   91  of RFC 822 (see section 3.1.1, “LONG HEADER FIELDS”); name=value is
   92  also accepted. Note that leading whitespace is removed from
   93  values. The optional values can contain format strings which refer to
   94  other values in the same section, or values in a special DEFAULT
   95  section. Additional defaults can be provided on initialization and
   96  retrieval. Lines beginning with '#' or ';' are ignored and may be
   97  used to provide comments.
   98 
   99  For example::
  100 
  101    [My Section]
  102    foodir = %(dir)s/whatever
  103    dir = frob
  104 
  105  would resolve the "%(dir)s" to the value of "dir" ("frob" in this case)
  106  resulting in "foodir" being "frob/whatever".
  107 
  108 __ https://docs.python.org/2/library/configparser.html
  109 
  110 Example configuration settings are below. This is a partial
  111 list. Documentation on all the settings is included in the
  112 ``config.ini`` file.
  113  
  114 .. index:: config.ini; sections main
  115 
  116 Section **main**
  117  database -- ``db``
  118   Database directory path. The path may be either absolute or relative
  119   to the directory containig this config file.
  120 
  121  templates -- ``html``
  122   Path to the HTML templates directory. The path may be either absolute
  123   or relative to the directory containing this config file.
  124 
  125  static_files -- default *blank*
  126   A list of space separated directory paths (or a single directory).
  127   These directories hold additional static files available via Web UI.
  128   These directories may contain sitewide images, CSS stylesheets etc. If
  129   a '-' is included, the list processing ends and the TEMPLATES
  130   directory is not searched after the specified directories.  If this
  131   option is not set, all static files are taken from the TEMPLATES
  132   directory.
  133 
  134  admin_email -- ``roundup-admin``
  135   Email address that roundup will complain to if it runs into trouble. If
  136   the email address doesn't contain an ``@`` part, the MAIL_DOMAIN defined
  137   below is used.
  138 
  139  dispatcher_email -- ``roundup-admin``
  140   The 'dispatcher' is a role that can get notified of new items to the
  141   database. It is used by the ERROR_MESSAGES_TO config setting. If the
  142   email address doesn't contain an ``@`` part, the MAIL_DOMAIN defined
  143   below is used.
  144 
  145  email_from_tag -- default *blank*
  146   Additional text to include in the "name" part of the From: address used
  147   in nosy messages. If the sending user is "Foo Bar", the From: line
  148   is usually: ``"Foo Bar" <issue_tracker@tracker.example>``
  149   the EMAIL_FROM_TAG goes inside the "Foo Bar" quotes like so:
  150   ``"Foo Bar EMAIL_FROM_TAG" <issue_tracker@tracker.example>``
  151 
  152  new_web_user_roles -- ``User``
  153   Roles that a user gets when they register with Web User Interface.
  154   This is a comma-separated list of role names (e.g. ``Admin,User``).
  155 
  156  new_email_user_roles -- ``User``
  157   Roles that a user gets when they register with Email Gateway.
  158   This is a comma-separated string of role names (e.g. ``Admin,User``).
  159 
  160  error_messages_to -- ``user``
  161   Send error message emails to the ``dispatcher``, ``user``, or ``both``?
  162   The dispatcher is configured using the DISPATCHER_EMAIL setting.
  163   Allowed values: ``dispatcher``, ``user``, or ``both``
  164 
  165  html_version -- ``html4``
  166   HTML version to generate. The templates are ``html4`` by default.
  167   If you wish to make them xhtml, then you'll need to change this
  168   var to ``xhtml`` too so all auto-generated HTML is compliant.
  169   Allowed values: ``html4``, ``xhtml``
  170 
  171  timezone -- ``0``
  172   Numeric timezone offset used when users do not choose their own
  173   in their settings.
  174 
  175  instant_registration -- ``yes``
  176   Register new users instantly, or require confirmation via
  177   email?
  178   Allowed values: ``yes``, ``no``
  179 
  180  email_registration_confirmation -- ``yes``
  181   Offer registration confirmation by email or only through the web?
  182   Allowed values: ``yes``, ``no``
  183 
  184  indexer_stopwords -- default *blank*
  185   Additional stop-words for the full-text indexer specific to
  186   your tracker. See the indexer source for the default list of
  187   stop-words (e.g. ``A,AND,ARE,AS,AT,BE,BUT,BY, ...``).
  188 
  189  umask -- ``02``
  190   Defines the file creation mode mask.
  191 
  192  csv_field_size -- ``131072``
  193   Maximum size of a csv-field during import. Roundups export
  194   format is a csv (comma separated values) variant. The csv
  195   reader has a limit on the size of individual fields
  196   starting with python 2.5. Set this to a higher value if you
  197   get the error 'Error: field larger than field limit' during
  198   import.
  199 
  200 .. index:: config.ini; sections tracker
  201 
  202 Section **tracker**
  203  name -- ``Roundup issue tracker``
  204   A descriptive name for your roundup instance.
  205 
  206  web -- ``http://host.example/demo/``
  207   The web address that the tracker is viewable at.
  208   This will be included in information sent to users of the tracker.
  209   The URL MUST include the cgi-bin part or anything else
  210   that is required to get to the home page of the tracker.
  211   You MUST include a trailing '/' in the URL.
  212 
  213  email -- ``issue_tracker``
  214   Email address that mail to roundup should go to.
  215 
  216  language -- default *blank*
  217   Default locale name for this tracker. If this option is not set, the
  218   language is determined by the environment variable LANGUAGE, LC_ALL,
  219   LC_MESSAGES, or LANG, in that order of preference.
  220 
  221 .. index:: config.ini; sections web
  222 
  223 Section **web**
  224  allow_html_file -- ``no``
  225   Setting this option enables Roundup to serve uploaded HTML
  226   file content *as HTML*. This is a potential security risk
  227   and is therefore disabled by default. Set to 'yes' if you
  228   trust *all* users uploading content to your tracker.
  229 
  230  http_auth -- ``yes``
  231   Whether to use HTTP Basic Authentication, if present.
  232   Roundup will use either the REMOTE_USER or HTTP_AUTHORIZATION
  233   variables supplied by your web server (in that order).
  234   Set this option to 'no' if you do not wish to use HTTP Basic
  235   Authentication in your web interface.
  236 
  237  use_browser_language -- ``yes``
  238   Whether to use HTTP Accept-Language, if present.
  239   Browsers send a language-region preference list.
  240   It's usually set in the client's browser or in their
  241   Operating System.
  242   Set this option to 'no' if you want to ignore it.
  243 
  244  debug -- ``no``
  245   Setting this option makes Roundup display error tracebacks
  246   in the user's browser rather than emailing them to the
  247   tracker admin."),
  248 
  249 .. index:: config.ini; sections rdbms
  250    single: config.ini; database settings
  251 
  252 Section **rdbms**
  253  Settings in this section are used to set the backend and configure
  254  addition settings needed by RDBMs like SQLite, Postgresql and
  255  MySQL backends.
  256 
  257  .. index::
  258     single: postgres; select backend in config.ini 
  259     single: mysql; select backend in config.ini
  260     single: sqlite; select backend in config.ini
  261     single: anydbm; select backend in config.ini
  262     see: database; postgres
  263     see: database; mysql
  264     see: database; sqlite
  265     see: database; anydbm
  266 
  267  backend -- set to value by init
  268   The database backend such as anydbm, sqlite, mysql or postgres.
  269 
  270  name -- ``roundup``
  271   Name of the database to use.
  272 
  273  host -- ``localhost``
  274   Database server host.
  275 
  276  port -- default *blank*
  277   TCP port number of the database server. Postgresql usually resides on
  278   port 5432 (if any), for MySQL default port number is 3306. Leave this
  279   option empty to use backend default.
  280 
  281  user -- ``roundup``
  282   Database user name that Roundup should use.
  283 
  284  password -- ``roundup``
  285   Database user password.
  286 
  287  read_default_file -- ``~/.my.cnf``
  288   Name of the MySQL defaults file. Only used in MySQL connections.
  289 
  290  read_default_group -- ``roundup``
  291   Name of the group to use in the MySQL defaults file. Only used in
  292   MySQL connections.
  293 
  294  .. index::
  295     single: sqlite; lock timeout
  296 
  297  sqlite_timeout -- ``30``
  298   Number of seconds to wait when the SQLite database is locked.
  299   Used only for SQLite.
  300 
  301  cache_size -- `100`
  302   Size of the node cache (in elements) used to keep most recently used
  303   data in memory.
  304 
  305 .. index:: config.ini; sections logging
  306    see: logging; config.ini, sections logging
  307 
  308 Section **logging**
  309  config -- default *blank*
  310   Path to configuration file for standard Python logging module. If this
  311   option is set, logging configuration is loaded from specified file;
  312   options 'filename' and 'level' in this section are ignored. The path may
  313   be either absolute or relative to the directory containig this config file.
  314 
  315  filename -- default *blank*
  316   Log file name for minimal logging facility built into Roundup.  If no file
  317   name specified, log messages are written on stderr. If above 'config'
  318   option is set, this option has no effect. The path may be either absolute
  319   or relative to the directory containig this config file.
  320 
  321  level -- ``ERROR``
  322   Minimal severity level of messages written to log file. If above 'config'
  323   option is set, this option has no effect.
  324   Allowed values: ``DEBUG``, ``INFO``, ``WARNING``, ``ERROR``
  325 
  326 .. index:: config.ini; sections mail
  327 
  328 Section **mail**
  329  Outgoing email options. Used for nosy messages, password reset and
  330  registration approval requests.
  331 
  332  domain -- ``localhost``
  333   Domain name used for email addresses.
  334 
  335  host -- default *blank*
  336   SMTP mail host that roundup will use to send mail
  337 
  338  username -- default *blank*
  339   SMTP login name. Set this if your mail host requires authenticated access.
  340   If username is not empty, password (below) MUST be set!
  341 
  342  password -- default *blank*
  343   SMTP login password.
  344   Set this if your mail host requires authenticated access.
  345 
  346  port -- default *25*
  347   SMTP port on mail host.
  348   Set this if your mail host runs on a different port.
  349 
  350  local_hostname -- default *blank*
  351   The fully qualified domain name (FQDN) to use during SMTP sessions. If left
  352   blank, the underlying SMTP library will attempt to detect your FQDN. If your
  353   mail host requires something specific, specify the FQDN to use.
  354 
  355  tls -- ``no``
  356   If your SMTP mail host provides or requires TLS (Transport Layer Security)
  357   then you may set this option to 'yes'.
  358   Allowed values: ``yes``, ``no``
  359 
  360  tls_keyfile -- default *blank*
  361   If TLS is used, you may set this option to the name of a PEM formatted
  362   file that contains your private key. The path may be either absolute or
  363   relative to the directory containig this config file.
  364 
  365  tls_certfile -- default *blank*
  366   If TLS is used, you may set this option to the name of a PEM formatted
  367   certificate chain file. The path may be either absolute or relative
  368   to the directory containig this config file.
  369 
  370  charset -- utf-8
  371   Character set to encode email headers with. We use utf-8 by default, as
  372   it's the most flexible. Some mail readers (eg. Eudora) can't cope with
  373   that, so you might need to specify a more limited character set
  374   (eg. iso-8859-1).
  375 
  376  debug -- default *blank*
  377   Setting this option makes Roundup to write all outgoing email messages
  378   to this file *instead* of sending them. This option has the same effect
  379   as environment variable SENDMAILDEBUG. Environment variable takes
  380   precedence. The path may be either absolute or relative to the directory
  381   containig this config file.
  382 
  383  add_authorinfo -- ``yes``
  384   Add a line with author information at top of all messages send by
  385   roundup.
  386 
  387  add_authoremail -- ``yes``
  388   Add the mail address of the author to the author information at the
  389   top of all messages.  If this is false but add_authorinfo is true,
  390   only the name of the actor is added which protects the mail address
  391   of the actor from being exposed at mail archives, etc.
  392 
  393 .. index:: config.ini; sections mailgw
  394    single: mailgw; config
  395    see: mail gateway; mailgw
  396 
  397 Section **mailgw**
  398  Roundup Mail Gateway options
  399 
  400  keep_quoted_text -- ``yes``
  401   Keep email citations when accepting messages. Setting this to ``no`` strips
  402   out "quoted" text from the message. Signatures are also stripped.
  403   Allowed values: ``yes``, ``no``
  404 
  405  leave_body_unchanged -- ``no``
  406   Preserve the email body as is - that is, keep the citations *and*
  407   signatures.
  408   Allowed values: ``yes``, ``no``
  409 
  410  default_class -- ``issue``
  411   Default class to use in the mailgw if one isn't supplied in email subjects.
  412   To disable, leave the value blank.
  413 
  414  language -- default *blank*
  415   Default locale name for the tracker mail gateway.  If this option is
  416   not set, mail gateway will use the language of the tracker instance.
  417 
  418  subject_prefix_parsing -- ``strict``
  419   Controls the parsing of the [prefix] on subject lines in incoming emails.
  420   ``strict`` will return an error to the sender if the [prefix] is not
  421   recognised. ``loose`` will attempt to parse the [prefix] but just
  422   pass it through as part of the issue title if not recognised. ``none``
  423   will always pass any [prefix] through as part of the issue title.
  424 
  425  subject_suffix_parsing -- ``strict``
  426   Controls the parsing of the [suffix] on subject lines in incoming emails.
  427   ``strict`` will return an error to the sender if the [suffix] is not
  428   recognised. ``loose`` will attempt to parse the [suffix] but just
  429   pass it through as part of the issue title if not recognised. ``none``
  430   will always pass any [suffix] through as part of the issue title.
  431 
  432  subject_suffix_delimiters -- ``[]``
  433   Defines the brackets used for delimiting the commands suffix in a subject
  434   line.
  435 
  436  subject_content_match -- ``always``
  437   Controls matching of the incoming email subject line against issue titles
  438   in the case where there is no designator [prefix]. ``never`` turns off
  439   matching. ``creation + interval`` or ``activity + interval`` will match
  440   an issue for the interval after the issue's creation or last activity.
  441   The interval is a standard Roundup interval.
  442 
  443  subject_updates_title -- ``yes``
  444   Update issue title if incoming subject of email is different.
  445   Setting this to ``no`` will ignore the title part of
  446   the subject of incoming email messages.
  447 
  448  refwd_re -- ``(\s*\W?\s*(fw|fwd|re|aw|sv|ang)\W)+``
  449   Regular expression matching a single reply or forward prefix
  450   prepended by the mailer. This is explicitly stripped from the
  451   subject during parsing.  Value is Python Regular Expression
  452   (UTF8-encoded).
  453 
  454  origmsg_re -- `` ^[>|\s]*-----\s?Original Message\s?-----$``
  455   Regular expression matching start of an original message if quoted
  456   in the body.  Value is Python Regular Expression (UTF8-encoded).
  457 
  458  sign_re -- ``^[>|\s]*-- ?$``
  459   Regular expression matching the start of a signature in the message
  460   body.  Value is Python Regular Expression (UTF8-encoded).
  461 
  462  eol_re -- ``[\r\n]+``
  463   Regular expression matching end of line.  Value is Python Regular
  464   Expression (UTF8-encoded).
  465 
  466  blankline_re -- ``[\r\n]+\s*[\r\n]+``
  467   Regular expression matching a blank line.  Value is Python Regular
  468   Expression (UTF8-encoded).
  469 
  470  ignore_alternatives -- ``no``
  471   When parsing incoming mails, roundup uses the first
  472   text/plain part it finds. If this part is inside a
  473   multipart/alternative, and this option is set, all other
  474   parts of the multipart/alternative are ignored. The default
  475   is to keep all parts and attach them to the issue.
  476 
  477 .. index:: config.ini; sections php
  478 
  479 Section **pgp**
  480  OpenPGP mail processing options
  481 
  482  enable -- ``no``
  483   Enable PGP processing. Requires gpg.
  484 
  485  roles -- default *blank*
  486   If specified, a comma-separated list of roles to perform PGP
  487   processing on. If not specified, it happens for all users.
  488 
  489  homedir -- default *blank*
  490   Location of PGP directory. Defaults to $HOME/.gnupg if not
  491   specified.
  492 
  493 
  494 .. index:: config.ini; sections nosy
  495 
  496 Section **nosy**
  497  Nosy messages sending
  498 
  499  messages_to_author -- ``no``
  500   Send nosy messages to the author of the message.
  501   If ``yes`` is used, then messages are sent to the author
  502   even if not on the nosy list, same for ``new`` (but only for new messages).
  503   When set to ``nosy``, the nosy list controls sending messages to the author.
  504   Allowed values: ``yes``, ``no``, ``new``, ``nosy``
  505 
  506  signature_position -- ``bottom``
  507   Where to place the email signature.
  508   Allowed values: ``top``, ``bottom``, ``none``
  509 
  510  add_author -- ``new``
  511   Does the author of a message get placed on the nosy list automatically?
  512   If ``new`` is used, then the author will only be added when a message
  513   creates a new issue. If ``yes``, then the author will be added on
  514   followups too. If ``no``, they're never added to the nosy.
  515   Allowed values: ``yes``, ``no``, ``new``
  516   
  517  add_recipients -- ``new``
  518   Do the recipients (``To:``, ``Cc:``) of a message get placed on the nosy
  519   list?  If ``new`` is used, then the recipients will only be added when a
  520   message creates a new issue. If ``yes``, then the recipients will be added
  521   on followups too. If ``no``, they're never added to the nosy.
  522   Allowed values: ``yes``, ``no``, ``new``
  523 
  524  email_sending -- ``single``
  525   Controls the email sending from the nosy reactor. If ``multiple`` then
  526   a separate email is sent to each recipient. If ``single`` then a single
  527   email is sent with each recipient as a CC address.
  528 
  529  max_attachment_size -- ``2147483647``
  530   Attachments larger than the given number of bytes won't be attached
  531   to nosy mails. They will be replaced by a link to the tracker's
  532   download page for the file.
  533 
  534 
  535 .. index:: single: roundup-admin; config.ini update
  536            single: roundup-admin; config.ini create
  537            single: config.ini; create
  538            single: config.ini; update
  539 
  540 You may generate a new default config file using the ``roundup-admin
  541 genconfig`` command. You can generate a new config file merging in
  542 existing settings using the ``roundup-admin updateconfig`` command.
  543 
  544 Configuration variables may be referred to in lower or upper case. In code,
  545 variables not in the "main" section are referred to using their section and
  546 name, so "domain" in the section "mail" becomes MAIL_DOMAIN.
  547 
  548 .. index:: pair: configuration; extensions
  549    pair: configuration; detectors
  550 
  551 Extending the configuration file
  552 --------------------------------
  553 
  554 You can't add new variables to the config.ini file in the tracker home but
  555 you can add two new config.ini files:
  556 
  557 - a config.ini in the ``extensions`` directory will be loaded and attached
  558   to the config variable as "ext".
  559 - a config.ini in the ``detectors`` directory will be loaded and attached
  560   to the config variable as "detectors".
  561 
  562 For example, the following in ``detectors/config.ini``::
  563 
  564     [main]
  565     qa_recipients = email@example.com
  566 
  567 is accessible as::
  568 
  569     db.config.detectors['QA_RECIPIENTS']
  570 
  571 Note that the name grouping applied to the main configuration file is
  572 applied to the extension config files, so if you instead have::
  573 
  574     [qa]
  575     recipients = email@example.com
  576 
  577 then the above ``db.config.detectors['QA_RECIPIENTS']`` will still work.
  578 
  579 .. index:: ! schema
  580 
  581 Tracker Schema
  582 ==============
  583 
  584 .. note::
  585    if you modify the schema, you'll most likely need to edit the
  586    `web interface`_ HTML template files and `detectors`_ to reflect
  587    your changes.
  588 
  589 A tracker schema defines what data is stored in the tracker's database.
  590 Schemas are defined using Python code in the ``schema.py`` module of your
  591 tracker.
  592 
  593 The ``schema.py`` and ``initial_data.py`` modules
  594 -------------------------------------------------
  595 
  596 The schema.py module is used to define what your tracker looks like
  597 on the inside, the schema of the tracker. It defines the Classes
  598 and properties on each class. It also defines the security for
  599 those Classes. The next few sections describe how schemas work
  600 and what you can do with them.
  601 
  602 The initial_data.py module sets up the initial state of your
  603 tracker. It’s called exactly once - by the ``roundup-admin initialise``
  604 command. See the start of the section on database content for more
  605 info about how this works.
  606 
  607 .. index:: schema; classic - description of
  608 
  609 The "classic" schema
  610 --------------------
  611 
  612 The "classic" schema looks like this (see section `setkey(property)`_
  613 below for the meaning of ``'setkey'`` -- you may also want to look into
  614 the sections `setlabelprop(property)`_ and `setorderprop(property)`_ for
  615 specifying (default) labelling and ordering of classes.)::
  616 
  617     pri = Class(db, "priority", name=String(), order=String())
  618     pri.setkey("name")
  619 
  620     stat = Class(db, "status", name=String(), order=String())
  621     stat.setkey("name")
  622 
  623     keyword = Class(db, "keyword", name=String())
  624     keyword.setkey("name")
  625 
  626     user = Class(db, "user", username=String(), organisation=String(),
  627         password=String(), address=String(), realname=String(),
  628         phone=String(), alternate_addresses=String(),
  629         queries=Multilink('query'), roles=String(), timezone=String())
  630     user.setkey("username")
  631 
  632     msg = FileClass(db, "msg", author=Link("user"), summary=String(),
  633         date=Date(), recipients=Multilink("user"),
  634         files=Multilink("file"), messageid=String(), inreplyto=String())
  635 
  636     file = FileClass(db, "file", name=String())
  637 
  638     issue = IssueClass(db, "issue", keyword=Multilink("keyword"),
  639         status=Link("status"), assignedto=Link("user"),
  640         priority=Link("priority"))
  641     issue.setkey('title')
  642 
  643 .. index:: schema; allowed changes
  644 
  645 What you can't do to the schema
  646 -------------------------------
  647 
  648 You must never:
  649 
  650 **Remove the users class**
  651   This class is the only *required* class in Roundup.
  652 
  653 **Remove the "username", "address", "password" or "realname" user properties**
  654   Various parts of Roundup require these properties. Don't remove them.
  655 
  656 **Change the type of a property**
  657   Property types must *never* be changed - the database simply doesn't take
  658   this kind of action into account. Note that you can't just remove a
  659   property and re-add it as a new type either. If you wanted to make the
  660   assignedto property a Multilink, you'd need to create a new property
  661   assignedto_list and remove the old assignedto property.
  662 
  663 
  664 What you can do to the schema
  665 -----------------------------
  666 
  667 Your schema may be changed at any time before or after the tracker has been
  668 initialised (or used). You may:
  669 
  670 **Add new properties to classes, or add whole new classes**
  671   This is painless and easy to do - there are generally no repercussions
  672   from adding new information to a tracker's schema.
  673 
  674 **Remove properties**
  675   Removing properties is a little more tricky - you need to make sure that
  676   the property is no longer used in the `web interface`_ *or* by the
  677   detectors_.
  678 
  679 
  680 
  681 Classes and Properties - creating a new information store
  682 ---------------------------------------------------------
  683 
  684 In the tracker above, we've defined 7 classes of information:
  685 
  686   priority
  687       Defines the possible levels of urgency for issues.
  688 
  689   status
  690       Defines the possible states of processing the issue may be in.
  691 
  692   keyword
  693       Initially empty, will hold keywords useful for searching issues.
  694 
  695   user
  696       Initially holding the "admin" user, will eventually have an entry
  697       for all users using roundup.
  698 
  699   msg
  700       Initially empty, will hold all e-mail messages sent to or
  701       generated by roundup.
  702 
  703   file
  704       Initially empty, will hold all files attached to issues.
  705 
  706   issue
  707       Initially empty, this is where the issue information is stored.
  708 
  709 We define the "priority" and "status" classes to allow two things:
  710 
  711   1. reduction in the amount of information stored on the issue
  712   2. more powerful, accurate searching of issues by priority and status
  713 
  714 By only requiring a link on the issue (which is stored as a single
  715 number) we reduce the chance that someone mis-types a priority or
  716 status - or simply makes a new one up.
  717 
  718 
  719 Class and Items
  720 ~~~~~~~~~~~~~~~
  721 
  722 A Class defines a particular class (or type) of data that will be stored
  723 in the database. A class comprises one or more properties, which gives
  724 the information about the class items.
  725 
  726 The actual data entered into the database, using ``class.create()``, are
  727 called items. They have a special immutable property called ``'id'``. We
  728 sometimes refer to this as the *itemid*.
  729 
  730 
  731 .. index:: schema; property types
  732 
  733 Properties
  734 ~~~~~~~~~~
  735 
  736 A Class is comprised of one or more properties of the following types:
  737 
  738   String
  739        properties are for storing arbitrary-length strings.
  740   Password
  741        properties are for storing encoded arbitrary-length strings.
  742        The default encoding is defined on the ``roundup.password.Password``
  743        class.
  744   Date
  745        properties store date-and-time stamps. Their values are Timestamp
  746        objects.
  747   Interval
  748        properties store time periods rather than absolute dates. For
  749        example 2 hours.
  750   Integer
  751        properties store integer values. (Number can store real/float values.)
  752   Number
  753        properties store numeric values. There is an option to use
  754        double-precision floating point numbers.
  755   Boolean
  756        properties store on/off, yes/no, true/false values.
  757   Link
  758        properties refers to a single other item selected from a
  759        specified class. The class is part of the property; the value is an
  760        integer, the id of the chosen item.
  761   Multilink
  762        properties refer to possibly many items in a specified
  763        class. The value is a list of integers.
  764 
  765 Properties can have additional attributes to change the default
  766 behaviour:
  767 
  768 .. index:: triple: schema; property attributes; required
  769    triple: schema; property attributes; default_value
  770    triple: schema; property attributes; quiet
  771 
  772 * All properties support the following attributes:
  773 
  774    - ``required``: see `design documentation`_. Adds the property to
  775      the list returned by calling get_required_props for the class.
  776    - ``default_value``: see `design documentation`_ Sets the default
  777      value if the property is not set.
  778    - ``quiet``: see `design documentation`_. Suppresses user visible
  779      to changes to this property. The property change is not reported:
  780 
  781       - in the change feedback/confirmation message in the web
  782         interface
  783       - the property change section of the nosy email
  784       - the web history at the bottom of an item's page
  785 
  786     This can be used to store state of the user interface (e.g. the
  787     names of elements that are collapsed or hidden from the
  788     user). Making properties that are updated as an indirect result of
  789     a user's change (e.g. updating a blockers property, counting
  790     number of times an issue was reopened or reassigned etc.) should
  791     not be displayed to the user as they can be confusing.
  792 
  793 .. index:: triple: schema; property attributes; indexme
  794 
  795 * String properties can have an ``indexme`` attribute that defines if the
  796   property should be part of the full text index. The default is 'no' but this
  797   can be set to 'yes' to allow a property's contents to be in the full
  798   text index.
  799 
  800 .. index:: triple: schema; property attributes; use_double
  801 
  802 * Number properties can have a ``use_double`` attribute that, when set
  803   to ``True``, will use double precision floating point in the database.
  804 * Link and Multilink properties can have several attributes:
  805 
  806   .. index:: triple: schema; property attributes; do_journal  
  807 
  808   - ``do_journal``: By default, every change of a link property is
  809     recorded in the item being linked to (or being unlinked). A typical
  810     use-case for setting ``do_journal='no'`` would be to turn off
  811     journalling of nosy list, message author and message recipient link
  812     and unlink events to prevent the journal from clogged with these
  813     events.
  814 
  815   .. index:: triple: schema; property attributes; try_id_parsing
  816 
  817   - ``try_id_parsing`` is turned on by default. If entering a number
  818     into a Link or Multilink field, roundup interprets this number as an
  819     ID of the item to link to. Sometimes items can have numeric names
  820     (like, e.g., product codes). For these roundup needs to match the
  821     numeric name and should never match an ID. In this case you can set
  822     ``try_id_parsing='no'``.
  823 
  824   .. index:: triple: schema; property attributes; rev_multilink
  825 
  826   - The ``rev_multilink`` option takes a property name to be inserted
  827     into the linked-to class. This property is a Multilink property that
  828     links back to the current class. The new Multilink is read-only (it
  829     is automatically modified if the Link or Multilink property defining
  830     it is modified). The new property can be used in normal searches
  831     using the "filter" method of the Class. This means it can be used
  832     like other Multilink properties when searching (in an index
  833     template) or via the REST and XMLRPC APIs.
  834 
  835     As a example, suppose you want to group multiple issues into a
  836     super issue. Each issue can be part of only one super issue. It is
  837     inefficient to find all of the issues that are part of the
  838     super issue by searching through all issues in the system looking
  839     at the part_of link property. To make this more efficient, you
  840     can declare an issue's part_of property as::
  841 
  842        issue = IssueClass(db, "issue",
  843                  ...
  844 		 part_of = Link("issue", rev_multilink="components"),
  845 		 ... )
  846 
  847     This automatically creates the ``components`` multilink on the issue
  848     class. The ``components`` multilink is never explicitly declared in
  849     the issue class, but it has the same effect as though you had
  850     declared the class as::
  851 
  852        issue = IssueClass(db, "issue",
  853                  ...
  854 		 part_of = Link("issue"),
  855 		 components = Multilink("issue"),
  856 		 ... )
  857 
  858     Then wrote a detector to update the components property on the
  859     corresponding issue. Writing this detector can be tricky. There is
  860     one other difference, you can not explicitly set/modify the
  861     ``components`` multilink.
  862 
  863     The effect of setting ``part_of = 3456`` on issue1234
  864     automatically adds "1234" to the ``components`` property on
  865     issue3456. You can search the ``components`` multilink just like a
  866     regular multilink, but you can't explicitly assign to it.
  867     Another difference of reverse multilinks to normal multilinks
  868     is that when a linked node is retired, the node vanishes from the
  869     multilink, e.g. in the example above, if an issue with ``part_of``
  870     set to another issue is retired this issue vanishes from the
  871     ``components`` multilink of the other issue.
  872 
  873     You can also link between different classes. So you can modify
  874     the issue definition to include::
  875 
  876        issue = IssueClass(db, "issue",
  877                  ...
  878 		 assigned_to = Link("user", rev_multilink="responsibleFor"),
  879 		 ... )
  880    
  881     This makes it easy to list all issues that the user is responsible
  882     for (aka assigned_to).
  883 
  884   .. index:: triple: schema; property attributes; msg_header_property
  885 
  886   - The ``msg_header_property`` is used by the mail gateway when sending
  887     out messages. When a link or multilink property of an issue changes,
  888     roundup creates email headers of the form::
  889 
  890         X-Roundup-issue-prop: value
  891 
  892     where ``value`` is the ``name`` property for the linked item(s).
  893     For example, if you have a multilink for attached_files in your
  894     issue, you will see a header::
  895 
  896         X-Roundup-issue-attached_files: MySpecialFile.doc, HisResume.txt
  897 
  898     when the class for attached files is defined as::
  899 
  900        file = FileClass(db, "file",
  901                 name=String())
  902 
  903     ``MySpecialFile.doc`` is the name for the file object.
  904 
  905     If you have an ``assigned_to`` property in your issue class that
  906     links to the user class and you want to add a header::
  907 
  908         X-Roundup-issue-assigned_to: ...
  909 
  910     so that the mail recipients can filter emails where
  911     ``X-Roundup-issue-assigned_to: name`` that contains their
  912     username. The user class is defined as::
  913 
  914        user = Class(db, "user",
  915                 username=String(),
  916                 password=Password(),
  917                 address=String(),
  918                 realname=String(),
  919                 phone=String(),
  920                 organisation=String(),
  921                 alternate_addresses=String(),
  922                 queries=Multilink('query'),
  923                 roles=String(),     # comma-separated string of Role names
  924                 timezone=String())
  925 
  926     Because there is no ``name`` parameter for the user class, there
  927     will be no header. However setting::
  928 
  929        assigned_to=Link("user", msg_header_property="username")
  930 
  931     will make the mail gateway generate an ``X-Roundup-issue-assigned_to``
  932     using the username property of the linked user.
  933 
  934     Assume assigned_to for an issue is linked to the user with
  935     username=joe_user, setting::
  936 
  937         msg_header_property="username"
  938 
  939     for the assigned_to property will generated message headers of the
  940     form::
  941 
  942         X-Roundup-issue-assigned_to: joe_user
  943 
  944     for emails sent on issues where joe_user has been assigned to the issue.
  945 
  946     If this property is set to the empty string "", it will prevent
  947     the header from being generated on outgoing mail.
  948 
  949 .. index:: triple: schema; class property; creator
  950    triple: schema; class property; creation
  951    triple: schema; class property; actor
  952    triple: schema; class property; activity
  953 
  954 All Classes automatically have a number of properties by default:
  955 
  956 *creator*
  957   Link to the user that created the item.
  958 *creation*
  959   Date the item was created.
  960 *actor*
  961   Link to the user that last modified the item.
  962 *activity*
  963   Date the item was last modified.
  964 
  965 
  966 .. index:: triple: schema; class property; content
  967    triple: schema; class property; type
  968 
  969 FileClass
  970 ~~~~~~~~~
  971 
  972 FileClasses save their "content" attribute off in a separate file from
  973 the rest of the database. This reduces the number of large entries in
  974 the database, which generally makes databases more efficient, and also
  975 allows us to use command-line tools to operate on the files. They are
  976 stored in the files sub-directory of the ``'db'`` directory in your
  977 tracker. FileClasses also have a "type" attribute to store the MIME
  978 type of the file.
  979 
  980 .. index:: triple: schema; class property; messages
  981    triple: schema; class property; files
  982    triple: schema; class property; nosy
  983    triple: schema; class property; superseder
  984 
  985 IssueClass
  986 ~~~~~~~~~~
  987 
  988 IssueClasses automatically include the "messages", "files", "nosy", and
  989 "superseder" properties.
  990 
  991 The messages and files properties list the links to the messages and
  992 files related to the issue. The nosy property is a list of links to
  993 users who wish to be informed of changes to the issue - they get "CC'ed"
  994 e-mails when messages are sent to or generated by the issue. The nosy
  995 reactor (in the ``'detectors'`` directory) handles this action. The
  996 superseder link indicates an issue which has superseded this one.
  997 
  998 They also have the dynamically generated "creation", "activity" and
  999 "creator" properties.
 1000 
 1001 The value of the "creation" property is the date when an item was
 1002 created, and the value of the "activity" property is the date when any
 1003 property on the item was last edited (equivalently, these are the dates
 1004 on the first and last records in the item's journal). The "creator"
 1005 property holds a link to the user that created the issue.
 1006 
 1007 .. index: triple: schema; class method; setkey
 1008 
 1009 setkey(property)
 1010 ~~~~~~~~~~~~~~~~
 1011 
 1012 .. index:: roundup-admin; setting assignedto on an issue
 1013 
 1014 Select a String property of the class to be the key property. The key
 1015 property must be unique, and allows references to the items in the class
 1016 by the content of the key property. That is, we can refer to users by
 1017 their username: for example, let's say that there's an issue in roundup,
 1018 issue 23. There's also a user, richard, who happens to be user 2. To
 1019 assign an issue to him, we could do either of::
 1020 
 1021      roundup-admin set issue23 assignedto=2
 1022 
 1023 or::
 1024 
 1025      roundup-admin set issue23 assignedto=richard
 1026 
 1027 Note, the same thing can be done in the web and e-mail interfaces. 
 1028 
 1029 .. index: triple: schema; class method; setlabelprop
 1030 
 1031 setlabelprop(property)
 1032 ~~~~~~~~~~~~~~~~~~~~~~
 1033 
 1034 Select a property of the class to be the label property. The label
 1035 property is used whereever an item should be uniquely identified, e.g.,
 1036 when displaying a link to an item. If setlabelprop is not specified for
 1037 a class, the following values are tried for the label: 
 1038 
 1039  * the key of the class (see the `setkey(property)`_ section above)
 1040  * the "name" property
 1041  * the "title" property
 1042  * the first property from the sorted property name list
 1043 
 1044 So in most cases you can get away without specifying setlabelprop
 1045 explicitly.
 1046 
 1047 You should make sure that users have View access to this property or
 1048 the id property for a class. If the property can not be viewed by a
 1049 user, looping over items in the class (e.g. messages attached to an
 1050 issue) will not work.
 1051 
 1052 .. index: triple: schema; class method; setorderprop
 1053 
 1054 setorderprop(property)
 1055 ~~~~~~~~~~~~~~~~~~~~~~
 1056 
 1057 Select a property of the class to be the order property. The order
 1058 property is used whenever using a default sort order for the class,
 1059 e.g., when grouping or sorting class A by a link to class B in the user
 1060 interface, the order property of class B is used for sorting.  If
 1061 setorderprop is not specified for a class, the following values are tried
 1062 for the order property:
 1063 
 1064  * the property named "order"
 1065  * the label property (see `setlabelprop(property)`_ above)
 1066 
 1067 So in most cases you can get away without specifying setorderprop
 1068 explicitly.
 1069 
 1070 .. index: triple: schema; class method; create
 1071 
 1072 create(information)
 1073 ~~~~~~~~~~~~~~~~~~~
 1074 
 1075 Create an item in the database. This is generally used to create items
 1076 in the "definitional" classes like "priority" and "status".
 1077 
 1078 .. index: schema; item ordering
 1079 
 1080 A note about ordering
 1081 ~~~~~~~~~~~~~~~~~~~~~
 1082 
 1083 When we sort items in the hyperdb, we use one of a number of methods,
 1084 depending on the properties being sorted on:
 1085 
 1086 1. If it's a String, Integer, Number, Date or Interval property, we
 1087    just sort the scalar value of the property. Strings are sorted
 1088    case-sensitively.
 1089 2. If it's a Link property, we sort by either the linked item's "order"
 1090    property (if it has one) or the linked item's "id".
 1091 3. Mulitlinks sort similar to #2, but we start with the first Multilink
 1092    list item, and if they're the same, we sort by the second item, and
 1093    so on.
 1094 
 1095 Note that if an "order" property is defined on a Class that is used for
 1096 sorting, all items of that Class *must* have a value against the "order"
 1097 property, or sorting will result in random ordering.
 1098 
 1099 
 1100 Examples of adding to your schema
 1101 ---------------------------------
 1102 
 1103 Some examples are in the :ref:`CustomExamples` section below.
 1104 
 1105 Also the `Roundup wiki`_ has additional examples of how schemas can be
 1106 customised to add new functionality.
 1107 
 1108 .. _Roundup wiki:
 1109    https://wiki.roundup-tracker.org/
 1110 
 1111 .. index:: !detectors
 1112 
 1113 Detectors - adding behaviour to your tracker
 1114 ============================================
 1115 .. _detectors:
 1116 
 1117 Detectors are initialised every time you open your tracker database, so
 1118 you're free to add and remove them any time, even after the database is
 1119 initialised via the ``roundup-admin initialise`` command.
 1120 
 1121 The detectors in your tracker fire *before* (**auditors**) and *after*
 1122 (**reactors**) changes to the contents of your database. They are Python
 1123 modules that sit in your tracker's ``detectors`` directory. You will
 1124 have some installed by default - have a look. You can write new
 1125 detectors or modify the existing ones. The existing detectors installed
 1126 for you are:
 1127 
 1128 .. index:: detectors; installed
 1129 
 1130 **nosyreaction.py**
 1131   This provides the automatic nosy list maintenance and email sending.
 1132   The nosy reactor (``nosyreaction``) fires when new messages are added
 1133   to issues. The nosy auditor (``updatenosy``) fires when issues are
 1134   changed, and figures out what changes need to be made to the nosy list
 1135   (such as adding new authors, etc.)
 1136 **statusauditor.py**
 1137   This provides the ``chatty`` auditor which changes the issue status
 1138   from ``unread`` or ``closed`` to ``chatting`` if new messages appear.
 1139   It also provides the ``presetunread`` auditor which pre-sets the
 1140   status to ``unread`` on new items if the status isn't explicitly
 1141   defined.
 1142 **messagesummary.py**
 1143   Generates the ``summary`` property for new messages based on the message
 1144   content.
 1145 **userauditor.py**
 1146   Verifies the content of some of the user fields (email addresses and
 1147   roles lists).
 1148 
 1149 If you don't want this default behaviour, you're completely free to change
 1150 or remove these detectors.
 1151 
 1152 See the detectors section in the `design document`__ for details of the
 1153 interface for detectors.
 1154 
 1155 __ design.html
 1156 
 1157 
 1158 .. index:: detectors; writing api
 1159 
 1160 Detector API
 1161 ------------
 1162 
 1163 .. index:: pair: detectors; auditors
 1164    single: auditors; function signature
 1165    single: auditors; defining
 1166    single: auditors; arguments
 1167 
 1168 Auditors are called with the arguments::
 1169 
 1170     audit(db, cl, itemid, newdata)
 1171 
 1172 where ``db`` is the database, ``cl`` is an instance of Class or
 1173 IssueClass within the database, and ``newdata`` is a dictionary mapping
 1174 property names to values.
 1175 
 1176 For a ``create()`` operation, the ``itemid`` argument is None and
 1177 newdata contains all of the initial property values with which the item
 1178 is about to be created.
 1179 
 1180 For a ``set()`` operation, newdata contains only the names and values of
 1181 properties that are about to be changed.
 1182 
 1183 For a ``retire()`` or ``restore()`` operation, newdata is None.
 1184 
 1185 .. index:: pair: detectors; reactor
 1186    single: reactors; function signature
 1187    single: reactors; defining
 1188    single: reactors; arguments
 1189 
 1190 Reactors are called with the arguments::
 1191 
 1192     react(db, cl, itemid, olddata)
 1193 
 1194 where ``db`` is the database, ``cl`` is an instance of Class or
 1195 IssueClass within the database, and ``olddata`` is a dictionary mapping
 1196 property names to values.
 1197 
 1198 For a ``create()`` operation, the ``itemid`` argument is the id of the
 1199 newly-created item and ``olddata`` is None.
 1200 
 1201 For a ``set()`` operation, ``olddata`` contains the names and previous
 1202 values of properties that were changed.
 1203 
 1204 For a ``retire()`` or ``restore()`` operation, ``itemid`` is the id of
 1205 the retired or restored item and ``olddata`` is None.
 1206 
 1207 .. index:: detectors; additional
 1208 
 1209 Additional Detectors Ready For Use
 1210 ----------------------------------
 1211 
 1212 Sample additional detectors that have been found useful will appear in
 1213 the ``'detectors'`` directory of the Roundup distribution. If you want
 1214 to use one, copy it to the ``'detectors'`` of your tracker instance:
 1215 
 1216 **irker.py**
 1217   This detector sends notification on IRC through an irker daemon
 1218   (http://www.catb.org/esr/irker/) when issues are created or messages
 1219   are added.  In order to use it you need to install irker, start the
 1220   irkerd daemon, and add an ``[irker]`` section in ``detectors/config.ini``
 1221   that contains a comma-separated list of channels where the messages should
 1222   be sent, e.g. ``channels = irc://chat.freenode.net/channelname``.
 1223 **newissuecopy.py**
 1224   This detector sends an email to a team address whenever a new issue is
 1225   created. The address is hard-coded into the detector, so edit it
 1226   before you use it (look for the text 'team@team.host') or you'll get
 1227   email errors!
 1228 **creator_resolution.py**
 1229   Catch attempts to set the status to "resolved" - if the assignedto
 1230   user isn't the creator, then set the status to "confirm-done". Note that
 1231   "classic" Roundup doesn't have that status, so you'll have to add it. If
 1232   you don't want to though, it'll just use "in-progress" instead.
 1233 **email_auditor.py**
 1234   If a file added to an issue is of type message/rfc822, we tack on the
 1235   extension .eml.
 1236   The reason for this is that Microsoft Internet Explorer will not open
 1237   things with a .eml attachment, as they deem it 'unsafe'. Worse yet,
 1238   they'll just give you an incomprehensible error message. For more 
 1239   information, see the detector code - it has a length explanation.
 1240 
 1241 
 1242 .. index:: auditors; rules for use
 1243    single: reactors; rules for use
 1244 
 1245 Auditor or Reactor?
 1246 -------------------
 1247 
 1248 Generally speaking, the following rules should be observed:
 1249 
 1250 **Auditors**
 1251   Are used for `vetoing creation of or changes to items`_. They might
 1252   also make automatic changes to item properties.
 1253 **Reactors**
 1254   Detect changes in the database and react accordingly. They should avoid
 1255   making changes to the database where possible, as this could create
 1256   detector loops.
 1257 
 1258 
 1259 Vetoing creation of or changes to items
 1260 ---------------------------------------
 1261 
 1262 Auditors may raise the ``Reject`` exception to prevent the creation of
 1263 or changes to items in the database.  The mail gateway, for example, will
 1264 not attach files or messages to issues when the creation of those files or
 1265 messages are prevented through the ``Reject`` exception. It'll also not create
 1266 users if that creation is ``Reject``'ed too.
 1267 
 1268 To use, simply add at the top of your auditor::
 1269 
 1270    from roundup.exceptions import Reject
 1271 
 1272 And then when your rejection criteria have been detected, simply::
 1273 
 1274    raise Reject('Description of error')
 1275 
 1276 Error messages raised with ``Reject`` automatically have any HTML content
 1277 escaped before being displayed to the user. To display an error message to the
 1278 user without performing any HTML escaping the ``RejectRaw`` should be used. All
 1279 security implications should be carefully considering before using
 1280 ``RejectRaw``.
 1281 
 1282 
 1283 Generating email from Roundup
 1284 -----------------------------
 1285 
 1286 The module ``roundup.mailer`` contains most of the nuts-n-bolts required
 1287 to generate email messages from Roundup.
 1288 
 1289 In addition, the ``IssueClass`` methods ``nosymessage()`` and
 1290 ``send_message()`` are used to generate nosy messages, and may generate
 1291 messages which only consist of a change note (ie. the message id parameter
 1292 is not required - this is referred to as a "System Message" because it
 1293 comes from "the system" and not a user).
 1294 
 1295 
 1296 .. index:: extensions
 1297 .. index:: extensions; add python functions to tracker
 1298 
 1299 Extensions - adding capabilities to your tracker
 1300 ================================================
 1301 .. _extensions:
 1302 
 1303 While detectors_ add new behavior by reacting to changes in tracked
 1304 objects, `extensions` add new actions and utilities to Roundup, which
 1305 are mostly used to enhance web interface.
 1306 
 1307 You can create an extension by creating Python file in your tracker
 1308 ``extensions`` directory.  All files from this dir are loaded when
 1309 tracker instance is created, at which point it calls ``init(instance)``
 1310 from each file supplying itself as a first argument.
 1311 
 1312 Note that at this point web interface is not loaded, but extensions still
 1313 can register actions for in tracker instance. This may be fixed in
 1314 Roundup 1.6 by introducing ``init_web(client)`` callback or a more
 1315 flexible extension point mechanism.
 1316 
 1317 
 1318   * ``instance.registerUtil`` is used for adding `templating utilities`_
 1319     (see `adding a time log to your issues`_ for an example)
 1320 
 1321   * ``instance.registerAction`` is used to add more actions to instance
 1322     and to web interface. See `Defining new web actions`_ for details.
 1323     Generic action can be added by inheriting from ``action.Action``
 1324     instead of ``cgi.action.Action``.
 1325 
 1326 interfaces.py - hooking into the core of roundup
 1327 ================================================
 1328 .. _interfaces.py:
 1329 
 1330 There is a magic trick for hooking into the core of roundup. Using
 1331 this you can:
 1332 
 1333   * modify class data structures
 1334   * monkey patch core code to add new functionality
 1335   * modify the email gateway
 1336   * add new rest endpoints
 1337 
 1338 but with great power comes great responsibility.
 1339 
 1340 Interfaces.py has been around since the earliest releases of roundup
 1341 and used to be the main way to get a lot of customization done. In
 1342 modern roundup, the extensions_ mechanism is used, but there are places
 1343 where interfaces.py is still useful.
 1344 
 1345 Example: Changing Cache-Control headers
 1346 ---------------------------------------
 1347 
 1348 The Client class in cgi/client.py has a lookup table that is used to
 1349 set the Cache-Control headers for static files. The entries in this
 1350 table are set from interfaces.py using::
 1351 
 1352   from roundup.cgi.client import Client
 1353 
 1354   Client.Cache_Control['text/css'] = "public, max-age=3600"
 1355   Client.Cache_Control['application/javascript'] = "public, max-age=30"
 1356   Client.Cache_Control['rss.xml'] = "public, max-age=900"
 1357   Client.Cache_Control['local.js'] = "public, max-age=7200"
 1358 
 1359 In this case static files delivered using @@file will have cache
 1360 headers set. These files are searched for along the `static_files`
 1361 path in the tracker's `config.ini`. In the example above:
 1362 
 1363   * a css file (e.g. @@file/style.css) will be cached for an hour
 1364   * javascript files (e.g. @@file/libraries/jquery.js) will be cached
 1365     for 30 seconds
 1366   * a file named rss.xml will be cached for 15 minutes
 1367   * a file named local.js will be cached for 2 hours
 1368 
 1369 Note that a file name match overrides the mime type settings.
 1370 
 1371 
 1372 Example: Implement password complexity checking
 1373 -----------------------------------------------
 1374 
 1375 .. index:: tracker; lib directory (example)
 1376 
 1377 This example uses the zxcvbn_ module that you can place in the zxcvbn
 1378 subdirectory of your tracker's lib directory.
 1379 
 1380 If you add this to the interfaces.py file in the root directory of
 1381 your tracker (same place as schema.py)::
 1382 
 1383     import roundup.password as password
 1384     from roundup.exceptions import Reject
 1385     from zxcvbn import zxcvbn
 1386 
 1387     # monkey patch the setPassword method with this method
 1388     # that checks password strength.
 1389     origPasswordFunc = password.Password.setPassword
 1390     def mpPasswordFunc(self, plaintext, scheme, config=None):
 1391 	""" Replace the password set function with one that
 1392 	    verifies that the password is complex enough. It
 1393 	    has to be done at this point and not in an auditor
 1394 	    as the auditor only sees the encrypted password.
 1395 	"""
 1396 	results = zxcvbn(plaintext)
 1397 	if results['score'] < 3:
 1398 	    l = []
 1399 	    map(l.extend, [[results['feedback']['warning']], results['feedback']['suggestions']])
 1400 	    errormsg = " ".join(l)
 1401 	    raise Reject ("Password is too easy to guess. " + errormsg)
 1402 	return origPasswordFunc(self, plaintext, scheme, config=config)
 1403 
 1404     password.Password.setPassword = mpPasswordFunc
 1405 
 1406 it replaces the setPassword method in the Password class. The new
 1407 version validates that the password is sufficiently complex. Then it
 1408 passes off the setting of password to the original method.
 1409 
 1410 Example: Enhance time intervals
 1411 -------------------------------
 1412 
 1413 To make the user interface easier to use, you may want to support
 1414 other forms for intervals. For example you can support an interval
 1415 like 1.5 by interpreting it the same as 1:30 (1 hour 30 minutes).
 1416 Also you can allow a bare integer (e.g. 45) as a number of minutes.
 1417 
 1418 To do this we intercept the from_raw method of the Interval class in
 1419 hyperdb.py with::
 1420 
 1421     import roundup.hyperdb as hyperdb
 1422     origFrom_Raw = hyperdb.Interval.from_raw
 1423 
 1424     def normalizeperiod(self, value, **kw):
 1425 	''' Convert alternate time forms into standard interval format
 1426 
 1427 	    [+-] [#y] [#m] [#w] [#d] [[[H]H:MM]:SS]
 1428 
 1429 	    if value is float, it's hour and fractional hours
 1430 	    if value is integer, it's number of minutes
 1431 	'''
 1432 	if ":" not in value:
 1433 	    # Not a specified interval
 1434 	    # if int consider number of minutes
 1435 	    try:
 1436 		isMinutes = int(value)
 1437 		minutes = isMinutes%60
 1438 		hours = (isMinutes - minutes) / 60
 1439 		value = "%d:%d"%(hours,minutes)
 1440 	    except ValueError:
 1441 		pass
 1442 	    # if float, consider it number of hours and fractional hours.
 1443 	    import math
 1444 	    try:
 1445 		afterdecimal, beforedecimal = math.modf(float(value))
 1446 		value = "%d:%d"%(beforedecimal,60*afterdecimal)
 1447 	    except ValueError:
 1448 		pass
 1449 
 1450 	return origFrom_Raw(self, value, **kw)
 1451 
 1452     hyperdb.Interval.from_raw = normalizeperiod
 1453 
 1454 any call to convert an interval from raw form now has two simpler
 1455 (and more friendly) ways to specify common time intervals.
 1456 
 1457 Example: Modifying the mail gateway
 1458 -----------------------------------
 1459 
 1460 One site receives email on a main gateway. The virtual alias delivery
 1461 table on the postfix server is configured with::
 1462 
 1463    test-issues@example.com  roundup-test@roundup-vm.example.com
 1464    test-support@example.com roundup-test+support-a@roundup-vm.example.com
 1465    test@support.example.com roundup-test+support-b@roundup-vm.example.com
 1466 
 1467 These modifications to the mail gateway for roundup allows anonymous
 1468 submissions. It hides all of the requesters under the "support"
 1469 user. It also makes some other modifications to the mail parser
 1470 allowing keywords to be set and prefixes to be defined based on the
 1471 delivery alias.
 1472 
 1473 This is the entry in interfaces.py::
 1474 
 1475     import roundup.mailgw
 1476     import email.utils
 1477 
 1478     class SupportTracker(object):
 1479 	def __init__(self, prefix=None, keyword=None):
 1480 	    self.prefix = prefix
 1481 	    self.keyword = keyword
 1482 
 1483     # Define new prefixes and keywords based on local address.
 1484     support_trackers = {
 1485 	### production instances ###
 1486 
 1487 	### test instances ###
 1488 	'roundup-test+support-a':
 1489 	    SupportTracker(prefix='Support 1', keyword='support1'),
 1490 	'roundup-test+support-b':
 1491 	    SupportTracker(prefix='Support 2', keyword='support2'),
 1492 	'roundup-test2+support-a':
 1493 	    SupportTracker(prefix='Support 1', keyword='support1'),
 1494 	'roundup-test2+support-b':
 1495 	    SupportTracker(prefix='Support 2', keyword='support2'),
 1496     }
 1497 
 1498     class parsedMessage(roundup.mailgw.parsedMessage):
 1499 	def __init__(self, mailgw, message, support_tracker):
 1500 	    roundup.mailgw.parsedMessage.__init__(self, mailgw, message)
 1501 	    if support_tracker.prefix:
 1502 		self.prefix = '%s: ' % support_tracker.prefix
 1503 	    else:
 1504 		self.prefix = ''
 1505 	    self.keywords = []
 1506 	    if support_tracker.keyword:
 1507 		try:
 1508 		    self.keywords = [
 1509 			self.db.keyword.lookup(support_tracker.keyword)]
 1510 		except KeyError:
 1511 		    pass
 1512 	    self.config.ADD_AUTHOR_TO_NOSY = 'no'
 1513 	    self.config.ADD_RECIPIENTS_TO_NOSY = 'no'
 1514 	    self.config.MAILGW_KEEP_QUOTED_TEXT = 'yes'
 1515 	    self.config.MAILGW_LEAVE_BODY_UNCHANGED = 'yes'
 1516 	    self.classname = 'issue'
 1517 	    self.pfxmode = 'loose'
 1518 	    self.sfxmode = 'none'
 1519 	    # set the support user id
 1520 	    self.fixed_author = self.db.user.lookup('support')
 1521 	    self.fixed_props = {
 1522 		'nosy': [self.fixed_author],
 1523 		'keyword': self.keywords,
 1524 	    }
 1525 
 1526 	def handle_help(self):
 1527 	    pass
 1528 
 1529 	def check_subject(self):
 1530 	    if not self.subject:
 1531 		self.subject = 'no subject'
 1532 
 1533 	def rego_confirm(self):
 1534 	    pass
 1535 
 1536 	def get_author_id(self):
 1537 	    # force the support user to be the author
 1538 	    self.author = self.fixed_author
 1539 
 1540 	def get_props(self):
 1541 	    self.props = {}
 1542 	    if not self.nodeid:
 1543 		self.props.update(self.fixed_props)
 1544 		self.props['title'] = ("%s%s" % (
 1545 		    self.prefix, self.subject.replace('[', '(').replace(']', ')')))
 1546 
 1547 	def get_content_and_attachments(self):
 1548 	    roundup.mailgw.parsedMessage.get_content_and_attachments(self)
 1549 	    if not self.content:
 1550 		self.content = 'no text'
 1551 	    intro = []
 1552 	    for header in ['From', 'To', 'Cc']:
 1553 		for addr in self.message.getaddrlist(header):
 1554 		    intro.append('%s: %s' % (header, email.utils.formataddr(addr)))
 1555 	    intro.append('Subject: %s' % self.subject)
 1556 	    intro.append('\n')
 1557 	    self.content = '\n'.join(intro) + self.content
 1558 
 1559     class MailGW(roundup.mailgw.MailGW):
 1560 	def parsed_message_class(self, mailgw, message):
 1561 	    support_tracker = None
 1562 	    # The delivered-to header is unique to postfix
 1563 	    # it is the target address:
 1564 	    #   roundup-test+support-a@roundup-vm.example.com
 1565             # rather than
 1566 	    #   test-support@example.com
 1567 	    recipients = message.getaddrlist('delivered-to')
 1568 	    if recipients:
 1569 		localpart = recipients[0][1].rpartition('@')[0]
 1570 		support_tracker = support_trackers.get(localpart)
 1571 	    if support_tracker:
 1572 	        # parse the mesage using the parsedMessage class
 1573 		# defined above.
 1574 		return parsedMessage(mailgw, message, support_tracker)
 1575 	    else:
 1576 		# parse the message normally
 1577 		return roundup.mailgw.parsedMessage(mailgw, message)
 1578 
 1579 This is the most complex example section. The mail gateway is also one
 1580 of the more complex subsystems in roundup, and modifying it is not
 1581 trivial.
 1582 
 1583 Other Examples
 1584 --------------
 1585 
 1586 See the `rest interface documentation`_ for instructions on how to add
 1587 new rest endpoints using interfaces.py.
 1588 
 1589 Database Content
 1590 ================
 1591 
 1592 .. note::
 1593    If you modify the content of definitional classes, you'll most
 1594    likely need to edit the tracker `detectors`_ to reflect your changes.
 1595 
 1596 Customisation of the special "definitional" classes (eg. status,
 1597 priority, resolution, ...) may be done either before or after the
 1598 tracker is initialised. The actual method of doing so is completely
 1599 different in each case though, so be careful to use the right one.
 1600 
 1601 **Changing content before tracker initialisation**
 1602     Edit the initial_data.py module in your tracker to alter the items
 1603     created using the ``create( ... )`` methods.
 1604 
 1605 **Changing content after tracker initialisation**
 1606     As the "admin" user, click on the "class list" link in the web
 1607     interface to bring up a list of all database classes. Click on the
 1608     name of the class you wish to change the content of.
 1609 
 1610     You may also use the ``roundup-admin`` interface's create, set and
 1611     retire methods to add, alter or remove items from the classes in
 1612     question.
 1613 
 1614 See "`adding a new field to the classic schema`_" for an example that
 1615 requires database content changes.
 1616 
 1617 
 1618 Security / Access Controls
 1619 ==========================
 1620 
 1621 A set of Permissions is built into the security module by default:
 1622 
 1623 - Create (everything)
 1624 - Edit (everything)
 1625 - View (everything)
 1626 - Register (User class only)
 1627 
 1628 These are assigned to the "Admin" Role by default, and allow a user to do
 1629 anything. Every Class you define in your `tracker schema`_ also gets an
 1630 Create, Edit and View Permission of its own. The web and email interfaces
 1631 also define:
 1632 
 1633 *Email Access*
 1634   If defined, the user may use the email interface. Used by default to deny
 1635   Anonymous users access to the email interface. When granted to the
 1636   Anonymous user, they will be automatically registered by the email
 1637   interface (see also the ``new_email_user_roles`` configuration option).
 1638 *Web Access*
 1639   If defined, the user may use the web interface. All users are able to see
 1640   the login form, regardless of this setting (thus enabling logging in).
 1641 *Web Roles*
 1642   Controls user access to editing the "roles" property of the "user" class.
 1643   TODO: deprecate in favour of a property-based control.
 1644 *Rest Access* and *Xmlrpc Access*
 1645   These control access to the Rest and Xmlrpc endpoints. The Admin and User
 1646   roles have these by default in the classic tracker. See the
 1647   `directions in the rest interface documentation`_ and the
 1648   `xmlrpc interface documentation`_.
 1649 
 1650 These are hooked into the default Roles:
 1651 
 1652 - Admin (Create, Edit, View and everything; Web Roles)
 1653 - User (Web Access; Email Access)
 1654 - Anonymous (Web Access)
 1655 
 1656 And finally, the "admin" user gets the "Admin" Role, and the "anonymous"
 1657 user gets "Anonymous" assigned when the tracker is installed.
 1658 
 1659 For the "User" Role, the "classic" tracker defines:
 1660 
 1661 - Create, Edit and View issue, file, msg, query, keyword 
 1662 - View priority, status
 1663 - View user
 1664 - Edit their own user record
 1665 
 1666 And the "Anonymous" Role is defined as:
 1667 
 1668 - Web interface access
 1669 - Register user (for registration)
 1670 - View issue, file, msg, query, keyword, priority, status
 1671 
 1672 Put together, these settings appear in the tracker's ``schema.py`` file::
 1673 
 1674     #
 1675     # TRACKER SECURITY SETTINGS
 1676     #
 1677     # See the configuration and customisation document for information
 1678     # about security setup.
 1679 
 1680     #
 1681     # REGULAR USERS
 1682     #
 1683     # Give the regular users access to the web and email interface
 1684     db.security.addPermissionToRole('User', 'Web Access')
 1685     db.security.addPermissionToRole('User', 'Email Access')
 1686     db.security.addPermissionToRole('User', 'Rest Access')
 1687     db.security.addPermissionToRole('User', 'Xmlrpc Access')
 1688 
 1689     # Assign the access and edit Permissions for issue, file and message
 1690     # to regular users now
 1691     for cl in 'issue', 'file', 'msg', 'query', 'keyword':
 1692         db.security.addPermissionToRole('User', 'View', cl)
 1693         db.security.addPermissionToRole('User', 'Edit', cl)
 1694         db.security.addPermissionToRole('User', 'Create', cl)
 1695     for cl in 'priority', 'status':
 1696         db.security.addPermissionToRole('User', 'View', cl)
 1697 
 1698     # May users view other user information? Comment these lines out
 1699     # if you don't want them to
 1700     db.security.addPermissionToRole('User', 'View', 'user')
 1701 
 1702     # Users should be able to edit their own details -- this permission
 1703     # is limited to only the situation where the Viewed or Edited item
 1704     # is their own.
 1705     def own_record(db, userid, itemid, **ctx):
 1706         '''Determine whether the userid matches the item being accessed.'''
 1707         return userid == itemid
 1708     p = db.security.addPermission(name='View', klass='user', check=own_record,
 1709         description="User is allowed to view their own user details")
 1710     db.security.addPermissionToRole('User', p)
 1711     p = db.security.addPermission(name='Edit', klass='user', check=own_record,
 1712         description="User is allowed to edit their own user details")
 1713     db.security.addPermissionToRole('User', p)
 1714 
 1715     #
 1716     # ANONYMOUS USER PERMISSIONS
 1717     #
 1718     # Let anonymous users access the web interface. Note that almost all
 1719     # trackers will need this Permission. The only situation where it's not
 1720     # required is in a tracker that uses an HTTP Basic Authenticated front-end.
 1721     db.security.addPermissionToRole('Anonymous', 'Web Access')
 1722 
 1723     # Let anonymous users access the email interface (note that this implies
 1724     # that they will be registered automatically, hence they will need the
 1725     # "Create" user Permission below)
 1726     # This is disabled by default to stop spam from auto-registering users on
 1727     # public trackers.
 1728     #db.security.addPermissionToRole('Anonymous', 'Email Access')
 1729 
 1730     # Assign the appropriate permissions to the anonymous user's Anonymous
 1731     # Role. Choices here are:
 1732     # - Allow anonymous users to register
 1733     db.security.addPermissionToRole('Anonymous', 'Create', 'user')
 1734 
 1735     # Allow anonymous users access to view issues (and the related, linked
 1736     # information)
 1737     for cl in 'issue', 'file', 'msg', 'keyword', 'priority', 'status':
 1738         db.security.addPermissionToRole('Anonymous', 'View', cl)
 1739 
 1740     # [OPTIONAL]
 1741     # Allow anonymous users access to create or edit "issue" items (and the
 1742     # related file and message items)
 1743     #for cl in 'issue', 'file', 'msg':
 1744     #   db.security.addPermissionToRole('Anonymous', 'Create', cl)
 1745     #   db.security.addPermissionToRole('Anonymous', 'Edit', cl)
 1746 
 1747 .. index::
 1748    single: roundup-admin; view class permissions
 1749 
 1750 You can use ``roundup-admin security`` to verify the permissions
 1751 defined in the schema. It also verifies that properties specified in
 1752 permissions are valid for the class. This helps detect typos that can
 1753 cause baffling permission issues.
 1754 
 1755 Automatic Permission Checks
 1756 ---------------------------
 1757 
 1758 Permissions are automatically checked when information is rendered
 1759 through the web. This includes:
 1760 
 1761 1. View checks for properties when being rendered via the ``plain()`` or
 1762    similar methods. If the check fails, the text "[hidden]" will be
 1763    displayed.
 1764 2. Edit checks for properties when the edit field is being rendered via
 1765    the ``field()`` or similar methods. If the check fails, the property
 1766    will be rendered via the ``plain()`` method (see point 1. for subsequent
 1767    checking performed)
 1768 3. View checks are performed in index pages for each item being displayed
 1769    such that if the user does not have permission, the row is not rendered.
 1770 4. View checks are performed at the top of item pages for the Item being
 1771    displayed. If the user does not have permission, the text "You are not
 1772    allowed to view this page." will be displayed.
 1773 5. View checks are performed at the top of index pages for the Class being
 1774    displayed. If the user does not have permission, the text "You are not
 1775    allowed to view this page." will be displayed.
 1776 
 1777 
 1778 New User Roles
 1779 --------------
 1780 
 1781 New users are assigned the Roles defined in the config file as:
 1782 
 1783 - NEW_WEB_USER_ROLES
 1784 - NEW_EMAIL_USER_ROLES
 1785 
 1786 The `users may only edit their issues`_ example shows customisation of
 1787 these parameters.
 1788 
 1789 
 1790 Changing Access Controls
 1791 ------------------------
 1792 
 1793 You may alter the configuration variables to change the Role that new
 1794 web or email users get, for example to not give them access to the web
 1795 interface if they register through email. 
 1796 
 1797 You may use the ``roundup-admin`` "``security``" command to display the
 1798 current Role and Permission configuration in your tracker.
 1799 
 1800 
 1801 Adding a new Permission
 1802 ~~~~~~~~~~~~~~~~~~~~~~~
 1803 
 1804 When adding a new Permission, you will need to:
 1805 
 1806 1. add it to your tracker's ``schema.py`` so it is created, using
 1807    ``security.addPermission``, for example::
 1808 
 1809     db.security.addPermission(name="View", klass='frozzle',
 1810         description="User is allowed to access frozzles")
 1811 
 1812    will set up a new "View" permission on the Class "frozzle".
 1813 2. enable it for the Roles that should have it (verify with
 1814    "``roundup-admin security``")
 1815 3. add it to the relevant HTML interface templates
 1816 4. add it to the appropriate xxxPermission methods on in your tracker
 1817    interfaces module
 1818 
 1819 The ``addPermission`` method takes a few optional parameters:
 1820 
 1821 **properties**
 1822   A sequence of property names that are the only properties to apply the
 1823   new Permission to (eg. ``... klass='user', properties=('name',
 1824   'email') ...``)
 1825 **props_only**
 1826   A boolean value (set to false by default) that is a new feature
 1827   in roundup 1.6.
 1828   A permission defined using:
 1829 
 1830      ``properties=('list', 'of', 'property', 'names')``
 1831 
 1832   is used to determine access for things other than just those
 1833   properties. For example a check for View permission on the issue
 1834   class or an issue item can use any View permission for the issue
 1835   class even if that permission has a property list.  This can be
 1836   confusing and surprising as you would think that a permission
 1837   including properties would be used only for determining the
 1838   access permission for those properties.
 1839 
 1840   ``roundup-admin security`` will report invalid properties for the
 1841   class. For example a permission with an invalid summary property is
 1842   presented as::
 1843 
 1844      Allowed to see content of object regardless of spam status
 1845         (View for "file": ('content', 'summary') only)
 1846 
 1847      **Invalid properties for file: ['summary']
 1848 
 1849   Setting ``props_only=True`` will make the permission valid only for
 1850   those properties.
 1851 
 1852   If you use a lot of permissions with property checks, it can be
 1853   difficult to change all of them. Calling the function:
 1854 
 1855     db.security.set_props_only_default(True)
 1856 
 1857   at the top of ``schema.py`` will make every permission creation
 1858   behave as though props_only was set to True. It is expected that the
 1859   default of True will become the default in a future Roundup release.
 1860 **check**
 1861   A function to be executed which returns boolean determining whether
 1862   the Permission is allowed. If it returns True, the permission is
 1863   allowed, if it returns False the permission is denied.  The function
 1864   can have one of two signatures::
 1865 
 1866      check(db, userid, itemid)
 1867 
 1868   or::
 1869 
 1870      check(db, userid, itemid, **ctx)
 1871 
 1872   where ``db`` is a handle on the open database, ``userid`` is
 1873   the user attempting access and ``itemid`` is the specific item being
 1874   accessed. If the second form is used the ``ctx`` dictionary is
 1875   defined with the following values::
 1876 
 1877      ctx['property'] the name of the property being checked or None if
 1878             it's a class check.
 1879 
 1880      ctx['classname'] the name of the class that is being checked
 1881             (issue, query ....).
 1882 
 1883      ctx['permission'] the name of the permission (e.g. View, Edit...).
 1884 
 1885 The second form is preferred as it makes it easier to implement more
 1886 complex permission schemes. An example of the use of ``ctx`` can be
 1887 found in the ``upgrading.txt`` or `upgrading.html`_ document.
 1888 
 1889 .. _`upgrading.html`: upgrading.html
 1890 
 1891 Example Scenarios
 1892 ~~~~~~~~~~~~~~~~~
 1893 
 1894 See the `examples`_ section for longer examples of customisation.
 1895 
 1896 **anonymous access through the e-mail gateway**
 1897  Give the "anonymous" user the "Email Access", ("Edit", "issue") and
 1898  ("Create", "msg") Permissions but do not not give them the ("Create",
 1899  "user") Permission. This means that when an unknown user sends email
 1900  into the tracker, they're automatically logged in as "anonymous".
 1901  Since they don't have the ("Create", "user") Permission, they won't
 1902  be automatically registered, but since "anonymous" has permission to
 1903  use the gateway, they'll still be able to submit issues. Note that
 1904  the Sender information - their email address - will not be available
 1905  - they're *anonymous*.
 1906 
 1907 **automatic registration of users in the e-mail gateway**
 1908  By giving the "anonymous" user the ("Register", "user") Permission, any
 1909  unidentified user will automatically be registered with the tracker
 1910  (with no password, so they won't be able to log in through
 1911  the web until an admin sets their password). By default new Roundup
 1912  trackers don't allow this as it opens them up to spam. It may be enabled
 1913  by uncommenting the appropriate addPermissionToRole in your tracker's
 1914  ``schema.py`` file. The new user is given the Roles list defined in the
 1915  "new_email_user_roles" config variable. 
 1916 
 1917 **only developers may be assigned issues**
 1918  Create a new Permission called "Fixer" for the "issue" class. Create a
 1919  new Role "Developer" which has that Permission, and assign that to the
 1920  appropriate users. Filter the list of users available in the assignedto
 1921  list to include only those users. Enforce the Permission with an
 1922  auditor. See the example 
 1923  `restricting the list of users that are assignable to a task`_.
 1924 
 1925 **only managers may sign off issues as complete**
 1926  Create a new Permission called "Closer" for the "issue" class. Create a
 1927  new Role "Manager" which has that Permission, and assign that to the
 1928  appropriate users. In your web interface, only display the "resolved"
 1929  issue state option when the user has the "Closer" Permissions. Enforce
 1930  the Permission with an auditor. This is very similar to the previous
 1931  example, except that the web interface check would look like::
 1932 
 1933    <option tal:condition="python:request.user.hasPermission('Closer')"
 1934            value="resolved">Resolved</option>
 1935  
 1936 **don't give web access to users who register through email**
 1937  Create a new Role called "Email User" which has all the Permissions of
 1938  the normal "User" Role minus the "Web Access" Permission. This will
 1939  allow users to send in emails to the tracker, but not access the web
 1940  interface.
 1941 
 1942 **let some users edit the details of all users**
 1943  Create a new Role called "User Admin" which has the Permission for
 1944  editing users::
 1945 
 1946     db.security.addRole(name='User Admin', description='Managing users')
 1947     p = db.security.getPermission('Edit', 'user')
 1948     db.security.addPermissionToRole('User Admin', p)
 1949 
 1950  and assign the Role to the users who need the permission.
 1951 
 1952 
 1953 Web Interface
 1954 =============
 1955 
 1956 .. contents::
 1957    :local:
 1958 
 1959 The web interface is provided by the ``roundup.cgi.client`` module and
 1960 is used by ``roundup.cgi``, ``roundup-server`` and ``ZRoundup``
 1961 (``ZRoundup``  is broken, until further notice). In all cases, we
 1962 determine which tracker is being accessed (the first part of the URL
 1963 path inside the scope of the CGI handler) and pass control on to the
 1964 ``roundup.cgi.client.Client`` class - which handles the rest of the
 1965 access through its ``main()`` method. This means that you can do pretty
 1966 much anything you want as a web interface to your tracker.
 1967 
 1968 
 1969 
 1970 Repercussions of changing the tracker schema
 1971 ---------------------------------------------
 1972 
 1973 If you choose to change the `tracker schema`_ you will need to ensure
 1974 the web interface knows about it:
 1975 
 1976 1. Index, item and search pages for the relevant classes may need to
 1977    have properties added or removed,
 1978 2. The "page" template may require links to be changed, as might the
 1979    "home" page's content arguments.
 1980 
 1981 
 1982 How requests are processed
 1983 --------------------------
 1984 
 1985 The basic processing of a web request proceeds as follows:
 1986 
 1987 1. figure out who we are, defaulting to the "anonymous" user
 1988 2. figure out what the request is for - we call this the "context"
 1989 3. handle any requested action (item edit, search, ...)
 1990 4. render the template requested by the context, resulting in HTML
 1991    output
 1992 
 1993 In some situations, exceptions occur:
 1994 
 1995 - HTTP Redirect  (generally raised by an action)
 1996 - SendFile       (generally raised by ``determine_context``)
 1997     here we serve up a FileClass "content" property
 1998 - SendStaticFile (generally raised by ``determine_context``)
 1999     here we serve up a file from the tracker "html" directory
 2000 - Unauthorised   (generally raised by an action)
 2001     here the action is cancelled, the request is rendered and an error
 2002     message is displayed indicating that permission was not granted for
 2003     the action to take place
 2004 - NotFound       (raised wherever it needs to be)
 2005     this exception percolates up to the CGI interface that called the
 2006     client
 2007 
 2008 
 2009 Roundup URL design
 2010 ------------------
 2011 
 2012 Each tracker has several hardcoded URLs. These three are equivalent and
 2013 lead to the main tracker page:
 2014 
 2015 1. ``/``
 2016 2. ``/index``
 2017 3. ``/home``
 2018 
 2019 The following prefix is used to access static resources:
 2020 
 2021 4. ``/@@file/``
 2022 
 2023 All other URLs depend on the classes configured in Roundup database.
 2024 Each class receives two URLs - one for the class itself and another
 2025 for specific items of that class. Example for class URL:
 2026 
 2027 5. ``/issue``
 2028 
 2029 This is usually used to show listings of class items. The URL for
 2030 for specific object of issue class with id 1 will look like:
 2031 
 2032 6. ``/issue1``
 2033 
 2034 
 2035 Determining web context
 2036 -----------------------
 2037 
 2038 To determine the "context" of a request (what request is for), we look at
 2039 the URL path after the tracker root and at ``@template`` request
 2040 parameter. Typical URL paths look like:
 2041 
 2042 1.  ``/tracker/issue``
 2043 2.  ``/tracker/issue1``
 2044 3.  ``/tracker/@@file/style.css``
 2045 4.  ``/cgi-bin/roundup.cgi/tracker/file1``
 2046 5.  ``/cgi-bin/roundup.cgi/tracker/file1/kitten.png``
 2047 
 2048 where tracker root is ``/tracker/`` or ``/cgi-bin/roundup.cgi/tracker/``
 2049 We're looking at "issue", "issue1", "@@file/style.css", "file1" and
 2050 "file1/kitten.png" in the cases above.
 2051 
 2052 a. with is no path we are in the "home" context. See `the "home"
 2053    context`_ below for details. "index" or "home" paths may also be used
 2054    to switch into "home" context.
 2055 b. for paths starting with "@@file" the additional path entry ("style.css"
 2056    in the example above) specifies the static file to be served
 2057    from the tracker TEMPLATES directory (or STATIC_FILES, if configured).
 2058    This is usually the tracker's "html" directory. Internally this works
 2059    by raising SendStaticFile exception.
 2060 c. if there is something in the path (as in example 1, "issue"), it
 2061    identifies the tracker class to display.
 2062 d. if the path is an item designator (as in examples 2 and 4, "issue1"
 2063    and "file1"), then we're to display a specific item.
 2064 e. if the path starts with an item designator and is longer than one
 2065    entry (as in example 5, "file1/kitten.png"), then we're assumed to be
 2066    handling an item of a ``FileClass``, and the extra path information
 2067    gives the filename that the client is going to label the download
 2068    with (i.e. "file1/kitten.png" is nicer to download than "file1").
 2069    This raises a ``SendFile`` exception.
 2070 
 2071 Neither b. or e. use templates and stop before the template is
 2072 determined. For other contexts the template used is specified by the
 2073 ``@template`` variable, which defaults to:
 2074 
 2075 - only classname supplied:       "index"
 2076 - full item designator supplied: "item"
 2077 
 2078 
 2079 The "home" Context
 2080 ------------------
 2081 
 2082 The "home" context is special because it allows you to add templated
 2083 pages to your tracker that don't rely on a class or item (ie. an issues
 2084 list or specific issue).
 2085 
 2086 Let's say you wish to add frames to control the layout of your tracker's
 2087 interface. You'd probably have:
 2088 
 2089 - A top-level frameset page. This page probably wouldn't be templated, so
 2090   it could be served as a static file (see `serving static content`_)
 2091 - A sidebar frame that is templated. Let's call this page
 2092   "home.navigation.html" in your tracker's "html" directory. To load that
 2093   page up, you use the URL:
 2094 
 2095     <tracker url>/home?@template=navigation
 2096 
 2097 
 2098 Serving static content
 2099 ----------------------
 2100 
 2101 See the previous section `determining web context`_ where it describes
 2102 ``@@file`` paths.
 2103 
 2104 
 2105 Performing actions in web requests
 2106 ----------------------------------
 2107 
 2108 When a user requests a web page, they may optionally also request for an
 2109 action to take place. As described in `how requests are processed`_, the
 2110 action is performed before the requested page is generated. Actions are
 2111 triggered by using a ``@action`` CGI variable, where the value is one
 2112 of:
 2113 
 2114 **login**
 2115  Attempt to log a user in.
 2116 
 2117 **logout**
 2118  Log the user out - make them "anonymous".
 2119 
 2120 **register**
 2121  Attempt to create a new user based on the contents of the form and then
 2122  log them in.
 2123 
 2124 **edit**
 2125  Perform an edit of an item in the database. There are some `special form
 2126  variables`_ you may use.
 2127 
 2128 **new**
 2129  Add a new item to the database. You may use the same `special form
 2130  variables`_ as in the "edit" action. Also you can set the
 2131  ``__redirect_to`` form variable to the URL that should be displayed after
 2132  the new item is created. This is useful if you want to create another
 2133  item rather than edit the newly created item.
 2134 
 2135 **retire**
 2136  Retire the item in the database.
 2137 
 2138 **editCSV**
 2139  Performs an edit of all of a class' items in one go. See also the
 2140  *class*.csv templating method which generates the CSV data to be
 2141  edited, and the ``'_generic.index'`` template which uses both of these
 2142  features.
 2143 
 2144 **search**
 2145  Mangle some of the form variables:
 2146 
 2147  - Set the form ":filter" variable based on the values of the filter
 2148    variables - if they're set to anything other than "dontcare" then add
 2149    them to :filter.
 2150 
 2151  - Also handle the ":queryname" variable and save off the query to the
 2152    user's query list.
 2153 
 2154 Each of the actions is implemented by a corresponding ``*XxxAction*`` (where
 2155 "Xxx" is the name of the action) class in the ``roundup.cgi.actions`` module.
 2156 These classes are registered with ``roundup.cgi.client.Client``. If you need
 2157 to define new actions, you may add them there (see `defining new
 2158 web actions`_).
 2159 
 2160 Each action class also has a ``*permission*`` method which determines whether
 2161 the action is permissible given the current user. The base permission checks
 2162 for each action are:
 2163 
 2164 **login**
 2165  Determine whether the user has the "Web Access" Permission.
 2166 **logout**
 2167  No permission checks are made.
 2168 **register**
 2169  Determine whether the user has the ("Create", "user") Permission.
 2170 **edit**
 2171  Determine whether the user has permission to edit this item. If we're
 2172  editing the "user" class, users are allowed to edit their own details -
 2173  unless they try to edit the "roles" property, which requires the
 2174  special Permission "Web Roles".
 2175 **new**
 2176  Determine whether the user has permission to create this item. No
 2177  additional property checks are made. Additionally, new user items may
 2178  be created if the user has the ("Create", "user") Permission.
 2179 **editCSV**
 2180  Determine whether the user has permission to edit this class.
 2181 **search**
 2182  Determine whether the user has permission to view this class.
 2183 
 2184 Protecting users from web application attacks
 2185 ---------------------------------------------
 2186 
 2187 There is a class of attacks known as Cross Site Request Forgeries
 2188 (CSRF). Malicious code running in the browser can making a
 2189 request to roundup while you are logged into roundup.  The
 2190 malicious code piggy backs on your existing roundup session to
 2191 make changes without your knowledge. Roundup 1.6 has support for
 2192 defending against this by analyzing the
 2193 
 2194 * Referer,
 2195 * Origin, and
 2196 * Host or 
 2197 * X-Forwarded-Host
 2198 
 2199 HTTP headers. It compares the headers to the value of the web setting
 2200 in the [tracker] section of the tracker's ``config.ini``.
 2201 
 2202 Also a per form token (also called a nonce) can be enabled for
 2203 the tracker using the ``csrf_enforce_token`` option in
 2204 config.ini.  When enabled, roundup will validate a hidden form
 2205 field called ``@csrf``. If the validation fails (or the token
 2206 is used more than once) the request is rejected.  The ``@csrf``
 2207 input field is added automatically by calling the ``submit``
 2208 function/path. It can also be added manually by calling
 2209 anti_csrf_nonce() directly. For example:
 2210 
 2211    <input name="@csrf" type="hidden"
 2212       tal:attributes="value python:utils.anti_csrf_nonce(lifetime=10)">
 2213 
 2214 By default a nonce lifetime is 2 weeks. However the lifetime (in
 2215 minutes) can be set by passing a lifetime argument as shown
 2216 above. The example above makes the nonce lifetime 10 minutes.
 2217  
 2218 Search for @csrf in this document for more examples. There are
 2219 more examples and information in ``upgrading.txt``.
 2220 
 2221 The token protects you because malicious code supplied by another
 2222 site is unable to obtain the token. Thus many attempts they make
 2223 to submit a request are rejected.
 2224 
 2225 The protection on the xmlrpc interface is untested, but is based
 2226 on a valid header check against the roundup url and the presence
 2227 of the ``X-REQUESTED-WITH`` header. Work to improve this is a
 2228 future project after the 1.6 release.
 2229 
 2230 The enforcement levels can be modified in ``config.ini``. Refer to
 2231 that file for details.
 2232 
 2233 Special form variables
 2234 ----------------------
 2235 
 2236 Item properties and their values are edited with html FORM
 2237 variables and their values. You can:
 2238 
 2239 - Change the value of some property of the current item.
 2240 - Create a new item of any class, and edit the new item's
 2241   properties,
 2242 - Attach newly created items to a multilink property of the
 2243   current item.
 2244 - Remove items from a multilink property of the current item.
 2245 - Specify that some properties are required for the edit
 2246   operation to be successful.
 2247 - Redirect to a different page after creating a new item (new action
 2248   only, not edit action). Usually you end up on the page for the
 2249   created item.
 2250 - Set up user interface locale.
 2251 
 2252 These operations will only take place if the form action (the
 2253 ``@action`` variable) is "edit" or "new".
 2254 
 2255 In the following, <bracketed> values are variable, "@" may be
 2256 either ":" or "@", and other text "required" is fixed.
 2257 
 2258 Two special form variables are used to specify user language preferences:
 2259 
 2260 ``@language``
 2261   value may be locale name or ``none``. If this variable is set to
 2262   locale name, web interface language is changed to given value
 2263   (provided that appropriate translation is available), the value
 2264   is stored in the browser cookie and will be used for all following
 2265   requests.  If value is ``none`` the cookie is removed and the
 2266   language is changed to the tracker default, set up in the tracker
 2267   configuration or OS environment.
 2268 
 2269 ``@charset``
 2270   value may be character set name or ``none``.  Character set name
 2271   is stored in the browser cookie and sets output encoding for all
 2272   HTML pages generated by Roundup.  If value is ``none`` the cookie
 2273   is removed and HTML output is reset to Roundup internal encoding
 2274   (UTF-8).
 2275 
 2276 Most properties are specified as form variables:
 2277 
 2278 ``<propname>``
 2279   property on the current context item
 2280 
 2281 ``<designator>"@"<propname>``
 2282   property on the indicated item (for editing related information)
 2283 
 2284 Designators name a specific item of a class.
 2285 
 2286 ``<classname><N>``
 2287     Name an existing item of class <classname>.
 2288 
 2289 ``<classname>"-"<N>``
 2290     Name the <N>th new item of class <classname>. If the form
 2291     submission is successful, a new item of <classname> is
 2292     created. Within the submitted form, a particular
 2293     designator of this form always refers to the same new
 2294     item.
 2295 
 2296 Once we have determined the "propname", we look at it to see
 2297 if it's special:
 2298 
 2299 ``@required``
 2300     The associated form value is a comma-separated list of
 2301     property names that must be specified when the form is
 2302     submitted for the edit operation to succeed.  
 2303 
 2304     When the <designator> is missing, the properties are
 2305     for the current context item.  When <designator> is
 2306     present, they are for the item specified by
 2307     <designator>.
 2308 
 2309     The "@required" specifier must come before any of the
 2310     properties it refers to are assigned in the form.
 2311 
 2312 ``@remove@<propname>=id(s)`` or ``@add@<propname>=id(s)``
 2313     The "@add@" and "@remove@" edit actions apply only to
 2314     Multilink properties.  The form value must be a
 2315     comma-separate list of keys for the class specified by
 2316     the simple form variable.  The listed items are added
 2317     to (respectively, removed from) the specified
 2318     property.
 2319 
 2320 ``@link@<propname>=<designator>``
 2321     If the edit action is "@link@", the simple form
 2322     variable must specify a Link or Multilink property.
 2323     The form value is a comma-separated list of
 2324     designators.  The item corresponding to each
 2325     designator is linked to the property given by simple
 2326     form variable.
 2327 
 2328 None of the above (ie. just a simple form value)
 2329     The value of the form variable is converted
 2330     appropriately, depending on the type of the property.
 2331 
 2332     For a Link('klass') property, the form value is a
 2333     single key for 'klass', where the key field is
 2334     specified in schema.py.  
 2335 
 2336     For a Multilink('klass') property, the form value is a
 2337     comma-separated list of keys for 'klass', where the
 2338     key field is specified in schema.py.  
 2339 
 2340     Note that for simple-form-variables specifiying Link
 2341     and Multilink properties, the linked-to class must
 2342     have a key field.
 2343 
 2344     For a String() property specifying a filename, the
 2345     file named by the form value is uploaded. This means we
 2346     try to set additional properties "filename" and "type" (if
 2347     they are valid for the class).  Otherwise, the property
 2348     is set to the form value.
 2349 
 2350     For Date(), Interval(), Boolean(), Integer() and Number()
 2351     properties, the form value is converted to the
 2352     appropriate
 2353 
 2354 Any of the form variables may be prefixed with a classname or
 2355 designator.
 2356 
 2357 Setting the form variable: ``__redirect_to=`` to a url when
 2358 @action=new redirects the user to the specified url after successfully
 2359 creating the new item. This is useful if you want the user to create
 2360 another item rather than edit the newly created item.  Note that the
 2361 url assigned to ``__redirect_to`` must be url encoded/quoted and be
 2362 under the tracker's base url. If the base_url uses http, you can set
 2363 the url to https.
 2364 
 2365 Two special form values are supported for backwards compatibility:
 2366 
 2367 @note
 2368     This is equivalent to::
 2369 
 2370         @link@messages=msg-1
 2371         msg-1@content=value
 2372 
 2373     except that in addition, the "author" and "date" properties of
 2374     "msg-1" are set to the userid of the submitter, and the current
 2375     time, respectively.
 2376 
 2377 @file
 2378     This is equivalent to::
 2379 
 2380         @link@files=file-1
 2381         file-1@content=value
 2382 
 2383     The String content value is handled as described above for file
 2384     uploads.
 2385 
 2386 If both the "@note" and "@file" form variables are
 2387 specified, the action::
 2388 
 2389         @link@msg-1@files=file-1
 2390 
 2391 is also performed.
 2392 
 2393 We also check that FileClass items have a "content" property with
 2394 actual content, otherwise we remove them from all_props before
 2395 returning.
 2396 
 2397 
 2398 Default templates
 2399 -----------------
 2400 
 2401 The default templates are html4 compliant. If you wish to change them to be
 2402 xhtml compliant, you'll need to change the ``html_version`` configuration
 2403 variable in ``config.ini`` to ``'xhtml'`` instead of ``'html4'``.
 2404 
 2405 Most customisation of the web view can be done by modifying the
 2406 templates in the tracker ``'html'`` directory. There are several types
 2407 of files in there. The *minimal* template includes:
 2408 
 2409 **page.html**
 2410   This template usually defines the overall look of your tracker. When
 2411   you view an issue, it appears inside this template. When you view an
 2412   index, it also appears inside this template. This template defines a
 2413   macro called "icing" which is used by almost all other templates as a
 2414   coating for their content, using its "content" slot. It also defines
 2415   the "head_title" and "body_title" slots to allow setting of the page
 2416   title.
 2417 **home.html**
 2418   the default page displayed when no other page is indicated by the user
 2419 **home.classlist.html**
 2420   a special version of the default page that lists the classes in the
 2421   tracker
 2422 **classname.item.html**
 2423   displays an item of the *classname* class
 2424 **classname.index.html**
 2425   displays a list of *classname* items
 2426 **classname.search.html**
 2427   displays a search page for *classname* items
 2428 **_generic.index.html**
 2429   used to display a list of items where there is no
 2430   ``*classname*.index`` available
 2431 **_generic.help.html**
 2432   used to display a "class help" page where there is no
 2433   ``*classname*.help``
 2434 **user.register.html**
 2435   a special page just for the user class, that renders the registration
 2436   page
 2437 **style.css**
 2438   a static file that is served up as-is
 2439 
 2440 The *classic* template has a number of additional templates.
 2441 
 2442 Remember that you can create any template extension you want to,
 2443 so if you just want to play around with the templating for new issues,
 2444 you can copy the current "issue.item" template to "issue.test", and then
 2445 access the test template using the "@template" URL argument::
 2446 
 2447    http://your.tracker.example/tracker/issue?@template=test
 2448 
 2449 and it won't affect your users using the "issue.item" template.
 2450 
 2451 You can also put templates into a subdirectory of the template
 2452 directory. So if you specify::
 2453 
 2454    http://your.tracker.example/tracker/issue?@template=test/item
 2455 
 2456 you will use the template at: ``test/issue.item.html``. If that
 2457 template doesn't exit it will try to use
 2458 ``test/_generic.item.html``. If that template doesn't exist
 2459 it will return an error.
 2460 
 2461 Implementing Modal Editing Using @template
 2462 ------------------------------------------
 2463 
 2464 Many item templates allow you to edit the item. They contain
 2465 code that renders edit boxes if the user has edit permissions.
 2466 Otherwise the template will just display the item information.
 2467 
 2468 In some cases you want to do a modal edit. The user has to take some
 2469 action (click a button or follow a link) to shift from display mode to
 2470 edit mode. When the changes are submitted, ending the edit mode,
 2471 the user is returned to display mode.
 2472 
 2473 Modal workflows usually slow things down and are not implemented by
 2474 default templates. However for some workflows a modal edit is useful.
 2475 For example a batch edit mode that allows the user to edit a number of
 2476 issues all from one form could be implemented as a modal workflow of:
 2477 
 2478 * search for issues to modify
 2479 * switch to edit mode and change values
 2480 * exit back to the results of the search
 2481 
 2482 To implement the modal edit, assume you have an issue.edit.html
 2483 template that implements an edit form.  On the display page (a version
 2484 of issue.item.html modified to only display information) add a link
 2485 that calls the display url, but adds ``@template=edit`` to the link.
 2486 
 2487 This will now display the edit page. On the edit page you want to add
 2488 a hidden text field to your form named ``@template`` with the value:
 2489 ``item|edit``.  When the form is submitted it is validated. If the
 2490 form is correct the user will see the item rendered using the item
 2491 template. If there is an error (validation failed) the item will be
 2492 rendered using the edit template. The edit template that is rendered
 2493 will display all the changes that the user made to the form before it
 2494 was submitted. The user can correct the error and resubmit the changes
 2495 until the form validates.
 2496 
 2497 If the form failed to validate but the ``@template`` field had the
 2498 value ``item`` the user would still see the error, but all of the data
 2499 the user entered would be discarded. The user would have to redo all
 2500 the edits again.
 2501 
 2502 
 2503 How the templates work
 2504 ----------------------
 2505 
 2506 
 2507 Templating engines
 2508 ~~~~~~~~~~~~~~~~~~
 2509 
 2510 Since version 1.4.20 Roundup supports two templating engines: the original
 2511 `Template Attribute Language`_ (TAL) engine from Zope and the standalone
 2512 Chameleon templating engine. Chameleon is intended as a replacement for the
 2513 original TAL engine, and supports the same syntax,
 2514 but they are not 100% compatible. The major (and most likely the only)
 2515 incompatibility is the default expression type being
 2516 ``python:`` instead of ``path:``. See also "Incompatibilities and
 2517 differences" section of `Chameleon documentation`__.
 2518 
 2519 **NOTE1**: For historical reasons, examples given below assumes path
 2520 expression as default expression type. With Chameleon you have to manually
 2521 resolve the path expressions. A Chameleon-based, z3c.pt, that is fully
 2522 compatible with the old TAL implementation, is planned to be included in a
 2523 future release.
 2524 
 2525 **NOTE2**: As of 1.4.20 Chameleon support is highly experimental and **not**
 2526 recommended for production use.
 2527 
 2528 .. _Chameleon:
 2529    https://pypi.org/project/Chameleon/
 2530 .. _z3c.pt:
 2531    https://pypi.org/project/z3c.pt/
 2532 __ https://chameleon.readthedocs.io/en/latest/reference.html?highlight=differences#incompatibilities-and-differences
 2533 .. _TAL:
 2534 .. _Template Attribute Language:
 2535    https://pagetemplates.readthedocs.io/en/latest/history/TALSpecification14.html
 2536 
 2537 
 2538 Basic Templating Actions
 2539 ~~~~~~~~~~~~~~~~~~~~~~~~
 2540 
 2541 Roundup's templates consist of special attributes on the HTML tags.
 2542 These attributes form the **Template Attribute Language**, or TAL.
 2543 The basic TAL commands are:
 2544 
 2545 **tal:define="variable expression; variable expression; ..."**
 2546    Define a new variable that is local to this tag and its contents. For
 2547    example::
 2548 
 2549       <html tal:define="title request/description">
 2550        <head><title tal:content="title"></title></head>
 2551       </html>
 2552 
 2553    In this example, the variable "title" is defined as the result of the
 2554    expression "request/description". The "tal:content" command inside the
 2555    <html> tag may then use the "title" variable.
 2556 
 2557 **tal:condition="expression"**
 2558    Only keep this tag and its contents if the expression is true. For
 2559    example::
 2560 
 2561      <p tal:condition="python:request.user.hasPermission('View', 'issue')">
 2562       Display some issue information.
 2563      </p>
 2564 
 2565    In the example, the <p> tag and its contents are only displayed if
 2566    the user has the "View" permission for issues. We consider the number
 2567    zero, a blank string, an empty list, and the built-in variable
 2568    nothing to be false values. Nearly every other value is true,
 2569    including non-zero numbers, and strings with anything in them (even
 2570    spaces!).
 2571 
 2572 **tal:repeat="variable expression"**
 2573    Repeat this tag and its contents for each element of the sequence
 2574    that the expression returns, defining a new local variable and a
 2575    special "repeat" variable for each element. For example::
 2576 
 2577      <tr tal:repeat="u user/list">
 2578       <td tal:content="u/id"></td>
 2579       <td tal:content="u/username"></td>
 2580       <td tal:content="u/realname"></td>
 2581      </tr>
 2582 
 2583    The example would iterate over the sequence of users returned by
 2584    "user/list" and define the local variable "u" for each entry. Using
 2585    the repeat command creates a new variable called "repeat" which you
 2586    may access to gather information about the iteration. See the section
 2587    below on `the repeat variable`_.
 2588 
 2589 **tal:replace="expression"**
 2590    Replace this tag with the result of the expression. For example::
 2591 
 2592     <span tal:replace="request/user/realname" />
 2593 
 2594    The example would replace the <span> tag and its contents with the
 2595    user's realname. If the user's realname was "Bruce", then the
 2596    resultant output would be "Bruce".
 2597 
 2598 **tal:content="expression"**
 2599    Replace the contents of this tag with the result of the expression.
 2600    For example::
 2601 
 2602     <span tal:content="request/user/realname">user's name appears here
 2603     </span>
 2604 
 2605    The example would replace the contents of the <span> tag with the
 2606    user's realname. If the user's realname was "Bruce" then the
 2607    resultant output would be "<span>Bruce</span>".
 2608 
 2609 **tal:attributes="attribute expression; attribute expression; ..."**
 2610    Set attributes on this tag to the results of expressions. For
 2611    example::
 2612 
 2613      <a tal:attributes="href string:user${request/user/id}">My Details</a>
 2614 
 2615    In the example, the "href" attribute of the <a> tag is set to the
 2616    value of the "string:user${request/user/id}" expression, which will
 2617    be something like "user123".
 2618 
 2619 **tal:omit-tag="expression"**
 2620    Remove this tag (but not its contents) if the expression is true. For
 2621    example::
 2622 
 2623       <span tal:omit-tag="python:1">Hello, world!</span>
 2624 
 2625    would result in output of::
 2626 
 2627       Hello, world!
 2628 
 2629 Note that the commands on a given tag are evaulated in the order above,
 2630 so *define* comes before *condition*, and so on.
 2631 
 2632 Additionally, you may include tags such as <tal:block>, which are
 2633 removed from output. Its content is kept, but the tag itself is not (so
 2634 don't go using any "tal:attributes" commands on it). This is useful for
 2635 making arbitrary blocks of HTML conditional or repeatable (very handy
 2636 for repeating multiple table rows, which would othewise require an
 2637 illegal tag placement to effect the repeat).
 2638 
 2639 
 2640 Templating Expressions
 2641 ~~~~~~~~~~~~~~~~~~~~~~
 2642 
 2643 Templating Expressions are covered by `Template Attribute Language
 2644 Expression Syntax`_, or TALES. The expressions you may use in the
 2645 attribute values may be one of the following forms:
 2646 
 2647 **Path Expressions** - eg. ``item/status/checklist``
 2648    These are object attribute / item accesses. Roughly speaking, the
 2649    path ``item/status/checklist`` is broken into parts ``item``,
 2650    ``status`` and ``checklist``. The ``item`` part is the root of the
 2651    expression. We then look for a ``status`` attribute on ``item``, or
 2652    failing that, a ``status`` item (as in ``item['status']``). If that
 2653    fails, the path expression fails. When we get to the end, the object
 2654    we're left with is evaluated to get a string - if it is a method, it
 2655    is called; if it is an object, it is stringified. Path expressions
 2656    may have an optional ``path:`` prefix, but they are the default
 2657    expression type, so it's not necessary.
 2658 
 2659    If an expression evaluates to ``default``, then the expression is
 2660    "cancelled" - whatever HTML already exists in the template will
 2661    remain (tag content in the case of ``tal:content``, attributes in the
 2662    case of ``tal:attributes``).
 2663 
 2664    If an expression evaluates to ``nothing`` then the target of the
 2665    expression is removed (tag content in the case of ``tal:content``,
 2666    attributes in the case of ``tal:attributes`` and the tag itself in
 2667    the case of ``tal:replace``).
 2668 
 2669    If an element in the path may not exist, then you can use the ``|``
 2670    operator in the expression to provide an alternative. So, the
 2671    expression ``request/form/foo/value | default`` would simply leave
 2672    the current HTML in place if the "foo" form variable doesn't exist.
 2673 
 2674    You may use the python function ``path``, as in
 2675    ``path("item/status")``, to embed path expressions in Python
 2676    expressions.
 2677 
 2678 **String Expressions** - eg. ``string:hello ${user/name}`` 
 2679    These expressions are simple string interpolations - though they can
 2680    be just plain strings with no interpolation if you want. The
 2681    expression in the ``${ ... }`` is just a path expression as above.
 2682 
 2683 **Python Expressions** - eg. ``python: 1+1`` 
 2684    These expressions give the full power of Python. All the "root level"
 2685    variables are available, so ``python:item.status.checklist()`` would
 2686    be equivalent to ``item/status/checklist``, assuming that
 2687    ``checklist`` is a method.
 2688 
 2689 Modifiers:
 2690 
 2691 **structure** - eg. ``structure python:msg.content.plain(hyperlink=1)``
 2692    The result of expressions are normally *escaped* to be safe for HTML
 2693    display (all "<", ">" and "&" are turned into special entities). The
 2694    ``structure`` expression modifier turns off this escaping - the
 2695    result of the expression is now assumed to be HTML, which is passed
 2696    to the web browser for rendering.
 2697 
 2698 **not:** - eg. ``not:python:1=1``
 2699    This simply inverts the logical true/false value of another
 2700    expression.
 2701 
 2702 .. _TALES:
 2703 .. _Template Attribute Language Expression Syntax:
 2704    https://pagetemplates.readthedocs.io/en/latest/history/TALESSpecification13.html
 2705 
 2706 
 2707 Template Macros
 2708 ~~~~~~~~~~~~~~~
 2709 
 2710 Macros are used in Roundup to save us from repeating the same common
 2711 page stuctures over and over. The most common (and probably only) macro
 2712 you'll use is the "icing" macro defined in the "page" template.
 2713 
 2714 Macros are generated and used inside your templates using special
 2715 attributes similar to the `basic templating actions`_. In this case,
 2716 though, the attributes belong to the `Macro Expansion Template
 2717 Attribute Language`_, or METAL. The macro commands are:
 2718 
 2719 **metal:define-macro="macro name"**
 2720   Define that the tag and its contents are now a macro that may be
 2721   inserted into other templates using the *use-macro* command. For
 2722   example::
 2723 
 2724     <html metal:define-macro="page">
 2725      ...
 2726     </html>
 2727 
 2728   defines a macro called "page" using the ``<html>`` tag and its
 2729   contents. Once defined, macros are stored on the template they're
 2730   defined on in the ``macros`` attribute. You can access them later on
 2731   through the ``templates`` variable, eg. the most common
 2732   ``templates/page/macros/icing`` to access the "page" macro of the
 2733   "page" template.
 2734 
 2735 **metal:use-macro="path expression"**
 2736   Use a macro, which is identified by the path expression (see above).
 2737   This will replace the current tag with the identified macro contents.
 2738   For example::
 2739 
 2740    <tal:block metal:use-macro="templates/page/macros/icing">
 2741     ...
 2742    </tal:block>
 2743 
 2744    will replace the tag and its contents with the "page" macro of the
 2745    "page" template.
 2746 
 2747 **metal:define-slot="slot name"** and **metal:fill-slot="slot name"**
 2748   To define *dynamic* parts of the macro, you define "slots" which may
 2749   be filled when the macro is used with a *use-macro* command. For
 2750   example, the ``templates/page/macros/icing`` macro defines a slot like
 2751   so::
 2752 
 2753     <title metal:define-slot="head_title">title goes here</title>
 2754 
 2755   In your *use-macro* command, you may now use a *fill-slot* command
 2756   like this::
 2757 
 2758     <title metal:fill-slot="head_title">My Title</title>
 2759 
 2760   where the tag that fills the slot completely replaces the one defined
 2761   as the slot in the macro.
 2762 
 2763 Note that you may not mix `METAL`_ and `TAL`_ commands on the same tag, but
 2764 TAL commands may be used freely inside METAL-using tags (so your
 2765 *fill-slots* tags may have all manner of TAL inside them).
 2766 
 2767 .. _METAL:
 2768 .. _Macro Expansion Template Attribute Language:
 2769    https://pagetemplates.readthedocs.io/en/latest/history/TALESSpecification13.html
 2770 
 2771 Information available to templates
 2772 ----------------------------------
 2773 
 2774 This is implemented by ``roundup.cgi.templating.RoundupPageTemplate``
 2775 
 2776 The following variables are available to templates.
 2777 
 2778 **context**
 2779   The current context. This is either None, a `hyperdb class wrapper`_
 2780   or a `hyperdb item wrapper`_
 2781 **request**
 2782   Includes information about the current request, including:
 2783    - the current index information (``filterspec``, ``filter`` args,
 2784      ``properties``, etc) parsed out of the form. 
 2785    - methods for easy filterspec link generation
 2786    - "form"
 2787      The current CGI form information as a mapping of form argument name
 2788      to value (specifically a cgi.FieldStorage)
 2789    - "env" the CGI environment variables
 2790    - "base" the base URL for this instance
 2791    - "user" a HTMLItem instance for the current user
 2792    - "language" as determined by the browser or config
 2793    - "classname" the current classname (possibly None)
 2794    - "template" the current template (suffix, also possibly None)
 2795 **config**
 2796   This variable holds all the values defined in the tracker config.ini
 2797   file (eg. TRACKER_NAME, etc.)
 2798 **db**
 2799   The current database, used to access arbitrary database items.
 2800 **templates**
 2801   Access to all the tracker templates by name. Used mainly in
 2802   *use-macro* commands.
 2803 **utils**
 2804   This variable makes available some utility functions like batching.
 2805 **nothing**
 2806   This is a special variable - if an expression evaluates to this, then
 2807   the tag (in the case of a ``tal:replace``), its contents (in the case
 2808   of ``tal:content``) or some attributes (in the case of
 2809   ``tal:attributes``) will not appear in the the output. So, for
 2810   example::
 2811 
 2812     <span tal:attributes="class nothing">Hello, World!</span>
 2813 
 2814   would result in::
 2815 
 2816     <span>Hello, World!</span>
 2817 
 2818 **default**
 2819   Also a special variable - if an expression evaluates to this, then the
 2820   existing HTML in the template will not be replaced or removed, it will
 2821   remain. So::
 2822 
 2823     <span tal:replace="default">Hello, World!</span>
 2824 
 2825   would result in::
 2826 
 2827     <span>Hello, World!</span>
 2828 
 2829 **true**, **false**
 2830   Boolean constants that may be used in `templating expressions`_
 2831   instead of ``python:1`` and ``python:0``.
 2832 **i18n**
 2833   Internationalization service, providing two string translation methods:
 2834 
 2835   **gettext** (*message*)
 2836     Return the localized translation of message
 2837   **ngettext** (*singular*, *plural*, *number*)
 2838     Like ``gettext()``, but consider plural forms. If a translation
 2839     is found, apply the plural formula to *number*, and return the
 2840     resulting message (some languages have more than two plural forms).
 2841     If no translation is found, return singular if *number* is 1;
 2842     return plural otherwise.
 2843 
 2844 The context variable
 2845 ~~~~~~~~~~~~~~~~~~~~
 2846 
 2847 The *context* variable is one of three things based on the current
 2848 context (see `determining web context`_ for how we figure this out):
 2849 
 2850 1. if we're looking at a "home" page, then it's None
 2851 2. if we're looking at a specific hyperdb class, it's a
 2852    `hyperdb class wrapper`_.
 2853 3. if we're looking at a specific hyperdb item, it's a
 2854    `hyperdb item wrapper`_.
 2855 
 2856 If the context is not None, we can access the properties of the class or
 2857 item. The only real difference between cases 2 and 3 above are:
 2858 
 2859 1. the properties may have a real value behind them, and this will
 2860    appear if the property is displayed through ``context/property`` or
 2861    ``context/property/field``.
 2862 2. the context's "id" property will be a false value in the second case,
 2863    but a real, or true value in the third. Thus we can determine whether
 2864    we're looking at a real item from the hyperdb by testing
 2865    "context/id".
 2866 
 2867 Hyperdb class wrapper
 2868 :::::::::::::::::::::
 2869 
 2870 This is implemented by the ``roundup.cgi.templating.HTMLClass``
 2871 class.
 2872 
 2873 This wrapper object provides access to a hyperdb class. It is used
 2874 primarily in both index view and new item views, but it's also usable
 2875 anywhere else that you wish to access information about a class, or the
 2876 items of a class, when you don't have a specific item of that class in
 2877 mind.
 2878 
 2879 We allow access to properties. There will be no "id" property. The value
 2880 accessed through the property will be the current value of the same name
 2881 from the CGI form.
 2882 
 2883 There are several methods available on these wrapper objects:
 2884 
 2885 =========== =============================================================
 2886 Method      Description
 2887 =========== =============================================================
 2888 properties  return a `hyperdb property wrapper`_ for all of this class's
 2889             properties that are searchable by the user. You can use
 2890             the argument cansearch=False to get all properties.
 2891 list        lists all of the active (not retired) items in the class.
 2892 csv         return the items of this class as a chunk of CSV text.
 2893 propnames   lists the names of the properties of this class.
 2894 filter      lists of items from this class, filtered and sorted. Two
 2895             options are available for sorting:
 2896 
 2897             1. by the current *request* filterspec/filter/sort/group args
 2898             2. by the "filterspec", "sort" and "group" keyword args.
 2899                "filterspec" is ``{propname: value(s)}``. "sort" and
 2900                "group" are an optionally empty list ``[(dir, prop)]``
 2901                where dir is '+', '-' or None
 2902                and prop is a prop name or None.
 2903 
 2904                The propname in filterspec and prop in a sort/group spec
 2905                may be transitive, i.e., it may contain properties of
 2906                the form link.link.link.name.
 2907 
 2908             eg. All issues with a priority of "1" with messages added in
 2909             the last week, sorted by activity date:
 2910             ``issue.filter(filterspec={"priority": "1",
 2911             'messages.creation' : '.-1w;'}, sort=[('activity', '+')])``
 2912 
 2913 filter_sql  **Only in SQL backends**
 2914 
 2915             Lists the items that match the SQL provided. The SQL is a
 2916             complete "select" statement.
 2917 
 2918             The SQL select must include the item id as the first column.
 2919 
 2920             This function **does not** filter out retired items, add
 2921             on a where clause "__retired__ <> 1" if you don't want
 2922             retired nodes.
 2923 
 2924 classhelp   display a link to a javascript popup containing this class'
 2925             "help" template.
 2926 
 2927             This generates a link to a popup window which displays the
 2928             properties indicated by "properties" of the class named by
 2929             "classname". The "properties" should be a comma-separated list
 2930             (eg. 'id,name,description'). Properties defaults to all the
 2931             properties of a class (excluding id, creator, created and
 2932             activity).
 2933 
 2934             You may optionally override the "label" displayed, the "width",
 2935             the "height", the number of items per page ("pagesize") and
 2936             the field on which the list is sorted ("sort").
 2937 
 2938             With the "filter" arg it is possible to specify a filter for
 2939             which items are supposed to be displayed. It has to be of
 2940             the format "<field>=<values>;<field>=<values>;...".
 2941 
 2942             The popup window will be resizable and scrollable.
 2943 
 2944             If the "property" arg is given, it's passed through to the
 2945             javascript help_window function. This allows updating of a
 2946             property in the calling HTML page.
 2947 
 2948             If the "form" arg is given, it's passed through to the
 2949             javascript help_window function - it's the name of the form
 2950             the "property" belongs to.
 2951 
 2952 submit      generate a submit button (and action and @csrf hidden elements)
 2953 renderWith  render this class with the given template.
 2954 history     returns 'New node - no history' :)
 2955 is_edit_ok  is the user allowed to Edit the current class?
 2956 is_view_ok  is the user allowed to View the current class?
 2957 =========== =============================================================
 2958 
 2959 Note that if you have a property of the same name as one of the above
 2960 methods, you'll need to access it using a python "item access"
 2961 expression. For example::
 2962 
 2963    python:context['list']
 2964 
 2965 will access the "list" property, rather than the list method.
 2966 
 2967 
 2968 Hyperdb item wrapper
 2969 ::::::::::::::::::::
 2970 
 2971 This is implemented by the ``roundup.cgi.templating.HTMLItem``
 2972 class.
 2973 
 2974 This wrapper object provides access to a hyperdb item.
 2975 
 2976 We allow access to properties. There will be no "id" property. The value
 2977 accessed through the property will be the current value of the same name
 2978 from the CGI form.
 2979 
 2980 There are several methods available on these wrapper objects:
 2981 
 2982 =============== ========================================================
 2983 Method          Description
 2984 =============== ========================================================
 2985 submit          generate a submit button (and action and @csrf hidden elements)
 2986 journal         return the journal of the current item (**not
 2987                 implemented**)
 2988 history         render the journal of the current item as
 2989                 HTML. By default properties marked as "quiet" (see 
 2990 		`design documentation`_) are not shown unless the
 2991 		function is called with the showall=True parameter.
 2992 		Properties that are not Viewable to the user are not
 2993 		shown.
 2994 renderQueryForm specific to the "query" class - render the search form
 2995                 for the query
 2996 hasPermission   specific to the "user" class - determine whether the
 2997                 user has a Permission. The signature is::
 2998 
 2999                     hasPermission(self, permission, [classname=],
 3000                         [property=], [itemid=])
 3001 
 3002                 where the classname defaults to the current context.
 3003 hasRole         specific to the "user" class - determine whether the
 3004                 user has a Role. The signature is::
 3005 
 3006                     hasRole(self, rolename)
 3007 
 3008 is_edit_ok      is the user allowed to Edit the current item?
 3009 is_view_ok      is the user allowed to View the current item?
 3010 is_retired      is the item retired?
 3011 download_url    generate a url-quoted link for download of FileClass
 3012                 item contents (ie. file<id>/<name>)
 3013 copy_url        generate a url-quoted link for creating a copy
 3014                 of this item.  By default, the copy will acquire
 3015                 all properties of the current item except for
 3016                 ``messages`` and ``files``.  This can be overridden
 3017                 by passing ``exclude`` argument which contains a list
 3018                 (or any iterable) of property names that shall not be
 3019                 copied.  Database-driven properties like ``id`` or
 3020                 ``activity`` cannot be copied.
 3021 =============== ========================================================
 3022 
 3023 Note that if you have a property of the same name as one of the above
 3024 methods, you'll need to access it using a python "item access"
 3025 expression. For example::
 3026 
 3027    python:context['journal']
 3028 
 3029 will access the "journal" property, rather than the journal method.
 3030 
 3031 
 3032 Hyperdb property wrapper
 3033 ::::::::::::::::::::::::
 3034 
 3035 This is implemented by subclasses of the
 3036 ``roundup.cgi.templating.HTMLProperty`` class (``HTMLStringProperty``,
 3037 ``HTMLNumberProperty``, and so on).
 3038 
 3039 This wrapper object provides access to a single property of a class. Its
 3040 value may be either:
 3041 
 3042 1. if accessed through a `hyperdb item wrapper`_, then it's a value from
 3043    the hyperdb
 3044 2. if access through a `hyperdb class wrapper`_, then it's a value from
 3045    the CGI form
 3046 
 3047 
 3048 The property wrapper has some useful attributes:
 3049 
 3050 =============== ========================================================
 3051 Attribute       Description
 3052 =============== ========================================================
 3053 _name           the name of the property
 3054 _value          the value of the property if any - this is the actual
 3055                 value retrieved from the hyperdb for this property
 3056 =============== ========================================================
 3057 
 3058 There are several methods available on these wrapper objects:
 3059 
 3060 =========== ================================================================
 3061 Method      Description
 3062 =========== ================================================================
 3063 plain       render a "plain" representation of the property. This method
 3064             may take two arguments:
 3065 
 3066             escape
 3067              If true, escape the text so it is HTML safe (default: no). The
 3068              reason this defaults to off is that text is usually escaped
 3069              at a later stage by the TAL commands, unless the "structure"
 3070              option is used in the template. The following ``tal:content``
 3071              expressions are all equivalent::
 3072  
 3073               "structure python:msg.content.plain(escape=1)"
 3074               "python:msg.content.plain()"
 3075               "msg/content/plain"
 3076               "msg/content"
 3077 
 3078              Usually you'll only want to use the escape option in a
 3079              complex expression.
 3080 
 3081             hyperlink
 3082              If true, turn URLs, email addresses and hyperdb item
 3083              designators in the text into hyperlinks (default: no). Note
 3084              that you'll need to use the "structure" TAL option if you
 3085              want to use this ``tal:content`` expression::
 3086   
 3087               "structure python:msg.content.plain(hyperlink=1)"
 3088 
 3089              The text is automatically HTML-escaped before the hyperlinking
 3090              transformation done in the plain() method.
 3091 
 3092 hyperlinked The same as msg.content.plain(hyperlink=1), but nicer::
 3093 
 3094               "structure msg/content/hyperlinked"
 3095 
 3096 field       render an appropriate form edit field for the property - for
 3097             most types this is a text entry box, but for Booleans it's a
 3098             tri-state yes/no/neither selection. This method may take some
 3099             arguments:
 3100 
 3101             size
 3102               Sets the width in characters of the edit field
 3103 
 3104             format (Date properties only)
 3105               Sets the format of the date in the field - uses the same
 3106               format string argument as supplied to the ``pretty`` method
 3107               below.
 3108 
 3109             popcal (Date properties only)
 3110               Include the Javascript-based popup calendar for date
 3111               selection. Defaults to on.
 3112 
 3113 stext       only on String properties - render the value of the property
 3114             as StructuredText (requires the StructureText module to be
 3115             installed separately)
 3116 multiline   only on String properties - render a multiline form edit
 3117             field for the property
 3118 email       only on String properties - render the value of the property
 3119             as an obscured email address
 3120 url_quote   only on String properties. It quotes any characters in the
 3121             string so it is safe to use in a url. E.G. a space is
 3122             replaced with %20.
 3123 confirm     only on Password properties - render a second form edit field
 3124             for the property, used for confirmation that the user typed
 3125             the password correctly. Generates a field with name
 3126             "name:confirm".
 3127 now         only on Date properties - return the current date as a new
 3128             property
 3129 reldate     only on Date properties - render the interval between the date
 3130             and now
 3131 local       only on Date properties - return this date as a new property
 3132             with some timezone offset, for example::
 3133             
 3134                 python:context.creation.local(10)
 3135 
 3136             will render the date with a +10 hour offset.
 3137 pretty      Date properties - render the date as "dd Mon YYYY" (eg. "19
 3138             Mar 2004"). Takes an optional format argument, for example::
 3139 
 3140                 python:context.activity.pretty('%Y-%m-%d')
 3141 
 3142             Will format as "2004-03-19" instead.
 3143 
 3144             Interval properties - render the interval in a pretty
 3145             format (eg. "yesterday"). The format arguments are those used
 3146             in the standard ``strftime`` call (see the `Python Library
 3147             Reference: time module`__)
 3148 popcal      Generate a link to a popup calendar which may be used to
 3149             edit the date field, for example::
 3150 
 3151               <span tal:replace="structure context/due/popcal" />
 3152 
 3153             you still need to include the ``field`` for the property, so
 3154             typically you'd have::
 3155 
 3156               <span tal:replace="structure context/due/field" />
 3157               <span tal:replace="structure context/due/popcal" />
 3158 
 3159 menu        only on Link and Multilink properties - render a form select
 3160             list for this property. Takes a number of optional arguments
 3161 
 3162             size
 3163                is used to limit the length of the list labels
 3164             height
 3165                is used to set the <select> tag's "size" attribute
 3166             showid
 3167                includes the item ids in the list labels
 3168             additional
 3169                lists properties which should be included in the label
 3170             sort_on
 3171                 indicates the property to sort the list on as (direction,
 3172                 (direction, property) where direction is '+' or '-'. A
 3173                 single string with the direction prepended may be used.
 3174                 For example: ('-', 'order'), '+name'.
 3175             value
 3176                 gives a default value to preselect in the menu
 3177 
 3178             The remaining keyword arguments are used as conditions for
 3179             filtering the items in the list - they're passed as the
 3180             "filterspec" argument to a Class.filter() call. For example::
 3181 
 3182              <span tal:replace="structure context/status/menu" />
 3183 
 3184              <span tal:replace="python:context.status.menu(order='+name",
 3185                                    value='chatting', 
 3186                                    filterspec={'status': '1,2,3,4'}" />
 3187 
 3188 sorted      only on Multilink properties - produce a list of the linked
 3189             items sorted by some property, for example::
 3190             
 3191                 python:context.files.sorted('creation')
 3192 
 3193             Will list the files by upload date. While
 3194 
 3195                python:context.files.sorted('creation', reverse=True)
 3196 
 3197             Will list the files by upload date in reverse order from
 3198             the prior example.
 3199 reverse     only on Multilink properties - produce a list of the linked
 3200             items in reverse order
 3201 isset       returns True if the property has been set to a value
 3202 =========== ================================================================
 3203 
 3204 __ https://docs.python.org/2/library/time.html
 3205 
 3206 All of the above functions perform checks for permissions required to
 3207 display or edit the data they are manipulating. The simplest case is
 3208 editing an issue title. Including the expression::
 3209 
 3210    context/title/field
 3211 
 3212 Will present the user with an edit field, if they have edit permission. If
 3213 not, then they will be presented with a static display if they have view
 3214 permission. If they don't even have view permission, then an error message
 3215 is raised, preventing the display of the page, indicating that they don't
 3216 have permission to view the information.
 3217 
 3218 
 3219 The request variable
 3220 ~~~~~~~~~~~~~~~~~~~~
 3221 
 3222 This is implemented by the ``roundup.cgi.templating.HTMLRequest``
 3223 class.
 3224 
 3225 The request variable is packed with information about the current
 3226 request.
 3227 
 3228 .. taken from ``roundup.cgi.templating.HTMLRequest`` docstring
 3229 
 3230 =========== ============================================================
 3231 Variable    Holds
 3232 =========== ============================================================
 3233 form        the CGI form as a cgi.FieldStorage
 3234 env         the CGI environment variables
 3235 base        the base URL for this tracker
 3236 user        a HTMLUser instance for this user
 3237 classname   the current classname (possibly None)
 3238 template    the current template (suffix, also possibly None)
 3239 form        the current CGI form variables in a FieldStorage
 3240 =========== ============================================================
 3241 
 3242 **Index page specific variables (indexing arguments)**
 3243 
 3244 =========== ============================================================
 3245 Variable    Holds
 3246 =========== ============================================================
 3247 columns     dictionary of the columns to display in an index page
 3248 show        a convenience access to columns - request/show/colname will
 3249             be true if the columns should be displayed, false otherwise
 3250 sort        index sort columns [(direction, column name)]
 3251 group       index grouping properties [(direction, column name)]
 3252 filter      properties to filter the index on
 3253 filterspec  values to filter the index on (property=value, eg
 3254             ``priority=1`` or ``messages.author=42``
 3255 search_text text to perform a full-text search on for an index
 3256 =========== ============================================================
 3257 
 3258 There are several methods available on the request variable:
 3259 
 3260 =============== ========================================================
 3261 Method          Description
 3262 =============== ========================================================
 3263 description     render a description of the request - handle for the
 3264                 page title
 3265 indexargs_form  render the current index args as form elements
 3266 indexargs_url   render the current index args as a URL
 3267 base_javascript render some javascript that is used by other components
 3268                 of the templating
 3269 batch           run the current index args through a filter and return a
 3270                 list of items (see `hyperdb item wrapper`_, and
 3271                 `batching`_)
 3272 =============== ========================================================
 3273 
 3274 The form variable
 3275 :::::::::::::::::
 3276 
 3277 The form variable is a bit special because it's actually a python
 3278 FieldStorage object. That means that you have two ways to access its
 3279 contents. For example, to look up the CGI form value for the variable
 3280 "name", use the path expression::
 3281 
 3282    request/form/name/value
 3283 
 3284 or the python expression::
 3285 
 3286    python:request.form['name'].value
 3287 
 3288 Note the "item" access used in the python case, and also note the
 3289 explicit "value" attribute we have to access. That's because the form
 3290 variables are stored as MiniFieldStorages. If there's more than one
 3291 "name" value in the form, then the above will break since
 3292 ``request/form/name`` is actually a *list* of MiniFieldStorages. So it's
 3293 best to know beforehand what you're dealing with.
 3294 
 3295 
 3296 The db variable
 3297 ~~~~~~~~~~~~~~~
 3298 
 3299 This is implemented by the ``roundup.cgi.templating.HTMLDatabase``
 3300 class.
 3301 
 3302 Allows access to all hyperdb classes as attributes of this variable. If
 3303 you want access to the "user" class, for example, you would use::
 3304 
 3305   db/user
 3306   python:db.user
 3307 
 3308 Also, the current id of the current user is available as
 3309 ``db.getuid()``. This isn't so useful in templates (where you have
 3310 ``request/user``), but it can be useful in detectors or interfaces.
 3311 
 3312 The access results in a `hyperdb class wrapper`_.
 3313 
 3314 
 3315 The templates variable
 3316 ~~~~~~~~~~~~~~~~~~~~~~
 3317 
 3318 This was implemented by the ``roundup.cgi.templating.Templates``
 3319 class before 1.4.20. In later versions it is the instance of appropriate
 3320 template engine loader class.
 3321 
 3322 This variable is used to access other templates in expressions and
 3323 template macros. It doesn't have any useful methods defined. The
 3324 templates can be accessed using the following path expression::
 3325 
 3326    templates/name
 3327 
 3328 or the python expression::
 3329 
 3330    templates[name]
 3331 
 3332 where "name" is the name of the template you wish to access. The
 3333 template has one useful attribute, namely "macros". To access a specific
 3334 macro (called "macro_name"), use the path expression::
 3335 
 3336    templates/name/macros/macro_name
 3337 
 3338 or the python expression::
 3339 
 3340    templates[name].macros[macro_name]
 3341 
 3342 The repeat variable
 3343 ~~~~~~~~~~~~~~~~~~~
 3344 
 3345 The repeat variable holds an entry for each active iteration. That is, if
 3346 you have a ``tal:repeat="user db/users"`` command, then there will be a
 3347 repeat variable entry called "user". This may be accessed as either::
 3348 
 3349     repeat/user
 3350     python:repeat['user']
 3351 
 3352 The "user" entry has a number of methods available for information:
 3353 
 3354 =============== =========================================================
 3355 Method          Description
 3356 =============== =========================================================
 3357 first           True if the current item is the first in the sequence.
 3358 last            True if the current item is the last in the sequence.
 3359 even            True if the current item is an even item in the sequence.
 3360 odd             True if the current item is an odd item in the sequence.
 3361 number          Current position in the sequence, starting from 1.
 3362 letter          Current position in the sequence as a letter, a through
 3363                 z, then aa through zz, and so on.
 3364 Letter          Same as letter(), except uppercase.
 3365 roman           Current position in the sequence as lowercase roman
 3366                 numerals.
 3367 Roman           Same as roman(), except uppercase.
 3368 =============== =========================================================
 3369 
 3370 
 3371 The utils variable
 3372 ~~~~~~~~~~~~~~~~~~
 3373 .. _templating utilities:
 3374 
 3375 This is implemented by the
 3376 ``roundup.cgi.templating.TemplatingUtils`` class, which may be extended
 3377 with additional methods by extensions_.
 3378 
 3379 =============== ========================================================
 3380 Method          Description
 3381 =============== ========================================================
 3382 Batch           return a batch object using the supplied list
 3383 url_quote       quote some text as safe for a URL (ie. space, %, ...)
 3384 html_quote      quote some text as safe in HTML (ie. <, >, ...)
 3385 html_calendar   renders an HTML calendar used by the
 3386                 ``_generic.calendar.html`` template (itself invoked by
 3387                 the popupCalendar DateHTMLProperty method
 3388 =============== ========================================================
 3389 
 3390 
 3391 Batching
 3392 ::::::::
 3393 
 3394 Use Batch to turn a list of items, or item ids of a given class, into a
 3395 series of batches. Its usage is::
 3396 
 3397     python:utils.Batch(sequence, size, start, end=0, orphan=0,
 3398     overlap=0)
 3399 
 3400 or, to get the current index batch::
 3401 
 3402     request/batch
 3403 
 3404 The parameters are:
 3405 
 3406 ========= ==============================================================
 3407 Parameter  Usage
 3408 ========= ==============================================================
 3409 sequence  a list of HTMLItems
 3410 size      how big to make the sequence.
 3411 start     where to start (0-indexed) in the sequence.
 3412 end       where to end (0-indexed) in the sequence.
 3413 orphan    if the next batch would contain less items than this value,
 3414           then it is combined with this batch
 3415 overlap   the number of items shared between adjacent batches
 3416 ========= ==============================================================
 3417 
 3418 All of the parameters are assigned as attributes on the batch object. In
 3419 addition, it has several more attributes:
 3420 
 3421 =============== ========================================================
 3422 Attribute       Description
 3423 =============== ========================================================
 3424 start           indicates the start index of the batch. *Unlike
 3425                 the argument, is a 1-based index (I know, lame)*
 3426 first           indicates the start index of the batch *as a 0-based
 3427                 index*
 3428 length          the actual number of elements in the batch
 3429 sequence_length the length of the original, unbatched, sequence.
 3430 =============== ========================================================
 3431 
 3432 And several methods:
 3433 
 3434 =============== ========================================================
 3435 Method          Description
 3436 =============== ========================================================
 3437 previous        returns a new Batch with the previous batch settings
 3438 next            returns a new Batch with the next batch settings
 3439 propchanged     detect if the named property changed on the current item
 3440                 when compared to the last item
 3441 =============== ========================================================
 3442 
 3443 An example of batching::
 3444 
 3445  <table class="otherinfo">
 3446   <tr><th colspan="4" class="header">Existing Keywords</th></tr>
 3447   <tr tal:define="keywords db/keyword/list"
 3448       tal:repeat="start python:range(0, len(keywords), 4)">
 3449    <td tal:define="batch python:utils.Batch(keywords, 4, start)"
 3450        tal:repeat="keyword batch" tal:content="keyword/name">
 3451        keyword here</td>
 3452   </tr>
 3453  </table>
 3454 
 3455 ... which will produce a table with four columns containing the items of
 3456 the "keyword" class (well, their "name" anyway).
 3457 
 3458 
 3459 Translations
 3460 ~~~~~~~~~~~~
 3461 
 3462 Should you wish to enable multiple languages in template content that you
 3463 create you'll need to add new locale files in the tracker home under a
 3464 ``locale`` directory. Use the instructions in the ``developer's guide`` to
 3465 create the locale files.
 3466 
 3467 
 3468 Displaying Properties
 3469 ---------------------
 3470 
 3471 Properties appear in the user interface in three contexts: in indices,
 3472 in editors, and as search arguments. For each type of property, there
 3473 are several display possibilities. For example, in an index view, a
 3474 string property may just be printed as a plain string, but in an editor
 3475 view, that property may be displayed in an editable field.
 3476 
 3477 
 3478 Index Views
 3479 -----------
 3480 
 3481 This is one of the class context views. It is also the default view for
 3482 classes. The template used is "*classname*.index".
 3483 
 3484 
 3485 Index View Specifiers
 3486 ~~~~~~~~~~~~~~~~~~~~~
 3487 
 3488 An index view specifier (URL fragment) looks like this (whitespace has
 3489 been added for clarity)::
 3490 
 3491     /issue?status=unread,in-progress,resolved&
 3492         keyword=security,ui&
 3493         @group=priority,-status&
 3494         @sort=-activity&
 3495         @filters=status,keyword&
 3496         @columns=title,status,fixer
 3497 
 3498 The index view is determined by two parts of the specifier: the layout
 3499 part and the filter part. The layout part consists of the query
 3500 parameters that begin with colons, and it determines the way that the
 3501 properties of selected items are displayed. The filter part consists of
 3502 all the other query parameters, and it determines the criteria by which
 3503 items are selected for display. The filter part is interactively
 3504 manipulated with the form widgets displayed in the filter section. The
 3505 layout part is interactively manipulated by clicking on the column
 3506 headings in the table.
 3507 
 3508 The filter part selects the union of the sets of items with values
 3509 matching any specified Link properties and the intersection of the sets
 3510 of items with values matching any specified Multilink properties.
 3511 
 3512 The example specifies an index of "issue" items. Only items with a
 3513 "status" of either "unread" or "in-progress" or "resolved" are
 3514 displayed, and only items with "keyword" values including both "security"
 3515 and "ui" are displayed. The items are grouped by priority arranged in
 3516 ascending order and in descending order by status; and within
 3517 groups, sorted by activity, arranged in descending order. The filter
 3518 section shows filters for the "status" and "keyword" properties, and the
 3519 table includes columns for the "title", "status", and "fixer"
 3520 properties.
 3521 
 3522 ============ =============================================================
 3523 Argument     Description
 3524 ============ =============================================================
 3525 @sort        sort by prop name, optionally preceeded with '-' to give
 3526              descending or nothing for ascending sorting. Several
 3527              properties can be specified delimited with comma.
 3528              Internally a search-page using several sort properties may
 3529              use @sort0, @sort1 etc. with option @sortdir0, @sortdir1
 3530              etc. for the direction of sorting (a non-empty value of
 3531              sortdir0 specifies reverse order).
 3532 @group       group by prop name, optionally preceeded with '-' or to sort
 3533              in descending or nothing for ascending order. Several
 3534              properties can be specified delimited with comma.
 3535              Internally a search-page using several grouping properties may
 3536              use @group0, @group1 etc. with option @groupdir0, @groupdir1
 3537              etc. for the direction of grouping (a non-empty value of
 3538              groupdir0 specifies reverse order).
 3539 @columns     selects the columns that should be displayed. Default is
 3540              all.                     
 3541 @filter      indicates which properties are being used in filtering.
 3542              Default is none.
 3543 propname     selects the values the item properties given by propname must
 3544              have (very basic search/filter).
 3545 @search_text if supplied, performs a full-text search (message bodies,
 3546              issue titles, etc)
 3547 ============ =============================================================
 3548 
 3549 
 3550 Searching Views
 3551 ---------------
 3552 
 3553 .. note::
 3554    if you add a new column to the ``@columns`` form variable potentials
 3555    then you will need to add the column to the appropriate `index views`_
 3556    template so that it is actually displayed.
 3557 
 3558 This is one of the class context views. The template used is typically
 3559 "*classname*.search". The form on this page should have "search" as its
 3560 ``@action`` variable. The "search" action:
 3561 
 3562 - sets up additional filtering, as well as performing indexed text
 3563   searching
 3564 - sets the ``@filter`` variable correctly
 3565 - saves the query off if ``@query_name`` is set.
 3566 
 3567 The search page should lay out any fields that you wish to allow the
 3568 user to search on. If your schema contains a large number of properties,
 3569 you should be wary of making all of those properties available for
 3570 searching, as this can cause confusion. If the additional properties are
 3571 Strings, consider having their value indexed, and then they will be
 3572 searchable using the full text indexed search. This is both faster, and
 3573 more useful for the end user.
 3574 
 3575 If the search view does specify the "search" ``@action``, then it may also
 3576 provide an additional argument:
 3577 
 3578 ============ =============================================================
 3579 Argument     Description
 3580 ============ =============================================================
 3581 @query_name  if supplied, the index parameters (including @search_text)
 3582              will be saved off as a the query item and registered against
 3583              the user's queries property. Note that the *classic* template
 3584              schema has this ability, but the *minimal* template schema
 3585              does not.
 3586 ============ =============================================================
 3587 
 3588 
 3589 Item Views
 3590 ----------
 3591 
 3592 The basic view of a hyperdb item is provided by the "*classname*.item"
 3593 template. It generally has three sections; an "editor", a "spool" and a
 3594 "history" section.
 3595 
 3596 
 3597 Editor Section
 3598 ~~~~~~~~~~~~~~
 3599 
 3600 The editor section is used to manipulate the item - it may be a static
 3601 display if the user doesn't have permission to edit the item.
 3602 
 3603 Here's an example of a basic editor template (this is the default
 3604 "classic" template issue item edit form - from the "issue.item.html"
 3605 template)::
 3606 
 3607  <table class="form">
 3608  <tr>
 3609   <th>Title</th>
 3610   <td colspan="3" tal:content="structure python:context.title.field(size=60)">title</td>
 3611  </tr>
 3612  
 3613  <tr>
 3614   <th>Priority</th>
 3615   <td tal:content="structure context/priority/menu">priority</td>
 3616   <th>Status</th>
 3617   <td tal:content="structure context/status/menu">status</td>
 3618  </tr>
 3619  
 3620  <tr>
 3621   <th>Superseder</th>
 3622   <td>
 3623    <span tal:replace="structure python:context.superseder.field(showid=1, size=20)" />
 3624    <span tal:replace="structure python:db.issue.classhelp('id,title')" />
 3625    <span tal:condition="context/superseder">
 3626     <br>View: <span tal:replace="structure python:context.superseder.link(showid=1)" />
 3627    </span>
 3628   </td>
 3629   <th>Nosy List</th>
 3630   <td>
 3631    <span tal:replace="structure context/nosy/field" />
 3632    <span tal:replace="structure python:db.user.classhelp('username,realname,address,phone')" />
 3633   </td>
 3634  </tr>
 3635  
 3636  <tr>
 3637   <th>Assigned To</th>
 3638   <td tal:content="structure context/assignedto/menu">
 3639    assignedto menu
 3640   </td>
 3641   <td>&nbsp;</td>
 3642   <td>&nbsp;</td>
 3643  </tr>
 3644  
 3645  <tr>
 3646   <th>Change Note</th>
 3647   <td colspan="3">
 3648    <textarea name=":note" wrap="hard" rows="5" cols="60"></textarea>
 3649   </td>
 3650  </tr>
 3651  
 3652  <tr>
 3653   <th>File</th>
 3654   <td colspan="3"><input type="file" name=":file" size="40"></td>
 3655  </tr>
 3656  
 3657  <tr>
 3658   <td>&nbsp;</td>
 3659   <td colspan="3" tal:content="structure context/submit">
 3660    submit button will go here
 3661   </td>
 3662  </tr>
 3663  </table>
 3664 
 3665 
 3666 When a change is submitted, the system automatically generates a message
 3667 describing the changed properties. As shown in the example, the editor
 3668 template can use the ":note" and ":file" fields, which are added to the
 3669 standard changenote message generated by Roundup.
 3670 
 3671 
 3672 Form values
 3673 :::::::::::
 3674 
 3675 We have a number of ways to pull properties out of the form in order to
 3676 meet the various needs of:
 3677 
 3678 1. editing the current item (perhaps an issue item)
 3679 2. editing information related to the current item (eg. messages or
 3680    attached files)
 3681 3. creating new information to be linked to the current item (eg. time
 3682    spent on an issue)
 3683 
 3684 In the following, ``<bracketed>`` values are variable, ":" may be one of
 3685 ":" or "@", and other text ("required") is fixed.
 3686 
 3687 Properties are specified as form variables:
 3688 
 3689 ``<propname>``
 3690   property on the current context item
 3691 
 3692 ``<designator>:<propname>``
 3693   property on the indicated item (for editing related information)
 3694 
 3695 ``<classname>-<N>:<propname>``
 3696   property on the Nth new item of classname (generally for creating new
 3697   items to attach to the current item)
 3698 
 3699 Once we have determined the "propname", we check to see if it is one of
 3700 the special form values:
 3701 
 3702 ``@required``
 3703   The named property values must be supplied or a ValueError will be
 3704   raised.
 3705 
 3706 ``@remove@<propname>=id(s)``
 3707   The ids will be removed from the multilink property.
 3708 
 3709 ``:add:<propname>=id(s)``
 3710   The ids will be added to the multilink property.
 3711 
 3712 ``:link:<propname>=<designator>``
 3713   Used to add a link to new items created during edit. These are
 3714   collected and returned in ``all_links``. This will result in an
 3715   additional linking operation (either Link set or Multilink append)
 3716   after the edit/create is done using ``all_props`` in ``_editnodes``.
 3717   The <propname> on the current item will be set/appended the id of the
 3718   newly created item of class <designator> (where <designator> must be
 3719   <classname>-<N>).
 3720 
 3721 Any of the form variables may be prefixed with a classname or
 3722 designator.
 3723 
 3724 Two special form values are supported for backwards compatibility:
 3725 
 3726 ``:note``
 3727   create a message (with content, author and date), linked to the
 3728   context item. This is ALWAYS designated "msg-1".
 3729 ``:file``
 3730   create a file, attached to the current item and any message created by
 3731   :note. This is ALWAYS designated "file-1".
 3732 
 3733 
 3734 Spool Section
 3735 ~~~~~~~~~~~~~
 3736 
 3737 The spool section lists related information like the messages and files
 3738 of an issue.
 3739 
 3740 TODO
 3741 
 3742 
 3743 History Section
 3744 ~~~~~~~~~~~~~~~
 3745 
 3746 The final section displayed is the history of the item - its database
 3747 journal. This is generally generated with the template::
 3748 
 3749  <tal:block tal:replace="structure context/history" />
 3750 
 3751 or::
 3752 
 3753    <tal:block
 3754        tal:replace="structure python:context.history(showall=True)" />
 3755 
 3756 if you want to show history entries for quiet properties.
 3757 
 3758 *To be done:*
 3759 
 3760 *The actual history entries of the item may be accessed for manual
 3761 templating through the "journal" method of the item*::
 3762 
 3763  <tal:block tal:repeat="entry context/journal">
 3764   a journal entry
 3765  </tal:block>
 3766 
 3767 *where each journal entry is an HTMLJournalEntry.*
 3768 
 3769 
 3770 Defining new web actions
 3771 ------------------------
 3772 
 3773 You may define new actions to be triggered by the ``@action`` form variable.
 3774 These are added to the tracker ``extensions`` directory and registered
 3775 using ``instance.registerAction``.
 3776 
 3777 All the existing Actions are defined in ``roundup.cgi.actions``.
 3778 
 3779 Adding action classes takes three steps; first you `define the new
 3780 action class`_, then you `register the action class`_ with the cgi
 3781 interface so it may be triggered by the ``@action`` form variable.
 3782 Finally you `use the new action`_ in your HTML form.
 3783 
 3784 See "`setting up a "wizard" (or "druid") for controlled adding of
 3785 issues`_" for an example.
 3786 
 3787 
 3788 Define the new action class
 3789 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 3790 
 3791 Create a new action class in your tracker's ``extensions`` directory, for
 3792 example ``myaction.py``::
 3793 
 3794  from roundup.cgi.actions import Action
 3795 
 3796  class MyAction(Action):
 3797      def handle(self):
 3798          ''' Perform some action. No return value is required.
 3799          '''
 3800 
 3801 The *self.client* attribute is an instance of ``roundup.cgi.client.Client``.
 3802 See the docstring of that class for details of what it can do.
 3803 
 3804 The method will typically check the ``self.form`` variable's contents.
 3805 It may then:
 3806 
 3807 - add information to ``self.client._ok_message``
 3808   or ``self.client._error_message`` (by using ``self.client.add_ok_message``
 3809   or ``self.client.add_error_message``, respectively)
 3810 - change the ``self.client.template`` variable to alter what the user will see
 3811   next
 3812 - raise Unauthorised, SendStaticFile, SendFile, NotFound or Redirect
 3813   exceptions (import them from roundup.cgi.exceptions)
 3814 
 3815 
 3816 Register the action class
 3817 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 3818 
 3819 The class is now written, but isn't available to the user until you register
 3820 it with the following code appended to your ``myaction.py`` file::
 3821 
 3822     def init(instance):
 3823         instance.registerAction('myaction', myActionClass)
 3824 
 3825 This maps the action name "myaction" to the action class we defined.
 3826 
 3827 
 3828 Use the new action
 3829 ~~~~~~~~~~~~~~~~~~
 3830 
 3831 In your HTML form, add a hidden form element like so::
 3832 
 3833   <input type="hidden" name="@action" value="myaction">
 3834 
 3835 where "myaction" is the name you registered in the previous step.
 3836 
 3837 Actions may return content to the user
 3838 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 3839 
 3840 Actions generally perform some database manipulation and then pass control
 3841 on to the rendering of a template in the current context (see `Determining
 3842 web context`_ for how that works.) Some actions will want to generate the
 3843 actual content returned to the user. Action methods may return their own
 3844 content string to be displayed to the user, overriding the templating step.
 3845 In this situation, we assume that the content is HTML by default. You may
 3846 override the content type indicated to the user by calling ``setHeader``::
 3847 
 3848    self.client.setHeader('Content-Type', 'text/csv')
 3849 
 3850 This example indicates that the value sent back to the user is actually
 3851 comma-separated value content (eg. something to be loaded into a
 3852 spreadsheet or database).
 3853 
 3854 
 3855 8-bit character set support in Web interface
 3856 --------------------------------------------
 3857 
 3858 The web interface uses UTF-8 default. It may be overridden in both forms
 3859 and a browser cookie.
 3860 
 3861 - In forms, use the ``@charset`` variable.
 3862 - To use the cookie override, have the ``roundup_charset`` cookie set.
 3863 
 3864 In both cases, the value is a valid charset name (eg. ``utf-8`` or
 3865 ``kio8-r``).
 3866 
 3867 Inside Roundup, all strings are stored and processed in utf-8.
 3868 Unfortunately, some older browsers do not work properly with
 3869 utf-8-encoded pages (e.g. Netscape Navigator 4 displays wrong
 3870 characters in form fields).  This version allows one to change
 3871 the character set for http transfers.  To do so, you may add
 3872 the following code to your ``page.html`` template::
 3873 
 3874  <tal:block define="uri string:${request/base}${request/env/PATH_INFO}">
 3875   <a tal:attributes="href python:request.indexargs_url(uri,
 3876    {'@charset':'utf-8'})">utf-8</a>
 3877   <a tal:attributes="href python:request.indexargs_url(uri,
 3878    {'@charset':'koi8-r'})">koi8-r</a>
 3879  </tal:block>
 3880 
 3881 (substitute ``koi8-r`` with appropriate charset for your language).
 3882 Charset preference is kept in the browser cookie ``roundup_charset``.
 3883 
 3884 ``meta http-equiv`` lines added to the tracker templates in version 0.6.0
 3885 should be changed to include actual character set name::
 3886 
 3887  <meta http-equiv="Content-Type"
 3888   tal:attributes="content string:text/html;; charset=${request/client/charset}"
 3889  />
 3890 
 3891 The charset is also sent in the http header.
 3892 
 3893 .. _CustomExamples:
 3894 
 3895 Examples
 3896 ========
 3897 
 3898 .. contents::
 3899    :local:
 3900    :depth: 2
 3901 
 3902 
 3903 Changing what's stored in the database
 3904 --------------------------------------
 3905 
 3906 The following examples illustrate ways to change the information stored in
 3907 the database.
 3908 
 3909 
 3910 Adding a new field to the classic schema
 3911 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 3912 
 3913 This example shows how to add a simple field (a due date) to the default
 3914 classic schema. It does not add any additional behaviour, such as enforcing
 3915 the due date, or causing automatic actions to fire if the due date passes.
 3916 
 3917 You add new fields by editing the ``schema.py`` file in you tracker's home.
 3918 Schema changes are automatically applied to the database on the next
 3919 tracker access (note that roundup-server would need to be restarted as it
 3920 caches the schema).
 3921 
 3922 .. index:: schema; example changes
 3923 
 3924 1. Modify the ``schema.py``::
 3925 
 3926     issue = IssueClass(db, "issue", 
 3927                     assignedto=Link("user"), keyword=Multilink("keyword"),
 3928                     priority=Link("priority"), status=Link("status"),
 3929                     due_date=Date())
 3930 
 3931 2. Add an edit field to the ``issue.item.html`` template::
 3932 
 3933     <tr> 
 3934      <th>Due Date</th> 
 3935      <td tal:content="structure context/due_date/field" /> 
 3936     </tr>
 3937     
 3938    If you want to show only the date part of due_date then do this instead::
 3939    
 3940     <tr> 
 3941      <th>Due Date</th> 
 3942      <td tal:content="structure python:context.due_date.field(format='%Y-%m-%d')" /> 
 3943     </tr>
 3944 
 3945 3. Add the property to the ``issue.index.html`` page::
 3946 
 3947     (in the heading row)
 3948       <th tal:condition="request/show/due_date">Due Date</th>
 3949     (in the data row)
 3950       <td tal:condition="request/show/due_date" 
 3951           tal:content="i/due_date" />
 3952           
 3953    If you want format control of the display of the due date you can
 3954    enter the following in the data row to show only the actual due date::
 3955     
 3956       <td tal:condition="request/show/due_date" 
 3957           tal:content="python:i.due_date.pretty('%Y-%m-%d')">&nbsp;</td>
 3958 
 3959 4. Add the property to the ``issue.search.html`` page::
 3960 
 3961      <tr tal:define="name string:due_date">
 3962        <th i18n:translate="">Due Date:</th>
 3963        <td metal:use-macro="search_input"></td>
 3964        <td metal:use-macro="column_input"></td>
 3965        <td metal:use-macro="sort_input"></td>
 3966        <td metal:use-macro="group_input"></td>
 3967      </tr>
 3968 
 3969 5. If you wish for the due date to appear in the standard views listed
 3970    in the sidebar of the web interface then you'll need to add "due_date"
 3971    to the columns and columns_showall lists in your ``page.html``::
 3972     
 3973     columns string:id,activity,due_date,title,creator,status;
 3974     columns_showall string:id,activity,due_date,title,creator,assignedto,status;
 3975 
 3976 Adding a new constrained field to the classic schema
 3977 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 3978 
 3979 This example shows how to add a new constrained property (i.e. a
 3980 selection of distinct values) to your tracker.
 3981 
 3982 
 3983 Introduction
 3984 ::::::::::::
 3985 
 3986 To make the classic schema of Roundup useful as a TODO tracking system
 3987 for a group of systems administrators, it needs an extra data field per
 3988 issue: a category.
 3989 
 3990 This would let sysadmins quickly list all TODOs in their particular area
 3991 of interest without having to do complex queries, and without relying on
 3992 the spelling capabilities of other sysadmins (a losing proposition at
 3993 best).
 3994 
 3995 
 3996 Adding a field to the database
 3997 ::::::::::::::::::::::::::::::
 3998 
 3999 This is the easiest part of the change. The category would just be a
 4000 plain string, nothing fancy. To change what is in the database you need
 4001 to add some lines to the ``schema.py`` file of your tracker instance.
 4002 Under the comment::
 4003 
 4004     # add any additional database schema configuration here
 4005 
 4006 add::
 4007 
 4008     category = Class(db, "category", name=String())
 4009     category.setkey("name")
 4010 
 4011 Here we are setting up a chunk of the database which we are calling
 4012 "category". It contains a string, which we are refering to as "name" for
 4013 lack of a more imaginative title. (Since "name" is one of the properties
 4014 that Roundup looks for on items if you do not set a key for them, it's
 4015 probably a good idea to stick with it for new classes if at all
 4016 appropriate.) Then we are setting the key of this chunk of the database
 4017 to be that "name". This is equivalent to an index for database types.
 4018 This also means that there can only be one category with a given name.
 4019 
 4020 Adding the above lines allows us to create categories, but they're not
 4021 tied to the issues that we are going to be creating. It's just a list of
 4022 categories off on its own, which isn't much use. We need to link it in
 4023 with the issues. To do that, find the lines 
 4024 in ``schema.py`` which set up the "issue" class, and then add a link to
 4025 the category::
 4026 
 4027     issue = IssueClass(db, "issue", ... ,
 4028         category=Multilink("category"), ... )
 4029 
 4030 The ``Multilink()`` means that each issue can have many categories. If
 4031 you were adding something with a one-to-one relationship to issues (such
 4032 as the "assignedto" property), use ``Link()`` instead.
 4033 
 4034 That is all you need to do to change the schema. The rest of the effort
 4035 is fiddling around so you can actually use the new category.
 4036 
 4037 
 4038 Populating the new category class
 4039 :::::::::::::::::::::::::::::::::
 4040 
 4041 If you haven't initialised the database with the
 4042 "``roundup-admin initialise``" command, then you
 4043 can add the following to the tracker ``initial_data.py``
 4044 under the comment::
 4045 
 4046     # add any additional database creation steps here - but only if you
 4047     # haven't initialised the database with the admin "initialise" command
 4048 
 4049 Add::
 4050 
 4051      category = db.getclass('category')
 4052      category.create(name="scipy")
 4053      category.create(name="chaco")
 4054      category.create(name="weave")
 4055 
 4056 .. index:: roundup-admin; create entries in class
 4057 
 4058 If the database has already been initalised, then you need to use the
 4059 ``roundup-admin`` tool::
 4060 
 4061      % roundup-admin -i <tracker home>
 4062      Roundup <version> ready for input.
 4063      Type "help" for help.
 4064      roundup> create category name=scipy
 4065      1
 4066      roundup> create category name=chaco
 4067      2
 4068      roundup> create category name=weave
 4069      3
 4070      roundup> exit...
 4071      There are unsaved changes. Commit them (y/N)? y
 4072 
 4073 
 4074 Setting up security on the new objects
 4075 ::::::::::::::::::::::::::::::::::::::
 4076 
 4077 By default only the admin user can look at and change objects. This
 4078 doesn't suit us, as we want any user to be able to create new categories
 4079 as required, and obviously everyone needs to be able to view the
 4080 categories of issues for it to be useful.
 4081 
 4082 We therefore need to change the security of the category objects. This
 4083 is also done in ``schema.py``.
 4084 
 4085 There are currently two loops which set up permissions and then assign
 4086 them to various roles. Simply add the new "category" to both lists::
 4087 
 4088     # Assign the access and edit permissions for issue, file and message
 4089     # to regular users now
 4090     for cl in 'issue', 'file', 'msg', 'category':
 4091         p = db.security.getPermission('View', cl)
 4092         db.security.addPermissionToRole('User', 'View', cl)
 4093         db.security.addPermissionToRole('User', 'Edit', cl)
 4094         db.security.addPermissionToRole('User', 'Create', cl)
 4095 
 4096 These lines assign the "View" and "Edit" Permissions to the "User" role,
 4097 so that normal users can view and edit "category" objects.
 4098 
 4099 This is all the work that needs to be done for the database. It will
 4100 store categories, and let users view and edit them. Now on to the
 4101 interface stuff.
 4102 
 4103 
 4104 Changing the web left hand frame
 4105 ::::::::::::::::::::::::::::::::
 4106 
 4107 We need to give the users the ability to create new categories, and the
 4108 place to put the link to this functionality is in the left hand function
 4109 bar, under the "Issues" area. The file that defines how this area looks
 4110 is ``html/page.html``, which is what we are going to be editing next.
 4111 
 4112 If you look at this file you can see that it contains a lot of
 4113 "classblock" sections which are chunks of HTML that will be included or
 4114 excluded in the output depending on whether the condition in the
 4115 classblock is met. We are going to add the category code at the end of
 4116 the classblock for the *issue* class::
 4117 
 4118   <p class="classblock"
 4119      tal:condition="python:request.user.hasPermission('View', 'category')">
 4120    <b>Categories</b><br>
 4121    <a tal:condition="python:request.user.hasPermission('Edit', 'category')"
 4122       href="category?@template=item">New Category<br></a>
 4123   </p>
 4124 
 4125 The first two lines is the classblock definition, which sets up a
 4126 condition that only users who have "View" permission for the "category"
 4127 object will have this section included in their output. Next comes a
 4128 plain "Categories" header in bold. Everyone who can view categories will
 4129 get that.
 4130 
 4131 Next comes the link to the editing area of categories. This link will
 4132 only appear if the condition - that the user has "Edit" permissions for
 4133 the "category" objects - is matched. If they do have permission then
 4134 they will get a link to another page which will let the user add new
 4135 categories.
 4136 
 4137 Note that if you have permission to *view* but not to *edit* categories,
 4138 then all you will see is a "Categories" header with nothing underneath
 4139 it. This is obviously not very good interface design, but will do for
 4140 now. I just claim that it is so I can add more links in this section
 4141 later on. However, to fix the problem you could change the condition in
 4142 the classblock statement, so that only users with "Edit" permission
 4143 would see the "Categories" stuff.
 4144 
 4145 
 4146 Setting up a page to edit categories
 4147 ::::::::::::::::::::::::::::::::::::
 4148 
 4149 We defined code in the previous section which let users with the
 4150 appropriate permissions see a link to a page which would let them edit
 4151 conditions. Now we have to write that page.
 4152 
 4153 The link was for the *item* template of the *category* object. This
 4154 translates into Roundup looking for a file called ``category.item.html``
 4155 in the ``html`` tracker directory. This is the file that we are going to
 4156 write now.
 4157 
 4158 First, we add an info tag in a comment which doesn't affect the outcome
 4159 of the code at all, but is useful for debugging. If you load a page in a
 4160 browser and look at the page source, you can see which sections come
 4161 from which files by looking for these comments::
 4162 
 4163     <!-- category.item -->
 4164 
 4165 Next we need to add in the METAL macro stuff so we get the normal page
 4166 trappings::
 4167 
 4168  <tal:block metal:use-macro="templates/page/macros/icing">
 4169   <title metal:fill-slot="head_title">Category editing</title>
 4170   <td class="page-header-top" metal:fill-slot="body_title">
 4171    <h2>Category editing</h2>
 4172   </td>
 4173   <td class="content" metal:fill-slot="content">
 4174 
 4175 Next we need to setup up a standard HTML form, which is the whole
 4176 purpose of this file. We link to some handy javascript which sends the
 4177 form through only once. This is to stop users hitting the send button
 4178 multiple times when they are impatient and thus having the form sent
 4179 multiple times::
 4180 
 4181     <form method="POST" onSubmit="return submit_once()"
 4182           enctype="multipart/form-data">
 4183 
 4184 Next we define some code which sets up the minimum list of fields that
 4185 we require the user to enter. There will be only one field - "name" - so
 4186 they better put something in it, otherwise the whole form is pointless::
 4187 
 4188     <input type="hidden" name="@required" value="name">
 4189 
 4190 To get everything to line up properly we will put everything in a table,
 4191 and put a nice big header on it so the user has an idea what is
 4192 happening::
 4193 
 4194     <table class="form">
 4195      <tr><th class="header" colspan="2">Category</th></tr>
 4196 
 4197 Next, we need the field into which the user is going to enter the new
 4198 category. The ``context.name.field(size=60)`` bit tells Roundup to
 4199 generate a normal HTML field of size 60, and the contents of that field
 4200 will be the "name" variable of the current context (namely "category").
 4201 The upshot of this is that when the user types something in
 4202 to the form, a new category will be created with that name::
 4203 
 4204     <tr>
 4205      <th>Name</th>
 4206      <td tal:content="structure python:context.name.field(size=60)">
 4207      name</td>
 4208     </tr>
 4209 
 4210 Then a submit button so that the user can submit the new category::
 4211 
 4212     <tr>
 4213      <td>&nbsp;</td>
 4214      <td colspan="3" tal:content="structure context/submit">
 4215       submit button will go here
 4216      </td>
 4217     </tr>
 4218 
 4219 The ``context/submit`` bit generates the submit button but also
 4220 generates the @action and @csrf hidden fields. The @action field is
 4221 used to tell roundup how to process the form. The @csrf field provides
 4222 a unique single use token to defend against CSRF attacks. (More about
 4223 anti-csrf measures can be found in ``upgrading.txt``.)
 4224 
 4225 Finally we finish off the tags we used at the start to do the METAL
 4226 stuff::
 4227 
 4228   </td>
 4229  </tal:block>
 4230 
 4231 So putting it all together, and closing the table and form we get::
 4232 
 4233  <!-- category.item -->
 4234  <tal:block metal:use-macro="templates/page/macros/icing">
 4235   <title metal:fill-slot="head_title">Category editing</title>
 4236   <td class="page-header-top" metal:fill-slot="body_title">
 4237    <h2>Category editing</h2>
 4238   </td>
 4239   <td class="content" metal:fill-slot="content">
 4240    <form method="POST" onSubmit="return submit_once()"
 4241          enctype="multipart/form-data">
 4242 
 4243     <table class="form">
 4244      <tr><th class="header" colspan="2">Category</th></tr>
 4245 
 4246      <tr>
 4247       <th>Name</th>
 4248       <td tal:content="structure python:context.name.field(size=60)">
 4249       name</td>
 4250      </tr>
 4251 
 4252      <tr>
 4253       <td>
 4254         &nbsp;
 4255         <input type="hidden" name="@required" value="name"> 
 4256       </td>
 4257       <td colspan="3" tal:content="structure context/submit">
 4258        submit button will go here
 4259       </td>
 4260      </tr>
 4261     </table>
 4262    </form>
 4263   </td>
 4264  </tal:block>
 4265 
 4266 This is quite a lot to just ask the user one simple question, but there
 4267 is a lot of setup for basically one line (the form line) to do its work.
 4268 To add another field to "category" would involve one more line (well,
 4269 maybe a few extra to get the formatting correct).
 4270 
 4271 
 4272 Adding the category to the issue
 4273 ::::::::::::::::::::::::::::::::
 4274 
 4275 We now have the ability to create issues to our heart's content, but
 4276 that is pointless unless we can assign categories to issues.  Just like
 4277 the ``html/category.item.html`` file was used to define how to add a new
 4278 category, the ``html/issue.item.html`` is used to define how a new issue
 4279 is created.
 4280 
 4281 Just like ``category.issue.html``, this file defines a form which has a
 4282 table to lay things out. It doesn't matter where in the table we add new
 4283 stuff, it is entirely up to your sense of aesthetics::
 4284 
 4285    <th>Category</th>
 4286    <td>
 4287     <span tal:replace="structure context/category/field" />
 4288     <span tal:replace="structure python:db.category.classhelp('name',
 4289                 property='category', width='200')" />
 4290    </td>
 4291 
 4292 First, we define a nice header so that the user knows what the next
 4293 section is, then the middle line does what we are most interested in.
 4294 This ``context/category/field`` gets replaced by a field which contains
 4295 the category in the current context (the current context being the new
 4296 issue).
 4297 
 4298 The classhelp lines generate a link (labelled "list") to a popup window
 4299 which contains the list of currently known categories.
 4300 
 4301 
 4302 Searching on categories
 4303 :::::::::::::::::::::::
 4304 
 4305 Now we can add categories, and create issues with categories. The next
 4306 obvious thing that we would like to be able to do, would be to search
 4307 for issues based on their category, so that, for example, anyone working
 4308 on the web server could look at all issues in the category "Web".
 4309 
 4310 If you look for "Search Issues" in the ``html/page.html`` file, you will
 4311 find that it looks something like 
 4312 ``<a href="issue?@template=search">Search Issues</a>``. This shows us
 4313 that when you click on "Search Issues" it will be looking for a
 4314 ``issue.search.html`` file to display. So that is the file that we will
 4315 change.
 4316 
 4317 If you look at this file it should begin to seem familiar, although it
 4318 does use some new macros. You can add the new category search code anywhere you
 4319 like within that form::
 4320 
 4321   <tr tal:define="name string:category;
 4322                   db_klass string:category;
 4323                   db_content string:name;">
 4324     <th>Priority:</th>
 4325     <td metal:use-macro="search_select"></td>
 4326     <td metal:use-macro="column_input"></td>
 4327     <td metal:use-macro="sort_input"></td>
 4328     <td metal:use-macro="group_input"></td>
 4329   </tr>
 4330 
 4331 The definitions in the ``<tr>`` opening tag are used by the macros:
 4332 
 4333 - ``search_select`` expands to a drop-down box with all categories using
 4334   ``db_klass`` and ``db_content``.
 4335 - ``column_input`` expands to a checkbox for selecting what columns
 4336   should be displayed.
 4337 - ``sort_input`` expands to a radio button for selecting what property
 4338   should be sorted on.
 4339 - ``group_input`` expands to a radio button for selecting what property
 4340   should be grouped on.
 4341 
 4342 The category search code above would expand to the following::
 4343 
 4344   <tr>
 4345     <th>Category:</th>
 4346     <td>
 4347       <select name="category">
 4348         <option value="">don't care</option>
 4349         <option value="">------------</option>      
 4350         <option value="1">scipy</option>
 4351         <option value="2">chaco</option>
 4352         <option value="3">weave</option>
 4353       </select>
 4354     </td>
 4355     <td><input type="checkbox" name=":columns" value="category"></td>
 4356     <td><input type="radio" name=":sort0" value="category"></td>
 4357     <td><input type="radio" name=":group0" value="category"></td>
 4358   </tr>
 4359 
 4360 Adding category to the default view
 4361 :::::::::::::::::::::::::::::::::::
 4362 
 4363 We can now add categories, add issues with categories, and search for
 4364 issues based on categories. This is everything that we need to do;
 4365 however, there is some more icing that we would like. I think the
 4366 category of an issue is important enough that it should be displayed by
 4367 default when listing all the issues.
 4368 
 4369 Unfortunately, this is a bit less obvious than the previous steps. The
 4370 code defining how the issues look is in ``html/issue.index.html``. This
 4371 is a large table with a form down at the bottom for redisplaying and so
 4372 forth. 
 4373 
 4374 Firstly we need to add an appropriate header to the start of the table::
 4375 
 4376     <th tal:condition="request/show/category">Category</th>
 4377 
 4378 The *condition* part of this statement is to avoid displaying the
 4379 Category column if the user has selected not to see it.
 4380 
 4381 The rest of the table is a loop which will go through every issue that
 4382 matches the display criteria. The loop variable is "i" - which means
 4383 that every issue gets assigned to "i" in turn.
 4384 
 4385 The new part of code to display the category will look like this::
 4386 
 4387     <td tal:condition="request/show/category"
 4388         tal:content="i/category"></td>
 4389 
 4390 The condition is the same as above: only display the condition when the
 4391 user hasn't asked for it to be hidden. The next part is to set the
 4392 content of the cell to be the category part of "i" - the current issue.
 4393 
 4394 Finally we have to edit ``html/page.html`` again. This time, we need to
 4395 tell it that when the user clicks on "Unassigned Issues" or "All Issues",
 4396 the category column should be included in the resulting list. If you
 4397 scroll down the page file, you can see the links with lots of options.
 4398 The option that we are interested in is the ``:columns=`` one which
 4399 tells roundup which fields of the issue to display. Simply add
 4400 "category" to that list and it all should work.
 4401 
 4402 Adding a time log to your issues
 4403 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 4404 
 4405 We want to log the dates and amount of time spent working on issues, and
 4406 be able to give a summary of the total time spent on a particular issue.
 4407 
 4408 1. Add a new class to your tracker ``schema.py``::
 4409 
 4410     # storage for time logging
 4411     timelog = Class(db, "timelog", period=Interval())
 4412 
 4413    Note that we automatically get the date of the time log entry
 4414    creation through the standard property "creation".
 4415 
 4416    You will need to grant "Creation" permission to the users who are
 4417    allowed to add timelog entries. You may do this with::
 4418 
 4419     db.security.addPermissionToRole('User', 'Create', 'timelog')
 4420     db.security.addPermissionToRole('User', 'View', 'timelog')
 4421 
 4422    If users are also able to *edit* timelog entries, then also include::
 4423 
 4424     db.security.addPermissionToRole('User', 'Edit', 'timelog')
 4425 
 4426 .. index:: schema; example changes
 4427 
 4428 2. Link to the new class from your issue class (again, in
 4429    ``schema.py``)::
 4430 
 4431     issue = IssueClass(db, "issue", 
 4432                     assignedto=Link("user"), keyword=Multilink("keyword"),
 4433                     priority=Link("priority"), status=Link("status"),
 4434                     times=Multilink("timelog"))
 4435 
 4436    the "times" property is the new link to the "timelog" class.
 4437 
 4438 3. We'll need to let people add in times to the issue, so in the web
 4439    interface we'll have a new entry field. This is a special field
 4440    because unlike the other fields in the ``issue.item`` template, it
 4441    affects a different item (a timelog item) and not the template's
 4442    item (an issue). We have a special syntax for form fields that affect
 4443    items other than the template default item (see the cgi 
 4444    documentation on `special form variables`_). In particular, we add a
 4445    field to capture a new timelog item's period::
 4446 
 4447     <tr> 
 4448      <th>Time Log</th> 
 4449      <td colspan=3><input type="text" name="timelog-1@period" /> 
 4450       (enter as '3y 1m 4d 2:40:02' or parts thereof) 
 4451      </td> 
 4452     </tr> 
 4453          
 4454    and another hidden field that links that new timelog item (new
 4455    because it's marked as having id "-1") to the issue item. It looks
 4456    like this::
 4457 
 4458      <input type="hidden" name="@link@times" value="timelog-1" />
 4459 
 4460    On submission, the "-1" timelog item will be created and assigned a
 4461    real item id. The "times" property of the issue will have the new id
 4462    added to it.
 4463    
 4464    The full entry will now look like this::
 4465    
 4466     <tr> 
 4467      <th>Time Log</th> 
 4468      <td colspan=3><input type="text" name="timelog-1@period" /> 
 4469       (enter as '3y 1m 4d 2:40:02' or parts thereof)
 4470       <input type="hidden" name="@link@times" value="timelog-1" /> 
 4471      </td> 
 4472     </tr> 
 4473    
 4474 
 4475 4. We want to display a total of the timelog times that have been
 4476    accumulated for an issue. To do this, we'll need to actually write
 4477    some Python code, since it's beyond the scope of PageTemplates to
 4478    perform such calculations. We do this by adding a module ``timespent.py``
 4479    to the ``extensions`` directory in our tracker. The contents of this
 4480    file is as follows::
 4481 
 4482     from roundup import date
 4483 
 4484     def totalTimeSpent(times):
 4485         ''' Call me with a list of timelog items (which have an
 4486             Interval "period" property)
 4487         '''
 4488         total = date.Interval('0d')
 4489         for time in times:
 4490             total += time.period._value
 4491         return total
 4492 
 4493     def init(instance):
 4494         instance.registerUtil('totalTimeSpent', totalTimeSpent)
 4495 
 4496    We will now be able to access the ``totalTimeSpent`` function via the
 4497    ``utils`` variable in our templates, as shown in the next step.
 4498 
 4499 5. Display the timelog for an issue::
 4500 
 4501      <table class="otherinfo" tal:condition="context/times">
 4502       <tr><th colspan="3" class="header">Time Log
 4503        <tal:block
 4504             tal:replace="python:utils.totalTimeSpent(context.times)" />
 4505       </th></tr>
 4506       <tr><th>Date</th><th>Period</th><th>Logged By</th></tr>
 4507       <tr tal:repeat="time context/times">
 4508        <td tal:content="time/creation"></td>
 4509        <td tal:content="time/period"></td>
 4510        <td tal:content="time/creator"></td>
 4511       </tr>
 4512      </table>
 4513 
 4514    I put this just above the Messages log in my issue display. Note our
 4515    use of the ``totalTimeSpent`` method which will total up the times
 4516    for the issue and return a new Interval. That will be automatically
 4517    displayed in the template as text like "+ 1y 2:40" (1 year, 2 hours
 4518    and 40 minutes).
 4519 
 4520 6. If you're using a persistent web server - ``roundup-server`` or
 4521    ``mod_wsgi`` for example - then you'll need to restart that to pick up
 4522    the code changes. When that's done, you'll be able to use the new
 4523    time logging interface.
 4524 
 4525 An extension of this modification attaches the timelog entries to any
 4526 change message entered at the time of the timelog entry:
 4527 
 4528 A. Add a link to the timelog to the msg class in ``schema.py``:
 4529 
 4530     msg = FileClass(db, "msg",
 4531                     author=Link("user", do_journal='no'),
 4532                     recipients=Multilink("user", do_journal='no'),
 4533                     date=Date(),
 4534                     summary=String(),
 4535                     files=Multilink("file"),
 4536                     messageid=String(),
 4537                     inreplyto=String(),
 4538                     times=Multilink("timelog"))
 4539 
 4540 B. Add a new hidden field that links that new timelog item (new
 4541    because it's marked as having id "-1") to the new message.
 4542    The link is placed in ``issue.item.html`` in the same section that
 4543    handles the timelog entry.
 4544    
 4545    It looks like this after this addition::
 4546 
 4547     <tr> 
 4548      <th>Time Log</th> 
 4549      <td colspan=3><input type="text" name="timelog-1@period" /> 
 4550       (enter as '3y 1m 4d 2:40:02' or parts thereof)
 4551       <input type="hidden" name="@link@times" value="timelog-1" />
 4552       <input type="hidden" name="msg-1@link@times" value="timelog-1" /> 
 4553      </td> 
 4554     </tr> 
 4555  
 4556    The "times" property of the message will have the new id added to it.
 4557 
 4558 C. Add the timelog listing from step 5. to the ``msg.item.html`` template
 4559    so that the timelog entry appears on the message view page. Note that
 4560    the call to totalTimeSpent is not used here since there will only be one
 4561    single timelog entry for each message.
 4562    
 4563    I placed it after the Date entry like this::
 4564    
 4565     <tr>
 4566      <th i18n:translate="">Date:</th>
 4567      <td tal:content="context/date"></td>
 4568     </tr>
 4569     </table>
 4570     
 4571     <table class="otherinfo" tal:condition="context/times">
 4572      <tr><th colspan="3" class="header">Time Log</th></tr>
 4573      <tr><th>Date</th><th>Period</th><th>Logged By</th></tr>
 4574      <tr tal:repeat="time context/times">
 4575       <td tal:content="time/creation"></td>
 4576       <td tal:content="time/period"></td>
 4577       <td tal:content="time/creator"></td>
 4578      </tr>
 4579     </table>
 4580     
 4581     <table class="messages">
 4582 
 4583 
 4584 Tracking different types of issues
 4585 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 4586 
 4587 Sometimes you will want to track different types of issues - developer,
 4588 customer support, systems, sales leads, etc. A single Roundup tracker is
 4589 able to support multiple types of issues. This example demonstrates adding
 4590 a system support issue class to a tracker.
 4591 
 4592 1. Figure out what information you're going to want to capture. OK, so
 4593    this is obvious, but sometimes it's better to actually sit down for a
 4594    while and think about the schema you're going to implement.
 4595 
 4596 2. Add the new issue class to your tracker's ``schema.py``. Just after the
 4597    "issue" class definition, add::
 4598 
 4599     # list our systems
 4600     system = Class(db, "system", name=String(), order=Number())
 4601     system.setkey("name")
 4602 
 4603     # store issues related to those systems
 4604     support = IssueClass(db, "support", 
 4605                     assignedto=Link("user"), keyword=Multilink("keyword"),
 4606                     status=Link("status"), deadline=Date(),
 4607                     affects=Multilink("system"))
 4608 
 4609 3. Copy the existing ``issue.*`` (item, search and index) templates in the
 4610    tracker's ``html`` to ``support.*``. Edit them so they use the properties
 4611    defined in the ``support`` class. Be sure to check for hidden form
 4612    variables like "required" to make sure they have the correct set of
 4613    required properties.
 4614 
 4615 4. Edit the modules in the ``detectors``, adding lines to their ``init``
 4616    functions where appropriate. Look for ``audit`` and ``react`` registrations
 4617    on the ``issue`` class, and duplicate them for ``support``.
 4618 
 4619 5. Create a new sidebar box for the new support class. Duplicate the
 4620    existing issues one, changing the ``issue`` class name to ``support``.
 4621 
 4622 6. Re-start your tracker and start using the new ``support`` class.
 4623 
 4624 
 4625 Optionally, you might want to restrict the users able to access this new
 4626 class to just the users with a new "SysAdmin" Role. To do this, we add
 4627 some security declarations::
 4628 
 4629     db.security.addPermissionToRole('SysAdmin', 'View', 'support')
 4630     db.security.addPermissionToRole('SysAdmin', 'Create', 'support')
 4631     db.security.addPermissionToRole('SysAdmin', 'Edit', 'support')
 4632 
 4633 You would then (as an "admin" user) edit the details of the appropriate
 4634 users, and add "SysAdmin" to their Roles list.
 4635 
 4636 Alternatively, you might want to change the Edit/View permissions granted
 4637 for the ``issue`` class so that it's only available to users with the "System"
 4638 or "Developer" Role, and then the new class you're adding is available to
 4639 all with the "User" Role.
 4640 
 4641 
 4642 .. _external-authentication:
 4643 
 4644 Using External User Databases
 4645 -----------------------------
 4646 
 4647 Using an external password validation source
 4648 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 4649 
 4650 .. note:: You will need to either have an "admin" user in your external
 4651           password source *or* have one of your regular users have
 4652           the Admin Role assigned. If you need to assign the Role *after*
 4653           making the changes below, you may use the ``roundup-admin``
 4654           program to edit a user's details.
 4655 
 4656 We have a centrally-managed password changing system for our users. This
 4657 results in a UN*X passwd-style file that we use for verification of
 4658 users. Entries in the file consist of ``name:password`` where the
 4659 password is encrypted using the standard UN*X ``crypt()`` function (see
 4660 the ``crypt`` module in your Python distribution). An example entry
 4661 would be::
 4662 
 4663     admin:aamrgyQfDFSHw
 4664 
 4665 Each user of Roundup must still have their information stored in the Roundup
 4666 database - we just use the passwd file to check their password. To do this, we
 4667 need to override the standard ``verifyPassword`` method defined in
 4668 ``roundup.cgi.actions.LoginAction`` and register the new class. The
 4669 following is added as ``externalpassword.py`` in the tracker ``extensions``
 4670 directory::
 4671 
 4672     import os, crypt
 4673     from roundup.cgi.actions import LoginAction    
 4674 
 4675     class ExternalPasswordLoginAction(LoginAction):
 4676         def verifyPassword(self, userid, password):
 4677             '''Look through the file, line by line, looking for a
 4678             name that matches.
 4679             '''
 4680             # get the user's username
 4681             username = self.db.user.get(userid, 'username')
 4682 
 4683             # the passwords are stored in the "passwd.txt" file in the
 4684             # tracker home
 4685             file = os.path.join(self.db.config.TRACKER_HOME, 'passwd.txt')
 4686 
 4687             # see if we can find a match
 4688             for ent in [line.strip().split(':') for line in
 4689                                                 open(file).readlines()]:
 4690                 if ent[0] == username:
 4691                     return crypt.crypt(password, ent[1][:2]) == ent[1]
 4692 
 4693             # user doesn't exist in the file
 4694             return 0
 4695 
 4696     def init(instance):
 4697         instance.registerAction('login', ExternalPasswordLoginAction)
 4698 
 4699 You should also remove the redundant password fields from the ``user.item``
 4700 template.
 4701 
 4702 
 4703 Using a UN*X passwd file as the user database
 4704 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 4705 
 4706 On some systems the primary store of users is the UN*X passwd file. It
 4707 holds information on users such as their username, real name, password
 4708 and primary user group.
 4709 
 4710 Roundup can use this store as its primary source of user information,
 4711 but it needs additional information too - email address(es), roundup
 4712 Roles, vacation flags, roundup hyperdb item ids, etc. Also, "retired"
 4713 users must still exist in the user database, unlike some passwd files in
 4714 which the users are removed when they no longer have access to a system.
 4715 
 4716 To make use of the passwd file, we therefore synchronise between the two
 4717 user stores. We also use the passwd file to validate the user logins, as
 4718 described in the previous example, `using an external password
 4719 validation source`_. We keep the user lists in sync using a fairly
 4720 simple script that runs once a day, or several times an hour if more
 4721 immediate access is needed. In short, it:
 4722 
 4723 1. parses the passwd file, finding usernames, passwords and real names,
 4724 2. compares that list to the current roundup user list:
 4725 
 4726    a. entries no longer in the passwd file are *retired*
 4727    b. entries with mismatching real names are *updated*
 4728    c. entries only exist in the passwd file are *created*
 4729 
 4730 3. send an email to administrators to let them know what's been done.
 4731 
 4732 The retiring and updating are simple operations, requiring only a call
 4733 to ``retire()`` or ``set()``. The creation operation requires more
 4734 information though - the user's email address and their Roundup Roles.
 4735 We're going to assume that the user's email address is the same as their
 4736 login name, so we just append the domain name to that. The Roles are
 4737 determined using the passwd group identifier - mapping their UN*X group
 4738 to an appropriate set of Roles.
 4739 
 4740 The script to perform all this, broken up into its main components, is
 4741 as follows. Firstly, we import the necessary modules and open the
 4742 tracker we're to work on::
 4743 
 4744     import sys, os, smtplib
 4745     from roundup import instance, date
 4746 
 4747     # open the tracker
 4748     tracker_home = sys.argv[1]
 4749     tracker = instance.open(tracker_home)
 4750 
 4751 Next we read in the *passwd* file from the tracker home::
 4752 
 4753     # read in the users from the "passwd.txt" file
 4754     file = os.path.join(tracker_home, 'passwd.txt')
 4755     users = [x.strip().split(':') for x in open(file).readlines()]
 4756 
 4757 Handle special users (those to ignore in the file, and those who don't
 4758 appear in the file)::
 4759 
 4760     # users to not keep ever, pre-load with the users I know aren't
 4761     # "real" users
 4762     ignore = ['ekmmon', 'bfast', 'csrmail']
 4763 
 4764     # users to keep - pre-load with the roundup-specific users
 4765     keep = ['comment_pool', 'network_pool', 'admin', 'dev-team',
 4766             'cs_pool', 'anonymous', 'system_pool', 'automated']
 4767 
 4768 Now we map the UN*X group numbers to the Roles that users should have::
 4769 
 4770     roles = {
 4771      '501': 'User,Tech',  # tech
 4772      '502': 'User',       # finance
 4773      '503': 'User,CSR',   # customer service reps
 4774      '504': 'User',       # sales
 4775      '505': 'User',       # marketing
 4776     }
 4777 
 4778 Now we do all the work. Note that the body of the script (where we have
 4779 the tracker database open) is wrapped in a ``try`` / ``finally`` clause,
 4780 so that we always close the database cleanly when we're finished. So, we
 4781 now do all the work::
 4782 
 4783     # open the database
 4784     db = tracker.open('admin')
 4785     try:
 4786         # store away messages to send to the tracker admins
 4787         msg = []
 4788 
 4789         # loop over the users list read in from the passwd file
 4790         for user,passw,uid,gid,real,home,shell in users:
 4791             if user in ignore:
 4792                 # this user shouldn't appear in our tracker
 4793                 continue
 4794             keep.append(user)
 4795             try:
 4796                 # see if the user exists in the tracker
 4797                 uid = db.user.lookup(user)
 4798 
 4799                 # yes, they do - now check the real name for correctness
 4800                 if real != db.user.get(uid, 'realname'):
 4801                     db.user.set(uid, realname=real)
 4802                     msg.append('FIX %s - %s'%(user, real))
 4803             except KeyError:
 4804                 # nope, the user doesn't exist
 4805                 db.user.create(username=user, realname=real,
 4806                     address='%s@ekit-inc.com'%user, roles=roles[gid])
 4807                 msg.append('ADD %s - %s (%s)'%(user, real, roles[gid]))
 4808 
 4809         # now check that all the users in the tracker are also in our
 4810         # "keep" list - retire those who aren't
 4811         for uid in db.user.list():
 4812             user = db.user.get(uid, 'username')
 4813             if user not in keep:
 4814                 db.user.retire(uid)
 4815                 msg.append('RET %s'%user)
 4816 
 4817         # if we did work, then send email to the tracker admins
 4818         if msg:
 4819             # create the email
 4820             msg = '''Subject: %s user database maintenance
 4821 
 4822             %s
 4823             '''%(db.config.TRACKER_NAME, '\n'.join(msg))
 4824 
 4825             # send the email
 4826             smtp = smtplib.SMTP(db.config.MAILHOST)
 4827             addr = db.config.ADMIN_EMAIL
 4828             smtp.sendmail(addr, addr, msg)
 4829 
 4830         # now we're done - commit the changes
 4831         db.commit()
 4832     finally:
 4833         # always close the database cleanly
 4834         db.close()
 4835 
 4836 And that's it!
 4837 
 4838 
 4839 Using an LDAP database for user information
 4840 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 4841 
 4842 A script that reads users from an LDAP store using
 4843 https://pypi.org/project/python-ldap/ and then compares the list to the users in the
 4844 roundup user database would be pretty easy to write. You'd then have it run
 4845 once an hour / day (or on demand if you can work that into your LDAP store
 4846 workflow). See the example `Using a UN*X passwd file as the user database`_
 4847 for more information about doing this.
 4848 
 4849 To authenticate off the LDAP store (rather than using the passwords in the
 4850 Roundup user database) you'd use the same python-ldap module inside an
 4851 extension to the cgi interface. You'd do this by overriding the method called
 4852 ``verifyPassword`` on the ``LoginAction`` class in your tracker's
 4853 ``extensions`` directory (see `using an external password validation
 4854 source`_). The method is implemented by default as::
 4855 
 4856     def verifyPassword(self, userid, password):
 4857         ''' Verify the password that the user has supplied
 4858         '''
 4859         stored = self.db.user.get(self.userid, 'password')
 4860         if password == stored:
 4861             return 1
 4862         if not password and not stored:
 4863             return 1
 4864         return 0
 4865 
 4866 So you could reimplement this as something like::
 4867 
 4868     def verifyPassword(self, userid, password):
 4869         ''' Verify the password that the user has supplied
 4870         '''
 4871         # look up some unique LDAP information about the user
 4872         username = self.db.user.get(self.userid, 'username')
 4873         # now verify the password supplied against the LDAP store
 4874 
 4875 
 4876 Changes to Tracker Behaviour
 4877 ----------------------------
 4878 
 4879 .. index:: single: auditors; how to register (example)
 4880    single: reactors; how to register (example)
 4881 
 4882 Preventing SPAM
 4883 ~~~~~~~~~~~~~~~
 4884 
 4885 The following detector code may be installed in your tracker's
 4886 ``detectors`` directory. It will block any messages being created that
 4887 have HTML attachments (a very common vector for spam and phishing)
 4888 and any messages that have more than 2 HTTP URLs in them. Just copy
 4889 the following into ``detectors/anti_spam.py`` in your tracker::
 4890 
 4891     from roundup.exceptions import Reject
 4892 
 4893     def reject_html(db, cl, nodeid, newvalues):
 4894         if newvalues['type'] == 'text/html':
 4895         raise Reject('not allowed')
 4896 
 4897     def reject_manylinks(db, cl, nodeid, newvalues):
 4898         content = newvalues['content']
 4899         if content.count('http://') > 2:
 4900         raise Reject('not allowed')
 4901 
 4902     def init(db):
 4903         db.file.audit('create', reject_html)
 4904         db.msg.audit('create', reject_manylinks)
 4905 
 4906 You may also wish to block image attachments if your tracker does not
 4907 need that ability::
 4908 
 4909     if newvalues['type'].startswith('image/'):
 4910         raise Reject('not allowed')
 4911 
 4912 
 4913 Stop "nosy" messages going to people on vacation
 4914 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 4915 
 4916 When users go on vacation and set up vacation email bouncing, you'll
 4917 start to see a lot of messages come back through Roundup "Fred is on
 4918 vacation". Not very useful, and relatively easy to stop.
 4919 
 4920 1. add a "vacation" flag to your users::
 4921 
 4922          user = Class(db, "user",
 4923                     username=String(),   password=Password(),
 4924                     address=String(),    realname=String(),
 4925                     phone=String(),      organisation=String(),
 4926                     alternate_addresses=String(),
 4927                     roles=String(), queries=Multilink("query"),
 4928                     vacation=Boolean())
 4929 
 4930 2. So that users may edit the vacation flags, add something like the
 4931    following to your ``user.item`` template::
 4932 
 4933      <tr>
 4934       <th>On Vacation</th> 
 4935       <td tal:content="structure context/vacation/field">vacation</td> 
 4936      </tr> 
 4937 
 4938 3. edit your detector ``nosyreactor.py`` so that the ``nosyreaction()``
 4939    consists of::
 4940 
 4941     def nosyreaction(db, cl, nodeid, oldvalues):
 4942         users = db.user
 4943         messages = db.msg
 4944         # send a copy of all new messages to the nosy list
 4945         for msgid in determineNewMessages(cl, nodeid, oldvalues):
 4946             try:
 4947                 # figure the recipient ids
 4948                 sendto = []
 4949                 seen_message = {}
 4950                 recipients = messages.get(msgid, 'recipients')
 4951                 for recipid in messages.get(msgid, 'recipients'):
 4952                     seen_message[recipid] = 1
 4953 
 4954                 # figure the author's id, and indicate they've received
 4955                 # the message
 4956                 authid = messages.get(msgid, 'author')
 4957 
 4958                 # possibly send the message to the author, as long as
 4959                 # they aren't anonymous
 4960                 if (db.config.MESSAGES_TO_AUTHOR == 'yes' and
 4961                         users.get(authid, 'username') != 'anonymous'):
 4962                     sendto.append(authid)
 4963                 seen_message[authid] = 1
 4964 
 4965                 # now figure the nosy people who weren't recipients
 4966                 nosy = cl.get(nodeid, 'nosy')
 4967                 for nosyid in nosy:
 4968                     # Don't send nosy mail to the anonymous user (that
 4969                     # user shouldn't appear in the nosy list, but just
 4970                     # in case they do...)
 4971                     if users.get(nosyid, 'username') == 'anonymous':
 4972                         continue
 4973                     # make sure they haven't seen the message already
 4974                     if nosyid not in seen_message:
 4975                         # send it to them
 4976                         sendto.append(nosyid)
 4977                         recipients.append(nosyid)
 4978 
 4979                 # generate a change note
 4980                 if oldvalues:
 4981                     note = cl.generateChangeNote(nodeid, oldvalues)
 4982                 else:
 4983                     note = cl.generateCreateNote(nodeid)
 4984 
 4985                 # we have new recipients
 4986                 if sendto:
 4987                     # filter out the people on vacation
 4988                     sendto = [i for i in sendto 
 4989                               if not users.get(i, 'vacation', 0)]
 4990 
 4991                     # map userids to addresses
 4992                     sendto = [users.get(i, 'address') for i in sendto]
 4993 
 4994                     # update the message's recipients list
 4995                     messages.set(msgid, recipients=recipients)
 4996 
 4997                     # send the message
 4998                     cl.send_message(nodeid, msgid, note, sendto)
 4999             except roundupdb.MessageSendError as message:
 5000                 raise roundupdb.DetectorError(message)
 5001 
 5002    Note that this is the standard nosy reaction code, with the small
 5003    addition of::
 5004 
 5005     # filter out the people on vacation
 5006     sendto = [i for i in sendto if not users.get(i, 'vacation', 0)]
 5007 
 5008    which filters out the users that have the vacation flag set to true.
 5009 
 5010 Adding in state transition control
 5011 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5012 
 5013 Sometimes tracker admins want to control the states to which users may
 5014 move issues. You can do this by following these steps:
 5015 
 5016 1. make "status" a required variable. This is achieved by adding the
 5017    following to the top of the form in the ``issue.item.html``
 5018    template::
 5019 
 5020      <input type="hidden" name="@required" value="status">
 5021 
 5022    This will force users to select a status.
 5023 
 5024 2. add a Multilink property to the status class::
 5025 
 5026      stat = Class(db, "status", ... , transitions=Multilink('status'),
 5027                   ...)
 5028 
 5029    and then edit the statuses already created, either:
 5030 
 5031    a. through the web using the class list -> status class editor, or
 5032    b. using the ``roundup-admin`` "set" command.
 5033 
 5034 3. add an auditor module ``checktransition.py`` in your tracker's
 5035    ``detectors`` directory, for example::
 5036 
 5037      def checktransition(db, cl, nodeid, newvalues):
 5038          ''' Check that the desired transition is valid for the "status"
 5039              property.
 5040          '''
 5041          if 'status' not in newvalues:
 5042              return
 5043          current = cl.get(nodeid, 'status')
 5044          new = newvalues['status']
 5045          if new == current:
 5046              return
 5047          ok = db.status.get(current, 'transitions')
 5048          if new not in ok:
 5049              raise ValueError('Status not allowed to move from "%s" to "%s"'%(
 5050                  db.status.get(current, 'name'), db.status.get(new, 'name')))
 5051 
 5052      def init(db):
 5053          db.issue.audit('set', checktransition)
 5054 
 5055 4. in the ``issue.item.html`` template, change the status editing bit
 5056    from::
 5057 
 5058     <th>Status</th>
 5059     <td tal:content="structure context/status/menu">status</td>
 5060 
 5061    to::
 5062 
 5063     <th>Status</th>
 5064     <td>
 5065      <select tal:condition="context/id" name="status">
 5066       <tal:block tal:define="ok context/status/transitions"
 5067                  tal:repeat="state db/status/list">
 5068        <option tal:condition="python:state.id in ok"
 5069                tal:attributes="
 5070                     value state/id;
 5071                     selected python:state.id == context.status.id"
 5072                tal:content="state/name"></option>
 5073       </tal:block>
 5074      </select>
 5075      <tal:block tal:condition="not:context/id"
 5076                 tal:replace="structure context/status/menu" />
 5077     </td>
 5078 
 5079    which displays only the allowed status to transition to.
 5080 
 5081 
 5082 Blocking issues that depend on other issues
 5083 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5084 
 5085 We needed the ability to mark certain issues as "blockers" - that is,
 5086 they can't be resolved until another issue (the blocker) they rely on is
 5087 resolved. To achieve this:
 5088 
 5089 1. Create a new property on the ``issue`` class:
 5090    ``blockers=Multilink("issue")``. To do this, edit the definition of
 5091    this class in your tracker's ``schema.py`` file. Change this::
 5092 
 5093     issue = IssueClass(db, "issue", 
 5094                     assignedto=Link("user"), keyword=Multilink("keyword"),
 5095                     priority=Link("priority"), status=Link("status"))
 5096 
 5097    to this, adding the blockers entry::
 5098 
 5099     issue = IssueClass(db, "issue", 
 5100                     blockers=Multilink("issue"),
 5101                     assignedto=Link("user"), keyword=Multilink("keyword"),
 5102                     priority=Link("priority"), status=Link("status"))
 5103 
 5104 2. Add the new ``blockers`` property to the ``issue.item.html`` edit
 5105    page, using something like::
 5106 
 5107     <th>Waiting On</th>
 5108     <td>
 5109      <span tal:replace="structure python:context.blockers.field(showid=1,
 5110                                   size=20)" />
 5111      <span tal:replace="structure python:db.issue.classhelp('id,title',
 5112                                   property='blockers')" />
 5113      <span tal:condition="context/blockers"
 5114            tal:repeat="blk context/blockers">
 5115       <br>View: <a tal:attributes="href string:issue${blk/id}"
 5116                    tal:content="blk/id"></a>
 5117      </span>
 5118     </td>
 5119 
 5120    You'll need to fiddle with your item page layout to find an
 5121    appropriate place to put it - I'll leave that fun part up to you.
 5122    Just make sure it appears in the first table, possibly somewhere near
 5123    the "superseders" field.
 5124 
 5125 3. Create a new detector module (see below) which enforces the rules:
 5126 
 5127    - issues may not be resolved if they have blockers
 5128    - when a blocker is resolved, it's removed from issues it blocks
 5129 
 5130    The contents of the detector should be something like this::
 5131 
 5132 
 5133     def blockresolution(db, cl, nodeid, newvalues):
 5134         ''' If the issue has blockers, don't allow it to be resolved.
 5135         '''
 5136         if nodeid is None:
 5137             blockers = []
 5138         else:
 5139             blockers = cl.get(nodeid, 'blockers')
 5140         blockers = newvalues.get('blockers', blockers)
 5141 
 5142         # don't do anything if there's no blockers or the status hasn't
 5143         # changed
 5144         if not blockers or 'status' not in newvalues:
 5145             return
 5146 
 5147         # get the resolved state ID
 5148         resolved_id = db.status.lookup('resolved')
 5149 
 5150         # format the info
 5151         u = db.config.TRACKER_WEB
 5152         s = ', '.join(['<a href="%sissue%s">%s</a>'%(
 5153                         u,id,id) for id in blockers])
 5154         if len(blockers) == 1:
 5155             s = 'issue %s is'%s
 5156         else:
 5157             s = 'issues %s are'%s
 5158 
 5159         # ok, see if we're trying to resolve
 5160         if newvalues['status'] == resolved_id:
 5161             raise ValueError("This issue can't be resolved until %s resolved."%s)
 5162 
 5163 
 5164     def resolveblockers(db, cl, nodeid, oldvalues):
 5165         ''' When we resolve an issue that's a blocker, remove it from the
 5166             blockers list of the issue(s) it blocks.
 5167         '''
 5168         newstatus = cl.get(nodeid,'status')
 5169 
 5170         # no change?
 5171         if oldvalues.get('status', None) == newstatus:
 5172             return
 5173 
 5174         resolved_id = db.status.lookup('resolved')
 5175 
 5176         # interesting?
 5177         if newstatus != resolved_id:
 5178             return
 5179 
 5180         # yes - find all the blocked issues, if any, and remove me from
 5181         # their blockers list
 5182         issues = cl.find(blockers=nodeid)
 5183         for issueid in issues:
 5184             blockers = cl.get(issueid, 'blockers')
 5185             if nodeid in blockers:
 5186                 blockers.remove(nodeid)
 5187                 cl.set(issueid, blockers=blockers)
 5188 
 5189     def init(db):
 5190         # might, in an obscure situation, happen in a create
 5191         db.issue.audit('create', blockresolution)
 5192         db.issue.audit('set', blockresolution)
 5193 
 5194         # can only happen on a set
 5195         db.issue.react('set', resolveblockers)
 5196 
 5197    Put the above code in a file called "blockers.py" in your tracker's
 5198    "detectors" directory.
 5199 
 5200 4. Finally, and this is an optional step, modify the tracker web page
 5201    URLs so they filter out issues with any blockers. You do this by
 5202    adding an additional filter on "blockers" for the value "-1". For
 5203    example, the existing "Show All" link in the "page" template (in the
 5204    tracker's "html" directory) looks like this::
 5205 
 5206     <a href="#"
 5207        tal:attributes="href python:request.indexargs_url('issue', {
 5208       '@sort': '-activity',
 5209       '@group': 'priority',
 5210       '@filter': 'status',
 5211       '@columns': columns_showall,
 5212       '@search_text': '',
 5213       'status': status_notresolved,
 5214       '@dispname': i18n.gettext('Show All'),
 5215      })"
 5216        i18n:translate="">Show All</a><br>
 5217 
 5218    modify it to add the "blockers" info to the URL (note, both the
 5219    "@filter" *and* "blockers" values must be specified)::
 5220 
 5221     <a href="#"
 5222        tal:attributes="href python:request.indexargs_url('issue', {
 5223       '@sort': '-activity',
 5224       '@group': 'priority',
 5225       '@filter': 'status,blockers',
 5226       '@columns': columns_showall,
 5227       '@search_text': '',
 5228       'status': status_notresolved,
 5229       'blockers': '-1',
 5230       '@dispname': i18n.gettext('Show All'),
 5231      })"
 5232        i18n:translate="">Show All</a><br>
 5233 
 5234    The above examples are line-wrapped on the trailing & and should
 5235    be unwrapped.
 5236 
 5237 That's it. You should now be able to set blockers on your issues. Note
 5238 that if you want to know whether an issue has any other issues dependent
 5239 on it (i.e. it's in their blockers list) you can look at the journal
 5240 history at the bottom of the issue page - look for a "link" event to
 5241 another issue's "blockers" property.
 5242 
 5243 Add users to the nosy list based on the keyword
 5244 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5245 
 5246 Let's say we need the ability to automatically add users to the nosy
 5247 list based
 5248 on the occurance of a keyword. Every user should be allowed to edit their
 5249 own list of keywords for which they want to be added to the nosy list.
 5250 
 5251 Below, we'll show that this change can be done with minimal
 5252 understanding of the Roundup system, using only copy and paste.
 5253 
 5254 This requires three changes to the tracker: a change in the database to
 5255 allow per-user recording of the lists of keywords for which he wants to
 5256 be put on the nosy list, a change in the user view allowing them to edit
 5257 this list of keywords, and addition of an auditor which updates the nosy
 5258 list when a keyword is set.
 5259 
 5260 Adding the nosy keyword list
 5261 ::::::::::::::::::::::::::::
 5262 
 5263 The change to make in the database, is that for any user there should be a list
 5264 of keywords for which he wants to be put on the nosy list. Adding a
 5265 ``Multilink`` of ``keyword`` seems to fullfill this. As such, all that has to
 5266 be done is to add a new field to the definition of ``user`` within the file
 5267 ``schema.py``.  We will call this new field ``nosy_keywords``, and the updated
 5268 definition of user will be::
 5269 
 5270     user = Class(db, "user", 
 5271                     username=String(),   password=Password(),
 5272                     address=String(),    realname=String(), 
 5273                     phone=String(),      organisation=String(),
 5274                     alternate_addresses=String(),
 5275                     queries=Multilink('query'), roles=String(),
 5276                     timezone=String(),
 5277                     nosy_keywords=Multilink('keyword'))
 5278  
 5279 Changing the user view to allow changing the nosy keyword list
 5280 ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
 5281 
 5282 We want any user to be able to change the list of keywords for which
 5283 he will by default be added to the nosy list. We choose to add this
 5284 to the user view, as is generated by the file ``html/user.item.html``.
 5285 We can easily 
 5286 see that the keyword field in the issue view has very similar editing
 5287 requirements as our nosy keywords, both being lists of keywords. As
 5288 such, we look for Keywords in ``issue.item.html``, and extract the
 5289 associated parts from there. We add this to ``user.item.html`` at the 
 5290 bottom of the list of viewed items (i.e. just below the 'Alternate
 5291 E-mail addresses' in the classic template)::
 5292 
 5293  <tr>
 5294   <th>Nosy Keywords</th>
 5295   <td>
 5296   <span tal:replace="structure context/nosy_keywords/field" />
 5297   <span tal:replace="structure python:db.keyword.classhelp(property='nosy_keywords')" />
 5298   </td>
 5299  </tr>
 5300   
 5301 
 5302 Addition of an auditor to update the nosy list
 5303 ::::::::::::::::::::::::::::::::::::::::::::::
 5304 
 5305 The more difficult part is the logic to add
 5306 the users to the nosy list when required. 
 5307 We choose to perform this action whenever the keywords on an
 5308 item are set (this includes the creation of items).
 5309 Here we choose to start out with a copy of the 
 5310 ``detectors/nosyreaction.py`` detector, which we copy to the file
 5311 ``detectors/nosy_keyword_reaction.py``. 
 5312 This looks like a good start as it also adds users
 5313 to the nosy list. A look through the code reveals that the
 5314 ``nosyreaction`` function actually sends the e-mail. 
 5315 We don't need this. Therefore, we can change the ``init`` function to::
 5316 
 5317     def init(db):
 5318         db.issue.audit('create', update_kw_nosy)
 5319         db.issue.audit('set', update_kw_nosy)
 5320 
 5321 After that, we rename the ``updatenosy`` function to ``update_kw_nosy``.
 5322 The first two blocks of code in that function relate to setting
 5323 ``current`` to a combination of the old and new nosy lists. This
 5324 functionality is left in the new auditor. The following block of
 5325 code, which handled adding the assignedto user(s) to the nosy list in
 5326 ``updatenosy``, should be replaced by a block of code to add the
 5327 interested users to the nosy list. We choose here to loop over all
 5328 new keywords, than looping over all users,
 5329 and assign the user to the nosy list when the keyword occurs in the user's
 5330 ``nosy_keywords``. The next part in ``updatenosy`` -- adding the author
 5331 and/or recipients of a message to the nosy list -- is obviously not
 5332 relevant here and is thus deleted from the new auditor. The last
 5333 part, copying the new nosy list to ``newvalues``, can stay as is.
 5334 This results in the following function::
 5335 
 5336     def update_kw_nosy(db, cl, nodeid, newvalues):
 5337         '''Update the nosy list for changes to the keywords
 5338         '''
 5339         # nodeid will be None if this is a new node
 5340         current = {}
 5341         if nodeid is None:
 5342             ok = ('new', 'yes')
 5343         else:
 5344             ok = ('yes',)
 5345             # old node, get the current values from the node if they haven't
 5346             # changed
 5347             if 'nosy' not in newvalues:
 5348                 nosy = cl.get(nodeid, 'nosy')
 5349                 for value in nosy:
 5350                     if value not in current:
 5351                         current[value] = 1
 5352 
 5353         # if the nosy list changed in this transaction, init from the new value
 5354         if 'nosy' in newvalues:
 5355             nosy = newvalues.get('nosy', [])
 5356             for value in nosy:
 5357                 if not db.hasnode('user', value):
 5358                     continue
 5359                 if value not in current:
 5360                     current[value] = 1
 5361 
 5362         # add users with keyword in nosy_keywords to the nosy list
 5363         if 'keyword' in newvalues and newvalues['keyword'] is not None:
 5364             keyword_ids = newvalues['keyword']
 5365             for keyword in keyword_ids:
 5366                 # loop over all users,
 5367                 # and assign user to nosy when keyword in nosy_keywords
 5368                 for user_id in db.user.list():
 5369                     nosy_kw = db.user.get(user_id, "nosy_keywords")
 5370                     found = 0
 5371                     for kw in nosy_kw:
 5372                         if kw == keyword:
 5373                             found = 1
 5374                     if found:
 5375                         current[user_id] = 1
 5376 
 5377         # that's it, save off the new nosy list
 5378         newvalues['nosy'] = list(current.keys())
 5379 
 5380 These two function are the only ones needed in the file.
 5381 
 5382 TODO: update this example to use the ``find()`` Class method.
 5383 
 5384 Caveats
 5385 :::::::
 5386 
 5387 A few problems with the design here can be noted:
 5388 
 5389 Multiple additions
 5390     When a user, after automatic selection, is manually removed
 5391     from the nosy list, he is added to the nosy list again when the
 5392     keyword list of the issue is updated. A better design might be
 5393     to only check which keywords are new compared to the old list
 5394     of keywords, and only add users when they have indicated
 5395     interest on a new keyword.
 5396 
 5397     The code could also be changed to only trigger on the ``create()``
 5398     event, rather than also on the ``set()`` event, thus only setting
 5399     the nosy list when the issue is created.
 5400 
 5401 Scalability
 5402     In the auditor, there is a loop over all users. For a site with
 5403     only few users this will pose no serious problem; however, with
 5404     many users this will be a serious performance bottleneck.
 5405     A way out would be to link from the keywords to the users who
 5406     selected these keywords as nosy keywords. This will eliminate the
 5407     loop over all users. See the ``rev_multilink`` attribute to make
 5408     this easier.
 5409 
 5410 Restricting updates that arrive by email
 5411 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5412 
 5413 Roundup supports multiple update methods:
 5414 
 5415 1. command line
 5416 2. plain email
 5417 3. pgp signed email
 5418 4. web access
 5419 
 5420 in some cases you may need to prevent changes to properties by some of
 5421 these methods. For example you can set up issues that are viewable
 5422 only by people on the nosy list. So you must prevent unauthenticated
 5423 changes to the nosy list.
 5424 
 5425 Since plain email can be easily forged, it does not provide sufficient
 5426 authentication in this senario.
 5427 
 5428 To prevent this we can add a detector that audits the source of the
 5429 transaction and rejects the update if it changes the nosy list.
 5430 
 5431 Create the detector (auditor) module and add it to the detectors
 5432 directory of your tracker::
 5433 
 5434    from roundup import roundupdb, hyperdb
 5435    
 5436    from roundup.mailgw import Unauthorized
 5437 
 5438    def restrict_nosy_changes(db, cl, nodeid, newvalues):
 5439        '''Do not permit changes to nosy via email.'''
 5440 
 5441        if 'nosy' not in newvalues:
 5442            # the nosy field has not changed so no need to check.
 5443            return
 5444 
 5445        if db.tx_Source in ['web', 'rest', 'xmlrpc', 'email-sig-openpgp', 'cli' ]:
 5446 	   # if the source of the transaction is from an authenticated
 5447 	   # source or a privileged process allow the transaction.
 5448 	   # Other possible sources: 'email'
 5449 	   return
 5450 
 5451        # otherwise raise an error
 5452        raise Unauthorized( \
 5453 	   'Changes to nosy property not allowed via %s for this issue.'%\
 5454            tx_Source)
 5455 	
 5456    def init(db):
 5457       ''' Install restrict_nosy_changes to run after other auditors. 
 5458 
 5459           Allow initial creation email to set nosy.
 5460           So don't execute: db.issue.audit('create', requestedbyauditor)
 5461 
 5462           Set priority to 110 to run this auditor after other auditors
 5463           that can cause nosy to change.
 5464       '''
 5465       db.issue.audit('set', restrict_nosy_changes, 110)
 5466 
 5467 This detector (auditor) will prevent updates to the nosy field if it
 5468 arrives by email. Since it runs after other auditors (due to the
 5469 priority of 110), it will also prevent changes to the nosy field that
 5470 are done by other auditors if triggered by an email.
 5471 
 5472 Note that db.tx_Source was not present in roundup versions before
 5473 1.4.22, so you must be running a newer version to use this detector.
 5474 Read the CHANGES.txt document in the roundup source code for further
 5475 details on tx_Source.
 5476 
 5477 Changes to Security and Permissions
 5478 -----------------------------------
 5479 
 5480 Restricting the list of users that are assignable to a task
 5481 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5482 
 5483 1. In your tracker's ``schema.py``, create a new Role, say "Developer"::
 5484 
 5485      db.security.addRole(name='Developer', description='A developer')
 5486 
 5487 2. Just after that, create a new Permission, say "Fixer", specific to
 5488    "issue"::
 5489 
 5490      p = db.security.addPermission(name='Fixer', klass='issue',
 5491          description='User is allowed to be assigned to fix issues')
 5492 
 5493 3. Then assign the new Permission to your "Developer" Role::
 5494 
 5495      db.security.addPermissionToRole('Developer', p)
 5496 
 5497 4. In the issue item edit page (``html/issue.item.html`` in your tracker
 5498    directory), use the new Permission in restricting the "assignedto"
 5499    list::
 5500 
 5501     <select name="assignedto">
 5502      <option value="-1">- no selection -</option>
 5503      <tal:block tal:repeat="user db/user/list">
 5504      <option tal:condition="python:user.hasPermission(
 5505                                 'Fixer', context._classname)"
 5506              tal:attributes="
 5507                 value user/id;
 5508                 selected python:user.id == context.assignedto"
 5509              tal:content="user/realname"></option>
 5510      </tal:block>
 5511     </select>
 5512 
 5513 For extra security, you may wish to setup an auditor to enforce the
 5514 Permission requirement (install this as ``assignedtoFixer.py`` in your
 5515 tracker ``detectors`` directory)::
 5516 
 5517   def assignedtoMustBeFixer(db, cl, nodeid, newvalues):
 5518       ''' Ensure the assignedto value in newvalues is used with the
 5519           Fixer Permission
 5520       '''
 5521       if 'assignedto' not in newvalues:
 5522           # don't care
 5523           return
 5524   
 5525       # get the userid
 5526       userid = newvalues['assignedto']
 5527       if not db.security.hasPermission('Fixer', userid, cl.classname):
 5528           raise ValueError('You do not have permission to edit %s'%cl.classname)
 5529 
 5530   def init(db):
 5531       db.issue.audit('set', assignedtoMustBeFixer)
 5532       db.issue.audit('create', assignedtoMustBeFixer)
 5533 
 5534 So now, if an edit action attempts to set "assignedto" to a user that
 5535 doesn't have the "Fixer" Permission, the error will be raised.
 5536 
 5537 
 5538 Users may only edit their issues
 5539 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5540 
 5541 In this case, users registering themselves are granted Provisional
 5542 access, meaning they
 5543 have access to edit the issues they submit, but not others. We create a new
 5544 Role called "Provisional User" which is granted to newly-registered users,
 5545 and has limited access. One of the Permissions they have is the new "Edit
 5546 Own" on issues (regular users have "Edit".)
 5547 
 5548 First up, we create the new Role and Permission structure in
 5549 ``schema.py``::
 5550 
 5551     #
 5552     # New users not approved by the admin
 5553     #
 5554     db.security.addRole(name='Provisional User',
 5555         description='New user registered via web or email')
 5556 
 5557     # These users need to be able to view and create issues but only edit
 5558     # and view their own
 5559     db.security.addPermissionToRole('Provisional User', 'Create', 'issue')
 5560     def own_issue(db, userid, itemid):
 5561         '''Determine whether the userid matches the creator of the issue.'''
 5562         return userid == db.issue.get(itemid, 'creator')
 5563     p = db.security.addPermission(name='Edit', klass='issue',
 5564         check=own_issue, description='Can only edit own issues')
 5565     db.security.addPermissionToRole('Provisional User', p)
 5566     p = db.security.addPermission(name='View', klass='issue',
 5567         check=own_issue, description='Can only view own issues')
 5568     db.security.addPermissionToRole('Provisional User', p)
 5569     # This allows the interface to get the names of the properties
 5570     # in the issue. Used for selecting sorting and grouping
 5571     # on the index page.
 5572     p = db.security.addPermission(name='Search', klass='issue')
 5573     db.security.addPermissionToRole ('Provisional User', p)
 5574 
 5575 
 5576     # Assign the Permissions for issue-related classes
 5577     for cl in 'file', 'msg', 'query', 'keyword':
 5578         db.security.addPermissionToRole('Provisional User', 'View', cl)
 5579         db.security.addPermissionToRole('Provisional User', 'Edit', cl)
 5580         db.security.addPermissionToRole('Provisional User', 'Create', cl)
 5581     for cl in 'priority', 'status':
 5582         db.security.addPermissionToRole('Provisional User', 'View', cl)
 5583 
 5584     # and give the new users access to the web and email interface
 5585     db.security.addPermissionToRole('Provisional User', 'Web Access')
 5586     db.security.addPermissionToRole('Provisional User', 'Email Access')
 5587 
 5588     # make sure they can view & edit their own user record
 5589     def own_record(db, userid, itemid):
 5590         '''Determine whether the userid matches the item being accessed.'''
 5591         return userid == itemid
 5592     p = db.security.addPermission(name='View', klass='user', check=own_record,
 5593         description="User is allowed to view their own user details")
 5594     db.security.addPermissionToRole('Provisional User', p)
 5595     p = db.security.addPermission(name='Edit', klass='user', check=own_record,
 5596         description="User is allowed to edit their own user details")
 5597     db.security.addPermissionToRole('Provisional User', p)
 5598 
 5599 Then, in ``config.ini``, we change the Role assigned to newly-registered
 5600 users, replacing the existing ``'User'`` values::
 5601 
 5602     [main]
 5603     ...
 5604     new_web_user_roles = Provisional User
 5605     new_email_user_roles = Provisional User
 5606 
 5607 
 5608 All users may only view and edit issues, files and messages they create
 5609 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5610 
 5611 Replace the standard "classic" tracker View and Edit Permission assignments
 5612 for the "issue", "file" and "msg" classes with the following::
 5613 
 5614     def checker(klass):
 5615         def check(db, userid, itemid, klass=klass):
 5616             return db.getclass(klass).get(itemid, 'creator') == userid
 5617         return check
 5618     for cl in 'issue', 'file', 'msg':
 5619         p = db.security.addPermission(name='View', klass=cl,
 5620             check=checker(cl))
 5621         db.security.addPermissionToRole('User', p)
 5622         p = db.security.addPermission(name='Edit', klass=cl,
 5623             check=checker(cl))
 5624         db.security.addPermissionToRole('User', p)
 5625         db.security.addPermissionToRole('User', 'Create', cl)
 5626 
 5627 
 5628 Moderating user registration
 5629 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5630 
 5631 You could set up new-user moderation in a public tracker by:
 5632 
 5633 1. creating a new highly-restricted user role "Pending",
 5634 2. set the config new_web_user_roles and/or new_email_user_roles to that
 5635    role,
 5636 3. have an auditor that emails you when new users are created with that
 5637    role using roundup.mailer
 5638 4. edit the role to "User" for valid users.
 5639 
 5640 Some simple javascript might help in the last step. If you have high volume
 5641 you could search for all currently-Pending users and do a bulk edit of all
 5642 their roles at once (again probably with some simple javascript help).
 5643 
 5644 
 5645 Changes to the Web User Interface
 5646 ---------------------------------
 5647 
 5648 Adding action links to the index page
 5649 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5650 
 5651 Add a column to the ``item.index.html`` template.
 5652 
 5653 Resolving the issue::
 5654 
 5655   <a tal:attributes="href
 5656      string:issue${i/id}?:status=resolved&:action=edit">resolve</a>
 5657 
 5658 "Take" the issue::
 5659 
 5660   <a tal:attributes="href
 5661      string:issue${i/id}?:assignedto=${request/user/id}&:action=edit">take</a>
 5662 
 5663 ... and so on.
 5664 
 5665 Colouring the rows in the issue index according to priority
 5666 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5667 
 5668 A simple ``tal:attributes`` statement will do the bulk of the work here. In
 5669 the ``issue.index.html`` template, add this to the ``<tr>`` that
 5670 displays the rows of data::
 5671 
 5672    <tr tal:attributes="class string:priority-${i/priority/plain}">
 5673 
 5674 and then in your stylesheet (``style.css``) specify the colouring for the
 5675 different priorities, as follows::
 5676 
 5677    tr.priority-critical td {
 5678        background-color: red;
 5679    }
 5680 
 5681    tr.priority-urgent td {
 5682        background-color: orange;
 5683    }
 5684 
 5685 and so on, with far less offensive colours :)
 5686 
 5687 Editing multiple items in an index view
 5688 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5689 
 5690 To edit the status of all items in the item index view, edit the
 5691 ``issue.item.html``:
 5692 
 5693 1. add a form around the listing table (separate from the existing
 5694    index-page form), so at the top it reads::
 5695 
 5696     <form method="POST" tal:attributes="action request/classname">
 5697      <table class="list">
 5698 
 5699    and at the bottom of that table::
 5700 
 5701      </table>
 5702     </form
 5703 
 5704    making sure you match the ``</table>`` from the list table, not the
 5705    navigation table or the subsequent form table.
 5706 
 5707 2. in the display for the issue property, change::
 5708 
 5709     <td tal:condition="request/show/status"
 5710         tal:content="python:i.status.plain() or default">&nbsp;</td>
 5711 
 5712    to::
 5713 
 5714     <td tal:condition="request/show/status"
 5715         tal:content="structure i/status/field">&nbsp;</td>
 5716 
 5717    this will result in an edit field for the status property.
 5718 
 5719 3. after the ``tal:block`` which lists the index items (marked by
 5720    ``tal:repeat="i batch"``) add a new table row::
 5721 
 5722     <tr>
 5723      <td tal:attributes="colspan python:len(request.columns)">
 5724       <input name="@csrf" type="hidden"		
 5725            tal:attributes="value python:utils.anti_csrf_nonce()">
 5726       <input type="submit" value=" Save Changes ">
 5727       <input type="hidden" name="@action" value="edit">
 5728       <tal:block replace="structure request/indexargs_form" />
 5729      </td>
 5730     </tr>
 5731 
 5732    which gives us a submit button, indicates that we are performing an
 5733    edit on any changed statuses, and provides a defense against cross
 5734    site request forgery attacks.
 5735 
 5736    The final ``tal:block`` will make sure that the current index view
 5737    parameters (filtering, columns, etc) will be used in rendering the
 5738    next page (the results of the editing).
 5739 
 5740 
 5741 Displaying only message summaries in the issue display
 5742 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5743 
 5744 Alter the ``issue.item`` template section for messages to::
 5745 
 5746  <table class="messages" tal:condition="context/messages">
 5747   <tr><th colspan="5" class="header">Messages</th></tr>
 5748   <tr tal:repeat="msg context/messages">
 5749    <td><a tal:attributes="href string:msg${msg/id}"
 5750           tal:content="string:msg${msg/id}"></a></td>
 5751    <td tal:content="msg/author">author</td>
 5752    <td class="date" tal:content="msg/date/pretty">date</td>
 5753    <td tal:content="msg/summary">summary</td>
 5754    <td>
 5755     <a tal:attributes="href string:?@remove@messages=${msg/id}&@action=edit">
 5756     remove</a>
 5757    </td>
 5758   </tr>
 5759  </table>
 5760 
 5761 
 5762 Enabling display of either message summaries or the entire messages
 5763 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5764 
 5765 This is pretty simple - all we need to do is copy the code from the
 5766 example `displaying only message summaries in the issue display`_ into
 5767 our template alongside the summary display, and then introduce a switch
 5768 that shows either the one or the other. We'll use a new form variable,
 5769 ``@whole_messages`` to achieve this::
 5770 
 5771  <table class="messages" tal:condition="context/messages">
 5772   <tal:block tal:condition="not:request/form/@whole_messages/value | python:0">
 5773    <tr><th colspan="3" class="header">Messages</th>
 5774        <th colspan="2" class="header">
 5775          <a href="?@whole_messages=yes">show entire messages</a>
 5776        </th>
 5777    </tr>
 5778    <tr tal:repeat="msg context/messages">
 5779     <td><a tal:attributes="href string:msg${msg/id}"
 5780            tal:content="string:msg${msg/id}"></a></td>
 5781     <td tal:content="msg/author">author</td>
 5782     <td class="date" tal:content="msg/date/pretty">date</td>
 5783     <td tal:content="msg/summary">summary</td>
 5784     <td>
 5785      <a tal:attributes="href string:?@remove@messages=${msg/id}&@action=edit">remove</a>
 5786     </td>
 5787    </tr>
 5788   </tal:block>
 5789 
 5790   <tal:block tal:condition="request/form/@whole_messages/value | python:0">
 5791    <tr><th colspan="2" class="header">Messages</th>
 5792        <th class="header">
 5793          <a href="?@whole_messages=">show only summaries</a>
 5794        </th>
 5795    </tr>
 5796    <tal:block tal:repeat="msg context/messages">
 5797     <tr>
 5798      <th tal:content="msg/author">author</th>
 5799      <th class="date" tal:content="msg/date/pretty">date</th>
 5800      <th style="text-align: right">
 5801       (<a tal:attributes="href string:?@remove@messages=${msg/id}&@action=edit">remove</a>)
 5802      </th>
 5803     </tr>
 5804     <tr><td colspan="3" tal:content="msg/content"></td></tr>
 5805    </tal:block>
 5806   </tal:block>
 5807  </table>
 5808 
 5809 
 5810 Setting up a "wizard" (or "druid") for controlled adding of issues
 5811 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 5812 
 5813 1. Set up the page templates you wish to use for data input. My wizard
 5814    is going to be a two-step process: first figuring out what category
 5815    of issue the user is submitting, and then getting details specific to
 5816    that category. The first page includes a table of help, explaining
 5817    what the category names mean, and then the core of the form::
 5818 
 5819     <form method="POST" onSubmit="return submit_once()"
 5820           enctype="multipart/form-data">
 5821        <input name="@csrf" type="hidden"
 5822           tal:attributes="value python:utils.anti_csrf_nonce()">
 5823       <input type="hidden" name="@template" value="add_page1">
 5824       <input type="hidden" name="@action" value="page1_submit">
 5825 
 5826       <strong>Category:</strong>
 5827       <tal:block tal:replace="structure context/category/menu" />
 5828       <input type="submit" value="Continue">
 5829     </form>
 5830 
 5831    The next page has the usual issue entry information, with the
 5832    addition of the following form fragments::
 5833 
 5834     <form method="POST" onSubmit="return submit_once()"
 5835           enctype="multipart/form-data"
 5836           tal:condition="context/is_edit_ok"
 5837           tal:define="cat request/form/category/value">
 5838 
 5839       <input name="@csrf" type="hidden"
 5840           tal:attributes="value python:utils.anti_csrf_nonce()">
 5841       <input type="hidden" name="@template" value="add_page2">
 5842       <input type="hidden" name="@required" value="title">
 5843       <input type="hidden" name="category" tal:attributes="value cat">
 5844        .
 5845        .
 5846        .
 5847     </form>
 5848 
 5849    Note that later in the form, I use the value of "cat" to decide which
 5850    form elements should be displayed. For example::
 5851 
 5852     <tal:block tal:condition="python:cat in '6 10 13 14 15 16 17'.split()">
 5853      <tr>
 5854       <th>Operating System</th>
 5855       <td tal:content="structure context/os/field"></td>
 5856      </tr>
 5857      <tr>
 5858       <th>Web Browser</th>
 5859       <td tal:content="structure context/browser/field"></td>
 5860      </tr>
 5861     </tal:block>
 5862 
 5863    ... the above section will only be displayed if the category is one
 5864    of 6, 10, 13, 14, 15, 16 or 17.
 5865 
 5866 3. Determine what actions need to be taken between the pages - these are
 5867    usually to validate user choices and determine what page is next. Now encode
 5868    those actions in a new ``Action`` class (see `defining new web actions`_)::
 5869 
 5870     from roundup.cgi.actions import Action
 5871 
 5872     class Page1SubmitAction(Action):
 5873         def handle(self):
 5874             ''' Verify that the user has selected a category, and then move
 5875                 on to page 2.
 5876             '''
 5877             category = self.form['category'].value
 5878             if category == '-1':
 5879                 self.client.add_error_message('You must select a category of report')
 5880                 return
 5881             # everything's ok, move on to the next page
 5882             self.client.template = 'add_page2'
 5883 
 5884     def init(instance):
 5885         instance.registerAction('page1_submit', Page1SubmitAction)
 5886 
 5887 4. Use the usual "new" action as the ``@action`` on the final page, and
 5888    you're done (the standard context/submit method can do this for you).
 5889 
 5890 
 5891 Silent Submit
 5892 ~~~~~~~~~~~~~
 5893 
 5894 When working on an issue, most of the time the people on the nosy list
 5895 need to be notified of changes. There are cases where a user wants to
 5896 add a comment to an issue and not bother other users on the nosy
 5897 list.
 5898 This feature is called Silent Submit because it allows the user to
 5899 silently modify an issue and not tell anyone.
 5900 
 5901 There are several parts to this change. The main activity part
 5902 involves editing the stock detectors/nosyreaction.py file in your
 5903 tracker. Insert the following lines near the top of the nosyreaction
 5904 function::
 5905 
 5906     # Did user click button to do a silent change?
 5907     try:
 5908         if db.web['submit'] == "silent_change":
 5909             return
 5910     except (AttributeError, KeyError) as err:
 5911         # The web attribute or submit key don't exist.
 5912         # That's fine. We were probably triggered by an email
 5913         # or cli based change.
 5914         pass
 5915   
 5916 This checks the submit button to see if it is the silent type. If there
 5917 are exceptions trying to make that determination they are ignored and
 5918 processing continues. You may wonder how db.web gets set. This is done
 5919 by creating an extension. Add the file extensions/edit.py with
 5920 this content::
 5921 
 5922   from roundup.cgi.actions import EditItemAction
 5923 
 5924   class Edit2Action(EditItemAction):
 5925     def handle(self):
 5926         self.db.web = {}  # create the dict
 5927         # populate the dict by getting the value of the submit_button
 5928         # element from the form.
 5929         self.db.web['submit'] = self.form['submit_button'].value
 5930 
 5931         # call the core EditItemAction to process the edit.
 5932         EditItemAction.handle(self)
 5933 
 5934   def init(instance):
 5935     '''Override the default edit action with this new version'''
 5936     instance.registerAction('edit', Edit2Action)
 5937 	
 5938 This code is a wrapper for the roundup EditItemAction. It checks the
 5939 form's submit button to save the value element. The rest of the changes
 5940 needed for the Silent Submit feature involves editing
 5941 html/issue.item.html to add the silent submit button. In
 5942 the stock issue.item.html the submit button is on a line that contains
 5943 "submit button". Replace that line with something like the following::
 5944 
 5945 	    <input type="submit" name="submit_button"
 5946 		   tal:condition="context/is_edit_ok"
 5947 		   value="Submit Changes">&nbsp;
 5948 	    <button type="submit" name="submit_button"
 5949 		    tal:condition="context/is_edit_ok"
 5950 		    title="Click this to submit but not send nosy email."
 5951 		    value="silent_change" i18n:translate="">
 5952 	      Silent Change</button>
 5953 
 5954 Note the difference in the value attribute for the two submit buttons.
 5955 The value "silent_change" in the button specification must match the
 5956 string in the nosy reaction function.
 5957 
 5958 Debugging Trackers
 5959 ==================
 5960 
 5961 There are three switches in tracker configs that turn on debugging in
 5962 Roundup:
 5963 
 5964 1. web :: debug
 5965 2. mail :: debug
 5966 3. logging :: level
 5967 
 5968 See the config.ini file or the `tracker configuration`_ section above for
 5969 more information.
 5970 
 5971 Additionally, the ``roundup-server.py`` script has its own debugging mode
 5972 in which it reloads edited templates immediately when they are changed,
 5973 rather than requiring a web server restart.
 5974 
 5975 
 5976 .. _`design documentation`: design.html
 5977 .. _`developer's guide`: developers.html
 5978 .. _`rest interface documentation`: rest.html#programming-the-rest-api
 5979 .. _`directions in the rest interface documentation`: rest.html#enabling-the-rest-api
 5980 .. _`xmlrpc interface documentation`: xmlrpc.html#through-roundup
 5981 .. _`zxcvbn`: https://github.com/dwolfhub/zxcvbn-python