"Fossies" - the Fresh Open Source Software Archive

Member "roundup-2.0.0/doc/upgrading.txt" (29 Jun 2020, 113763 Bytes) of package /linux/www/roundup-2.0.0.tar.gz:


The requested HTML page contains a <FORM> tag that is unusable on "Fossies" in "automatic" (rendered) mode so that page is shown as HTML source code (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. See also the latest Fossies "Diffs" side-by-side code changes report for "upgrading.txt": 1.6.1_vs_2.0.0.

    1 .. index:: Upgrading
    2 
    3 ======================================
    4 Upgrading to newer versions of Roundup
    5 ======================================
    6 
    7 Please read each section carefully and edit your tracker home files
    8 accordingly. Note that there is information about upgrade procedures in the
    9 `administration guide`_.
   10 
   11 If a specific version transition isn't mentioned here (eg. 0.6.7 to 0.6.8)
   12 then you don't need to do anything. If you're upgrading from 0.5.6 to
   13 0.6.8 though, you'll need to check the "0.5 to 0.6" and "0.6.x to 0.6.3"
   14 steps.
   15 
   16 **IMPORTANT** The v1.5.x releases of Roundup were the last to support
   17 Python v2.5 and v2.6.  Starting with the v1.6 releases of Roundup
   18 v2.7.2 is required to run newer releases of Roundup.
   19 
   20 Contents:
   21 
   22 .. contents::
   23    :local:
   24 
   25 .. index:: Upgrading; 1.6.x to 2.0.0
   26 
   27 Migrating from 1.6.X to 2.0.0
   28 =============================
   29 
   30 .. index:: roundup-admin; updateconfig subcommand
   31 
   32 
   33 Python 2 MYSQL users MUST READ
   34 ------------------------------
   35 
   36 To fix issues with encoding of data and text searching, roundup now
   37 explicitly sets the database connection character set. Roundup prior
   38 to 2.0 used the default character set which was not always utf-8. All
   39 roundup data is manipulated in utf-8. This mismatch causes issues with
   40 searches and result in corrupted data in the database if it was not
   41 properly represented across the charset conversions.
   42 
   43 This issue exists when running roundup under python 2. Note that there
   44 are more changes required for running roundup 2.0 if you choose to use
   45 python3. See `Python 3 support`_.
   46 
   47 In an upgraded ``config.ini`` (see next section) the ``[rdbms]``
   48 section has a key ``mysql_charset`` set by default to ``utf-8``.
   49 
   50 It should be possible to change utf-8 to any mysql charset. So if you
   51 know what charset is enabled (e.g. via a setting in ~roundup/.my.cnf,
   52 or the default charset for the database) you can set it in
   53 ``config.ini`` and not need to covert the database. However the
   54 underlying issues with misconverted data and bad searches will still
   55 exist if they did before.
   56 
   57 None of the roundup developers run mysql, so the exact steps to take
   58 during the upgrade were tested with test and not production databases.
   59 
   60 **Before doing anything else:**
   61 
   62    Backup the mysql database using mysql dump or other mysql
   63    supported tool.
   64 
   65    Backup roundup using your current backup tool and take the roundup
   66    instance offline.
   67 
   68 Then the following steps (similar to the conversion in needed for
   69 Python 3) should work:
   70 
   71 1. Export the tracker database
   72    using your **current** 1.6 instance::
   73 
   74      roundup-admin -i <trackerdir> exporttables <export_dir>
   75 
   76    replacing tracker_dir and export_dir as appropriate.
   77 
   78 2. Import the exported database using the **new** 2.0 roundup::
   79 
   80      roundup-admin -i <trackerdir> importtables <export_dir>
   81 
   82    replacing tracker_dir and export_dir as appropriate.
   83 
   84 The imported data should overwrite the original data. Note it is
   85 critically important that the ``exporttables`` be done with the *old
   86 tracker* and the ``importtables`` be done with the *new tracker*. An
   87 import/export cycle between roundup 1.6.0 and roundup 2.0 has been
   88 done successfully. So the export format for 1.6 and 2.0 should be
   89 compatible.
   90 
   91 Note that ``importtables`` is new in roundup-2.0, so you will not be
   92 able to import the result of ``exporttables`` using any 1.x version of
   93 roundup.
   94 
   95 Following the same sequence as above using ``export`` and ``import``
   96 should also work, but it will export all the files and messages. This
   97 will take longer but may be worth trying if the ``exporttables`` and
   98 ``importtables`` method fails for some reason.
   99 
  100 Another way that should be faster, but is untested is to use mysql
  101 dump to dump the database.
  102 https://makandracards.com/makandra/595-dumping-and-importing-from-to-mysql-in-an-utf-8-safe-way
  103 recommends::
  104 
  105    Note that when your MySQL server is not set to UTF-8 you need to do
  106    mysqldump --default-character-set=latin1 (!) to get a correctly
  107    encoded dump. In that case you will also need to remove the SET
  108    NAMES='latin1' comment at the top of the dump, so the target machine
  109    won't change its UTF-8 charset when sourcing.
  110 
  111 Then import the dump. Removing ``SET NAMES`` should allow the import
  112 to use UTF-8.
  113 
  114 Please report success or issues with this conversion to the
  115 roundup-users AT lists.sourceforge.net mailing list.
  116 
  117 As people report successful or unsuccessful conversions, we will update
  118 the errata page at: https://wiki.roundup-tracker.org/ReleaseErrata.
  119 
  120 Upgrade tracker's config.ini file
  121 ---------------------------------
  122 
  123 Once you have installed the new roundup, use::
  124 
  125   roundup-admin -i /path/to/tracker updateconfig newconfig.ini
  126 
  127 to generate a new ini file preserving all your settings. You can then
  128 merge any local comments from the tracker's ``config.ini`` into
  129 ``newconfig.ini``. Compare the old and new files and configure any new
  130 settings as you want. Then replace ``config.ini`` with the
  131 ``newconfig.ini`` file.
  132 
  133 Python 3 support
  134 ----------------
  135 
  136 Many of the ``.html`` and ``.py`` files from Roundup that are copied
  137 into tracker directories have changed for Python 3 support.  If you
  138 wish to move an existing tracker to Python 3, you need to merge in
  139 those changes. Also you need to make sure that locally created python
  140 code in the tracker is correct for Python 3.
  141 
  142 If your tracker uses the ``anydbm`` or ``mysql`` backends, you also
  143 need to export the tracker contents using ``roundup-admin export``
  144 running under Python 2, and them import them using ``roundup-admin
  145 import`` running under Python 3. This is detailed in the documention
  146 for migrating to a different backend.  If using the ``sqlite`` backend,
  147 you do not need to export and import, but need to delete the
  148 ``db/otks`` and ``db/sessions`` files when changing Python version.
  149 If using the ``postgresql`` backend, you do not need to export and
  150 import and no other special database-related steps are needed.
  151 
  152 If you use the whoosh indexer, you will need to reindex. It looks like
  153 a database created with Python 2 leads to Unicode decode errors when
  154 accessed by Python 3. Reindexing can take a while (see details below
  155 look for "reindexing").
  156 
  157 Octal values in config.ini change from the Python 2 representation
  158 with a leading ``0`` (``022``). They now use a leading ``0o``
  159 (``0o22``). Note that the ``0o`` format is properly handled under
  160 python 2. You can use the ``newconfig.ini`` generated using ``python3
  161 roundup-admin -i ... updateconfig newconfig.ini`` if you want to go
  162 back to using python 2. (Note going back to Python 2 will require
  163 the same steps as moving from 2 to 3 except using Python 3 to perform
  164 the export.)
  165 
  166 Rate Limit New User Registration
  167 --------------------------------
  168 
  169 The new user registration form can be abused by bots to allow
  170 automated registration for spamming. This can be limited by using the
  171 new ``config.ini`` ``[web]`` option called
  172 ``registration_delay``. The default is 4 and is the number of seconds
  173 between the time the form was generated and the time the form is
  174 processed.
  175 
  176 If you do not modify the ``user.register.html`` template in your
  177 tracker's html directory, you *must* set this to 0. Otherwise you will
  178 see the error:
  179 
  180   Form is corrupted, missing: opaqueregister.
  181 
  182 If set to 0, the rate limit check is disabled.
  183 
  184 If you want to use this, you can change your ``user.register.html``
  185 file to include::
  186 
  187  <input type="hidden" name="opaqueregister" tal:attributes="value python: utils.timestamp()">
  188 
  189 The hidden input field can be placed right after the form declaration
  190 that starts with::
  191 
  192    <form method="POST" onSubmit="return submit_once()"
  193 
  194 If you have applied Erik Forsberg's tracker level patch to implement
  195 (see: https://hg.python.org/tracker/python-dev/rev/83477f735132), you
  196 can back the code out of the tracker. You must change the name of the
  197 field in the html template to ``opaqueregistration`` from ``opaque``
  198 in order to use the core code.
  199 
  200 PGP mail processing
  201 -------------------
  202 
  203 Roundup now uses the ``gpg`` module instead of ``pyme`` to process PGP
  204 mail.  If you have PGP processing enabled, make sure the ``gpg``
  205 module is installed.
  206 
  207 MySQL client module
  208 -------------------
  209 
  210 Although the ``MySQLdb`` module from
  211 https://pypi.org/project/MySQL-python/ is still supported, it is
  212 recommended to switch to the updated module from
  213 https://pypi.org/project/mysqlclient/.
  214 
  215 XMLRPC Access Role
  216 ------------------
  217 
  218 A new permission has been added to control access to the XMLRPC
  219 endpoint. If the user doesn't have the new "Xmlrpc Access" permission,
  220 they will not be able to log in using the /xmlrpc end point.  To add
  221 this new permission to the "User" role you should change your
  222 tracker's schema.py and add::
  223 
  224     db.security.addPermissionToRole('User', 'Xmlrpc Access')
  225 
  226 This is usually included near where other permissions like "Web Access"
  227 or "Email Access" are assigned.
  228 
  229 New values for db.tx_Source
  230 ---------------------------
  231 
  232 The database attribute tx_Source reports "xmlrpc" and "rest" when the
  233 /xmlrpc and /rest web endpoints are used. Check all code (extensions,
  234 detectors, lib) in trackers looking for tx_Source. If you have code
  235 like::
  236 
  237     if db.tx_Source == "web":
  238 
  239 or::
  240 
  241     if db.tx_Source in ['web', 'email-sig-openpgp', 'cli' ]:
  242 
  243 you may need to change these to include matches to "rest" and
  244 "xmlrpc". For example::
  245 
  246     if db.tx_Source in [ "web", "rest", "xmlrpc" ]
  247 
  248 or::
  249 
  250     if db.tx_Source in ['web', 'rest', 'xmlrpc', 'email-sig-openpgp', 'cli' ]:
  251 
  252 
  253 CSV export changes
  254 ------------------
  255 
  256 The original Roundup CSV export function for indexes reported id
  257 numbers for links. The wiki had a version that resolved the id's to
  258 names, so it would report ``open`` rather than ``2`` or
  259 ``user2;user3`` rather than ``[2,3]``.
  260 
  261 Many people added the enhanced version to their extensions directory.
  262 
  263 The enhanced version was made the default in roundup 2.0.  If you want
  264 to use the old version (that returns id's), you can replace references
  265 to ``export_csv`` with ``export_csv_id`` in templates.
  266 
  267 Both core csv export functions have been changed to force quoting of
  268 all exported fields. To incorporate this change in any CSV export
  269 extension you may have added, change references in your code from::
  270 
  271     writer = csv.writer(wfile)
  272 
  273 to::
  274 
  275     writer = csv.writer(wfile, quoting=csv.QUOTE_NONNUMERIC)
  276 
  277 this forces all (non-numeric) fields to be quoted and empty quotes to
  278 be added for missing parameters.
  279 
  280 This turns exported values that may look like formulas into strings so
  281 some versions of Excel won't try to interpret them as a formula.
  282 
  283 Update userauditor.py to restrict usernames
  284 -------------------------------------------
  285 
  286 A username can be created with embedded commas and < and >
  287 characters. Even though the < and > are usually escaped when
  288 displayed, the embedded comma makes it difficult to edit lists of
  289 users as they are comma separated.
  290 
  291 If you have not modified your tracker's userauditor.py, you can just
  292 copy the userauditor.py from the classic template into your tracker's
  293 detectors directory. Otherwise merge the changes from the template
  294 userauditor.py. https://issues.roundup-tracker.org/issue2550921 may be
  295 helpful.
  296 
  297 Consider reindexing if you use European languages
  298 -------------------------------------------------
  299 
  300 A couple of bugs dealing with incorrect indexing of European languages
  301 (Russian and German were reported) have been fixed. Note reindexing
  302 all your data may take a long time. See:
  303 https://issues.roundup-tracker.org/issue1195739 and
  304 https://issues.roundup-tracker.org/issue1344046 for a description of
  305 the problem. If you determine that this a problem for your tracker,
  306 you can use::
  307 
  308   roundup-admin -i /path/to/tracker reindex
  309 
  310 to rewrite your full text indexes. The tracker used for reindex timing
  311 had 140MB of file/message data and 2500 issues with a slow 5400RPM
  312 SATA drive. Using native indexing with sqlite took about 45
  313 minutes. Using whoosh took about 2 hours. Using xapian took about 6
  314 hours. All examples were with Python 2. Anecdotal evidence shows
  315 Python 3 is faster, but YMMV.
  316 
  317 Merge improvements in statusauditor.py
  318 --------------------------------------
  319 
  320 By default the detector statusauditor.py will change the status from
  321 "unread" to "chatting" when a second message is added to an issue.
  322 The distributed classic and jinja templates implement this feature in
  323 their copies of ``detectors/statusauditor.py``.
  324 
  325 This can be a problem. Consider a person sending email to create an
  326 issue. Then the person sends a followup message to add some additional
  327 information to the issue. The followup message will trigger the status
  328 change from "unread" to "chatting". This is misleading since the
  329 person is "chatting" with themselves.
  330 
  331 Statusauditor.py has been enhanced to prevent the status from changing
  332 to "chatting" until a second user (person) adds a message.  If you
  333 want this functionality, you need to merge the distributed
  334 statusauditor.py with your tracker's statusauditor.py. If you have not
  335 customized your tracker's statusauditor.py, copy the one from the
  336 distibuted template. In addition to the python file, you also must
  337 copy/merge the distributed ``detectors/config.ini`` into your
  338 tracker's detectors directory. Most people can copy
  339 ``detectors/config.ini`` from the distributed templates as they won't
  340 have a ``detectors/config.ini`` file. (Note this is
  341 ``detectors/config.ini`` do not confuse it with the main
  342 ``config.ini`` file at the root of the tracker home.)
  343 
  344 This enhancement is disabled by default. Enable it by changing the
  345 value in ``detectors/config.ini`` from:
  346 
  347    chatting_requires_two_users = False
  348 
  349 to
  350 
  351    chatting_requires_two_users = True
  352 
  353 (the values ``no`` and ``yes`` can also be used). Restart the tracker
  354 to enable the change.
  355 
  356 If you don't do this quite right you will see one of two error
  357 messages in the web interface when you try to update an issue with a
  358 message::
  359 
  360    Edit Error: Unsupported configuration option: Option
  361       STATUSAUDITOR_CHATTING_REQUIRES_TWO_USERS not found in
  362       detectors/config.ini.
  363    Contact tracker admin to fix.
  364 
  365 This happens if detectors/config.ini is not found or is missing the
  366 ``chatting_requires_two_users`` option in the ``statusauditor``
  367 section.
  368 
  369 If you have an incorrect value (say you use ``T`` rather than
  370 ``True``) you see a different error::
  371 
  372    Edit Error: Invalid value for
  373      DETECTOR::STATUSAUDITOR_CHATTING_REQUIRES_TWO_USERS: 'T'
  374    Allowed values: yes, no
  375 
  376 to fix this set the value to ``yes`` (True) or ``no`` (False).
  377 
  378 Responsive template changes
  379 ---------------------------
  380 
  381 There have been some changes to the responsive template. You can
  382 diff/merge these changes into your responsive template based tracker.
  383 
  384 Jinja template changes
  385 ----------------------
  386 
  387 Auto escaping has been enabled in the jinja template engine, this
  388 means it is no longer necessary to manually escape dynamic strings
  389 with "\|e", but strings that should not be escaped need to be marked
  390 with "\|safe" (e.g. "{{ context.history()|u|safe }}"). Also, the i18n
  391 extension has been enabled and the template has been updated to use
  392 the extension for translatable text instead of explicit "i18n.gettext"
  393 calls:
  394 
  395    {% trans %}List of issues{% endtrans %}
  396 
  397 instead of:
  398 
  399    {{ i18n.gettext('List of issues')|u }}
  400 
  401 The jinja template has been upgraded to use bootstrap 4.1.3 (from
  402 2.2.2). You can diff/merge changes into your jinja template based
  403 tracker.
  404 
  405 Also search _generic.index.html, navigation.html and file.index.html
  406 in the html directory of your tracker. Look for::
  407 
  408    <input type="hidden" name="@action"
  409 
  410 where the value is a jinja expression that calls i18n.gettext. Set the
  411 value to the argument of the gettext call. E.G. replace::
  412 
  413     <input type="hidden" name="@action" value="{{ i18n.gettext('editCSV')|u }}">
  414 
  415 with::
  416 
  417     <input type="hidden" name="@action" value="editCSV">
  418 
  419 The action keywords should not be translated.
  420 
  421 .. index:: Upgrading; 1.5.1 to 1.6.0
  422 
  423 Migrating from 1.5.1 to 1.6.0
  424 =============================
  425 
  426 Update tracker config file
  427 --------------------------
  428 
  429 After installing the new version of roundup, you should
  430 update the ``config.ini`` file for your tracker. To do this:
  431 
  432  1. backup your existing ``config.ini`` file
  433  2. using the newly installed code, run::
  434 
  435        roundup-admin -i /path/to/tracker updateconfig config.ini.new
  436 
  437     to create the file config.ini.new. Replace
  438     ``/path/to/tracker`` with the path to your tracker.
  439  3. replace your tracker's config.ini with config.ini.new
  440 
  441 Using updateconfig keeps all the settings from your
  442 tracker's config.ini file and adds settings for all the new
  443 options.
  444 
  445 If you have added comments to your original config.ini file,
  446 merge the added comments into the config.ini.new file.  Then
  447 replace your tracker's config.ini with config.ini.new.
  448 
  449 Read the new config.ini and configure it to enable new
  450 features. Details on using these features can be found in
  451 this section.
  452 
  453 Make sure that user can view labelprop on classes (REQUIRED)
  454 ------------------------------------------------------------
  455 
  456 If you have View permissions that use ```properties=...```,
  457 make sure that the labelprop for the class is listed in the
  458 properties list.
  459 
  460 The first one of these that exists must must be in the list:
  461 
  462  1. the property set by a call to setlabelprop for the class
  463  2. the key of the class (as set by setkey())
  464  3. the "name" property (if it exists)
  465  4. the "title" property (if it exists)
  466 
  467 if none of those apply, you must allow
  468 
  469  * the "id" property
  470 
  471 E.G. If your class does a setlabelprop("foo") you must include "foo"
  472 in the properties list even if the class has name or title properties.
  473 
  474 See:
  475 http://www.roundup-tracker.org/docs/customizing.html#setlabelprop-property
  476 for further details on the labelprop.
  477 
  478 If you don't do this, you will find that multilinks (and possibly
  479 links) may not be displayed properly. E.G. templates that iterate over
  480 a mutlilink field (with tal:repeat for example) may not show any
  481 content.
  482 
  483 See: https://sourceforge.net/p/roundup/mailman/message/35763294/
  484 for the initial discussion of the issue.
  485 
  486 Cross Site Request Forgery Detection Added
  487 ------------------------------------------
  488 
  489 Roundup 1.6. supports a number of defenses against CSRF.
  490 
  491 Http header verification against the tracker's ``web``
  492 setting in the ``[tracker]`` section of config.ini for the
  493 following headers:
  494 
  495  1. Analyze the ``Referer`` HTTP header to make sure it
  496     includes the web setting.
  497  2. Analyze the ``Origin`` HTTP header to make sure the
  498     schema://host matches the web setting.
  499  3. Analyze the ``X-Forwarded-Host`` header set by a proxy
  500     running in front of roundup to make sure it agrees with
  501     the host part of the web setting.
  502  4. Analyze the ``Host`` header to make sure it agrees with
  503     the host part of the web setting. This is not done if
  504     ``X-Forwarded-Host`` is set.
  505 
  506 By default roundup 1.6 does not require any specific header
  507 to be present. However at least one of the headers above
  508 *must* pass validation checks (usually ``Host`` or
  509 ``Referer``) or the submission is rejected with an error.
  510 If any header fails validation, the submission is
  511 rejected. (Note the user's form keeps all the data they
  512 entered if it was rejected.)
  513 
  514 Also the admin can include unique csrf tokens for all forms
  515 submitted using the POST method. (Delete and put methods are also
  516 included, but not currently used by roundup.) The csrf
  517 token (nonce) is tied to the user's session. When the user
  518 submits the form and nonce, the nonce is checked to make
  519 sure it was issued to the user and the same session. If this
  520 is not true the post is rejected and the user is notified.
  521 
  522 The standard context/submit templating item creates CSRF tokens by
  523 default. If you have forms using the POST method that are not using
  524 the standard submit routine, you should add the following field to all
  525 forms:
  526 
  527    <input name="@csrf" type="hidden"
  528       tal:attributes="value python:utils.anti_csrf_nonce()">
  529 
  530 A unique random token is generated by every call to
  531 utils.anti_csrf_nonce() and is put in a database to be
  532 retreived if the token is used. Token lifetimes are 2 weeks
  533 by default but can be configured in config.ini. Roundup will
  534 automatically prune old tokens. Calling anti_csrf_nonce with
  535 an integer lifetime, for example:
  536 
  537    <input name="@csrf" type="hidden"
  538       tal:attributes="value python:utils.anti_csrf_nonce(lifetime=10)">
  539 
  540 sets the lifetime of that nonce to 10 minutes.
  541 
  542 If you want to change the default settings, you have to
  543 update the web section in your tracker's config.ini file. Follow the
  544 section above to generate an updated config.ini file. Then
  545 look for settings that start with csrf. The updated config.ini
  546 file includes detailed descriptions of the settings.
  547 
  548 In general one of four values can be set for these
  549 settings. The default is ``yes``, which validates the header
  550 or nonce and blocks access if the validation fails. If the
  551 field/header is missing it allows access. Setting these
  552 fields to ``required`` blocks access if the header/nonce is
  553 missing.
  554 
  555 It is recommended that you change your templates so every form
  556 that is not submitted via GET has an @csrf field. Then change
  557 the csrf_enforce_token setting to 'required'.
  558 
  559 Errors and Troubleshooting - @csrf in url
  560 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  561 
  562 If you see the @csrf nonce in the URL, you have added the value to a
  563 form that uses the GET method. You should remove the @csrf token from
  564 these forms as it is not needed.
  565 
  566 Errors and Troubleshooting - AttributeError list object no attribute value
  567 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  568 If you get an error:
  569 
  570   AttributeError: 'list' object has no attribute 'value'
  571 
  572 in handle_csrf, you have more than one @csrf token for the form. This
  573 usually occurs because the form uses the standard context/submit
  574 element but you also added an explicit @csrf statement. Simply remove
  575 the @csrf element for that form.
  576 
  577 Errors and Troubleshooting - xmlrpc Required Header Missing
  578 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  579 When performing and xmlrpc call, if you see something like:
  580 
  581   xmlrpclib.Fault: <Fault 1: "<class
  582      'roundup.exceptions.UsageError'>:Required Header Missing">
  583 
  584 change the setting of csrf_enforce_header_x-requested-with in
  585 config.ini to no. So it looks like:
  586 
  587   csrf_enforce_header_x-requested-with = no
  588 
  589 Alternatively change your xmlrpc client to add appropriate headers to
  590 the request including the:
  591 
  592   X-Requested-With:
  593 
  594 header as well as any other required csrf headers (e.g. referer, origin)
  595 configured in config.ini. See the advanced python client at the end of 
  596 the `xmlrpc guide`_.
  597 
  598 Support for SameSite cookie option for session cookie
  599 -----------------------------------------------------
  600 
  601 Support for serving the session cookie using the SameSite cookie option
  602 has been added. By default it is set to lax to provide a better user
  603 experience. But this can be changes to strict or the option can be
  604 removed entirely.
  605 
  606 Using the process for merging config.ini changes described in
  607 `Cross Site Request Forgery Detection Added`_ you can add the
  608 ``samesite_cookie_setting`` to the ``[web]`` section of the config
  609 file.
  610 
  611 Fix for path traversal changes template resolution
  612 --------------------------------------------------
  613 
  614 The templates in the tracker's html subdirectory must not be
  615 symbolic links that lead outside of the html directory.
  616 
  617 If you don't use symbolic links for templates in your html
  618 subdirectory you don't have to make any changes. Otherwise you need to
  619 replace the symbolic links with hard links to the files or replace the
  620 symbolic links with the files.
  621 
  622 This is a side effect of fixing a path traversal security issue.  The
  623 security issue required a directory with a specific unusual name. This
  624 made it difficult to exploit. However allowing the use of
  625 subdirectories to organize the templates required that it be fixed.
  626 
  627 
  628 Database back end specified in config.ini (REQUIRED)
  629 ----------------------------------------------------
  630 
  631 The ``db/backend_name`` file is no longer used to configure the database
  632 backend being used for a tracker. The backend is now configured in the
  633 ``config.ini`` file using the ``backend`` option located in the ``[rdbms]``
  634 section. For example if ``db/backend_name`` file contains ``sqlite``, a new
  635 entry in the tracker's ``config.ini`` will need to be created::
  636 
  637   [rdbms]
  638 
  639   ...
  640 
  641   # Database backend.
  642   # Default:
  643   backend = sqlite
  644 
  645 Once the ``config.ini`` file has been updated with the new ``backend`` option,
  646 you can safely delete the ``db/backend_name`` file.
  647 
  648 Note: the ``backend_name`` file may be located in a directory other than
  649 ``db/`` if you have configured the ``database`` option in the ``[main]``
  650 section of the ``config.ini`` file to be something other than ``db``.
  651 
  652 Note 2: if you are using the anydbm back end, you still set
  653 it using the backend option in the rdbms section of the
  654 config.ini file.
  655 
  656 New config file option 'indexer' added
  657 --------------------------------------
  658 
  659 This release added support for the Whoosh indexer, so a new
  660 config file option has been
  661 added. You can force Roundup to use a particular text indexer by
  662 setting this value in the [main] section of the tracker's
  663 ``config.ini`` file (usually placed right before indexer_stopwords)::
  664 
  665   [main]
  666 
  667   ...
  668 
  669   # Force Roundup to use a particular text indexer.
  670   # If no indexer is supplied, the first available indexer
  671   # will be used in the following order:
  672   # Possible values: xapian, whoosh, native (internal).
  673   indexer =
  674 
  675 Errors and Troubleshooting - Full text searching not working
  676 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  677 
  678 If after the upgrade full text searching is not working try changing
  679 the indexer value. If this is failing most likely you need to set
  680 '''indexer = native''' to use the rdbms or db text indexing systems.
  681 
  682 Alternatively you can do a
  683 '''roundup-admin -i /path/to/tracker reindex'''
  684 to generate a new index using roundup's preferred indexer from the
  685 list above.
  686 
  687 Xapian error with flint when reindexing
  688 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  689 If you reindex and are using xapian, you may get the error that
  690 "flint" is not supported (looks like flint was removed after xapian
  691 1.2.x). To fix this, you can delete the full text search database
  692 located in the tracker home directory in the file '''db/text-index'''
  693 and then perform a reindex.
  694 
  695 Stemming improved in Xapian Indexer
  696 -----------------------------------
  697 
  698 Stemming allows a search for "silent" also match silently.  The Porter
  699 stemmer in Xapian works with lowercase English text. In this release we
  700 lowercase the documents as they are put into the indexer.
  701 
  702 This means capitalization is not preserved, but produces more hits by
  703 using the stemmer.
  704 
  705 You will need to do a roundup-admin reindex if you are using the
  706 Xapian full text indexer on your tracker.
  707 
  708 
  709 New config file option 'replyto_address' added
  710 ----------------------------------------------
  711 
  712 A new config file option has been added to let you control the
  713 Reply-To header on nosy messages.
  714 
  715 Edit your tracker's ``config.ini`` and place the following after
  716 the email entry in the tracker section::
  717 
  718   [tracker]
  719   ...
  720 
  721   # Controls the reply-to header address used when sending
  722   # nosy messages.
  723   # If the value is unset (default) the roundup tracker's
  724   # email address (above) is used.
  725   # If set to "AUTHOR" then the primary email address of the
  726   # author of the change will be used as the reply-to
  727   # address. This allows email exchanges to occur outside of
  728   # the view of roundup and exposes the address of the person
  729   # who updated the issue, but it could be useful in some
  730   # unusual circumstances.
  731   # If set to some other value, the value is used as the reply-to
  732   # address. It must be a valid RFC2822 address or people will not be
  733   # able to reply.
  734   # Default: 
  735   replyto_address = 
  736 
  737 Login from a search or after logout works better (REQUIRED)
  738 -----------------------------------------------------------
  739 
  740 The login form has been improved to work with some back end code
  741 changes. Now when a user logs in they stay on the same page where they
  742 started the login. To make this work, you must change the tal that is
  743 used to set the ``__came_from`` form variable. Note that the url
  744 assigned to __came_from must be url encoded/quoted and be under the
  745 tracker's base url. If the base_url uses http, you can set the url to 
  746 https.
  747 
  748 Replace the existing code in the tracker's html/page.html page that
  749 looks similar to (look for name="__came_from")::
  750 
  751   <input type="hidden" name="__came_from" tal:attributes="value string:${request/base}${request/env/PATH_INFO}">
  752 
  753 with the following::
  754 
  755   <input type="hidden" name="__came_from"
  756     tal:condition="exists:request/env/QUERY_STRING"
  757     tal:attributes="value string:${request/base}${request/env/PATH_INFO}?${request/env/QUERY_STRING}">
  758   <input type="hidden" name="__came_from"
  759     tal:condition="not:exists:request/env/QUERY_STRING"
  760     tal:attributes="value string:${request/base}${request/env/PATH_INFO}">
  761 
  762 Now search backwards for the nearest form statement before the code
  763 that sets __came_from. If it looks like::
  764 
  765      <form method="post" action="#">
  766 
  767 replace it with::
  768 
  769      <form method="post" tal:attributes="action request/base">
  770 
  771 or with::
  772 
  773      <form method="post" tal:attributes="action string:${request/env/PATH_INFO}">
  774 
  775 the important part is that the action field **must not** include any query
  776 parameters ('#' includes query params).
  777 
  778 Errors and Troubleshooting - Unrecognized scheme in ...
  779 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  780 
  781 One symptom of failing to do this is getting an error:
  782 
  783    Unrecognized scheme in ....
  784 
  785 where the .... changes depending on the url path. You can see this
  786 when logging in from any screen other than the main index.
  787 
  788 Option to make adding multiple keywords more convenient
  789 -------------------------------------------------------
  790 
  791 In the classic tracker, after adding a new keyword you are redirected
  792 to the page for the new keyword so you can change the keyword's
  793 name. This is usually not desirable as you usually correctly set the
  794 keyword's name when creating the keyword. The new classic tracker has
  795 a new checkbox (checked by default) that keeps you on the same page so
  796 you can add a new keywords one after the other.
  797 
  798 To add this to your own tracker, add the following code (prefixed with
  799 a +) after the entry box for the new keyword in html/keyword.item.html::
  800 
  801       <tr>
  802        <th i18n:translate="">Keyword</th>
  803        <td tal:content="structure context/name/field">name</td>
  804    +   <td tal:condition="not:context/id">
  805    +     <tal:comment tal:replace="nothing">
  806    +       If we get here and do not have an id, we are creating a new
  807    +       keyword. It would be nice to provide some mechanism to
  808    +       determine the preferred state of the "Continue adding keywords"
  809    +       checkbox. By default it is enabled.
  810    +     </tal:comment>
  811    +     <input type="checkbox" id="continue_new_keyword"
  812    +           name="__redirect_to"
  813    +           tal:attributes="value
  814    +                           string:${request/base}${request/env/PATH_INFO}?@template=item;
  815    +                           checked python:True" />
  816    +     <label for="continue_new_keyword" i18n:translate="">Continue adding keywords.</label>
  817    +   </td>
  818       </tr>
  819 
  820 Note remove the leading '+' when adding this to the templates.
  821 
  822 The key component here is support for the '__redirect_to' query
  823 property. It is a url which can be used when creating any new item
  824 (issue, user, keyword ....). It controls the next page displayed after
  825 creating the item. If '__redirect_to' is not set, then you end up on
  826 the page for the newly created item. The url value assigned to
  827 __redirect_to must start with the tracker's base url and must be properly
  828 url encoded.
  829 
  830 Helper popups trigger change events on the original page
  831 --------------------------------------------------------
  832 
  833 The helper popups used to set dates (from a calendar), change lists of
  834 users or lists of issues did not notify the browser that the fields
  835 had been changed. This release adds code to trigger the change event.
  836 
  837 To add the change event to the calendar popup, you don't need to do
  838 any changes to the tracker. It is all done in the roundup python code
  839 in templating.py.
  840 
  841 To add the change event when updating users using the help-submit
  842 template, copy
  843 share/roundup/templates/devel/html/_generic.help-submit.html and
  844 replace your tracker's html/_generic.help-submit.html.  If you have
  845 done local changes to this file, change your file to include the code
  846 that defines the onclick event for the input field with
  847 id="btn_apply".
  848 
  849 To add the change event when updating lists of issues copy
  850 share/roundup/templates/devel/html/help_controls.js to your tracer's
  851 html directory. If you have made local changes to the javascript file,
  852 merge the two if/else blocks labeled::
  853 
  854    /* trigger change event on the field we changed */
  855 
  856 into your help_controls.js
  857 
  858 html/_generic.404.html in trackers use page template
  859 ----------------------------------------------------
  860 
  861 The original generic 404 error pages for many trackers did not use the
  862 standard page layout. This change replaces the html/_generic.404.html
  863 page with one that uses the page template.
  864 
  865 If your deployed tracker is based on: classic, minimal, responsive or
  866 devel templates and has not changed the html/_generic.404.html file,
  867 you can copy in the new file to get this additional functionality.
  868 
  869 Organize templates into subdirectories
  870 --------------------------------------
  871 
  872 The @template parameter to the web interface allows the use of
  873 subdirectories. So a setting of @template=view/view for an issue would
  874 use the template in the tracker's html/view/issue.view.html. Similarly
  875 for a caller class, you could put all the templates under the
  876 html/caller directory with names like: html/caller/caller.item.html,
  877 html/caller/caller.index.html etc. You may want to symbolically link the
  878 html/_generic* templates into your subdirectory so that missing
  879 templates (e.g. a missing caller.edit.html template) can be satisfied
  880 by the _generic.edit.html template.
  881 
  882 Properly quote query dispname (displayed name) in page.html
  883 -----------------------------------------------------------
  884 
  885 A new method has been added to HTMLStringProperty called url_quote.
  886 The default templates have been updated to use this in the "Your
  887 Query" section of the trackers html/page.html file. You will want to
  888 change your template. Lines starting with - are the original line and
  889 you want to change it to match the line starting with the + (remove
  890 the + from the line)::
  891 
  892       <tal:block tal:repeat="qs request/user/queries">
  893   -    <a href="#" tal:attributes="href string:${qs/klass}?${qs/url}&@dispname=${qs/name}"
  894   +    <a href="#" tal:attributes="href string:${qs/klass}?${qs/url}&@dispname=${qs/name/url_quote}"
  895 	  tal:content="qs/name">link</a><br>
  896       </tal:block>
  897 
  898 Find the tal:repeat line that loops over all queries. Then
  899 change the value assigned to @dispname in the href attribute from
  900 ${qs/name} to ${qs/name/url_quote}. Note that you should *not* change
  901 the value for tal:content.
  902 
  903 Allow "Show Unassigned" issues link to work for Anonymous user
  904 --------------------------------------------------------------
  905 
  906 In this release the anonymous user is allowed to search the user
  907 class. The following was added to the schema for all templates that
  908 provide the search option::
  909 
  910    p = db.security.addPermission(name='Search', klass='user')
  911    db.security.addPermissionToRole ('Anonymous', p)
  912 
  913 If you are running a tracker that **does not** allow read access for
  914 anonymous, you should remove this entry as it can be used to perform
  915 a username guessing attack against a roundup install.
  916 
  917 Errors and Troubleshooting - Unassigned issues for anonymous
  918 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  919 
  920 If you notice that the "Unassigned Issues" search on page.html
  921 is displaying assigned issues for users with the Anonymous role,
  922 you need to allow search permissions for the user class.
  923 
  924 Improvements in Classic Tracker query.edit.html template
  925 --------------------------------------------------------
  926 
  927 There is a new query editing template included in the distribution at:
  928 
  929    ``share/roundup/templates/classic/html/query.edit.html``
  930 
  931 This template fixes:
  932 
  933  * public query could not be removed from "Your Queries" once it was added.
  934    Trying to do so would cause a permissions error.
  935  * private yes/no dropdown always showed "yes" regardless of
  936    underlying state
  937  * query Delete button did not work.
  938  * same query being displayed multiple times
  939 
  940 It also adds:
  941  * the table layout displays queries created by the user first,
  942    then available public queries.
  943  * public query owners are shown
  944  * better support for deleted queries. When a query is deleted, it is
  945    still available for those who added it to their query list. If you
  946    are the query owner, you can restore (undelete) the query. If you
  947    are not the owner you can remove it from your query list.
  948    (If a query is deleted and nobody had it in their query list, it
  949    will not show up in the "Active retired queries" section. You will
  950    have to use the class editor or roundup_admin command line to
  951    restore it.)
  952  * notifies the user that delete/restore requires javascript. It
  953    always did, but that requirement wasn't displayed.
  954 
  955 To use the new template, you must add Restore permission on queries to
  956 allow the user to restore queries (see below).
  957 
  958 If you have not modified the query.edit.html template in your tracker,
  959 you should be able to copy the new version from the location above.
  960 Otherwise you will have to merge the changes into your modified template.
  961 
  962 Add the query Restore permission for the User role to your tracker's
  963 schema.py file.  Place it right after the query retire permission for
  964 the user role. After the change it should look like::
  965 
  966   p = db.security.addPermission(name='Retire', klass='query', check=edit_query,
  967       description="User is allowed to retire their queries")
  968   db.security.addPermissionToRole('User', p)
  969   p = db.security.addPermission(name='Restore', klass='query',
  970       check=edit_query,
  971       description="User is allowed to restore their queries")
  972   db.security.addPermissionToRole('User', p)
  973 
  974 where the last four lines are the ones you need to add.
  975 
  976 Usually you can add this to your User role. If all users have the User
  977 role in common then all logged in users should be ok. If you have
  978 users who do not include the User role (e.g. they may only have a
  979 Provisional role), you should add the search permission to that role
  980 (e.g. Provisional) as well if you allow them to edit their list of
  981 queries.
  982 
  983 Also see the `new search permissions for query in 1.4.17`_ section
  984 discussing search permission requirements for editing queries. The
  985 fixes in this release require the ability to search the creator of all
  986 queries to work correctly.
  987 
  988 If the test script for the `new search permissions for query in
  989 1.4.17`_ doesn't report that a role has the ability to search queries
  990 or at least search the creator property for queries, add the following
  991 permissions to your schema.py::
  992 
  993   s = db.security.addPermission(name='Search', klass='query',
  994     properties=['creator'],
  995     description="User is allowed to Search queries for creator")
  996   db.security.addPermissionToRole('User', s)
  997 
  998 Errors and Troubleshooting - Public queries listed twice when editing
  999 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 1000 
 1001 If you do not do this, public queries will be listed twice in the edit
 1002 interface. Once in the "Queries I created" section and again in the
 1003 "Queries others created" section of the query edit page
 1004 (``http..../query?@template=edit``).
 1005 
 1006 Fix security issues in query.item.html template
 1007 -----------------------------------------------
 1008 The default query.item.html template allows anybody to view all
 1009 queries.
 1010 
 1011 This has been updated in the classic, devel and responsive templates
 1012 to only allow people to view queries they creates or queries that are
 1013 publicly viewable.
 1014 
 1015 If you haven't modified you query.item.html template, simply copy the
 1016 query.item.html template from one of the above default templates to
 1017 your tracker's html directory.
 1018 
 1019 Enhancement to check command for Permissions
 1020 --------------------------------------------
 1021 
 1022 A new form of check function is permitted in permission definitions.
 1023 The three argument form is still supported and will work the same
 1024 as it always has (although it may be depricated in the future).
 1025 
 1026 If the check function is defined as::
 1027 
 1028       check(db, userid, itemid, **ctx)
 1029 
 1030 the ctx variable will have the context to use when determining access
 1031 rights::
 1032 
 1033      ctx['property'] the name of the property being checked or None if
 1034             it's a class check.
 1035 
 1036      ctx['classname'] the name of the class that is being checked
 1037             (issue, query ....).
 1038 
 1039      ctx['permission'] the name of the permission (e.g. View, Edit...).
 1040 
 1041 This should make defining complex permissions much easier. Consider::
 1042 
 1043     def issue_private_access(db, userid, itemid, **ctx):
 1044         if not db.issue.get(itemid, 'private'):
 1045            # allow access to everything if not private	
 1046 	   return True
 1047 
 1048 	# It is a private issue hide nosy list
 1049 	# Note that the nosy property *must* be listed
 1050 	# in permissions argument to the addPermission
 1051 	# definition otherwise this check command
 1052 	# is not run.
 1053 	if ctx['property'] == 'nosy':
 1054 	   return False # deny access to this property
 1055 
 1056 	# allow access for editing, viewing etc. of the class
 1057 	return True
 1058 
 1059 
 1060     e = db.security.addPermission(name='Edit', klass='issue',
 1061                                   check=issue_private_access,
 1062 				  properties=['nosy'],
 1063                                   description="Edit issue checks")
 1064 
 1065 It is suggested that you change your checks to use the ``**ctx``
 1066 parameter. This is expected to be the preferred form in the future.
 1067 You do not need to use the ``ctx`` parameter in the function if you do
 1068 not need it.
 1069 
 1070 Changes to property permissions
 1071 -------------------------------
 1072 
 1073 If you create a permission:
 1074 
 1075     db.security.addPermission(name='View', klass='user',
 1076        properties=['theme'], check=own_record,
 1077        description="User is allowed to view their own theme")
 1078 
 1079 that combines checks and properties, the permission also matches a
 1080 permission check for the View permission on the user class.  So this
 1081 also allows the user to see their user record. It is unexpected that
 1082 checking for access without a property would match this permission.
 1083 
 1084 This release adds support for making a permission like above only be
 1085 used during property permission tests. See ``customizing.txt`` and
 1086 search for props_only and set_props_only_default in the section
 1087 'Adding a new Permission'
 1088 
 1089 Improve query editing
 1090 ---------------------
 1091 
 1092 If a user creates a query with the same name as one of their existing
 1093 queries, the query editing interface will now report an error. By
 1094 default the query editing page (issue.search.html) displays the index
 1095 page when the search is triggered. This is usually correct since the
 1096 user expects to see the results of the query. But now that
 1097 the code properly checks for duplicate search names, the user should
 1098 stay on the search page if there is an error. To add this to your
 1099 existing issue.search.html page, add the following line after the
 1100 hidden field @old-queryname:
 1101 
 1102   <input type="hidden" name="@template" value="index|search"/>
 1103 
 1104 With this addition, the index template is displayed if there is no
 1105 error, and the user stays on the search template if there is an error.
 1106 
 1107 New -L (loghttpvialogger) option to roundup-server
 1108 --------------------------------------------------
 1109 
 1110 Http request logs from roundup-server are sent to stderr or
 1111 can be recorded in a log file (if -l or the logfile options
 1112 is used). However there is no way to rotate the logfile
 1113 without shutting down and restarting the roundup-server.
 1114 
 1115 If the -L flag is used, the python logging module is used
 1116 for logging the http requests. The name for the log
 1117 (qualname) is 'roundup.http'. You can direct these messages
 1118 to a rotating log file by putting the following::
 1119 
 1120    [loggers]
 1121    keys=roundup.http
 1122 
 1123    [logger_roundup.http]
 1124    level=INFO
 1125    handlers=rotate_weblog
 1126    qualname=roundup.http
 1127    propagate=0
 1128 
 1129    [handlers]
 1130    keys=rotate_weblog
 1131 
 1132    [handler_rotate_weblog]
 1133    class=logging.handlers.RotatingFileHandler
 1134    args=('httpd.log','a', 512000, 2)
 1135    formatter=plain
 1136 
 1137    [formatters]
 1138    keys=plain
 1139 
 1140    [formatter_plain]
 1141    format=%(message)s
 1142 
 1143 into a file (e.g. logging.ini). Then reference this file in
 1144 the 'config' value of the [logging] section in the trackers
 1145 config.ini file.
 1146 
 1147 Note the log configuration above is an example and can be
 1148 merged into a more full featured logging config file for
 1149 your tracker if you wish. It will create a new file in the
 1150 current working directory called 'httpd.log' and will rotate
 1151 the log file at 500K and keep two old copies of the file.
 1152 
 1153 .. index:: Upgrading; 1.5.0 to 1.5.1
 1154 
 1155 Migrating from 1.5.0 to 1.5.1
 1156 =============================
 1157 
 1158 User data visibility
 1159 --------------------
 1160 
 1161 For security reasons you should change the permissions on the user
 1162 class. We previously shipped a configuration that allowed users to see
 1163 too many of other users details, including hashed passwords under
 1164 certain circumstances. In schema.py in your tracker, replace the line::
 1165 
 1166    db.security.addPermissionToRole('User', 'View', 'user')
 1167 
 1168 with::
 1169 
 1170    p = db.security.addPermission(name='View', klass='user',
 1171        properties=('id', 'organisation', 'phone', 'realname',
 1172        'timezone', 'username'))
 1173    db.security.addPermissionToRole('User', p)
 1174 
 1175 Note that this removes visibility of user emails, if you want emails to
 1176 be visible you can add 'address' and 'alternate_addresses' to the list
 1177 above.
 1178 
 1179 XSS protection for custom actions
 1180 ---------------------------------
 1181 
 1182 If you have defined your own cgi actions in your tracker instance
 1183 (e.g. in a custom ``extensions/spambayes.py`` file) you need to modify
 1184 all cases where client.error_message or client.ok_message are modified
 1185 directly. Instead of::
 1186 
 1187   self.client.ok_message.append(...)
 1188 
 1189 you need to call::
 1190 
 1191   self.client.add_ok_message(...)
 1192 
 1193 and the same for::
 1194 
 1195   self.client.error_message.append(...)
 1196 
 1197 vs.::
 1198 
 1199   self.client.add_error_message(...)
 1200 
 1201 The new calls escape the passed string by default and avoid XSS security
 1202 issues.
 1203 
 1204 .. index:: Upgrading; 1.4.20 to 1.4.21
 1205 
 1206 Migrating from 1.4.20 to 1.4.21
 1207 ===============================
 1208 
 1209 The ``_generic.calendar.html`` page of the instance has been updated to include
 1210 ``<meta name="robots" content="noindex, nofollow" />``.  This prevents
 1211 robots to follow all the links in the calendar.  If you haven't modified the
 1212 page on your local instance, you can simply replace it with the one in
 1213 ``share/roundup/templates/classic/html/_generic.calendar.html``; if you did,
 1214 you can add the tag manually.  See issue2550765 and changeset a099ff2ceff3.
 1215 
 1216 If you are using the xml-rpc interface, there is a change
 1217   in accessing it. You can not send text/xml data to any
 1218   roundup url and get a response, you must use the /xmlrpc
 1219   url. For example, if you used to send your xmlrpc request to:
 1220 
 1221    http://myroundup.com/roundup
 1222 
 1223   you need to change the url to read:
 1224 
 1225    http://myroundup.com/roundup/xmlrpc
 1226 
 1227   to invoke the xmlrpc handler. This allows us to send xml
 1228   data to roundup for other handlers (e.g. REST, SOAP ...)
 1229   in the future.
 1230 
 1231 
 1232 .. index:: upgrading; 1.4.19 to 1.4.20
 1233 
 1234 Migrating from 1.4.19 to 1.4.20
 1235 ===============================
 1236 
 1237 Roundup used to allow certain HTML-Tags in OK- and Error-messages. Since
 1238 these messages are passed via the URL (due to roundup redirecting after
 1239 an edit), we did have security-issues (see issue2550724).
 1240 
 1241 If you have customized the OK or Error messages in your
 1242 roundup-installation and you were using features like bold or italic
 1243 in the message, you will have to do without this highlighting and
 1244 remove HTML tags from messages.
 1245 
 1246 If you were using <br> tags for multi-line messages, you now should use
 1247 newlines instead, these will be replaced with <br/> during formatting.
 1248 
 1249 Note that the previous implementation also allowed links inside
 1250 messages. Since these links could be set by an attacker, no links in
 1251 roundup messages are supported anymore. This does *not* affect the
 1252 "clear this message" link in OK-messages as it is generated by the
 1253 template and is not part of the OK-message.
 1254 
 1255 If you have not modified any roundup messages, you need not do anything,
 1256 the templates shipped with roundup did not use HTML tags in messages for
 1257 highlighting.
 1258 
 1259 
 1260 .. index:: upgrading; 1.4.17 to 1.4.18
 1261 
 1262 Migrating from 1.4.17 to 1.4.18
 1263 ===============================
 1264 
 1265 There was a bug in 1.4.17 where files were unlinked from issues if a
 1266 mail without attachment was received via the mail interface. The
 1267 following script will list likely issues being affected by the bug.
 1268 The date in the script is the date of the 1.4.17 release. If you have
 1269 installed 1.4.17 later than this date, you can change the date
 1270 appropriately to your installation date. Run the script in the directory
 1271 of your tracker::
 1272 
 1273   #!/usr/bin/python
 1274   import os
 1275   from roundup import instance
 1276   from roundup.date import Date
 1277   dir     = os.getcwd ()
 1278   tracker = instance.open (dir)
 1279   db      = tracker.open ('admin')
 1280   # you may want to change this to your install date to find less candidates
 1281   last_release = Date('2011-05-13')
 1282   affected = {}
 1283   for i in db.issue.getnodeids():
 1284       for j in db.issue.history(i):
 1285           if i in affected:
 1286               break
 1287           if j[1] < last_release or j[3] != 'set' or 'files' not in j[4]:
 1288               continue
 1289           for op, p in j[4]['files']:
 1290               if op == '-':
 1291                   affected [i] = 1
 1292                   break
 1293   print(', '.join(sorted(affected.keys())))
 1294 
 1295 To find out which files where attached before you can look in the
 1296 history of the affected issue.  For fixing issues you can re-attach the
 1297 files in question using the "set" command of roundup-admin, e.g., if the
 1298 list of files attached to an issue should be files 5, 17, 23 for issue42
 1299 you will set this using
 1300 
 1301 roundup-admin -i /path/to/your/tracker set issue42 files=5,17,23
 1302 
 1303 .. index:: upgrading; 1.4.x to 1.4.17
 1304 
 1305 Migrating from 1.4.x to 1.4.17
 1306 ==============================
 1307 
 1308 There is a new config-option `migrate_passwords` in section `web` to
 1309 auto-migrate passwords at web-login time to a more secure storage
 1310 scheme. Default for the new option is "yes" so if you don't want that
 1311 passwords are auto-migrated to a more secure password scheme on user
 1312 login, set this to "no" before running your tracker(s) after the
 1313 upgrade.
 1314 
 1315 The standalone roundup-server now defaults to listening on localhost (no
 1316 longer on all network interfaces). This will not affect you if you're
 1317 already using a configuration file for roundup-server. If you are using
 1318 an empty setting for the `host` parameter in the config-file you should
 1319 explicitly put 0.0.0.0 there as the use of an empty string to specify
 1320 listening to all interfaces is deprecated and will go away in a future
 1321 version.  If you are starting the server without a configuration file
 1322 and want to explicitly listen to all network interface, you should
 1323 specify the -n option with the address `0.0.0.0`.
 1324 
 1325 .. _new search permissions for query in 1.4.17:
 1326 
 1327 Searching now requires either read-permission without a check method, or
 1328 you will have to add a "Search" permission for a class or a list of
 1329 properties for a class (if you want to allow searching). For the classic
 1330 template (or other templates derived from it) you want to add the
 1331 following lines to your `schema.py` file::
 1332 
 1333   p = db.security.addPermission(name='Search', klass='query')
 1334   db.security.addPermissionToRole('User', p)
 1335 
 1336 This is needed, because for the `query` class users may view only their
 1337 own queries (or public queries). This is implemented with a `check`
 1338 method, therefore the default search permissions will not allow
 1339 searching and you'll have to add an explicit search permission.
 1340 If you have modified your schema, you can check if you're missing any
 1341 search permissions with the following script, run it in your tracker
 1342 directory, it will list for each Class and Property the roles that may
 1343 search for this property::
 1344 
 1345     #!/usr/bin/python
 1346     from __future__ import print_function
 1347     import os
 1348     from roundup import instance
 1349 
 1350     tracker = instance.open(os.getcwd ())
 1351     db = tracker.open('admin')
 1352 
 1353     for cl in sorted(db.getclasses()):
 1354         print("Class:", cl)
 1355         for p in sorted(db.getclass(cl).getprops(protected=True).keys()):
 1356             print("    Property:", p)
 1357             roles = []
 1358             for role in sorted(db.security.role.keys()):
 1359                 if db.security.roleHasSearchPermission(cl,p,role):
 1360                     roles.append(role)
 1361             print("        roles may search:", ', '.join(roles))
 1362 
 1363 
 1364 .. index:: upgrading; 1.4.x to 1.4.12
 1365 
 1366 Migrating from 1.4.x to 1.4.12
 1367 ==============================
 1368 
 1369 Item creation now checks the "Create" permission instead of the "Edit"
 1370 permission for individual properties. If you have modified your tracker
 1371 permissions from the default distribution, you should check that
 1372 "Create" permissions exist for all properties you want users to be able
 1373 to create.
 1374 
 1375 
 1376 Fixing some potential security holes
 1377 ------------------------------------
 1378 
 1379 Enhanced checking was added to the user registration auditor. If you
 1380 run a public tracker you should update your tracker's
 1381 ``detectors/userauditor.py`` using the new code from
 1382 ``share/roundup/templates/classic/detectors/userauditor.py``. In most
 1383 cases you may just copy the file over, but if you've made changes to
 1384 the auditor in your tracker then you'll need to manually integrate
 1385 the new code.
 1386 
 1387 Some HTML templates were found to have formatting security problems:
 1388 
 1389 ``html/page.html``::
 1390 
 1391   -tal:replace="request/user/username">username</span></b><br>
 1392   +tal:replace="python:request.user.username.plain(escape=1)">username</span></b><br>
 1393 
 1394 ``html/_generic.help-list.html``::
 1395 
 1396   -tal:content="structure python:item[prop]"></label>
 1397   +tal:content="python:item[prop]"></label>
 1398 
 1399 The lines marked "+" should be added and lines marked "-" should be
 1400 deleted (minus the "+"/"-" signs).
 1401 
 1402 
 1403 Some HTML interface tweaks
 1404 --------------------------
 1405 
 1406 You may wish to copy the ``user_utils.js`` and ``style.css` files from the
 1407 source distribution ``share/roundup/templates/classic/html/`` directory to the
 1408 ``html`` directory of your trackers as it includes a small improvement.
 1409 
 1410 If you have made local changes to those files you'll need to manually work
 1411 the differences in to your versions or ignore the changes.
 1412 
 1413 
 1414 .. index:: upgrading; 1.4.x to 1.4.11
 1415 
 1416 Migrating from 1.4.x to 1.4.11
 1417 ==============================
 1418 
 1419 Close potential security hole
 1420 -----------------------------
 1421 
 1422 If your tracker has untrusted users you should examine its ``schema.py``
 1423 file and look for the section granting the "Edit" permission to your users.
 1424 This should look something like::
 1425 
 1426     p = db.security.addPermission(name='Edit', klass='user', check=own_record,
 1427         description="User is allowed to edit their own user details")
 1428 
 1429 and should be modified to restrict the list of properties they are allowed
 1430 to edit by adding the ``properties=`` section like::
 1431 
 1432     p = db.security.addPermission(name='Edit', klass='user', check=own_record,
 1433         properties=('username', 'password', 'address', 'realname', 'phone',
 1434             'organisation', 'alternate_addresses', 'queries', 'timezone'),
 1435         description="User is allowed to edit their own user details")
 1436 
 1437 Most importantly the "roles" property should not be editable - thus not
 1438 appear in that list of properties.
 1439 
 1440 
 1441 Grant the "Register" permission to the Anonymous role
 1442 -----------------------------------------------------
 1443 
 1444 A separate "Register" permission has been introduced to allow
 1445 anonymous users to register. This means you will need to add the
 1446 following to your tracker's ``schema.py`` to add the permission and
 1447 assign it to the Anonymous role (replacing any previously assigned
 1448 "Create user" permission for the Anonymous role)::
 1449 
 1450   +db.security.addPermission(name='Register', klass='user',
 1451   +     description='User is allowed to register new user')
 1452 
 1453    # Assign the appropriate permissions to the anonymous user's Anonymous
 1454    # Role. Choices here are:
 1455    # - Allow anonymous users to register
 1456   -db.security.addPermissionToRole('Anonymous', 'Create', 'user')
 1457   +db.security.addPermissionToRole('Anonymous', 'Register', 'user')
 1458 
 1459 The lines marked "+" should be added and lines marked "-" should be
 1460 deleted (minus the "+"/"-" signs).
 1461 
 1462 You should also modify the ``html/page.html`` template to change the
 1463 permission tested there::
 1464 
 1465    -tal:condition="python:request.user.hasPermission('Create', 'user')"
 1466    +tal:condition="python:request.user.hasPermission('Register', 'user')"
 1467 
 1468 
 1469 Generic class editor may now restore retired items
 1470 --------------------------------------------------
 1471 
 1472 The instructions for doing so won't be present in your tracker unless you copy
 1473 the ``_generic.index.html`` template from the roundup distribution in
 1474 ``share/roundup/templates/classic/html`` to your tracker's ``html`` directory.
 1475 
 1476 
 1477 .. index:: upgrading; 1.4.x to 1.4.9
 1478 
 1479 Migrating from 1.4.x to 1.4.9
 1480 =============================
 1481 
 1482 Customized MailGW Class
 1483 -----------------------
 1484 
 1485 If you have customized the MailGW class in your tracker: The new MailGW
 1486 class opens the database for each message in the method handle_message
 1487 (instance.open) instead of passing the opened database as a parameter to
 1488 the MailGW constructor. The old handle_message has been renamed to
 1489 _handle_message. The new method opens the database and wraps the call to
 1490 the old method into a try/finally.
 1491 
 1492 Your customized MailGW class needs to mirror this behavior.
 1493 
 1494 Fix the "remove" button in issue files and messages lists
 1495 ---------------------------------------------------------
 1496 
 1497 The "remove" button(s) in the issue messages list needs to be altered. Find
 1498 the following in your tracker's ``html/issue.item.html`` template::
 1499 
 1500   <td>
 1501    <form style="padding:0" tal:condition="context/is_edit_ok"
 1502          tal:attributes="action string:issue${context/id}">
 1503     <input type="hidden" name="@remove@files" tal:attributes="value file/id">
 1504 
 1505 and add ``method="POST"`` as shown below::
 1506 
 1507   <td>
 1508    <form style="padding:0" method="POST" tal:condition="context/is_edit_ok"
 1509          tal:attributes="action string:issue${context/id}">
 1510     <input type="hidden" name="@remove@files" tal:attributes="value file/id">
 1511 
 1512 Then also find::
 1513 
 1514   <td>
 1515    <form style="padding:0" tal:condition="context/is_edit_ok"
 1516          tal:attributes="action string:issue${context/id}">
 1517     <input type="hidden" name="@remove@messages" tal:attributes="value msg/id">
 1518 
 1519 and add ``method="POST"`` as shown below::
 1520 
 1521   <td>
 1522    <form style="padding:0" method="POST" tal:condition="context/is_edit_ok"
 1523          tal:attributes="action string:issue${context/id}">
 1524     <input type="hidden" name="@remove@messages" tal:attributes="value msg/id">
 1525 
 1526 
 1527 Fixing the "retire" button in user management list
 1528 --------------------------------------------------
 1529 
 1530 Some previous versions of this upgrading document missed ``method="POST"``
 1531 in the change to the "retire" link in the user management list
 1532 in section `Migrating from 1.4.x to 1.4.7`_.
 1533 Make sure the change is done as listed below in this document.
 1534 
 1535 
 1536 .. index:: upgrading; 1.4.x to 1.4.7
 1537 
 1538 Migrating from 1.4.x to 1.4.7
 1539 =============================
 1540 
 1541 Several security issues were addressed in this release. Some aspects of your
 1542 trackers may no longer function depending on your local customisations. Core
 1543 functionality that will need to be modified:
 1544 
 1545 Grant the "retire" permission to users for their queries
 1546 --------------------------------------------------------
 1547 
 1548 Users will no longer be able to retire their own queries. To remedy this you
 1549 will need to add the following to your tracker's ``schema.py`` just under the
 1550 line that grants them permission to edit their own queries::
 1551 
 1552    p = db.security.addPermission(name='Edit', klass='query', check=edit_query,
 1553       description="User is allowed to edit their queries")
 1554    db.security.addPermissionToRole('User', p)
 1555  + p = db.security.addPermission(name='Retire', klass='query', check=edit_query,
 1556  +    description="User is allowed to retire their queries")
 1557  + db.security.addPermissionToRole('User', p)
 1558    p = db.security.addPermission(name='Create', klass='query',
 1559       description="User is allowed to create queries")
 1560    db.security.addPermissionToRole('User', p)
 1561 
 1562 The lines marked "+" should be added, minus the "+" sign.
 1563 
 1564 
 1565 Fix the "retire" link in the users list for admin users
 1566 -------------------------------------------------------
 1567 
 1568 The "retire" link found in the file ``html/user.index.html``::
 1569 
 1570   <td tal:condition="context/is_edit_ok">
 1571    <a tal:attributes="href string:user${user/id}?@action=retire&@template=index"
 1572     i18n:translate="">retire</a>
 1573 
 1574 Should be replaced with::
 1575 
 1576   <td tal:condition="context/is_retire_ok">
 1577      <form style="padding:0" method="POST"
 1578            tal:attributes="action string:user${user/id}">
 1579       <input type="hidden" name="@template" value="index">
 1580       <input type="hidden" name="@action" value="retire">
 1581       <input type="submit" value="retire" i18n:attributes="value">
 1582      </form>
 1583 
 1584 
 1585 Fix for Python 2.6+ users
 1586 -------------------------
 1587 
 1588 If you use Python 2.6 you should edit your tracker's
 1589 ``detectors/nosyreaction.py`` file to change::
 1590 
 1591    import sets
 1592 
 1593 at the top to::
 1594 
 1595    from roundup.anypy.sets_ import set
 1596 
 1597 and then all instances of ``sets.Set()`` to ``set()`` in the later code.
 1598 
 1599 
 1600 
 1601 Trackers currently allowing HTML file uploading
 1602 -----------------------------------------------
 1603 
 1604 Trackers which wish to continue to allow uploading of HTML content against issues
 1605 will need to set a new configuration variable in the ``[web]`` section of the
 1606 tracker's ``config.ini`` file:
 1607 
 1608    # Setting this option enables Roundup to serve uploaded HTML
 1609    # file content *as HTML*. This is a potential security risk
 1610    # and is therefore disabled by default. Set to 'yes' if you
 1611    # trust *all* users uploading content to your tracker.
 1612    # Allowed values: yes, no
 1613    # Default: no
 1614    allow_html_file = no
 1615 
 1616 
 1617 
 1618 .. index:: upgrading; 1.4.2 to 1.4.3
 1619 
 1620 Migrating from 1.4.2 to 1.4.3
 1621 =============================
 1622 
 1623 If you are using the MySQL backend you will need to replace some indexes
 1624 that may have been created by version 1.4.2.
 1625 
 1626 You should to access your MySQL database directly and remove any indexes
 1627 with a name ending in "_key_retired_idx". You should then re-add them with
 1628 the same spec except the key column name needs a size. So an index on
 1629 "_user (__retired, _name)" should become "_user (__retired, _name(255))".
 1630 
 1631 
 1632 .. index:: upgrading; 1.4.x to 1.4.2
 1633 
 1634 Migrating from 1.4.x to 1.4.2
 1635 =============================
 1636 
 1637 .. index:: roundup-admin; migrate subcommand
 1638 
 1639 You should run the "roundup-admin migrate" command for your tracker once
 1640 you've installed the latest codebase.
 1641 
 1642 Do this before you use the web, command-line or mail interface and before
 1643 any users access the tracker.
 1644 
 1645 This command will respond with either "Tracker updated" (if you've not
 1646 previously run it on an RDBMS backend) or "No migration action required"
 1647 (if you have run it, or have used another interface to the tracker,
 1648 or are using anydbm).
 1649 
 1650 It's safe to run this even if it's not required, so just get into the
 1651 habit.
 1652 
 1653 
 1654 .. index:: upgrading; 1.3.3 to 1.4.0
 1655 
 1656 Migrating from 1.3.3 to 1.4.0
 1657 =============================
 1658 
 1659 Value of the "refwd_re" tracker configuration option (section "mailgw")
 1660 is treated as UTF-8 string.  In previous versions, it was ISO8859-1.
 1661 
 1662 If you have running trackers based on the classic template, please
 1663 update the messagesummary detector as follows::
 1664 
 1665     --- detectors/messagesummary.py 17 Apr 2003 03:26:38 -0000      1.1
 1666     +++ detectors/messagesummary.py 3 Apr 2007 06:47:21 -0000       1.2
 1667     @@ -8,7 +8,7 @@
 1668      if newvalues.has_key('summary') or not newvalues.has_key('content'):
 1669          return
 1670 
 1671     -    summary, content = parseContent(newvalues['content'], 1, 1)
 1672     +    summary, content = parseContent(newvalues['content'], config=db.config)
 1673      newvalues['summary'] = summary
 1674 
 1675 In the latest version we have added some database indexes to the
 1676 SQL-backends (mysql, postgresql, sqlite) for speeding up building the
 1677 roundup-index for full-text search. We recommend that you create the
 1678 following database indexes on the database by hand::
 1679 
 1680  CREATE INDEX words_by_id ON __words (_textid);
 1681  CREATE UNIQUE INDEX __textids_by_props ON __textids (_class, _itemid, _prop);
 1682 
 1683 .. index:: upgrading; 1.2.x to 1.3.0
 1684 
 1685 Migrating from 1.2.x to 1.3.0
 1686 =============================
 1687 
 1688 1.3.0 Web interface changes
 1689 ---------------------------
 1690 
 1691 Some of the HTML files in the "classic" and "minimal" tracker templates
 1692 were changed to fix some bugs and clean them up. You may wish to compare
 1693 them to the HTML files in your tracker and apply any changes.
 1694 
 1695 
 1696 .. index:: upgrading; 1.1.2 to 1.2.0
 1697 
 1698 Migrating from 1.1.2 to 1.2.0
 1699 =============================
 1700 
 1701 1.2.0 Sorting and grouping by multiple properties
 1702 -------------------------------------------------
 1703 
 1704 Starting with this version, sorting and grouping by multiple properties
 1705 is possible. This means that request.sort and request.group are now
 1706 lists. This is reflected in several places:
 1707 
 1708  * ``renderWith`` now has list attributes for ``sort`` and ``group``,
 1709    where you previously wrote::
 1710 
 1711     renderWith(... sort=('-', 'activity'), group=('+', 'priority')
 1712 
 1713    you write now::
 1714 
 1715     renderWith(... sort=[('-', 'activity')], group=[('+', 'priority')]
 1716 
 1717  * In templates that permit to edit sorting/grouping, request.sort and
 1718    request.group are (possibly empty) lists. You can now sort and group
 1719    by multiple attributes. For an example, see the classic template. You
 1720    may want search for the variable ``n_sort`` which can be set to the
 1721    number of sort/group properties.
 1722 
 1723  * Templates that diplay new headlines for each group of items with
 1724    equal group properties can now use the modified ``batch.propchanged``
 1725    method that can take several properties which are checked for
 1726    changes. See the example in the classic template which makes use of
 1727    ``batch.propchanged``.
 1728 
 1729 .. index:: upgrading; 1.1.0 to 1.1.1
 1730 
 1731 Migrating from 1.1.0 to 1.1.1
 1732 =============================
 1733 
 1734 1.1.1 "Clear this message"
 1735 --------------------------
 1736 
 1737 In 1.1.1, the standard ``page.html`` template includes a "clear this message"
 1738 link in the green "ok" message bar that appears after a successful edit
 1739 (or other) action.
 1740 
 1741 To include this in your tracker, change the following in your ``page.html``
 1742 template::
 1743 
 1744  <p tal:condition="options/ok_message | nothing" class="ok-message"
 1745     tal:repeat="m options/ok_message" tal:content="structure m">error</p>
 1746 
 1747 to be::
 1748 
 1749  <p tal:condition="options/ok_message | nothing" class="ok-message">
 1750    <span tal:repeat="m options/ok_message"
 1751       tal:content="structure string:$m <br/ > " />
 1752     <a class="form-small" tal:attributes="href request/current_url"
 1753        i18n:translate="">clear this message</a>
 1754  </p>
 1755 
 1756 
 1757 If you implemented the "clear this message" in your 1.1.0 tracker, then you
 1758 should change it to the above and it will work much better!
 1759 
 1760 
 1761 .. index:: upgrading; 1.0.x to 1.1.0
 1762 
 1763 Migrating from 1.0.x to 1.1.0
 1764 =============================
 1765 
 1766 1.1 Login "For Session Only"
 1767 ----------------------------
 1768 
 1769 In 1.1, web logins are alive for the length of a session only, *unless* you
 1770 add the following to the login form in your tracker's ``page.html``::
 1771 
 1772     <input type="checkbox" name="remember" id="remember">
 1773     <label for="remember" i18n:translate="">Remember me?</label><br>
 1774 
 1775 See the classic tracker ``page.html`` if you're unsure where this should
 1776 go.
 1777 
 1778 
 1779 1.1 Query Display Name
 1780 ----------------------
 1781 
 1782 The ``dispname`` web variable has been renamed ``@dispname`` to avoid
 1783 clashing with other variables of the same name. If you are using the
 1784 display name feature, you will need to edit your tracker's ``page.html``
 1785 and ``issue.index.html`` pages to change ``dispname`` to ``@dispname``.
 1786 
 1787 A side-effect of this change is that the renderWith method used in the
 1788 ``home.html`` page may now take a dispname argument.
 1789 
 1790 
 1791 1.1 "Clear this message"
 1792 ------------------------
 1793 
 1794 In 1.1, the standard ``page.html`` template includes a "clear this message"
 1795 link in the green "ok" message bar that appears after a successful edit
 1796 (or other) action.
 1797 
 1798 To include this in your tracker, change the following in your ``page.html``
 1799 template::
 1800 
 1801  <p tal:condition="options/ok_message | nothing" class="ok-message"
 1802     tal:repeat="m options/ok_message" tal:content="structure m">error</p>
 1803 
 1804 to be::
 1805 
 1806  <p tal:condition="options/ok_message | nothing" class="ok-message">
 1807    <span tal:repeat="m options/ok_message"
 1808       tal:content="structure string:$m <br/ > " />
 1809     <a class="form-small" tal:attributes="href string:issue${context/id}"
 1810        i18n:translate="">clear this message</a>
 1811  </p>
 1812 
 1813 
 1814 .. index:: upgrading; 0.8.x to 1.0
 1815 
 1816 Migrating from 0.8.x to 1.0
 1817 ===========================
 1818 
 1819 1.0 New Query Permissions
 1820 -------------------------
 1821 
 1822 New permissions are defined for query editing and viewing. To include these
 1823 in your tracker, you need to add these lines to your tracker's
 1824 ``schema.py``::
 1825 
 1826  # Users should be able to edit and view their own queries. They should also
 1827  # be able to view any marked as not private. They should not be able to
 1828  # edit others' queries, even if they're not private
 1829  def view_query(db, userid, itemid):
 1830      private_for = db.query.get(itemid, 'private_for')
 1831      if not private_for: return True
 1832      return userid == private_for
 1833  def edit_query(db, userid, itemid):
 1834      return userid == db.query.get(itemid, 'creator')
 1835  p = db.security.addPermission(name='View', klass='query', check=view_query,
 1836      description="User is allowed to view their own and public queries")
 1837  db.security.addPermissionToRole('User', p)
 1838  p = db.security.addPermission(name='Edit', klass='query', check=edit_query,
 1839      description="User is allowed to edit their queries")
 1840  db.security.addPermissionToRole('User', p)
 1841  p = db.security.addPermission(name='Create', klass='query',
 1842      description="User is allowed to create queries")
 1843  db.security.addPermissionToRole('User', p)
 1844 
 1845 and then remove 'query' from the line::
 1846 
 1847  # Assign the access and edit Permissions for issue, file and message
 1848  # to regular users now
 1849  for cl in 'issue', 'file', 'msg', 'query', 'keyword':
 1850 
 1851 so it looks like::
 1852 
 1853  for cl in 'issue', 'file', 'msg', 'keyword':
 1854 
 1855 
 1856 .. index:: upgrading; 0.8.0 to 0.8.3
 1857 
 1858 Migrating from 0.8.0 to 0.8.3
 1859 =============================
 1860 
 1861 0.8.3 Nosy Handling Changes
 1862 ---------------------------
 1863 
 1864 A change was made to fix a bug in the ``nosyreaction.py`` standard
 1865 detector. To incorporate this fix in your trackers, you will need to copy
 1866 the ``nosyreaction.py`` file from the ``templates/classic/detectors``
 1867 directory of the source to your tracker's ``templates`` directory.
 1868 
 1869 If you have modified the ``nosyreaction.py`` file from the standard
 1870 version, you will need to roll your changes into the new file.
 1871 
 1872 
 1873 .. index:: upgrading; 0.7.1 to 0.8.0
 1874 
 1875 Migrating from 0.7.1 to 0.8.0
 1876 =============================
 1877 
 1878 You *must* fully uninstall previous Roundup version before installing
 1879 Roundup 0.8.0.  If you don't do that, ``roundup-admin install``
 1880 command may fail to function properly.
 1881 
 1882 0.8.0 Backend changes
 1883 ---------------------
 1884 
 1885 Backends 'bsddb' and 'bsddb3' are removed.  If you are using one of these,
 1886 you *must* migrate to another backend before upgrading.
 1887 
 1888 
 1889 0.8.0 API changes
 1890 -----------------
 1891 
 1892 Class.safeget() was removed from the API. Test your item ids before calling
 1893 Class.get() instead.
 1894 
 1895 
 1896 0.8.0 New tracker layout
 1897 ------------------------
 1898 
 1899 The ``config.py`` file has been replaced by ``config.ini``. You may use the
 1900 roundup-admin command "genconfig" to generate a new config file::
 1901 
 1902   roundup-admin genconfig <tracker home>/config.ini
 1903 
 1904 and modify the values therein based on the contents of your old config.py.
 1905 In most cases, the names of the config variables are the same.
 1906 
 1907 The ``select_db.py`` file has been replaced by a file in the ``db``
 1908 directory called ``backend_name``. As you might guess, this file contains
 1909 just the name of the backend. To figure what the contents of yours should
 1910 be, use the following table:
 1911 
 1912   ================================ =========================
 1913   ``select_db.py`` contents        ``backend_name`` contents
 1914   ================================ =========================
 1915   from back_anydbm import ...      anydbm
 1916   from back_metakit import ...     metakit
 1917   from back_sqlite import ...      sqlite
 1918   from back_mysql import ...       mysql
 1919   from back_postgresql import ...  postgresql
 1920   ================================ =========================
 1921 
 1922 The ``dbinit.py`` file has been split into two new files,
 1923 ``initial_data.py`` and ``schema.py``. The contents of this file are:
 1924 
 1925 ``initial_data.py``
 1926   You don't need one of these as your tracker is already initialised.
 1927 
 1928 ``schema.py``
 1929   Copy the body of the ``def open(name=None)`` function from your old
 1930   tracker's ``dbinit.py`` file to this file. As the lines you're copying
 1931   aren't part of a function definition anymore, one level of indentation
 1932   needs to be removed (remove only the leading four spaces on each
 1933   line).
 1934 
 1935   The first few lines -- those starting with ``from roundup.hyperdb
 1936   import ...`` and the ``db = Database(config, name)`` line -- don't
 1937   need to be copied. Neither do the last few lines -- those starting
 1938   with ``import detectors``, down to ``return db`` inclusive.
 1939 
 1940 You may remove the ``__init__.py`` module from the "detectors" directory as
 1941 it is no longer used.
 1942 
 1943 There's a new way to write extension code for Roundup. If you have code in
 1944 an ``interfaces.py`` file you should move it. See the `customisation
 1945 documentation`_ for information about how extensions are now written.
 1946 Note that some older trackers may use ``interfaces.py`` to customise the
 1947 mail gateway behaviour. You will need to keep your ``interfaces.py`` file
 1948 if this is the case.
 1949 
 1950 
 1951 0.8.0 Permissions Changes
 1952 -------------------------
 1953 
 1954 The creation of a new item in the user interfaces is now controlled by the
 1955 "Create" Permission. You will need to add an assignment of this Permission
 1956 to your users who are allowed to create items. The most common form of this
 1957 is the following in your ``schema.py`` added just under the current
 1958 assignation of the Edit Permission::
 1959 
 1960     for cl in 'issue', 'file', 'msg', 'query', 'keyword':
 1961         p = db.security.getPermission('Create', cl)
 1962         db.security.addPermissionToRole('User', p)
 1963 
 1964 You will need to explicitly let anonymous users access the web interface so
 1965 that regular users are able to see the login form. Note that almost all
 1966 trackers will need this Permission. The only situation where it's not
 1967 required is in a tracker that uses an HTTP Basic Authenticated front-end.
 1968 It's enabled by adding to your ``schema.py``::
 1969 
 1970     p = db.security.getPermission('Web Access')
 1971     db.security.addPermissionToRole('Anonymous', p)
 1972 
 1973 Finally, you will need to enable permission for your users to edit their
 1974 own details by adding the following to ``schema.py``::
 1975 
 1976     # Users should be able to edit their own details. Note that this
 1977     # permission is limited to only the situation where the Viewed or
 1978     # Edited item is their own.
 1979     def own_record(db, userid, itemid):
 1980         '''Determine whether the userid matches the item being accessed.'''
 1981         return userid == itemid
 1982     p = db.security.addPermission(name='View', klass='user', check=own_record,
 1983         description="User is allowed to view their own user details")
 1984     p = db.security.addPermission(name='Edit', klass='user', check=own_record,
 1985         description="User is allowed to edit their own user details")
 1986     db.security.addPermissionToRole('User', p)
 1987 
 1988 
 1989 0.8.0 Use of TemplatingUtils
 1990 ----------------------------
 1991 
 1992 If you used custom python functions in TemplatingUtils, they must
 1993 be moved from interfaces.py to a new file in the ``extensions`` directory.
 1994 
 1995 Each Function that should be available through TAL needs to be defined
 1996 as a toplevel function in the newly created file. Furthermore you
 1997 add an inititialization function, that registers the functions with the
 1998 tracker.
 1999 
 2000 If you find this too tedious, donfu wrote an automatic init function that
 2001 takes an existing TemplatingUtils class, and registers all class methods
 2002 that do not start with an underscore. The following hack should be placed
 2003 in the ``extensions`` directory alongside other extensions::
 2004 
 2005     class TemplatingUtils:
 2006          # copy from interfaces.py
 2007 
 2008     def init(tracker):
 2009          util = TemplatingUtils()
 2010 
 2011          def setClient(tu):
 2012              util.client = tu.client
 2013              return util
 2014 
 2015          def execUtil(name):
 2016              return lambda tu, *args, **kwargs: \
 2017                      getattr(setClient(tu), name)(*args, **kwargs)
 2018 
 2019          for name in dir(util):
 2020              if callable(getattr(util, name)) and not name.startswith('_'):
 2021                   tracker.registerUtil(name, execUtil(name))
 2022 
 2023 
 2024 0.8.0 Logging Configuration
 2025 ---------------------------
 2026 
 2027 See the `administration guide`_ for information about configuring the new
 2028 logging implemented in 0.8.0.
 2029 
 2030 
 2031 .. index:: upgrading; 0.7.2 to 0.7.3
 2032 
 2033 Migrating from 0.7.2 to 0.7.3
 2034 =============================
 2035 
 2036 0.7.3 Configuration
 2037 -------------------
 2038 
 2039 If you choose, you may specify the directory from which static files are
 2040 served (those which use the URL component ``@@file``). Currently the
 2041 directory defaults to the ``TEMPLATES`` configuration variable. You may
 2042 define a new variable, ``STATIC_FILES`` which overrides this value for
 2043 static files.
 2044 
 2045 
 2046 .. index:: upgrading; 0.7.0 to 0.7.2
 2047 
 2048 Migrating from 0.7.0 to 0.7.2
 2049 =============================
 2050 
 2051 0.7.2 DEFAULT_TIMEZONE is now required
 2052 --------------------------------------
 2053 
 2054 The DEFAULT_TIMEZONE configuration variable is now required. Add the
 2055 following to your tracker's ``config.py`` file::
 2056 
 2057     # You may specify a different default timezone, for use when users do not
 2058     # choose their own in their settings.
 2059     DEFAULT_TIMEZONE = 0            # specify as numeric hour offest
 2060 
 2061 .. index:: upgrading; 0.7.0 to 0.7.1
 2062 
 2063 Migrating from 0.7.0 to 0.7.1
 2064 =============================
 2065 
 2066 0.7.1 Permission assignments
 2067 ----------------------------
 2068 
 2069 If you allow anonymous access to your tracker, you might need to assign
 2070 some additional View (or Edit if your tracker is that open) permissions
 2071 to the "anonymous" user. To do so, find the code in your ``dbinit.py`` that
 2072 says::
 2073 
 2074     for cl in 'issue', 'file', 'msg', 'query', 'keyword':
 2075         p = db.security.getPermission('View', cl)
 2076         db.security.addPermissionToRole('User', p)
 2077         p = db.security.getPermission('Edit', cl)
 2078         db.security.addPermissionToRole('User', p)
 2079     for cl in 'priority', 'status':
 2080         p = db.security.getPermission('View', cl)
 2081         db.security.addPermissionToRole('User', p)
 2082 
 2083 Add add a line::
 2084 
 2085         db.security.addPermissionToRole('Anonymous', p)
 2086 
 2087 next to the existing ``'User'`` lines for the Permissions you wish to
 2088 assign to the anonymous user.
 2089 
 2090 
 2091 .. index:: upgrading; versions earlier than 0.7
 2092 
 2093 Migrating from 0.6 to 0.7
 2094 =========================
 2095 
 2096 0.7.0 Permission assignments
 2097 ----------------------------
 2098 
 2099 Due to a change in the rendering of web widgets, permissions are now
 2100 checked on Classes where they previously weren't (this is a good thing).
 2101 
 2102 You will need to add some additional Permission assignments for your
 2103 regular users, or some displays will break. After the following in your
 2104 tracker's ``dbinit.py``::
 2105 
 2106     # Assign the access and edit Permissions for issue, file and message
 2107     # to regular users now
 2108     for cl in 'issue', 'file', 'msg', 'query', 'keyword':
 2109         p = db.security.getPermission('View', cl)
 2110         db.security.addPermissionToRole('User', p)
 2111         p = db.security.getPermission('Edit', cl)
 2112         db.security.addPermissionToRole('User', p)
 2113 
 2114 add::
 2115 
 2116     for cl in 'priority', 'status':
 2117         p = db.security.getPermission('View', cl)
 2118         db.security.addPermissionToRole('User', p)
 2119 
 2120 
 2121 0.7.0 Getting the current user id
 2122 ---------------------------------
 2123 
 2124 The Database.curuserid attribute has been removed.
 2125 
 2126 Any code referencing this attribute should be replaced with a
 2127 call to Database.getuid().
 2128 
 2129 
 2130 0.7.0 ZRoundup changes
 2131 ----------------------
 2132 
 2133 The templates in your tracker's html directory will need updating if you
 2134 wish to use ZRoundup. If you've not modified those files (or some of them),
 2135 you may just copy the new versions from the Roundup source in the
 2136 templates/classic/html directory.
 2137 
 2138 If you have modified the html files, then you'll need to manually edit them
 2139 to change all occurances of special form variables from using the colon ":"
 2140 special character to the at "@" special character. That is, variables such
 2141 as::
 2142 
 2143   :action :required :template :remove:messages ...
 2144 
 2145 should become::
 2146 
 2147   @action @required @template @remove@messages ...
 2148 
 2149 Note that ``tal:`` statements are unaffected. So are TAL expression type
 2150 prefixes such as ``python:`` and ``string:``. Please ask on the
 2151 roundup-users mailing list for help if you're unsure.
 2152 
 2153 
 2154 0.7.0 Edit collision detection
 2155 ------------------------------
 2156 
 2157 Roundup now detects collisions with editing in the web interface (that is,
 2158 two people editing the same item at the same time).
 2159 
 2160 You must copy the ``_generic.collision.html`` file from Roundup source in
 2161 the ``templates/classic/html`` directory. to your tracker's ``html``
 2162 directory.
 2163 
 2164 
 2165 Migrating from 0.6.x to 0.6.3
 2166 =============================
 2167 
 2168 0.6.3 Configuration
 2169 -------------------
 2170 
 2171 You will need to copy the file::
 2172 
 2173   templates/classic/detectors/__init__.py
 2174 
 2175 to your tracker's ``detectors`` directory, replacing the one already there.
 2176 This fixes a couple of bugs in that file.
 2177 
 2178 
 2179 
 2180 Migrating from 0.5 to 0.6
 2181 =========================
 2182 
 2183 
 2184 0.6.0 Configuration
 2185 -------------------
 2186 
 2187 Introduced EMAIL_FROM_TAG config variable. This value is inserted into
 2188 the From: line of nosy email. If the sending user is "Foo Bar", the
 2189 From: line is usually::
 2190 
 2191      "Foo Bar" <issue_tracker@tracker.example>
 2192 
 2193 the EMAIL_FROM_TAG goes inside the "Foo Bar" quotes like so::
 2194 
 2195      "Foo Bar EMAIL_FROM_TAG" <issue_tracker@tracker.example>
 2196 
 2197 I've altered the mechanism in the detectors __init__.py module so that it
 2198 doesn't cross-import detectors from other trackers (if you run more than one
 2199 in a single roundup-server). This change means that you'll need to copy the
 2200 __init__.py from roundup/templates/classic/detectors/__init__.py to your
 2201 <tracker home>/detectors/__init__.py. Don't worry, the "classic" __init__ is a
 2202 one-size-fits-all, so it'll work even if you've added/removed detectors.
 2203 
 2204 0.6.0 Templating changes
 2205 ------------------------
 2206 
 2207 The ``user.item`` template (in the tracker home "templates" directory)
 2208 needs to have the following hidden variable added to its form (between the
 2209 ``<form...>`` and ``</form>`` tags::
 2210 
 2211   <input type="hidden" name=":template" value="item">
 2212 
 2213 
 2214 0.6.0 Form handling changes
 2215 ---------------------------
 2216 
 2217 Roundup's form handling capabilities have been significantly expanded. This
 2218 should not affect users of 0.5 installations - but if you find you're
 2219 getting errors from form submissions, please ask for help on the Roundup
 2220 users mailing list:
 2221 
 2222   https://sourceforge.net/projects/roundup/lists/roundup-users
 2223 
 2224 See the customisation doc section on `Form Values`__ for documentation of the
 2225 new form variables possible.
 2226 
 2227 __ customizing.html#form-values
 2228 
 2229 
 2230 0.6.0 Multilingual character set support
 2231 ----------------------------------------
 2232 
 2233 Added internationalization support. This is done via encoding all data
 2234 stored in roundup database to utf-8 (unicode encoding). To support utf-8 in
 2235 web interface you should add the folowing line to your tracker's html/page
 2236 and html/_generic.help files inside <head> tag::
 2237 
 2238     <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
 2239 
 2240 Since latin characters in utf-8 have the same codes as in ASCII table, this
 2241 modification is optional for users who use only plain latin characters.
 2242 
 2243 After this modification, you will be able to see and enter any world
 2244 character via web interface. Data received via mail interface also converted
 2245 to utf-8, however only new messages will be converted. If your roundup
 2246 database contains some of non-ASCII characters in one of 8-bit encoding,
 2247 they will not be visible in new unicode environment. Some of such data (e.g.
 2248 user names, keywords, etc)  can be edited by administrator, the others
 2249 (e.g. messages' contents) is not editable via web interface. Currently there
 2250 is no tool for converting such data, the only solution is to close
 2251 appropriate old issues and create new ones with the same content.
 2252 
 2253 
 2254 0.6.0 User timezone support
 2255 ---------------------------
 2256 
 2257 From version 0.6.0 roundup supports displaying of Date data in user' local
 2258 timezone if he/she has provided timezone information. To make it possible
 2259 some modification to tracker's schema and HTML templates are required.
 2260 First you must add string property 'timezone' to user class in dbinit.py
 2261 like this::
 2262 
 2263     user = Class(db, "user",
 2264                     username=String(),   password=Password(),
 2265                     address=String(),    realname=String(),
 2266                     phone=String(),      organisation=String(),
 2267                     alternate_addresses=String(),
 2268                     queries=Multilink('query'), roles=String(),
 2269                     timezone=String())
 2270 
 2271 And second - html interface. Add following lines to
 2272 $TRACKER_HOME/html/user.item template::
 2273 
 2274      <tr>
 2275       <th>Timezone</th>
 2276       <td tal:content="structure context/timezone/field">timezone</td>
 2277      </tr>
 2278 
 2279 After that all users should be able to provide their timezone information.
 2280 Timezone should be a positive or negative integer - offset from GMT.
 2281 
 2282 After providing timezone, roundup will show all dates values, found in web
 2283 and mail interfaces in local time. It will also accept any Date info in
 2284 local time, convert and store it in GMT.
 2285 
 2286 
 2287 0.6.0 Search page structure
 2288 ---------------------------
 2289 
 2290 In order to accomodate query editing the search page has been restructured. If
 2291 you want to provide your users with query editing, you should update your
 2292 search page using the macros detailed in the customisation doc section
 2293 `Searching on categories`__.
 2294 
 2295 __ customizing.html#searching-on-categories
 2296 
 2297 Also, the url field in the query class no longer starts with a '?'. You'll need
 2298 to remove this question mark from the url field to support queries. There's
 2299 a script in the "tools" directory called ``migrate-queries.py`` that should
 2300 automatically change any existing queries for you. As always, make a backup
 2301 of your database before running such a script.
 2302 
 2303 
 2304 0.6.0 Notes for metakit backend users
 2305 -------------------------------------
 2306 
 2307 Roundup 0.6.0 introduced searching on ranges of dates and intervals. To
 2308 support it, some modifications to interval storing routine were made. So if
 2309 your tracker uses metakit backend and your db schema contains intervals
 2310 property, searches on that property will not be accurate for db items that
 2311 was stored before roundup' upgrade. However all new records should be
 2312 searchable on intervals.
 2313 
 2314 It is possible to convert your database to new format: you can export and
 2315 import back all your data (consult "Migrating backends" in "Maintenance"
 2316 documentation). After this operation all your interval properties should
 2317 become searchable.
 2318 
 2319 Users of backends others than metakit should not worry about this issue.
 2320 
 2321 
 2322 Migrating from 0.4.x to 0.5.0
 2323 =============================
 2324 
 2325 This has been a fairly major revision of Roundup:
 2326 
 2327 1. Brand new, much more powerful, flexible, tasty and nutritious templating.
 2328    Unfortunately, this means all your current templates are useless. Hopefully
 2329    the new documentation and examples will be enough to help you make the
 2330    transition. Please don't hesitate to ask on roundup-users for help (or
 2331    complete conversions if you're completely stuck)!
 2332 2. The database backed got a lot more flexible, allowing Metakit and SQL
 2333    databases! The only decent SQL database implemented at present is sqlite,
 2334    but others shouldn't be a whole lot more work.
 2335 3. A brand new, highly flexible and much more robust security system including
 2336    a system of Permissions, Roles and Role assignments to users. You may now
 2337    define your own Permissions that may be checked in CGI transactions.
 2338 4. Journalling has been made less storage-hungry, so has been turned on
 2339    by default *except* for author, recipient and nosy link/unlink events. You
 2340    are advised to turn it off in your trackers too.
 2341 5. We've changed the terminology from "instance" to "tracker", to ease the
 2342    learning curve/impact for new users.
 2343 6. Because of the above changes, the tracker configuration has seen some
 2344    major changes. See below for the details.
 2345 
 2346 Please, **back up your database** before you start the migration process. This
 2347 is as simple as copying the "db" directory and all its contents from your
 2348 tracker to somewhere safe.
 2349 
 2350 
 2351 0.5.0 Configuration
 2352 -------------------
 2353 
 2354 First up, rename your ``instance_config.py`` file to just ``config.py``.
 2355 
 2356 Then edit your tracker's ``__init__.py`` module. It'll currently look
 2357 like this::
 2358 
 2359  from instance_config import *
 2360  try:
 2361      from dbinit import *
 2362  except ImportError:
 2363      pass # in installdir (probably :)
 2364  from interfaces import *
 2365 
 2366 and it needs to be::
 2367 
 2368  import config
 2369  from dbinit import open, init
 2370  from interfaces import Client, MailGW
 2371 
 2372 Due to the new templating having a top-level ``page`` that defines links for
 2373 searching, indexes, adding items etc, the following variables are no longer
 2374 used:
 2375 
 2376 - HEADER_INDEX_LINKS
 2377 - HEADER_ADD_LINKS
 2378 - HEADER_SEARCH_LINKS
 2379 - SEARCH_FILTERS
 2380 - DEFAULT_INDEX
 2381 - UNASSIGNED_INDEX
 2382 - USER_INDEX
 2383 - ISSUE_FILTER
 2384 
 2385 The new security implementation will require additions to the dbinit module,
 2386 but also removes the need for the following tracker config variables:
 2387 
 2388 - ANONYMOUS_ACCESS
 2389 - ANONYMOUS_REGISTER
 2390 
 2391 but requires two new variables which define the Roles assigned to users who
 2392 register through the web and e-mail interfaces:
 2393 
 2394 - NEW_WEB_USER_ROLES
 2395 - NEW_EMAIL_USER_ROLES
 2396 
 2397 in both cases, 'User' is a good initial setting. To emulate
 2398 ``ANONYMOUS_ACCESS='deny'``, remove all "View" Permissions from the
 2399 "Anonymous" Role. To emulate ``ANONYMOUS_REGISTER='deny'``, remove the "Web
 2400 Registration" and/or the "Email Registration" Permission from the "Anonymous"
 2401 Role. See the section on customising security in the `customisation
 2402 documentation`_ for more information.
 2403 
 2404 Finally, the following config variables have been renamed to make more sense:
 2405 
 2406 - INSTANCE_HOME -> TRACKER_HOME
 2407 - INSTANCE_NAME -> TRACKER_NAME
 2408 - ISSUE_TRACKER_WEB -> TRACKER_WEB
 2409 - ISSUE_TRACKER_EMAIL -> TRACKER_EMAIL
 2410 
 2411 
 2412 0.5.0 Schema Specification
 2413 --------------------------
 2414 
 2415 0.5.0 Database backend changes
 2416 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 2417 
 2418 Your select_db module in your tracker has changed a fair bit. Where it used
 2419 to contain::
 2420 
 2421  # WARNING: DO NOT EDIT THIS FILE!!!
 2422  from roundup.backends.back_anydbm import Database
 2423 
 2424 it must now contain::
 2425 
 2426  # WARNING: DO NOT EDIT THIS FILE!!!
 2427  from roundup.backends.back_anydbm import Database, Class, FileClass, IssueClass
 2428 
 2429 Yes, I realise the irony of the "DO NOT EDIT THIS FILE" statement :)
 2430 Note the addition of the Class, FileClass, IssueClass imports. These are very
 2431 important, as they're going to make the next change work too. You now need to
 2432 modify the top of the dbinit module in your tracker from::
 2433 
 2434  import instance_config
 2435  from roundup import roundupdb
 2436  from select_db import Database
 2437 
 2438  from roundup.roundupdb import Class, FileClass
 2439 
 2440  class Database(roundupdb.Database, select_db.Database):
 2441      ''' Creates a hybrid database from:
 2442           . the selected database back-end from select_db
 2443           . the roundup extensions from roundupdb
 2444      '''
 2445      pass
 2446 
 2447  class IssueClass(roundupdb.IssueClass):
 2448      ''' issues need the email information
 2449      '''
 2450      pass
 2451 
 2452 to::
 2453 
 2454  import config
 2455  from select_db import Database, Class, FileClass, IssueClass
 2456 
 2457 Yes, remove the Database and IssueClass definitions and those other imports.
 2458 They're not needed any more!
 2459 
 2460 Look for places in dbinit.py where ``instance_config`` is used too, and
 2461 rename them ``config``.
 2462 
 2463 
 2464 0.5.0 Journalling changes
 2465 ~~~~~~~~~~~~~~~~~~~~~~~~~
 2466 
 2467 Journalling has been optimised for storage. Journalling of links has been
 2468 turned back on by default. If your tracker has a large user base, you may wish
 2469 to turn off journalling of nosy list, message author and message recipient
 2470 link and unlink events. You do this by adding ``do_journal='no'`` to the Class
 2471 initialisation in your dbinit. For example, your *msg* class initialisation
 2472 probably looks like this::
 2473 
 2474     msg = FileClass(db, "msg",
 2475                     author=Link("user"), recipients=Multilink("user"),
 2476                     date=Date(),         summary=String(),
 2477                     files=Multilink("file"),
 2478                     messageid=String(),  inreplyto=String())
 2479 
 2480 to turn off journalling of author and recipient link events, add
 2481 ``do_journal='no'`` to the ``author=Link("user")`` part of the statement,
 2482 like so::
 2483 
 2484     msg = FileClass(db, "msg",
 2485                     author=Link("user", do_journal='no'),
 2486                     recipients=Multilink("user", do_journal='no'),
 2487                     date=Date(),         summary=String(),
 2488                     files=Multilink("file"),
 2489                     messageid=String(),  inreplyto=String())
 2490 
 2491 Nosy list link event journalling is actually turned off by default now. If you
 2492 want to turn it on, change to your issue class' nosy list, change its
 2493 definition from::
 2494 
 2495     issue = IssueClass(db, "issue",
 2496                     assignedto=Link("user"), topic=Multilink("keyword"),
 2497                     priority=Link("priority"), status=Link("status"))
 2498 
 2499 to::
 2500 
 2501     issue = IssueClass(db, "issue", nosy=Multilink("user", do_journal='yes'),
 2502                     assignedto=Link("user"), topic=Multilink("keyword"),
 2503                     priority=Link("priority"), status=Link("status"))
 2504 
 2505 noting that your definition of the nosy Multilink will override the normal one.
 2506 
 2507 
 2508 0.5.0 User schema changes
 2509 ~~~~~~~~~~~~~~~~~~~~~~~~~
 2510 
 2511 Users have two more properties, "queries" and "roles". You'll have something
 2512 like this in your dbinit module now::
 2513 
 2514     user = Class(db, "user",
 2515                     username=String(),   password=Password(),
 2516                     address=String(),    realname=String(),
 2517                     phone=String(),      organisation=String(),
 2518                     alternate_addresses=String())
 2519     user.setkey("username")
 2520 
 2521 and you'll need to add the new properties and the new "query" class to it
 2522 like so::
 2523 
 2524     query = Class(db, "query",
 2525                     klass=String(),     name=String(),
 2526                     url=String())
 2527     query.setkey("name")
 2528 
 2529     # Note: roles is a comma-separated string of Role names
 2530     user = Class(db, "user",
 2531                     username=String(),   password=Password(),
 2532                     address=String(),    realname=String(),
 2533                     phone=String(),      organisation=String(),
 2534                     alternate_addresses=String(),
 2535                     queries=Multilink('query'), roles=String())
 2536     user.setkey("username")
 2537 
 2538 The "queries" property is used to store off the user's favourite database
 2539 queries. The "roles" property is explained below in `0.5.0 Security
 2540 Settings`_.
 2541 
 2542 
 2543 0.5.0 Security Settings
 2544 ~~~~~~~~~~~~~~~~~~~~~~~
 2545 
 2546 See the `security documentation`_ for an explanation of how the new security
 2547 system works. In a nutshell though, the security is handled as a four step
 2548 process:
 2549 
 2550 1. Permissions are defined as having a name and optionally a hyperdb class
 2551    they're specific to,
 2552 2. Roles are defined that have one or more Permissions,
 2553 3. Users are assigned Roles in their "roles" property, and finally
 2554 4. Roundup checks that users have appropriate Permissions at appropriate times
 2555    (like editing issues).
 2556 
 2557 Your tracker dbinit module's *open* function now has to define any
 2558 Permissions that are specific to your tracker, and also the assignment
 2559 of Permissions to Roles. At the moment, your open function
 2560 ends with::
 2561 
 2562     import detectors
 2563     detectors.init(db)
 2564 
 2565     return db
 2566 
 2567 and what we need to do is insert some commands that will set up the security
 2568 parameters. Right above the ``import detectors`` line, you'll want to insert
 2569 these lines::
 2570 
 2571     #
 2572     # SECURITY SETTINGS
 2573     #
 2574     # new permissions for this schema
 2575     for cl in 'issue', 'file', 'msg', 'user':
 2576         db.security.addPermission(name="Edit", klass=cl,
 2577             description="User is allowed to edit "+cl)
 2578         db.security.addPermission(name="View", klass=cl,
 2579             description="User is allowed to access "+cl)
 2580 
 2581     # Assign the access and edit permissions for issue, file and message
 2582     # to regular users now
 2583     for cl in 'issue', 'file', 'msg':
 2584         p = db.security.getPermission('View', cl)
 2585         db.security.addPermissionToRole('User', p)
 2586         p = db.security.getPermission('Edit', cl)
 2587         db.security.addPermissionToRole('User', p)
 2588     # and give the regular users access to the web and email interface
 2589     p = db.security.getPermission('Web Access')
 2590     db.security.addPermissionToRole('User', p)
 2591     p = db.security.getPermission('Email Access')
 2592     db.security.addPermissionToRole('User', p)
 2593 
 2594     # May users view other user information? Comment these lines out
 2595     # if you don't want them to
 2596     p = db.security.getPermission('View', 'user')
 2597     db.security.addPermissionToRole('User', p)
 2598 
 2599     # Assign the appropriate permissions to the anonymous user's Anonymous
 2600     # Role. Choices here are:
 2601     # - Allow anonymous users to register through the web
 2602     p = db.security.getPermission('Web Registration')
 2603     db.security.addPermissionToRole('Anonymous', p)
 2604     # - Allow anonymous (new) users to register through the email gateway
 2605     p = db.security.getPermission('Email Registration')
 2606     db.security.addPermissionToRole('Anonymous', p)
 2607     # - Allow anonymous users access to the "issue" class of data
 2608     #   Note: this also grants access to related information like files,
 2609     #         messages, statuses etc that are linked to issues
 2610     #p = db.security.getPermission('View', 'issue')
 2611     #db.security.addPermissionToRole('Anonymous', p)
 2612     # - Allow anonymous users access to edit the "issue" class of data
 2613     #   Note: this also grants access to create related information like
 2614     #         files and messages etc that are linked to issues
 2615     #p = db.security.getPermission('Edit', 'issue')
 2616     #db.security.addPermissionToRole('Anonymous', p)
 2617 
 2618     # oh, g'wan, let anonymous access the web interface too
 2619     p = db.security.getPermission('Web Access')
 2620     db.security.addPermissionToRole('Anonymous', p)
 2621 
 2622 Note in the comments there the places where you might change the permissions
 2623 to restrict users or grant users more access. If you've created additional
 2624 classes that users should be able to edit and view, then you should add them
 2625 to the "new permissions for this schema" section at the start of the security
 2626 block. Then add them to the "Assign the access and edit permissions" section
 2627 too, so people actually have the new Permission you've created.
 2628 
 2629 One final change is needed that finishes off the security system's
 2630 initialisation. We need to add a call to ``db.post_init()`` at the end of the
 2631 dbinit open() function. Add it like this::
 2632 
 2633     import detectors
 2634     detectors.init(db)
 2635 
 2636     # schema is set up - run any post-initialisation
 2637     db.post_init()
 2638     return db
 2639 
 2640 You may verify the setup of Permissions and Roles using the new
 2641 "``roundup-admin security``" command.
 2642 
 2643 
 2644 0.5.0 User changes
 2645 ~~~~~~~~~~~~~~~~~~
 2646 
 2647 To support all those schema changes, you'll need to massage your user database
 2648 a little too, to:
 2649 
 2650 1. make sure there's an "anonymous" user - this user is mandatory now and is
 2651    the one that unknown users are logged in as.
 2652 2. make sure all users have at least one Role.
 2653 
 2654 If you don't have the "anonymous" user, create it now with the command::
 2655 
 2656   roundup-admin create user username=anonymous roles=Anonymous
 2657 
 2658 making sure the capitalisation is the same as above. Once you've done that,
 2659 you'll need to set the roles property on all users to a reasonable default.
 2660 The admin user should get "Admin", the anonymous user "Anonymous"
 2661 and all other users "User". The ``fixroles.py`` script in the tools directory
 2662 will do this. Run it like so (where python is your python 2+ binary)::
 2663 
 2664   python tools/fixroles.py -i <tracker home> fixroles
 2665 
 2666 
 2667 
 2668 0.5.0 CGI interface changes
 2669 ---------------------------
 2670 
 2671 The CGI interface code was completely reorganised and largely rewritten. The
 2672 end result is that this section of your tracker interfaces module will need
 2673 changing from::
 2674 
 2675  from roundup import cgi_client, mailgw
 2676  from roundup.i18n import _
 2677 
 2678  class Client(cgi_client.Client):
 2679      ''' derives basic CGI implementation from the standard module,
 2680          with any specific extensions
 2681      '''
 2682      pass
 2683 
 2684 to::
 2685 
 2686  from roundup import mailgw
 2687  from roundup.cgi import client
 2688 
 2689  class Client(client.Client):
 2690      ''' derives basic CGI implementation from the standard module,
 2691          with any specific extensions
 2692      '''
 2693      pass
 2694 
 2695 You will also need to install the new version of roundup.cgi from the source
 2696 cgi-bin directory if you're using it.
 2697 
 2698 
 2699 0.5.0 HTML templating
 2700 ---------------------
 2701 
 2702 You'll want to make a backup of your current tracker html directory. You
 2703 should then copy the html directory from the Roundup source "classic" template
 2704 and modify it according to your local schema changes.
 2705 
 2706 If you need help with the new templating system, please ask questions on the
 2707 roundup-users mailing list (available through the roundup web page on
 2708 sourceforge, http://www.roundup-tracker.org/.
 2709 
 2710 
 2711 0.5.0 Detectors
 2712 ---------------
 2713 
 2714 The nosy reactor has been updated to handle the tracker not having an
 2715 "assignedto" property on issues. You may want to copy it into your tracker's
 2716 detectors directory. Chances are you've already fixed it though :)
 2717 
 2718 
 2719 Migrating from 0.4.1 to 0.4.2
 2720 =============================
 2721 
 2722 0.4.2 Configuration
 2723 -------------------
 2724 The USER_INDEX definition introduced in 0.4.1 was too restrictive in its
 2725 allowing replacement of 'assignedto' with the user's userid. Users must change
 2726 the None value of 'assignedto' to 'CURRENT USER' (the string, in quotes) for
 2727 the replacement behaviour to occur now.
 2728 
 2729 The new configuration variables are:
 2730 
 2731 - EMAIL_KEEP_QUOTED_TEXT
 2732 - EMAIL_LEAVE_BODY_UNCHANGED
 2733 - ADD_RECIPIENTS_TO_NOSY
 2734 
 2735 See the sample configuration files in::
 2736 
 2737  <roundup source>/roundup/templates/classic/instance_config.py
 2738 
 2739 and::
 2740 
 2741  <roundup source>/roundup/templates/extended/instance_config.py
 2742 
 2743 and the `customisation documentation`_ for information on how they're used.
 2744 
 2745 
 2746 0.4.2 Changes to detectors
 2747 --------------------------
 2748 You will need to copy the detectors from the distribution into your instance
 2749 home "detectors" directory. If you used the classic schema, the detectors
 2750 are in::
 2751 
 2752  <roundup source>/roundup/templates/classic/detectors/
 2753 
 2754 If you used the extended schema, the detectors are in::
 2755 
 2756  <roundup source>/roundup/templates/extended/detectors/
 2757 
 2758 The change means that schema-specific code has been removed from the
 2759 mail gateway and cgi interface and made into auditors:
 2760 
 2761 - nosyreactor.py has now got an updatenosy auditor which updates the nosy
 2762   list with author, recipient and assignedto information.
 2763 - statusauditor.py makes the unread or resolved -> chatting changes and
 2764   presets the status of an issue to unread.
 2765 
 2766 There's also a bug or two fixed in the nosyreactor code.
 2767 
 2768 0.4.2 HTML templating changes
 2769 -----------------------------
 2770 The link() htmltemplate function now has a "showid" option for links and
 2771 multilinks. When true, it only displays the linked item id as the anchor
 2772 text. The link value is displayed as a tooltip using the title anchor
 2773 attribute. To use in eg. the superseder field, have something like this::
 2774 
 2775    <td>
 2776     <display call="field('superseder', showid=1)">
 2777     <display call="classhelp('issue', 'id,title', label='list', width=500)">
 2778     <property name="superseder">
 2779      <br>View: <display call="link('superseder', showid=1)">
 2780     </property>
 2781    </td>
 2782 
 2783 The stylesheets have been cleaned up too. You may want to use the newer
 2784 versions in::
 2785 
 2786  <roundup source>/roundup/templates/<template>/html/default.css
 2787 
 2788 
 2789 
 2790 Migrating from 0.4.0 to 0.4.1
 2791 =============================
 2792 
 2793 0.4.1 Files storage
 2794 -------------------
 2795 
 2796 Messages and files from newly created issues will be put into subdierectories
 2797 in thousands e.g. msg123 will be put into files/msg/0/msg123, file2003
 2798 will go into files/file/2/file2003. Previous messages are still found, but
 2799 could be put into this structure.
 2800 
 2801 0.4.1 Configuration
 2802 -------------------
 2803 
 2804 To allow more fine-grained access control, the variable used to check
 2805 permission to auto-register users in the mail gateway is now called
 2806 ANONYMOUS_REGISTER_MAIL rather than overloading ANONYMOUS_REGISTER. If the
 2807 variable doesn't exist, then ANONYMOUS_REGISTER is tested as before.
 2808 
 2809 Configuring the links in the web header is now easier too. The following
 2810 variables have been added to the classic instance_config.py::
 2811 
 2812   HEADER_INDEX_LINKS   - defines the "index" links to be made available
 2813   HEADER_ADD_LINKS     - defines the "add" links
 2814   DEFAULT_INDEX        - specifies the index view for DEFAULT
 2815   UNASSIGNED_INDEX     - specifies the index view for UNASSIGNED
 2816   USER_INDEX           - specifies the index view for USER
 2817 
 2818 See the <roundup source>/roundup/templates/classic/instance_config.py for more
 2819 information - including how the variables are to be set up. Most users will
 2820 just be able to copy the variables from the source to their instance home. If
 2821 you've modified the header by changing the source of the interfaces.py file in
 2822 the instance home, you'll need to remove that customisation and move it into
 2823 the appropriate variables in instance_config.py.
 2824 
 2825 The extended schema has similar variables added too - see the source for more
 2826 info.
 2827 
 2828 0.4.1 Alternate E-Mail Addresses
 2829 --------------------------------
 2830 
 2831 If you add the property "alternate_addresses" to your user class, your users
 2832 will be able to register alternate email addresses that they may use to
 2833 communicate with roundup as. All email from roundup will continue to be sent
 2834 to their primary address.
 2835 
 2836 If you have not edited the dbinit.py file in your instance home directory,
 2837 you may simply copy the new dbinit.py file from the core code. If you used
 2838 the classic schema, the interfaces file is in::
 2839 
 2840  <roundup source>/roundup/templates/classic/dbinit.py
 2841 
 2842 If you used the extended schema, the file is in::
 2843 
 2844  <roundup source>/roundup/templates/extended/dbinit.py
 2845 
 2846 If you have modified your dbinit.py file, you need to edit the dbinit.py
 2847 file in your instance home directory. Find the lines which define the user
 2848 class::
 2849 
 2850     user = Class(db, "msg",
 2851                     username=String(),   password=Password(),
 2852                     address=String(),    realname=String(),
 2853                     phone=String(),      organisation=String(),
 2854                     alternate_addresses=String())
 2855 
 2856 You will also want to add the property to the user's details page. The
 2857 template for this is the "user.item" file in your instance home "html"
 2858 directory. Similar to above, you may copy the file from the roundup source if
 2859 you haven't modified it. Otherwise, add the following to the template::
 2860 
 2861    <display call="multiline('alternate_addresses')">
 2862 
 2863 with appropriate labelling etc. See the standard template for an idea.
 2864 
 2865 
 2866 
 2867 Migrating from 0.3.x to 0.4.0
 2868 =============================
 2869 
 2870 0.4.0 Message-ID and In-Reply-To addition
 2871 -----------------------------------------
 2872 0.4.0 adds the tracking of messages by message-id and allows threading
 2873 using in-reply-to. Most e-mail clients support threading using this
 2874 feature, and we hope to add support for it to the web gateway. If you
 2875 have not edited the dbinit.py file in your instance home directory, you may
 2876 simply copy the new dbinit.py file from the core code. If you used the
 2877 classic schema, the interfaces file is in::
 2878 
 2879  <roundup source>/roundup/templates/classic/dbinit.py
 2880 
 2881 If you used the extended schema, the file is in::
 2882 
 2883  <roundup source>/roundup/templates/extended/dbinit.py
 2884 
 2885 If you have modified your dbinit.py file, you need to edit the dbinit.py
 2886 file in your instance home directory. Find the lines which define the msg
 2887 class::
 2888 
 2889     msg = FileClass(db, "msg",
 2890                     author=Link("user"), recipients=Multilink("user"),
 2891                     date=Date(),         summary=String(),
 2892                     files=Multilink("file"))
 2893 
 2894 and add the messageid and inreplyto properties like so::
 2895 
 2896     msg = FileClass(db, "msg",
 2897                     author=Link("user"), recipients=Multilink("user"),
 2898                     date=Date(),         summary=String(),
 2899                     files=Multilink("file"),
 2900                     messageid=String(),  inreplyto=String())
 2901 
 2902 Also, configuration is being cleaned up. This means that your dbinit.py will
 2903 also need to be changed in the open function. If you haven't changed your
 2904 dbinit.py, the above copy will be enough. If you have, you'll need to change
 2905 the line (round line 50)::
 2906 
 2907     db = Database(instance_config.DATABASE, name)
 2908 
 2909 to::
 2910 
 2911     db = Database(instance_config, name)
 2912 
 2913 
 2914 0.4.0 Configuration
 2915 --------------------
 2916 ``TRACKER_NAME`` and ``EMAIL_SIGNATURE_POSITION`` have been added to the
 2917 instance_config.py. The simplest solution is to copy the default values
 2918 from template in the core source.
 2919 
 2920 The mail gateway now checks ``ANONYMOUS_REGISTER`` to see if unknown users
 2921 are to be automatically registered with the tracker. If it is set to "deny"
 2922 then unknown users will not have access. If it is set to "allow" they will be
 2923 automatically registered with the tracker.
 2924 
 2925 
 2926 0.4.0 CGI script roundup.cgi
 2927 ----------------------------
 2928 The CGI script has been updated with some features and a bugfix, so you should
 2929 copy it from the roundup cgi-bin source directory again. Make sure you update
 2930 the ROUNDUP_INSTANCE_HOMES after the copy.
 2931 
 2932 
 2933 0.4.0 Nosy reactor
 2934 ------------------
 2935 The nosy reactor has also changed - copy the nosyreactor.py file from the core
 2936 source::
 2937 
 2938    <roundup source>/roundup/templates/<template>/detectors/nosyreactor.py
 2939 
 2940 to your instance home "detectors" directory.
 2941 
 2942 
 2943 0.4.0 HTML templating
 2944 ---------------------
 2945 The field() function was incorrectly implemented - links and multilinks now
 2946 display as text fields when rendered using field(). To display a menu (drop-
 2947 down or select box) you need to use the menu() function.
 2948 
 2949 
 2950 
 2951 Migrating from 0.2.x to 0.3.x
 2952 =============================
 2953 
 2954 0.3.x Cookie Authentication changes
 2955 -----------------------------------
 2956 0.3.0 introduces cookie authentication - you will need to copy the
 2957 interfaces.py file from the roundup source to your instance home to enable
 2958 authentication. If you used the classic schema, the interfaces file is in::
 2959 
 2960  <roundup source>/roundup/templates/classic/interfaces.py
 2961 
 2962 If you used the extended schema, the file is in::
 2963 
 2964  <roundup source>/roundup/templates/extended/interfaces.py
 2965 
 2966 If you have modified your interfaces.Client class, you will need to take
 2967 note of the login/logout functionality provided in roundup.cgi_client.Client
 2968 (classic schema) or roundup.cgi_client.ExtendedClient (extended schema) and
 2969 modify your instance code apropriately.
 2970 
 2971 
 2972 0.3.x Password encoding
 2973 -----------------------
 2974 This release also introduces encoding of passwords in the database. If you
 2975 have not edited the dbinit.py file in your instance home directory, you may
 2976 simply copy the new dbinit.py file from the core code. If you used the
 2977 classic schema, the interfaces file is in::
 2978 
 2979  <roundup source>/roundup/templates/classic/dbinit.py
 2980 
 2981 If you used the extended schema, the file is in::
 2982 
 2983  <roundup source>/roundup/templates/extended/dbinit.py
 2984 
 2985 
 2986 If you have modified your dbinit.py file, you may use encoded passwords:
 2987 
 2988 1. Edit the dbinit.py file in your instance home directory
 2989    a. At the first code line of the open() function::
 2990 
 2991        from roundup.hyperdb import String, Date, Link, Multilink
 2992 
 2993       alter to include Password, as so::
 2994 
 2995        from roundup.hyperdb import String, Password, Date, Link, Multilink
 2996 
 2997    b. Where the password property is defined (around line 66)::
 2998 
 2999        user = Class(db, "user",
 3000                        username=String(),   password=String(),
 3001                        address=String(),    realname=String(),
 3002                        phone=String(),      organisation=String())
 3003        user.setkey("username")
 3004 
 3005       alter the "password=String()" to "password=Password()"::
 3006 
 3007        user = Class(db, "user",
 3008                        username=String(),   password=Password(),
 3009                        address=String(),    realname=String(),
 3010                        phone=String(),      organisation=String())
 3011        user.setkey("username")
 3012 
 3013 2. Any existing passwords in the database will remain cleartext until they
 3014    are edited. It is recommended that at a minimum the admin password be
 3015    changed immediately::
 3016 
 3017       roundup-admin -i <instance home> set user1 password=<new password>
 3018 
 3019 
 3020 0.3.x Configuration
 3021 -------------------
 3022 FILTER_POSITION, ANONYMOUS_ACCESS, ANONYMOUS_REGISTER have been added to
 3023 the instance_config.py. Simplest solution is to copy the default values from
 3024 template in the core source.
 3025 
 3026 MESSAGES_TO_AUTHOR has been added to the IssueClass in dbinit.py. Set to 'yes'
 3027 to send nosy messages to the author. Default behaviour is to not send nosy
 3028 messages to the author. You will need to add MESSAGES_TO_AUTHOR to your
 3029 dbinit.py in your instance home.
 3030 
 3031 
 3032 0.3.x CGI script roundup.cgi
 3033 ----------------------------
 3034 There have been some structural changes to the roundup.cgi script - you will
 3035 need to install it again from the cgi-bin directory of the source
 3036 distribution. Make sure you update the ROUNDUP_INSTANCE_HOMES after the
 3037 copy.
 3038 
 3039 
 3040 .. _`customisation documentation`: customizing.html
 3041 .. _`security documentation`: security.html
 3042 .. _`administration guide`: admin_guide.html
 3043 .. _`xmlrpc guide`: xmlrpc.html