"Fossies" - the Fresh Open Source Software Archive

Member "automake-1.16.3/t/README" (15 Nov 2020, 11783 Bytes) of package /linux/misc/automake-1.16.3.tar.xz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. See also the latest Fossies "Diffs" side-by-side code changes report for "README": 1.16.2_vs_1.16.3.

    1 			    The Automake test suite
    2 
    3 
    4 User interface
    5 ==============
    6 
    7 
    8 Running the tests
    9 -----------------
   10 
   11   To run all tests:
   12 
   13     make -k check
   14 
   15   By default, verbose output of a test 't/foo.sh' or 't/foo.tap' is retained
   16   in the log file 't/foo.log'.  Also, a summary log is created in the file
   17   'test-suite.log' (in the top-level directory).
   18 
   19   You can use '-jN' for faster completion (it even helps on a uniprocessor
   20   system, due to unavoidable sleep delays, as noted below):
   21 
   22     make -k -j4
   23 
   24   To rerun only failed tests:
   25 
   26     make -k recheck
   27 
   28   To run only tests that are newer than their last results:
   29 
   30     make -k check RECHECK_LOGS=
   31 
   32   To run only selected tests:
   33 
   34     make -k check TESTS="t/foo.sh t/bar.tap"           (GNU make)
   35     env TESTS="t/foo.sh t/bar.tap" make -e -k check    (non-GNU make)
   36 
   37  To run the tests in cross-compilation mode, you should first configure
   38  the automake source tree to a cross-compilation setup.  For example, to
   39  run with a Linux-to-MinGW cross compiler, you will need something like
   40  this:
   41 
   42    ./configure --host i586-mingw32msvc --build i686-pc-linux-gnu
   43 
   44  To avoid possible spurious error, you really have to *explicitly* specify
   45  '--build' in addition to '--host'; the 'lib/config.guess' script can help
   46  determine the correct value to pass to '--build'.
   47  Then you can just run the testsuite in the usual way, and the test cases
   48  using a compiler should automatically use a cross-compilation setup.
   49 
   50 
   51 Interpretation
   52 --------------
   53 
   54   Successes:
   55     PASS  - success
   56     XFAIL - expected failure
   57 
   58   Failures:
   59     FAIL  - failure
   60     XPASS - unexpected success
   61 
   62   Other:
   63     SKIP  - skipped tests (third party tools not available)
   64     ERROR - some unexpected error condition
   65 
   66 
   67 About the tests
   68 ---------------
   69 
   70   There are two kinds of tests in the Automake testsuite (both implemented
   71   as shell scripts).  The scripts with the '.sh' suffix are "simple"
   72   tests, their outcome completely determined by their exit status.  Those
   73   with the '.tap' suffix use the TAP protocol.
   74 
   75   If you want to run a test by hand, you should be able to do so using the
   76   'runtest' script provided in the Automake distribution:
   77 
   78       ./runtest t/nogzip.sh
   79       ./runtest t/add-missing.tap
   80 
   81   This will run the test using the correct shell, and should also work in
   82   VPATH builds.  Note that, to run the TAP tests this way, you'll need to
   83   have the prove(1) utility available in $PATH.
   84 
   85 
   86 Supported shells
   87 ----------------
   88 
   89   By default, the tests are run by a proper shell detected at configure
   90   time.  Here is how you can run the tests with a different shell, say
   91   '/bin/my-sh':
   92 
   93     # Running through the makefile test driver.
   94     make check AM_TEST_RUNNER_SHELL=/bin/my-sh         (GNU make)
   95     AM_TEST_RUNNER_SHELL=/bin/my-sh make -e check      (non-GNU make)
   96 
   97     # Run a test directly from the command line.
   98     AM_TEST_RUNNER_SHELL=/bin/my-sh ./runtest t/foo.sh
   99 
  100   The test scripts are written with portability in mind, and should run
  101   with any decent POSIX shell.  However, it is worth nothing that older
  102   versions of Zsh (pre-4.3) exhibited several bugs and incompatibilities
  103   with our uses, and are thus not supported for running Automake's test
  104   scripts.
  105 
  106 
  107 Reporting failures
  108 ------------------
  109 
  110   Send verbose output, i.e., the contents of test-suite.log, of failing
  111   tests to <bug-automake@gnu.org>, along with the usual version numbers
  112   (which Automake, which Autoconf, which operating system, which make
  113   version, which shell, etc.)
  114 
  115 
  116 
  117 Writing test cases
  118 ==================
  119 
  120 * If you plan to fix a bug, write the test case first.  This way you'll
  121   make sure the test catches the bug, and that it succeeds once you have
  122   fixed the bug.
  123 
  124 * Add a copyright/license paragraph.
  125 
  126 * Explain what the test does, i.e., which features it checks, which
  127   invariants it verifies, or what bugs/issues it guard against.
  128 
  129 * Cite the PR number (if any), and the original reporter (if any), so
  130   we can find or ask for information if needed.
  131 
  132 * If a test checks examples or idioms given in the documentation, make
  133   sure the documentation reference them appropriately in comments, as
  134   with:
  135 
  136     @c Keep in sync with autodist-config-headers.sh
  137     @example
  138     ...
  139     @end example
  140 
  141 * Use "required=..." for required tools.  Do not explicitly require
  142   tools which can be taken for granted because they're listed in the
  143   GNU Coding Standards (for example, 'gzip').
  144 
  145 * Include 'test-init.sh' in every test script (see existing tests for
  146   examples of how to do this).
  147 
  148 * Use the 'skip_' function to skip tests, with a meaningful message if
  149   possible.  Where convenient, use the 'warn_' function to print generic
  150   warnings, the 'fail_' function for test failures, and the 'fatal_'
  151   function for hard errors.  In case a hard error is due to a failed
  152   set-up of a test scenario, you can use the 'framework_fail_' function
  153   instead.
  154 
  155 * For those tests checking the Automake-provided test harnesses that
  156   are expected to work also when the 'serial-tests' Automake option
  157   is used (thus causing the serial testsuite harness to be used in the
  158   generated Makefile), place a line containing "try-with-serial-tests"
  159   somewhere in the file (usually in a comment).
  160   That will ensure that the 'gen-testsuite-part' script generates a
  161   sibling of that test which uses the serial harness instead of the
  162   parallel one.  For those tests that are *not* meant to work with the
  163   parallel testsuite harness at all (these should be very very few),
  164   set the shell variable 'am_serial_tests' to "yes" before including
  165   test-init.sh.
  166 
  167 * Some tests in the Automake testsuite are auto-generated; those tests
  168   might have custom extensions, but their basename (that is, with such
  169   extension stripped) is expected to end with "-w" string, optionally
  170   followed by decimal digits.  For example, the name of a valid
  171   auto-generated test can be 'color-w.sh' or 'tap-signal-w09.tap'.
  172   Please don't name hand-written tests in a way that could cause them
  173   to be confused with auto-generated tests; for example, 'u-v-w.sh'
  174   or 'option-w0.tap' are *not* valid name for hand-written tests.
  175 
  176 * test-init.sh brings in some commonly required files, and sets a skeleton
  177   configure.ac.  If possible, append to this file.  In some cases you'll
  178   have to overwrite it, but this should be the exception.  Note that
  179   configure.ac registers Makefile.in but do not output anything by
  180   default.  If you need ./configure to create Makefile, append AC_OUTPUT
  181   to configure.ac.  In case you don't want your test directory to be
  182   pre-populate by test-init.sh (this should be a rare occurrence), set
  183   the 'am_create_testdir' shell variable to "empty" before sourcing
  184   test-init.sh.
  185 
  186 * By default, the testcases are run with the errexit shell flag on,
  187   to make it easier to catch failures you might not have thought of.
  188   If  this is undesirable in some testcase, you can use "set +e" to
  189   disable the errexit flag (but please do so only if you have a very
  190   good reason).
  191 
  192 * End the test script with a ':' command.  Otherwise, when somebody
  193   changes the test by adding a failing command after the last command,
  194   the test will spuriously fail because '$?' is nonzero at the end.
  195   Note that this is relevant even if the errexit shell flag is on, in
  196   case the test contains commands like "grep ... Makefile.in && exit 1"
  197   (and there are indeed a lot of such tests).
  198 
  199 * Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER,
  200   $PERL, $MAKE, $EGREP, and $FGREP, instead of the corresponding
  201   commands.
  202 
  203 * When you want to redirect the output from a make invocation, use the
  204   'run_make' function rather than calling $MAKE directly.  Not only is
  205   this more idiomatic, but it also avoid possible spurious racy failures
  206   when the make invocations in the testsuite are run in parallel mode
  207   (as with "make check AM_TESTSUITE_MAKE='make -j4'").
  208 
  209 * Do not override Makefile variables using make arguments, as in e.g.:
  210 
  211     $MAKE prefix=/opt install           # BAD
  212 
  213   This is not portable for recursive targets (with non-GNU make,
  214   targets that call a sub-make may not pass "prefix=/opt" along).
  215   Instead, use the 'run_make' function, which automatically uses
  216   the AM_MAKEFLAGS to propagate the variable definitions along to
  217   sub-make:
  218 
  219     run_make prefix=/opt install        # GOOD
  220 
  221 * Use '$sleep' when you have to make sure that some file is newer
  222   than another.
  223 
  224 * Use cat or grep or similar commands to display (part of) files that
  225   may be interesting for debugging, so that when a user send a verbose
  226   output we don't have to ask him for more details.  Display stderr
  227   output on the stderr file descriptor.  If some redirected command is
  228   likely to fail, display its output even in the failure case, before
  229   exiting.
  230 
  231 * Use '$PATH_SEPARATOR', not hard-coded ':', as the separator of
  232   PATH's entries.
  233 
  234 * It's more important to make sure that a feature works, than make
  235   sure that Automake's output looks correct.  It might look correct
  236   and still fail to work.  In other words, prefer running 'make' over
  237   grepping Makefile.in (or do both).
  238 
  239 * If you run $ACLOCAL, $AUTOMAKE or $AUTOCONF several times in the
  240   same test and change configure.ac by the meantime, do
  241 
  242     rm -rf autom4te*.cache
  243 
  244   before the following runs.  On fast machines the new configure.ac
  245   could otherwise have the same timestamp as the old autom4te.cache.
  246 
  247 * Use filenames with two consecutive spaces when testing that some
  248   code preserves filenames with spaces.  This will catch errors like
  249   `echo $filename | ...`.
  250 
  251 * Make sure your test script can be used to faithfully check an
  252   installed version of automake (as with "make installcheck").  For
  253   example, if you need to copy or grep an automake-provided script,
  254   do not assume that they can be found in the '$top_srcdir/lib'
  255   directory, but use '$am_scriptdir' instead.  The complete list of
  256   such "$am_...dir" variables can be found in the 't/ax/test-defs.in'
  257   file.
  258 
  259 * When writing input for lex, include the following in the definitions
  260   section:
  261 
  262     %{
  263     #define YY_NO_UNISTD_H 1
  264     %}
  265 
  266   to accommodate non-ANSI systems, since GNU flex generates code that
  267   includes unistd.h otherwise.  Also add:
  268 
  269     int isatty (int fd) { return 0; }
  270 
  271   to the definitions section if the generated code is to be compiled
  272   by a C++ compiler, for similar reasons (i.e., the isatty(3) function
  273   from that same unistd.h header would be required otherwise).
  274 
  275 * Add any new test to handwritten_TESTS in 't/list-of-tests.mk', and
  276   to XFAIL_TESTS in addition if needed (that is, if the test is expected
  277   to fail).
  278 
  279 * In test scripts, prefer using POSIX constructs over their old
  280   Bourne-only equivalents:
  281 
  282     - use $(...), not `...`, for command substitution;
  283     - use $((...)), not `expr ...`, for arithmetic processing;
  284     - liberally use '!' to invert the exit status of a command, e.g.,
  285       in idioms like "if ! CMD; then ...", instead of relying on clumsy
  286       paraphrases like "if CMD; then :; else ...".
  287     - prefer use of ${param%pattern} and ${param#pattern} parameter
  288       expansions over processing by 'sed' or 'expr'.
  289 
  290 * Note however that, when writing Makefile recipes or shell code in a
  291   configure.ac, you should still use `...` instead, because the Autoconf
  292   generated configure scripts do not ensure they will find a truly POSIX
  293   shell (even though they will prefer and use it *if* it's found).
  294 
  295 * Do not test an Automake error with "$AUTOMAKE && exit 1", or in three
  296   years we'll discover that this test failed for some other bogus reason.
  297   This happened many times.  Better use something like
  298 
  299      AUTOMAKE_fails
  300      grep 'expected diagnostic' stderr
  301 
  302   Note this doesn't prevent the test from failing for another reason,
  303   but at least it makes sure the original error is still here.