Next: Debugging a test case, Previous: Board configuration file values, Up: Extending DejaGnu
The easiest way to prepare a new test case is to base it on an existing one for a similar situation. There are two major categories of tests: batch-oriented and interactive. Batch-oriented tests are usually easier to write.
The GCC tests are a good example of batch-oriented tests. All GCC tests consist primarily of a call to a single common procedure, since all the tests either have no output, or only have a few warning messages when successfully compiled. Any non-warning output constitutes a test failure. All the C code needed is kept in the test directory. The test driver, written in Tcl, need only get a listing of all the C files in the directory, and compile them all using a generic procedure. This procedure and a few others supporting for these tests are kept in the library module lib/c-torture.exp of the GCC testsuite. Most tests of this kind use very few Expect features, and are coded almost purely in Tcl.
Writing the complete suite of C tests, then, consisted of these steps:
Testing interactive programs is intrinsically more complex. Tests for most interactive programs require some trial and error before they are complete.
However, some interactive programs can be tested in a simple fashion reminiscent of batch tests. For example, prior to the creation of DejaGnu, the GDB distribution already included a wide-ranging testing procedure. This procedure was very robust, and had already undergone much more debugging and error checking than many recent DejaGnu test cases. Accordingly, the best approach was simply to encapsulate the existing GDB tests, for reporting purposes. Thereafter, new GDB tests built up a family of Tcl procedures specialized for GDB testing.
It is safest to write patterns that match all the output generated by
the tested program; this is called closure. If a pattern does not match
the entire output, any output that remains will be examined by the next
expect
command. In this situation, the precise boundary that
determines which expect
command sees what is very sensitive to
timing between the Expect task and the task running the tested tool. As
a result, the test may sometimes appear to work, but is likely to have
unpredictable results. (This problem is particularly likely for
interactive tools, but can also affect batch tools—especially for
tests that take a long time to finish.) The best way to ensure closure
is to use the -re
option for the expect
command to write
the pattern as a full regular expressions; then you can match the end of
output using a $. It is also a good idea to write patterns that
match all available output by using .*\ after the text of
interest; this will also match any intervening blank lines. Sometimes
an alternative is to match end of line using \r or \n, but
this is usually too dependent on terminal settings.
Always escape punctuation, such as ( or ", in your patterns; for example, write \(. If you forget to escape punctuation, you will usually see an error message like:
extra characters after close-quote
If you have trouble understanding why a pattern does not match the
program output, try using the --debug
option to runtest
,
and examine the debug log carefully.
Be careful not to neglect output generated by setup rather than by the
interesting parts of a test case. For example, while testing GDB, I
issue a send set height 0\n command. The purpose is simply to
make sure GDB never calls a paging program. The set height
command in GDB does not generate any output; but running any command
makes GDB issue a new (gdb) prompt. If there were no
expect
command to match this prompt, the output (gdb)
begins the text seen by the next expect
command—which might
make that pattern fail to match.
To preserve basic sanity, I also recommended that no test ever pass if
there was any kind of problem in the test case. To take an extreme
case, tests that pass even when the tool will not spawn are
misleading. Ideally, a test in this sort of situation should not fail
either. Instead, print an error message by calling one of the DejaGnu
procedures error
or warning
.
Next: Debugging a test case, Previous: Board configuration file values, Up: Extending DejaGnu