Remember that when it comes to testing three things should
come to mind and should be explained to the reader of the tests:
1) What is the expected output?
2) What is the observed output?
3) Is the observed output the same as the expected output?
Why and/or why not?
Here are some general rules of thumb that you should consider when designing, implementing and running your tests. 1) Consider to whom you are trying to demonstrate your program (i.e., think about the marker). What type of and how much information will they want to see in your testing? For example, pretend that your are the project leader working in a large company. Once a month there is a large meeting of technical staff in which each of the project leaders has 10 minutes to give a status report on their project. This means that you have 10 minutes to demonstrate using the output of your testing what parts of your project do and do not work correctly. (Remember showing what does NOT work is just as important as showing what does work - perhaps more so). 2) Be smart about your testing. If the program you write which performs your tests (your test program) is more than 3 or 4 pages long it is likely too long. If the output produced from running the test program is more than 2 or 3 pages your test program is probably not designed very well. Think about the number of different things that need to be tested and how much time it will take to look at the output. A good test program should demonstrate simply and easily that the program does or does not work using minimal output. 3) Even though the test program is written only for the purpose of testing the implementation of other modules, classes or programs, it still needs to be clear, concise, and documented. In particular, the test program should specify what values you expect it to return (this can be done using comments where needed). 4) The test program should be bug free. Take some time when writing the test program to ensure that it is correct. Too often people waste a bunch of time trying to fix the modules they are working on when in fact the modules are fine but there is a bug in the test program (or they misunderstand how the test works or the return values that are expected). 5) If all the tests pass, then the test program need only print a single message to this effect. If a test fails, then useful information needs to be printed. One way of doing this is to make extensive use of the Turing assert statement. Recall that an assert statement in OOT automatically stops the program with useful information as to which statement failed and the cause of the failure. The goal is to reason about the state of the testing program and the underlying modules being tested and to use knowledge about what you know must be true if the modules being tested are functioning correctly to insert enough useful assert statements so that if all the assertions pass, then the program known to be correct. (Although our test program may not be able to PROVE that the entire program and/or the modules are correct we'd like to provide strong evidence that they are very likely correct.) 6) Someone should be able to spend about 1 minute looking at the output of each test program to determine whether all of the tests in the program failed or succeeded. 7) Use multiple test programs to test different aspects of your system. (In assignment 1 these different tests are performed during the different task. For example, your interactive tests performed with driver.dem should differ from the tests you perform using tester.dem.) 8) Test special cases. Ideally a production quality program should be completely bullet proof (not that many really are :-). No user, even malicious ones, should be able to crash the program. 9) Check the return values of function calls. That is check that the call has completed successfully - if it has not handle the error intelligently.
Updated for 1030: January 6, 1997