Marmocreate Improve Pit

Marmocreate is a program that wraps a collection of RST test suites in a fashion that can be understood by Marmoset. It has only been used to demonstrate that this is possible; as such, a number of very useful features are missing.

Retaining Extra Files

The model RST uses is more general than that of Marmoset. One feature is that extra files are often retained via keepFile or printFile; for now, these will silently be dropped. The Marmocreate utility should probably add an extra test at the end of every suite run, worth 0 marks, which is simply a dump of the contents of OUTPUT.txt. It may be preferable to dump OUTPUT.pdf; however, it appears Marmoset is only capable of displaying text so this is the assumed output format for now.

A note about the positioning of test results and output comparisons

RST encourages a model where files showing differences between files (for example, a side by side comparison of generated and expected outputs), if generated, are included in the OUTPUT file. In a Marmoset display as encouraged above, this would mean the comparison related to a test would not appear with that test.

There are arguments for showing the comparison to output files with the test, so all feedback and marks for the test are coupled together. There's also an argument, though, for listing the test results in a compact form so they can be more readily absorbed as a whole, and then putting the more elaborate results below. It's really just a matter of aesthetic taste.

Automatic Mark Allocation

Marmocreate determines the test names that will be given to Marmoset based on the test names in the file generated by writeToTestResults. This file already contains the number of marks each test is worth, but Marmoset still requests that this information be provided manually via a web form. If there is a way to automate this information transfer, it should be utilized.

Correctness Assurance

Marmocreate currently generates a thin wrapper for RST which will depend on the configuration settings RST uses. If these change during the term, it can break test information generated by Marmocreate. To guard against this, Marmocreate should hardcode all of the relevant RST parameters in the generated scripts automatically to ensure stability (for example, output file separators, etc.).

Result directory accumulation

A test run will generate output directories and leave them around as garbage. This should really be cleaned up when Marmoset is done doing the testing. It is not very natural in the general case, but if the "retaining extra files" test recommended above is implemented and run as the very last test, this would be a natural place to put the cleanup directions.

It appears Marmoset may be generating two results directories when it does RST runs. This would be very strange, and should be confirmed and explained if it is true.