Special Cases and Common Issues

This covers miscellaneous helpful topics and general tips that can be used while setting up public/private tests.

CommonIssues

Many of these sections will be labeled with an error that is displayed after running rst. Not all of these errors are fully understood. As a last resort, you can always contact the Computer Science Computing Facility (CSCF) for help. Please keep in mind that provided files are written in the full racket.

When Required Module is Not Found

 Required module lib.rkt not found;

The test script cannot find the library needed. It may be that the file name was misspelled in the options file or not named correctly in the provided. Or, the file may just be missing from the provided. It can actually still run properly with the error message if the student submitted a file of the same name into their folder.

When Using Disallowed Functions

 Disallowed function xxx called

This one is straightforward. If you are stuck, check all files for use of the disallowed function. This includes the submitted files and the test.rkt files. If the function is NOT supposed to be disallowed, comment out the function from disallowed.rkt as shown here.

Error with Creating Evaluators

 Evaluator creation error: FAILED - Error creating evaluator, [insert error message] 

There are many different things that can lead to this issue. The best thing to do is read the entire error message; it's usually a long and scary block of characters followed by a small helpful message at the very end. The list below is causes and fixes from most likely to least likely.

• The submission is requiring a library that is not in the list of modules in the options.rkt file. To fix this add the file's name into the modules line in the options file for that question.

• The submission may be using a banned special form. Some special forms will give the disallowed function error message and some will give a funny error message. Double-check the allowed use of lambda and local from the assignment specifications and what the disallowed.rkt file states.

• One of the provided files may be requiring a standard racket library. It is unknown why this causes problems. It also seems that there is only an issue for some racket libraries. If it is something just for fun, like a drawing library, then comment out any code related to it and make functions that use it just return empty. Contact the instructor or CSCF if this is not a valid solution.

• One of the provided libraries may be requiring another file in the provided directory. This error has to do with the load order of the modules. The fix that has been shown to work is to not do this. As a workaround, you can copy the code into one big file and provide that as a module. Or, check the order that you specified your modules in the (modules ...) line.

If the error message includes 'Timeout of 60 seconds reached; assuming infinite recursion" when the function clearly should not time out, be careful when banning some functions. For some reason, what should say "FAILED - Error creating evaluator, Disallowed function reverse called" becomes a timeout message when the function called format is not allowed. Make sure to allow the format to prevent this issue; it's a fairly trivial function that seems to be used by the autotesting scripts.

GeneralFAQ

How would I account for students who already have a (require ...) line because it's actually part of the assignment?

For most occasions, students might leave (require htdp-trace) at the top of the file. If they have not actually used the feature of htdp-trace, define/trace, the fastest way to fix this is to comment out the (require ...) line in ~/handin/axx_autotest/userid. If they have used define/trace, provide cs135-trace-correctness.rkt (located in ~/marking/provided-library) into the modules and make sure a copy of the provided file is in our test.0/provided/ folder.

If the students are allowed to require their own files, it gets more complicated. In order to use their version of the required file, you do not put anything in test.0/provided/ and only add the name to the (modules ...) line. In general, allowing students to require their own files is not recommended, as it can cause issues with names being re-defined when running the check-testcases script. Be careful asking for similar types of questions to be submitted in different files in this case, as they may have names repeated.

Can students keep their check-expects in their files to get marks for test case coverage without interfering with correctness?

Although DrRacket is an interpreted (actually, JIT-compiled) language, there is still some "compilation" that takes place. The application performs an initial syntax check for things like missing brackets without running anything. Then, it runs the code sequentially, interrupting mid-execution if an error pops up. Finally, check-expect is a special form which is only evaluated after these first two steps are completed. In short, the correctness tests will run fine even if there are incorrect check-expects in the file, because the functions can be called as long as they've been defined at some point. Of course, the tests will still fail if the check-expect has an important character missing or calls a function that the student didn't even write.

How do I fix and re-run test cases if TA marking has already started?

The fastest way is probably to use the set_marks_rst command to re-assign the marks on MarkUs. You can create a folder separate from test.pt and test.0, like test.1.fix_q2_test_002, with only the Q2 folder present to make rst/distrst run faster. Then you can use set_marks_rst with the test.1.fix_q2_test_002.AUTOTESTRESULTS folder targeted.

How can I allow list outputs to be considered correct regardless of order?

Use order-not-matter.rkt located in ~/marking/axx/provided/ instead of equal?.

SpecialCases

Combining Correctness Test Results in OUTPUT.txt

RST requires having different folders for each question. If function names are different or if stepper questions have different values, we must create different directories for each. If there seem to be too many subquestions for one big question, we can combine the test results using underscore _. Usuage:

QuestionNum_SubQuestionNum

Make sure QuestionNum stays the same for all questions that need to be combined. Just like RST, the same folder with the same QuestionNum and SubQuestionNum will not be valid (e.g. 1_b 1_b will likely cause an error). SubQuestionNum does not need to be necessarily meaningful. However, keep in mind that the SubQuestionNum will be shown as a part of OUTPUT.txt.

Examples:

If we have question folders like below,

 1a 1b 

The OUTPUT.txt will look like this:

x/2 Total Mark

** Question 1a: x/1
** Question 1b: x/1
(Question 1a, Test 001, 1 marks): Checking xxx:...
...
(Question 1b, Test 001, 1 marks): Checking xxx:...
...

On the other hand, when we use underscore (_),

 1_a 1_b 

The OUTPUT.txt will look like this:

x/2 Total Mark

** Question 1: x/2
(Question 1, Test a_001, 1 marks): Checking xxx:...
...
(Question 1, Test b_001, 1 marks): Checking xxx:...
...

Dealing with Inexact Numbers in Full Racket

It is recommended to work in full Racket for any provided modules you write. However, in the full racket, (equal? 0 0.0) will be false while (= 0 0.0) outputs true. Thus, when the lambda function is used in testings, it should NOT be defined in full Racket. For other constants which output is inexact, please make sure to use them with check-within-cs135? (in ~/marking/provided-library) or (< (abs (- (functions...) output)) tolerance).

First Solution: Define the constant and helpers directly in test.rkt. Although the file will become longer if the constants are too big, this is a safe way to avoid any issues.

Example:

(result (= (function-name (lambda (x) (* 1/2 (expt x 3))) ...) ...))

Second Solution: Define in a file that is set to the appropriate language that the assignment uses Then, define the full racket module which allows that racket-defined module can be used. For example,

Step 1: Define functions/constants/helpers that will be used in testing in a file set to the appropriate language. Add the (require ...) line below. If we use Intermediate Lambda language,

#reader(lib "htdp-intermediate-lambda<-reader.ss" "lang")
       ((modname FILENAME) (read-case-sensitive #t) (teachpacks ()) (htdp-settings #(#t constructor repeating-dec)))
(require "get-provide.rkt")
(cs135-provide FunctionNames...)
(define ...)

Step 2: Create "get-provide.rkt" (or any name) in a full racket language, which allows the file made in step one to be used. It only requires two lines.

#lang racket
(provide (rename-out [provide cs135-provide])) 

Please note that rename-out part is not necessary. However, it is safe to have it there in case students name something "provide". Make sure both of the files are added to options.rkt modules.

Lastly, please keep in mind that when decimals or fractions have to be used, it is recommended to keep them as fractions to avoid any potential issues.

How to Provide Our Own Function Definition

If you want to provide your own function definition, say the student is doing part 2b) and you want to provide your own definition for part 2a) to be used in 2b) (So that the student messing up their own definition of 2a, doesn't mess up the definition of 2b as well). Then:

• You should put the file 2a) is defined in the provided directory of the test setup so it can be used by the student 2b) submission and include it in modules for the options.rkt for 2b) (/in/2b/options.rkt)
• Be very careful not to include it for 2a) (so it shouldn't be included in modules for any parent options.rkt file) since then instead of testing the student's function it will use our own version of it, as the filenames will be the same and all the students will pass regardless of what they submitted.
• You also need to use exit-early.sh which is in marking/provided-library/analyze-helpers directory as an instructor script for 2a). So copy it over to the provided folder for the assignment, make necessary changes, and add the following line to in/2a/options.rkt

(instructor-script "provided/exit-early.sh")

This is because if a student does not submit 2a) then it will automatically use our version of 2a) even if it was not explicitly included (yes it is annoying) and give the student full marks anyway. So we don't let bittersuite handle it, and instead just exit early if the student did not submit.

• If you want to enforce that students require our provided file and use our version for 2b no matter what then use the checkprovided option from this script: CheckingHelpersRecursion

Comments