Basic Tests

The basic tests are designed to help students detect any simple issues with their code. If the solution's output is in the wrong format, the name of the function does not match the name given in the assignment or there is an error in the file, the basic tests should catch it. That being said, every instructor will have different preferences when it comes to how basic the basic tests should be. The important thing to remember is that you should always ask the instructor before releasing the assignment.

The practice in the past has been to include a very simple test case. This can be the example given in the assignment or something else that is trivial. We have to be careful with basic tests, because a student who failed can see that they failed and gets to see what the expected output was. We do not want to help them write their functions by showing them additional examples of expected behaviour. However, some instructors may want the tests to be even simpler. For example, they may want to only type check the output, without even checking if the answer is correct. On the opposite end of the spectrum, they may want to analyze the students output and give them hints on how it is wrong. You can make test.rkt do something like this:

(result (cond [(boolean? (student-fn 2)) true]

[else (error "Function must produce a boolean")]))

(expected true)

This throws an error for the evaluator to catch, producing a fairly clean message without revealing the expected value as usual.

If you are uncertain on how complicated to make the basic tests ask the instructor in charge of the assignment. If there are multiple instructors making assignments be sure to ask each one what their preferences are.

A common thing for the basic tests to check is any misspellings which are syntactically correct and don't crash the program. This often comes up when dealing with structs, struct fields, and symbols, often affecting CheckTestcases as well. Your basic tests should be slightly more comprehensive so that students can easily realize they spelled something wrong before they end up getting 0 on correctness. For example:

(cond [(symbol? x 'conservatave) ...] ;; is conservative=

=[else ...] ;; is liberal=

If the only basic test was a case with 'liberal, it does not tell the student that they spelled 'conservative incorrectly. Instructors may feel this is unfair and something that the basic test should catch.

A similar principle can be used when dealing with complicated data structures. You can check that the student is treating the data definition correctly by doing a complicated test case, but simply checking something like (not (empty? ...)) or (integer? ...) as the actual test. This way, the basic test tells the student if their program crashes without revealing too much information about the case itself. This type-checking technique can also help catch calls to disallowed functions; remember that basic tests only fail if the banned function is actually called. If you test a simple base case like empty and nothing else, it will not alert the student to their use of a banned function in the non-base cases.

(pre Dec/23

The basic tests are designed to help students detect any simple issues with their code. If the solution's output is in the wrong format or there is an error in the file, the basic tests should catch it. That being said, every instructor will have different preferences when it comes to how basic the basic tests should be. The important thing to remember is that you should always ask the instructor before releasing the assignment.

The practice in the past has been to include a very simple test case. This can be the example given in the assignment or something else that is trivial. We have to be careful with basic tests, because a student who failed can see that they failed and gets to see what the expected output was. We do not want to help them write their functions by showing them additional examples of expected behaviour. However, some instructors may want the tests to be even simpler. For example, they may want to only type check the output, without even checking if the answer is correct. On the opposite end of the spectrum, they may want to analyze the students output and give them hints on how it is wrong. You can make test.rkt do something like this:

(result (cond [(boolean? (student-fn 2)) true]

=[else (error "Function must produce a boolean")]))=

(expected true)

This throws an error for the evaluator to catch, producing a fairly clean message without revealing the expected value as usual.

If you are uncertain on how complicated to make the basic tests ask the instructor in charge of the assignment. If there are multiple instructors making assignments be sure to ask each one what their preferences are.

A common thing for the basic tests to check is any misspellings which are syntactically correct and don't crash the program. This often comes up when dealing with structs, struct fields, and symbols, often affecting CheckTestcases as well. Your basic tests should be slightly more comprehensive so that students can easily realize they spelled something wrong before they end up getting 0 on correctness. For example:

(cond [(symbol? x 'conservatave) ...] ;; is conservative=

=[else ...] ;; is liberal=

If the only basic test was a case with 'liberal, it does not tell the student that they spelled 'conservative incorrectly. Instructors may feel this is unfair and something that the basic test should catch.

A similar principle can be used when dealing with complicated data structures. You can check that the student is treating the data definition correctly by doing a complicated test case, but simply checking something like (not (empty? ...)) or (integer? ...) as the actual test. This way, the basic test tells the student if their program crashes without revealing too much information about the case itself. This type-checking technique can also help catch calls to disallowed functions; remember that basic tests only fail if the banned function is actually called. If you test a simple base case like empty and nothing else, it will not alert the student to their use of a banned function in the non-base cases.)

Testing Basic Tests

(Dec 23 / update)

It is very important to test the basic tests before releasing the assignment. One simple way to do this is to follow the steps below

• Set up MarkUs for the assignment and make the assignment hidden to students

• Switch to your student account (i.e. userid-student)

• Submit your files to MarkUs

• Run rst on your questId

If there are issues with MarkUs for some reason, use this method

• Make a temporary directory using mkdir ~/handin/aXX/questid

• Place the solutions you would like to test in this directory

• Run a command like rst –t t –s questid aXX pt test1

• Delete the directory created for aXX after testing is done

To be safe, use your own id for the questid. The pt in the rst command tells the script to run the basic tests, for correctness tests use 0 instead of pt. Deleting the directory in handin will prevent any conflicts with MarkUs. Be sure to test both correct and incorrect solutions. This will ensure that the basic tests are working as expected.

(pre Dec 23

It is very important to test the basic tests before releasing the assignment. One way to do this is to follow the steps below.

• Make a temporary directory using mkdir ~/handin/aXX/questid

• Place the solutions you would like to test in this directory

• Run a command like rst –t t –s questid aXX pt test1

• Delete the directory created for aXX after testing is done

To be safe, use your own id for the questid. The pt in the rst command tells the script to run the basic tests, for correctness tests use 0 instead of pt. Deleting the directory in handin will prevent any conflicts with MarkUs. Be sure to test both correct and incorrect solutions. This will ensure that the basic tests are working as expected.)

Comments