This is a general guide for testing student's assignment submissions in CS135. Basic knowledge of the racket programming language, command line navigation and testing practices is expected. The best way to read this guide is to have examples from previous terms open in another window. You do not have to remember everything in this manual, just remember that it exists and that it may have answers to your questions throughout the term.

If there is anything you come across that is not in this guide, please add it. It will benefit future ISA's if this guide is kept up to date with as much knowledge as possible.

There is also a short summary of the tasks at CompSci135TutorDuties.

Layout and Format

This section will go over the basics of the layout of the marking directory. This includes the file tree, descriptions of file layouts and how to add new tests.

File Layout

This diagram presents an overview of the file structure in the course account.

Most of what you need is under the marking directory in the course account. From here you can go into a specific assignment. The convention is to label the assignment directories aXX, where the XX is the number of the assignment. Under each assignment directory should be a test.0 and a test.pt directory.

There may also be some test.0.something and test.pt.something directories. The "something" can be any string, and is just used as a label. These directories contains test results for students. One level down will be directories labeled with student's quest ids. These contain the results for that individual student.

The test.pt directory contains the basic tests and the test.0 contains the correctness tests. Basic tests are created before the assignment is released to students. They include a test for each question so that students can catch simple errors. Correctness tests are used to fully test and grade a students submission. They must be finished before the assignment is due (or at least collected and tested). Both testing directories have the same structure, so only the test.0 is shown in detail.

One level into test.0 there is a provided and an in directory, as well as some other files. The header file contains the text that will show up at the top of all test results for the assignment. There is also a computeMarks file not shown on the diagram. This is part of the script that runs tests on students; it should not need any modification. Other files not listed in the diagram should not need any modification.

The provided directory is where any libraries or helpful modules will go. For basic details on including libraries see the options.rkt section. For more in-depth details, or if you are running into errors, see the correctness test section.

in contains directories for every question on the assignment, this is where the tests go. Within each question directory will be directories starting at 001 going up to the last test that you want to write for a question. These directories contain a test.rkt file which is an individual test for that question. Basic tests will usually only need just one test, so there will only be a 001 for that question. Stepper questions have a different format described in the Banning Functions section.

test.rkt

The format of these files are very simple, the contents have the following form.

(result (student-function our-test-input)) 
(expected expected-output)      

The expression in the result line will be evaluated in the language chosen in the options file. The value produced will then be compared to the value in the expected line, using the function equal?. If these two values are the same, and there were no errors while evaluating any expressions, the submission will pass the test. Unfortunately, this is not a built-in form in Racket, so you can only check them in check-expect form or use rst to actually run them on the server. You can still call other Racket functions in the test case, even something like a cond with multiple cases depending on what the student's function does. There will be more tips on writing tests in the basic and correctness test sections.

options.rkt

There are two different kinds of options.rkt files. The one directly under the in directory contains the global options for the entire assignment. It will have the following form.

(language scheme/LANGCHOICE) 
(timeout 60) 
(memory 256)  
(value 1)        

LANGCHOICE is the language level allowed. This will be set depending on the assignment specifications. The possible options are beginner, beginner-abbr, intermediate, or intermediate-lambda.

The next line contains the time limit in seconds that the students are allowed for each test (default 60). Please note that this means that each individual test has this time limit, not all tests for one question.

The memory line has the allowed memory per test in MB; it can be omitted, defaulting to 128. If the default is not working then you may need to add this line to increase the memory limit, or instructors may want a more restrictive memory limit.

We cannot control the file size of the students' actual submissions, but Nick from CSCF had it set to 5 MB on MarkUs in Fall 2018.

Lastly, the value line should not need to be changed.

The second type of options files are the ones under each question. These files have specifications for each individual question. The general form is below.

(desc "string describing the test") 
(modules "lib1.rkt" "lib2.rkt" "disallowed.rkt") 
(loadcode "filename.rkt")         

The desc in the first line stands for description, and contains a string. There should be a convention to follow throughout the term. For example, you may want to use something like "Checking 2b".

The modules line contains all of the libraries that need to be included. Each of these arguments will be converted into a (require "libX.rkt") placed at the top of the student's file in the same order, and makes sure the file gets copied to the temp folder where the testing takes place. The files listed here are strings with spaces between each other. Also note that a file must be located in the provided directory if you want to include it as a standard module; alternatively, if it does not exist in the provided directory, the student's version in the handin folder will be used. If the students are given a library as part of the assignment, that library also needs to be included in the modules list. The disallowed.rkt file should be last, because it usually disallows the function require, which gets called when any subsequent modules are provided into the student's file. There are no filename requirements to follow, but we try to use names like "disallowed-except-reverse.rkt" and "provided-3b.rkt" for consistency.

The loadcode specifies the file name that the students solution is expected to be in. There should only be one file; put other relevant student files in the (modules ...) line.

It is also possible to specify any lines from the global options in the options for a single question. For example, you may want every question to use a 60 second time limit and beginning student, but allow 3b to have only 30 seconds and beginning student with list abbreviations. In this case the options.rkt under 3b may look like the following.

(desc "Checking 3b") 
(modules "disallowed.rkt") 
(loadcode "anagram.rkt") 
(language scheme/beginner-abbr) 
(timeout 30) 

Banning Functions

Students are often restricted on what built in functions they can use. The point of the assignments is to find a solution to the problem, not to look for a built in function that will solve the problem for them.

Format of disallowed.rkt

For every assignment there will be a file containing a list of banned functions and special forms, most likely disallowed.rkt. It uses the define-syntax or "macro" feature of full Racket to make any banned functions throw an error when they are called. The file has a similar form to below.

(define-syntax disallow 
    Some-Racket-Code  ) 

(disallow  ;require 
           lambda 
           local 
           if 
           ;append 
           ...) 

As you can see, the function append has a semi-colon before it, meaning it is commented out. This means that the this function is not included in the disallowed list and the students can use it freely in their solutions. The special form local is in the disallowed list so this means that students will get an error if they use it.

If you want to ban functions that are not listed in the file simply add it to the list, preferably in the correct place alphabetically. There may be cases where adding some special forms will not work. In the past, cond has presented issues. A solution for cond would be to copy the below lines of code exactly as is, including the "...", adding it just below the #lang racket line.

(provide cond) 
(define-syntax-rule (cond ...) (error "cond is disallowed"))

This may also work with other special forms not in the disallowed list, but it has not been tested.

Tips for Banning Functions

If you believe there are functions that should be banned, ask the instructor before banning them. It may be that the instructor was unaware of a potential solutions involving built in functions. It may also be the case that the instructor believes a solutions using the built in function is acceptable; for example using string->number and string-ith to deal with digits of a number, when strings haven't been explicitly allowed on the assignment.

Adding a function to the disallowed list will disallow any direct calls to it in the student's submission file or test.rkt. For example, if you wanted to ban the function "and" because they must implement it with nested conds, you would not be allowed to use "and" directly in test.rkt. See the "Working with Racket languages" section for workarounds to this.

If the instructors want to give the students a library to use, make sure that require is commented out so they can work with it without having to copy and paste.

A very useful thing to do is make different disallowed files for different questions. It is common to ban or allow functions for specific questions. The way to do this is copy the disallowed file, rename it for the question, for example disallowed3c.rkt and keep it in the provided directory. Then change disallowed3c.rkt to follow the assignment specifications. Lastly, make sure to change the module in the options.rkt of the question to include the correct version of the disallowed file.

Effect on Students using Banned Functions

A final thing to note is what happens when a student uses a function in the disallowed list. The way the disallow works is that it redefines any functions in the list to throw an error. This means that if a students code does not run the function they will not lose any marks on that test. For example, let's say a students solution looked like the following.

(define (student-fn x)
  (cond
    [(even? x) (banned-function x)]
    [(odd? x) (something-ok x)])) 

In this case the student would get half the marks, assuming half of the tests used odd numbers. This may or may not be the intended result. One argument is that the deduction they get for using the banned function is fair. Another argument is that they should lose all marks since the assignment fully specified the problem. Be sure to check with the instructor of the course on what marking scheme should be used. If using the first one, then no additional work is required. If the instructor would prefer to dock all marks for using disallowed functions then you will need to find all the students that have used it, even if they only failed one test case from it. One way to do this would be to read the error message the student receives when failing a test from a disallowed function. Finaly use the following command in the directory with the test results to get all of the students who have used the disallowed function.

grep -r 'error message of test'

Basic Tests

The basic tests are designed to help students detect any simple issues with their code. If the solution's output is in the wrong format or there is an error in the file, the basic tests should catch it. That being said, every instructor will have different preferences when it comes to how basic the basic tests should be. The important thing to remember is that you should always ask the instructor before releasing the assignment.

The practice in the past has been to include a very simple test case. This can be the example given in the assignment or something else that is trivial. We have to be careful with basic tests, because a student who failed can see that they failed and gets to see what the expected output was. We do not want to help them write their functions by showing them additional examples of expected behaviour. However, some instructors may want the tests to be even simpler. For example, they may want to only type check the output, without even checking if the answer is correct. On the opposite end of the spectrum, they may want to analyze the students output and give them hints on how it is wrong. You can make test.rkt do something like this:

(result (cond [(boolean? (student-fn 2)) true]
              [else (error "Function must produce a boolean")]))
(expected true)

This throws an error for the evaluator to catch, producing a fairly clean message without revealing the expected value as usual.

If you are uncertain on how complicated to make the basic tests ask the instructor in charge of the assignment. If there are multiple instructors making assignments be sure to ask each one what their preferences are.

A common thing for the basic tests to check is any misspellings which are syntactically correct and don't crash the program. This often comes up when dealing with structs, struct fields, and symbols, often affecting CheckTestcases as well. Your basic tests should be slightly more comprehensive so that students can easily realize they spelled something wrong before they end up getting 0 on correctness. For example:

(cond [(symbol=? x 'conservatave) ...] ;; is conservative
      [else ...] ;; is liberal

If the only basic test was a case with 'liberal, it does not tell the student that they spelled 'conservative incorrectly. Instructors may feel this is unfair and something that the basic test should catch.

A similar principle can be used when dealing with complicated data structures. You can check that the student is treating the data definition correctly by doing a complicated test case, but simply checking something like (not (empty? ...)) or (integer? ...) as the actual test. This way, the basic test tells the student if their program crashes without revealing too much information about the case itself. This type-checking technique can also help catch calls to disallowed functions; remember that basic tests only fail if the banned function is actually called. If you test a simple base case like empty and nothing else, it will not alert the student to their use of a banned function in the non-base cases.

Stepping Questions

The options.rkt file for a stepping problem should not need to be changed in any way. It has the following format.

(language external) 
(desc "Stepping Problem Results")
(value 4) 

Instead of a numbered directory under the question, there will be a directory called Stepper. Inside this directory should be a file test.exe. There are three lines in this file that will need to be changed. The first one will me similar to the following.

my $duetime = 1511316000;

The number should be set to the time that the assignment is due in Unix epoch time. You can use the date command as shown in the file to get this (either something like date +%s -d "Oct 16 21:00:00 EDT 2018" or date +%s -d "Dec 16 21:00:00 EST 2018" depending on whether daylight savings is active). If it still does not work try to remove the option that specifies the time zone or just look up the Unix epoch time online.

The second line to change looks like the following.

next unless $F[3] =~ /^assignment8q1([a-d])$/; 

The assignment should be changed to the correct number and the correct question. The [a-d] part should have bounds that represents the number of stepper problems on the assignment. In this example, there are four stepper problems on assignment eight, question one.

Finally, this line is just for aesthetic purposes:

print THREE (($#plist + 1) * 100), "/4\n";

The "/4\n" should correspond to how many steppers the question is "out of" - in this case, we had a-d so it was out of 4.

Testing Basic Tests

It is very important to test the basic tests before releasing the assignment. One way to do this is to follow the steps below.

  • Make a temporary directory using mkdir ~/handin/aXX/questid

  • Place the solutions you would like to test in this directory

  • Run a command like rst –t t –s questid aXX pt test1

  • Delete the directory created for aXX after testing is done

To be safe, use your own id for the questid. The pt in the rst command tells the script to run the basic tests, for correctness tests use 0 instead of pt. Deleting the directory in handin will prevent any conflicts with MarkUs. Be sure to test both correct and incorrect solutions. This will ensure that the basic tests are working as expected.

Correctness Tests

Correctness tests are used to completely test every students code. The mark they get on an assignment are heavily dependent on the results of the correctness tests. Testing methodologies is a huge topic in computer science. The main focus on this section will be tips and tricks specific to the testing software you will be working with.

Test your Test

Testing the test you have written is by far the most important thing to do. This can be done with the rst command as in step three of the previous section; just change the pt to a 0. Make sure to test solutions other than your own. These are some useful steps to go through.

  • Use the instructors solution and other ISA solutions if available. This will help ensure that you understood the questions properly and wrote correct tests.

  • Test incorrect code against the test you have written. Check that this code missed a few tests and determine if the mark received was reasonable.

  • Lastly, test your correctness tests on a few students that have submitted. This will further help to catch any errors that may have been made.

It is impossible to prevent every issue that can happen. Be prepared to fix something that goes wrong and make it a learning experience for the future. If you would like to re-evaluate your tests after assignments have already been collected, you can make a copy of the test.0 folder called something like test.1 or test.0.q2_check and make changes in that folder before using it for rst/distrst.

If there are a lot of test cases before the question you're actually testing and rst is being slow, you can make your question be the one tested first. Since rst goes through the testing folders in order by name, you can do a command like "cp -r Q4 Q0004" so that the results for Q0004 are the ones run first. Remember to delete the temporary folder once you're done.

Keep it Fair

Remember that RST and MarkUs do not distinguish between your test cases; every test you add is given a weight of exactly 1. As such, it is best to make the tests as directed and equally spread-out as possible. What this means is that one test should have a single, distinct purpose. This may be testing a specific edge case, a largish input or something else. The important thing is to not overlap multiple edge cases with one test. This should actually be made into a different test called a corner case. It may be possible that four tests can fully tests a question for correctness, but a suite of directed tests is 16 tests. In this case a student could get 0/4 test right when they really deserve 12/16. It is often challenging to develop directed tests, but is crucial in giving students a fair evaluation. If you have multiple tests which consider different cases but follow the same idea, you can combine their weight into a single test case using an (and (equal? test1 expected1) (equal? test2 expected2) ...) statement, so that part of the question is not too heavily weighted.

Test valid input only. Surprisingly, it is a very common place for issues to arise. After developing a test suite, re-read the assignment and all clarifications from the instructors on Piazza, then use that to determine if every test contains input that is valid and/or fair for the students to consider. Doing black box testing is another good way to make sure you are testing valid input.

Finally, always ask first before testing efficiency. Efficiency is a messy topic in CS135. It is never explained formally, yet students are sometimes expected to have an idea of how to make something "more" efficient. By default, do not test efficiency. If you think that a problem should have one or two efficiency tests you must ask the instructor before including it. Testing efficiency usually involves creating a very large input and possibly lowering the time or memory limits for a question (see the section describing options.rkt on how to do this).

Make Testing Easy

Write tests in DrRacket as you do the assignment, this will just save you time in the long run. It is much easier to make thorough tests when you have your head wrapped around the problem. It is also more convenient to validate your tests in DrRacket than it is using rst on the course account. Lastly, you can easily convert you solutions to the sample solutions, just remove some of the harder tests you have written.

Do not be afraid to make constants and helper functions. For example, if the student has to write a solution to a card game, then a constant for every card and a few hands may be convenient. You can also make functions to compare the students result with the expected result. This is very helpful if your check for correctness needs to be more complicated than an equality statement. Making test constants and helper functions will be helpful on some of the earlier assignments, and necessary by the last assignment. You can even make functions which generate large test constants, like a height 10 BST (make sure they're valid!). If the assignment does not involve any structures it is very easy to make constants and helper functions following the steps bellow.

  • Create a racket file, say lib.rkt, containing constants and functions you want to use in the test.rkt file.

  • Add a provide line at the top of lib.rkt and specify all the constants and functions needed.

  • Add lib.rkt under the provided directory.

  • Add "lib.rkt" in the modules option for the options.rkt file for the question.

For more details on creating helper libraries see the example at the end.

You may want to ask instructors if they are reusing questions from previous terms. If they are you can search the archives for the old test suites, just be sure to test them to make sure they work.

Other Helpful Tips

Testing for correctness is black box testing. This means that you should assume nothing about the implementation of the question and test based only off of the assignment specifications. This creates a fair and rigorous test suite for all students. With many students, comes many different solutions. Basing the correctness tests off of a particular solution only ensures a good test suite for solutions with a similar implementation.

Always pair test cases for empty and Boolean values with a different case at the same time. This is to make sure students do not get marks for a trivial function. For example, suppose students were asked to write a predicate good?. Have a look below at a potential submission:

(define (good? x) false) 

This should probably receive no correctness marks. However, if half of the tests expected false this student would get half the marks for doing almost nothing. The way to fix this is to write test similar to below.

(result (and (good? input1) 
  (not (good? input2)))) 
(expected true)  

If a student always outputs the same answer they would fail a test like this. With a correct implementation the student will still get the test correct. For the "helper" test, try to stick to really obvious cases which interfere as little as possible with the actual result. The point of the second test is to make sure that there exists a case where the student's function produces a different value, and it should definitely not introduce another check which makes it harder to pass the case. A basic test is always a good candidate to add in, because you can be relatively confident that the student passed it. For more complicated data structures, you may want to pre-emptively deploy two basic tests (one for true and one for false) so you can use them in the private tests and work under the assumption that students already passed them.

Another good rule is to never use the examples given in the assignment as correctness tests. Students may hardcode information from the example into their solution if they do not understand the question. This should be penalized; the easiest way to do that is to test using input that is not related to the given example in any way.

The course generally avoids inexact numbers, but they may pop up in the early assignments. Simply use a range-based check that emulates check-within for this:

(result (< (abs (- (some-function 2) 0.01155)) 0.0001))
(expected true)

Providing Libraries and Helper Files

This section should be your go to reference if there are any issues with providing a library or creating helper functions for testing. For the most part, all you need to do is put the file in the provided folder, then make sure it's in full Racket and have a line like (provide test-constant test-function) at the top for the identifiers you want to make available in the student's file and/or test.rkt. Add the name of the file to (modules ...) in options.rkt and you're all set.

One thing to be careful of is to make your provided names unique. If students happen to also call one of their constants "test-constant," you can get troublesome double define errors. Try doing something like (provide test-constant-cs135 test-function-cs135) to prevent this.

Working with Racket languages

You should be aware of how the various language levels and their function libraries interact across files. The test.rkt file is always evaluated in the same language level as the assignment, i.e. what is specified in the options.rkt file. You will not be able to call things like sort and local on Assignment 2, when the class is still in Beginning Student! However, the function (require …) is always available no matter which level of Racket you’re in, which is exactly what using the (modules …) line in options.rkt does. When you require another Racket module, that file is made visible to your working file, but you can only call on the identifiers which have been "revealed" in a (provide …) line.

It is recommended to work in full Racket for any provided modules you write. You can get this by selecting “Determine language from source” in DrRacket and having a “#lang racket” line at the top of your file. You’ll be able to use all the important functions from the teaching languages, plus lots of other useful functions. There won’t be any conflicts stemming from the language set in options.rkt, because a Racket file calls functions from a provided module with their definitions “hidden,” - the language level in your provided file operates independently of the language level where its functions are called. For example, you can have a provided file like this if the language is Beginning Student for the assignment:

(provide my-sort)

(define (my-sort lst pred) (sort lst pred))

Now, we haven’t talked about the disallowed functions from disallowed.rkt, but this actually doesn’t introduce any new issues. A disallowed.rkt file is used like any other module, and included at the top of the student’s submission file. It does not check for the banned functions in your provided files, even if those functions are called from within the student’s solution. However, in general, we try to have the disallowed file be the last module in the (modules …) line, because if it bans require, the testing will break once it tries to include any subsequent modules using require.

There are slight differences in full Racket that the test writer should get to know when making provided files. An important one is that structs are opaque (or private, if you’ve done OOP) by default, and you must add #:transparent if you want to access the fields.

Another one is that full Racket actually recognizes types (even though variables are not typed), so be especially careful working with floats: (= 3.0 3) is true, but (equal? 3.0 3) is false in full Racket! Even (equal? (list 1.0 2.0 3.0) (list 1.0 2.0 3)) is false! In test.rkt, the result and expected are compared using equal? in whatever teaching language is used, so you probably won’t have to worry about this. Incidentally, #i3 in the teaching languages is equivalent to what 3.0 does in full Racket, so you can have it go the other way where (equal? #i3 3.0) is true in full Racket but false in Beginning Student (because the 3.0 gets treated as the exact value, 3).

Common Issues

Many of these sections will be labeled with an error that is displayed after running rst. Not all of these errors are fully understood. As a last resort, you can always contact the Computer Science Computing Facility (CSCF) for help. Please keep in mind that provided files are written in full racket.

 Required module lib.rkt not found;

The test script cannot find the library needed. It may be that the file name was misspelled in the options file or not named correctly in the provided. Or, the file may just be missing from the provided. It can actually still run properly with the error message, if the student submitted a file of the same name into their folder.

 Disallowed function some-fun called

This one is straight forward. If you are stuck check all files for use of the disallowed function. This includes the submitted file, the test.rkt file and all files used from provided.

 Evaluator creation error: FAILED - Error creating evaluator, [insert error message] 

There are many different things that can lead to this issue. The best thing to do is read the entire error message; it's usually a long and scary block of characters followed by a small helpful message at the very end. The list bellow are causes and fixes from most likely to least likely.

  • The submission is requiring a library that is not in the list of modules in the options.rkt file. To fix this add the file's name into the modules line in the options file for that question.

  • The submission may be using a banned special form. Some special forms will give the disallowed function error message and some will give a funny error message. Double check the allowed use of lambda and local from the assignment specifications and the what the disallowed.rkt file states.

  • One of the provided files may be requiring a standard racket library. It is unknown why this causes problems. It also seems that there is only an issue for some racket libraries. If it is something just for fun, like a drawing library, then comment out any code related to it and make functions that use it just return empty. Contact the instructor or CSCF if this is not a valid solution.

  • One of the provided libraries may be requiring another file in the provided directory. This error has to do with the load order of the modules. The fix that has been shown to work is to not do this. As a workaround you can copy the code into one big file and provide that as a module. Or, check the order that you specified your modules in the (modules ...) line.

Test Constants and Helper Functions

When a student has to define a structure in their submitted file it is hard to write test constants and helper functions for that structure. One way to do this is to make your own version of the structure and convert everything into vectors. Suppose the students are asked to write some function that outputs a list of cells. The students must also define the cell struct in their submitted file. This means that you cannot use the cell structure in any provided files.

To get around this follow the example provided in the appendix. Any functions or constants that are provided have a random string after them. This is done to avoid naming conflicts with students code. In full racket structs must be transparent if you want to access the fields. Next, define functions that access a structs fields by converting it to a vector. This is the most important part because it allows for fields of an undefined struct to be accessed. For convenience, they are wrapped by a function with the same name that struct cell would have. With this you can now define any test constants and helper functions needed. The constant input1_645 could be a possible input to the students function, with output1_645 as the expected output. The test.rkt file may look something like the following.

(result (list-equiv?_645 (student-fn input1_645)
(output1_645)))
(expected true)

Remember that any helper files you write will be in full racket. Also make sure not to use functions that are disallowed.

Example file

for test constants and helper functions.

#lang racket 

(provide list-equiv?_645 input1_645 output1_645) 

(define-struct my-cell (x y used?) #:transparent) 
;; A Cell is a (make-my-cell Nat Nat Bool) 

;; Defining these functions allows you to directly transfer any code 
;; in your solutions into this file 
(define make-cell make-my-cell) 
(define (cell-x c) (vector-ref (struct->vector c) 1)) 
(define (cell-y c) (vector-ref (struct->vector c) 2)) 
(define (cell-used? c) (vector-ref (struct->vector c) 3)) 

(define c1 (make-cell 0 0 false)) 
(define c2 (make-cell 1 0 false)) 
(define c3 (make-cell 2 0 false)) 
(define c4 (make-cell 1 0 true)) 
(define c5 (make-cell 2 0 true)) 

(define input1_645 (list c1 c2 c3)) 
(define output1_645 (list c1 c4 c5)) 

;; In practice this function will be comparing a cell defined in this file 
;; with a cell define in a studens file. 

(define (cell-equiv? cell1 cell2) 
 (and (equal? (cell-x cell1) (cell-x cell2)) 
      (equal? (cell-y cell1) (cell-y cell2)) 
      (equal? (cell-used? cell1) (cell-used? cell2)))) 

;; (list-equiv? lst1 lst2) determines if two lists contain all of the 
;; same elements regardless of order. 
;; list-equiv?: (listof Cell) (listof Cell) -> Bool 
;; This function will only work if abstract list functions are allowed 

(define (list-equiv?_645 lst1 lst2) 
 (local 
    [;; like member?, but comparing using cell-equiv? instead of equal? 
    (define (cell-mem? cell lst) 
      (ormap (lambda (c) (cell-equiv? cell c)) lst))]
      (and (= (length lst1) (length lst2)) 
              (andmap (lambda (c) (cell-mem? c lst2)) lst1) 
              (andmap (lambda (c) (cell-mem? c lst1)) lst2))))

  • examplelib.rkt: This is a file containing the example at the bottom of the page

Alternative solutions

I could not get the above method to work in my term; it seems like you would have to do define-syntax, not just define, on all the functions associated with cell. Otherwise, the student’s file will crash when it tries to call things like cell-x on something that is a my-cell and not a cell.

If the assignment has reached Intermediate Student level when functions can be used as values, you can define your own version of the struct, plus a function which converts your struct into the student’s struct, using the constructor and your struct as arguments. For example:

(define (cast-cell ctor other-cell)
  (ctor (my-cell-x other-cell)
        (my-cell-y other-cell)
        (my-cell-used? other-cell)))

(cast-cell make-cell test-cell-1)

If you are working with recursive definitions like trees, use abstract list functions to help write your cast function! Hopefully, if you are still in Beginning Student, the structures are simple enough that it’s quick to just write them directly in test.rkt.

Miscellaneous notes

Why is there a message "Evaluator creation error: FAILED - Error creating evaluator, Timeout of 60 seconds reached; assuming infinite recursion." when the function clearly should not time out?

Be careful when banning string functions. For some reason, what should say "FAILED - Error creating evaluator, Disallowed function reverse called" becomes a timeout message when the function called format is not allowed. Make sure to allow format to prevent this issue; it's a fairly trivial function which seems to be used by the autotesting scripts.

How would I account for students who already have a (require ...) line because it's actually part of the assignment?

This most likely happens when we provide a library (like 2htdp/image) that the students need to do the questions. Simply make sure a copy of the provided file is in our test.0/provided/ folder and add the name of the file to the (modules ...) line, so it's made available.

If the students are allowed to require their own files, it gets more complicated. In order to use their version of the required file, you do not put anything in test.0/provided/ and only add the name to the (modules ...) line. In general, allowing students to require their own files is not recommended, as it can cause issues with names being re-defined when running the check-testcases script. Be careful asking for similar types of questions to be submitted in different files in this case, as they may have names repeated.

Can students keep their check-expects in their files to get marks for test case coverage without interfering with correctness?

Although DrRacket is an interpreted (actually, JIT-compiled) language, there is still some "compilation" that takes place. The application performs an initial syntax check for things like missing brackets without running anything. Then, it runs the code sequentially, interrupting mid-execution if an error pops up. Finally, check-expect is a special form which is only evaluated after these first two steps are completed. In short, the correctness tests will run fine even if there are incorrect check-expects in the file, because the functions can be called as long as they've been defined at some point. Of course, the tests will still fail if the check-expect has an important character missing or calls a function that the student didn't even write.

How do I fix and re-run test cases if TA marking has already started?

The fastest way is probably to use the set_marks_rst command to re-assign the marks on MarkUs. You can create a folder separate from test.pt and test.0, like test.1.fix_q2_test_002, with only the Q2 folder present to make rst/distrst run faster. Then you can use set_marks_rst with the test.1.fix_q2_test_002.AUTOTESTRESULTS folder targeted.

How can I allow list outputs to be considered correct regardless of order?

There are lots of ways to do this. The simplest is to use sort or quicksort on both the student's output and the expected output, and compare with equal?. You can provide a my-sort or my-quicksort if they aren't allowed, which hides the call to sort/quicksort in a provided file. If you like ALFs, you can check that the student's list is a subset of the expected list using andmap and member?. Then do it again vice-versa, because two sets that are subsets of each other are necessarily equivalent. Finally, the "true" way to store unordered sets is with Racket's hash table type; you can convert both lists into hash tables using hash or make-hash (i.e. value-frequency pairs), and simply compare using equal? (it will ignore ordering once it recognizes you are comparing hash tables).

-- DustinFirman - 2017-12-21

Topic attachments
I Attachment Action Size Date Who Comment
Unknown file formatrkt examplelib.rkt manage 1.6 K 2017-12-21 - 10:12 DustinFirman This is a file containing the example at the bottom of the page
Topic revision: r7 - 2018-12-20 - BillLiu
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback