Scripts Available for Banning Functions

There are four different scripts available for banning functions on assignments. Two are whitelist scripts, and two are blacklist scripts. Note that the two scripts written by Daniel Holtby are designed to be used together and is the highly reccommended option to use when banning functions.

Daniel Holtby’s Scripts

Daniel Holtby’s Whitelist->Blacklist Script

A script that takes in the list of allowed functions in the preamble of the assignment (permitted-functions.rkt provided by the instructor in charge of the assignment) and turns that into a blacklist.

You need four files in total to run the script. The four files are:

• disallow-srcloc.rkt

• language-defs.rktd

• make-blacklist.rkt

• permitted-functions.rkt (Get this file from the instructor in charge of the assignment)

How to Use the Script

Step 1: Put all four files into one directory

Step 2: Open the make-blacklist.rkt file

Step 3: Add the line (whitelist->blacklist 'beginner-abbr "permitted-functions.rkt"). It doesn’t have to be beginner-abbr. The ISA should set it to be the language level that the assignment specifies. The value of the first parameter should be the language level that the assignment specifies (specified as a symbol), and the value of the second parameter should be the racket file that contains all permitted functions that the assignment specifies (specified as a string). The different language levels you can use and how they should be typed in can be found in the language-defs.rktd file.

Step 4: Verify that all the functions you want students to be allowed to use are placed in the permitted-functions.rkt file. Any built-in functions not placed in this file, will automatically be considered a disallowed function by the end of this process, based on the language level specified in Step 3.

Step 5: To run make-blacklist.rkt, type "racket make-blacklist.rkt" in your terminal and it will print a (disallow ...) expression to be put into a disallowed.rkt file for the relevant question.

Step 6: Copy the printed disallow expression and paste it into disallow-srcloc.rkt. ISAs can then save this new file as disallowed.rkt to use in basic/correctness tests.

If there are any bugs, please send the details about the bugs the instructor Daniel Holtby’s email so that he can fix it.

Daniel Holtby’s Blacklist Script

This is the instructor Daniel Holtby’s version of the Blacklist script that scans a student’s entire file and will fail them if they end up using any of the functions listed on the blacklist.

This script should be used in conjunction with the whitelist->blacklist script.

If there are any bugs, please send the details about the bugs to the instructor Daniel Holtby’s email so that he can fix it.

Format of disallowed.rkt

For every assignment there will be a file containing a list of banned functions and special forms, most likely disallowed.rkt. It uses the define-syntax or "macro" feature of full Racket to make any banned functions throw an error when they are called. The file has a similar form to below.

(define-syntax disallow 
    Some-Racket-Code  ) 

(disallow  ;require 
           lambda 
           local 
           if 
           ;append 
           ...) 

As you can see, the function append has a semi-colon before it, meaning it is commented out. This means that the this function is not included in the disallowed list and the students can use it freely in their solutions. The special form local is in the disallowed list so this means that students will get an error if they use it.

If you want to ban functions that are not listed in the file simply add it to the list, preferably in the correct place alphabetically. There may be cases where adding some special forms will not work. In the past, cond has presented issues. A solution for cond would be to copy the below lines of code exactly as is, including the "...", adding it just below the #lang racket line.

(provide cond) 
(define-syntax-rule (cond ...) (error "cond is disallowed"))

This may also work with other special forms not in the disallowed list, but it has not been tested.

Tips for Banning Functions

• The first tip is only applicable when a whitelist (permitted-functions.rkt) is not provided to the ISAs (e.g. asking instructors which functions should be banned. No need to ask if a whitelist is provided). The rest of the tips should be applicable whenever disallowed.rkt is used.

• If you believe there are functions that should be banned, ask the instructor before banning them. It may be that the instructor was unaware of a potential solution involving built in functions. It may also be the case that the instructor believes a solution using the built in function is acceptable; for example using string->number and string-ith to deal with digits of a number, when strings haven't been explicitly allowed on the assignment.

• Adding a function to the disallowed list will disallow any direct calls to it in the student's submission file or test.rkt. For example, if you wanted to ban the function "and" because they must implement it with nested conds, you would not be allowed to use "and" directly in test.rkt. See the "Working with Racket languages'' section for workarounds to this.

• If the instructors want to give the students a library to use, make sure that “require” is commented out so they can work with it without having to copy and paste.

• A very useful thing to do is make different disallowed files for different questions. It is common to ban or allow functions for specific questions. The way to do this is copy the disallowed file, rename it for the question, for example disallowed3c.rkt and keep it in the provided directory. Then change disallowed3c.rkt to follow the assignment specifications. Lastly, make sure to change the module in the options.rkt of the question to include the correct version of the disallowed file.

• When disallowing the "/" function, this will only disallow the function from being used for division, but using fraction literals will still be allowed (Ex. (/ 4 7) will be disallowed but 4/7 will not).

• When using disallowed.rkt, it will check the whole file and is not able to only check a specific portion of a file. This means that when you have multiple question parts contained in one file, they should all have the same disallowed file.

• When the function, "member?", is permitted and in the permitted-functions.rkt file, then the function "member" is automatically permitted as well when make-blacklist.rkt is run. However, when "member" is permitted, "member?" will not automatically be permitted.

• It may be wise to allow the use of the "require" function regularly, even though it is not often on the permitted list of functions given by instructors. This is because students may require certain libraries like htdp-trace to help them debug their code and we do not want students losing all their marks because they forgot to remove it.

• Be careful when disallowing functions. If it is not in the right language level, then it is possible that a student may unknowingly define a function with the same name as a built-in function and if that name is contained in the disallowed.rkt file, they will be told the function they have defined is disallowed.

Cameron John Morland’s Whitelist Python Script

A whitelist script that takes in the list of allowed functions in the preamble of the assignment (permitted-functions.rkt) and ensures that students will fail basic/correctness tests if they use functions that are not on the whitelist script.

Multiple copies of Cameron John Morland’s whitelist script can be created for an assignment to whitelist different questions.

How the Script Works

Please see the documentation for the script.

How to Use and Set the Script Up

Step 1: Get the permitted-functions.rkt file from the instructor in charge of the assignment

Step 2: Put the permitted-functions.rkt file in the racket-whitelist-scripts directory in test.pt and test.0

Step 3: Change the options.rkt file in the "in" directory. Remove the disallowed.rkt file in the module and add the line (instructor-script "racket-whitelist-scripts/check-racket-whitelist.sh")

Possible Problems and Bugs

The example test suite below has an example of whitelisting Racket functions. In this example, Q1 and Q2 merely demonstrates how to use the instructor-script option in options.rkt. Q3 actually runs the Python scripts to check for the whitelisted functions listed in permitted-functions.rkt.

You can find the most up to date script here https://git.uwaterloo.ca/cjmorland/rackettools. If there are any bugs with the script, Please send the details about the bugs to the instructor Cameron John Morland’s email so that he can fix it.

Disallowed-old.rkt

This is the old version of the blacklist script that will only catch banned functions if our test cases ends up running the section of the code with the banned function

This script is error prone due to the fact that the basic tests (which are very basic in nature) will most likely not end up running the section of the code that contains the banned function

This script can be useful in certain cases (e.g. one subpart of a question bans a certain function, but the other subparts of the same question do not. In this case, you do not want to use Dan’s (dank) disallowed file since it will scan the entire file for the banned function and will fail all of the subparts for it being used. Be sure to document this example).

-- Yuying Li - 2020-12-23

Effect on Students using Banned Functions (Applicable to disallowed-old.rkt ONLY)

A final thing to note is what happens when a student uses a function in the disallowed list. The way the disallow works is that it redefines any functions in the list to throw an error. This means that if a students code does not run the function they will not lose any marks on that test. For example, let's say a students solution looked like the following.

(define (student-fn x)
  (cond
    [(even? x) (banned-function x)]
    [(odd? x) (something-ok x)])) 

In this case the student would get half the marks, assuming half of the tests used odd numbers. This may or may not be the intended result. One argument is that the deduction they get for using the banned function is fair. Another argument is that they should lose all marks since the assignment fully specified the problem. Be sure to check with the instructor of the course on what marking scheme should be used. If using the first one, then no additional work is required. If the instructor would prefer to dock all marks for using disallowed functions then you will need to find all the students that have used it, even if they only failed one test case from it. One way to do this would be to read the error message the student receives when failing a test from a disallowed function. Finaly use the following command in the directory with the test results to get all of the students who have used the disallowed function.

grep -r 'error message of test'