RSTPublicTests

NOTE: In August 2014, the name "public test" was changed to "basic test", which is more meaningful for students.

The public tests is a facility that students can use to make sure their code conforms to the assignment specifications. Before the assignment is due, students first submit their code (for example using MarkUs). Then the public test will check the student's code, and email them the results (for example, the student may get an email saying "2/4 tests failed"). Students can repeat this procedure until they pass all the checks.

This page describes how the public tests work and how to fix problems. For information on specific scripts, see ISGScriptsManPages

Related Pages

• Possible Improvements
• MarkSchemesForRSTPublicTests
• WebPublicTestFrontEnd

How Public Tests Work

Making a Public Test Request

Before requesting a public test, you need two things:

• A test suite folder called test.pt, which should be placed in /u/csXXX/marking/aYY/ ( csXXX is your course, and aYY is the assignment the public test is for). Thus the full path is /u/csXXX/marking/aYY/test.pt. For information on what to put in the test.pt folder, see BitterSuite and BitterSuiteWalkthrough.
• The student's files that the public test should check. The files should be placed in the directory /u/csXXX/handin/aYY/questid/ where questid is the Quest ID of the student to run the public test on.

To request a public test, you can run this command from a terminal while logged in to the course account: /u/isg/bin/submit/submit_pub_test_hook course assign questid

• course is the course account name. Example: cs135
• assign is the assignment number. Example: a04
• questid is the student for whom you want to run the public tests for. Example: j99smith

You can run the command from any directory, as long as you're logged in to the course account.

MarkUs automatically runs the submit_pub_test_hook program when students submit. If your course uses submit and you want the public tests to run automatically for every submission, you should consider using the SubmitPublicTestHook.

Submission (for example using MarkUs or submit command) is a separate step from running the public test. In fact, you can skip the submission step and manually put files in the handin folder (for example by drag & drop in Mac Finder window). Then you can run the command above to request a public test. Doing this is useful when you're developing the public test suite ( test.pt folder); it can save time because you don't need to go to MarkUs or use submit.

The submit_pub_test_hook program just runs another program, pub_test_logger, which is located in /u/isg/bin/public_test/student_interaction/. pub_test_logger will create an empty file named csXXX.questid.aYY.request in the directory /u/isg/public_test_request_files/requests/. That is essentially all pub_test_logger does; it just creates this file. This file is used to keep track of public test requests. Another program, the public test runners, will detect that this file has been created, and it will continue the public test procedure.

A student can have only one public test running at a time, for a particular assignment and course. When pub_test_logger is run, it will check if the student already has a file in the requests folder (for the particular assignment and course). If so, then it means there's already a public test running, so pub_test_logger will just exit. This can be a problem if students submit too quickly in MarkUs (see below).

Public Test Runners

At the start of the term, CSCF will launch some public test runners. Public test runners are programs that run forever, waiting for students to request a public test. When a request is made, a file is created in the requests directory, as described earlier. The public test runners repeatedly scan this folder for files; that's how the runners know a public test request has been made.

• When there are no public test requests (ie the requests folder is empty), the runners just "sleep", doing nothing.
• When a public test request appears (ie a file appears in the requests folder), a "valid" public test runner will process that request by ssh into the course account and running ca_pub_test_service. A public test runner is "valid" if it's specified in the course's .rstrc file (see next section). When the runner is done, it'll delete the file in the requests folder.

The ca_pub_test_service script is located in /u2/isg/bin/public_test/back_end/. It's responsible for running the rst command to test the student's code, and emailing the public test results to the student.

The public test runners are launched on the isg user account with the pub_test_launch command, which launches pub_test_runner processes on several different servers. The ISAs should never have to launch public test runners; CSCF or the ISCs will take care of this.

Public test runner is the program /u/isg/bin/public_test/back_end/pub_test_runner. For security reasons, only CSCF and ISCs can view the source code.

You can see if the public test runners are running on a particular server by ssh into the server, and running the Bash command: ps -U isg -u isg u from any directory as any user. If you see a bunch of sleep and bash /u2/isg/bin/public_test/back_end/pub_test_runner_impl, then it means the public test runners are running on that server. pub_test_runner and pub_test_runner_impl are two different files, but pub_test_runner is just a wrapper for pub_test_runner_impl.

In Spring 2014, the public test runners are running on the servers ubuntu1204-004 and ubuntu1204-006. On each server, there are N public test runners running, where N is the number of processors the server has (N=48 for both ubuntu1204-004 and ubuntu1204-006).

If the public test runners stop running for whatever reason, they will automatically start again in about 30 minutes.

Configuring Public Tests ( =.rstrc file)

You can configure the public tests by editing the file /u/csXXX/.rstrc. The .rstrc is a hidden file (it starts with a period), so you may have to enable hidden files in Mac to see it.

The .rstrc file sets some variables; you can read about them in RstRc, but the only one you may want to change is test_servers.

If a public test runner sees a public test request (ie it sees a file in the requests folder), the runner will check if the server it's running on is listed in test_servers. If the server is NOT listed, then the public test runner will ignore the request, and let another public test runner handle the request. Otherwise, it will process the request.

IMPORTANT: The entries in test_servers should be the same as the output of the hostname command. For example:

• ubuntu1204-004.student.cs is incorrect, because it's not what hostname prints
• linux.student.cs is incorrect, because it's not what hostname prints
• ubuntu1204-004 is OK, because it's what hostname prints

Normally, you will list all the servers with public test runners running. For Spring 2014, this is ubuntu1204-004 and ubuntu1204-006. However, if a particular server isn't working right, you can temporarily remove it from test_servers until CSCF fixes the problem (examples are below).

distrst also uses the servers you list in the test_servers variable.

Public Test Logs

You can find logs of public test requests in /u/csXXX/course/publictest/term/ where term is a four digit code for the term. The most significant digit is the century (1), the middle two digits are the last two digits in 20xx, and the last digit is either 1 (Winter), 5 (Spring), or 9 (Fall). For example, for Fall 2014, term should be 1149.

In the /u/csXXX/course/publictest/term/ folder, you will see several log files for each assignment. You can look at the aYY.completedrun.log files, but be careful not to modify them. In the .completedrun.log file, the number at the start of each line is a timestamp, with format MMDDHHMMSS (the first MM is for month, the second MM is for the minute). After the timestamp is the student's quest ID, the server that ran the public test, and the mark for that public test run.

If you need a more detailed log, you can open /u/csXXX/sent which contains all the public test emails sent out to students. Be careful when doing any kind of processing on the sent file, as the file can grow very big.

When Public Tests Stop Working

The public tests may stop working during the term. If that happens, you should first identify where the problem is, and then attempt to fix it. As an ISA, you will not be able to fix all problems, either because you don't have the necessary permissions (ie. no admin/root access) or you don't know how (which is fine). If you cannot fix the problem, you should contact your ISC or CSCF point of contact.

Sometimes students report they cannot receive the public test email. Here are some common causes of this problem:

• The student is not checking his or her email properly. For example:
  • The public test email is in the spam folder.
  • The student's inbox is not sorted by Date, so the student doesn't see the email.
  • The student did not set his or her email correctly in WatIAM.
• Check if submission (MarkUs or submit command) is working. Try submitting something as a student. The submitted files should be saved to the /u/csXXX/handin/aYY/ folder. If the submitted files do not appear there, then the problem is with MarkUs or submit.
• If you're using MarkUs, make sure that the markus user can ssh into the course account, and vice versa. To do this, log into the course account and run ssh markus@linux.student.cs. You should be able to get into the markus account without being asked for a password. As markus user, try ssh'ing into the course account with ssh csXXX@linux.student.cs. This should not ask you for a password, and it should put you back into the course account. If this isn't happening, then you'll need to update the SSH keys ( .ssh/authorized_keys) on the markus and course accounts.
• Sometimes the email server goes down. In the /u/csXXX/.rstrc file, there should be a variable called test_servers which lists the servers the public test uses. For each of the servers, ssh into it and run mailq. If mailq reports an error, then the email server is down. For CS135, students can view the public tests online so this isn't a problem. But for other courses, you should report the problem to CSCF immediatedly.
• If ssh is not working, then there is nothing you (the ISA) can do. You should report the problem to CSCF immediatedly.

If students are receiving the public test email, but the mark is incorrect or there are obscure errors, make sure that your test.pt folder and files are correct (for example, make sure there are no mis-matched brackets in the test.rkt files). Try running the public test manually using rst command from a terminal. If you get errors with rst command, then the public test will not work (since the public test runs rst command).