Winter2010ScriptTestingFeedback < ISG

ISG Web>ISGScripts>Testing>ImprovePit>Winter2010ScriptTestingFeedback (2011-04-29, HumaZafar) (raw view)
%TOC%

This page is meant for feedback about ISG's scripts at the end of Winter 2010
to help ensure that the modified versions are ready to go in Spring 2010.
The goal is to catch any potential errors now before they manifest after the
start of the Spring term.


---+ Outstanding errors

---++ Missing packages on linux006.student.cs

Utilities complain about the following missing packages on the new Linux systems:
   * perl-5.8.1
   * bzip2-1.0
   * cups-1.2.X
   * javajdk-1.5
   * lynx-2.8.5
   * mfcf-misc
   * mutt-1.4
   * python-2.5
   * tar-1.19
   * tetex-1.0
   * wget-1.8
   * xfig-3.2


---++ Run errors

The following use cases that used to work no longer work on both cpu-solaris.student.cs
and linux006.student.cs.






---+++ The RST family of commands no longer accepts whitespace-delimited glob patterns

<pre>> ~isg/test/distrst a05 m 0 'bc2cui m66zhang binzhou' isg1                                                                                     
Reading configuration files........ completed.                                                                                                 
>                                                                                                                                              
</pre>

This just hangs for a while and then terminates without even producing a results directory.

The =rst= command instead appears only to execute test the first of the specified ids.

This should be fixable, but it is nontrivial.

---+++ The =distrst= command fails when given a =-s= option with RSTVersionTwo semantics

Unfortunately, there is insufficient time to track down this error.
The default ("test all students") still functions correctly.


---++ Unexpected cross-system behaviour

The following use cases have one type of behaviour on cpu-solaris.student.cs
but a different behaviour on linux006.student.cs.


---+++ Login cannot differentiate users on linux

Another consequence of rsh vs. ssh; ssh does not provide the =REMOTEUSERS= environment variable.

This is both good and bad; for relatively novice users, auto-personalized configuration would be good.
A discussion of how to provide this via an alternate command is taking place at RT #72227:
https://www.cs.uwaterloo.ca/cscf/internal/request/UpdateRequest?72227




---+ Errors presumed fixed
---+++ RST dies when trying to run tests

---++++ Phase 1: rsh is ssh on linux

<pre>**** bc2cui ********************************************************************                                                               
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                                                                                    
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @                                                                                    
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                                                                                    
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!                                                                                          
Someone could be eavesdropping on you right now (man-in-the-middle attack)!                                                                    
It is also possible that the RSA host key has just been changed.                                                                               
The fingerprint for the RSA key sent by the remote host is                                                                                     
31:46:98:f6:bd:d0:ee:7f:c1:f1:ca:71:04:48:bb:92.                                                                                               
Please contact your system administrator.                                                                                                      
Add correct host key in /u3/cs135/.ssh/known_hosts to get rid of this message.                                                                 
Offending key in /u3/cs135/.ssh/known_hosts:1                                                                                                  
RSA host key for localhost has changed and you have requested strict checking.                                                                 
Host key verification failed.                                                                                                                  
ERROR: aborting on line 404 of rst                                                                                                             
</pre>

This is an artifact of the way =rst= connects to the testing account by default:
<pre>rsh localhost -l <em>test-ac</em> ...</pre>

Recent modifications making this configurable mean that it should be safe to change the default from =rsh= to =ssh=, 
explicitly passing the flag that tells ssh not to care about RSA keys on =localhost= (which will not be available via =rsh= on Solaris).


---++++ Phase 2: Linux rejects ssh keys of <em>new</em> testing accounts

*PLEASE NOTE THAT Solaris machines should not be used for connecting to course accounts as of Spring 2011.*

The ssh-key distribution mechanism that CSCF supplies should be helping to ensure a smooth transition from using =rsh= to using =ssh=.

However, while Solaris is capable of using ssh to connect to course accounts:

<pre>[cs135@cpu18]:~$ ssh -o NoHostAuthenticationForLocalhost=yes -o NumberOfPasswordPrompts=0 localhost -l cs135t                                 
Last login: Fri Apr 23 13:17:31 2010 from localhost
Terminal type is xterm-color
cpu18:~> hostname
cpu18.student.cs
</pre>

it appears that, for some reason, the Linux machines are *not*:

<pre>[cs135@linux006]:~$ ssh -o NoHostAuthenticationForLocalhost=yes -o NumberOfPasswordPrompts=0 localhost -l cs135t
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
[cs135@linux006]:~$ hostname
linux006.student.cs
</pre>

There is no clear reason for this at the present time.  Fixing this issue is critical to allowing testing to happen on Linux servers.
If =ssh= cannot be used, another option that would provide stability improvements is =sudo=; however, this would have to be managed by CSCF on every single server instead of globally like =ssh= is.

<b>NB:</b> This problem may only be isolated to the new testing accounts.  If the following lines are added to the =.rstrc= file of the =cs115= account:

<pre>testac=cs115-t
testgrp=cs115-t
</pre>

then RST functions correctly.


---+++ Test account access via rsh denied on Solaris machines

For example, running rst on the cs135 account:

<pre>Reading configuration files........ completed.                                  
Using run_id isg1                                                               
**** bc2cui ********************************************************************            
rshd: cpu08.student.cs: cs135: Permission denied to become cs135t; using        
jqpublic                                                                        
cpu08.student.cs: jqpublic shell:                                               
/u2/isg/test/rst-files/testacc/testSubmission                                   
/tmp/.cs135.a10.t.0.isg1/bc2cui/11253QtaO.v                                     
/u3/cs135/marking/a10/test.0/runTests /u3/cs135/marking/a10/test.0              
/tmp/.cs135.a10.t.0.isg1/bc2cui/11220niaG6v                                     
/tmp/.cs135.a10.t.0.isg1/bc2cui/11215g5a45v bc2cui a10 cs135: Permission        
denied                                                                          
ERROR: aborting on line 235 of /u2/isg/test/rst-files/rst-common (via line      
50 of rst_impl) with exit status 204                                            
</pre>


The likely explanation is that <em><tt>localhost cs135</tt></em> is not in the =.rhosts= file of =cs135t=,
which is a requirement for RST to operate in the traditional fashion.

<pre>$ rsh localhost -l cs135-t whoami
cs135-t
$ rsh localhost -l cs135t whoami
rshd: cpu18.student.cs: cs135: Permission denied to become cs135t; using jqpublic
cpu18.student.cs: jqpublic shell: whoami: Permission denied
$ rsh localhost -l cs135-t fgrep cs135 /u/cs135-t/.rhosts
localhost cs135
$ 
</pre>

The auto-propagation tool discussed at CF.CourseAccountSshKeyAccess should then be run on
userids from =.rhosts=, allowing ssh access from course accounts to testing accounts as
well.  This ssh access is *required* from Linux systems, as rsh is really ssh.



---+++ moss_makereadable perl errors


From =~cs136/mossoutput/test.sh=:

<pre>#!/bin/sh

# This will confuse(?) makereadable
perl -e '$count=0; while (1) { print "r\nRENAMED" . ++$count . "\n"; }' | /u/isg/test/runMoss pA10P1 "-l c -d"
# Seems like this does not
# yes d | /u/isg/test/runMoss pA10P1 "-l c -d"

cd html

/u/isg/test/moss_makereadable ./index.html 5 > readableIndex.html</pre>

The errors from the =moss_makereadable= line are:
<pre>Use of uninitialized value $person2 in concatenation (.) or string at /org/cs/u2/isg/test/moss_scripts/moss_fixuseridshelper.pl line 65, <> line 44.
Use of uninitialized value $person2 in concatenation (.) or string at /org/cs/u2/isg/test/moss_scripts/moss_fixuseridshelper.pl line 65, <> line 63.
Use of uninitialized value $person2 in concatenation (.) or string at /org/cs/u2/isg/test/moss_scripts/moss_fixuseridshelper.pl line 65, <> line 75.
Use of uninitialized value $person2 in concatenation (.) or string at /org/cs/u2/isg/test/moss_scripts/moss_fixuseridshelper.pl line 65, <> line 89.
Use of uninitialized value $person2 in concatenation (.) or string at /org/cs/u2/isg/test/moss_scripts/moss_fixuseridshelper.pl line 65, <> line 139.
Use of uninitialized value $person2 in concatenation (.) or string at /org/cs/u2/isg/test/moss_scripts/moss_fixuseridshelper.pl line 65, <> line 186.
Use of uninitialized value $person2 in concatenation (.) or string at /org/cs/u2/isg/test/moss_scripts/moss_fixuseridshelper.pl line 65, <> line 215.
Use of uninitialized value $person2 in concatenation (.) or string at /org/cs/u2/isg/test/moss_scripts/moss_fixuseridshelper.pl line 65, <> line 244.
</pre>


---+++ Public test runners fail to parse filenames properly on linux systems

This appears to be a backward-incompatibility with =bash-3.0=; the Linux systems run v4 flavours of bash.

An installation of =bash-3.0= has fixed this problem.



---+++ RST dies when generating postscripts on linux006

This is due to a difference in the behaviour of the version of dvips in the =tetex-1.0= package and the default on linux006.

An installation of =tetex-1.0= should fix this problem.  There is no such package on Linux for now, so a very unstable hackaround has been concocted that allows it to appear to be working properly on both Solaris 8 and Ubuntu 9.10.



---+++ Errors appear at the end of an RST run

<pre>Running Scheme computeMarks postprocessor from directory /u2/isg/bittersuite3/languages/scheme/intermediate-lambda                             
computeMarks complete.                                                                                                                         
sh: /tmp/.cs135.a10.t.0.isg1/bc2cui/9038CEaWPr/OUTPUT.txt: cannot create                                                                       
sh: /tmp/.cs135.a10.t.0.isg1/bc2cui/9038CEaWPr/OUTPUT.txt: cannot create                                                                       
sh: /tmp/.cs135.a10.t.0.isg1/bc2cui/9038CEaWPr/OUTPUT.txt: cannot create                                                                       
sh: /tmp/.cs135.a10.t.0.isg1/bc2cui/9038CEaWPr/OUTPUT.txt: cannot create                                                                       
sh: /tmp/.cs135.a10.t.0.isg1/bc2cui/9038CEaWPr/OUTPUT.txt: cannot create                                                                       
</pre>

However, the =OUTPUT.txt= file still appears in the save directory.

The problem was that this was a bad mix of assumptions in RST, BitterSuite, and MarkUs;
the =OUTPUT.txt= was a _previously generated_ output file instead of something up to date.
This has now been corrected.


---++ Packages now on Ubuntu 9.10 systems

These packages are available via the new software installations on the =cs_build= account:

   * bash-3.0


---+ Issues that are not errors

---++ Distrst cannot run because there is no =test_servers= array

<pre>> ~isg/test/distrst a05 m 0 '[ bc2cui m66zhang binzhou ]' isg1                                                                                 
Reading configuration files........ completed.                                                                                                 
Error: distrst cannot be run                                                                                                                   
The test_servers array must be populated in an appropriate .rstrc file                                                                         
/u2/isg/test/rst-files/distributed-rst/distrst_impl: line 144: [: -ne: unary operator expected                                                 
Usage: distrst-master.py <assign> <type> <suite> <students> <run_id> <logdir> <student_glob> <submitdir_glob> <allowed_host1> [...             
<allowed_hostN>] [<RSToption1>...<RSToptionN>]                                                                                                 
ERROR: aborting on line 149 of /u2/isg/test/rst-files/distributed-rst/distrst_impl with exit status 255                                        
</pre>

In an environment with multiple architectures with multiple versions of software, the =distrst= command cannot make any assumptions about which servers it should run on and which it should avoid.

As a result, it requires the specification of the =test_servers= array in the course/assignment/suite-specific =.rstrc= file.

This array is also read by the public test runners to determine where code should be run.
Topic revision: r15 - 2011-04-29 - HumaZafar
ISG Web
ISG Web Home
- Changes
- Index
- Search
Webs
- AIMAS
- CERAS
- CF
- CrySP
- External
- Faqtest
- HCI
- Himrod
- ISG
- Main
- Multicore
- Sandbox
- TWiki
- TestNewSandbox
- TestWebS
- UW
My links
- People
- CERAS
- WatForm
- Tetherless lab
- Ubuntu Main.HowTo
- eDocs
- RGG NE notes
- RGG
- CS infrastructure
- Grad images
Edit
Instructional Support Group, David R. Cheriton School of Computer Science, University of Waterloo