How to Run Shadow on RIPPLE

Shadow used to have some issues with running on RIPPLE machines, but these days, the constraints are fairly standard for any Shadow install. The first thing to keep in mind is the system and user limits, which you should check yourself before starting any experiments, and ask Lori to increase them if any are too low. Another limit not currently documented there is the maximum number of processes the Linux kernel can run (since tids are really just pids wearing a disguise, that means threads), which is about 4 million for x86-64, so you'll need to keep the scale low enough you don't run into that limit.

Do not run Shadow experiments from your RIPPLE home directory. The /home directory on RIPPLE is network-backed, and Shadow experiments do a lot of file IO, so for the sake of experiment performance and the sake of everyone else using RIPPLE, find someplace else on the file system to run from. Since any experiment that isn't extremely tiny will use far more RAM than storage anyway, a good option is to make a directory in /dev/shm and run it from there to get all the performance benefit of a RAM disk (make a new directory even if /dev/shm itself is currently empty, processes use that folder for IPC so it can suddenly fill with files at any moment). Do note that since /dev/shm is volatile memory, once the machine reboots, all those files will be lost. The typical workflow is therefore: generate the network on persistent storage (home is fine for this step if need be, though slow), copy the experiment(s) to the folder in /dev/shm, run the experiment(s), archive and copy the results to home. You can automatically do that last part by adding something like

&& tar caf $EXPERIMENT_NAME.tar.xz $EXPERIMENT_PATH && mv $EXPERIMENT_NAME.tar.xz ~/.

to the end of your shadow command. You may also want to export XZ_OPT='-T0' first if you do this, replacing 0 with the number of threads you want to use while compressing (0 sets it to the number of CPU threads).

As per usual for long-running experiments, you'll want to do all this in a screen or tmux session. Experiments can be run in parallel in different shells in the session, likely using the numactl command to bind the different experiments to the CPUs and memory of the different NUMA nodes (if the machine you're running on doesn't have multiple NUMA nodes, you can follow Shadow's parallel simulation docs instead, though numactl should still work for this if it's installed). There are many ways to organize this, but a decent one is to set the numbers of the bindings in environment variables exported to a particular associated shell, so that, e.g., all the experiments that will use NUMA nodes 0 and 1 run in screen shell 0, all the experiments that will use NUMA nodes 2 and 3 run in screen shell 2, etc. You can also run experiments on different machines, but only if you keep in mind that process behaviour (e.g., tor) can differ based on OS, libc version, and installed RAM. This means you should only mix-and-match experiments that were run on identically configured machines.