Linux: Assigning IRQ's and Scheduling Processes to Different CPUS

Tim Brecht, November 2002.

Note: these are preliminary test results.

In web servers that are trying to operate under overload conditions a situation occurs where the server's queues that track TCP SYNs and the listenq become full. When this happens incoming TCP SYN packets interrupt the kernel, and it ends up dropping the packet.

Below I've conducted 3 experiments labelled:

1. APIC any with Server any.
   (Interrupts are handled by any processor and the server runs on any processor.)
2. APIC 0 with Server any.
   (Interrupts are handled by any processor 0 and the server runs on any processor.)
3. APIC 0 with Server 1.
   (Interrupts are handled by any processor 0 and the server runs on processor 1.)

The tools that enable these types of assignments on Linux are:

• /proc/irq/*/smp_affinity
  which is a bitmap inidicating which processor handles each interrupt
• /proc/interrupts
  Shows how many interrupts of each type have been handled by each processor.
• Processor Affinity Module cpu_affinity-0.2.2.tgz
  Allows the cpu scheduling mask to be set so that the scheduler only runs the given process on the subset of processors indicted in the mask.

The theory was that if it was only the interrupt processing overhead that was causing the decrease in throughput, assigning one processor to handle all of the interrupts and the other to run the web server should permit performance to flatten out as the request rate increases.

The results show that pinnning the interrupts to processor 0 help performance significantly. As can be seen in the graph below. However, pinning the interrrupts to processor 0 and the server to processor 1 does not result in the flatten out performance that we'd hoped for.