Lecture 5 - Universal Circuits & Computing First-Order Partial Derivatives
Universal Circuits
In the same way that in Boolean complexity theory, we have the notion of universal Turing machines, in algebraic complexity theory, we have the notion of universal circuits. Universal circuits are circuits that can simulate any circuit of a given size. We now give a formal definition of universal circuits.
Definition 1 (Universal Circuits): A circuit $\Phi$ is said to be a universal circuit for $n$ inputs and $n$ output circuits of size $s$, computing forms of degree $d$, if the following holds:
For any $n$ forms $f_1(x_1, \dots, x_n), \dots, f_n(x_1, \dots, x_n)$ of degree at most $d$ that can be simultaneously computed by a circuit of size at most $s$, there is a circuit $\Psi$ computing the forms $f_1, \dots, f_n$, such that the computation graph of $\Psi$ is the same as the computation graph of $\Phi$.
In other words, $\Psi$ is a projection of $\Phi$.
By the results above on efficient homogenization, it is enough to construct universal circuits for homogeneous circuits computing forms.
In order prove the existence of efficient universal circuits, it is good to first put a bit more structure on the circuits we are considering. This will be done by considering circuits in normal-homogeneous form.
Definition 2 (Normal-Homogeneous Form): A homogeneous circuit $\Psi$ is said to be in normal-homogeneous form if the following holds:
- All inputs are labeled by a variable
- All edges leaving an input gate are connected to sum gates
- Output gates are sum gates
- Non-input gates are alternating: that is, a product gate is connected to a sum gate, and a sum gate is connected to a product gate.
- The fan-in of each product gate is exactly $2$. (we do not restrict fan-in of sum gates)
- The fan-out of each addition gate is at most $1$.
We will first show that any homogeneous circuit can be efficiently transformed into a circuit in normal-homogeneous form.
Lemma 1: For any homogeneous circuit $\Gamma$ of size $s$, there is a circuit $\Psi$ in normal-homogeneous form of size $O(s)$ that computes all forms in any gate of $\Gamma$.
We are now ready to prove that we can efficiently construct universal circuits.
Theorem 3 (Universal Circuits): For any integers $s \geq n \geq 1$ and $d \geq 1$, we can construct in $\mathrm{poly}(s,d)$ time a circuit $\Phi$ in normal-homogeneous form of size $O(d^4s)$ that is universal for $n$ input and $n$ output circuits of size at most $s$ computing forms of degree at most $d$.
Proof: We
Computing First-Order Partial Derivatives
We will now prove the following seminal result in algebraic complexity theory, due to Baur and Strassen, which is also known as backpropagation in Machine Learning.
Theorem 4 (Baur-Strassen): Let $f(x_1, \dots, x_n)$ be a polynomial that can be computed by an algebraic circuit of size $s$ and depth $\Delta$. Then, $f$ and the first-order partial derivatives of $f$ can be computed by an algebraic circuit of size $5s$ and depth $O(\Delta)$.
The main idea is to note that the chain rule for differentiation can be efficiently implemented in algebraic circuits, since each gate in the circuit has fanin at most $2$.
Proof: We prove this theorem by induction on the size of the circuit computing $f$.
Base case: If $f$ is a single variable, then the partial derivative of $f$ is simply $1$. If $f$ is a constant, then the partial derivative of $f$ is $0$. Both of these can be computed by a circuit of size $5$ and depth $1$.
Inductive step: Suppose that the theorem holds for all circuits of size at most $s$. Let $\Phi$ be a circuit of size $s+1$ computing $f$. Label the gates of $\Phi$ as $g_1, \dots, g_{s+1}$ in the order in which they are computed.