Positive Semidefinite Matrices & Semidefinite Programming

Symmetric Matrices & Spectral Theorem

A matrix $A\in {\mathbb{R}}^{n\times n}$ is called symmetric if $A=A^T$. A complex number $\lambda \in \mathbb{C}$ is called an eigenvalue of $A$ if there exists a vector $u\in {\mathbb{C}}^n$ such that $Au=\lambda u$. In this case, $u$ is called an eigenvector of $A$ corresponding to $\lambda$.

Spectral Theorem: If $A$ is a symmetric matrix, then

1. $A$ has $n$ eigenvalues (counted with multiplicity),
2. all eigenvalues of $A$ are real,
3. there exists an orthonormal basis of ${\mathbb{R}}^n$ consisting of eigenvectors of $A$.

In other words, we can write $$A=\sum _{i=1}^n{\lambda }_i{u}_i{u}_i^T,$$ where ${\lambda }_1,\dots ,{\lambda }_n\in \mathbb{R}$ are the eigenvalues of $A$, and $u_1,\dots ,u_n\in {\mathbb{R}}^n$ are the corresponding (orthonormal) eigenvectors.

If a symmetric matrix $A$ has only non-negative eigenvalues, then we say that $A$ is positive semidefinite, and write $A⪰0$. There are several equivalent definitions of positive semidefinite matrices:

1. all eigenvalues of $A$ are non-negative.
2. $A=B^{T}B$ for some matrix $B\in {\mathbb{R}}^{d\times n}$, where $d\le n$. The smallest value of $d$ is the rank of $A$.
3. $x^{T}Ax\ge 0$ for all $x\in {\mathbb{R}}^n$.
4. $A=LD{L}^T$ for some diagonal matrix $D$ with non-negative diagonal entries and some lower triangular matrix $L$ with diagonal elements equal to $1$.
5. $A$ is in the convex hull of the set of rank-one matrices $u{u}^T$ for $u\in {\mathbb{R}}^n$.
6. $A=U^{T}DU$ for some diagonal matrix $D$ with non-negative diagonal entries and some orthonormal matrix $U$.
7. $A$ is symmetric and all principal minors of $A$ are non-negative. Here, by principal minors we mean the determinants of the submatrices of $A$ obtained by deleting the same set of rows and columns.

Semidefinite Programming

Let ${\mathcal{S}}^m:={\mathcal{S}}^m(\mathbb{R})$ denote the set of $m\times m$ real symmetric matrices.

A semidefinite program (SDP) is an optimization problem of the form $$\begin{array}{ll}\text{minimize}& ⟨C,X⟩\\ \text{subject to}& ⟨A_i,X⟩=b_i,i=1,\dots ,m\\ & X⪰0,\end{array}$$ where $C,A_1,\dots ,A_m\in {\mathcal{S}}^n$ and $b_1,\dots ,b_m\in \mathbb{R}$. Moreover, $⟨A,B⟩:=\text{tr}(A^{T}B)$ is the trace inner product.

We can write and SDP in a way similar to a linear program as follows: $$\begin{array}{ll}\text{minimize}& c^{T}x\\ \text{subject to}& A_1{x}_1+\cdots +A_n{x}_n⪰B\\ & x\in {\mathbb{R}}^n,\end{array}$$ where $A_1,\dots ,A_n,B\in {\mathcal{S}}^m$ and $c\in {\mathbb{R}}^n$, and we use $C⪰D$ to denote that $C-D⪰0$.

If the matrices $A_i,B$ are diagonal matrices, then the SDP is equivalent to a linear program. Thus, we see that SDPs generalize linear programs.

In a similar way to linear programs, the following are important structural and algorithmic questions for SDPs:

1. When is a given SDP feasible? That is, is there a solution to the constraints at all?
2. When is a given SDP bounded? Is there a minimum? Is it achievable? If so, how can we find it?
3. Can we characterize optimality?
   • How can we know that a given solution is optimal?
   • Do the optimal solutions have a nice description?
   • Do the solutions have small bit complexity?
4. How can we solve SDPs efficiently?

To understand better these questions and the structure of SDPs, we will need to learn a bit about convex algebraic geometry.

Convex Algebraic Geometry

Spectrahedra

To understand the geometry of SDPs, we will need to understand their feasible regions, which are called spectrahedra and are described by Linear Matrix Inequalities (LMIs).

Definition 1 (Linear Matrix Inequality (LMI)): An LMI is an inequality of the form $$A_0+\sum _{i=1}^n{A}_i{x}_i⪰0,$$ where $A_0,\dots ,A_n\in {\mathcal{S}}^m$.

Definition 2 (Spectrahedron): A spectrahedron is a set of the form $$S=x\in {\mathbb{R}}^n:A_0+\sum _{i=1}^n{A}_i{x}_i⪰0,$$ where $A_0,\dots ,A_n\in {\mathcal{S}}^m$.

Note that spectrahedra are convex sets, since they are defined by LMIs, which are convex constraints. Moreover, several important convex sets are spectrahedra, including all polyhedra, circles/spheres, hyperbola, (sections of) elliptic curves, among others.

When considering SDPs, it is enough to work with a more general class of convex sets, which we call spectrahedral shadows. Spectrahedral shadows are simply projections of spectrahedra onto lower-dimensional spaces.

Testing Membership in Spectrahedra

To be able to solve SDPs efficiently, a first step is to be able to test membership in spectrahedra efficiently. That is, given a spectrahedron $S=x\in {\mathbb{R}}^n:A_0+\sum _{i=1}^n{A}_i{x}_i⪰0$ and a point $x\in {\mathbb{R}}^n$, we want to determine whether $x\in S$. Since $x\in S$ if and only if $A_0+\sum _{i=1}^n{A}_i{x}_i⪰0$, this is equivalent to testing whether a given symmetric matrix is positive semidefinite.

More succinctly, we have the following decision problem:

• Input: symmetric matrix $A\in {\mathcal{S}}^m$
• Output: YES if $A⪰0$, NO otherwise.

An efficient algorithm for this problem is the symmetric gaussian elimination algorithm, which runs in time $O(m^3)$. The algorithm will proceed just as in gaussian elimination, by performing elementary row operations (without row swapping) to reduce $A$ to an upper triangular form. However, in this case, every time we apply a row operation, which can be encoded by a lower unitriangular matrix $L$, we also apply the same operation to the columns of $A$ by right-multiplying $A$ by $L^T$. This way, we ensure that $A$ remains symmetric throughout the process.

As the product of lower (or upper) unitriangular matrices is again a lower (or upper) unitriangular matrix, we can see that the symmetric gaussian elimination algorithm will always output a diagonal matrix $D$ with non-negative diagonal entries iff $A⪰0$.

To see that $D⪰0\iff A⪰0$, first note that $D=LA{L}^T$, where $L$ is a lower unitriangular matrix. Now, $A⪰0\iff z^{T}Az\ge 0$ for all $z\in {\mathbb{R}}^m\iff (L^{T}z{)}^{T}D(L^{T}z)\ge 0$ for all $z\in {\mathbb{R}}^m\iff D⪰0$.

The above proves that our algorithm is correct, and the running time is $O(m^3)$, since we need to perform $m^2$ elementary row operations, each of which takes $O(m)$ time.

Application: Control Theory

not required material - to be written here later - please see slides and references for this part

SDPs are used in many areas of mathematics and engineering, including control theory, combinatorial optimization, and quantum information theory. Today we will see an application of SDPs to control theory, in particular to the problem of stabilizing a linear, discrete-time dynamical system.