Linear Programming Relaxations and Rounding
Many important optimization problems are NP-hard. A prominent example is the Integer Linear Programming (ILP) problem, which is defined as follows: $$ \min c^T x$$ $$ \text{subject to } Ax \leq b, x \in \mathbb{N}^n$$ where $A \in \mathbb{R}^{m \times n}$, $b \in \mathbb{R}^m$, and $c \in \mathbb{R}^n$ are given. The goal is to find an integer vector $x$ that minimizes the objective function while satisfying the constraints.
One advantage of ILPs is that they can model a wide range of combinatorial optimization problems. However, this very same property makes them computationally intractable in general. But we do know how to solve linear programming (LP) problems efficiently. So, a natural question is: can we get good solutions when we relax the integrality constraints from an ILP and solve the corresponding LP instead? Can we round the solution we obtained from the LP to get a good solution for the ILP?
If we manage to do the above, we would be getting “pretty good” solutions for a wide range of combinatorial optimization problems. This is the idea behind the method of linear programming relaxations and rounding: we are content with approximately optimal solutions that we can find efficiently.
Here is a high-level overview of the method:
-
Formulate the combinatorial optimization problem as an ILP. (say by minimizing some objective function)
-
Relax the integrality constraints to get an LP. (This is called the linear programming relaxation of the ILP.)
2.1. We are still minimizing the same objective function, but over a (potentially) larger feasible region. Hence $$ opt(LP) \leq opt(ILP)$$
-
Solve the LP to get a fractional solution. This can be done efficiently.
3.1. If the fractional solution is already integral, we are done, as it will be a solution to the ILP.
3.2. Otherwise, we need to round the fractional solution to get an integral solution. This is the tricky part. In this case, we need to design a rounding algorithm that transforms the fractional solution into an integral solution while preserving the objective function value as much as possible. Thus, we would like to find a rounding algorithm and a value $\alpha \geq 1$ such that $$ opt(LP) \leq \text{ value of the rounded solution} \leq \alpha \cdot opt(LP) \leq \alpha \cdot opt(ILP).$$
Example: Vertex Cover
Given a weighted graph $G = (V, E, w)$, where $w : E \to \mathbb{R}_+$, we would like to find a minimum weight vertex cover. That is, we want to find a set $S \subseteq V$ such that for every edge ${u, v} \in E$, at least one of $u$ or $v$ is in $S$, and the total weight of the vertices in $S$ is minimized.
Following our strategy, we can formulate the problem as an ILP: $$ \min \sum_{v \in V} w(v) x_v$$ $$ \text{subject to } \quad x_u + x_v \geq 1 \quad \text{ for every edge } {u, v} \in E$$ $$ x_v \in \{0, 1\} \quad \text{ for every } v \in V$$
We can interpret the above ILP as follows: $x_v = 1$ if and only if vertex $v$ is in the vertex cover $S$. The inequality constraints ensure that for every edge ${u, v}$, at least one of $u$ or $v$ is in the vertex cover. The objective function is the total weight of the vertices in the vertex cover, which we want to minimize.
The linear programming relaxation of the above ILP is obtained by relaxing the integrality constraints on $x_v$. This gives us the following LP: $$ \min \sum_{v \in V} w(v) x_v$$ $$ \text{subject to } \quad x_u + x_v \geq 1 \quad \text{ for every edge } {u, v} \in E$$ $$ 0 \leq x_v \leq 1 \quad \text{ for every } v \in V$$
Now we use an efficient LP solver to solve the above LP. Let $z$ the the optimal solution we obtain. Now we need to devise a rounding algorithm that transforms the fractional solution $z$ into an integral solution $S$.
In this case, we can use the following simple rounding algorithm: round each $z_v$ to the nearest integer. Let $y_v$ be the rounded value of $z_v$. Then, $y_v = 1$ if $z_v \geq 1/2$, and $y_v = 0$ otherwise. Note that $y_v \leq 2 z_v$, for every vertex $v \in V$. Moreover, we can see that $y$ encodes a vertex cover: for every edge ${u, v}$, since $z_u + z_v \geq 1$, we have $y_u + y_v \geq 1$, which means that at least one of $u$ or $v$ is in the vertex cover encoded by $y$.
Finally, we can analyze the approximation guarantee of the above rounding algorithm. The cost of the solution given by $y$ is: $$ \sum_{v \in V} w(v) y_v \leq 2 \sum_{v \in V} w(v) z_v \leq 2 \cdot opt(LP) \leq 2 \cdot opt(ILP).$$
Thus, we have obtained a 2-approximation algorithm for the vertex cover problem using linear programming relaxations and rounding.
Main Example: Set Cover
The set cover problem is a classic combinatorial optimization problem, generalizing the vertex cover problem.
- Input: A finite set $U$ (the universe) and a collection $\mathcal{S} = \{S_1, S_2, \ldots, S_n\}$ of subsets of $U$.
- Output: A minimum-size subcollection $\mathcal{C} \subseteq \mathcal{S}$ such that $$\bigcup_{S \in \mathcal{C}} S = U.$$
We could also have a weighted version of the set cover problem, where each set $S_i$ has a non-negative weight $w_i$ associated with it, and the goal is to minimize the total weight of the sets in the subcollection $\mathcal{C}$.
Let us now formulate the weighted set cover problem as an ILP:
$$ \min \sum_{i=1}^n w_i x_i$$ $$ \text{subject to } \quad \sum_{i: v \in S_i} x_i \geq 1 \quad \text{ for every } v \in U$$ $$ x_i \in \{0, 1\} \quad \text{ for every } i = 1, 2, \ldots, n$$
The above ILP can be interpreted as follows: $x_i = 1$ if and only if set $S_i$ is in the subcollection $\mathcal{C}$. The inequality constraints ensure that every element $v \in U$ is covered by at least one set in $\mathcal{C}$. The objective function is the total weight of the sets in the subcollection $\mathcal{C}$, which we want to minimize.
Now we can proceed with our method of linear programming relaxations and rounding. The linear programming relaxation of the above ILP is obtained by relaxing the integrality constraints on $x_i$. With this, we get the following LP:
$$ \min \sum_{i=1}^n w_i x_i$$ $$ \text{subject to } \quad \sum_{i: v \in S_i} x_i \geq 1 \quad \text{ for every } v \in U$$ $$ 0 \leq x_i \leq 1 \quad \text{ for every } i = 1, 2, \ldots, n$$ Suppose we solve the above LP and obtain an optimal solution $z$. We now need to come up with a rounding algorithm that transforms the fractional solution $z$ into an integral solution $\mathcal{C}$.
Can we just round each $z_i$ to the nearest integer as we did in the vertex cover example? Not really. Say a given vertex $v$ is an element of 20 of the sets in $\mathcal{S}$, and the optimal solution $z$ assigns a value of $1/20$ to each of these sets. Then the above rounding algorithm would round each of these values to 0, which is not good, as it would not cover $v$.
Instead, we can think of $z_i$ as the “probability” that we would pick set $S_i$ in an optimal solution. This way, $z$ is describing a set of “optimal probability distributions” over each of the sets in $\mathcal{S}$. Given this interpretation, we can use the following randomized rounding algorithm:
Algorithm 1: Random Pick
- Input: A fractional solution $z$ to the LP relaxation of the weighted set cover problem.
- Output: A subcollection $\mathcal{C}$ of sets in $\mathcal{S}$. (which hopefully covers $U$)
- Set $\mathcal{C} = \emptyset$.
- For $i = 1, 2, \ldots, n$:
- Pick set $S_i$ with probability $z_i$.
- If $S_i$ is picked, add it to $\mathcal{C}$.
- Return $\mathcal{C}$.
Note that the expected weight of the subcollection $\mathcal{C}$ output by the above algorithm is exactly the value of the fractional solution $z$. But will be $\mathcal{C}$ be a set cover?
Let us consider the Random Pick algorithm from the perspective of an element $v \in U$. Let $v \in S_1, S_2, \ldots, S_k$ (for simplicity) and $v \notin S_{k+1}, S_{k+2}, \ldots, S_n$. As long as we select at least one of $S_1, S_2, \ldots, S_k$, we are good (with respect to $v$). Note that we select set $S_i$ with probability $z_i$, and we know that $$ \sum_{i =1}^k z_i \geq 1,$$ as $z$ is a feasible solution to the LP. What is the probability that $v$ is covered by Random Pick? It is definitely not 1, as you can see from the case where $k=2$ and $z_1 = z_2 = 1/2$, for instance.
The following lemma gives us a bound on the probability that an element $v \in U$ is covered by the Random Pick algorithm:
Lemma 1 (Probability of covering an element): in a sequence of $k$ independent experiments, in which the $i^{th}$ experiment succeeds with probability $p_i$, and $$ \sum_{i=1}^k p_i \geq 1,$$ the probability that at least one of the experiments succeeds is at least $1 - 1/e$.
Proof: The probability that no experiment succeeds is $$ \prod_{i=1}^k (1 - p_i) \leq e^{-\sum_{i=1}^k p_i} \leq e^{-1}.$$
Thus, the probability that at least one experiment succeeds is at least $1 - 1/e$.
By the above lemma, we see that the probability that an element $v \in U$ is covered by the Random Pick algorithm is at least $1 - 1/e$. But we could still have many elements that are not covered by $\mathcal{C}$. How do we deal with this? By perseverance!
Algorithm 2: Randomized Rounding
- Input: A fractional solution $z$ to the LP relaxation of the weighted set cover problem.
- Output: A subcollection $\mathcal{C}$ of sets in $\mathcal{S}$ that covers $U$.
- Set $\mathcal{C} = \emptyset$.
- While there is an uncovered element $v \in U$:
- $\mathcal{C} = \mathcal{C} \cup \text{Random Pick}(z)$.
- Return $\mathcal{C}$.
The Randomized Rounding algorithm repeatedly applies the Random Pick algorithm until all elements in $U$ are covered. To show that the above is a good algorithm, we need to show that we will not execute the Random Pick algorithm too many times (with high probability). This is captured by the following lemma:
Lemma 2 (Probability decay): Let $t \in \mathbb{N}$. The probability that the Randomized Rounding algorithm executes the Random Pick algorithm more than $t + \ln(|U|)$ times is at most $e^{-t}$.
Proof: The probability that the Randomized Rounding algorithm executes the Random Pick algorithm more than $t + \ln(|U|)$ times is the probability that there is an uncovered element after $t + \ln(|U|)$ iterations.
Let $v \in U$. For each iteration, by Lemma 1, the probability that $v$ is covered is at least $1 - 1/e$. Hence, the probability that $v$ is not covered after $t + \ln(|U|)$ iterations is at most $$1/e^{\ln(|U|) + t} = \dfrac{1}{|U|} \cdot e^{-t}.$$
By the union bound, the probability that there is an uncovered element after $t + \ln(|U|)$ iterations is at most $e^{-t}$.
Now that we know that we wil cover $U$ with high probability, we need to bound the cost of the solution we came up with. Suppose that the Randomized Rounding algorithm executes the Random Pick algorithm $T$ times. Let $X$ be the total weight of the subcollection $\mathcal{C}$ output by the Randomized Rounding algorithm.
At each implementation of the Random Pick process, the expected weight of the subcollection $\mathcal{C}$ output is exactly the value of the fractional solution $z$. That is $\sum_{i=1}^n w_i z_i$.
After $T$ calls to Random Pick, expected total weight of the subcollection $\mathcal{C}$ is at most $$ \mathbb{E}[X] = T \cdot \sum_{i=1}^n w_i z_i$$ By Markov: $$ \mathbb{P}[X \geq 4 \cdot \mathbb{E}[X]] \leq 1/4.$$
We are now ready to get a bound on the cost of the solution output by the Randomized Rounding algorithm:
Lemma 3 (Cost of Rounding): Given a fractional solution $z$ to the LP relaxation of the weighted set cover problem, the Randomized Rounding algorithm outputs, with probability at least $0.7$, a set cover $\mathcal{C}$ of weight at most $4 \cdot (\ln(|U|) + 3) \cdot OPT(ILP)$.
Proof: Let $T := \lceil \ln(|U|) + 3 \rceil$. By Lemma 2, there is a probability of at most $e^{-3} < 0.05$ that the Randomized Rounding algorithm executes the Random Pick algorithm more than $T$ times. After $T$ calls to Random Pick, the expected total weight of the subcollection $\mathcal{C}$ is $T \cdot \sum_{i=1}^n w_i z_i \leq T \cdot OPT(ILP)$. By Markov, the probability that the total weight of the subcollection $\mathcal{C}$ is at least $4 \cdot T \cdot OPT(ILP)$ is at most $1/4$. By the union bound, with probability at most $1/4 + 0.05 = 0.3$, the algorithm either makes more than $T$ calls to Random Pick or our solution has weight $> 4 T \cdot OPT(ILP)$.
Thus, with probability $\geq 0.7$ we stop at $T$ iterations and construct a solution with weight at most $4 \cdot T \cdot OPT(ILP)$.