Amortized Analysis

In your first data structures course, you learned how to analyze data structures using either the worst-case or the average-case per query paradigm. For instance, you may have seen the following results:

Where the last two rows are examples of data structures that are analyzed using the average-case per query paradigm. The worst-case analysis for balanced binary search trees is $\Theta(\text{height of tree})$ for all three operations.

In this lecture, we will learn how to analyze data structures using the amortized analysis paradigm. In amortized analysis, we analyze the average cost of a sequence of operations, rather than the cost of a single operation. So, while the worst-case per query might be high for one particular query, the amortized cost of a sequence of operations might be low, provided that most queries have low cost.

Remark 1: Amortized analysis is a worst-case analysis, but it is a worst-case analysis of a sequence of operations, rather than a single operation.

Remark 2: data structures with great amortized performance are ideal for internal processes, such as graph algorithms (for instance finding minimum spanning trees). However, these data structures are not so great in the client-server model (such as internet queries), where the user is waiting for a response. In the latter setting, we want to guarantee that the worst-case response time per query is low.

Types of Amortized Analysis

There are three types of amortized analysis:

1. Aggregate Analysis: We analyze the total cost $T(n)$ of a sequence of $n$ operations, and then divide by the number of operations. Thus, in this case the amortized cost of a sequence of $n$ operations is $T(n)/n$.

2. Accounting Method: We assign a certain charge to each operation (independent of the actual cost of the operation). In the accounting method, we must show that the total charges to a sequence of operations is an upper bound on the total true cost of the sequence of operations. The idea is that if the cost of an operation is cheaper than the charge, then we can use the extra charge to pay for more expensive (later) operations.

3. Potential Method: We come up with a potential function that maps the state of the data structure to a real number. Each operation will then cahnge the state of the data structure, and thus change the potential function. The amortized cost of an operation is then the true cost of the operation plus the change in the potential function.

Example: Binary Counter

We will use the binary counter data structure as an example.

Input: A binary counter $C$ of $r$ bits, initially set to $0$.

Output: increment this counter up to the integer $n$, where $n < 2^r$.

The counter $C$ is represented as an array of $r$ bits, where $C[i]$ is the $i$th bit of the counter. The counter is incremented by the following procedure:

increment($C$): for each bit $C[i]$ of $C$, starting from the least significant bit, do the following: if $C[i] = 0$, then set $C[i] = 1$ and stop; otherwise, set $C[i] = 0$ and continue to the next bit.

The question of interest is: how many bit operations does it take to increment the counter from $0$ to $n$?

Note that, per query, the worst-case cost of incrementing the counter from $0$ to $n$ is $\Theta(\log n)$, since we may have to flip $\log n$ bits of the counter. Thus, an upper bound on the total cost of incrementing the counter from $0$ to $n$ is $O(n \log n)$.

However, upon closer inspection, we see that the most significant bits of the counter are flipped less frequently than the least significant bits. So, is the above analysis tight?

Aggregate Analysis

With aggregate analysis, we analyze the total cost of incrementing the counter from $0$ to $n$, and then divide by $n$. How can we count the total number of bit flips? Let $c_i$ be the number of bit flips that occur when the counter is incremented from $i-1$ to $i$ (that is, the cost of the $i^{th}$ operation). Then, the total number of bit flips is $T(n) = \sum_{i=1}^n c_i$. However, this way of counting the total number of bit flips is not very useful, since we don’t know how to compute $c_i$.

Now, let $b_k$ be the total number of of times that the $k$th bit of the counter is flipped when we increment the counter from $0$ to $n$. Then, $T(n) = \sum_{k=1}^r b_k$, since the total number of bit flips is the sum of the number of times that each bit is flipped.

Since the $k$th bit is flipped once every $2^k$ increments, we have $b_k = \lfloor n/2^{k-1} \rfloor$.

Hence,
$$T(n) = \sum_{i=1}^r \lfloor n/2^{k-1} \rfloor \leq \sum_{j = 0}^\infty n/2^j = 2n.$$

Since $T(n) \geq n$, as we always flip a bit, we have $T(n) = \Theta(n)$.

Accounting Method

In the accounting method, we assign a charge to each operation. In the accounting method, we will assign a charge $\gamma_i$ to the $i^{th}$ operation, so that $$\sum_{i=1}^m \gamma_i \geq \sum_{i=1}^m c_i, \ \forall m \in \mathbb{N}$$ where $c_i$ is the true cost of the $i^{th}$ operation.

Thus, the total charge is an upper bound on the total true cost of the sequence of operations.

How do we assign charges? Let us charge the cost of “clearing a bit” (i.e. changing the bit from $1$ to $0$) to the operation that sets the bit (i.e. changes the bit from $0$ to $1$). Thus, we have $\gamma_i = 2$ for each iteration $i$. The reason for this is that in each iteration, we only set exactly one bit from $0$ to $1$, and we must clear all the bits to the right of this bit.

Thus, if at the end of the $i^{th}$ iteration we have flipped $k$ bits, then we have already charged $k-1$ of the bit flips to earlier operations. This is because to change $k$ bits, we must have cleared $k-1$ bits (for the carry over to the next bit). So, instead of charging $k$ bit flips to the $i^{th}$ operation, we will charge exactly $2$ to this operation, as we must finally set one bit to $1$.

Note that since we have charged the clearing beforehand, and we have charged $2$ for setting the last bit to $1$, we still have “more charge” left over. Thus the above analysis shows that the total charge is always an upper bound on the total true cost of the sequence of operations.

In the end, we have $T(n) \leq \sum_{i=1}^n \gamma_i = 2n$, and we are done again.

Potential Method

In the potential method, we define a potential function $\Phi$ that maps the state of the data structure to a real number. Then, the amortized cost of operation $i$, denoted by $\gamma_i$, is defined as $$\gamma_i = c_i + \Phi(D_i) - \Phi(D_{i-1})$$ where $c_i$ is the true cost of the $i^{th}$ operation, and $D_i$ is the state of the data structure after the $i^{th}$ operation. Hence, the total amortized cost of a sequence of $n$ operations is $$\sum_{i=1}^n \gamma_i = \sum_{i=1}^n (c_i + \Phi(D_i) - \Phi(D_{i-1})) = \Phi(D_n) - \Phi(D_0) + \sum_{i=1}^n c_i .$$ Thus, so long as $\Phi(D_m) - \Phi(D_0)$ for all $m \in \mathbb{N}$, we have that the total amortized cost is an upper bound on the total true cost.

Remark 3: a potential function $\Phi$ which satisfies $\Phi(D_m) - \Phi(D_0) \geq 0$ for all $m \in \mathbb{N}$ is called a valid potential function.

For the binary counter, we define the potential function $\Phi(D_i) := $ number of bits with value $1$ at step $i$. Note that $\Phi(D_0) = 0$ and $\Phi(D_m) \geq 1$ for all $m > 0$. Hence, $\Phi$ is a valid potential function.

Amortized cost of the $i^{th}$ operation:

• $c_i = (\text{# bits } 0 \rightarrow 1) + (\text{# bits } 1 \rightarrow 0) $
• $ \Phi(D_i) - \Phi(D_{i-1}) = (\text{# bits } 0 \rightarrow 1) - ( \text{# bits } 1 \rightarrow 0)$
• $\gamma_i = c_i + \Phi(D_i) - \Phi(D_{i-1}) = 2 \times (\text{# bits } 0 \rightarrow 1) = 2$

Hence, the total amortized cost is $2n$, and we are done again.