Time Complexity

Still in preparation. Please check the references at the end of the document.

Time Complexity

Even though a language may be decidable, it may not be feasible to compute it in practice, if the solution requires too many computational resources (such as storage space, time, etc.). Today we will study one kind of computational resource: time. To define the time complexity of a problem, we first need to define the running time of a (single-tape) Turing machine.

The time cost of a Turing machine $M$ on input $x$ is the number of steps that $M$ takes to halt when given input $x$. If $M$ does not halt on input $x$, then the time cost is infinite. From now on, we will only consider the time cost of Turing machines that halt on all inputs.

Now that we have defined the time cost of a Turing machine, we can define its time complexity.

Definition 1: The (worst-case) time complexity of a (single-tape) Turing machine $M$ is the function $t_M:\mathbb{N}\to \mathbb{N}$ defined by $$t_M(n)=\underset{x\in \{0,1{\}}^n}{max}\{\text{time cost of }M\text{ on input }x\}.$$

Now that we have defined the time complexity of a Turing machine, we can define the time complexity of a language.

Definition 2: Let $t:\mathbb{N}\to \mathbb{N}$ be a function. The class $\mathtt{\text{TIME}}(t(n))$ is the set of languages that can be decided by a deterministic, single-tape Turing machine $M$ such that $t_M(n)=O(t(n))$.

From the above definition, we have a number of natural complexity classes:

Observations

Time complexity classes can be defined in terms of other models of computation, such as multi-tape Turing machines, or other models that we have not discussed. Depending on the choice of computational model, the time complexity classes may change. However, the choice of model does not affect the time complexity classes in a significant way.

For instance, if we wanted to define time complexity classes in terms of multi-tape Turing machines, one can show that the time complexity classes that arise are equivalent up to a polynomial factor. This is captured by the following proposition, whose proof can be found in [S13, Theorem 7.8].

Proposition 1: For any function $f:\mathbb{N}\to \mathbb{N}$, where $f(n)\ge n$, any $f(n)$-time multi-tape Turing machine has an equivalent $O(f(n{)}^2)$-time single-tape Turing machine.

Time Hierarchy Theorem

Now that we have defined time complexity classes, it is natural to ask: are these classes distinct from one another? In other words, does giving more time to a Turing machine allow it to solve more problems? As we will now see, the answer is yes. But before we do so, we need one more definition.

Definition 3: A function $f:\mathbb{N}\to \mathbb{N}$, where $f(n)\ge n\log n$, is time-constructible if there exists a Turing machine $M$ that, on input $1^n$, halts in $O(f(n))$ steps and outputs the binary representation of $f(n)$.

Examples of time-constructible functions: $n$, $\lfloor n\sqrt{n}\rfloor$, $n^c$ for any $c\in \mathbb{N}$, $\lfloor n\log n\rfloor$, $2^n$, etc.

Theorem 1: $\mathtt{\text{TIME}}(n)⊊\mathtt{\text{TIME}}(n^6)$.

Proof idea: we will use the fact that the universal Turing machine can be efficiently implemented (i.e., with small overhead) to simulate other Turing machines. With this at hand, together with the observation from last lecture of the principle that a Turing machine cannot predict the behavior of another Turing machine without simulating it, we can prove the theorem.

Proof: Consider the following language: $$A_{TM}^{n^2}:=\{x:=⟨M⟩0^k\mid M\text{ is a TM that accepts }x\text{ in at most }|x{|}^2\text{ steps}\}.$$ We will now show that $A_{TM}^{n^2}\in \mathtt{\text{TIME}}(n^6)$. Consider the following Turing machine $B$:

On input $x$:

1. Let $n=|x|$.
2. Compute $n^2$ (which is possible as $n^2$ is time-constructible) and store it in a binary counter. We will decrement this counter every time we execute a basic instruction of $M$ in step 4. If the counter ever reaches zero, reject.
3. If $x$ is not of the form $⟨M⟩0^k$, reject.
4. Simulate (using the universal Turing machine) $M$ on $x$.
5. If $M$ accepts $x$, accept. If $M$ rejects $x$, reject.

It suffices to show that $B$ is a multi-tape Turing machine which runs in $O(n^3)$ time, as by Proposition 1, this is equivalent to a single-tape Turing machine running in $O(n^6)$ time. We will use 4 tapes to implement $B$, where one tape is the input tape. Let us now describe the other tapes. One of $B$’s tapes will be used to store the binary counter, another tape of $B$ will contain the description of $M$, and the other tape will be used to simulate $M$ on input $x$.

• Since $n^2$ is time-constructible, we can implement the binary counter in $O(n^2)$ time on its tape.
• Checking if $x$ is of the form $⟨M⟩0^k$ can be done in $O(n)$ time.
• Now, let us analyze the simulation of $M$ on input $x$.
  • Each step of $M$ can be simulated in $O(n)$ time, as the description of $M$ has length $\le n$.
  • Since $M$ runs for at most $n^2$ steps (since we will always halt in case $M$ has not decided in $n^2$ steps), the total time to simulate $M$ on input $x$ is $O(n^3)$.
• Step 5 can be done in $O(1)$ time.

Thus, $B$ is a multi-tape Turing machine that runs in $O(n^3)$ time, and hence $A_{TM}^{n^2}\in \mathtt{\text{TIME}}(n^6)$.

Now, to finish the proof of the theorem, we need to show that $A_{TM}^{n^2}\notin \mathtt{\text{TIME}}(n)$. Suppose, for the sake of contradiction, that $A_{TM}^{n^2}\in \mathtt{\text{TIME}}(n)$. Then there exists a Turing machine $D$ that decides $A_{TM}^{n^2}$ in $O(n)$ time. That is, there is a constant $c>1$ such that $D$ decides $A_{TM}^{n^2}$ in at most $cn$ steps.

Consider the following Turing machine $E$:

On input $x$:

1. Let $n=|x|$.
2. If $x$ is not of the form $⟨M⟩0^k$, reject.
3. Run $D$ on input $x$, and do the opposite. That is, if $D$ accepts, reject. If $D$ rejects, accept.

By construction, $E$ runs in time $\le 2cn$. Now, consider the input $y=⟨E⟩0^k$, where $n:=|y|$. For $k$ large enough, we have $2cn\le n^2$. Thus, we know that $E$ runs in time $\le n^2$ on input $y$. By definition of $D$, we have that $D$ correctly decides whether $E$ accepts $y$ in at most $cn$ steps. However, this is a contradiction, as (by construction) $E$ does the opposite of what $D$ decides.

By being more careful with the implementation of each step in the above argument (as well as the implementation of the universal Turing machine), one can show the following improved version of the Time Hierarchy Theorem. A proof can be found in [S13, Theorem 9.10].

Theorem 2 (Full Time Hierarchy Theorem): For any two functions $f,g:\mathbb{N}\to \mathbb{N}$ such that $f(n)=o(g/\log g)$, and $g$ is time-constructible, we have that $$\mathtt{\text{TIME}}(f(n))⊊\mathtt{\text{TIME}}(g(n)).$$

The proof of the above theorem follows the same ideas as the proof of Theorem 1, but with a more careful implementation of the universal Turing machine.

Polynomial Time

Acknowledgements & References

This lecture was based on these resources:

• Lecture notes by Prof. Eric Blais.
• [S13, Sections 7.1, 7.2, 9.1]