Lecture 1: Computational Models & Complexity Classes
In this course, we will study the following important questions:
- What is computation?
- What is a computer?
- What are the fundamental limits of computation?
In other words, what problems can be solved by a computer and what problems cannot be solved by a computer?
- What are the fundamental limits of efficient computation?
In other words, what problems can be solved by a computer with a reasonable amount of resources? Common resources include time, space, randomness, and interaction with an oracle.
To be able to address the above questions, we will need to establish more mathematically precise definitions for the terms “computation”, “computer”, “problem”, what it means to “solve” a problem, and what it means for a computer to be “efficient”.
Computation can be defined as any process which produces an output from a given input. The object carrying out the computation is called a computer. A problem is a question or task that we want to decide or solve: given an input, we want to decide whether it has a certain property (or we want to find a solution to the desired task starting from the input).
The above definitions are quite general and encompass a wide range of processes, but they still do not allow us to work with it mathematically. Hence, to be able to formally define a computer, and thus characterize computation, we need to specify how to represent the input, the output, what operations the computer can perform, and what does it mean for the computer to solve a problem.
Languages
To be able to describe problems and tasks, we need to introduce a language to represent them.
An alphabet is a finite set of predetermined symbols. The following proposition shows that we can assume, without loss of generality, that the alphabet is $\{0,1\}$.
Proposition 1: For any finite set $\Sigma$ with $m$ elements, there is a one-to-one encoding function $h : \Sigma \to \{0,1\}^{\lceil \log_2 m \rceil}$.
Proof: Fix an arbitrary ordering of the elements of $\Sigma$, denoted by $\sigma_1,\ldots,\sigma_m$. One possible encoding function is the following: $h(\sigma_i) = bin(i)$, where $bin(i)$ is the binary representation of $i$.
A string is a finite sequence of symbols from an alphabet. As we are assuming from now on that the alphabet is $\{0,1\}$, we will denote the set of all strings over this alphabet by $$\{0,1\}^* := \bigcup_{i \geq 0} \{0,1\}^i.$$
For (most of) this course we will make a simplifying assumption on the types of problems that we will be studying: we will only consider decision problems. These are problems that have a yes/no answer for each given input. We will represent the “yes” answers by the symbol $1$ and the “no” answers by the symbol $0$.
Remark 1: theoretical computer science is not limited to decision problems, and problems other than decision problems are also studied, such as counting problems, optimization problems, etc. We will see some examples of these more general problems later in the course.
Any decision problem divides the set of all strings into two disjoint sets: the set of strings for which the answer is “yes” and the set of strings for which the answer is “no”. Thus, any decision problem $\mathcal{P}$ can be represented by a subset of $L_{\mathcal{P}} \subseteq \{0,1\}^*$. We will take the convention that the set of strings for which the answer is “yes” (or equivalently $1$) is the one that is represented by the subset $L_{\mathcal{P}}$.
More generally, we define a language to be any subset of $\{0,1\}^*$.
For instance, let us consider the problem $\texttt{PAL}$ of deciding whether a given string is a palindrome. The set of strings that are palindromes is $L_{\texttt{PAL}} = \{0,1\}^* \cap \{w \in \{0,1\}^* \mid w = w^R\}$, where $w^R$ denotes the reverse of the string $w$. We will refer to the subset $L_{\texttt{PAL}}$ as the language of palindromes.
Now, as we have defined languages, we can relate any language $\mathcal{L}$ to a function $f_{\mathcal{L}} : \{0,1\}^* \to \{0,1\}$, where $f_{\mathcal{L}}(x) = 1$ if, and only if, $x \in \mathcal{L}$.
Any function $f : \{0,1\}^* \to \{0,1\}$ can be represented by a family of boolean functions $\{f_n : \{0,1\}^n \to \{0,1\} \mid n \geq 0\}$, where $f_n$ is the restriction of $f$ to the set $\{0,1\}^n$. In that manner, we see that a function $f$ corresponds to a language $\mathcal{L}_f := \bigcup_{n \geq 0} f_n^{-1}(1)$.
Cardinality of Languages
Counting arguments (or the probabilistic method) are often used in theoretical computer science to show the limitations of certain computational models.
Before we apply our first counting argument, let us introduce/review the following definitions:
- A set $S$ is said to be finite if there is $n \in \mathbb{N}$ such that there is an injective map $f: S \rightarrow \{1,\ldots,n\}$. Otherwise, $S$ is said to be infinite.
- A set $S$ is said to be countable if there is an injective map $f: S \rightarrow \mathbb{N}$. Otherwise, $S$ is said to be uncountable.
For other equivalent definitions of countability, see this Wikipedia page.
For any $n \in \mathbb{N}$, the set $\{0,1\}^n$ is finite, as there is a bijection between $\{0,1\}^n$ and $\{0, 1, \ldots, 2^n-1\}$.
We will now show that the set of all strings $\{0,1\}^*$ is countable. In the following proof (and for the rest of the course), we will use the notation $|w|$ to denote the length of the string $w$.
Proposition 2: The set $\{0,1\}^*$ is countable.
Proof: Let $f : \{0,1\}^* \to \mathbb{N}$ be the function defined by $f(w) = 2^{|w|} + \sum_{i=1}^{|w|} w_i2^{i-1}$, where $w_i$ is the $i$-th symbol of the string $w$. This function is simply the correspondence between a string $w$ and the positive number corresponding to the binary representation of the string $1w$.
We will now show that the set of all languages is uncountable. The proof of this fact is a standard diagonalization argument, which is a common technique used in theoretical computer science.
Theorem 1: The set of all languages is uncountable.
Proof: Suppose, for the sake of contradiction, that the set of all languages is countable. Then, we can enumerate all languages as $\mathcal{L}_1, \mathcal{L}_2, \ldots$. We will now construct a language $\mathcal{N}$ that is not in this enumeration.
Since the set of all strings $\{0,1\}^*$ is countable, we can enumerate all strings as $w_1, w_2, \ldots$.
Now, we can construct a table $T$ whose rows are indexed by the languages $\mathcal{L}_i$ and whose columns are indexed by the strings $w_j$. The entry $T_{i,j}$ is $1$ if $w_j \in \mathcal{L}_i$ and $0$ otherwise. Note that any language is completely specified by the row corresponding to it in this table.
Now, let $\mathcal{N}$ be the language defined by the following rule: for any string $w_i \in \{0,1\}^*$, enumerated according to the above enumeration of all strings, $\mathcal{N}$ contains $w_i$ if, and only if, $w_i \notin \mathcal{L}_i$. In other words, $w_i \in \mathcal{N}$ if, and only if, $T_{i,i} = 0$.
Now, if the set of all languages is countable, then $\mathcal{N}$ must be in the enumeration, say $\mathcal{N} = \mathcal{L}_k$. However, by the definition of $\mathcal{N}$, we have that $w_k \in \mathcal{N}$ if, and only if, $w_k \notin \mathcal{L}_k$, which is a contradiction. Thus, the set of all languages is uncountable.
Uncomputability
We are now ready to prove our first result on the limitations of computation, even without having defined what a computer is!
Let’s stay with our intuitive notion of what a computer program is: a computer program is a finite sequence of symbols from an alphabet.
Also, let us say that a computer program $\mathcal{P}$ computes a language $\mathcal{L}$ if, for any input $x \in \{0,1\}^*$, the program $\mathcal{P}$ outputs $1$ if $x \in \mathcal{L}$ and $0$ otherwise.
Our first uncomputability result will come from the simple observation that the set of all possible “programs” in a particular language is countable, as it is a subset of the set of all strings, which is countable.
Proposition 3: There is a language $L$ such that $L$ cannot be computed by any Python program.
Proof: Any Python program can be encoded as a string over the alphabet $\{0,1\}$ (its binary representation). Thus, the set of all Python programs is countable. Now, each Python program $\mathcal{P}$ can be associated with a language $L_{\mathcal{P}}$ of the strings for which the program $\mathcal{P}$ outputs $1$. Thus, the set of all languages that can be computed by a Python program is countable.
Now, Theorem 1 tells us that the set of all languages is uncountable. Thus, there must be a language $L$ that cannot be computed by any Python program.
This result is quite surprising, as it shows that there are problems that cannot be solved by a computer, even if we allow the computer to run for an arbitrary amount of time.
Also, note that there is nothing special about Python in the above result: we could have replaced Python with any other programming language, and the result would still hold. In particular, we could have replaced Python with any fixed machine model and the result would still hold, as we only need to be able to describe the encoding of all possible programs in the machine model.
The above result, while elegant and surprising, is still far from satisfactory. Here are some reasons why:
- Non-construtive: it only shows that there is a language that cannot be computed by any Python program, but it does not tell us what this language is, or how to identify such a language.
- Non-universal: the result only tells us that there is a language that cannot be computed by any Python program, but it does not tell us whether there is a language that cannot be computed by any computer program (i.e., any machine model). As of now, it could just be that some problem is undecidable for Python, but it is decidable for some other machine model.
We will address these issues in the next lecture. More precisely, we will show that there is a specific language that cannot be computed by any computer, no matter how we define the computer.
Acknowledgements & References
- This material was based on these lecture notes by Prof. Eric Blais.