1 - Computing Models and The Power of Writing
1 - Computing Models and The Power of Writing
1 - Computing Models and The Power of Writing
1
Part I
Deterministic Algorithms and
Machines
2
1 Language Recognition
Let be a …nite set. We can use the elements of as symbols of an alphabet,
and construct …nite strings using these symbols. Let be the set of all …nite
strings that can be constructed with the symbols of : We can use those in…nitely
many strings to encode inputs of algorithms. One of the most basic algorithmic
tasks related to those codes is to distinguish between the codes that hold some
speci…c property and the codes that do not hold this property.
Problem 2 Recognition(L)
Input: w 2 :
Problem: decide if w 2 L:
We focus on the above type of problems, which are called decision problems.
Then, when we think in constructing an abstract model of a computing machine,
we think in devices that deal with decision problems.
we have that
w 1 = k1 k2 km and w2 = n1 n2 nm :
3
We get that the computation of any computable function can be reduced to
an appropriate decision problem.
pal = w 2 f0; 1g : w = wR ;
where wR denotes the reverse of w (if w = w [1] w [m] ; then wR = w [m] w [1]).
Our …rst goal is to design a model of automata that can solve the problem
Recognition(pal) : The goal does not seem to be very demanding. Then, we
could begin trying with the most elementary devices.
References
[1] M. Sipser. Introduction to the Theory of Computation. PWS Publishing,
Boston, 1997.
4
2 Finite State Automata
At the beginning there was only read-once automata...
Finite-state machines are a simple type of automata that is studied in au-
tomata theory and the theory of computation. This model was introduced by
W. McCulloch, W. Pitts who though of it as a combinatorial model of human
neurons and neural activity [1].
5
2. Q is a …nite set of inner states used to save information about the char-
acters that have been read. If M is in state q after reading the string w;
then q corresponds to the summary taken by M about the text w:
3. q0 2 Q is the initial state. When M is ready to start the scanning of w;
this automaton should be in a state that represents the fact that M has
not read a single character, this state is the initial state q0 :
4. A Q is the set of accepting states.
5. is the transition function of M, which is a function from the set Q
into the set Q: Notice that an element of Q is a pair (q; a) that can
represent the data that can be accessed by the automaton in a given instant
of the computation: the character a that is written on the cell that is
being read, and the state q that is written on its inner memory at this
time instant. On the other hand, an element of Q is a state p that can
represent the e¤ ect exerted by those data on the computation: a change
in the content of the inner memory that switches from the state q to the
state p:
1. At instant 1 the string w is written on the input tape, the i-th character
is written on the i-th cell. The scanning head is placed on the …rst cell,
and the inner state is equal to q0 :
2. At instant 2 the scanning head is placed on cell 2; and the inner state has
changed to (q0 ; w [1]) :
3. At instant i the scanning head is placed on cell i: The inner state, at this
time instant, is equal to b (q0 ; w [1; :::; i]) ; where b is the extension of to
the set Q that is de…ned by the following recursion:
For all q 2 Q and for all a 2 the equality b (q; a) = (q; a) holds.
n
For all q 2 Q and for all u 2 the equality
holds.
4. At instant jwj + 1 the scanning head is placed on the …rst empty cell. This
is the halting condition for M, which is always reached after jwj + 1 steps.
The state reached by the automaton is the state b (q0 ; w) : We say that M
accepts the input, if and only if, state b (q0 ; w) belongs to A:
6
1. Finite state automata cannot write. Those automata can only read and
save a bounded amount of information from its readings (at most log (jQj)
bits).
2. Finite state automata are read-once devices, that is: the i-th cell is scanned
only once, at the i-th instant of the computation.
We ask: which are the languages that can be recognized by …nite state
automata?
We say that a language L is a regular language, if and only if, there exists M
such that L = L (M) :
We use the symbol REG to denote the set of all regular languages. The set
REG is closed under several operations.
Exercise 8 Prove that REG is closed under unions, intersections and taking
complements.
7
Exercise 10 De…ne the star of L as the language
We have to conclude, given the above facts, that REG is a large set that
could contain many more languages that we though at a …rst glance. Does REG
contain the language pal?
1. b q0 ; wwR 2 A:
2. uwR 2
= pal:
3. b q0 ; wwR = b q0 ; uwR (see the exercise below).
References
[1] W. McCulloch, W. Pitts. A Logical Calculus of the Ideas Immanent in Ner-
vous Activity. Bulletin of Mathematical Biophysics. 5 (4): 115 - 133, 1943.
8
3 Myhill-Nerode
Let v; w be two strings and suppose that the equality b (q0 ; v) = b (q0 ; w)
holds; we get that for all string X the equality b (q0 ; vX) = b (q0 ; wX) also
holds, and we get that for all string X the condition
holds.
Proof. We have already proved one of the two implications. Let us focus on
the second one. Thus, suppose that SL is a …nite-index relation. Let QSL be
the …nite set SL : De…ne MSL as QSL ; ; q0SL ; ASL ; SL ; where:
1. State q0SL is equal to ["]SL ; where " is the empty string and given w 2
the symbol [w]SL denotes the equivalence class of w:
2. The set of accepting states is the set
ASL = [w]SL : w 2 L :
9
3. The transition function of MSL is de…ned by the equation
SL [w]SL ; a = [wa]SL :
References
[1] A. Nerode. Linear Automaton Transformations. Proceedings of the AMS, 9,
JSTOR 2033204, 1958.
10
4 2DFA, Shepherdson
And then Michael Rabin and Dana Scott discovered the reverse...
Can we recognize palindromes using a read-only machine that can move
backwards and revisit cells that were previously visited?
A two-way deterministic …nite automaton (2DFA, for short) is an automaton
that can revisit characters already processed. As with DFA’s, there are a …nite
number of states with transitions between them based on the current character,
but each transition is also labelled with a value indicating whether the machine
will move its head to the left, right, or stay at the same position.
Does there exist a 2DFA M that recognizes the language pal? 2DFAs were
introduced in a seminal 1959 paper by Rabin and Scott, who proved them to
have equivalent power to one-way DFAs [1]. That is, any formal language which
can be recognized by a 2DFA can be recognized by a deterministic …nite state
automaton. Since DFAs are obviously a special case of 2DFAs, this implies that
both kinds of machines recognize precisely the class of regular languages. The
proof of Rabin and Scott is long and involved. Shepherdson found, short time
after the communication of Rabin-Scott result, a simpler proof of this fact [2],
(actually, both papers were published in the same journal issue, see below). We
will discuss a proof of this theorem that is based on Shepherdson’s proof.
11
Suppose that Alice and Bob want to make some collaborative computation:
they want to run M, on the input xy; where x is given to Alice, and y is given
to Bob. Alice begins running M, on the input x: When the automaton leaves
x to the right, Alice communicates to Bob the state reached by M, and he uses
this information to continue with the computation: he runs M, on the input
y, but he starts this computation with the scanning head placed on the …rst
character of y; and he set the initial inner state to be equal to the state that
was communicated by Alice. When the scanning head reaches the left end of y;
Bob communicates to Alice the state reached by M, and then she takes charge
of the computation. The computation continues in this way until a halting state
is reached. We have to observe that, the number of messages communicated by
Alice and Bob depends on the length of x and y.
Now suppose that Alice has to go and quit the collaboration. She has to
communicate some bits to Bob, which allow him to take charge of the whole
computation. How many bits must Alice communicate to Bob? It su¢ ces
if Alice communicates O (jxj) bits: it su¢ ces if Alice communicates string x:
Can Alice do better? Suppose that M is a DFA. Then, it su¢ ces if Alice
communicates O (log (jQj)) bits, where Q is the set of states of M: Alice does
her job, if she communicates to Bob the state that is reached by M when the
scanning head reaches the left end of x.
Let C be a class of automata. Consider the following interactive computation.
The spoiler asks Alice and Bob to decide wether M accepts xy: They have
to communicate as few bits as possible.
We say that C is an easy class, if and only if, given M 2 C there exists a
constant CM such that it su¢ ces if Alice communicates CM bits to Bob, and
this number of bits does not depend on the pair x; y: We showed that the class
DFA is easy. We have
fL : 9M (M 2 C ^ L = L (M))g REG
holds.
We have that:
12
We get that L(M) 2N :
Now suppose that the containment
fL : 9M (M 2 C ^ L = L (M))g REG
holds. Let M 2 C. We have that L (M) is regular. Let
and suppose that Alice and Bob has to solve the triple (M; x; y), that is: Alice
gets x; Bob gets y and they are asked to decide wether M will accept the input
xy:
They can solve the triple (M; x; y) by communicating a number of bits that
does not depend on (x; y) : They proceed as follows:
1. Alice computes the number i N for which the equivalence x L(M) wi
holds, and then she communicates to Bob the string wi .
2. Bob simulates the computation of M on the input wi y: Note that this
computation produces the same output than the computation of M on
input xy:
Let K = max fjwi j : i N g : We get that Alice must communicate some-
thing like K log (j j) bits, and this quantity does not depend on the pair (x; y) :
We conclude that C is easy.
We get that, according to the above theorem, proving Rabin-Scott result is
equivalent to prove that the class 2DFA is easy.
Theorem 22 2DFA is an easy class.
Proof. Let M = (Q; ; q0 ; T; A; ) be a 2DFA, and suppose that Alice and Bob
are asked to run the computation of M, on the input xy: We suppose that Alice
is given string x; while Bob is given string y:
Alice has to tell Bob what is the state reached by M the …rst time the
scanning head leaves x and goes to the right. Let qR be this state.
Let q 2 Q: Suppose that M is in the state q scanning the …rst character
of y: Suppose also that (q; y [1]) is equal to (a; b; 1) : Then, in the next
time instant the scanning head will enter the pre…x of the input string
that is hidden for Bob, and the automaton will stay some time processing
characters of this pre…x. If Bob wants to continue with the computation,
without knowing x; he will have to ask Alice about the result of this part
of the computation that is hidden for him. This result could be a halting
state, if the automaton reaches such a state without leaving the pre…x x:
This result could also be a nonterminating state, if the automaton reaches
the right end of x and then moves one further position to the right. Note
that in both cases the result of the computation is just a state M. Let us
use the symbol q b to denote this state.
13
Note that it su¢ ces if Alice communicates to Bob the state qR as well as
the function fM;x : Q ! Q that is de…ned by the equation fM;x (q) = q b :
Then, the total number of bits that must be communicated is upperbounded by
log (jQj) + jQj log (jQj) and this quantity does not depend on the pair (x; y) ; it
only depends on the size of M. The theorem is proved.
How many states are necessary to recognize Lk using a DFA? How many states
are necessary to recognize the same language using a 2DFA?
References
[1] M. Rabin, D. Scott. Finite automata and their decision problems. IBM Jour-
nal of Research and Development 3 (2): 114–125, 1959.
[2] J. Shepherdson. The reduction of two-way automata to one-way automata.
IBM Journal of Research and Development 3(2): 198 - 200, 1959.
14
5 Minimal Writing Power: Inkdot Automata
There was a time when automata used spots to solve problems...
We think that it was a big surprise for the reader to know that the ability of
moving to the left, and check previously visited cells, does not provide computa-
tional power beyond the power of deterministic …nite state automata (1-DFA’s
for short). However, we have to observe that the scanning head cannot recognize
a previously visited cell, unless this cell is located at a short distance from one
of the ends of the input string, or, unless this cells got marked someway in the
previous visits. Note that checking previously visited cells can be useless if one
cannot recognize those cells. Let us try with marking mechanisms that allow
the automaton to mark cells that were previously visited.
Marking is the weakest form of writing. In the next lectures we will inves-
tigate the computational power of di¤erent forms of writing. We consider that
the weakest form of marking power corresponds to provide the automaton with
a bounded number of ink drops, which can be placed on a cell by the scanning
head, which can be sensed by this head, and which cannot be removed.
We study in this lecture the computational power of automata that are
provided with a constant number of ink drops, say k ink drops, and which can
do at most k marking actions along their computations.
15
Proof. The proof is similar to the proof of Theorem 22.
Let M = (Q; ; q0 ; T; A; ) be a Ink-1-2DFA, and suppose that Alice is given
string x; while Bob is given string y:
Let i n = jxj ; and let x(i) be the string that is obtained from x after
dropping the inkdot at position i: Suppose that M is processing input xy; and
suppose that M drops the inkdot at position i: We can think of the remainder
of the computation of M, on the input xy; as the computation of a 2DFA
on the string x(i) y: We know that Bob can take charge of this computation
after receiving the appropriate information from Alice. What information? The
string x(i) y determines the function fi : Q ! Q that is de…ned by:
Function fi is the navigation chart that must be known by Bob, and this naviga-
tion chart can be described using O (jQj log (jQj)) bits. The symbol f0 represents
the function that is determined by x:
We also have to consider the segment of the computation that occurs before
the dropping of the inkdot, and which determines the place at which the inkdot
is dropped. Each time the scanning head enters the pre…x x while carrying the
ink drop, it could drops the ink before leaving this pre…x. The occurrence or
not occurrence of this dropping is determined by the state at which the machine
enters the pre…x x; and this state also determines the position at which the ink
is dropped. Bob must be informed wether the dropping occurs, and he must
also be informed about the position at which the ink is dropped. The important
information about this position i is the function fi that it determines.
If we put all the above pieces together we get that: it su¢ ces if Alice com-
putes the function
Let (q; 0) 2 Q f0; 1g : The pair (q; 0) stands that the scanning head
enters the pre…x x from the right, without the ink drop. The equality
tells us that: if the ink dot is not in x; then f0 (q) is the outcome of this
segment of the computation that is hidden to Bob.
16
Let (q; 1) 2 Q f0; 1g : The pair (q; 1) stands that the scanning head enters
the pre…x x from the right with the ink drop. Suppose that the equality
holds. The triple (f0 (q) ; 1; f0 ) tells us that the scanning head will leave
x carrying the ink drop, and that f0 (q) will be the inner state of M at
this precise instant.
Let (q; 1) 2 Q f0; 1g : Suppose that the equality
M;x (q; 1) = (q ; 0; f )
holds. The triple (q ; 0; f ) tells us that the scanning head will leave x
without the ink drop, that q will be the inner state of M at this precise
instant, and that the ink drop will placed at a position of x; say position
i; such that f = fi . The function f is the function that must be used by
Bob in the future: if some time in the future, the scanning head would
enter x from the right while being in the state q; then the outcome of this
segment of the computation, that is hidden to Bob, will be equal to f (q) :
We claim that function M;x can be used by Bob to solve the following task:
Bob is given a string y; and he is asked to simulate the computation of M
on input xy:
2
We get that it su¢ ces if Alice communicates O jQj log (jQj) bits to Bob,
since this number of bits is enough to describe the function M;x : The theorem
is proved.
We get, from the latter exercise, that the class Ink-1-2DFA is strictly more
powerful than the class DFA. Note that this additional computational power
comes from the ability of those automata to make an unbounded number of
marking actions along its computations. It is questionable wether an unbounded
tank …lled with in…nitely many ink drops is a realistic resource. On the other
hand, the ability of making an unbounded number of marking actions can be
achieved using a bounded number of removable and reusable marks that we
call pebbles. We consider, in the next few lectures, an elementary architecture
for automata that gives place to devices that can make an unbounded number
of marking actions using a bounded number of pebbles. We claim that those
automata are able of nontrivial writing, and we investigate their computational
power.
17
References
18
6 One-Pebble Automata
And then, Manuel Blum invented pebbles...
We study in this lecture the computational power of automata that are
provided with a single pebble, which can be placed on the tape, which can be
sensed, and which can be removed and placed again as many times as necessary.
19
It su¢ ces if Alice computes the function
Q
M;x :Q f0; 1g ! Q f0; 1g (Q f0; 1g)
that is de…ned by:
Let (q; 0) 2 Q f0; 1g : The pair (q; 0) represents that the scanning head
enters the pre…x x from the right, without the pebble. The equality
M;x (q; 1) = (q ; 0; g)
holds. The triple (q ; 0; g) tells us that the scanning head will leave x
without the pebble, that q will be the inner state of M at this precise
instant, and that the pebble will be placed at a position of x; say position
i; such that g = gi .
We claim that function M;x can be used by Bob to solve his task. We get
2
that it su¢ ces if Alice communicates O jQj log (jQj) bits to Bob, since this
number of bits is enough to describe the function M;x : The theorem is proved.
Remark 31 Blum and Hewitt proved that the languages recognized by Pebble-
1-2DFA’s are regular languages. The proof uses the following result: Let N be a
one-tape Turing machine that runs in linear time, the language recognized by N
is a regular language. We think that it was worth to present a proof of this result
that follows from basic principles and the very de…nition of Pebble-1-2DFAs.
Exercise 32 Homework 6. Prove that one cannot recognize the language pal
using a 2DFA that is equipped with a pebble and k ink drops.
References
[1] M. Blum, Ch. Hewitt. Automata on a 2-Dimensional Tape. Proceedings of
SWAT (FOCS) 1967: 155-160.
20
7 Two, Three,... A Bounded Number of Pebbles
At the beginning there was only one pebble, but then a second pebble came to
light...
Why are Pebble-1-2DFAs unable to recognize the language pal? Suppose
we get a string w; and suppose we are asked to decide wether this string is
a palindrome. Processing w forces us to work on both sides of w; scanning
characters on the left and then on the right, and then once again on the left, on
then on the right and so on. Thus, it seems that we need at least two pebbles
to recognize palindromes. Do two pebbles su¢ ce?
where tM (w) denotes the number of transitions that constitute the computation
of M, on the input w: Prove that the running time of the above automaton is
equal to n2 : Is it possible to recognize pal using a Pebble-2-2DFA that runs
in subquadratic time? Hint: investigate about Hennie’s Theorem.
21
3. Which are the languages recognized by Pebble-k-2DFAs?
Pebble automata can be used to recognize complex languages:
Exercise 35 Prove that three pebbles su¢ ce to recognize the language
n n o
a2 : n 1 :
And it seems that the addition of pebbles allows the recognition of more
complex languages:
Exercise 36 Prove that one can use three pebbles to recognize the language
fap : p is a prime numberg :
Exercise 37 Prove that one can use four pebbles to recognize the language
n n
o
an #a2 : n 1 :
Conclude that one can use four pebbles to compute the function 2n :
Exercise 38 Homework 8. Prove that one can use …ve pebbles to recognize
the language n o
n
an #ak #ak : n 1 ;
conclude that …ve pebbles su¢ ce to make arithmetic.
Thus, it seems that the addition of pebbles provides computing power. Let
k 2; does there exist a language which can be recognized using k + 1 pebbles,
but which cannot be recognized using k pebbles?
22
Let i 2m, automaton Mk+2 uses k + 1 pebbles to check wether wi 2 Lk :
Automaton Mk+2 uses the remaining pebble to keep track of the number
of i’s such that wi 2 Lk :
Hsia and Ye proved that Lk+1 requires k + 2 pebbles [1]. Let Mk+2 be a
Pebble-k + 2-2DFA automaton recognizing Lk+1 and let m > 0: It is easy to
prove that there is a string w1 #w2 # #w2m ; there is i 2m; and there is an
instant of the computation of Mk+2 on this string such that Mk+2 places k + 1
pebbles on wi : This implies that Mk+2 has to use k + 1 pebbles to recognize
the i’s such that wi 2 Lk ; and it requires an additional pebble to keep track of
the number of those i’s.
It is important to observe that j k j = k + 1: This means that we are us-
ing increasing alphabets to construct the sequence fLk : k 1g : Moreover, the
recursive nature of this sequence makes it hard to visualize the languages in-
cluded in it. It would be great if we could construct a more uniform sequence
of languages separating the levels of the pebble hierarchy.
We have that pal1 is equal to pal; and we have that pal1 requires two pebbles
to be accepted by a pebble automaton. It seems that pal2 requires three pebbles.
It is natural to conjecture that the language palk requires k+1 pebbles. However,
we have to observe that
Proof. Let w 2 f0; 1g : Let M4 be the pebble automaton that uses the
following divide and conquer strategy:
M4 uses one pebble to divide the input w into two factors, a pre…x
w [1; :::; i] and a su¢ x w [i + 1; :::; jwj] :.
M4 uses the remaining three pebbles to check wether w [1; :::; i] belongs
to pal2 . If this …rst check is positive M4 moves those three pebbles to the
right and checks wether w [1; :::; i] belongs to pal2 : If this latter check is
also positive M4 halts and accepts the input, otherwise it moves the …rst
pebble one position to the right and repeats the above work with the pair
w [1; :::; i + 1] and w [i + 2; :::; jxj].
If the …rst pebble reaches the right end of the input string the automaton
halts and rejects the input.
23
We note that we can use a similar strategy to prove that palk 2 REG log(k)+2 :
We conjecture that palk 2
= REG log(k)+1 :
References
[1] P. Hsia, R. Ye. Marker Automata. Information Sciences 8: 11-88, 1975.
24
8 Simulating Automata by Automata and the
Emergence of Programmable Machines
Pebbles are computational resources that can be smartly used to implement
non-naive programming strategies. Pebbles can also be used for programming.
We say that an automaton is a programmable machine, if and only if, it can
be used to execute an in…nite number of programs. We show that one can use
a …nite number of pebbles to search the transition tables of in…nitely many
automata, as well as to execute the programs consigned in those tables.
25
To begin with the construction we have to de…ne a encoding of DFAs as
strings over a suitable alphabet. Let 0 be equal to f0; 1; #g : Let M =
(Q; ; q0 ; T; A; ) : We can suppose that
and
A = fm + k; m + k + 1; :::; ng :
We also suppose that = f1; :::; lg : We associate to M a string wM 2 0 that
provides a full description of automaton M. Given i 2 N we use the symbol
bi to denote the binary code of i: String wM is a suitable concatenation of the
following strings:
1. The string
wstate = b
1#b
2# #b
n
2. The string
walphabet = b
1#b
2# #b
l:
3. The string
\
waccepting = m + k#m \
+ k + 1# #b
n:
4. The string w that describes the transition function . Note that can
be described by its table, which is …nite list of triples of the form (p; a; q) ;
where p; q 2 f1; :::; ng and a 2 f1; :::; lg : The triple (p; a; q) can be de-
scribed by the string pb#ba#bq : If is equal to
we set
We set
ud
[1]#ud
[2]# #ud
[k] 2 0:
26
Now, we have to construct a computing device that can execute all the
aforementioned programs, that is: we have to construct a Pebble-3-2DFA, that
we will denote with the symbol U0 ; and which satis…es the following
M accepts u; if and only if, U0 accepts the input string wM #### [u] :
Let us also suppose that the state-pebble of U0 is placed over the substring
b
k; while the scanning-pebble is placed over the substring the encodes the
i-th character u: This pebble con…guration suitably represents the con-
…guration reached by M after i transitions. Thus, we are assuming that
the simulation has worked well until this point. We can suppose that the
inner state of U0 is indicating to the machine that it has just …nished the
simulation of the i-th transition of M, and that it is time to begin with the
simulation of the (i + 1)-step. Then, the automaton moves the scanning
pebble to the substring that encodes the (i + 1)-character of u: Suppose
that this character is equal to a: Note that U0 can use the search-pebble
and the scanning-pebble to recognize the substrings of w that encode
triples of the form ( ; a; ) : Note also that U0 can use the search-pebble
and the state-pebble to recognize the substrings of w that encode triples
of the form (q; ; ) : Then, U0 can use those three pebbles to locate the
substring of w that encodes the unique triple of the form (q; a; ) :
27
Suppose that the substring qb#ba#b p has been located. Note that U0 can
use the search-pebble and the state-pebble to place the latter over the
substring pb. Notice that, by doing so, U0 is simulating a further step of
the computation of M on the input u:
Remark 44 The language pal has been intensively studied along the historical
development of automata theory. We claim that there are good reasons for this
prominence of pal; the most important of which is the following one:
The ability of recognizing palindromes and squares is closely related to the
ability of searching tables and executing the programs that are consigned in those
tables.
Note that U0 can search for the right triple, and then for the right state, only
because this automaton can use pairs of pebbles to identify pairs of substrings
that are identical. U0 uses those pebbles to identify the right substrings using
a zig-zag strategy that is analogous to the siz-zag strategies that are used to
recognize palindromes and squares: recognizing palindromes is a prerequisite for
the emergence of programmability.
References
[1] T. Koetsier. On the prehistory of programmable machines: musical au-
tomata, looms, calculators. Mechanism and Machine Theory, Elsevier, 36
(5): 589–603, 2001.
28
9 Unbounded pebbles: Counter Automata
The constructors of automata were accumulating pebbles for some time until
they could proceed with the construction of a supermachine, an automaton with
an unbounded number of pebbles...
What can be done with an unbounded number of pebbles? There are many
things that can be done, and we will employ some time discovering those pos-
sibilities. The …rst thing that comes to mind is to stack pebbles in a pile, and
use this pile to count items beyond any possible bound. Note that this was
the …rst computational application of pebbles in the history of computing: the
construction of counters.
We can use pebbles to simulate counters. To this end we can stack pebbles
on the …rst empty cell, and use this stack as a counter, or we can horizontally
stack those pebbles, using the empty cells of the tape, placing at most one pebble
on each one of those cells. We have to observe that COU N T is a (relatively)
small set that does not contain the language pal: Then, we ask: do we need an
unbounded stock of pebbles to simulate a counter? We have
29
Exercise 51 De…ne COUN T k as the class of problems that can be recognized
using k pebbles and one counter. Prove that
Note that we can use U1 to simulate all the automata considered so far: U1
can simulate 1DFAs, 2DFAs, one-counter automata, pebble automata. Then, it
can be argued that U1 is an universal automaton. Is U1 universal? An a¢ r-
mative answer would imply that any problem that can be solved by automata
and computing machines can be solved by pebble automata.
References
[1] L. Valiant, M. Paterson. Deterministic one-counter automata. Automaten
Theorie und Formale Sprachen: 104-115, 1973.
30
10 Two Counters, Colored Pebbles and Deter-
ministic Pushdown Automata
Once upon the time in Neverland someone dreamed with colored pebbles...
31
1. Q is a …nite set of states.
2. q0 2 Q is the initial state
3. is a …nite set, the input alphabet of M.
4. is a …nite set, the stack-alphabet of M.
5. A Q is the set of accepting states.
6. The transition function is a function from the set Q into the set
P ow (Q fp; w; ng ) : The equality
means that: if the inner state is equal to q; the input-head is scanning the
character a; and the stack-head is scanning the character x; then automa-
ton M nondeterministically chooses one the triples in the set
and executes the order encoded by this triple. Suppose that M chooses
(qi ; "i ; xi ) : Then, M switches from q to qi ; advances its input-head one
step to the right, and makes some work in the pushdown stack. The work
that is done in the pushdown stack is determined by the pair ("i ; xi ) : If
"i is equal to p; the stack-head pops the character that it is reading at this
moment. If "i is equal to w; the stack-head push the character xi on the
top of the stack. If "i is equal to n; the stack-head does nothing.
w 2 f0; 1g : 9i < jwj 9w1 9wi (jw1 j ; :::; jwi j > 1 & w1 ; :::; wi 2 pal) :
32
10.2 Pushdown Acceptors and Context-Free Languages
Pushdown automata were not created by an engineer, those automata were
…rst designed by a linguistic, Noam Chomsky, who was looking for a reason-
able mathematical model of natural languages such as English, French, etc.
Pushdown acceptors correspond to the machine-characterization of context-free
languages, which are the languages generated by context-free grammars. The
concept of pushdown acceptors was in the air since 1954, and it was …rst formal-
ized by Chomsky [1] and Evey [2]. In the late 1960, it was discovered that the
"ALGOL-like" languages, were identical with the context-free languages. This
fact led to a ‡urry of activity in the theoretical development of context-free
languages. Much of this work was done by those concerned either with natural
languages in connection with computers, or with programming languages. The
remainder was done by mathematicians and logicians intrigued by the inherent
problems, techniques, and results.
Pushdown stacks were not introduced because of the likely advantages of
their unlikely architectures. Pushdown stacks were introduced because this is
the architecture that captures the computation of strings using context-free
grammars. Those bizarre automata determine a set of computable languages,
the set of context-free languages, that we can use to test the universality of U1 :
33
Pebble-k-2DFA are a very restricted type of logspace Turing machines, and it
is very unlikely that any context-free language can be recognized using logspace.
However, it is still an open problem whether context-free languages can be
decided in logarithmic space. The best known result shows that they can be
decided in deterministic space O log2 (n) . It is also known that: if all context-
free languages can be decided in deterministic logarithmic space, then L = N L
[5]. The equality between logarithmic space and nondeterministic logarithmic
space is considered to be very unlikely. We have that it is very unlikely that
all context-free languages can be recognized using a …nite number of pebbles.
This suggests that there are computable problems beyond the power of pebble
automata, which could be tackled by automata provided with an unbounded
number of pebbles. We would like to exhibit a context-free language that cannot
be recognized by pebble automata. Recall a previous conjecture: there does not
exist k such that all the powers of pal belong to REG k : We have:
Theorem 57 Suppose that there does not exist k such that for all i 1 the
language pal(i) belongs to REG k : Let pal be equal to the language
n o
w 2 f0; 1; 2g : there exists i 1 such that w = 2i u and u 2 pal(i) :
[
We have that pal is a context-free language that does not belong to REG k :
k 1
The above theorem indicates that our previous conjecture deserves some
consideration. We will discuss some additional facts related to this conjecture
in a future lecture.
34
Let M be a pushdown automaton whose stack alphabet is equal to f1; :::; kg.
We can use i pebbles to simulate the character i; k + 1 pebbles to simulate the
stack-head, and 2k + 1 pebbles to simulate the input-head. Thus, suppose that
the stack content is equal to w1 wn ; and suppose that the input head is
located at position i > n: This con…guration of M is represented by a tape that
contains w1 pebbles placed on the …rst cell,...,wn + k + 1 pebbles placed on the
n-th cell, and 2k + 1 pebbles located on the i-th cell.
References
[1] N.Chomsky. Context-free grammars and pushdown storage. MIT Res. Lab.
Electron. Quart. Prog. Rept. 65, 1962.
[2] R. Evey. The theory and applications of pushdown store machines. Mathe-
matical Linguistics and Automatic Translation, Harvard Univ. Computation
Lab. Rept. NSF-IO, May, 1963.
35
11 Using the Input Tape to Write: Linear Bounded
AutomataXXXVan 12
Someone asked - why do we need a pushdown stack? -. He added - we can use
the cells of the input tape to construct small monochromatic stacks... And use
the concatenation of those stacks to simulate a pushdown stack...
6. The transition function is a function from the set Q into the set
Q fw; ng f 1; 0; 1g : The equality
means that: if the inner state is equal to q and the input-head is scanning
the character a; then the machine executes the order encoded by (p; "; y; ) ;
that is: M switches its inner state from q to p; it makes some work on the
cell its work-head is placed in, and then it moves its input-head according
to : The work that is done on the aforementioned cell is determined by
36
the pair ("; y) : if " is equal to n the machine does nothing, otherwise it
deletes the actual character and overwrites y: Moreover, the equality
implies that = 1: Note that this equality implies that the machine
cannot use the empty cells as work space.
Prove that this problem can be solved by a LBA. What is the running time
of the LBA solving problem SAT?
References
[1] J. Myhill. Linear Bounded Automata (WADD Technical Note). Wright Pat-
terson AFB, Wright Air Development Division, Ohio, 1960.
[2] P. Landweber. Three Theorems on Phrase Structure Grammars of Type 1.
Information and Control 6(2): 131–136, 1963.
[3] S. Kuroda. Classes of languages and linear-bounded automata. Information
and Control 7 (2): 207–223, 1964.
37
12 Turing Machines
Alan Turing was an heroic character. He observed that we could venture into
the obscure region that was made of empty cells...
M = (Q; q0 ; ; ; H; A; ) ;
where:
means that: if the inner state is equal to q and the scanning heads are
reading the tuple (a1 ; :::; ak ) ; then the machine executes the order encoded
by (p; "; y; 1 ; :::; k ) ; that is: M switches its inner state from q to p; makes
some work on the cell that is occupied by the head with label 1; and then
moves all its heads according to the tuple 1 ; :::; k : The work that is done
on the aforementioned cell is determined by the pair ("; y) : if " is equal
to n, then the machine does nothing on this cell, otherwise it deletes the
actual character and overwrites y:
38
Warning. The equality
We can simulate multitape Turing machines using the same ideas used in the
simulation of linear bounded automata automata. Thus, given a Turing machine
M there exists a pebble automaton PM , provided with in…nitely many pebbles,
and such that PM can simulate the computations of M.
3. 1915, a …nite state automaton constructed by IBM left an oil-spot over its
tape. Inkdot automata were accidentally discovered. The elusive search
for an automaton recognizing pal remained un…nished.
4. 1916, M. Blum and Ch. Hewitt invented pebble automata. They con-
structed a two pebble automaton recognizing the language pal:
39
7. 1932, J. Myhill began to use pebbles to write arbitrarily long strings on
the input tape. The writing machine was invented.
8. 1937, A. Turing discovered the power of the twilight zone. He constructed
an universal Turing machine, proved that the halting problem is undecid-
able and formulated the Church-Turing-Pebble Thesis [1].
11. 1985, D. Deutsch replaced pebbles by spins [3]. The Quantum Turing
Machine was invented.
40
References
[1] A. Turing. On Computable Numbers, with an Application to the Entschei-
dungsproblem. Proceedings of the London Mathematical Society. 42(2): 230
- 265, 1937.
[2] S. Cook. The complexity of theorem proving procedures. Proceedings of the
Third Annual ACM Symposium on Theory of Computing. pp. 151–158, 1971.
[3] D. Deutsch. Quantum theory, the Church-Turing principle and the universal
quantum computer. Proceedings of the Royal Society A. 400 (1818): 97–117,
1985.
41
13 Why Do We Need Turing Machines and Other
Computer Architectures
We have pebbles, why do we need Turing machines?
Pebble automata constitute one of the most rudimentary and ine¢ cient ar-
chitectures, and this makes those automata become inappropriate for theoretical
analysis. On the other hand, The Turing Machine architecture is an ine¢ cient
architecture that is versatile enough to be appropriate for the theoretical analy-
sis of algorithmic problems. Let us focus on the problem of recognizing the
language pal: What are the perfomances that we can expect from those two
models of universal computation?
We can recognize the language pal in time 32 n + O (1) using a two-head
Turing machine M that works, on input w; as follows:
On the other hand, we know that we can recognize pal using a pebble au-
tomaton that runs in time n2 and which implements the naive zig-zag strat-
egy discussed before. It is important to observe that this zig-zag automaton cor-
responds to the simulation of M using pebbles. Thus, we have that the solution
using pebbles is a parsimonious version of the solution using Turing machines.
Does there exists a better solution using pebbles? We prove that any solution
using pebbles runs in time n2 : The proof of this fact mimics the proof of
Hennie’s Theorem [1].
Remark 64 Hennie’s Theorem estates that any one-head Turing machine recog-
nizing pal runs in time n2 : This result of Hennie is one of the few known
superlinear lower bounds that hold for an universal model of computation, and
it is also one of the oldest lower bounds that can be found in the computing
literature. Thus, we are …nishing this part of the book with our pebble version
of one of the earliest results of complexity theory.
Proof. The proof shows that a pebble automaton recognizing pal is an au-
tomaton that implements a strategy that is very close to the zig-zag strategy
discussed before. Thus, the proof shows that any pebble automaton recognizing
pal is an automaton that has to go so many times from left to right and from
right to left that it spends quadratic time processing its inputs.
n
Let M be a pebble automaton recognizing the language pal: Let w 2 f0; 1g ,
and let sw = w0n wR : Suppose that M is processing input sw : We de…ne Sw
42
as the ordered sequence of states that are reached by w each time the scanning
head of M goes from cell n + 1 to cell n after visiting the su¢ x wR :
References
[1] F. Hennie. Crossing sequences and o- ine Turing machine computations.
Proceedings of 6th Annual Symposium on Switching Circuit Theory and Log-
ical Design (SWCT 1965).
43
Part II
Nondeterminism: a Crash
Introduction to Complexity
Theory
44
14 The Power of Nondeterminism
Pushdown automata exhibit a characteristic that tells them apart from all the
other models of automata considered in this work. We are referring to the
following nontrivial fact: let M = (Q; q0 ; ; ; A; ) be a PDA, the transition
function of M is a function from Q into P ow (Q fp; w; ng ) : This
implies that an instant con…guration (q; a; x) does not determine a single action
but a …nite set of possible actions, say the set f(qi ; ri ; xi ) : i ng ; which are
all equally likely, and which do not lead to the same outcomes. Thus, the
nondeterministic character of those machines make them either unreliable or
unfeasible:
Suppose that w 2 L (M). The string w could determine exponentially many
computations of M. Suppose that this is the case, and suppose that the ac-
cepting computations constitute a small subset of the former set. If we run the
automaton only once we will see a rejecting computation with a high probabil-
ity, but the outcome of this computation is clearly wrong. Then, if we want to
obtain a de…nitive answer from M, we will have to simulate all the exponen-
tially many computations of M on w: The latter can be done in exponential
time using a Turing machine, and it can also be done in linear time using expo-
nentially many processors working in parallel. We need, in those both cases, a
exponential amount of computational resources.
It can be said that nondeterministic machines work on paper and cannot be
used in practice. Nondeterministic machines are machines that do not …t our
most naive conceptions about computation. We have to ask: how found this
bizarre concept its way into computing theory?
45
Thus, we have to ask: how could this bizarre notion of nondeterministic
computation survive to the fact that it does not yield computational power
to the devices it was applied? Rabin and Scott proved that nondeterministic
transitions allow the construction of automata with fewer states.
Exercise 70 Let Lk be the language
w 2 f0; 1g : w [jwj k + 1] = 0 :
Prove that Lk can be recognized by a NFA with k states. Prove that a DFA
recognizing Lk has at least 2k states (use Myhill-Nerode Theorem).
The above fact is a very important one: we have to take into account that
memory-size (number of states) is the single computational resource of …nite
automata, and it is the only computational resource that we should spare. Thus,
Rabin and Scott showed that nondeterminism allows the construction of more
e¢ cient automata. They also showed that nondeterminism allows more natural
constructions, and they exempli…ed this fact by providing natural constructions
of NFAs for the concatenation of two regular languages and for the Kleene star
of a regular language.
Exercise 71 Construct the aforementioned NFAs.
However, it can(could) be argued that the aforementioned advantages of
nondeterminism were not enough to accept this bizarre concept. Let us ask:
does nondeterminism provide computational power?
46
It can be proved that, if L is a deterministic context-free language then min (L)
and max (L) are also deterministic context-free languages. Use this to prove
that pal cannot be recognized by a deterministic pushdown automata (Hint. you
will have to prove that either min (pal) is not context-free or max (pal) is not
context-free, to prove the latter you can use the pumping lemma for context-free
languages).
w2wR : w 2 f0; 1g :
References
[1] M. Rabin, D. Scott. Finite automata and their decision problems. IBM Jour-
nal of Research and Development 3 (2): 114–125, 1959.
[2] N.Chomsky. Context-free grammars and pushdown storage. MIT Res. Lab.
Electron. Quart. Prog. Rept. 65, 1962.
[3] M. Schutzenberger. On context-free languages and pushdown automata. Inf.
and Control, 6(3): 246 - 264, 1963.
47
15 Nondeterministic Pushdown Automata and
Context-Free Grammars
Linguists describe the grammars of natural languages in terms of their block
structure, and describe how sentences are recursively built up from smaller
phrases, and eventually individual words or word elements. Context-free gram-
mars provide a simple and mathematically precise mechanism for describing
the methods by which phrases are built from smaller blocks. The simplicity of
context-free grammars makes the formalism amenable to rigorous mathematical
study. On the other hand, the expressive power of context-free grammars allows
them to describe the basic recursive structure of sentences, the way in which
clauses nest inside other clauses, and the way in which lists of adjectives and
adverbs are swallowed by nouns and verbs
S ! X;
X ! : (X) j ((X) con (X)) j var;
con ! ^ j _;
var ! 0var j 1var j ";
where the symbol " denotes the empty string, and the rule var ! "
corresponds to a deletion rule.
48
Let G = (V; T; S; R) be a context-free grammar and let w 2 T : A derivation
of w is a …nite sequence 1 ; :::; n such that:
w1 wi 1 wi+1 wn
1. S;
2. X;
6. ((var) _ (var)) ;
7. ((1var) _ (var)) ;
8. ((1var) _ (1var)) ;
9. ((1) _ (1var)) ;
De…nition 76 Let G be a grammar, we use the symbol L (G) to denote the set
of strings that can be derived in G:
49
Remark 78 Block structure was introduced into computer programming lan-
guages by the Algol project (1957–1960), which, as a consequence, also featured
a context-free grammar to describe the resulting Algol syntax. This became
a standard feature of computer languages. The "block structure" aspect that
context-free grammars capture is so fundamental to grammar that the terms
syntax and grammar are often identi…ed with context-free grammar rules, espe-
cially in computer science. Formal constraints not captured by the grammar are
then considered to be part of the "semantics" of the language.
Exercise 79 The elementary functions are the functions of calculus, which are
constructed from the monomials, the trigonometric functions, the exponential
and the logarithmic functions using sums, products, divisions and compositions.
Construct a context-free grammar GCalculus for the language constituted by all
those functions.
ww : w 2 f0; 1g ;
use the pumping lemma to prove that this language is not context-free.
The language SQU ARES require a di¤erent type of grammars, the so called
context-sensitive grammars.
50
Context-sensitive grammars were also studied by Chomsky, who placed them
in the top level of his hierarchy of generative grammars, the so called Chom-
sky Hierarchy, and which is constituted by the regular grammars (regular lan-
guages), context-free grammars (context-free languages), and the context-sensitive
grammars (context-sensitive languages) [3].
Exercise 83 Can you characterize the context-free grammars that give place to
regular languages?
If the construction of a grammar for SQU ARES was a hard task, then the
reader will become scared if we ask him to prove that the following language is
context sensitive
The reader can easily solve this latter task, if she(he) takes into account the
following important result by Kuroda [1].
References
[1] S. Kuroda. Classes of languages and linear-bounded automata. Information
and Control 7(2): 207–223, 1964.
[2] N. Chomsky. Three models for the description of language. IEEE Transac-
tions on Information Theory, 2(3): 113–124, 1956.
51
16 Parsing
Let G be a context-free grammar. Consider the problem:
Input: w 2 T :
Problem: decide if w 2 L (G) :
The proof of the theorem of Chomsky is constructive, and this implies that
we can assume that we are given a PDA MG that recognizes the language L (G) :
Suppose that L (G) is a nondeterministic context-free language that cannot be
recognized by a deterministic PDA. We get that MG is a nondeterministic PDA.
Perhaps the reader is tempted to use MG to solve the recognition problem for
L (G) : However, we have to take into account the nondeterministic nature of
MG : nondeterministic machines are not intended to be used in practice. If
we insist in use MG we will have to …gure out a method for simulating all
the computations of MG on the input string. This can be done, but it could
have an exponential cost. Thus, we have to conclude that MG provides us
with a (nondeterministic) solution to the recognition problem, but it could be
a very ine¢ cient solution: nondeterministic algorithms can be simulated by
deterministic ones, but this simulation can have an (super)exponential slow
down. Can we e¢ ciently solve the recognition problem for L (G)? If L (G) is
deterministic, then we can use a deterministic PDA that solves the recognition
problem, and this PDA provides us with a real-time solution. Thus, let us ask:
can be e¢ ciently solve the recognition problem for nondeterministic context-free
languages?
We can try to solve a harder problem. Consider the following problem:
Input: w 2 T :
Problem: decide if w 2 L (G) ; in that case compute a derivation of w in
G:
Or, even better, we can try to solve the problem of computing parse trees
for the strings in L (G) that are given as input.
52
This is a parse tree for the formula (:p ^ q) ) (p ^ (q _ :r)) : One could argue
that the above parse tree is the meaning of this formula:
The meaning of a formula is its extension, which is equal to the set of its
models. Then, it can be argued that the meaning of a formula is a speci…cation of
the way this formula behaves with respect to all the possible truth assignments.
The above parse tree can be understood as such speci…cation. Given a truth
assignment for the formula, the value of this truth assignment corresponds to
the propagation (from the bottom to the root) of the truth values assigned to
the leaves of this tree.
Let w 2 L (GP L ) : A parse tree for w is a labeled tree (T; r; L) ; that satis…es:
1. r is the root of T:
2. L is a labeling function, that assigns to each node of T an element of
(V [ T ) :
3. L (r) = S:
4. The labels assigned to the leaves of T belong to T:
5. If one reads those leaves from left to right, he reads the string w:
Exercise 88 Let be the formula (((1) ^ (2)) _ (: ((3) ^ (: (2))))) that belongs
to L (GP L ) : Construct a parse tree for this formula.
53
16.2.1 The Algorithm
Let G be a context-free grammar. We say that G is in Chomsky normal form,
if and only if, any production rule in G has either the form A ! or the
form A ! BC; where 2 T and B; C 2 V: It is not hard to prove that any
context-free grammar G that does not produce the empty string is equivalent
to a grammar in Chomsky normal form. Moreover, given G as above, one can
compute G such that G is in Chomsky normal form and G is equivalent to
G :
The CYK algorithm receives as input a Chomsky grammar G and a string w;
and it outputs either a parse tree for w or a message indicating that w 2
= L (G) :
This algorithm is a dynamic programming algorithm, which means that it com-
putes an array, of which some entries can be easily computed at the beginning of
the computation, while the remaining entries are computed in a dynamic way
propagating the data that are encoded into the previously computed entries.
The computed array is equal to [Pl;s;v ]l s jwj;v2V ; where
Pl;s;v = 1; if and only if, the string w [l; s] can be derived from the variable v:
Note that it is easy to compute all the entries of the form Pl;l;v : To do
this, one simply has to check if G contains the production rule v ! w [l] : To
propagate the previously computed values one uses the following rule
It is easy to decide, after computing the whole array, wether w 2 L (G) : one
only has to check if the equality P1;jwj;S = 1 holds.
54
subcubic algorithm for context-free recognition. The dependence on e¢ cient
matrix multiplication cannot be avoided altogether: Lee has proved that any
parser for context-free grammars working in time O n3 " jGj can be e¤ectively
converted into an algorithm computing the product of boolean matrices in time
"
O n3 3 (see [3]).
References
[1] T. Kasami. An e¢ cient recognition and syntax-analysis algorithm for
context-free languages- Technical report AFCRL. 65-758, 1965.
[2] L. Valiant. General context-free recognition in less than cubic time. J. Com-
put. Syst. Sci. 10 (2): 308–314, 1975.
[3] L. Lee. Fast context-free grammar parsing requires fast Boolean matrix mul-
tiplication. J. ACM. 49(1): 1–15, 2002.
55
17 The LBA Problems: Unreliable Machines Can-
not be E¢ ciently Simulated by Reliable Ma-
chines
Let G be a context-sensitive grammar. Consider the problem:
Input: w 2 T :
Problem: decide if w 2 L (G) :
56
can use the CYK deterministic algorithm instead of those PDAs. We can also
dispense us of using nondeterministic linear bounded automata, because we can
use the naive simulation-algorithm sketched in the previous section. However,
we have to note that we will pay a very high prize if we use this simulation
algorithm: the running time, and the space requirements of this algorithm are
astronomical. Then, we have to look for better algorithms retaining some of the
advantages of the original NLBAs. We have to ask: what are those advantages?
Suppose that we are given a NPDA M that recognizes L (G) : The running time
of this automaton could be exponential. Then, we have to conclude that the
strength of M is not related to its running time. On the other hand, we have
to observe that a linear bounded automaton is a Turing machine that satis…es
a further constraint that is related to the amount of memory used by those
automata.
De…nition 92 Let M be a nondeterministic Turing machine which is provided
with a read-only input tape and a …nite number of work tapes, one head per tape.
We say that M is a multitape Turing machine. Let w be an input of M, we
de…ne sM (w) as
max f#cells (c) : c is a computation of M on wg ;
where #cells (c) is equal to the maximum number of cells of the work tapes that
are simultaneously occupied along the execution of c:
We de…ne SM : N ! N by the equation
n
SM (n) = max fsM (w) : w 2 g:
Function SM measures the memory requirements of the machine M.
Note that linear bounded automata are essentially the same as multitape
Turing machines that satisfy the inequality SM (n) n:
Exercise 93 Let C > 0; and let M be a multitapeTuring machine that satis-
…es the inequality SM (n) Cn. Prove that there exists a machine M that
recognizes the language L (M) and which satis…es the inequality SM (n) n:
Let f : N ! N be a non-decreasing function. We de…ne two complexity
classes:
57
17.2 The LBA Questions
The two LBA questions go back to Kuroda, who asked:
The …rst of the above two questions remains open. Let us discuss in brief
the second question.
Let M be a deterministic machine, and let co-L (M) be the language
fw : w is rejected by Mg :
Can we recognize co-L (M) using a machine that satis…es the same time bounds
and the same memory requirements? Yes, we can, and it is enough if we inter-
change the role of the accepting and rejecting states of M. Now suppose that
M is a nondeterministic machine. It is not clear if we can recognize co-L (M)
using the same type of machines. Note that:
References
[1] R. Floyd. Nondeterministic Algorithms. Journal of the ACM. 14 (4): 636 -
644, 1967.
58
18 The Immerman-Szelepcsényi Theorem
The Immerman–Szelepcsényi theorem states that nondeterministic space com-
plexity classes are closed under complementation. It was proven independently
by Neil Immerman and Róbert Szelepcsényi in 1987, for which they shared the
1995 Gödel Prize (see [1] and [2]). In its general form the theorem states that
NSPACE(s (n)) = co-NSPACE(s (n)) for any function s (n) log (n) : We are
interested in the special case s (n) = n:
Set x = 0:
n
For i = 1 to n jQj j j generate the i-th possible con…guration of the ma-
chine and call it ci .
59
– Guess a path from s to ci : Verify that it is a proper path.
– If the guessed path is proper set x = x + 1:
This procedure is a good example of the guess and verify technique used in
nondeterministic algorithms. All we did was exploit our de…nition of nondeter-
minism. We looked at all possible con…gurations and counted those which were
reachable from the starting con…guration.
Set n0 = 1.
Set i = 0:
Repeat
– i = i + 1.
– ni = 0.
– Set m = 0:
n
– For j = 1 to n jQj j j generate sj : Guess whether sj can be reached
in i steps or less. If a path from sw to sj is veri…able then ni = ni +1:
If reached in less than i steps then m = m + 1:
– Verify that m = ni 1 (otherwise reject).
until ni = ni 1:
Proof. Most of the work has already been done. To built a machine which
accepts the complement of L (M) involves
putting the previous two procedures together. First …nd out exactly how
many con…gurations are reachable. Then examine all of them, and if any ac-
cepting con…gurations are encountered, reject. Otherwise accept. The theorem
is proved.
60
References
[1] N. Immerman. Nondeterministic space is closed under complementation.
SIAM Journal on Computing, 17(5): 935 - 938, 1987.
61
19 L vs NL
There are problems that can be solved using a small amount of workspace, and
these are precisely the problems that can be e¤ectively parallelized to be solved
in polylogarithmic time.
Exercise 103 Prove that MC( ) can be solved using a logspace Turing ma-
chine.
62
19.3 Complete Problems for NL
The L vs NL question is one of the most important problems of theoretical
computer science. This question, as well as the P vs NP question, has been
attacked using suitable notions of reducibility and completeness.
Exercise 108 Let L be a language that is complete for NL. We have that L is
equal to NL, if and only if, L belongs to L.
The above exercise indicates that the notion of NL-completeness can help
us to settle the L vs NL question. However, we have to ensure that this notion
is nonempty. Do there exist problems that are complete for NL? Consider the
following problem:
63
Proof. Let L be a language in NL, and let M be a logspace Turing machine
that recognizes the language L: Let w be an instance of L: Let sw be the initial
con…guration of M determined by the input w: The set of con…gurations of M
that can be accessed from w has polynomial size with respect to jwj ; and this
implies that all its elements can be described using a logarithmic number of bits.
Moreover, the whole graph can be constructed using logspace. Let GM;w be this
graph. Note that sw 2 V (GM;w ) : Let Tw be the set of accepting con…gurations
that are contained in the set V (GM;w ) : We have that w 2 L; if and only if,
(GM;w ; sw ; Tw ) is a YES-instance of GA. The function that sends w in the triple
(GM;w ; sw ; Tw ) is a logspace reduction of L into GA. The theorem is proved.
Savitch realized that the LHS paper had a fundamental idea, an idea that was
very clever, yet an idea that even the authors did not completely understand.
In proving that every context-free language was accepted using O log2 (n)
nondeterministic work space, Lewis et al had used a clever method for keeping
track of the pushdown states. Essentially, Savitch saw that when separated
from context-free languages, what they were doing was exactly what he need to
prove his theorem.
64
language, the same algorithm may be implemented with the same asymptotic
space bound on a Turing machine.
References
[1] W. Savitch. Relationships between nondeterministic and deterministic tape
complexities. Journal of Computer and System Sciences, 4(2): 177–192, 1970.
[2] P. Lewis, R. Stearns, J. Hartmanis. Memory bounds for recognition of
context-free and context-sensitive languages. SWCT (FOCS) 1965: 191-202.
65
20 The Pebble Hierarchy Revisited
A nondeterministic k-pebble automaton is a k-pebble automaton whose transi-
tion function is nondeterministic. We have to ask: what is the relation between
the set of languages accepted by deterministic k-pebble automata and the set
of languages recognized by nondeterministic k-pebble automata. We use the
symbol N REG k to denote the latter class.
Let k 0: It is easy to check that any k-pebble automaton can be sim-
ulated by a logspace Turing machine: one can use logarithmic space to track
the position of the k-heads. An analogous fact holds true for nondeterministic
pebble automata: those automata can be simulated by nondeterministic Turing
machines. It is a little bit more surprising that the reciprocal of those two facts
also hold true. It follows form the work of Petersen [1] and Sudborough [2] that:
[
1. L = REG k :
i 0
[
2. NL = N REG k :
i 0
pal : ci w : w 2 pali :
We have:
66
Lemma 114 The language pal belongs to N REG 4 :
67
Exercise 117 It is possible to simulate two non-sensing heads using two pebbles
(it is possible to simulate k heads using k pebbles).
It is very much harder to prove that one can simulate k pebbles using k + 1
nonsensing heads. Let REG h2 be the set of languages that can be recognized
using two nonsensing heads. We have that
References
[1] H. Petersen. The Equivalence of Pebbles and Sensing Heads for Finite Au-
tomata. FCT 1997: 400-410.
[2] I. Sudborough. On tape bounded complexity classes and multihead …nite
automata. J. Compt. System Sci. 10:338 - 345, 1975.
68
21 P vs NP
With this lecture we arrive to one of the most important problems of theoretical
computer science P 6= NP.
We will assume the above point of view in the remainder of this work, that
is: we will assume that the complexity class P is constituted by all the tractable
problems.
Guess a candidate.
Check that this candidates holds the intended property.
If the set of candidates is in…nite the above strategy gives place to nonhalting
algorithms. If the set of candidates is a large …nite set, this strategy gives place
69
to ine¢ cient brute force algorithms, and if the set of candidates is small (can
be made small) we get an e¢ cient algorithm.
Suppose that the input is a …nite structure, and suppose that we are asked
to compute a substructure exhibiting some special property. Observe that this
scenario corresponds to the above second category: the set of substructures has
exponential size. Most problems of combinatorial optimization have this kind
of structure: most of those problems ask for the construction of substructures
exhibiting some special properties. Most of those problems also have a second
special feature: the structural property the de…nes the problem is easy to check.
Consider the following examples:
We can solve any one of those problems by guessing a substructure and then
checking that this substructure holds the property that de…nes the problem. If
the candidate does not satisfy this property, then we try a second candidate,
and then a third, and so on. We halt when, either we …nd a substructure that
satis…es the property, or we try all the possible candidates without …nding a
solution to the problem. By doing so, we are solving the existential problem
by performing exponential many checks, that can be done, each one of them,
in polynomial time. We can work in a sequential fashion and use exponential
time, or we can use an exponential number of processors and solve the problem
in polynomial time, or we can work nondeterministically with a machine that
always do the correct guess. We have to observe that none of those three options
is feasible. Then, it is worth to consider a fourth option, namely: can the search
spaces of all those existential problems be reduced to sets of polynomial size?
The reader should think in the Eulerian problem. Let G be an instance of
the problem. The set of possible Eulerian trails for G is equal to the set of
permutations of E (G). However, the Theorem of Euler tells us that this large
3
set can be replaced by the set V (G) ; and that the special individual that must
be searched, in this latter very much smaller space, is a triple such that all
its three nodes are odd degree nodes. This theorem yields a polynomial time
algorithm for the problem. We ask: do any existential problem like the …ve
listed above count with an Euler-like Theorem? We would like to deal with this
question. To this end we have to de…ne, in a precise way, what do we mean with
the expression: a problem like the …ve listed above.
De…nition 119 The complexity class NP is the set of languages that can be
recognized by nondeterministic polynomial time Turing machines.
70
Exercise 120 Explain why the complexity class NP is also equal to the set of
languages whose instances can be correctly classi…ed by doing an exponential
number of easy checks.
Exercise 121 The complexity calls NP can be presented in terms of the notion
of certi…cate. Let R be a binary relation. We say that R is p-balanced,
if and only if, the following two conditions are satis…ed:
LR = fx#y : (x; y) 2 Rg
Let L9R be the language fx : 9y ((x; y) 2 R)g : Prove that L belongs to NP,
if and only if, there exists a p-balanced relation R such that L = L9R :
Let x 2 L9R ; and let y be a string such that (x; y) 2 R; we say that y is a
certi…cate for x:
71
Exercise 124 Suppose that L is polynomial time reducible T (L 4pt T ). It can
be argued that T is at least as hard as L : if T belongs to P, the language L
also belongs to P.
We would say that L is the hardest problem in NP, if and only if, any problem
in NP can be polynomial time reduced to L:
Exercise 125 Choose one of your favorite problems in NP, and reduce it to
SAT.
It seems that many people did the same exercise around 1970, and it seems
that most of those people arrived to the same conclusion we arrived: it is easy
to reduce a problem in NP to the problem SAT. This suggests that SAT is one
of the hardest problems in NP.
Remark 129 Before Cook’s paper there was not much interest in problems
computable in polynomial time by nondeterministic Turing machines. The class
NP only became an interesting class after Cook’s Theorem.
Exercise 130 Suppose that L is NP-hard, and also suppose that L 4pt T: Prove
that T is NP-hard.
72
Problem 131 Integer Factorization
This problem is one of the few problems that is known to belong to both
NP and co-NP (but not known to be in P). It is easy to check that the problem
belongs to NP: if m does have such a factor then the factor itself is a certi…cate.
Membership in co-NP is also straightforward: one can just list the prime factors
of m, all greater or equal to n, which the veri…er can con…rm to be valid by
multiplication and the AKS primality test.
References
[1] A. Cobham. The Recognition Problem for the Set of Perfect Squares. SWAT
(FOCS) 1966: 78 - 87, 1973.
[2] L. Levin. Universal search problems. Problems of Information Transmission
(in Russian) 9 (3): 115116.
[3] N. Chomsky. On Certain Formal Properties of Grammars. Inf. Control. 2(2):
137 - 167, 1959.
[4] N. Chomsky. A Note on Phrase Structure Grammars. Inf. Control. 2(4): 393
- 395, 1959.
73
22 NP-Completeness: Automata Synchroniza-
tion
Let L be a problem. Suppose that we suspect that L is NP-hard. This conjecture
leads us to search for a NP-hardness proof for L. It follows from the de…nition
that L is NP-hard, if and only if, SAT is polynomial time reducible to L. It also
follows from the de…nition of NP-hardness that problem SAT is as reducible
to L as any other NP-complete problem, that is: theory tells us that it is the
same to look for a reduction of SAT into L than to look for a reduction into L
of a NP-hard problem that seems to be closely related to this latter problem.
However, theory tells us one thing, and practice tells us a completely di¤erent
thing: if we want to prove that L is NP-hard, we should look into the enormous
list of problems that are known to be NP-complete, and then choose those
problems that exhibit structural analogies with problem L: Sometimes one can
easily …gure out a reduction of SAT (CNF-SAT) into L: Sometimes it is very
much easier to …gure out a reduction of a suitable NP-complete problem into L
than a reduction from SAT.
We would like to study two algorithmic problems related to automata. Those
two problems are NP-hard, and they represent two opposite sides of the wide
spectrum of possibilities. The …rst one is the minimal-synchronizing word prob-
lem (MSS, for short), and the second one is the separating words problem (SW,
for short).
It can be easily proved that MSS is NP-hard. The NP-hardness proof for
this problem is a straightforward reduction of CNF-SAT into it (this reduction
is one the most crystal-clear reduction this author knows). It can also be proved
that SW is NP-hard. However, this second result is a recent result, and the NP-
hardness proof is a reduction of a technical problem about …nite semigroups into
SW. It is true that one can use this reduction, and other auxiliary reductions,
to construct a reduction of SAT into SW. We can search the literature and
reconstruct the path that goes from SAT to L: However, if we do the latter we
will have to compose no less than four nontrivial polynomial time reductions.
V (GM ) = Q:
Given p; q 2 Q there exists an edge (p; q) with label a; if and only if, the
equality (p; a) = q holds.
74
agent x is placed at node q; and suppose that it has to execute the program
w 2 ; then after i time units this agent will be placed at node b (p; w [1; :::; i]) :
Now suppose that we have several agents scattered over GM ; suppose that we
do not know the speci…c position of each agent, and suppose that we want to
send a message w 2 , the same message for all the agents, that leads all
those agents to same node q: What we need is a synchronizing string for the
automaton M.
De…nition 132 Let M = (Q; ; ) be a DFA (without initial and accepting
states), and let w 2 : We say that w is a synchronizing string for M; if and
only if, for all p; q 2 Q the equality b (p; w) = b (q; w) holds. We say that M
is a synchronizing automaton, if and only if, there exists a synchronizing string
for M.
There are DFAs that are not synchronizing.
Exercise 133 Show that any unary automaton without a sink state is not syn-
chronizing. Prove also that any automaton with more than one sink is not
synchronizing.
Consider the following problem:
can be accessed from any other state of M(2) : Thus, the problem of recog-
nizing the automata that can be synchronized is polynomial time reducible
to an accessibility problem about digraphs. The theorem is proved.
75
Let M be a synchronizing automaton. We de…ne the reset treshold of M as
We have:
Let us set
we have that r (n) 2 O n3 : The famous Cerný Conjecture estates that rt (n)
2
is equal to (n 1) (see [1]). Then, if this conjecture is true, there is a large gap
between the length of the shortest synchronizing strings, and the length of the
strings that can be constructed using the above sketched naive strategy. One of
the ubiquitous questions of computer science is the question about the existence
of algorithms that beat the naive algorithmic solutions. Thus, it is natural to
ask wether there exists a polynomial time algorithm for the construction of
shortest synchronizing strings. Consider the problem:
Remark 138 If P6=NP there exist synchronizing automata whose reset length
is lesser than the length of the synchronizing strings that are constructed using
the naive pair-synchronization strategy.
76
Proof. Consider the propositional formula
If Xi does not occur in cj ; then there is not shorcut at all, and the two
edges that go out from qji are directed toward the next node.
7! (G ; n)
Exercise 140 Write down a detailed description of the reduction used in the
proof of the above theorem.
References
[1] J. Cerný, A. Pirická, B. Rosenauerová. On directable automata. Kybernetika
7(4): 289 - 298, 1971.
[2] D. Eppstein. Reset Sequences for Monotonic Automata. SIAM J. Comput.
19(3): 500 - 510, 1990.
77
23 NP-completeness: Discerning I
We study the problem of separating words with …nite automata. This problem
can be regarded as “the simplest computational problem you could ask to solve”.
To begin with we suppose that there are two parties, say Alice and Bob,
which can communicate with each other using a noiseless communication chan-
nel. We also suppose that those two parties have unlimited computational
power.
n
First, we suppose that Alice has a string w 2 f0; 1g , and that she wants
to communicate this string to Bob. What is the minimum number of bits that
must be transmitted by Alice in order to achieve her goal? This is the question
that is addressed by Kolmogorov Complexity [1]. We know that the answer is
O (K (w)) ; where K (w) is the Kolmogorov complexity of string w: Recall that
K (w) is equal to the length of the smartest (shortest) program that separates
the string w:
Exercise 142 Suppose for an instant that Alice is not allowed to compute smart
programs separating the string w (she cannot compute unrestricted Turing ma-
chines), and that she is restricted to compute deterministic …nite state automata.
We have that in this latter case she is obligated to communicate (n) bits: prove
that a minimal DFA separating the string w has n + 2 states.
Let us consider a second scenario. In this second scenario, Alice has a string
n n
w 2 f0; 1g , Bob has a string u 2 f0; 1g , and they must decide if those two
strings are either equal or di¤erent. What is the minimum number of bits that
must be communicated by Alice and Bob in order to achieve their common goal?
This is the question that is addressed by Communication Complexity [2]. To
begin with we notice that there is a naive solution to the problem that is being
faced by Alice and Bob: Alice communicates to Bob K (w) bits describing the
string w: However, we know that there exists nondeterministic communication
protocols that allow Alice and Bob to solve their problem by communication
very much fewer bits. Consider the following protocol:
2. Alice sends to Bob the string biw [i], where bi is the binary code of i:
78
3. Bob checks if the equality w [i] = u [i] holds.
Notice that, if the equality w = u holds, Alice and Bob will agree that those
two strings are equal. On the other hand, if w 6= u Alice can guess a position
i n for which w [i] 6= u [i] : In the latter case Alice and Bob will agree that those
two strings are di¤erent. Thus, we can conclude that the above nondeterministic
protocol computes the equality function using O (log (n)) communication bits.
It is known that the latter problem requires (log (n)) communication bits to
be correctly solved in the nondeterministic sense [2].
Let us consider a third scenario. Suppose that Alice and Bob have an un-
n
ordered pair of strings, say the strings w; u 2 f0; 1g , and suppose that Alice
chooses one of those two strings. What is the minimum number of bits that
must be communicated to Bob in order to let he knows which is the string cho-
sen by Alice? It happens that the latter question is closely related to the former
two questions:
If Alice wants to solve the latter problem e¢ ciently, then she has to compute
the smartest program separating u from w; and then she has to communicate
a succinct description of this program to Bob.
Remark 143 We say that program M separates u from w, if and only if, the
…nal states of the computations of M on those two inputs are di¤ erent states.
Observe that the above solution to the discerning problem, as we call our
problem, can be turned into a nondeterministic communication protocol that
computes the equality function using (roughly speaking) the same number of
communication bits. Consider the following protocol:
79
23.1 Separating Strings with Finite Automata
Suppose for an instant that Alice is not allowed to compute smart programs,
and that she is restricted to compute DFA’s. We use the symbol SDF A (u; v) to
denote the number of bits that must be communicated by Alice in this latter
scenario. We are interested in the asymptotic behavior of function SDF A .
Notice that any n-state DFA can be fully described using O (n log (n)) bits.
We can forget the logarithmic factor and focus our attention on the functions
Sep : ! N and sep : N ! N that are de…ned by
We are interested in upper and lower bounds for function sep: Let us begin
with two naive bounds.
Lemma 144 Let f : N ! N be a function such that f (n) log (f (n)) belongs to
(log (n)), we have that:
Proof. The lower bound comes from the connections between the discerning
problem and nondeterministic communication complexity that were discussed
at the end of the previous section. The upper bound is obvious.
Observe that the gap between those two bounds is exponential. Thus, there
is a lot of room for improvements. It seems that the …rst published reference on
this subject is an old paper of Goralcik and Koubek [3]. This paper includes the
…rst non-trivial upper bound for function sep : Goralcik and Koubek proved that
2 3
sep (n) 2 o (n). Robson proved, shortly after, that sep (n) 2 O n 5 log 5 (n) ,
and it is the best known upper bound for function sep [4].
Notice that the gap between those two bounds is still exponential, and notice
that those bounds have not been improved after more than thirty years. We
heard that Je¤rey Shallit o¤ered 100 pounds to the …rst person that improves
one of those bounds.
Notice that the di¤erences between those strings are somewhat obvious, and
then one is tempted to conjecture that there must be pairs of strings that are
very much harder to separate. Therefore, it is a little bit frustrating to know
80
that (log (n)) is the best known lower bound for function sep; and this bound
is achieved by the GK pairs.
Can we shorten the gap between upper and lower bounds for function sep? It
could happens that one of the two bounds is sharp, either the lower bound or the
upper bound. Thus, if we want to shorten the gap between those two bounds,
we will have to carefully decide which of those two bounds can be improved.
Can complexity theory help us to take a suitable decision?
References
[1] M. Li, P. Vitanyi. An introduction to Kolmogorov Complexity and Its Ap-
plications. Springer Verlag, Berlin, 2008.
[2] E. Kushilevitz, N. Nisan. Communication Complexity. Cambridge University
Press, 1996.
[3] P. Goralcik, V. Koubek. On discerning words by automata. In L. Kott,
editor, Proc. 13th Int’l Conf. on Automata, Languages, and Programming
(ICALP), volume 226 of Lecture Notes in Computer Science, pages 116-122.
Springer-Verlag, 1986.
[4] J. Robson. Separating strings with small automata. Inform. Process. Lett,
30: 209-214, 1989.
81
24 NP-Completeness: Discerning II
Consider the algorithmic problem
Input: (u; v; k), where u; v are binary strings of the same length and k is
a positive integer.
Problem: decide if sep (u; v) k:
Proposition 146 Suppose that there exists m such that sep (n) 2 O (logm (n)) ;
and suppose that SEP is NP-hard. Then, given a problem L in NP there exists
c
cL such that L can be solved in time O 2log L (n) .
It is widely believed that the problems that are NP-hard cannot be solved
in pseudopolynomial time, that is: it is widely believed that given a NP-hard
problem L; there does not exist cL > 0; such that L can be solved in time
c
O 2log L (n) .
Exercise 148 Suppose that the above hypothesis holds, and suppose that SEP
is NP-hard, conclude that the containment sep (n) 2 (logm (n)) holds for all
m > 0.
82
it not possible to adapt this proof, which uses in a very strong way that the
sets to be separated could be very large. We claim that it is hard to …gure out
a reduction of the dfa consistency problem into the problem SEP. Is SEP NP-
hard? Yes, the problem SEP is NP-hard, and we will able to prove this result
thanks to an important connection between automata and …nite transformation
semigroups.
De…nition 150 We say that a pair of strings, say the pair (u; v) ; is an identity
for Tk , if and only if, for all interpretation
: f0; 1g ! Tk
Volkov et al proved that IDENT is co-NP hard [2]. We use this result to
prove that SEP is NP-hard.
83
Remark 152 The NP-hardness of SEP is mentioned for the …rst time in [?],
where a proof of this fact is barely sketched. It seems that the authors of the
aforementioned preprint were not aware of the (conditional) implications of this
result on the asymptotic behavior of function I: they closed their paper claiming
that they have few hopes about obtaining, in the near future, superlogarithmic
lower bounds for function I:
Proof. Let (u; v; k) be an instance of IDENT. We only have to prove that the
latter pair is an identity of Tk , if and only if, this pair cannot be separated using
k states. The latter equivalence follows from the well studied relation between
…nite state automata an transformation semigroups [2]. Given
Notice that Mf0 ;f1 separates the pair (u; v) ; if and only if, the equality
fu; = fv; does not hold, with the interpretation given by the equalities
We get the …rst implication: if (u; v) is an identity for Tk , then the pair
(u; v) cannot be separated with k states. On the other hand, given a n-state
automaton M; we can always suppose that QM is equal to f1; :::; kg : If we do the
latter, we get that automaton M determines an interpretation M : f0; 1g ! Tk
which is given by:
M (i) (s) = M (s; i) :
Notice that M separates the pair (u; v) ; if and only if, the equality fu; M =
fv; M
does not hold. We get the second implication, and the theorem is proved.
The NP hardness of SEP can also be obtained from a reduction of the prob-
lem SAT into the former problem. However, it seems that the most natural
of those reductions is the composition of no less than three reductions that
involve some technical problems about …nite groups and …nite transformation
semigroups.
References
[1] M. Gold. Complexity of automata identi…cation from given data. Informa-
tion and control, 378(3): 302-320, 1978.
84
[2] J. Almeida, M. Volkov, S. Goldberg. Complexity of the identity checking
problem for nite semigroups. J. Math. Sciences, 158(5): 605 - 614, 2009.
[3] A. Bulatov, O. Karpova, A. Shur, K. Startsev. Lower Bounds on Words
Separation: Are There Short Identities in Transformation Semigroups? The
Electronic Journal of Combinatorics 24(3), 2017.
85