Problem Sheet 6 Solutions: Continuous–Time Markov Chains

Question 6
Let Xi ∼ Exp(λ) be the length of time the machine functions for the ith time before it breaks
down again (for i = 1, 2, . . . ). Let Yi ∼ Exp(ν) denote the length of time the machine is in the
being repaired state for the ith time (for i = 1, 2, . . . ). All Xi and Yi are independent. (i) Then
1 3 2
E(X1 ) = , E(X1 + X2 + X3 + Y1 + Y2 ) = + .
λ λ ν
(ii)
1 3 2
V ar(X1 ) = , V ar(X1 + X2 + X3 + Y1 + Y2 ) = + 2.
λ2 λ2 ν

Question 7
(i) For each individual we have:

P(it wakes up in (t, t + δ]|it was asleep at t) = βδ + o(δ)

P(it falls asleep in (t, t + δ]|it was awake at t) = νδ + o(δ)

There are N individuals, thus if there are i awake, there are N − i asleep; the ‘birth’ and ‘death’
rates are

λi = (N − i)β
µi = iν,

for i = 0, . . . , N . Thus the generator matrix is

 
−N β Nβ 0 0 ···
 ν
 −ν − β(N − 1) β(N − 1) 0 ··· 

 0
 2ν −2ν − β(N − 2) β(N − 2) 

G=
 . .. . .. .. .

 . 

 (N − 1)ν −(N − 1)ν −β β 

 
Nν −N ν

(ii) For the stationary distribution, solve πG = 0, from notes, for a general birth and death
process we have
λ0 . . . λn−1
πn = π0 , n ≥ 1
µ1 . . . µn
(N β)(β(N − 1)) × β(N − n + 1)
= π0
ν(2ν) . . . (nν)
βn N
 
= π0 .
νn n

1
 PN
Also, since n=0 πn = 1, giving
1
π0 = PN βn N

.
1+ n=1 ν n n

(iii) For one individual  

−β β
G= .
ν −ν
(iv) From the forward equations P′t = Pt G, it follows

d
(1 − psw (t)) = −β(1 − psw (t)) + νpsw (t)
dt
d
⇔ − psw (t) = psw (t)(β + ν) − β
dt
d
⇔ psw (t) + psw (t)(β + ν) = β
dt

Using the integrating factor M (x) = exp((β + ν)x) and since psw (0) = 0, the constant C = 0 and
thus Z t
β
psw (t) = β exp((β + ν)u)du · exp(−(β + ν)t) = (1 − e−(β+ν)t ).
0 β+ν
In a similar manner as before
d
(1 − pww (t)) = −β(1 − pww (t)) + νpww (t)
dt
with the boundary condition pww (0) = 1, so

β + νe−(β+ν)t
pww (t) = .
β+ν

(v) Xm,t denotes the number of individuals awake at time t > 0, given that at time 0 there were
m < N individuals awake. Recall that all individuals behave independently of each other. We get
that
Xm,t = Ym,t + Zm,t ,
where
Ym,t ∼ Binomial(m, pww (t)), Zm,t ∼ Binomial(N − m, psw (t)).
Hence, we have two random variables with Binomial distribution, where the probability of success
is the probability that at time t, the individual will be awake. We have that

E(Ym,t ) = mpww (t), E(Zm,t ) = (N − m)psw (t).

Altogether we get

E[Xm,t ] = mpww (t) + (N − m)psw (t)

Nβ
= me−(β+ν)t + (1 − e−(β+ν)t ).
β+ν

Question 8
Let Tn be the first time the population drops to size n or below, i.e. Tn = min{t : Nt ≤ n} (for a
non-negative integer n ≤ n0 ). Then NT0 = 0 and T = T0 . We write T as

T = (T0 − T1 ) + (T1 − T2 ) + · · · (Tn0 −1 − Tn0 ),

2
where Tn0 = 0. Then T = X1 + · · · + Xn0 where Xi = Ti−1 − Ti is the time the population has
size i before dropping to size i − 1. We have that X1 , . . . , Xn0 are independent with Xi ∼ Exp(µi)
for i ∈ {1, . . . , n0 }. Using standard results in probability and the moments of an exponential
distribution it follows that
 
1 1 1
E[T ] = 1 + + ··· +
µ 2 n0
 
1 1 1
Var[T ] = 1 + + · · · + .
µ2 4 n20

Question 9
If n = 0, via the forward equations

p′0 (t) = −λp0 (t) + βp1 (t).

If n ≥ 1:
p′n (t) = λpn−1 (t) − (λ + β)pn (t) + βpn+1 (t).
Let t → +∞. Then the derivatives going to 0 implies

−λp̂0 + β p̂1 = 0
λp̂n−1 − (λ + β)p̂n + β p̂n+1 = 0.

We try a solution of the form p̂n = Cθn and discover that

−Cλ + βCθ = 0
n−1 n n+1
λCθ − (λ + β)Cθ + βCθ = 0
P∞ P∞ n
which tells us that θ = λ/β. Using the fact that 1 = n=0 p̂n = C n=0 θ , we see that
C = (1 − λ/β), so that   n
λ λ
p̂n = 1 − .
β β

Question 10
Let Xt denote the number of representatives busy at time t. Let pn (t) = P(Xt = n), and p0 (0) = 1.
The forward equations are

p0 (t + δ) = (1 − 2δ)p0 (t) + δp1 (t) + o(δ)

p1 (t + δ) = (1 − 3δ)p1 (t) + 2δp0 (t) + 2δp2 (t) + o(δ)
p2 (t + δ) = 2δp1 (t) + (1 − 2δ)p2 (t) + o(δ).

Hence

p′0 (t) = −2p0 (t) + p1 (t)

p′1 (t) = 2p0 (t) − 3p1 (t) + 2p2 (t) (1)
p′2 (t) = 2p1 (t) − 2p2 (t). (2)

Note that you could derive the forward equations using the generator matrix: We know that the
forward equations are given by P′t = Pt G. The corresponding generator is
 
−2 2 0
G =  1 −3 2 
0 2 −2

3
this can be justified by arguing that we have an exponential clocks running in each state: if no
customer is present, the next customer arrives at rate 2; if one customer is present, the rate 2
arrival competes with the rate 1 service time; if two customers are present, new customers are
ignored and the two service times compete to give rate 2.
Substitute p2 (t) = 1 − p0 (t) − p1 (t) in (1), which yields

p′1 (t) = 2 − 5p1 (t)

⇔ p′1 (t) + 5p1 (t) = 2

Using the integrating factor approach, we obtain M (x) = exp(5x) and since p1 (0) = 0, the
corresponding constant is given by C = 0. Altogether, we get
Z t
2
p1 (t) = 2 exp(5x)dx · exp(−5t) = (1 − e−5t ).
0 5

Consider (2):
4
(1 − e−5t ).
p′2 (t) + 2p2 (t) =
5
This is an ODE that can be solved by the integrating factor approach. The integrating factor is
Z x 
M (x) = exp 2dz = e2x .
0

Then using the boundary condition, p2 (0) = 0, it follows that the corresponding constant is given
by C = 0. Thus Z t 
−2t 4 −5u 2u
p2 (t) = e (1 − e )e du .
0 5

The integral is Z t  
4 −5u 2u 4 1 2t 1 −3t 5
(1 − e )e du = e + e − .
0 5 5 2 3 6
Hence
2 4 2
p2 (t) = + e−5t − e−2t
5 15 3
as required.