Skip to content

Markov Processes - Exercises

Exercise 1

Markov Chain \(X\) on the canonical space with transition probability matrix

\[ p= \begin{pmatrix} 1 & 0 & 0 & 0 \\ 1/2 & 0 & 1/2 & 0 \\ 0 & 1/2 & 0 & 1/2 \\ 0 & 0 & 0 & 1 \\ \end{pmatrix}, \]
  1. As in the lecture, provide a graph that shows how the process evolves between the four states. Hereby, you can visualize which sequences are possible or not.
  2. Compute the probability \(P_x[X_t=4, \text{ for some time }t]\) where \(x\) is a state. That is the probability that \(X\) reaches at some moment the state \(4\) starting from \(x\).

  3. Let \(\tau_{1,4}=\inf\{t\colon X_t=1\text{ or }X_t=4\}\) be the first "visiting" time of the Markov Chain of the states \(1\) or \(4\). Compute for every state \(x\).

    \[E_{P_x}\left[ \tau_{1,4} \right]\]

Exercise 2

number \(b\) of black balls. Starting from time \(1\), we pick a ball at random from the urn, and put the ball back together with another one of the same color. Each ball in the urn has the same probability of being picked. For instance at time \(1\), if I pick a black ball, I put it back in the urn together with an additional black one. Hence at time one the urn contains \(w\) white balls and \(b+1\) black balls. Denote now \(B_t\) the number of black balls at time \(t=0,1,\ldots\) in the urn and by \(X_t\) the proportion of black balls in the urn at time \(t\), that is

\[X_t=\frac{B_t}{t+w+b}, \quad t=0,1, \ldots\]

and let \(\mathcal{F}_t=\sigma(X_s\colon s\leq t)\). Hence the conditional probability given \(\mathcal{F}_t\) to pick a black ball at time \(t\) is \(X_t\). Show that

  1. \(X\) is a martingale. (Astonishing? In particular if you start with a large proportion of black :)) Hint: Note that per definition, \(P[B_{t+1}=B_t+1|\mathcal{F}_t]=X_t\) and \(P[B_{t+1}=B_t|\mathcal{F}_t]=1-X_t\).
  2. Show that there exists a random variable \(X\) such that \(X_t\to X\) in \(L^p\) for \(1 < p<\infty\).
  3. Assuming that \(b=w=1\), compute \(P[B_t=k]\) for \(k=1,\ldots, t+1\) for every \(t\). And determine the distribution of \(X\). Hint: guess the form for \(t=1,2,3\) and show per recursion of every \(t\).