Expectation

Integration, Lebesgue Convergence Theorem

Definition: Simple Random Variables

A random variable $X$ is said to be simple if there exists a finite family $(A_k{)}_{1\le k\le n}$ of pairwise disjoint events and real nonzero numbers $({\alpha }_k{)}_{1\le k\le n}$ such that:

$$X=\sum _{k=1}^n{\alpha }_k{1}_{A_k}$$

We denote by ${\mathcal{L}}^{0,step}$ the set of all simple random variables.

Warning

Clearly, ${\mathcal{L}}^{0,step}$ is a subset of ${\mathcal{L}}^0$. However, the decomposition of a simple random variable $X$ into $(A_k{)}_{1\le k\le n}$ and $({\alpha }_k{)}_{1\le k\le n}$ is not unique. Indeed, let $A$ and $B$ be two elements of $\mathcal{F}$ such that $A\subseteq B$. It follows that $X=1_B=1_{B\setminus A}+1_A$.

Lemma

The spaces ${\mathcal{L}}^0$ as well as ${\mathcal{L}}^{0,step}$ are vector spaces.

Proof

The proof is left as an exercise.

Let $X=\sum _{1\le k\le n}{\alpha }_k{1}_{A_k}$ and $Y=\sum _{1\le l\le m}{\beta }_l{1}_{B_l}$ be two simple random variables. It is possible to find a finite family $(C_r{)}_{1\le j\le r}$ of pairwise disjoint elements in $\mathcal{F}$ and two finite families $(\widetilde{{\alpha }_j}{)}_{1\le j\le r}$ and $({\tilde{\beta }}_j{)}_{1\le j\le r}$ of nonzero numbers such that:

$$X=\sum _{1\le j\le r}{\tilde{\alpha }}_j{1}_{C_j},Y=\sum _{1\le j\le r}{\tilde{\beta }}_j{1}_{C_j}$$

Indeed, let $C_{kl}=A_k\cap B_l$, which constitutes a finite family of pairwise disjoint elements in $\mathcal{F}$. It follows that:

$$X=\sum _{1\le k\le n}\sum _{1\le l\le m}{\alpha }_k{1}_{C_{kl}},Y=\sum _{1\le k\le n}\sum _{1\le l\le m}{\beta }_l{1}_{C_{kl}}$$

In other words, any two simple random variables can be decomposed on the same common family of pairwise disjoint elements.

Definition: Expectation 1.0

We define the expectation of a simple random variable $X=\sum _{1\le k\le n}{\alpha }_k{1}_{A_k}$ with respect to $P$ as:

$$\hat{E}[X]:=\sum _{k\le n}{\alpha }_{k}P[A_k]$$

Exercise

Show that the definition of expectation is a well-defined operator on ${\mathcal{L}}^{0,step}$. Indeed, the decomposition of $X$ is not unique.

Proposition

On ${\mathcal{L}}^{0,step}$, the following properties hold:

• Monotonicity: $\hat{E}[X]\le \hat{E}[Y]$ whenever $X\le Y$.
• Linearity: $\hat{E}$ is a linear operator on ${\mathcal{L}}^{0,step}$.

Proof

Let $X$ and $Y$ be two simple random variables. Since those two simple random variables can be decomposed on a common set of events we can write:

$$X=\sum _{k\le n}{\alpha }_k{1}_{A_k},Y=\sum _{k\le m}{\beta }_k{1}_{A_k}$$

If $X\le Y$, it follows that ${\alpha }_k=X(\omega )\le Y(\omega )={\beta }_k$ for every state $\omega$ in $A_k$. Hence:

$$\hat{E}[X]=\sum _{k\le n}{\alpha }_{k}P[A_k]\le \sum _{k\le n}{\beta }_{k}P[A_k]=\hat{E}[Y]$$

For real numbers $a$ and $b$, it holds:

$$\hat{E}[aX+bY]=\sum _{k\le n}(a{\alpha }_k+b{\beta }_k)P[A_k]=a\sum _{k\le n}{\alpha }_{k}P[A_k]+b\sum _{k\le n}{\beta }_{k}P[A_k]=a\hat{E}[X]+b\hat{E}[Y]$$

This proposition is important in so far that ir shows that the expectation is a linear operator which is monotone. This monotonicity property allows to extends naturally from below the expectation to the class of positive random variables.

• First Approximation

  ---

  

• Second Approximation

  ---

  

Definition: Expectation 2.0

For any positive extended random variable $X$ in ${\overline{\mathcal{L}}}_{+}^0$, we define the expectation of which as

$$E[X]:=sup\{\hat{E}[Y]:Y\le X,Y\in {\mathcal{L}}_{+}^{0,step}\}$$

---

A random variable $X$ is called integrable if $E[X^{+}]<\mathrm{\infty }$ and $E[X^{-}]<\mathrm{\infty }$. The set of integrable random variables is denoted by ${\mathcal{L}}^1$ and the expectation of which is defined as

$$E[X]:=E[X^{+}]-E[X^{-}]$$

Remark

• Show as an exercise that for a positive extended random variable $X$ where $P[X=\mathrm{\infty }]>0$, then it follows that $E[X]=\mathrm{\infty }$;
• Clearly ${\mathcal{L}}^{0,step}\subseteq {\mathcal{L}}^1$;
• Also, by definition and monotonicity of $\hat{E}$, for every $X\in {\mathcal{L}}^{0,step}$, it holds that $E[X]=\hat{E}[X]$. In other terms, $E$ is an extension of $\hat{E}$ to the space ${\overline{\mathcal{L}}}_{+}^0$. We therefore remove the hat on the top of the expectation symbol everywhere.

Lemma

For every $X$ and $Y$ in ${\overline{\mathcal{L}}}_{+}^0$ and $a,b$ positive numbers, it holds:

• $E[X]\le E[Y]$ whenever $X\le Y$.
• $E[aX+bY]=aE\left[X\right]+bE\left[Y\right]$.

Proof

The proof is left as an exercise.

Theorem: Lebegue's Monotone Convergence Theorem

Let $(X_n)$ be an increasing sequence of positive random variables. Denote by $X=lim{X}_n=sup{X}_n$ the resulting extended positive random variable limit of the sequence. Then it holds

$$E[X]=E[lim{X}_n]=limE[X_n]$$

Proof

By monotonicity, we clearly have $E[X_n]\le E[X]$ for every $n$, therefore $supE[X_n]\le E[X]$. Reciprocally, suppose that $E[X]<\mathrm{\infty }$ and pick $\epsilon >0$ and some simple positive random variable $Y$ such that $Y\le X$ and $E[X]-\epsilon \le E[Y]$. For $0<c<1$ define the sets $A_n=\{X_n\ge cY\}$. Since $X^n$ is increasing to $X$, it follows that $A_n$ is an increasing sequence of events. Furthermore, since $cY\le Y\le X$ and $cY<X$ on $\{X>0\}$, it follows that $\cup A_n=\mathrm{\Omega }$. By non-negativity of $X_n$ and monotonicity, it follows that

$$cE[1_{A_n}Y]\le E[1_{A_n}{X}_n]\le E[X_n]$$

and so

$$csupE[1_{A_n}Y]\le supE[X_n]$$

Since $Y=\sum _{l\le k}{\alpha }_l{1}_{B_l}$ for ${\alpha }_1,\dots ,{\alpha }_k\in {\mathbb{R}}_{+}$ and $B_1,\dots ,B_k\in \mathcal{F}$, it follows that

$$E\left[1_{A_n}Y\right]=\sum _{l\le k}{\alpha }_{l}P[A_n\cap B_l].$$

However, since $P$ is a probability measure, and $A_n$ is increasing to $\mathrm{\Omega }$, it follows from the lower semi-continuity of probability measures that $P[A_n\cap B_l]\nearrow P[\mathrm{\Omega }\cap B_l]=P[B_l]$, and so

$$supE[1_{A_n}Y]=\sum _{l\le k}{\alpha }_{l}supP[A_n\cap B_l]=\sum {\alpha }_{l}P[B_l]=E[Y].$$

Consequently

$$E[X]\ge limE[X_n]=supE[X_n]\ge cE[Y]\ge cE[X]-c\epsilon$$

which by letting $c$ converging to $1$ and $\epsilon$ to $0$ yields the result. The case where $E[X]=\mathrm{\infty }$ is similar and left to the reader.

As the previous figure suggests, it is actually possible to construct by hand a sequential approximation of positive random variables by simple ones.

Proposition: Approximation by Simple Random Variables

For any positive random variabel $X$, there exists an increasing sequence of simple positive random variables $(X_n)$ such that $X_n(\omega )\nearrow X(\omega )$ and uniformly on each set $\{X\le M\}$ where $M\in \mathbb{R}$.

Proof

Let $A_k^n=\{(k-1)/2^n\le X<k/2^n\}$ for $k=1,\dots ,n{2}^n$ and define

$$X_n:=\sum _{k=1}^{n{2}^n}\frac{k-1}{2^n}{1}_{A_k^n}+n{1}_{\{X>n\}}$$

From the definition, it follows that $X_n\le X$ for every $n$ and $X(\omega )-2^{-n}\le X_n(\omega )$ for every $\omega \in \{X\le n\}$. This, along with the monotonicity, concludes the proof.

Proposition

For $X$ and $Y$ in ${\mathcal{L}}^1$, a real number $a$ and two disjoint events $A,B$ in $\mathcal{F}$. The following assertions hold:

1. $1_{A}X$, $X+Y$, $aX$ and $|X+Y|$ are integrable.
2. $E[(1_A+1_B)X]=E[1_{A}X]+E[1_{B}Y]$.
3. $E[X+Y]=E[X]+E[Y]$ and $E[aX]=aE[X]$.
4. $E[X]\le E[Y]$ whenever $X\le Y$.
5. If $X\ge 0$ and $E[X]=0$, then $P[X=0]=1$.
6. If $P[X\ne Y]=0$, then $E[X]=E[Y]$.
7. If $Z$ is a random variable such that $|Z|\le X$, then $Z$ is integrable.

Remark

In particular, ${\mathcal{L}}^1$ is a vector space and the expectation operator $E:{\mathcal{L}}^1\to \mathbb{R}$ is a monotone, positive, and linear functional.

Proof

1. It holds $|X+Y|\le |X|+|Y|$. According to Lemma \ref{lem:linearityL0+}, it follows that $E[|X+Y|]\le E[|X|+|Y|]=E[|X|]+E[|Y|]<\mathrm{\infty }$, showing that $X+Y$ and $|X+Y|$ are integrable. The argumentation for $1_{A}X$ and $aX$ follows the same line.

2. It holds $(1_A+1_B)X=(1_A+1_B)X^{+}-(1_A+1_B)X^{-}$. From the linearity on ${\mathcal{L}}_{+}^0$, it follows that $E[(1_A+1_B)X^{\pm }]=E[1_A{X}^{\pm }]+E[1_B{X}^{\pm }]$, showing that $E[(1_A+1_B)X]=E[1_{A}X]+E[1_{B}X]$.

3. Without loss of generality, assume that $a\ge 0$. Here again, it follows from $aX=a{X}^{+}-a{X}^{-}$ and from the linearity on ${\mathcal{L}}_{+}^0$ that $E[a{X}^{\pm }]=aE[X^{\pm }]$. Also, since $X+Y=(X^{+}+Y^{+})-(X^{-}+Y^{-})=(X+Y{)}^{+}-(X+Y{)}^{-}$, it follows that $(X^{+}+Y^{+})+(X+Y{)}^{-}=(X^{-}+Y^{-})+(X+Y{)}^{+}$. However, again from the linearity on ${\mathcal{L}}_0^{+}$, it holds $E[(X^{+}+Y^{+})+(X+Y{)}^{-}]=E[X^{+}]+E[Y^{+}]+E[(X+Y{)}^{-}]$ and $E[(X^{-}+Y^{-})+(X+Y{)}^{+}]=E[X^{-}]+E[Y^{-}]+E[(X+Y{)}^{+}]$, showing that $E[X+Y]=E[(X+Y{)}^{+}]-E[(X+Y{)}^{-}]=E[X^{+}]-E[X^{-}]+E[Y^{+}]-E[Y^{-}]=E[X]+E[Y]$.

4. If $X\le Y$, it follows that $0\le Y-X$. According to the proposition stating the approximation from below, let $(Z_n)$ be an increasing sequence of positive simple random variables such that $Z_n\nearrow Y-X$. It follows from the monotone convergence Theorem that $0\le E[Z^n]\le supE[Z_n]=E[Y-X]$. Applying the previous point, we get $E[Y-X]=E[Y]-E[X]$, yielding the assertion.

5. Let $A_n=\{X\ge 1/n\}$ which is an increasing sequence of events such that $\cup A_n=\{X>0\}$. It follows that $1_{A_n}1/n\le 1_{A_n}X\le X$ since $X$ is positive. Monotonicity from the previous point yields $P[A_n]/n\le E[1_{A_n}X]\le E[X]=0$, showing that $P[A_n]=0$ for every $n$. By the lower semi-continuity property of measures, it follows that $P[A]=supP[A_n]=0$, showing that $P[X>0]=0$.

6. Suppose that $P[X\ne 0]=0$ and define $X_n=|X|1_{\{X=0\}}+(|X|\wedge n)1_{A_n}$ where $A_n=\{|X|\ge 1/n\}$. On the one hand, by definition, $A_n$ is an increasing sequence such that $\cup A_n=\{|X|\ne 0\}$. Hence, $0\le X_n\nearrow |X|$, which by the monotone convergence Theorem implies that $E[X_n]\nearrow E[|X|]$. On the other hand, $A_n\subseteq \{X\ne 0\}$, which by monotonicity of the measure implies that $P[A_n]=0$ for every $n$. Hence, $E[X_n]\le nP[A_n]=0$ for every $n$. We conclude that $E[|X|]=0$, which implies that $E[X]=0$.

7. Follows directly from the linearity on ${\mathcal{L}}_{+}^0$.

Remark

Note that for $X\in {\overline{\mathcal{L}}}^0$ with $X^{-}\in {\mathcal{L}}^1$, and $Y\in {\mathcal{L}}^1$, the same argumentation as above yields that:

$$-\mathrm{\infty }<E[X-Y]=E[X^{+}-X^{-}-Y]=E[X^{+}]-E[X^{-}]-E[Y]=E[X]-E[Y]$$

We finish this section with two of the most important assertions of integration theory.

Theorem: Fatou's Lemma and Lebegue's Dominated Convergence

Let $(X_n)$ be a sequence in ${\mathcal{L}}^0$.

• Fatou's Lemma: Suppose that $X_n\ge Y$ for some $Y\in {\mathcal{L}}^1$. Then it holds

  $$E[lim inf{X}_n]\le lim infE\left[X_n\right].$$

• Dominated Convergence Theorem: Suppose that $|X_n|\le Y$ for some $Y\in {\mathcal{L}}^1$ and $X_n(\omega )\to X(\omega )$ for any state $\omega$. Then it holds

  $$E\left[X\right]=limE\left[X_n\right].$$

Proof

By linearity, up to the variable change $X_n-Y$, we can assume that $X_n$ is positive since $E[lim inf{X}_n-Y]=E[lim inf{X}_n]-Y$ and $E[X_n-Y]=E[X_n]-Y$ for every $n$. Let $Y_n=\underset{k\ge n}{inf}{X}_n$, which is an increasing sequence of positive random variables that converges to $lim inf{X}_n=\underset{n}{sup}\underset{k\ge n}{inf}{X}_k$. Notice also that $Y_n\le X_k$ for every $k\ge n$, and therefore by monotonicity of the expectation $E[Y_n]\le \underset{k\ge n}{inf}E[X_k]$. We conclude Fatou's lemma with the monotone convergence theorem as follows:

$$E[lim inf{X}_n]=limE\left[Y_n\right]=supE\left[Y_n\right]\le \underset{n}{sup}\underset{k\ge n}{inf}E[X_k]=lim infE[X_n].$$

A simple sign change shows that Fatou's lemma holds in the other direction. That is, if $X_n\le Y$ for some $Y\in {\mathcal{L}}^1$, then it holds

$$lim supE\left[X_n\right]\le E[lim sup{X}_n].$$

Now the dominated convergence theorem assumptions yield that $-Y\le X_n\le Y$ for some $Y\in {\mathcal{L}}^1$. Hence, since $X=lim{X}_n=lim inf{X}_n=lim sup{X}_n$, it follows that

$$lim supE\left[X_n\right]\le E[lim sup{X}_n]=E\left[X\right]=E[lim inf{X}_n]\le lim infE\left[X_n\right].$$

However, $lim infE\left[X_n\right]\le lim supE[X_n]$, showing that $E[X_n]$ converges, and

$$E[X]=lim infE[X_n]=lim supE[X_n]=limE[X_n].$$

which ends the proof.

Example: Defining a Probability Measure from a Density

The concept of density is quite often used in statistics as it defines new measures. Let us formalize it using dominated convergence.

On a probability space $(\mathrm{\Omega },\mathcal{F},P)$, consider a positive integrable random variable $Z$ such that $E[Z]=1$. We define the set function

$$\begin{array}{rl}Q:\mathcal{F}& ⟼[0,1]\\ A& ⟼Q[A]=E[Z{1}_A]\end{array}$$

which is clearly well defined and mapping to $[0,1]$ since $Z$ is positive and $E[Z{1}_A]\le E[Z]=1$.

It follows that $Q$ defined as such is a new probability measure. Indeed

• $Q[\mathrm{\varnothing }]=E[Z{1}_{\mathrm{\varnothing }}]=E[0]=0$, $Q[\mathrm{\Omega }]=E[Z{1}_{\mathrm{\Omega }}]=E[Z]=1$;

• $\sigma$-additivity: Let $(A_n)$ be a sequence of disjoint events. It follows that

  $$1_{{\cup }_{k\le n}{A}_k}=\sum _{k\le n}{1}_{A_k}\nearrow \sum 1_{A_n}=1_{\cup A_n}$$

  By monotone convergence

  $$\begin{array}{rl}\sum Q[A_n]& =lim\sum _{k\le n}Q[A_k]\\ & =lim\sum _{k\le n}E[Z{1}_{A_k}]\\ & =limE\left[Z\sum _{k\le n}{1}_{A_k}\right]\\ & =E[Z\sum 1_{A_n}]\\ & =E[Z{1}_{\cup A_n}]=Q[\cup A_n]\end{array}$$

It can be shown using step functions that integration under $P$ and $Q$ are related through the formula

$$E^Q[X]=E^P\left[ZX\right]$$

for any bounded random variable $X$ or any $X$ with sufficient integrability under $Q$.

Another particular property of the probability measure $Q$ so defined is that it is absolutely continuous with respect to $P$ in the sense that

$$P[A]=0\text{implies}Q[A]=0$$