Probability functions and densities, distribution functions, mathematical expectation, marginal and conditional distributions, conditional expectation

1. Introduction to Probability Distributions

In statistics, a probability distribution describes how likely it is that a random variable will take on each of its possible values. It's a fundamental concept that helps us understand and model random phenomena. We often deal with two main types of random variables: discrete and continuous.

A discrete random variable can only take on a finite number of values or a countably infinite number of values. Examples include the number of heads in three coin flips or the number of defective items in a batch.

A continuous random variable can take on any value within a given range. Examples include height, weight, or temperature. The way we describe probability distributions differs slightly for these two types of variables.

2. Distribution Functions

The distribution function, also known as the cumulative distribution function (CDF), provides the probability that a random variable X takes on a value less than or equal to a specific value 'x'. It is denoted by F(x) = P(X ≤ x).

2.1. Cumulative Distribution Function (CDF) for Discrete Random Variables

For a discrete random variable X with possible values x₁, x₂, x₃, ..., and corresponding probabilities P(X = x_i) = p_i, the CDF is given by:

F(x) = P(X ≤ x) = Σ_{xi ≤ x} P(X = x_i)

The CDF for a discrete random variable is a step function. It is non-decreasing, right-continuous, and its values range from 0 to 1.

Example: Consider a random variable X representing the number of heads in two coin flips. The possible values are 0, 1, and 2, with probabilities P(X=0) = 1/4, P(X=1) = 1/2, P(X=2) = 1/4.

• F(0) = P(X ≤ 0) = P(X=0) = 1/4
• F(1) = P(X ≤ 1) = P(X=0) + P(X=1) = 1/4 + 1/2 = 3/4
• F(2) = P(X ≤ 2) = P(X=0) + P(X=1) + P(X=2) = 1/4 + 1/2 + 1/4 = 1
• For any x < 0, F(x) = 0. For any x ≥ 2, F(x) = 1.

2.2. Cumulative Distribution Function (CDF) for Continuous Random Variables

For a continuous random variable X, the CDF is defined as:

F(x) = P(X ≤ x) = ∫_-∞^x f(t) dt

Here, f(t) is the probability density function (PDF) of X. The CDF for a continuous random variable is a continuous, non-decreasing function ranging from 0 to 1.

Key properties of CDF F(x):

• 0 ≤ F(x) ≤ 1 for all x.
• F(x) is non-decreasing: If x₁ < x₂, then F(x₁) ≤ F(x₂).
• lim_x→-∞ F(x) = 0
• lim_x→+∞ F(x) = 1

3. Probability Functions and Densities

Probability functions and densities are used to describe the likelihood of specific outcomes for random variables.

3.1. Probability Mass Function (PMF) for Discrete Random Variables

The probability mass function (PMF) gives the probability that a discrete random variable is exactly equal to some value. It is denoted by p(x) or P(X=x).

For a discrete random variable X, the PMF satisfies:

• p(x) ≥ 0 for all x.
• Σ_x p(x) = 1, where the sum is over all possible values of x.

Example: A fair six-sided die is rolled. The random variable X is the outcome. The PMF is p(x) = 1/6 for x ∈ {1, 2, 3, 4, 5, 6}, and p(x) = 0 otherwise.

3.2. Probability Density Function (PDF) for Continuous Random Variables

For a continuous random variable X, the probability of X taking on any single specific value is zero. Instead, we use the probability density function (PDF), denoted by f(x), to describe the relative likelihood for this random variable to take on a given value.

The PDF f(x) satisfies:

• f(x) ≥ 0 for all x.
• ∫_-∞^+∞ f(x) dx = 1.

The probability that X falls within an interval [a, b] is given by the integral of the PDF over that interval:

P(a ≤ X ≤ b) = ∫_a^b f(x) dx

It's important to note that f(x) itself is not a probability. It's a density, and its value can be greater than 1. The probability is obtained by integrating the density.

Example: A uniform distribution on the interval [0, 1] has a PDF f(x) = 1 for 0 ≤ x ≤ 1, and f(x) = 0 otherwise. The probability of X being between 0.2 and 0.7 is ∫_0.2^0.7 1 dx = [x]_0.2^0.7 = 0.7 - 0.2 = 0.5.

4. Mathematical Expectation

Mathematical expectation, often called the expected value, is the weighted average of all possible values that a random variable can take. The weights are the probabilities of those values. It represents the long-run average value of the random variable if the experiment were repeated many times.

4.1. Expected Value of a Discrete Random Variable

If X is a discrete random variable with PMF p(x) and possible values x₁, x₂, ..., the expected value of X, denoted by E(X) or μ, is:

E(X) = Σ_i x_i * p(x_i)

Example: For the two coin flips (X = number of heads): E(X) = (0 * 1/4) + (1 * 1/2) + (2 * 1/4) = 0 + 1/2 + 1/2 = 1. The expected number of heads is 1.

4.2. Expected Value of a Continuous Random Variable

If X is a continuous random variable with PDF f(x), the expected value of X is:

E(X) = ∫_-∞^+∞ x * f(x) dx

Example: For the uniform distribution on [0, 1] with f(x) = 1 for 0 ≤ x ≤ 1:

E(X) = ∫₀¹ x * 1 dx = ∫₀¹ x dx = [x²/2]₀¹ = (1²/2) - (0²/2) = 1/2. The expected value is 0.5.

4.3. Expected Value of a Function of a Random Variable

If Y = g(X) is a function of a random variable X, then the expected value of Y can be calculated without explicitly finding the distribution of Y.

For a discrete random variable X: E(g(X)) = Σ_i g(x_i) * p(x_i)

For a continuous random variable X: E(g(X)) = ∫_-∞^+∞ g(x) * f(x) dx

This is known as the Law of the Unconscious Statistician (LOTUS).

Example: If X is the outcome of a die roll (PMF p(x)=1/6 for x=1..6) and we want E(X²):

E(X²) = (1² * 1/6) + (2² * 1/6) + (3² * 1/6) + (4² * 1/6) + (5² * 1/6) + (6² * 1/6) E(X²) = (1 + 4 + 9 + 16 + 25 + 36) / 6 = 91 / 6 ≈ 15.17

4.4. Properties of Expectation

• E(c) = c, where c is a constant.
• E(cX) = cE(X), where c is a constant.
• E(X + Y) = E(X) + E(Y) (Linearity of Expectation). This holds regardless of whether X and Y are independent.
• E(aX + bY + c) = aE(X) + bE(Y) + c, where a, b, and c are constants.

5. Marginal and Conditional Distributions

When we have multiple random variables, we often want to understand their individual behaviors or how they relate to each other.

5.1. Joint Distributions

A joint distribution describes the probabilities of two or more random variables occurring simultaneously.

For two discrete random variables X and Y, the joint PMF is denoted by p(x, y) = P(X=x, Y=y).

For two continuous random variables X and Y, the joint PDF is denoted by f(x, y). The probability of (X, Y) falling into a region A is ∫∫_A f(x, y) dx dy.

5.2. Marginal Distributions

The marginal distribution of one variable (e.g., X) from a joint distribution is its individual probability distribution, ignoring the other variable(s).

For discrete random variables X and Y with joint PMF p(x, y):

Marginal PMF of X: p_X(x) = P(X=x) = Σ_y p(x, y) (sum over all possible values of y).

Marginal PMF of Y: p_Y(y) = P(Y=y) = Σ_x p(x, y) (sum over all possible values of x).

For continuous random variables X and Y with joint PDF f(x, y):

Marginal PDF of X: f_X(x) = ∫_-∞^+∞ f(x, y) dy (integrate over all possible values of y).

Marginal PDF of Y: f_Y(y) = ∫_-∞^+∞ f(x, y) dx (integrate over all possible values of x).

Example: Suppose we have a joint PMF for two dice rolls (X, Y) where X is the first die and Y is the second. If X and Y are independent, p(x, y) = p_X(x) * p_Y(y) = (1/6) * (1/6) = 1/36 for all x, y in {1, ..., 6}.

The marginal PMF of X is p_X(x) = Σ_y=1⁶ (1/36) = 6 * (1/36) = 1/6, which is expected for a single die.

5.3. Conditional Distributions

A conditional distribution describes the probability distribution of one random variable given that another random variable has taken a specific value.

For discrete random variables X and Y with joint PMF p(x, y) and marginal PMF p_Y(y) > 0:

Conditional PMF of X given Y=y: p_X|Y(x|y) = P(X=x | Y=y) = p(x, y) / p_Y(y).

Similarly, for Y given X=x: p_Y|X(y|x) = P(Y=y | X=x) = p(x, y) / p_X(x), provided p_X(x) > 0.

For continuous random variables X and Y with joint PDF f(x, y) and marginal PDF f_Y(y) > 0:

Conditional PDF of X given Y=y: f_X|Y(x|y) = f(x, y) / f_Y(y).

Similarly, for Y given X=x: f_Y|X(y|x) = f(x, y) / f_X(x), provided f_X(x) > 0.

Example: Consider two variables X and Y, where X is the number of heads in 2 flips and Y is the result of a third flip (1 for heads, 0 for tails).

Possible (X, Y) pairs are (0,0), (0,1), (1,0), (1,1), (2,0), (2,1).

P(X=0) = 1/4 (TT), P(X=1) = 1/2 (HT, TH), P(X=2) = 1/4 (HH).

P(Y=0) = 1/2, P(Y=1) = 1/2.

If X and Y are independent, joint PMF p(x, y) = p_X(x) * p_Y(y). For example, p(1, 1) = P(X=1 and Y=1) = P(X=1) * P(Y=1) = (1/2) * (1/2) = 1/4.

What is the conditional probability P(X=1 | Y=1)?

p_X|Y(1|1) = p(1, 1) / p_Y(1) = (1/4) / (1/2) = 1/2.

This makes sense: knowing the third flip was heads doesn't change the probability of getting one head in the first two flips.

6. Conditional Expectation

Conditional expectation is the expected value of a random variable calculated with respect to its conditional distribution. It's the expected value of a random variable given that some other random variable(s) have taken specific values.

6.1. Conditional Expectation for Discrete Random Variables

If X and Y are discrete random variables with joint PMF p(x, y), the conditional expectation of X given Y=y is:

E(X | Y=y) = Σ_x x * p_X|Y(x|y) = Σ_x x * [ p(x, y) / p_Y(y) ]

Similarly, E(Y | X=x) = Σ_y y * p_Y|X(y|x).

Example: Using the two dice roll example where X and Y are independent (p(x,y)=1/36):

E(X | Y=3) = Σ_x=1⁶ x * p_X|Y(x|3)

Since X and Y are independent, p_X|Y(x|3) = p_X(x) = 1/6.

E(X | Y=3) = Σ_x=1⁶ x * (1/6) = (1+2+3+4+5+6)/6 = 21/6 = 3.5.

This shows that if X and Y are independent, the conditional expectation of X given Y=y is just the unconditional expectation of X, E(X).

6.2. Conditional Expectation for Continuous Random Variables

If X and Y are continuous random variables with joint PDF f(x, y), the conditional expectation of X given Y=y is:

E(X | Y=y) = ∫_-∞^+∞ x * f_X|Y(x|y) dx = ∫_-∞^+∞ x * [ f(x, y) / f_Y(y) ] dx

Similarly, E(Y | X=x) = ∫_-∞^+∞ y * f_Y|X(y|x) dy.

6.3. Properties of Conditional Expectation

Conditional expectation is itself a random variable because it depends on the value of the conditioning variable (e.g., Y).

• Iterated Expectation (Law of Total Expectation): E[E(X | Y)] = E(X). This is a very powerful property. It states that the expected value of the conditional expectation of X given Y is equal to the overall expected value of X.
• E(c | Y) = c, where c is a constant.
• E(g(Y)X | Y) = g(Y)E(X | Y). If a function of Y is multiplied by X, it can be pulled out of the conditional expectation.
• E(X + Z | Y) = E(X | Y) + E(Z | Y).

Example using Iterated Expectation: Consider the previous example of X (number of heads in 2 flips) and Y (result of 3rd flip). We found E(X | Y=1) = 1/2 and E(X | Y=0) = 1/2.

Using the iterated expectation property:

E[E(X | Y)] = E(X | Y=0) * P(Y=0) + E(X | Y=1) * P(Y=1) E[E(X | Y)] = (1/2) * (1/2) + (1/2) * (1/2) = 1/4 + 1/4 = 1/2.

Let's calculate E(X) directly: X can be 0, 1, 2 with probabilities 1/4, 1/2, 1/4. E(X) = (0 * 1/4) + (1 * 1/2) + (2 * 1/4) = 0 + 1/2 + 1/2 = 1.

Wait, there's a discrepancy! Let's re-evaluate the example. X = number of heads in first 2 flips. Y = result of 3rd flip (1=H, 0=T). X and Y are independent.

P(X=0)=1/4, P(X=1)=1/2, P(X=2)=1/4. E(X)=1.

P(Y=0)=1/2, P(Y=1)=1/2. E(Y)=1/2.

Since X and Y are independent: E(X | Y=y) = E(X) = 1 for any y.

So, E(X | Y=0) = 1 and E(X | Y=1) = 1.

Now, let's check iterated expectation: E[E(X | Y)] = E(X | Y=0) * P(Y=0) + E(X | Y=1) * P(Y=1) E[E(X | Y)] = (1) * (1/2) + (1) * (1/2) = 1/2 + 1/2 = 1.

This matches E(X) = 1. The property holds. The previous calculation error was in assuming E(X|Y=y) was different for different y when X and Y were independent.