Probability Theory and Random Variables

1. Introduction to Probability Theory

Probability theory is a branch of mathematics that studies the phenomena governed by randomness. It provides a framework for quantifying uncertainty and making informed decisions in situations where outcomes are not certain. This field is fundamental to statistics, data science, finance, physics, and many other disciplines.

The basic concept of probability assigns a numerical value between 0 and 1 to the likelihood of an event occurring. A probability of 0 means the event is impossible, while a probability of 1 means the event is certain.

1.1 Basic Concepts

Experiment: An action or process that produces observable outcomes. For example, flipping a coin, rolling a die, or measuring the height of a person.

Sample Space (S): The set of all possible outcomes of an experiment.

• Example: For a coin flip, the sample space is S = {Heads, Tails}.
• Example: For rolling a standard six-sided die, S = {1, 2, 3, 4, 5, 6}.

Event (E): A subset of the sample space, representing a specific outcome or a collection of outcomes.

• Example: If the experiment is rolling a die, the event of getting an even number is E = {2, 4, 6}.
• Example: The event of getting a number greater than 4 is E = {5, 6}.

Elementary Event: An event that consists of a single outcome. For example, rolling a 3 on a die is an elementary event.

Compound Event: An event that consists of more than one outcome. For example, rolling an even number on a die is a compound event.

Complementary Event (E'): The event that E does not occur. The union of an event and its complement covers the entire sample space. P(E') = 1 - P(E).

• Example: If E is the event of rolling an even number {2, 4, 6}, then E' is the event of rolling an odd number {1, 3, 5}.

Impossible Event: An event that cannot occur. Its probability is 0. For example, rolling a 7 on a standard six-sided die.

Certain Event: An event that is sure to occur. Its probability is 1. For example, rolling a number less than 7 on a standard six-sided die.

1.2 Approaches to Probability

There are three main ways to define or interpret probability:

1. Classical (or Theoretical) Probability: This approach assumes all outcomes in the sample space are equally likely. The probability of an event is the ratio of the number of favorable outcomes to the total number of possible outcomes.

   Formula: P(E) = (Number of outcomes favorable to E) / (Total number of possible outcomes)

   Example: The probability of drawing an Ace from a standard deck of 52 cards. There are 4 Aces, and 52 cards in total. So, P(Ace) = 4/52 = 1/13.

   Limitations: This definition is only applicable when all outcomes are equally likely, which is not always the case.

2. Empirical (or Frequentist) Probability: This approach is based on observed data from experiments or past events. The probability of an event is the ratio of the number of times the event occurred to the total number of trials.

   Formula: P(E) = (Number of times event E occurred) / (Total number of trials)

   Example: If a coin is tossed 1000 times and lands heads 510 times, the empirical probability of getting heads is P(Heads) = 510/1000 = 0.51.

   This probability approaches the theoretical probability as the number of trials increases (Law of Large Numbers).

3. Subjective Probability: This approach is based on personal belief, judgment, or intuition. It is used when outcomes are not equally likely and there is no historical data.

   Example: A weather forecaster stating there is a 70% chance of rain tomorrow based on their expertise and current atmospheric conditions.

1.3 Axioms of Probability

Kolmogorov's axioms provide a rigorous mathematical foundation for probability theory. Let S be the sample space and P be a probability measure.

1. Non-negativity: For any event E, P(E) ≥ 0. The probability of an event cannot be negative.
2. Normalization: The probability of the sample space is 1. P(S) = 1. The event that *something* in the sample space occurs is certain.
3. Additivity: For any sequence of mutually exclusive events E₁, E₂, E₃, ..., the probability that at least one of them occurs is the sum of their individual probabilities.

   P(E₁ ∪ E₂ ∪ E₃ ∪ ...) = P(E₁) + P(E₂) + P(E₃) + ...

   For two events E and F: P(E ∪ F) = P(E) + P(F) - P(E ∩ F). If E and F are mutually exclusive (cannot happen at the same time), then P(E ∩ F) = 0, and P(E ∪ F) = P(E) + P(F).

1.4 Rules of Probability

Based on the axioms, several important rules can be derived:

• P(∅) = 0 (The probability of the impossible event is 0).
• P(E') = 1 - P(E) (Probability of the complement).
• If E ⊆ F, then P(E) ≤ P(F).
• P(E ∪ F) = P(E) + P(F) - P(E ∩ F) (General addition rule).
• P(E ∩ F) = P(E|F) * P(F) = P(F|E) * P(E) (Multiplication rule for dependent events).
• P(E ∩ F) = P(E) * P(F) (Multiplication rule for independent events).

1.5 Conditional Probability

Conditional probability is the probability of an event occurring given that another event has already occurred. It is denoted as P(A|B), read as "the probability of A given B".

Formula: P(A|B) = P(A ∩ B) / P(B), provided P(B) > 0.

Example: Consider drawing two cards from a standard deck without replacement. What is the probability that the second card is a King, given the first card was a King?

• Let A be the event that the second card is a King.
• Let B be the event that the first card is a King.
• P(B) = 4/52.
• P(A ∩ B) = Probability that both cards are Kings = (4/52) * (3/51).
• P(A|B) = P(A ∩ B) / P(B) = [(4/52) * (3/51)] / (4/52) = 3/51.

This makes sense: after drawing one King, there are 3 Kings left in the remaining 51 cards.

1.6 Independence of Events

Two events A and B are independent if the occurrence of one does not affect the probability of the other.

Mathematically, A and B are independent if:

• P(A|B) = P(A)
• P(B|A) = P(B)
• P(A ∩ B) = P(A) * P(B)

Example: Flipping a coin twice. The outcome of the first flip does not affect the outcome of the second flip. The events are independent. P(Heads on 2nd flip | Heads on 1st flip) = P(Heads on 2nd flip) = 0.5.

Example: Drawing two cards with replacement from a deck. The first card is put back before drawing the second. The events are independent.

Dependence: If events are not independent, they are dependent. Example: Drawing two cards without replacement. The outcome of the second draw depends on the outcome of the first draw.

1.7 Bayes' Theorem

Bayes' Theorem is a fundamental result in probability theory that describes how to update the probability of a hypothesis based on new evidence. It is widely used in statistical inference, machine learning, and medical diagnosis.

Let A and B be two events. Bayes' Theorem states:

P(A|B) = [P(B|A) * P(A)] / P(B)

Often, P(B) is expanded using the law of total probability. If we have a set of mutually exclusive and exhaustive events B₁, B₂, ..., B_n (a partition of the sample space), then:

P(B) = Σ P(B|B_i) * P(B_i) for i = 1 to n

So, Bayes' Theorem becomes:

P(A|B) = [P(B|A) * P(A)] / Σ P(B|B_i) * P(B_i)

Example: Medical Diagnosis

• Hypothesis A: A patient has a specific disease.
• Evidence B: The patient tests positive for the disease.
• P(A) = Prior probability of having the disease (prevalence in the population). Let's say 0.01 (1%).
• P(B|A) = Probability of testing positive given the patient has the disease (sensitivity of the test). Let's say 0.95 (95%).
• P(B|A') = Probability of testing positive given the patient does NOT have the disease (false positive rate). Let's say 0.05 (5%).
• A' is the event that the patient does NOT have the disease. P(A') = 1 - P(A) = 0.99.
• We want to find P(A|B): the probability the patient actually has the disease given a positive test result.

Using Bayes' Theorem:

P(A|B) = [P(B|A) * P(A)] / [P(B|A) * P(A) + P(B|A') * P(A')]

P(A|B) = [0.95 * 0.01] / [0.95 * 0.01 + 0.05 * 0.99]

P(A|B) = 0.0095 / [0.0095 + 0.0495]

P(A|B) = 0.0095 / 0.0590 ≈ 0.161 or 16.1%

This result shows that even with a positive test, the probability of actually having the disease is only about 16.1% if the disease is rare. This highlights the importance of the prior probability and the false positive rate.

2. Random Variables

A random variable is a variable whose value is a numerical outcome of a random phenomenon. It essentially assigns a number to each outcome in the sample space. Random variables are crucial for modeling real-world data and performing statistical analysis.

2.1 Types of Random Variables

Random variables are broadly classified into two types:

1. Discrete Random Variable: A variable that can only take a finite number of values or a countably infinite number of values. The values are typically integers.
   • Example: The number of heads when flipping a coin 3 times (0, 1, 2, 3).
   • Example: The number of defective items in a sample of 100.
   • Example: The number of cars passing a certain point on a highway in an hour (can be 0, 1, 2, ... infinitely).
2. Continuous Random Variable: A variable that can take any value within a given range or interval. The values are not restricted to specific points.
   • Example: The height of a person.
   • Example: The temperature of a room.
   • Example: The time it takes for a machine to fail.

2.2 Probability Distributions

A probability distribution describes how the probabilities are distributed over the possible values of a random variable.

2.2.1 Probability Distribution for Discrete Random Variables

For a discrete random variable X, its probability distribution is given by the Probability Mass Function (PMF), denoted as P(X=x) or p(x). The PMF must satisfy:

• p(x) ≥ 0 for all possible values x.
• Σ p(x) = 1 (The sum of probabilities over all possible values is 1).

Cumulative Distribution Function (CDF): For any random variable (discrete or continuous), the CDF, denoted F(x), gives the probability that the random variable X takes on a value less than or equal to x.

F(x) = P(X ≤ x)

Properties of CDF:

• 0 ≤ F(x) ≤ 1
• F(x) is non-decreasing.
• lim_x→-∞ F(x) = 0
• lim_x→+∞ F(x) = 1

For a discrete random variable:

F(x) = Σ_t≤x p(t)

2.2.2 Probability Distribution for Continuous Random Variables

For a continuous random variable X, its probability distribution is described by the Probability Density Function (PDF), denoted as f(x). Unlike PMF, f(x) does not represent a probability directly. Instead, the probability of X falling within an interval [a, b] is given by the integral of the PDF over that interval.

The PDF must satisfy:

• f(x) ≥ 0 for all x.
• ∫_-∞^∞ f(x) dx = 1 (The total area under the curve is 1).

Probability for an interval:

P(a ≤ X ≤ b) = ∫_a^b f(x) dx

Important Note: For a continuous random variable, the probability of X taking any single specific value is zero.

P(X = c) = 0 for any constant c.

This means P(a ≤ X ≤ b) = P(a < X ≤ b) = P(a ≤ X < b) = P(a < X < b).

The CDF for a continuous random variable is:

F(x) = P(X ≤ x) = ∫_-∞^x f(t) dt

The PDF can be obtained from the CDF by differentiation: f(x) = F'(x).

2.3 Common Discrete Probability Distributions

These are fundamental distributions used to model various real-world scenarios.

1. Bernoulli Distribution: Models a single trial with two possible outcomes (success or failure).
   • Let X be a Bernoulli random variable with parameter p (probability of success).
   • P(X=1) = p (success)
   • P(X=0) = 1-p (failure)
   • PMF: p(x) = p^x(1-p)^1-x for x = 0, 1.
   • Mean (Expected Value): E[X] = p
   • Variance: Var(X) = p(1-p)
   Shortcut: Think of a single coin flip where 'Heads' is success (p).
2. Binomial Distribution: Models the number of successes in a fixed number of independent Bernoulli trials.
   • Let X ~ Binomial(n, p), where n is the number of trials and p is the probability of success in each trial.
   • X can take values 0, 1, 2, ..., n.
   • PMF: P(X=k) = C(n, k) * p^k * (1-p)^n-k, where C(n, k) = n! / (k! * (n-k)!) is the binomial coefficient.
   • Mean: E[X] = np
   • Variance: Var(X) = np(1-p)
   Shortcut: "Number of successes in 'n' independent trials, each with probability 'p'." C(n,k) is the number of ways to choose k successes out of n trials.
   Example: The probability of getting exactly 3 heads in 5 coin flips (n=5, p=0.5). P(X=3) = C(5, 3) * (0.5)³ * (0.5)² = 10 * 0.125 * 0.25 = 0.3125.
3. Poisson Distribution: Models the number of events occurring in a fixed interval of time or space, given a constant average rate and independence of events. It's often used for rare events.
   • Let X ~ Poisson(λ), where λ (lambda) is the average number of events in the interval.
   • X can take values 0, 1, 2, ... (countably infinite).
   • PMF: P(X=k) = (e^-λ * λ^k) / k!, for k = 0, 1, 2, ...
   • Mean: E[X] = λ
   • Variance: Var(X) = λ
   Shortcut: Use for counts of events over time/space, especially when the probability of an event in a very small interval is proportional to the interval length. Think "number of calls received per hour."
   Example: If a call center receives an average of 10 calls per hour (λ=10), the probability of receiving exactly 5 calls in an hour is P(X=5) = (e^-10 * 10⁵) / 5! ≈ 0.0378. Note: The Binomial distribution can be approximated by the Poisson distribution when n is large and p is small (np = λ).
4. Geometric Distribution: Models the number of trials needed to achieve the first success in a series of independent Bernoulli trials.
   • Let X ~ Geometric(p), where p is the probability of success.
   • X can take values 1, 2, 3, ... (number of trials).
   • PMF: P(X=k) = (1-p)^k-1 * p, for k = 1, 2, 3, ...
   • Mean: E[X] = 1/p
   • Variance: Var(X) = (1-p) / p²
   Shortcut: "How many trials until the FIRST success?" The (k-1) term represents the failures before the first success.
   Example: Probability that the first head appears on the 3rd flip of a fair coin (p=0.5). X=3. P(X=3) = (1-0.5)^3-1 * 0.5 = (0.5)² * 0.5 = 0.125.
5. Hypergeometric Distribution: Models the probability of k successes (random draws for which the object drawn has a specified feature) in n draws, without replacement, from a finite population of size N that contains exactly K objects with that feature.
   • Population size: N
   • Number of success states in population: K
   • Number of draws (sample size): n
   • Number of observed successes: k
   • PMF: P(X=k) = [C(K, k) * C(N-K, n-k)] / C(N, n)
   • Mean: E[X] = n * (K/N)
   • Variance: Var(X) = n * (K/N) * (1 - K/N) * (N-n)/(N-1)
   Shortcut: Use when sampling WITHOUT replacement from a finite population. Think "drawing balls from an urn" or "quality control inspection."
   Example: An urn contains 10 red balls and 5 blue balls. If 3 balls are drawn without replacement, what is the probability that exactly 2 are red? (N=15, K=10, n=3, k=2). P(X=2) = [C(10, 2) * C(5, 1)] / C(15, 3) = [45 * 5] / 455 = 225 / 455 ≈ 0.495. Note: When N is very large compared to n, the Hypergeometric distribution can be approximated by the Binomial distribution.

2.4 Common Continuous Probability Distributions

These distributions are essential for modeling continuous measurements.

1. Uniform Distribution: All values within a given interval are equally likely.
   • Let X ~ Uniform(a, b).
   • PDF: f(x) = 1 / (b-a) for a ≤ x ≤ b, and 0 otherwise.
   • Mean: E[X] = (a+b) / 2
   • Variance: Var(X) = (b-a)² / 12
   Shortcut: Think of a random number generator that produces numbers between 'a' and 'b' with equal chance.
   Example: A bus arrives every 30 minutes between 2:00 PM and 2:30 PM. If you arrive at a random time, the arrival time of the bus is Uniform(0, 30) minutes past 2:00 PM. The probability of the bus arriving within the first 10 minutes is P(0 ≤ X ≤ 10) = 10 / 30 = 1/3.
2. Exponential Distribution: Models the time until an event occurs in a Poisson process (i.e., time between events). It has the "memoryless" property.
   • Let X ~ Exponential(λ), where λ is the rate parameter (often related to the rate of events in a Poisson process).
   • PDF: f(x) = λe^-λx for x ≥ 0, and 0 otherwise.
   • CDF: F(x) = 1 - e^-λx for x ≥ 0.
   • Mean: E[X] = 1/λ
   • Variance: Var(X) = 1/λ²
   Shortcut: "Time until the next event" in a process where events happen at a constant average rate (like customer arrivals, radioactive decay). Key property: Memoryless (P(X > s+t | X > s) = P(X > t)).
   Example: If the average time between customers arriving at a store is 5 minutes (λ = 1/5 customers per minute), the probability that the next customer arrives within 3 minutes is P(X ≤ 3) = 1 - e^-(1/5)*3 = 1 - e^-0.6 ≈ 1 - 0.5488 = 0.4512.
3. Normal (Gaussian) Distribution: The most important continuous distribution, characterized by its bell shape. Many natural phenomena follow this distribution.
   • Let X ~ Normal(μ, σ²), where μ is the mean and σ² is the variance (σ is the standard deviation).
   • PDF: f(x) = [1 / (σ√(2π))] * e^{-[ (x-μ)2 / (2σ2) ]}
   • The standard normal distribution has μ=0 and σ=1, denoted Z ~ N(0, 1). Its PDF is φ(z) = [1 / √(2π)] * e^-(z2/2).
   • The CDF of the standard normal distribution is denoted Φ(z).
   • To find probabilities for any normal distribution, we standardize it: Z = (X - μ) / σ. Then P(a ≤ X ≤ b) = P((a-μ)/σ ≤ Z ≤ (b-μ)/σ) = Φ((b-μ)/σ) - Φ((a-μ)/σ).
   • Mean: E[X] = μ
   • Variance: Var(X) = σ²
   Shortcut: The "bell curve." Symmetrical around the mean. Approximately 68% of data falls within 1 standard deviation (μ ± σ), 95% within 2 standard deviations (μ ± 2σ), and 99.7% within 3 standard deviations (μ ± 3σ). Use Z-tables or calculators for probabilities.
   Example: If IQ scores are normally distributed with mean 100 and standard deviation 15, what is the probability that a randomly selected person has an IQ between 85 and 115? Z₁ = (85 - 100) / 15 = -1. Z₂ = (115 - 100) / 15 = 1. P(85 ≤ X ≤ 115) = P(-1 ≤ Z ≤ 1) ≈ 0.68 (from the 68-95-99.7 rule). Using a Z-table, Φ(1) - Φ(-1) ≈ 0.8413 - 0.1587 = 0.6826.
4. Central Limit Theorem (CLT): Not a distribution itself, but a fundamental theorem. It states that the distribution of the sample means (or sums) of a large number of independent, identically distributed random variables will be approximately normally distributed, regardless of the original distribution of the variables.

   If X₁, X₂, ..., X_n are independent and identically distributed random variables with mean μ and finite variance σ², then for large n (typically n ≥ 30), the sample mean X̄ = (Σ X_i) / n is approximately normally distributed with mean μ and variance σ²/n.

   Significance: This is why the normal distribution is so prevalent. It allows us to use normal distribution theory for inference about population means even when the population itself is not normally distributed.

3. Properties of Random Variables

Key characteristics used to describe and compare random variables.

3.1 Expected Value (Mean)

The expected value, denoted E[X] or μ, represents the average value of a random variable over many repetitions of the experiment. It's a measure of the central tendency of the distribution.

• For a discrete random variable: E[X] = Σ [x * P(X=x)] over all possible x.
• For a continuous random variable: E[X] = ∫ [x * f(x)] dx over the entire range of x.

Properties of Expected Value:

• E[c] = c (Expected value of a constant is the constant itself).
• E[cX] = cE[X] (Linearity).
• E[X + Y] = E[X] + E[Y] (Linearity, holds for any random variables X and Y).
• E[X + c] = E[X] + c.
• If X and Y are independent, E[XY] = E[X]E[Y].

Example: For a fair die roll, E[X] = (1 * 1/6) + (2 * 1/6) + ... + (6 * 1/6) = (1+2+3+4+5+6) / 6 = 21/6 = 3.5.

3.2 Variance and Standard Deviation

Variance, denoted Var(X) or σ², measures the spread or dispersion of the random variable's values around its mean. A higher variance indicates greater spread.

Formula: Var(X) = E[(X - μ)²] = E[X²] - (E[X])².

The Standard Deviation, denoted σ, is the square root of the variance. It is often preferred because it is in the same units as the random variable.

σ = √Var(X)

Properties of Variance:

• Var(c) = 0 (Variance of a constant is zero).
• Var(cX) = c²Var(X).
• Var(X + c) = Var(X).
• If X and Y are independent, Var(X + Y) = Var(X) + Var(Y).
• If X and Y are independent, Var(X - Y) = Var(X) + Var(Y).

Example: For a fair die roll, E[X²] = (1²*1/6) + (2²*1/6) + ... + (6²*1/6) = (1+4+9+16+25+36)/6 = 91/6. Var(X) = E[X²] - (E[X])² = 91/6 - (3.5)² = 91/6 - 12.25 = 15.1667 - 12.25 = 2.9167 ≈ 35/12. Standard Deviation σ = √(35/12) ≈ 1.708.

3.3 Covariance

Covariance, denoted Cov(X, Y), measures the degree to which two random variables change together.

Formula: Cov(X, Y) = E[(X - E[X])(Y - E[Y])] = E[XY] - E[X]E[Y].

• If Cov(X, Y) > 0, X and Y tend to move in the same direction (positive correlation).
• If Cov(X, Y) < 0, X and Y tend to move in opposite directions (negative correlation).
• If Cov(X, Y) = 0, X and Y are uncorrelated. This does NOT necessarily mean they are independent, but if they are independent, they are uncorrelated.

Correlation Coefficient (ρ): A standardized measure of linear association, ranging from -1 to +1.

ρ_XY = Cov(X, Y) / (σ_X * σ_Y)

3.4 Moments

Moments are statistical measures that describe the shape and characteristics of a distribution.

• Raw Moments (about the origin): The k-th raw moment is E[X^k].
  • k=1: Mean (E[X]).
  • k=2: E[X²] (used in variance calculation).
• Central Moments (about the mean): The k-th central moment is E[(X - μ)^k].
  • k=1: E[(X - μ)¹] = 0.
  • k=2: E[(X - μ)²] = Variance (σ²).
  • k=3: Measures Skewness (asymmetry of the distribution).
  • k=4: Measures Kurtosis (tailedness or peakedness of the distribution).