Probability Theory

Probability theory is the mathematical framework for quantifying uncertainty. It provides the language and tools to analyze random phenomena, which are ubiquitous in economics, from predicting consumer behavior to understanding market fluctuations. At its core, probability theory deals with events and their likelihood of occurrence.

Basic Concepts

To understand probability, we first need to define some fundamental terms.

Random Experiment

A random experiment is a process or activity whose outcome cannot be predicted with certainty before it is performed, but it is known that all possible outcomes are predetermined. For example, tossing a coin is a random experiment because we don't know if it will land on heads or tails, but we know the only possible outcomes are heads or tails. In economics, a random experiment could be observing the price of a stock tomorrow.

Sample Space (S)

The sample space is the set of all possible outcomes of a random experiment. Each individual outcome is called a sample point. For the coin toss experiment, the sample space is S = {Head, Tail}. If we roll a fair six-sided die, the sample space is S = {1, 2, 3, 4, 5, 6}. In econometrics, if we are observing the income of a household, the sample space is all possible non-negative income values.

Event (E)

An event is a subset of the sample space. It represents a specific outcome or a collection of outcomes in which we are interested. For the die roll experiment, the event of rolling an even number is E = {2, 4, 6}. The event of getting more than 4 is E = {5, 6}. In economics, an event could be "inflation rate is above 5%" or "GDP growth is positive."

Types of Events

• Simple Event (or Elementary Event): An event that consists of exactly one outcome from the sample space. For example, rolling a '3' on a die is a simple event.
• Compound Event: An event that consists of two or more outcomes. For example, rolling an even number on a die {2, 4, 6} is a compound event.
• Certain Event: An event that is sure to occur. Its probability is 1. For example, rolling a number less than 7 on a six-sided die.
• Impossible Event: An event that cannot occur. Its probability is 0. For example, rolling a 7 on a six-sided die.
• Mutually Exclusive Events: Two or more events are mutually exclusive if they cannot occur at the same time. For example, when tossing a coin once, the events "getting a head" and "getting a tail" are mutually exclusive.
• Exhaustive Events: A set of events is exhaustive if at least one of them must occur. In the coin toss example, {Head, Tail} are exhaustive events.
• Independent Events: Two events are independent if the occurrence of one does not affect the probability of the occurrence of the other. For example, tossing a coin twice; the outcome of the first toss does not influence the outcome of the second toss.
• Dependent Events: Two events are dependent if the occurrence of one affects the probability of the occurrence of the other. For example, drawing two cards from a deck without replacement. The probability of drawing a second card of a certain rank depends on what the first card was.

Probability of an Event

The probability of an event E, denoted by P(E), is a measure of the likelihood that the event will occur. It is a number between 0 and 1, inclusive.

There are several ways to define and calculate probability:

1. Classical (or Theoretical) Probability

This definition applies when all outcomes in the sample space are equally likely. The probability of an event E is the ratio of the number of favorable outcomes to the total number of possible outcomes.

$$ P(E) = \frac{\text{Number of outcomes favorable to E}}{\text{Total number of possible outcomes}} = \frac{n(E)}{n(S)} $$ where $n(E)$ is the number of outcomes in event E, and $n(S)$ is the total number of outcomes in the sample space.

Example: When rolling a fair die, the probability of rolling a 4 is $P(\text{rolling a 4}) = \frac{1}{6}$ because there is one favorable outcome (rolling a 4) and six total possible outcomes {1, 2, 3, 4, 5, 6}.

2. Empirical (or Statistical) Probability

This definition is based on observed data from experiments or past events. The probability of an event E is the ratio of the number of times the event occurred to the total number of trials.

$$ P(E) = \frac{\text{Number of times event E occurred}}{\text{Total number of trials}} $$ This is often used when outcomes are not equally likely or when the sample space is infinite.

Example: If a factory produces 1000 light bulbs and 10 are found to be defective, the empirical probability of a light bulb being defective is $P(\text{defective}) = \frac{10}{1000} = 0.01$. This is frequently used in economics for forecasting or risk assessment based on historical data.

3. Subjective Probability

This is a personal assessment of the likelihood of an event occurring, based on individual beliefs, experience, and intuition. It is often used when there is no objective data available. For instance, an economist might assign a subjective probability to the likelihood of a recession based on their expert judgment.

Axioms of Probability

Probability theory is built upon three fundamental axioms, proposed by Andrey Kolmogorov:

1. Non-negativity: The probability of any event E is non-negative. $P(E) \ge 0$ for all events E.
2. Normalization: The probability of the sample space S is 1. $P(S) = 1$. This means that one of the possible outcomes must occur.
3. Additivity: For any sequence of mutually exclusive events $E_1, E_2, E_3, \dots$, the probability that at least one of them occurs is the sum of their individual probabilities. $P(E_1 \cup E_2 \cup E_3 \cup \dots) = P(E_1) + P(E_2) + P(E_3) + \dots$.

Basic Theorems of Probability

From these axioms, several important theorems can be derived:

• Probability of the Impossible Event: $P(\emptyset) = 0$, where $\emptyset$ is the empty set (impossible event).
• Complement Rule: The probability of the complement of an event E (denoted as $E^c$ or $E'$) is $P(E^c) = 1 - P(E)$. The complement of an event E is the event that E does not occur.
• Addition Rule for Two Events: For any two events A and B, the probability that either A or B or both occur is given by: $$ P(A \cup B) = P(A) + P(B) - P(A \cap B) $$ where $P(A \cap B)$ is the probability that both A and B occur. If A and B are mutually exclusive, then $P(A \cap B) = 0$, and the formula simplifies to $P(A \cup B) = P(A) + P(B)$.

Example: Suppose in a class, 40% of students like Math, 30% like Statistics, and 10% like both. What is the probability that a randomly selected student likes Math or Statistics? Let M be the event "likes Math" and S be the event "likes Statistics". $P(M) = 0.40$, $P(S) = 0.30$, $P(M \cap S) = 0.10$. Using the addition rule: $P(M \cup S) = P(M) + P(S) - P(M \cap S) = 0.40 + 0.30 - 0.10 = 0.60$. So, 60% of students like Math or Statistics.

Conditional Probability

Conditional probability measures the probability of an event occurring given that another event has already occurred. It is denoted by $P(A|B)$, the probability of event A given that event B has occurred.

The formula for conditional probability is: $$ P(A|B) = \frac{P(A \cap B)}{P(B)} $$ provided that $P(B) > 0$.

Example: In the previous example, what is the probability that a student likes Math given that they like Statistics? $P(M|S) = \frac{P(M \cap S)}{P(S)} = \frac{0.10}{0.30} = \frac{1}{3} \approx 0.333$. This means that if we know a student likes Statistics, there is a 33.3% chance they also like Math.

Multiplication Rule

The multiplication rule is derived from the conditional probability formula and is used to find the probability of the intersection of two events.

For any two events A and B: $$ P(A \cap B) = P(A|B)P(B) $$ or $$ P(A \cap B) = P(B|A)P(A) $$

If A and B are independent events, then $P(A|B) = P(A)$ and $P(B|A) = P(B)$, so the rule simplifies to $P(A \cap B) = P(A)P(B)$.

Bayes' Theorem

Bayes' theorem is a fundamental result in probability theory that describes how to update the probability of a hypothesis based on new evidence. It relates conditional probabilities and is crucial in econometrics for updating beliefs.

If we have an event A and a partition of the sample space $B_1, B_2, \dots, B_n$ (meaning the $B_i$ are mutually exclusive and their union is S), then Bayes' theorem states: $$ P(B_i|A) = \frac{P(A|B_i)P(B_i)}{P(A)} $$ The term $P(A)$ in the denominator can be expanded using the law of total probability: $$ P(A) = \sum_{j=1}^{n} P(A|B_j)P(B_j) $$ So, the expanded form is: $$ P(B_i|A) = \frac{P(A|B_i)P(B_i)}{\sum_{j=1}^{n} P(A|B_j)P(B_j)} $$

Example: Suppose there are two factories, Factory 1 (F1) and Factory 2 (F2), producing a certain product. Factory 1 produces 60% of the output, and Factory 2 produces 40%. The defect rate for Factory 1 is 3%, and for Factory 2 is 5%. If a randomly selected product is found to be defective, what is the probability it came from Factory 1? Let D be the event "defective product". We are given: $P(F1) = 0.60$, $P(F2) = 0.40$ $P(D|F1) = 0.03$, $P(D|F2) = 0.05$ We want to find $P(F1|D)$. Using Bayes' Theorem: $P(F1|D) = \frac{P(D|F1)P(F1)}{P(D|F1)P(F1) + P(D|F2)P(F2)}$ $P(F1|D) = \frac{(0.03)(0.60)}{(0.03)(0.60) + (0.05)(0.40)}$ $P(F1|D) = \frac{0.018}{0.018 + 0.020} = \frac{0.018}{0.038} = \frac{18}{38} = \frac{9}{19} \approx 0.474$ So, if a product is defective, there is approximately a 47.4% chance it came from Factory 1.

Probability Distributions

A probability distribution describes the likelihood of obtaining each possible value that a random variable can take. Random variables can be either discrete or continuous.

Discrete Probability Distributions

A discrete random variable is one that can only take on a finite number of values or a countably infinite number of values (like integers). A discrete probability distribution is typically represented by a probability mass function (PMF).

Probability Mass Function (PMF)

For a discrete random variable X, the PMF, denoted by $P(X=x)$, gives the probability that X is exactly equal to some value x. The PMF must satisfy two conditions:

• $P(X=x) \ge 0$ for all possible values of x.
• $\sum P(X=x) = 1$ over all possible values of x.

Common Discrete Distributions

1. Bernoulli Distribution: This distribution describes a single trial with two possible outcomes, usually labeled "success" (with probability p) and "failure" (with probability 1-p).
   • Random Variable X: 1 for success, 0 for failure.
   • PMF: $P(X=1) = p$, $P(X=0) = 1-p$.
   • Mean (Expected Value): $E(X) = p$.
   • Variance: $Var(X) = p(1-p)$.
   Example: A single coin toss (Head = success, Tail = failure, $p=0.5$). A consumer deciding to buy a product or not (Buy = success, Don't buy = failure).
2. Binomial Distribution: This distribution describes the number of successes in a fixed number of independent Bernoulli trials, each with the same probability of success.
   • Random Variable X: Number of successes in n trials.
   • Parameters: n (number of trials), p (probability of success in a single trial).
   • PMF: $P(X=k) = \binom{n}{k} p^k (1-p)^{n-k}$, for $k = 0, 1, \dots, n$. Here, $\binom{n}{k} = \frac{n!}{k!(n-k)!}$ is the binomial coefficient, representing the number of ways to choose k successes from n trials.
   • Mean: $E(X) = np$.
   • Variance: $Var(X) = np(1-p)$.
   Example: The number of heads in 10 coin tosses. The number of defective items in a batch of 50, if the probability of a single item being defective is 0.05. In economics, this could model the number of firms in a sample that increase prices, given a certain probability.
   Binomial Memory Trick: Think "BInomial" for "Between n trials" and "p" for probability.
3. Poisson Distribution: This distribution describes the number of events occurring in a fixed interval of time or space, if these events occur with a known constant average rate and independently of the time since the last event. It's often used for rare events.
   • Random Variable X: Number of events in an interval.
   • Parameter: $\lambda$ (lambda), the average number of events in the interval ($\lambda > 0$).
   • PMF: $P(X=k) = \frac{e^{-\lambda} \lambda^k}{k!}$, for $k = 0, 1, 2, \dots$. Here, $e$ is the base of the natural logarithm (approximately 2.71828).
   • Mean: $E(X) = \lambda$.
   • Variance: $Var(X) = \lambda$.
   Example: The number of customers arriving at a store per hour. The number of traffic accidents in a city per week. The number of typos on a page. In economics, it can model the number of defaults on loans per month.
   Poisson Memory Trick: Think "Poisson" for "POssibly rare" events, and remember Mean = Variance = $\lambda$.

Continuous Probability Distributions

A continuous random variable can take any value within a given range. Its probability distribution is described by a probability density function (PDF).

Probability Density Function (PDF)

For a continuous random variable X, the PDF, denoted by $f(x)$, describes the relative likelihood for X to take on a given value. Unlike PMF, $f(x)$ is not the probability of X=x (which is zero for continuous variables). Instead, the probability that X falls within an interval [a, b] is given by the integral of the PDF over that interval:

$$ P(a \le X \le b) = \int_{a}^{b} f(x) dx $$ The PDF must satisfy two conditions:

• $f(x) \ge 0$ for all x.
• $\int_{-\infty}^{\infty} f(x) dx = 1$.

Cumulative Distribution Function (CDF)

The CDF, denoted by $F(x)$, gives the probability that the random variable X takes on a value less than or equal to x.

$$ F(x) = P(X \le x) = \int_{-\infty}^{x} f(t) dt $$ For a continuous random variable, $P(X \le x) = P(X < x)$. Also, $P(a \le X \le b) = F(b) - F(a)$.

Common Continuous Distributions

1. Uniform Distribution: This distribution describes a situation where all values within a given interval [a, b] are equally likely.
   • Parameters: a (lower bound), b (upper bound).
   • PDF: $f(x) = \frac{1}{b-a}$ for $a \le x \le b$, and 0 otherwise.
   • Mean: $E(X) = \frac{a+b}{2}$.
   • Variance: $Var(X) = \frac{(b-a)^2}{12}$.
   Example: The time a bus arrives, assuming it arrives at any point within a 15-minute interval with equal probability. The random selection of a number between 0 and 1.
2. Exponential Distribution: This distribution describes the time until an event occurs in a Poisson process, i.e., the time between events in a sequence of events occurring randomly in time. It is memoryless.
   • Parameter: $\lambda$ (rate parameter, usually related to the mean number of events per unit time, $\lambda > 0$). The PDF is often written in terms of the mean $\mu = 1/\lambda$.
   • PDF: $f(x) = \lambda e^{-\lambda x}$ for $x \ge 0$. Or $f(x) = \frac{1}{\mu} e^{-x/\mu}$ for $x \ge 0$.
   • Mean: $E(X) = \frac{1}{\lambda} = \mu$.
   • Variance: $Var(X) = \frac{1}{\lambda^2} = \mu^2$.
   Example: The time between customer arrivals. The lifespan of a component. The time until the next phone call. In economics, it can model the duration of unemployment spells.
   Exponential Memory Trick: Think "Exponential" for "Exponential decay" in the PDF, and remember it's memoryless (past failures don't affect future probabilities). Mean = $\mu$, Variance = $\mu^2$.
3. Normal Distribution (Gaussian Distribution): This is arguably the most important distribution in statistics and probability. It is bell-shaped and symmetric. Many natural phenomena and economic variables approximate a normal distribution.
   • Parameters: $\mu$ (mean), $\sigma^2$ (variance), where $\sigma > 0$ is the standard deviation.
   • PDF: $f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}$, for $-\infty < x < \infty$.
   • Mean: $E(X) = \mu$.
   • Variance: $Var(X) = \sigma^2$.
   Standard Normal Distribution: A special case where $\mu = 0$ and $\sigma^2 = 1$. Denoted by Z. Its PDF is $f(z) = \frac{1}{\sqrt{2\pi}} e^{-z^2/2}$. Probabilities for any normal distribution can be found by standardizing it: $Z = \frac{X-\mu}{\sigma}$.
   Normal Distribution Properties:

   • Symmetric around the mean ($\mu$).
   • The mean, median, and mode are all equal.
   • The total area under the curve is 1.
   • The curve approaches the x-axis asymptotically.
   • The Empirical Rule (68-95-99.7 Rule):
     • Approx. 68% of data falls within 1 standard deviation of the mean ($[\mu-\sigma, \mu+\sigma]$).
     • Approx. 95% of data falls within 2 standard deviations ($[\mu-2\sigma, \mu+2\sigma]$).
     • Approx. 99.7% of data falls within 3 standard deviations ($[\mu-3\sigma, \mu+3\sigma]$).
   Example: Heights of people, measurement errors, IQ scores. In economics, many macroeconomic variables like inflation rates or growth rates are often modeled as normally distributed, especially in the short term.
   Normal Memory Trick: Bell curve, symmetric, mean=median=mode. Remember the 68-95-99.7 rule for standard deviations. Z-score: $Z = (X-\mu)/\sigma$.
4. Chi-Squared ($\chi^2$) Distribution: This distribution is important in inferential statistics, particularly in hypothesis testing for variance and goodness-of-fit tests. It is related to the normal distribution.
   • Parameter: k (degrees of freedom, $k > 0$).
   • It is the distribution of the sum of $k$ squared independent standard normal random variables.
   • Mean: $E(X) = k$.
   • Variance: $Var(X) = 2k$.
   • It is always non-negative and right-skewed (especially for small k).
   Example: Used in testing if a sample variance is equal to a hypothesized population variance.
5. t-Distribution (Student's t-distribution): Similar to the normal distribution but with heavier tails, making it more robust to outliers. It is used in hypothesis testing when the population standard deviation is unknown and the sample size is small.
   • Parameter: k (degrees of freedom, $k > 0$).
   • It approaches the standard normal distribution as k approaches infinity.
   • Mean: $E(X) = 0$ (for $k>1$).
   • Variance: $Var(X) = \frac{k}{k-2}$ (for $k>2$).
   Example: Used in t-tests for comparing means when population variance is unknown.
6. F-Distribution: Used primarily in analysis of variance (ANOVA) and regression analysis to compare variances of two populations.
   • Parameters: $k_1$ (numerator degrees of freedom), $k_2$ (denominator degrees of freedom).
   • It is the distribution of the ratio of two independent chi-squared random variables, scaled by their degrees of freedom.
   • It is always non-negative and right-skewed.
   Example: Used in testing if the variances of two normal populations are equal.

Moments of a Distribution

Moments are statistical measures that describe the shape and characteristics of a probability distribution. They generalize concepts like mean, variance, skewness, and kurtosis.

Moments about the Origin (Raw Moments)

The k-th moment about the origin for a random variable X is the expected value of $X^k$.

For a discrete random variable: $\mu_k' = E(X^k) = \sum x^k P(X=x)$. For a continuous random variable: $\mu_k' = E(X^k) = \int_{-\infty}^{\infty} x^k f(x) dx$.

• The 1st moment about the origin is the mean: $E(X) = \mu_1'$.
• The 2nd moment about the origin is $E(X^2)$.

Moments about the Mean (Central Moments)

The k-th moment about the mean for a random variable X is the expected value of $(X - \mu)^k$, where $\mu = E(X)$ is the mean.

For a discrete random variable: $\mu_k = E[(X-\mu)^k] = \sum (x-\mu)^k P(X=x)$. For a continuous random variable: $\mu_k = E[(X-\mu)^k] = \int_{-\infty}^{\infty} (x-\mu)^k f(x) dx$.

• The 1st central moment is always zero: $E(X-\mu) = E(X) - E(\mu) = \mu - \mu = 0$.
• The 2nd central moment is the variance: $\mu_2 = Var(X) = E[(X-\mu)^2]$. This measures the spread or dispersion of the distribution.
• The 3rd central moment is related to skewness.
• The 4th central moment is related to kurtosis.

Skewness

Skewness measures the asymmetry of the probability distribution.

• Positive Skewness (Right Skewed): The tail on the right side of the distribution is longer or fatter than the left side. The mean is typically greater than the median, which is greater than the mode. The 3rd central moment ($\mu_3$) is positive.
• Negative Skewness (Left Skewed): The tail on the left side is longer or fatter than the right side. The mean is typically less than the median, which is less than the mode. The 3rd central moment ($\mu_3$) is negative.
• Zero Skewness: The distribution is symmetric (like the normal distribution). The 3rd central moment ($\mu_3$) is zero.

The standardized skewness is often calculated as: $$ \text{Skewness} = \frac{E[(X-\mu)^3]}{\sigma^3} = \frac{\mu_3}{\sigma^3} $$

Kurtosis

Kurtosis measures the "tailedness" or "peakedness" of the probability distribution relative to a normal distribution.

• Mesokurtic: A distribution with kurtosis equal to that of the normal distribution. The kurtosis value is 3 (using the raw 4th moment definition) or 0 (using the excess kurtosis definition).
• Leptokurtic: A distribution with kurtosis greater than that of the normal distribution. It has heavier tails and a sharper peak. The 4th central moment ($\mu_4$) is large relative to $\sigma^4$.
• Platykurtic: A distribution with kurtosis less than that of the normal distribution. It has lighter tails and a flatter peak. The 4th central moment ($\mu_4$) is small relative to $\sigma^4$.

The most common measure is excess kurtosis, which is Kurtosis - 3. $$ \text{Excess Kurtosis} = \frac{E[(X-\mu)^4]}{\sigma^4} - 3 = \frac{\mu_4}{\sigma^4} - 3 $$ A normal distribution has an excess kurtosis of 0. Leptokurtic distributions have positive excess kurtosis, and platykurtic distributions have negative excess kurtosis.

Example in Economics: Financial returns often exhibit leptokurtosis (positive excess kurtosis), meaning extreme events (large gains or losses) are more likely than predicted by a normal distribution. This is known as the "fat tails" phenomenon.

Central Limit Theorem (CLT)

The Central Limit Theorem is one of the most profound and useful theorems in probability and statistics. It states that, under certain conditions, the distribution of the sample means (or sums) of independent random variables approaches a normal distribution, regardless of the original distribution of the variables themselves, as the sample size gets large enough.

Statement of the Theorem

Let $X_1, X_2, \dots, X_n$ be a sequence of $n$ independent and identically distributed (i.i.d.) random variables, each with a finite mean $\mu$ and a finite variance $\sigma^2$. Let $\bar{X}_n$ be the sample mean: $$ \bar{X}_n = \frac{X_1 + X_2 + \dots + X_n}{n} $$ As $n$ approaches infinity ($n \to \infty$), the distribution of the standardized sample mean approaches the standard normal distribution: $$ Z_n = \frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} N(0, 1) $$ where $\xrightarrow{d}$ denotes convergence in distribution.

This means that for a sufficiently large sample size $n$, the random variable $Z_n$ can be approximated by a standard normal distribution $N(0, 1)$.

Alternatively, the sum $S_n = X_1 + X_2 + \dots + X_n$ also approaches a normal distribution with mean $n\mu$ and variance $n\sigma^2$: $$ S_n \xrightarrow{d} N(n\mu, n\sigma^2) $$

Conditions for the CLT

The key conditions for the CLT to hold are:

• The random variables must be independent.
• The random variables must be identically distributed (drawn from the same distribution).
• Each random variable must have a finite mean ($\mu$) and a finite variance ($\sigma^2$).

Significance and Applications in Economics

The CLT is fundamental to statistical inference and econometrics for several reasons:

• Justification for Normal Distribution: It explains why the normal distribution appears so frequently in nature and social sciences. Many observed phenomena are the result of the sum or average of many small, independent effects. For example, a person's income might be influenced by numerous small factors (education, luck, job market conditions, etc.), leading to a distribution that approximates normality.
• Statistical Inference: It allows us to make inferences about population parameters (like the population mean $\mu$) using sample statistics (like the sample mean $\bar{X}_n$). Even if the population distribution is unknown or non-normal, we can use the normal distribution to construct confidence intervals and perform hypothesis tests for the mean, provided the sample size is large enough.
• Regression Analysis: In econometrics, the CLT is implicitly used when assuming that the error terms in a regression model are normally distributed, or when using large-sample properties of estimators.
• Approximation: It provides a powerful approximation tool. We can approximate the distribution of sample means from non-normal distributions (like uniform, exponential, or even skewed distributions) using the normal distribution.

Rule of Thumb for Sample Size

What constitutes a "sufficiently large" sample size ($n$) depends on the underlying distribution.

• If the original distribution is already close to normal, $n$ as small as 10 or 15 might be sufficient.
• If the original distribution is symmetric but not normal, $n$ around 30 is often considered adequate.
• If the original distribution is highly skewed or has heavy tails, a much larger sample size (e.g., $n > 30$ or even $n > 50$) may be required for the normal approximation to be reliable.

Example: Suppose we want to estimate the average income of a city. We know incomes are typically right-skewed (many people with moderate incomes, a few with very high incomes). If we take a sample of 100 people, the CLT tells us that the distribution of the *sample means* of income will be approximately normal, even though the distribution of individual incomes is not. This allows us to use normal distribution theory to calculate confidence intervals for the average income.