Bayes' Theorem and Simple Probability Distributions

Bayes' Theorem

Bayes' theorem is a fundamental concept in probability theory that describes how to update the probability of a hypothesis based on new evidence. It's named after Reverend Thomas Bayes, an 18th-century statistician and theologian.

In essence, Bayes' theorem allows us to revise our existing beliefs (prior probabilities) in light of new data to arrive at a new, updated belief (posterior probability).

Formal Statement of Bayes' Theorem

Let A and B be two events. Bayes' theorem states that the conditional probability of event A occurring given that event B has occurred is:

$P(A|B) = \frac{P(B|A) P(A)}{P(B)}$

Where:

• $P(A|B)$ is the posterior probability: the probability of hypothesis A given the evidence B.
• $P(B|A)$ is the likelihood: the probability of evidence B given that hypothesis A is true.
• $P(A)$ is the prior probability: the initial probability of hypothesis A before observing any evidence.
• $P(B)$ is the probability of the evidence B: the total probability of observing the evidence, regardless of the hypothesis.

Expanding the Denominator $P(B)$

The term $P(B)$ in the denominator is often expanded using the law of total probability. If we have a set of mutually exclusive and exhaustive hypotheses ($A_1, A_2, ..., A_n$) that cover all possibilities, then $P(B)$ can be calculated as:

$P(B) = P(B|A_1)P(A_1) + P(B|A_2)P(A_2) + ... + P(B|A_n)P(A_n)$

This can be written more compactly using summation notation:

$P(B) = \sum_{i=1}^{n} P(B|A_i)P(A_i)$

Substituting this expanded form into Bayes' theorem gives us:

$P(A_i|B) = \frac{P(B|A_i) P(A_i)}{\sum_{j=1}^{n} P(B|A_j)P(A_j)}$

Understanding the Components with an Example

Imagine a factory produces light bulbs. There are two machines, Machine X and Machine Y. Machine X produces 60% of the bulbs, and 5% of its bulbs are defective. Machine Y produces 40% of the bulbs, and 10% of its bulbs are defective.

Let:

• A be the event that a bulb was produced by Machine X.
• B be the event that a bulb was produced by Machine Y.
• D be the event that a bulb is defective.

We are given the following probabilities:

• $P(A) = 0.60$ (Machine X produces 60% of bulbs)
• $P(B) = 0.40$ (Machine Y produces 40% of bulbs)
• $P(D|A) = 0.05$ (5% of bulbs from Machine X are defective)
• $P(D|B) = 0.10$ (10% of bulbs from Machine Y are defective)

Now, suppose we pick a bulb at random, and it turns out to be defective. We want to find the probability that this defective bulb was produced by Machine X. This is $P(A|D)$.

First, we need to calculate $P(D)$, the overall probability of a bulb being defective. Using the law of total probability:

$P(D) = P(D|A)P(A) + P(D|B)P(B)$

$P(D) = (0.05)(0.60) + (0.10)(0.40)$

$P(D) = 0.030 + 0.040$

$P(D) = 0.070$

So, 7% of all bulbs produced are defective.

Now, we can apply Bayes' theorem to find $P(A|D)$:

$P(A|D) = \frac{P(D|A) P(A)}{P(D)}$

$P(A|D) = \frac{(0.05)(0.60)}{0.070}$

$P(A|D) = \frac{0.030}{0.070}$

$P(A|D) \approx 0.4286$

This means that if we find a defective bulb, there is approximately a 42.86% chance it was produced by Machine X. Initially, we thought there was a 60% chance a bulb came from Machine X. The evidence (the bulb being defective) has updated our belief, making it less likely that the bulb came from Machine X, because Machine Y has a higher defect rate.

Applications of Bayes' Theorem

Bayes' theorem has wide-ranging applications:

• Spam Filtering: Email clients use it to classify emails as spam or not spam based on the words they contain.
• Medical Diagnosis: Doctors use it to update the probability of a disease given a patient's symptoms and test results.
• Machine Learning: It's the basis for Naive Bayes classifiers, used in various classification tasks.
• Finance: Used for risk assessment and portfolio management.
• Scientific Research: Used to update hypotheses based on experimental data.

Simple Probability Distributions

A probability distribution describes the likelihood of obtaining the possible values that a random variable can take. For example, if you roll a die, the probability distribution tells you the chance of getting a 1, a 2, etc.

We can classify probability distributions into two main types:

• Discrete Probability Distributions: For random variables that can only take a finite number of values or a countably infinite number of values (e.g., number of heads in three coin flips, number of cars passing a point in an hour).
• Continuous Probability Distributions: For random variables that can take any value within a given range (e.g., height of a person, temperature).

We will focus on some simple, commonly used discrete probability distributions.

1. Bernoulli Distribution

The Bernoulli distribution models a single trial of an experiment that has only two possible outcomes: success or failure. It's the simplest discrete probability distribution.

Let X be a Bernoulli random variable. The probability mass function (PMF) is:

• $P(X=1) = p$ (Probability of success)
• $P(X=0) = 1-p$ (Probability of failure)

Here, 'p' is the parameter of the distribution, representing the probability of success, where $0 \le p \le 1$.

Examples:

• Flipping a coin once (Heads is success, Tails is failure).
• A single free-throw attempt in basketball (Made is success, Missed is failure).
• A single test of a manufactured item (Defective is success, Not defective is failure).

Mean (Expected Value): $E(X) = p$

Variance: $Var(X) = p(1-p)$

2. Binomial Distribution

The Binomial distribution is an extension of the Bernoulli distribution. It describes the probability of obtaining a specific number of successes in a fixed number of independent Bernoulli trials, where each trial has the same probability of success.

Let X be a Binomial random variable, denoted as $X \sim B(n, p)$, where:

• 'n' is the number of trials.
• 'p' is the probability of success on each individual trial.

The probability mass function (PMF) for the Binomial distribution is:

$P(X=k) = \binom{n}{k} p^k (1-p)^{n-k}$

Where:

• $k$ is the number of successes ($0 \le k \le n$).
• $\binom{n}{k}$ is the binomial coefficient, read as "n choose k", calculated as $\frac{n!}{k!(n-k)!}$. It represents the number of ways to choose $k$ successes from $n$ trials.
• $p^k$ is the probability of getting $k$ successes.
• $(1-p)^{n-k}$ is the probability of getting $n-k$ failures.

Conditions for a Binomial Distribution:

1. The experiment consists of a fixed number of trials, $n$.
2. Each trial has only two possible outcomes: success or failure.
3. The probability of success, $p$, is the same for each trial.
4. The trials are independent of each other.

Example:

What is the probability of getting exactly 3 heads in 5 coin flips, assuming a fair coin?

• Here, $n=5$ (number of flips).
• $p=0.5$ (probability of getting heads on a fair coin).
• $k=3$ (number of desired heads).

Using the Binomial PMF:

$P(X=3) = \binom{5}{3} (0.5)^3 (1-0.5)^{5-3}$

$P(X=3) = \frac{5!}{3!(5-3)!} (0.5)^3 (0.5)^2$

$P(X=3) = \frac{5 \times 4}{2 \times 1} (0.5)^5$

$P(X=3) = 10 \times 0.03125$

$P(X=3) = 0.3125$

So, there is a 31.25% chance of getting exactly 3 heads in 5 flips.

Mean (Expected Value): $E(X) = np$

Variance: $Var(X) = np(1-p)$

3. Poisson Distribution

The Poisson distribution is used to model the probability of a given number of events occurring in a fixed interval of time or space, if these events occur with a known constant mean rate and independently of the time since the last event.

Let X be a Poisson random variable, denoted as $X \sim Poisson(\lambda)$, where:

• $\lambda$ (lambda) is the average number of events in the given interval (also known as the rate parameter).

The probability mass function (PMF) for the Poisson distribution is:

$P(X=k) = \frac{\lambda^k e^{-\lambda}}{k!}$

Where:

• $k$ is the number of occurrences of the event ($k = 0, 1, 2, ...$).
• $\lambda$ is the average rate of occurrence.
• $e$ is Euler's number, approximately 2.71828.
• $k!$ is the factorial of $k$.

Conditions for a Poisson Distribution:

1. Events occur one at a time.
2. The occurrence of an event does not affect the probability of another event occurring.
3. The average rate of events ($\lambda$) is constant over the interval.

Example:

Suppose a call center receives an average of 5 calls per hour. What is the probability that in a given hour, the center receives exactly 7 calls?

• Here, $\lambda = 5$ (average calls per hour).
• $k=7$ (number of calls we are interested in).

Using the Poisson PMF:

$P(X=7) = \frac{5^7 e^{-5}}{7!}$

$P(X=7) = \frac{78125 \times e^{-5}}{5040}$

Using $e^{-5} \approx 0.006738$:

$P(X=7) \approx \frac{78125 \times 0.006738}{5040}$

$P(X=7) \approx \frac{526.40625}{5040}$

$P(X=7) \approx 0.1044$

So, there is approximately a 10.44% chance of receiving exactly 7 calls in an hour.

Relationship to Binomial: The Poisson distribution can be used as an approximation to the Binomial distribution when $n$ is very large and $p$ is very small. In such cases, $\lambda \approx np$. This is useful because calculating binomial probabilities with very large $n$ can be computationally intensive.

Mean (Expected Value): $E(X) = \lambda$

Variance: $Var(X) = \lambda$

Notice that for a Poisson distribution, the mean and variance are equal.

Summary Table of Distributions

Distribution	Notation	Parameters	PMF	Mean	Variance	Use Case
Bernoulli	$X \sim Bernoulli(p)$	$p$ (probability of success)	$P(X=1)=p, P(X=0)=1-p$	$p$	$p(1-p)$	Single trial with two outcomes.
Binomial	$X \sim B(n, p)$	$n$ (number of trials), $p$ (prob. of success)	$\binom{n}{k} p^k (1-p)^{n-k}$	$np$	$np(1-p)$	Number of successes in $n$ independent trials.
Poisson	$X \sim Poisson(\lambda)$	$\lambda$ (average rate)	$\frac{\lambda^k e^{-\lambda}}{k!}$	$\lambda$	$\lambda$	Number of events in an interval with a constant average rate.

Connection between Distributions and Bayes' Theorem

Bayes' theorem is a tool for updating probabilities. Probability distributions provide the framework for those probabilities. For instance, if we assume our prior belief about a parameter (like 'p' in a Binomial distribution) follows a certain distribution, we can use Bayes' theorem to update that belief based on observed data, resulting in a posterior distribution.

This forms the basis of Bayesian statistics, where instead of fixed parameters, we work with probability distributions for parameters themselves.