Probability - Discrete Sample Space, Events, Classical, Relative Frequency and Axiomatic Approaches, Conditional Probability, Independence, Bayes' Theorem

1. Introduction to Probability

Probability is a branch of mathematics that deals with the quantification of uncertainty. It provides a framework for understanding and analyzing random phenomena. In simple terms, it is the measure of the likelihood that an event will occur. This measure is expressed as a number between 0 and 1, where 0 indicates impossibility and 1 indicates certainty.

The study of probability is fundamental in many fields, including statistics, physics, engineering, economics, and computer science. It allows us to make informed decisions and predictions in situations where outcomes are not deterministic.

2. Sample Space and Events

2.1 Sample Space (S)

The sample space is the set of all possible outcomes of a random experiment. Each outcome is an element of the sample space. We denote the sample space by 'S'.

Example: When a fair coin is tossed, the possible outcomes are 'Heads' (H) and 'Tails' (T). So, the sample space is S = {H, T}.

Example: When a fair six-sided die is rolled, the possible outcomes are the numbers 1, 2, 3, 4, 5, and 6. So, the sample space is S = {1, 2, 3, 4, 5, 6}.

2.2 Discrete Sample Space

A sample space is called discrete if it contains a finite number of outcomes or a countably infinite number of outcomes. This means we can list the outcomes or count them, even if the list is infinitely long.

Example: The number of heads when tossing a coin 10 times. The possible outcomes are 0, 1, 2, ..., 10. This is a finite discrete sample space.

Example: The number of coin tosses required to get the first head. The possible outcomes are 1, 2, 3, ... . This is a countably infinite discrete sample space.

2.3 Events

An event is a subset of the sample space. It represents a specific outcome or a collection of outcomes of interest from a random experiment.

We denote events by capital letters like A, B, C, etc.

Example: Consider rolling a fair die. S = {1, 2, 3, 4, 5, 6}.

Let A be the event of rolling an even number. Then A = {2, 4, 6}.

Let B be the event of rolling a number greater than 4. Then B = {5, 6}.

Let C be the event of rolling a 7. Then C = {} (the empty set), which is an impossible event.

Let D be the event of rolling a number less than 7. Then D = {1, 2, 3, 4, 5, 6} = S, which is a certain event.

2.4 Types of Events

• Simple Event: An event that consists of a single outcome. Example: Rolling a 3 on a die (A = {3}).
• Compound Event: An event that consists of two or more outcomes. Example: Rolling an even number on a die (A = {2, 4, 6}).
• Impossible Event: An event that cannot occur. It is represented by the empty set, {}. Its probability is 0.
• Certain Event: An event that is sure to occur. It is represented by the sample space, S. Its probability is 1.
• Mutually Exclusive Events: Two events A and B are mutually exclusive if they cannot occur at the same time. This means their intersection is empty (A ∩ B = {}).
• Independent Events: Two events A and B are independent if the occurrence of one does not affect the probability of the other.
• Dependent Events: Two events are dependent if the occurrence of one affects the probability of the other.

3. Approaches to Probability

There are several ways to define and calculate probability. The main approaches are classical, relative frequency, and axiomatic.

3.1 Classical Approach (A Priori Probability)

This approach is used when all outcomes in the sample space are equally likely. It is based on logical reasoning before an experiment is conducted.

Definition: If an experiment has 'n' equally likely outcomes, and an event 'A' consists of 'm' of these outcomes, then the probability of event A is given by:

P(A) = (Number of favorable outcomes for A) / (Total number of equally likely outcomes)

P(A) = m / n

Conditions for use:

• The total number of outcomes (n) must be finite.
• All outcomes must be equally likely.
• There should be no bias in the experiment.

Example: What is the probability of drawing a King from a standard deck of 52 playing cards?

Total number of outcomes = 52 (total cards).

Number of favorable outcomes (Kings) = 4.

P(King) = 4 / 52 = 1 / 13.

Limitations: The classical approach cannot be applied if the outcomes are not equally likely (e.g., a biased coin) or if the total number of outcomes is infinite.

3.2 Relative Frequency Approach (Empirical or A Posteriori Probability)

This approach is used when outcomes are not equally likely or when dealing with experiments where the number of outcomes is very large or infinite. It is based on observations from past experiments.

Definition: The probability of an event A is estimated by the relative frequency of its occurrence in a large number of trials.

P(A) ≈ (Number of times event A occurred) / (Total number of trials)

This is often written as P(A) = lim (N→∞) [f(A) / N], where f(A) is the frequency of event A and N is the total number of trials. As the number of trials increases, the relative frequency approaches the true probability.

Example: A factory produces light bulbs. Over the past year, they produced 100,000 bulbs, and 500 of them were found to be defective. What is the probability that a randomly selected bulb is defective?

Number of defective bulbs = 500.

Total number of bulbs = 100,000.

P(Defective) ≈ 500 / 100,000 = 1 / 200 = 0.005.

Conditions for use:

• The experiment must be repeatable under identical conditions.
• The number of trials should be large.

Limitations: This approach provides an estimate, not an exact value. It's also difficult to apply to unique events that cannot be repeated.

3.3 Axiomatic Approach (Modern Approach)

Developed by Andrey Kolmogorov, this is the most general and mathematically rigorous approach to probability. It defines probability based on a set of axioms (fundamental assumptions) rather than definitions.

Let S be the sample space of a random experiment.

A probability measure P is a function that assigns a real number P(A) to each event A in the sample space, satisfying the following axioms:

1. Non-negativity: For any event A, P(A) ≥ 0. (Probability cannot be negative).
2. Normalization: The probability of the sample space is 1. P(S) = 1. (The event that something happens is certain).
3. Additivity (for mutually exclusive events): If A₁, A₂, A₃, ... are a sequence of mutually exclusive events (i.e., Aᵢ ∩ Aⱼ = {} for i ≠ j), then the probability of their union is the sum of their probabilities:

P(A₁ ∪ A₂ ∪ A₃ ∪ ...) = P(A₁) + P(A₂) + P(A₃) + ...

For a finite number of mutually exclusive events A and B:

P(A ∪ B) = P(A) + P(B) if A ∩ B = {}.

Consequences of the Axioms:

• P({}) = 0 (Probability of the impossible event is 0).
• For any event A, 0 ≤ P(A) ≤ 1.
• If A is a subset of B (A ⊆ B), then P(A) ≤ P(B).
• P(A ∪ B) = P(A) + P(B) - P(A ∩ B) (General addition rule for any two events).

The axiomatic approach provides a solid mathematical foundation that encompasses both the classical and relative frequency interpretations as special cases or interpretations.

4. Basic Probability Rules

4.1 Complement Rule

The probability of the complement of an event A (denoted as A' or A^c), which is the event that A does not occur, is given by:

P(A') = 1 - P(A)

Example: If the probability of rain tomorrow is 0.3, the probability of no rain is P(No Rain) = 1 - P(Rain) = 1 - 0.3 = 0.7.

4.2 Addition Rule

This rule helps calculate the probability of the union of two events (A or B occurring).

For mutually exclusive events: If A and B cannot happen together, P(A ∪ B) = P(A) + P(B).

For non-mutually exclusive events: If A and B can happen together, we must subtract the probability of both occurring to avoid double-counting.

P(A ∪ B) = P(A) + P(B) - P(A ∩ B)

Where P(A ∩ B) is the probability that both A and B occur.

Example: What is the probability of drawing an Ace or a Heart from a standard deck of 52 cards?

Let A be the event of drawing an Ace. P(A) = 4/52.

Let H be the event of drawing a Heart. P(H) = 13/52.

The event of drawing an Ace of Hearts (A ∩ H) occurs once. P(A ∩ H) = 1/52.

Since the Ace of Hearts is both an Ace and a Heart, these events are not mutually exclusive.

P(A ∪ H) = P(A) + P(H) - P(A ∩ H)

P(A ∪ H) = 4/52 + 13/52 - 1/52 = 16/52 = 4/13.

5. Conditional Probability

Conditional probability is the probability of an event occurring given that another event has already occurred. It is denoted as P(A|B), which reads "the probability of A given B".

Definition: For two events A and B, where P(B) > 0, the conditional probability of A given B is:

P(A|B) = P(A ∩ B) / P(B)

This formula makes intuitive sense: we are restricting our sample space to only those outcomes where B occurred (the denominator P(B)), and within that restricted space, we are looking at the proportion where A also occurred (the numerator P(A ∩ B)).

Example: Suppose a bag contains 5 red balls and 3 blue balls. Two balls are drawn without replacement.

What is the probability that the second ball drawn is blue, given that the first ball drawn was red?

Let R₁ be the event that the first ball is red.

Let B₂ be the event that the second ball is blue.

We want to find P(B₂|R₁).

Initially, there are 8 balls (5 red, 3 blue).

If the first ball drawn is red (R₁), then there are 7 balls left: 4 red and 3 blue.

The probability of drawing a blue ball as the second ball, given the first was red, is:

P(B₂|R₁) = (Number of blue balls remaining) / (Total number of balls remaining) = 3 / 7.

Using the formula:

P(R₁) = 5/8.

P(R₁ ∩ B₂) = Probability of drawing red first AND blue second = (5/8) * (3/7) = 15/56.

P(B₂|R₁) = P(R₁ ∩ B₂) / P(R₁) = (15/56) / (5/8) = (15/56) * (8/5) = (15 * 8) / (56 * 5) = 120 / 280 = 12 / 28 = 3 / 7. (Matches)

5.1 Multiplication Rule

The formula for conditional probability can be rearranged to find the probability of the intersection of two events:

P(A ∩ B) = P(A|B) * P(B)

Or, symmetrically:

P(A ∩ B) = P(B|A) * P(A)

This rule is useful for calculating the probability of a sequence of events occurring.

Example: Using the previous example, what is the probability that the first ball is red AND the second ball is blue?

P(R₁ ∩ B₂) = P(B₂|R₁) * P(R₁) = (3/7) * (5/8) = 15/56.

6. Independence of Events

Two events A and B are said to be independent if the occurrence of one event does not affect the probability of the other event occurring. Mathematically, A and B are independent if:

P(A|B) = P(A)

And also:

P(B|A) = P(B)

If this condition holds, then substituting into the multiplication rule:

P(A ∩ B) = P(A|B) * P(B) = P(A) * P(B)

So, the key condition for independence is:

P(A ∩ B) = P(A) * P(B)

How to check for independence:

1. Calculate P(A ∩ B).
2. Calculate P(A) and P(B).
3. Check if P(A ∩ B) is equal to P(A) * P(B).

Example: Is drawing a King from a deck of cards independent of drawing a Heart?

Let K be the event of drawing a King. P(K) = 4/52 = 1/13.

Let H be the event of drawing a Heart. P(H) = 13/52 = 1/4.

P(K ∩ H) = Probability of drawing the King of Hearts = 1/52.

Now, check if P(K ∩ H) = P(K) * P(H).

P(K) * P(H) = (1/13) * (1/4) = 1/52.

Since P(K ∩ H) = P(K) * P(H), the events of drawing a King and drawing a Heart are independent.

Example of Dependent Events: Drawing two cards from a deck without replacement.

Let A be the event that the first card is a King. P(A) = 4/52.

Let B be the event that the second card is a King.

If the first card was a King (event A occurred), there are only 3 Kings left and 51 total cards. So, P(B|A) = 3/51.

If the first card was NOT a King (event A did not occur), there are still 4 Kings left and 51 total cards. So, P(B|A') = 4/51.

Since P(B|A) ≠ P(B|A'), the event B is dependent on event A.

Let's calculate P(B). P(B) = P(B|A)P(A) + P(B|A')P(A') = (3/51)*(4/52) + (4/51)*(48/52) = (12 + 192) / (51*52) = 204 / 2652 = 4/52 = 1/13. So P(B)=1/13.

Now check P(A ∩ B) = P(A) * P(B)?

P(A ∩ B) = P(B|A) * P(A) = (3/51) * (4/52) = 12 / 2652 = 1 / 221.

P(A) * P(B) = (4/52) * (4/52) = (1/13) * (1/13) = 1/169.

Since 1/221 ≠ 1/169, the events are dependent.

7. Bayes' Theorem

Bayes' theorem provides a way to update the probability of a hypothesis based on new evidence. It relates conditional probabilities and is derived from the multiplication rule.

Suppose we have an event B (evidence) and we want to find the probability of event A given B, P(A|B). Bayes' theorem states:

P(A|B) = [ P(B|A) * P(A) ] / P(B)

For a situation with multiple mutually exclusive and exhaustive events (hypotheses) H₁, H₂, ..., H<0xE2><0x82><0x99> and an observed evidence B, Bayes' theorem is typically written as:

P(Hᵢ|B) = [ P(B|Hᵢ) * P(Hᵢ) ] / P(B)

Where P(B) can be expanded using the law of total probability:

P(B) = Σ P(B|H<0xE2><0x82><0x9D>) * P(H<0xE2><0x82><0x9D>)

So, the full form is:

P(Hᵢ|B) = [ P(B|Hᵢ) * P(Hᵢ) ] / [ Σ P(B|H<0xE2><0x82><0x9D>) * P(H<0xE2><0x82><0x9D>) ]

Key Terms:

• P(Hᵢ): Prior probability of hypothesis Hᵢ (our belief before seeing evidence B).
• P(B|Hᵢ): Likelihood of observing evidence B given hypothesis Hᵢ is true.
• P(Hᵢ|B): Posterior probability of hypothesis Hᵢ given evidence B (our updated belief).
• P(B): Probability of the evidence.

Example: Medical Diagnosis

Suppose a rare disease affects 1% of the population. A diagnostic test for this disease is 95% accurate for people who have the disease (i.e., it correctly identifies 95% of those with the disease - true positive rate). However, it also gives a positive result for 2% of people who do not have the disease (false positive rate).

Let D be the event that a person has the disease.

Let D' be the event that a person does not have the disease.

Let T+ be the event that the test is positive.

We are given:

• P(D) = 0.01 (Prior probability of having the disease)
• P(D') = 1 - P(D) = 0.99 (Prior probability of not having the disease)
• P(T+|D) = 0.95 (True positive rate - probability of testing positive given you have the disease)
• P(T+|D') = 0.02 (False positive rate - probability of testing positive given you don't have the disease)

We want to find P(D|T+), the probability that a person actually has the disease given that they tested positive.

Using Bayes' Theorem:

P(D|T+) = [ P(T+|D) * P(D) ] / P(T+)

First, we need to calculate P(T+), the overall probability of testing positive. We use the law of total probability:

P(T+) = P(T+|D) * P(D) + P(T+|D') * P(D')

P(T+) = (0.95 * 0.01) + (0.02 * 0.99)

P(T+) = 0.0095 + 0.0198

P(T+) = 0.0293

Now, substitute this back into Bayes' Theorem:

P(D|T+) = (0.95 * 0.01) / 0.0293

P(D|T+) = 0.0095 / 0.0293

P(D|T+) ≈ 0.324

Interpretation: Even though the test is 95% accurate, if you test positive, there is only about a 32.4% chance you actually have the disease. This is because the disease is rare (low prior probability), and the false positive rate, while small, is applied to a much larger group of healthy individuals.