Curve Fitting and Theory of Attributes

1. Curve Fitting

Curve fitting is a statistical method used to find a mathematical equation that best represents the relationship between two or more variables. This process involves plotting observed data points and then drawing a curve that passes as close as possible to these points. The goal is to establish a functional relationship that can be used for prediction, interpolation, or understanding underlying trends.

1.1. Introduction to Curve Fitting

In many real-world scenarios, data collected from experiments or observations do not perfectly align with a simple mathematical function. Curve fitting helps us approximate these complex relationships with simpler, manageable functions. This approximation is crucial for making informed decisions based on data, forecasting future outcomes, and simplifying complex phenomena. For instance, in economics, we might fit a curve to historical sales data to predict future sales. In engineering, we might fit a curve to stress-strain data to understand material properties.

1.2. Types of Curves

The choice of curve depends on the visual pattern observed in the data plot (scatter diagram) and the theoretical relationship expected between the variables. Common types of curves include:

• Straight Line (Linear): $y = a + bx$. This is the simplest form, used when the relationship appears linear.
• Parabola (Quadratic): $y = a + bx + cx^2$. Used for relationships with a single turning point.
• Exponential Curve: $y = ae^{bx}$ or $y = ab^x$. Used for data that grows or decays at an increasing or decreasing rate.
• Logarithmic Curve: $y = a + b \log(x)$. Used when the dependent variable changes rapidly at first and then slows down.
• Power Curve: $y = ax^b$. Used when one variable is proportional to some power of another.

1.3. Methods of Curve Fitting

The most common method for curve fitting is the Method of Least Squares. This method aims to minimize the sum of the squares of the vertical distances (residuals) between the observed data points and the fitted curve. These distances represent the errors in our approximation.

1.4. Method of Least Squares

Let the observed data points be $(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)$. We want to find the parameters of a function $y = f(x, a, b, c, \dots)$ such that the sum of the squares of the errors, $S$, is minimized.

The error for the $i$-th point is $e_i = y_i - f(x_i, a, b, c, \dots)$.

The sum of squares is $S = \sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n} [y_i - f(x_i, a, b, c, \dots)]^2$.

To minimize $S$, we take the partial derivatives of $S$ with respect to each parameter ($a, b, c, \dots$) and set them to zero. These equations are called the normal equations.

1.4.1. Linear Regression (Fitting a Straight Line)

Consider fitting a straight line $y = a + bx$ to the data points $(x_1, y_1), \dots, (x_n, y_n)$.

The sum of squares is $S = \sum_{i=1}^{n} (y_i - (a + bx_i))^2$.

To minimize $S$, we find the partial derivatives with respect to $a$ and $b$ and set them to zero:

$\frac{\partial S}{\partial a} = \sum_{i=1}^{n} 2(y_i - a - bx_i)(-1) = 0 \implies \sum y_i - na - b \sum x_i = 0$

$\frac{\partial S}{\partial b} = \sum_{i=1}^{n} 2(y_i - a - bx_i)(-x_i) = 0 \implies \sum x_i y_i - a \sum x_i - b \sum x_i^2 = 0$

These give us the normal equations for linear regression:

1. $na + b \sum x_i = \sum y_i$
2. $a \sum x_i + b \sum x_i^2 = \sum x_i y_i$

These are two linear equations with two unknowns, $a$ and $b$. They can be solved simultaneously.

Often, these equations are rewritten by dividing by $n$:

1. $a + b \bar{x} = \bar{y}$ (where $\bar{x} = \frac{\sum x_i}{n}$ and $\bar{y} = \frac{\sum y_i}{n}$)
2. $a \bar{x} + b \frac{\sum x_i^2}{n} = \frac{\sum x_i y_i}{n}$

From the first equation, $a = \bar{y} - b\bar{x}$. Substituting this into the second equation:

$(\bar{y} - b\bar{x})\bar{x} + b \frac{\sum x_i^2}{n} = \frac{\sum x_i y_i}{n}$

$ \bar{y}\bar{x} - b\bar{x}^2 + b \frac{\sum x_i^2}{n} = \frac{\sum x_i y_i}{n}$

$ b \left( \frac{\sum x_i^2}{n} - \bar{x}^2 \right) = \frac{\sum x_i y_i}{n} - \bar{y}\bar{x}$

Recall that variance of x, $\sigma_x^2 = \frac{\sum x_i^2}{n} - \bar{x}^2$ and covariance of x and y, $\text{Cov}(x,y) = \frac{\sum x_i y_i}{n} - \bar{x}\bar{y}$.

So, $b \sigma_x^2 = \text{Cov}(x,y)$, which gives $b = \frac{\text{Cov}(x,y)}{\sigma_x^2}$.

This is often expressed using sums directly:

$b = \frac{n \sum x_i y_i - (\sum x_i)(\sum y_i)}{n \sum x_i^2 - (\sum x_i)^2}$

And $a = \bar{y} - b\bar{x} = \frac{\sum y_i - b \sum x_i}{n}$.

1.4.2. Non-linear Regression (Transformable to Linear)**

Some non-linear relationships can be transformed into linear ones, allowing the use of the least squares method. This is often easier than deriving normal equations for non-linear functions directly.

Example 1: Exponential Curve $y = ae^{bx}$

Taking the natural logarithm of both sides:

$\ln(y) = \ln(a) + bx$

Let $Y = \ln(y)$ and $A = \ln(a)$. The equation becomes $Y = A + bx$, which is linear in terms of $x$ and $Y$. We can fit a straight line to the transformed data points $(x_i, \ln(y_i))$ to find $A$ and $b$. Once $A$ and $b$ are found, $a = e^A$.

Example 2: Power Curve $y = ax^b$

Taking the logarithm (base 10 or natural log) of both sides:

$\log(y) = \log(a) + b \log(x)$

Let $Y = \log(y)$, $X = \log(x)$, and $A = \log(a)$. The equation becomes $Y = A + bX$, which is linear. We fit a straight line to the transformed data points $(\log(x_i), \log(y_i))$ to find $A$ and $b$. Once $A$ and $b$ are found, $a = 10^A$ (if using log base 10).

Example 3: $y = a + b \log(x)$

Let $Y = y$ and $X = \log(x)$. The equation becomes $Y = a + bX$, which is linear. We fit a straight line to the data points $(\log(x_i), y_i)$ to find $a$ and $b$. No transformation is needed for $y$. If using $\log_{10}$, $X = \log_{10}(x)$.

1.5. Goodness of Fit

After fitting a curve, it's important to assess how well the curve represents the data. Several measures can be used:

• Visual Inspection: Plotting the fitted curve along with the data points.
• Residual Analysis: Examining the differences between observed and predicted values. Ideally, residuals should be randomly scattered around zero.
• Coefficient of Determination ($R^2$): This value ranges from 0 to 1 and indicates the proportion of the variance in the dependent variable that is predictable from the independent variable(s). A value closer to 1 indicates a better fit.

For a linear regression $y = a + bx$, $R^2$ can be calculated as:

$R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}$

where $y_i$ are the observed values, $\hat{y}_i = a + bx_i$ are the predicted values, and $\bar{y}$ is the mean of the observed $y$ values.

1.6. Limitations of Curve Fitting

Curve fitting provides an approximation, not an exact representation. Extrapolation (predicting values outside the range of the observed data) can be highly unreliable. The choice of the curve type significantly impacts the fit; an inappropriate model can lead to misleading conclusions. It assumes that the errors are independent and have constant variance (homoscedasticity).

2. Theory of Attributes

The Theory of Attributes deals with qualitative data, also known as categorical data. Unlike quantitative data (which can be measured numerically, like height or weight), attributes are qualities or characteristics that cannot be measured numerically but can be classified into distinct categories (e.g., color, gender, nationality, pass/fail). The theory focuses on analyzing frequencies and relationships between these attributes.

2.1. Introduction to Attributes

In statistics, we often encounter data that describes qualities rather than quantities. For example, classifying people by their eye color (blue, brown, green) or by their employment status (employed, unemployed). The Theory of Attributes provides methods to organize, analyze, and draw conclusions from such data. It's particularly useful in social sciences, market research, and surveys where qualitative information is abundant.

2.2. Basic Concepts

Attribute: A quality or characteristic that can be classified into two or more categories. Examples: Sex (Male/Female), Color (Black/White/Red), Intelligence (Intelligent/Dull).

Classes: The categories within an attribute. For Sex, the classes are Male and Female.

Variate: A characteristic that can be measured numerically. Example: Height, Age.

Simple Attribute: An attribute that can only be classified into two mutually exclusive classes. Example: Literacy (Literate/Illiterate).

Manifold Attribute: An attribute that can be classified into more than two mutually exclusive classes. Example: Color (Black/White/Red/Blue).

Degree of a Class: The number of attributes in a class. A class with one attribute is of the first degree, two attributes is of the second degree, and so on.

Frequencies: The number of observations falling into each class.

2.3. Notations Used

Let 'A' and 'B' be two attributes. We use specific notations to represent frequencies:

• $N$: Total number of observations.
• $[A]$: Number of observations possessing attribute A.
• $[B]$: Number of observations possessing attribute B.
• $[AB]$: Number of observations possessing both attributes A and B.
• $[A']$: Number of observations not possessing attribute A (complement of A).
• $[B']$: Number of observations not possessing attribute B (complement of B).
• $[A'B]$: Number of observations not possessing A but possessing B.
• $[AB']$: Number of observations possessing A but not possessing B.
• $[A'B']$: Number of observations possessing neither A nor B.

Note that $N = [A] + [A']$. Similarly, $N = [B] + [B']$.

Also, $[A] = [AB] + [AB']$ and $[B] = [AB] + [A'B]$.

And $N = [AB] + [AB'] + [A'B] + [A'B']$.

2.4. Association of Attributes

Association refers to the relationship between two or more attributes. We investigate whether attributes occur together more or less often than would be expected by chance.

2.5. Yule's Coefficient of Association (Q)**

Yule's coefficient of association is used to measure the degree of association between two attributes, especially when they are dichotomous (having only two classes each). It is calculated using the frequencies of occurrence and non-occurrence of the attributes.

Given two attributes A and B, and their complements A' and B', the frequencies are:

• $[AB]$: Number possessing both A and B.
• $[A'B]$: Number possessing B but not A.
• $[AB']$: Number possessing A but not B.
• $[A'B']$: Number possessing neither A nor B.

Yule's coefficient of association, denoted by $Q$, is given by:

$Q = \frac{[AB][A'B'] - [A'B][AB']}{[AB][A'B'] + [A'B][AB']}$

The value of $Q$ ranges from -1 to +1.

• If $Q = 1$, there is perfect positive association (A and B always occur together).
• If $Q = -1$, there is perfect negative association (A and B never occur together).
• If $Q = 0$, there is no association between A and B.
• If $Q > 0$, there is positive association.
• If $Q < 0$, there is negative association.

Interpretation: $Q$ measures the relative difference between the number of concordant pairs ($[AB][A'B']$) and discordant pairs ($[A'B][AB']$).

2.6. Yule's Coefficient of Colligation (Y)**

Yule's coefficient of colligation, denoted by $Y$, is another measure of association. It is related to $Q$ by the formula:

$Y = \frac{1 - \sqrt{1 - Q^2}}{1 + \sqrt{1 - Q^2}}$ or $Q = \frac{2Y}{1+Y^2}$

The value of $Y$ ranges from 0 to 1.

• If $Y = 1$, there is perfect association.
• If $Y = 0$, there is no association.

While $Q$ indicates the direction of association (positive or negative), $Y$ only indicates the strength of association.

2.7. Independence and Association

Two attributes A and B are said to be independent if the frequency of their joint occurrence is exactly what would be expected if they were unrelated. If $N$ is the total number of observations:

Attributes A and B are independent if $[AB] = \frac{[A][B]}{N}$.

If $[AB] > \frac{[A][B]}{N}$, the attributes are positively associated.

If $[AB] < \frac{[A][B]}{N}$, the attributes are negatively associated.

2.8. Consistency of Data (Contradiction)**

Sometimes, we are given frequencies for various classes, and we need to check if these frequencies are consistent or if they contradict each other. A fundamental principle is that the frequency of a class cannot be negative. Also, the frequency of a sub-class cannot be greater than the frequency of the class it belongs to.

For two attributes A and B:

• The frequency of any class must be non-negative. For example, $[AB] \ge 0$, $[A'B] \ge 0$, etc.
• The sum of frequencies of mutually exclusive classes must not exceed the total frequency. For example, $[AB] + [AB'] \le N$, which simplifies to $[A] \le N$. This is always true.
• More importantly, for any two attributes, the following inequalities must hold:
  • $[A] + [B] - N \le [AB] \le \min([A], [B])$
  • $[A'] + [B'] - N \le [A'B'] \le \min([A'], [B'])$

  These inequalities help determine the possible range of frequencies for joint attributes and check for consistency.

2.9. Theory of Attributes with Three Attributes

The concepts can be extended to three attributes: A, B, and C. We would then consider frequencies like $[ABC]$, $[AB'C]$, $[A'BC]$, $[A'B'C]$, $[ABC']$, $[AB'C']$, $[A'BC']$, $[A'B'C']$.

The total number of observations $N$ is the sum of all these eight mutually exclusive classes:

$N = [ABC] + [ABC'] + [AB'C] + [AB'C'] + [A'BC] + [A'BC'] + [A'B'C] + [A'B'C']$

Similar relationships and consistency conditions apply. For example, for three attributes, the consistency condition for $[ABC]$ is:

$\max(0, [A]+[B]+[C]-2N) \le [ABC] \le \min([A], [B], [C])$

More complex consistency rules involving pairs of attributes are also used:

• $[AB] + [AC] + [BC] - 2[ABC] \le N$
• $[A] - [AB] - [AC] + [ABC] \ge 0$ (and similar for B and C)

2.10. Applications of Theory of Attributes

The Theory of Attributes is applied in various fields:

• Social Surveys: Analyzing relationships between variables like education level, income, and employment status.
• Market Research: Understanding consumer preferences based on demographics, brand loyalty, and purchasing habits.
• Medical Studies: Investigating the association between risk factors (e.g., smoking) and diseases (e.g., lung cancer).
• Quality Control: Assessing defect rates based on different production factors.