9  Regression analysis

Regression analysis is one of the most important tools in statistics, used to understand and quantify the relationships between variables. In essence, regression helps us answer questions such as: “How does a change in one variable (like fertilizer usage) affect another variable (like crop yield)?” It provides a mathematical framework to explore these relationships based on observed data (Montgomery, Peck, and Vining 2021; Draper and Smith 1998).

Regression analysis involves two types of variables:

• Dependent variable (y): The dependent variable (usually denoted as y) is the variable whose values are to be explained, predicted, or modeled based on one or more independent variables. It represents the outcome or response of interest and is assumed to depend on the values of the explanatory variables. For example, crop yield is often the dependent variable in agricultural experiments. In regression analysis, it is commonly referred to as the response variable, outcome variable, criterion variable, target variable, explained variable, regressand, or simply the response.

• Independent variable (x): The independent variable (usually denoted as x) is a variable used to explain, predict, or account for variation in the dependent variable. It is considered the explanatory or predictor variable in the model and is used as an input to estimate the response. Examples include fertilizer dose, irrigation level, rainfall, temperature, or plant density. In regression analysis, it is also known as the predictor variable, explanatory variable, regressor, covariate, feature (in machine learning), input variable, or simply the predictor.

Why use regression analysis?

Regression analysis is particularly useful because it allows you to:

1. Quantify relationships: It measures how strongly one or more independent variables are associated with the dependent variable.

2. Predict outcomes: Once the relationship is understood, regression can be used to predict the dependent variable for new values of the independent variables.

3. Identify key factors: It can highlight which variables have the most significant impact on the dependent variable, guiding decision-making.

4. Control for multiple factors: By including several independent variables, regression helps isolate the effect of each variable while controlling for the others.

Types of regression

There are different types of regression techniques, depending on the nature of the data and the relationship between variables:

• Simple linear regression: Examines the relationship between one independent variable and one dependent variable, assuming a straight-line relationship.

• Multiple linear regression: An extension of simple linear regression, allowing for the analysis of the relationship between one dependent variable (y) and multiple independent variables (more than one x). It is used when the outcome is influenced by more than one factor.

• Nonlinear regression: Deals with situations where the relationship between variables is not a straight line.

• Logistic regression: Used when the dependent variable is categorical, such as predicting whether a plant will survive (yes or no) based on environmental factors.

Practical applications

Regression analysis has a wide range of applications across fields:

• In agriculture, it can be used to study the effect of irrigation, soil nutrients, and weather on crop yield.

• In economics, it helps analyse the impact of income, education, and employment on consumer behaviour.

• In medicine, regression can predict health outcomes based on patient characteristics.

By the end of this chapter, you will learn how to perform regression analysis, interpret its results, and understand its assumptions and limitations.

9.1 Simple linear regression

Regression can be simply defined as a technique of fitting a line of best fit to estimate the value of one variable on the basis of another variable. But what is a line of best fit? To understand this concept, consider the data presented in Table 8.2 of Example 8.2, which shows the average daily soil moisture content and the corresponding monetary yield from crops. This example helps visualise how the relationship between two variables i.e. soil moisture content (independent variable) and crop yield (dependent variable) can be captured by a line.

We can use regression analysis to answer the following questions: What will the crop yield be (in rupees) when soil moisture content is maintained at 20%? What is the functional form of the relationship between soil moisture content and monetary crop yield?

Refer to the scatter diagram of Table 8.2 in Figure 8.3. To represent the relationship between soil moisture content and monetary crop yield, we might attempt to draw a line through the data points, as illustrated in Figure 9.1. However, as shown in Figure 9.1, it is possible to draw numerous lines through the same set of data points. This raises the question: which line is the best fit?

(a)

(b)

(c)

Figure 9.1: Possible lines drawn to show the functional relationship between soil moisture and yield

The best fit line is the one that minimises the distances between the observed data points and the line itself. These distances are minimised collectively using a specific mathematical criterion. The regression technique provides a systematic approach to determine and draw this best fit line. Before going further into regression, it is essential to understand the concepts of error and residual, which play a critical role in determining the best fit line.

9.1.1 Error and residual

In regression analysis, an error represents the difference between an observed value and the value based on the true regression line. True regression line here what we meant is the line that reflects the actual relationship between variables in the entire population. Since the true regression line is based on the whole population and is usually unknown, the error is a theoretical concept that cannot be directly measured.

A residual is the difference between an observed value and the value predicted by the regression line fitted from sample data. For a given data point, the residual is calculated as:

\[\text{Residual} = \text{Observed value} - \text{Predicted value}\]

Residuals are measurable because they are derived from the observed data and the fitted regression line. Unlike the errors which are theoretical deviations from the true underlying model, the residuals provide a practical estimate of these deviations, allowing us to assess the goodness of fit and identify any patterns or discrepancies in the model.

In essence, a residual serves as an estimate of the error. From Figure 9.2, you can see the residual \(e_i\) of the \(i\)th observation in a fitted regression line.

Figure 9.2: Residual and the best fit line

The distance of the \(i\)th observation (\(e_i\)) from the fitted line is the residual. The best fit line is obtained by minimising these distances, which is achieved using the principle of least squares, discussed in Section 9.1.3. Before identifying the best fit line, it is useful to recall the concept of a straight line.

9.1.2 Straight lines

A straight line is the simplest figure in geometry. The mathematical equation of a straight line is:

\[Y = \alpha + \beta X \tag{9.1}\]

A line has two important features: the intercept (\(\alpha\)) and the slope (\(\beta\)). The intercept \(\alpha\) is the value of \(Y\) at the point where the line crosses the \(Y\)-axis (i.e., when \(X = 0\)). The slope \(\beta\) is a number that measures the steepness of the line, it is the change in \(Y\) for a one-unit change in \(X\). In regression, \(\beta\) is called the regression coefficient, explained further in Section 9.1.4.

Intercept and slope

(a) Intercept of a straight line

(b) Slope of a straight line

Figure 9.3: Intercept and slope of a straight line

The values \(\alpha\) and \(\beta\) act as a fingerprint of a line, with these two values, we can uniquely identify any straight line. So our problem reduces to finding the line of best fit by estimating \(\alpha\) and \(\beta\) such that the error \(e_i\) of each observation is minimised. For this, we use the method of least squares.

9.1.3 Method of least squares

Including the error term \(e_i\), the equation of the regression line is:

\[y_i = \alpha + \beta x_i + e_i \tag{9.2}\]

where \(e_i\) is the \(i\)th error term corresponding to \(y_i\), for \(i = 1, 2, \ldots, n\).

Note

One way to obtain the line of best fit is by estimating \(\alpha\) and \(\beta\) by minimising the error sum \(\sum_{i=1}^{n} e_i\). However, by theorem, \(\sum_{i=1}^{n} e_i = 0\) for any fitted line, so this does not uniquely determine \(\alpha\) and \(\beta\). Instead, we estimate \(\alpha\) and \(\beta\) by minimising \(\sum_{i=1}^{n} e_i^2\). The term \(\sum_{i=1}^{n} e_i^2\) is called the error sum of squares. Since we are minimising the sum of the squared error terms, this process is known as the principle of least squares.

Principle of least squares

The principle of least squares is the statistical method used to determine the line of best fit by minimising the sum of squares of the errors, i.e., minimising \(\sum_{i=1}^{n} e_i^2\) (Draper and Smith 1998).

Starting from Equation 9.2:

\[e_i = y_i - (\alpha + \beta x_i) \tag{9.3}\]

\[e_i^2 = [y_i - (\alpha + \beta x_i)]^2 \tag{9.4}\]

Let \(E = \sum_{i=1}^{n} e_i^2\).

\[E=\sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n} [y_i - (\alpha + \beta x_i)]^2 \tag{9.5}\]

We want to minimise Equation 9.5 to estimate \(\alpha\) and \(\beta\). This is done by taking the partial derivative of \(\sum_{i=1}^{n} e_i^2\) with respect to \(\alpha\) and \(\beta\) separately, and equating each to zero. Doing so yields two equations called the normal equations. Solving these normal equations gives the formulas for estimating \(\alpha\) and \(\beta\).

Differentiating with respect to \(\alpha\) and equating to zero:

\[\frac{\partial E}{\partial \alpha} = \sum_{i=1}^n 2 \left[ y_i - (\alpha + \beta x_i) \right](-1) = 0 \tag{9.6}\]

\[\Rightarrow -2 \sum_{i=1}^n \left[ y_i - \alpha - \beta x_i \right] = 0 \tag{9.7}\]

Simplifying Equation 9.7:

\[\sum_{i=1}^n y_i - n\alpha - \beta \sum_{i=1}^n x_i = 0 \tag{9.8}\]

Rearranging gives the first normal equation:

\[\sum_{i=1}^n y_i = n\alpha + \beta \sum_{i=1}^n x_i \tag{9.9}\]

Differentiating \(E\) with respect to \(\beta\) and equating to zero:

\[\frac{\partial E}{\partial \beta} = \sum_{i=1}^n 2 \left[ y_i - (\alpha + \beta x_i) \right](-x_i) = 0 \tag{9.10}\]

\[\Rightarrow -2 \sum_{i=1}^n x_i \left[ y_i - \alpha - \beta x_i \right] = 0 \tag{9.11}\]

Simplifying Equation 9.11:

\[\sum_{i=1}^n x_i y_i - \alpha \sum_{i=1}^n x_i - \beta \sum_{i=1}^n x_i^2 = 0 \tag{9.12}\]

Rearranging gives the second normal equation:

\[\sum_{i=1}^n x_i y_i = \alpha \sum_{i=1}^n x_i + \beta \sum_{i=1}^n x_i^2 \tag{9.13}\]

Solving normal equations Equation 9.9 and Equation 9.13 together gives the formulas for estimating \(\alpha\) and \(\beta\). Since \(\alpha\) and \(\beta\) are population parameters and are usually unknown, we estimate them from sample data. The estimated values are denoted \(\hat{\alpha}\) and \(\hat{\beta}\) (pronounced “alpha hat” and “beta hat”), and are used as approximations of the true population parameters.

Solving for \(\hat{\beta}\)

From the first normal equation (Equation 9.9), divide both sides by \(n\):

\[\frac{\sum_{i=1}^n y_i}{n} = \alpha + \beta \frac{\sum_{i=1}^n x_i}{n}\]
\[\overline{y} = \alpha + \beta \overline{x} \tag{9.14}\]
This gives us:

\[\alpha = \overline{y} - \beta \overline{x} \tag{9.15}\]

Substitute Equation 9.15 into the second normal equation (Equation 9.13):

\[\sum_{i=1}^n x_i y_i = (\overline{y} - \beta \overline{x})\sum_{i=1}^n x_i + \beta \sum_{i=1}^n x_i^2\]
\[\sum_{i=1}^n x_i y_i = \overline{y}\sum_{i=1}^n x_i - \beta \overline{x}\sum_{i=1}^n x_i + \beta \sum_{i=1}^n x_i^2\]

Collecting the \(\beta\) terms on the right:

\[\sum_{i=1}^n x_i y_i - \overline{y}\sum_{i=1}^n x_i = \beta \left(\sum_{i=1}^n x_i^2 - \overline{x}\sum_{i=1}^n x_i\right)\] Since \(\overline{x} = \frac{\sum x_i}{n}\), we have \(\overline{x} \sum_{i=1}^n x_i = \frac{\left(\sum_{i=1}^n x_i\right)^2}{n}\). Similarly, \(\overline{y} \sum_{i=1}^n x_i = \frac{\sum_{i=1}^n y_i \sum_{i=1}^n x_i}{n}\). Substituting:

\[\sum_{i=1}^n x_i y_i - \frac{\sum_{i=1}^n y_i \sum_{i=1}^n x_i}{n} = \beta \left(\sum_{i=1}^n x_i^2 - \frac{\left(\sum_{i=1}^n x_i\right)^2}{n}\right)\] Solving for \(\beta\):

\[\hat{\beta} = \frac{\sum_{i=1}^n x_i y_i - \dfrac{\sum_{i=1}^n x_i \sum_{i=1}^n y_i}{n}}{\sum_{i=1}^n x_i^2 - \dfrac{\left(\sum_{i=1}^n x_i\right)^2}{n}} \tag{9.16}\]

This is the formula for \(\hat{\beta}\) used for hand calculations. It can be written more compactly by recognising that:

\[\sum_{i=1}^n x_i y_i - \frac{\sum_{i=1}^n x_i \sum_{i=1}^n y_i}{n} = n \cdot cov(x, y)\]

\[\sum_{i=1}^n x_i^2 - \frac{\left(\sum_{i=1}^n x_i\right)^2}{n} = n \cdot var(x)\] Therefore:

\[\hat{\beta} = \frac{cov(x,y)}{var(x)} \tag{9.17}\]
Solving for \(\hat{\alpha}\)

Once \(\hat{\beta}\) is known, the intercept is obtained directly from Equation 9.15:

\[\hat{\alpha} = \overline{y} - \hat{\beta}\,\overline{x} \tag{9.18}\]

where \(\overline{y}\) = mean of \(y\) and \(\overline{x}\) = mean of \(x\).

Note

Once the estimates \(\hat{\alpha}\) and \(\hat{\beta}\) are obtained using Equation 9.17 and Equation 9.18, the estimated regression line can be written as:

\[\hat{y} = \hat{\alpha} + \hat{\beta}\, x \tag{9.19}\]

Note that the predicted value of \(y\) is written as \(\hat{y}\) (y-hat) to distinguish it from the observed value \(y\). Also note that the regression line always passes through the point \((\overline{x},\, \overline{y})\), which can be verified by substituting \(x = \overline{x}\) into Equation 9.19.

9.1.4 Regression coefficient

The regression coefficient \(\beta\) in linear regression represents the slope of the regression line. It quantifies the relationship between the independent variable (\(x\)) and the dependent variable (\(y\)). Specifically, \(\hat{\beta}\) indicates the expected change in \(y\) for a one-unit increase in \(x\), when all other factors are held constant (Montgomery, Peck, and Vining 2021).

Regression coefficients can take any real value from \(-\infty\) to \(+\infty\). A positive \(\hat{\beta}\) implies a direct relationship (as \(x\) increases, \(y\) increases), while a negative \(\hat{\beta}\) implies an inverse relationship (as \(x\) increases, \(y\) decreases). A coefficient of zero suggests no linear relationship between the variables.

The formula for \(\hat{\beta}\) is obtained by solving normal equations Equation 9.9 and Equation 9.13, and is given by Equation 9.16, which can be used for hand calculations:

\[\hat{\beta}=\frac{\sum_{i = 1}^{n}{y_{i}x_{i} - \frac{\sum_{i = 1}^{n}{y_{i}\sum_{i = 1}^{n}x_{i}}}{n}}}{\sum_{i = 1}^{n}x_{i}^{2} - \frac{\left( \sum_{i = 1}^{n}x_{i} \right)^{2}}{n}}\]

Equation 9.16 can be written more compactly as Equation 9.17:

\[\hat{\beta} = \frac{cov(x,y)}{var(x)}\]

9.1.5 Intercept

The intercept \(\alpha\) represents the value of the dependent variable \(y\) when the independent variable \(x\) equals zero. It is the point at which the regression line crosses the \(y\)-axis, and provides a baseline value for \(y\) before any influence from \(x\) is considered.

The intercept can take any real value (\(-\infty\) to \(+\infty\)). Its practical interpretation depends on the specific context. In many agricultural datasets, \(x = 0\) may not lie within the range of observed data, in which case the intercept may not carry a meaningful practical interpretation.

The formula for \(\hat{\alpha}\), from Equation 9.18, is

\[\hat{\alpha} = \overline{y} - \hat{\beta}\,\overline{x}\]

where \(\overline{y}\) = mean of \(y\) and \(\overline{x}\) = mean of \(x\).

9.1.6 Assumptions

For the results of a regression analysis to be reliable and meaningful, certain underlying assumptions must be met. These ensure that the estimates are accurate, predictions are unbiased, and conclusions drawn from the model are valid (Montgomery, Peck, and Vining 2021; Draper and Smith 1998).

1. Linearity
   The relationship between the independent variable(s) and the dependent variable is linear, changes in \(y\) are proportional to changes in \(x\).

2. Independence
   The observations in the dataset are independent of each other, and the residuals (errors) are also independent.

3. Homoscedasticity
   The variance of the residuals is constant across all levels of the independent variable(s). The spread of the residuals should remain consistent and not show patterns of increasing or decreasing variance.

4. Normality of residuals
   The residuals are normally distributed. This is particularly important for hypothesis testing and constructing confidence intervals. The normality assumption does not strongly influence the estimation of regression coefficients themselves, but it matters for inference.

5. No multicollinearity
   In the case of multiple regression, the independent variables should not be highly correlated with each other. Multicollinearity can distort the estimates of regression coefficients and make them difficult to interpret.

6. No autocorrelation
   There should be no autocorrelation in the residuals, the residual of one observation should not be correlated with the residual of another.

7. Correct model specification
   The model should include all relevant variables and exclude irrelevant ones. The functional form of the relationship between variables should be correctly specified.

Violations of these assumptions can lead to biased, inconsistent, or inefficient estimates, affecting the validity of the regression analysis.

Note

The essence of the assumptions in linear regression can be summarised as \(e \sim \text{i.i.d.}(0, \sigma^2)\). This denotes that the errors are independent and identically distributed (i.i.d.), with a mean of zero and a constant variance \(\sigma^2\).

• Independence ensures that the error for one observation does not influence the error for another.
• Identically distributed means all errors are drawn from the same probability distribution.
• Mean of zero ensures that errors do not introduce systematic bias into the model’s predictions.
• Constant variance (homoscedasticity) means the errors maintain a consistent level of variability across all values of the independent variable(s).

9.2 Two lines of regression

Consider the data presented in Table 8.2, showing average daily soil moisture content and the corresponding monetary crop yield from Example 8.2. For any two variables \(x\) and \(y\), we can draw two lines of regression :- one by treating \(x\) as independent and \(y\) as dependent, and the other by interchanging their roles as shown in Figure 9.4.

(a) Moisture on X-axis, yield on Y-axis (regression of y on x)

(b) Yield on X-axis, moisture on Y-axis (regression of x on y)

Figure 9.4: Two lines of regression

From Figure 9.4 it is clear that two distinct lines of regression are possible: the regression of \(y\) on \(x\), and the regression of \(x\) on \(y\).

Regression of \(y\) on \(x\)

When \(y\) is the dependent variable and \(x\) is the independent variable, the regression equation is:

\[y = \alpha + \beta_{yx}\, x \tag{9.20}\]

This is used to predict the unknown value of \(y\) when the value of \(x\) is known. The regression coefficient here is denoted \(\beta_{yx}\) and is obtained using:

\[\beta_{yx} = \frac{cov(x,y)}{var(x)} \tag{9.21}\]

Regression of \(x\) on \(y\)

When \(x\) is the dependent variable and \(y\) is the independent variable, the regression equation is:

\[x = \alpha_1 + \beta_{xy}\, y \tag{9.22}\]

This is used to predict the unknown value of \(x\) when the value of \(y\) is known. The regression coefficient here is denoted \(\beta_{xy}\) and is obtained using:

\[\beta_{xy} = \frac{cov(x,y)}{var(y)} \tag{9.23}\]

As seen from Equation 9.21 and Equation 9.23, the two regression coefficients are different. It is the experimenter’s responsibility to choose which variable is treated as dependent and which as independent, based on the objective of the study. In Example 8.2, predicting soil moisture based on monetary crop yield may not be meaningful in practice, the choice of dependent and independent variable must reflect the scientific question being asked.

9.2.1 Properties of Regression Coefficients

1. Relationship with the correlation coefficient

   The product of the two regression coefficients equals the square of the correlation coefficient:

   \[\beta_{yx}\beta_{xy}=r_{xy}^{\,2}\]

   Hence,

   \[|r_{xy}|=\sqrt{\beta_{yx}\beta_{xy}}\]

   Since the regression coefficients always have the same sign,

   \[ r_{xy}= \begin{cases} \sqrt{\beta_{yx}\beta_{xy}}, & \text{if } \beta_{yx},\beta_{xy}>0,\\[4pt] -\sqrt{\beta_{yx}\beta_{xy}}, & \text{if } \beta_{yx},\beta_{xy}<0. \end{cases} \]

2. Both regression coefficients have the same sign

   The two regression coefficients always possess the same sign.

   • If \(\beta_{yx}>0\), then \(\beta_{xy}>0\), and the correlation coefficient is positive.
   • If \(\beta_{yx}<0\), then \(\beta_{xy}<0\), and the correlation coefficient is negative.
   • If one regression coefficient is zero, the other is also zero, implying \(r_{xy}=0\).

3. Regression coefficients are independent of change of origin but not of scale

   • Adding or subtracting a constant from either variable (change of origin) does not affect the regression coefficients (slopes), although it changes the intercepts.
   • Multiplying or dividing a variable by a constant (change of scale) changes the regression coefficients.

   If

   \[ X=\frac{x-a}{h},\qquad Y=\frac{y-b}{k}, \]

   then

   \[ \beta_{YX}=\frac{h}{k}\beta_{yx}, \qquad \beta_{XY}=\frac{k}{h}\beta_{xy}. \]

4. Relationship with unity

   Since

   \[\beta_{yx}\beta_{xy}=r^2\le1,\]

   • if one regression coefficient is greater than unity, the other must be less than unity;
   • both regression coefficients may be less than one;
   • both regression coefficients cannot exceed one simultaneously.

5. Regression coefficients are not bounded

   Unlike the correlation coefficient, whose value lies between \(-1\) and \(+1\), regression coefficients can take any real value. Thus,

   \[-\infty<\beta<+\infty.\]

6. Regression coefficients in terms of correlation and standard deviations

   The regression coefficients are related to the correlation coefficient and the standard deviations by

   \[ \beta_{yx}=r\frac{\sigma_y}{\sigma_x}, \qquad \beta_{xy}=r\frac{\sigma_x}{\sigma_y}. \]

   Thus, the magnitude of a regression coefficient depends on both the strength of association and the relative variability of the two variables.

7. Independence property

   If the variables \(x\) and \(y\) are statistically independent,

   \[r_{xy}=0,\]

   and therefore

   \[\beta_{yx}=\beta_{xy}=0.\]

   In this case, neither variable can be used to predict the other through a linear regression model.

8. Perfect correlation

   If the variables are perfectly correlated (\(r=\pm1\)),

   \[\beta_{yx}\beta_{xy}=1.\]

   The two regression lines coincide, indicating a perfect linear relationship between the variables.

9.2.2 Properties of Regression Lines

1. Regression lines are the lines of best fit

   Each regression line is obtained using the least squares principle, which minimises the sum of the squared deviations between the observed and predicted values.

   • The regression line of \(y\) on \(x\) minimises the sum of squared vertical deviations.
   • The regression line of \(x\) on \(y\) minimises the sum of squared horizontal deviations.

2. Both regression lines pass through the point of means

   The regression lines of \(y\) on \(x\) and \(x\) on \(y\) always intersect at the point

   \[(\bar{x},\,\bar{y}),\]

   where \(\bar{x}\) and \(\bar{y}\) are the arithmetic means of the variables.

3. The position of the regression lines depends on the correlation coefficient

   The angle between the two regression lines depends on the magnitude of the correlation coefficient (\(r\)).

   • If \(|r|=1\), the two regression lines coincide, indicating a perfect linear relationship.
   • If \(r=0\), the regression lines are perpendicular to each other, indicating no linear relationship.
   • For \(0<|r|<1\), the two regression lines intersect at an angle between \(0^\circ\) and \(90^\circ\).

   The position and angle of the regression lines therefore reflect the strength of the linear relationship between the variables (see Figure 9.5).

4. Regression lines are unique

   For a given dataset, there is only one regression line of \(y\) on \(x\) and one regression line of \(x\) on \(y\).

5. The two regression lines are identical only under perfect correlation

   The regression lines coincide only when

   \[r=\pm1,\]

   indicating a perfect positive or perfect negative linear relationship. Otherwise, they are distinct.

6. Prediction depends on the direction of regression

   The regression line of \(y\) on \(x\) is used to predict the value of \(y\) from a given value of \(x\), whereas the regression line of \(x\) on \(y\) is used to predict the value of \(x\) from a given value of \(y\). These two regression equations are generally different unless \(|r|=1\).

Figure 9.5: Effect of correlation strength on the position of regression lines

9.3 Uses of regression

• Prediction
  Regression is used to predict the value of a dependent variable (\(y\)) based on one or more independent variables (\(x\)). Examples in agricultural research include:
  • Predicting crop yield based on weather parameters such as temperature, rainfall, and humidity.
  • Estimating soil nutrient levels using remote sensing data or environmental variables.
  • Forecasting pest or disease outbreaks based on climatic and ecological conditions.
• Identifying the strength of relationships
  Regression helps quantify the strength of the relationship between variables, which is essential for identifying influential factors in agricultural research. Examples include:
  • Determining the effect of fertilizer dosage on crop yield.
  • Analysing the relationship between irrigation frequency and plant growth.
  • Understanding the impact of livestock feed composition on milk production.
• Forecasting effects of changes
  Regression models allow researchers to evaluate how changes in one or more independent variables affect the dependent variable. For example:
  • Assessing how seed quality impacts overall harvest productivity.
  • Analysing the effects of varying water availability on crop output in drought-prone areas.
  • Estimating the economic benefits of adopting precision farming techniques.
• Predicting trends and future values
  Regression is valuable for modelling trends and forecasting future values. Applications include:
  • Predicting future crop yields under different climate change scenarios.
  • Estimating long-term price trends for agricultural commodities such as rice, wheat, or coffee.
  • Forecasting the adoption rates of new agricultural technologies among farmers.

Note

Multiple regression is an extension of simple linear regression that models the relationship between a dependent variable and two or more independent variables (Montgomery, Peck, and Vining 2021). It allows researchers to account for the combined effect of multiple factors on an outcome. For instance, crop yield can be predicted based on a combination of soil nutrients, rainfall, temperature, and fertilizer application.

In a multiple regression model, the relationship between the dependent variable \(y\) and independent variables \(x_1, x_2, \ldots, x_k\) is expressed as:

\[y = \alpha + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_k x_k + e \tag{9.24}\]

Where \(y\) is the dependent variable; \(\alpha\) is the intercept; \(\beta_1, \beta_2, \ldots, \beta_k\) are the coefficients for each independent variable; \(x_1, x_2, \ldots, x_k\) are the independent variables; and \(e\) is the error term.

Example 9.1 Using the data in Table 8.2 (average daily soil moisture content and monetary crop yield from Example 8.2), answer the following:

1. What is the functional form of the relationship between soil moisture and monetary crop yield?
2. What will be the estimated monetary crop yield when average daily soil moisture is maintained at 20%?

Solution

Step 1: Fit a model treating monetary crop yield as the dependent variable (\(y\)) and average soil moisture as the independent variable (\(x\)). Fitting a model means estimating \(\hat{\beta}\) using Equation 9.17 and \(\hat{\alpha}\) using Equation 9.18.

Step 2: Substitute \(x = 20\) into the fitted equation to obtain the predicted crop yield.

Table 9.1: Calculation table for regression

Sl No.	Soil moisture (\(x\))	Crop yield in Rs (\(y\))	\((x_{i}-\overline{x})\)	\((y_{i}-\overline{y})\)	\((x_{i}-\overline{x})(y_{i}-\overline{y})\)	\((x_{i}-\overline{x})^2\)
1	14.2	215	-4.48	-187.42	838.69	20.03
2	16.4	325	-2.28	-77.42	176.12	5.18
3	11.9	185	-6.78	-217.42	1473.00	45.90
4	15.2	332	-3.48	-70.42	244.70	12.08
5	18.5	406	-0.18	3.58	-0.63	0.03
6	22.1	522	3.43	119.58	409.57	11.73
7	19.4	412	0.73	9.58	6.95	0.53
8	25.1	614	6.43	211.58	1359.42	41.28
9	23.4	544	4.73	141.58	668.98	22.33
10	18.1	421	-0.58	18.58	-10.69	0.33
11	22.6	445	3.93	42.58	167.14	15.41
12	17.2	408	-1.48	5.58	-8.24	2.18
SUM	224.1	4829	0.00	0.00	5325.03	176.98

\(n = 12\)

\[\overline{x} = \frac{224.1}{12} = 18.675\]

\[\overline{y} = \frac{4829}{12} = 402.416\]

\[cov(x,y) = \frac{1}{n}\sum_{i=1}^{n}(x_i - \overline{x})(y_i - \overline{y}) = \frac{5325.03}{12} = 443.752\]

\[var(x) = \frac{1}{n}\sum_{i=1}^{n}(x_i - \overline{x})^2 = \frac{176.983}{12} = 14.749\]

Using Equation 9.17:

\[\hat{\beta} = \frac{cov(x,y)}{var(x)} = \frac{443.752}{14.749} = 30.088\]

Using Equation 9.18:

\[\hat{\alpha} = \overline{y} - \hat{\beta}\,\overline{x} = 402.416 - 30.088 \times 18.675 = -159.477\]

The estimated regression equation is therefore:

\[\hat{y} = -159.477 + 30.088\, x\]

\[\text{Crop yield (Rs)} = -159.477 + 30.088 \times \text{(soil moisture \%)}\]

For a soil moisture content of 20% (\(x = 20\)):

\[\hat{y} = -159.477 + 30.088 \times 20 = 442.28\]

The predicted monetary crop yield at an average soil moisture of 20% is Rs 442.28.

9.4 Correlation and regression

Correlation and regression are closely related concepts in statistics, often used together to explore and model relationships between variables. While both techniques examine how variables relate to one another, they differ in purpose, interpretation, and methodology.

Correlation focuses on measuring the strength and direction of an association between two variables, without assuming causation. Regression goes a step further by modelling the relationship, enabling predictions of one variable based on another. Table 9.2 below provides a comparison of the two approaches.

Table 9.2: Correlation versus regression

Aspect	Correlation	Regression
Definition	A statistical measure that quantifies the strength and direction of the linear relationship between two variables.	A statistical technique that models the relationship between a dependent variable and one or more independent variables for explanation or prediction.
Primary objective	To measure the degree of association between variables.	To estimate or predict the value of the dependent variable from the independent variable(s).
Nature of relationship	Describes association only.	Describes the functional relationship between variables through a fitted mathematical model.
Causation	Does not imply causation.	Does not establish causation by itself; causal interpretation requires appropriate study design and valid assumptions.
Variables involved	Treats both variables equally; no distinction between dependent and independent variables.	Clearly distinguishes between the dependent (response) variable and the independent (predictor) variable(s).
Symmetry	Symmetric: \(r_{xy}=r_{yx}\).	Asymmetric: the regression of \(y\) on \(x\) differs from the regression of \(x\) on \(y\).
Equation	Summarised by the correlation coefficient, e.g., \(r=\dfrac{\operatorname{Cov}(x,y)}{\sigma_x\sigma_y}\).	Produces a regression equation, e.g., \(\hat{y}=\hat{\alpha}+\hat{\beta}x\).
Output	Produces a single correlation coefficient (\(r\)).	Produces regression coefficients (intercept and slope), fitted values, residuals, and prediction equations.
Range	The correlation coefficient lies between \(-1\) and \(+1\).	Regression coefficients are not bounded and may take any real value (\(-\infty\) to \(+\infty\)).
Units	Dimensionless (unit-free).	Regression coefficients depend on the units of measurement of the variables.
Effect of interchanging variables	Unchanged; the value of \(r\) remains the same.	Changes the regression equation and regression coefficients.
Prediction	Cannot be used directly for prediction.	Specifically designed for prediction and estimation of the dependent variable.
Applications	Assessing the strength and direction of relationships between variables.	Prediction, estimation, trend analysis, forecasting, and quantifying the effect of explanatory variables.

9.5 Chapter Summary

Fill in the blanks

Answers are given at the end of the chapter.

1. Regression analysis is used to understand and quantify the __________ between variables.

2. The variable whose value is explained, predicted, or modelled is called the __________ variable.

3. The variable used to explain or predict the dependent variable is called the __________ variable.

4. The dependent variable is commonly denoted by __________.

5. The independent variable is commonly denoted by __________.

6. In agricultural research, crop yield is commonly treated as the __________ variable.

7. A regression involving one dependent variable and one independent variable is called __________ linear regression.

8. Regression involving one dependent variable and two or more independent variables is called __________ linear regression.

9. Regression used when the relationship between variables is not a straight line is called __________ regression.

10. Regression used when the dependent variable is categorical is called __________ regression.

11. A fitted line used to represent the relationship between two variables is called the line of __________ fit.

12. The difference between an observed value and the value predicted by the fitted regression line is called a __________.

13. The difference between an observed value and the value based on the true regression line is called an __________.

14. Residuals are __________, whereas errors are theoretical and generally unknown.

15. The principle used to obtain the line of best fit by minimising the sum of squared residuals is called the principle of __________ squares.

16. The equation of a straight line is \(Y=\alpha+\beta X\), where \(\alpha\) is the __________ and \(\beta\) is the __________.

17. The slope of a regression line represents the expected change in \(Y\) for a __________-unit increase in \(X\).

18. The regression coefficient is denoted by __________.

19. The intercept represents the value of \(Y\) when \(X\) is equal to __________.

20. The fitted regression equation is written as \(\hat{y}=\) __________.

21. The regression line always passes through the point __________.

22. The two normal equations are obtained by differentiating the sum of squared errors with respect to __________ and __________.

23. The regression coefficient \(\hat{\beta}\) can be expressed as the ratio of __________ to __________.

24. A positive regression coefficient indicates a __________ relationship between \(x\) and \(y\).

25. A negative regression coefficient indicates an __________ relationship between \(x\) and \(y\).

26. Regression coefficients can take any real value from __________ to __________.

27. The regression of \(y\) on \(x\) is used to predict __________ from __________.

28. The regression of \(x\) on \(y\) is used to predict __________ from __________.

29. The regression coefficient of \(y\) on \(x\) is denoted by __________.

30. The regression coefficient of \(x\) on \(y\) is denoted by __________.

31. The product of the two regression coefficients is equal to the __________ of the correlation coefficient.

32. The two regression coefficients always have the same __________.

33. If one regression coefficient is greater than one, the other must be __________ than one.

34. Both regression coefficients cannot exceed __________ simultaneously.

35. Regression coefficients are independent of change of __________ but not of change of __________.

36. If \(r=0\), both regression coefficients are __________.

37. If \(r=\pm1\), the two regression lines __________.

38. The regression lines of \(y\) on \(x\) and \(x\) on \(y\) intersect at the point __________.

39. The regression line of \(y\) on \(x\) minimises the sum of squared __________ deviations.

40. The regression line of \(x\) on \(y\) minimises the sum of squared __________ deviations.

41. One of the assumptions of linear regression is __________.

42. Constant variance of residuals is called __________.

43. In multiple regression, the independent variables should not be highly correlated with each other. This condition is called absence of __________.

44. The errors in a regression model should not exhibit __________.

45. The errors are often summarised as \(e\sim\text{i.i.d.}(0,\sigma^2)\), where i.i.d. means __________ and __________ distributed.

46. Regression does not establish __________ by itself.

47. Correlation treats the two variables __________, whereas regression distinguishes between dependent and independent variables.

48. Correlation is __________ when the two variables are interchanged, whereas regression is not.

49. The correlation coefficient is bounded between __________ and __________, whereas regression coefficients are unbounded.

50. The regression coefficient can be expressed in terms of correlation and standard deviations as \(\beta_{yx}=\) __________.

Short-answer questions

1. Define regression analysis and explain its importance.

2. Distinguish between dependent and independent variables.

3. Explain the different types of regression.

4. What is simple linear regression?

5. What is multiple linear regression?

6. What is nonlinear regression?

7. What is logistic regression?

8. What is a line of best fit?

9. Explain the difference between error and residual.

10. Define the principle of least squares.

11. Why is the sum of errors not minimised directly in least squares estimation?

12. Explain the meaning of intercept and slope in a straight line.

13. Derive the normal equations for simple linear regression.

14. Explain the estimation of the regression coefficient using the normal equations.

15. Explain how the intercept of a regression line is estimated.

16. What is the meaning of a regression coefficient?

17. What is the interpretation of a positive regression coefficient?

18. What is the interpretation of a negative regression coefficient?

19. Explain the assumptions of simple linear regression.

20. What is homoscedasticity?

21. What is multicollinearity?

22. What is autocorrelation?

23. Why is normality of residuals important in regression analysis?

24. What is meant by correct model specification?

25. Explain the two lines of regression.

26. Distinguish between regression of \(y\) on \(x\) and regression of \(x\) on \(y\).

27. State the important properties of regression coefficients.

28. State the relationship between regression coefficients and correlation coefficient.

29. Explain the relationship between regression coefficients and standard deviations.

30. What happens to regression coefficients when the origin is changed?

31. What happens to regression coefficients when the scale is changed?

32. Explain the relationship between regression coefficients and unity.

33. State the properties of regression lines.

34. Why do the two regression lines always pass through \((\bar{x},\bar{y})\)?

35. Under what condition do the two regression lines coincide?

36. Explain the uses of regression analysis in agricultural research.

37. Distinguish between correlation and regression.

Numerical and conceptual questions

Answers are given at the end of the chapter.

1. The fitted regression equation is \(\hat{y}=10+2x\). Interpret the intercept and regression coefficient.

2. For the regression equation \(\hat{y}=25-3x\), interpret the regression coefficient.

3. If \(\bar{x}=10\), \(\bar{y}=30\), and \(\hat{\beta}=2\), calculate the intercept.

4. If \(\bar{x}=15\), \(\bar{y}=50\), and \(\hat{\beta}=-2\), calculate the intercept.

5. If \(\hat{\alpha}=5\) and \(\hat{\beta}=3\), calculate the predicted value of \(y\) when \(x=10\).

6. The regression equation is \(\hat{y}=-159.477+30.088x\). Estimate the crop yield when soil moisture is \(20\%\).

7. If \(cov(x,y)=20\) and \(var(x)=5\), calculate the regression coefficient of \(y\) on \(x\).

8. If \(cov(x,y)=18\) and \(var(y)=9\), calculate the regression coefficient of \(x\) on \(y\).

9. If \(\beta_{yx}=0.8\) and \(\beta_{xy}=0.45\), calculate the correlation coefficient.

10. If \(\beta_{yx}=-0.8\) and \(\beta_{xy}=-0.45\), calculate the correlation coefficient.

11. If \(r=0.6\) and \(\sigma_y=12\), \(\sigma_x=8\), calculate \(\beta_{yx}\).

12. If \(r=0.6\), \(\sigma_x=8\), and \(\sigma_y=12\), calculate \(\beta_{xy}\).

13. If \(r=0\), what are the values of the two regression coefficients?

14. If \(r=1\), what is the product of the two regression coefficients?

15. If \(r=-1\), what is the product of the two regression coefficients?

16. If \(\beta_{yx}=2\) and \(\beta_{xy}=0.25\), verify the relationship between the regression coefficients and the correlation coefficient.

17. If one regression coefficient is 1.5, what can be said about the other regression coefficient?

18. If both regression coefficients are negative, what can be said about the sign of the correlation coefficient?

19. A fitted regression equation is \(\hat{y}=20+4x\). Estimate \(y\) for \(x=5\), \(x=10\), and \(x=15\).

20. A regression equation has \(\bar{x}=20\), \(\bar{y}=80\), and slope \(=2.5\). Find the intercept and write the regression equation.

21. Given the data below, calculate the regression coefficient of \(y\) on \(x\).

\(x\)	2	4	6	8	10
\(y\)	4	8	12	16	20

22. Using the data in Question 21, obtain the regression equation of \(y\) on \(x\).

23. Explain why the regression of \(y\) on \(x\) and regression of \(x\) on \(y\) are generally different.

24. Explain why correlation cannot be directly used to predict the value of \(y\) for a given value of \(x\).

25. If the residual for an observation is \(-5\), what does this indicate about the observed and predicted values?

26. If the residuals become increasingly spread out as \(x\) increases, which regression assumption may be violated?

27. If the residuals show a systematic pattern against \(x\), what does this suggest about the regression model?

28. If two independent variables in a multiple regression model are highly correlated, what problem may arise?

Important formulae

Straight-line equation:

\[ Y=\alpha+\beta X \]

Simple linear regression model:

\[ y_i=\alpha+\beta x_i+e_i \]

Residual:

\[ e_i=y_i-(\alpha+\beta x_i) \]

Sum of squared errors:

\[ E=\sum_{i=1}^{n}e_i^2 \]

Sum of squared errors in terms of the regression model:

\[ E=\sum_{i=1}^{n}[y_i-(\alpha+\beta x_i)]^2 \]

First normal equation:

\[ \sum_{i=1}^{n}y_i=n\alpha+\beta\sum_{i=1}^{n}x_i \]

Second normal equation:

\[ \sum_{i=1}^{n}x_iy_i=\alpha\sum_{i=1}^{n}x_i+\beta\sum_{i=1}^{n}x_i^2 \]

Mean relationship:

\[ \bar{y}=\alpha+\beta\bar{x} \]

Intercept:

\[ \hat{\alpha}=\bar{y}-\hat{\beta}\bar{x} \]

Regression coefficient of \(y\) on \(x\):

\[ \hat{\beta}_{yx}= \frac{\sum x_iy_i-\frac{\sum x_i\sum y_i}{n}} {\sum x_i^2-\frac{(\sum x_i)^2}{n}} \]

Regression coefficient using covariance and variance:

\[ \hat{\beta}_{yx}=\frac{cov(x,y)}{var(x)} \]

Regression equation of \(y\) on \(x\):

\[ \hat{y}=\hat{\alpha}+\hat{\beta}_{yx}x \]

Regression coefficient of \(x\) on \(y\):

\[ \hat{\beta}_{xy}=\frac{cov(x,y)}{var(y)} \]

Regression equation of \(x\) on \(y\):

\[ \hat{x}=\hat{\alpha}_1+\hat{\beta}_{xy}y \]

Relationship between regression coefficients and correlation:

\[ \beta_{yx}\beta_{xy}=r_{xy}^2 \]

Correlation coefficient from regression coefficients:

\[ |r_{xy}|=\sqrt{\beta_{yx}\beta_{xy}} \]

If both regression coefficients are positive:

\[ r_{xy}=\sqrt{\beta_{yx}\beta_{xy}} \]

If both regression coefficients are negative:

\[ r_{xy}=-\sqrt{\beta_{yx}\beta_{xy}} \]

Regression coefficient in terms of correlation and standard deviations:

\[ \beta_{yx}=r\frac{\sigma_y}{\sigma_x} \]

\[ \beta_{xy}=r\frac{\sigma_x}{\sigma_y} \]

Change of origin and scale:

\[ X=\frac{x-a}{h},\qquad Y=\frac{y-b}{k} \]

Transformed regression coefficients:

\[ \beta_{YX}=\frac{h}{k}\beta_{yx} \]

\[ \beta_{XY}=\frac{k}{h}\beta_{xy} \]

Multiple linear regression model:

\[ y=\alpha+\beta_1x_1+\beta_2x_2+\cdots+\beta_kx_k+e \]

Quick revision

• Regression → models the relationship between a dependent variable and one or more independent variables.
• Dependent variable → response or outcome to be predicted or explained.
• Independent variable → predictor or explanatory variable.
• Simple linear regression → one dependent variable and one independent variable.
• Multiple linear regression → one dependent variable and two or more independent variables.
• Nonlinear regression → relationship is not a straight line.
• Logistic regression → categorical dependent variable.
• Line of best fit → fitted line obtained using least squares.
• Error → theoretical difference from the true regression relationship.
• Residual → observed value minus fitted value.
• Least squares → minimises the sum of squared residuals.
• Slope \(\beta\) → expected change in \(y\) for a one-unit increase in \(x\).
• Intercept \(\alpha\) → value of \(y\) when \(x=0\).
• Regression coefficient can range from \(-\infty\) to \(+\infty\).
• Positive regression coefficient → direct relationship.
• Negative regression coefficient → inverse relationship.
• Regression of \(y\) on \(x\) → predict \(y\) from \(x\).
• Regression of \(x\) on \(y\) → predict \(x\) from \(y\).
• The regression line passes through \((\bar{x},\bar{y})\).
• Both regression coefficients have the same sign.
• \(\beta_{yx}\beta_{xy}=r^2\).
• If \(r=0\), both regression coefficients are zero.
• If \(r=\pm1\), the two regression lines coincide.
• If one regression coefficient exceeds 1, the other must be less than 1.
• Both regression coefficients cannot exceed 1 simultaneously.
• Regression coefficients are independent of change of origin but not of change of scale.
• Regression coefficients depend on the correlation and the relative standard deviations of the variables.
• Linearity, independence, homoscedasticity, normality of residuals, absence of multicollinearity, absence of autocorrelation, and correct model specification are important assumptions.
• Regression lines are unique for a given dataset.
• Correlation measures association; regression models the relationship and allows prediction.
• Correlation is symmetric; regression is asymmetric.
• Correlation coefficient is unit-free and lies between \(-1\) and \(+1\).
• Regression coefficients have units and are not bounded.
• Regression does not establish causation by itself.

Answers to fill in the blanks

1. Relationships 2. Dependent 3. Independent 4. \(y\) 5. \(x\) 6. Dependent 7. Simple 8. Multiple 9. Nonlinear 10. Logistic 11. Best 12. Residual 13. Error 14. Measurable 15. Least 16. Intercept; Slope 17. One 18. \(\beta\) 19. Zero 20. \(\hat{\alpha}+\hat{\beta}x\) 21. \((\bar{x},\bar{y})\) 22. \(\alpha\); \(\beta\) 23. Covariance; Variance 24. Direct 25. Inverse 26. \(-\infty\); \(+\infty\) 27. \(y\); \(x\) 28. \(x\); \(y\) 29. \(\beta_{yx}\) 30. \(\beta_{xy}\) 31. Square 32. Sign 33. Less 34. One 35. Origin; Scale 36. Zero 37. Coincide 38. \((\bar{x},\bar{y})\) 39. Vertical 40. Horizontal 41. Linearity 42. Homoscedasticity 43. Multicollinearity 44. Autocorrelation 45. Independent; identically 46. Causation 47. Equally 48. Symmetric 49. \(-1\); \(+1\) 50. \(r\frac{\sigma_y}{\sigma_x}\)

Solutions to numerical and conceptual questions

1. In \(\hat{y}=10+2x\), the intercept \(\hat{\alpha}=10\) is the predicted \(y\) when \(x=0\), and the coefficient \(\hat{\beta}=2\) means \(y\) increases by 2 units for every one-unit increase in \(x\).

2. In \(\hat{y}=25-3x\), the coefficient \(\hat{\beta}=-3\) means \(y\) decreases by 3 units for every one-unit increase in \(x\).

3. Using Equation 9.18, \(\hat{\alpha}=\bar{y}-\hat{\beta}\bar{x}=30-2(10)=10\).

4. Using Equation 9.18, \(\hat{\alpha}=50-(-2)(15)=80\).

5. Using Equation 9.19, \(\hat{y}=5+3(10)=35\).

6. Using Equation 9.19, \(\hat{y}=-159.477+30.088(20)=442.28\) (Rs).

7. Using Equation 9.21, \(\hat{\beta}_{yx}=\frac{cov(x,y)}{var(x)}=\frac{20}{5}=4\).

8. Using Equation 9.23, \(\hat{\beta}_{xy}=\frac{cov(x,y)}{var(y)}=\frac{18}{9}=2\).

9. Since \(r=\sqrt{\beta_{yx}\beta_{xy}}\), \(r=\sqrt{0.8\times0.45}=0.6\).

10. Since both coefficients are negative, \(r=-\sqrt{\beta_{yx}\beta_{xy}}=-\sqrt{0.8\times0.45}=-0.6\).

11. Using \(\beta_{yx}=r\frac{\sigma_y}{\sigma_x}=0.6\times\frac{12}{8}=0.9\).

12. Using \(\beta_{xy}=r\frac{\sigma_x}{\sigma_y}=0.6\times\frac{8}{12}=0.4\).

13. If \(r=0\), both regression coefficients are zero, so \(\beta_{yx}=\beta_{xy}=0\).

14. If \(r=1\), \(\beta_{yx}\beta_{xy}=r^2=1\).

15. If \(r=-1\), \(\beta_{yx}\beta_{xy}=r^2=1\).

16. \(\beta_{yx}\beta_{xy}=2\times0.25=0.5=r^2\), so \(r=\sqrt{0.5}=0.707\), verifying \(\beta_{yx}\beta_{xy}=r^2\).

17. Since \(\beta_{yx}\beta_{xy}=r^2\le1\), if one coefficient is \(1.5\) the other must be less than 1 (here at most \(1/1.5=0.667\)).

18. If both regression coefficients are negative, the correlation coefficient is also negative.

19. Using Equation 9.19 with \(\hat{y}=20+4x\): \(\hat{y}=40\) at \(x=5\), \(\hat{y}=60\) at \(x=10\), and \(\hat{y}=80\) at \(x=15\).

20. Using Equation 9.18, \(\hat{\alpha}=80-2.5(20)=30\), so the regression equation is \(\hat{y}=30+2.5x\).

21. Using Equation 9.21, \(\bar{x}=6\), \(\bar{y}=12\), \(cov(x,y)=16\), \(var(x)=8\), so \(\hat{\beta}_{yx}=\frac{16}{8}=2\).

22. Using Equation 9.18, \(\hat{\alpha}=12-2(6)=0\), so the regression equation is \(\hat{y}=2x\).

23. The two lines use different coefficients, \(\beta_{yx}=\frac{cov(x,y)}{var(x)}\) and \(\beta_{xy}=\frac{cov(x,y)}{var(y)}\), so the regression of \(y\) on \(x\) and of \(x\) on \(y\) are generally different (they coincide only when \(|r|=1\)).

24. Correlation gives only the strength and direction of association, not an equation; regression produces a fitted equation \(\hat{y}=\hat{\alpha}+\hat{\beta}x\) that can predict \(y\) for a given \(x\).

25. A residual \(e=y-\hat{y}=-5\) means the observed value lies 5 units below the predicted value.

26. Residuals spreading out as \(x\) increases violates the assumption of constant variance, i.e. homoscedasticity.

27. A systematic pattern in the residuals suggests the model is misspecified (wrong functional form, omitted variables, or non-linearity).

28. If two independent variables are highly correlated, multicollinearity arises, making the coefficient estimates unstable and hard to interpret.

Historical Insights

Regression and the Study of Heights

The story of regression begins with Sir Francis Galton’s groundbreaking work on heredity in the late nineteenth century. While studying the heights of parents and their children, Galton noticed a fascinating pattern: tall parents tended to have slightly shorter children, and shorter parents tended to have slightly taller children. He called this phenomenon “regression toward the mean” - the offspring’s heights seemed to move closer to the population average, not further away from it. This observation not only introduced the term “regression” into statistics but also inspired the development of tools for studying relationships between variables. (Galton 1886)

Galton’s work was later placed on a firm mathematical foundation by Karl Pearson, who formalised the method of least squares in the context of regression and introduced the product-moment correlation coefficient - the same \(r\) we use today. (Pearson 1896) Together, their contributions laid the foundation for modern regression analysis, which remains an essential technique in fields ranging from agriculture to space exploration.

Quotes to Inspire

“Statistics is the art of never having to say you’re wrong.”
- Robert P. Abelson

Draper, Norman R., and Harry Smith. 1998. Applied Regression Analysis. 3rd ed. Wiley Series in Probability and Statistics. New York: John Wiley & Sons.

Galton, Francis. 1886. “Regression Towards Mediocrity in Hereditary Stature.” The Journal of the Anthropological Institute of Great Britain and Ireland 15: 246–63. https://doi.org/10.2307/2841583.

Montgomery, Douglas C., Elizabeth A. Peck, and G. Geoffrey Vining. 2021. Introduction to Linear Regression Analysis. 6th ed. Hoboken, NJ: John Wiley & Sons.

Pearson, Karl. 1896. “Mathematical Contributions to the Theory of Evolution. III. Regression, Heredity, and Panmixia.” Philosophical Transactions of the Royal Society of London, Series A 187: 253–318. https://doi.org/10.1098/rsta.1896.0007.