Robust regression, like robust statistics in general, is an approach that tries to minimize the effect of outliers. Traditional least squares regression is sensitive to noisy data—if one point in the data is way off from the others, purely by chance, it can badly distort the results of the regression compared to what you’d obtain from analyzing other samples from the same population.

One example of the many techniques for robust regression is the method of least absolute deviation (LAD). Given:

• a response vector [math]y = (y_1, \ldots, y_n)[/math] of length [math]n[/math],
• a predictor matrix [math]X[/math] of size [math]n \times p[/math], and
• a coefficient vector [math]\beta[/math] of length [math]p[/math],

let [math]\hat{y} = X \beta[/math].

If you check the linear algebra, you’ll see that this means [math]\hat{y}[/math] is also a vector of length [math]n[/math]. Least squares regression attempts to find [math]\beta[/math] to minimize:

[math]\sum_{i=1}^n \left( y_i - \hat{y}_i \right)^2[/math]

while LAD attempts to find [math]\beta[/math] to minimize:

[math]\sum_{i=1}^n \left| y_i - \hat{y}_i \right|.[/math]

Because LAD is summing over absolute differences rather than squared differences, outliers don’t make as much of a difference. There are a number of algorithms for this—none of which, unfortunately, are anywhere near as simple or elegant as the normal equations for linear regression. But it’s implemented in the L1pack package for R, and probably in other packages for other languages.

This is really only the tip of the iceberg as far as robust regression goes. Because of the problems outliers pose for least-squares methods, it’s been a subject of research for decades, and probably will continue to be for some time to come.

Ordinary Least Squares (OLS) is a general method for deciding what parameter estimates provide the ‘best’ solution. “Least squares” refers to what you want to minimize: the sum of squared prediction errors (SSE) The ‘best’ or optimal model is often defined as the one that minimizes sum of squared prediction errors (see note at end for more information).

Multiple regression is one of many statistical techniques for which parameter estimates can be obtained based on OLS method. Other statistics (such as the sample mean M) can also found using OLS methods.

For simple models, the same parameter estimates that work best can also be obtained using Maximum Likelihood (ML) estimation methods (in other words, OLS and ML are two of the most common methods of paramer estimation, there are many others).

For complicated models, such as Structural Equation Models, OLS methods do not work; instead, we have to use ML or other estimation methods.

MORE DETAIL:

I won’t go into all the math, but: You can set up an equation that gives the value of SSE as a funciton of b in a bivariate regression Y’= b0 + b1*X, as shown in the figure below:

The derivative of this function is the point where the curve reaches its minimum value; solving for the derivative gives the equation to estimate b.

M is the value for which Sum of (X-M) = 0 and Sum of Squares of deviations from the mean has the minimum possible value.

Ordinary Least Squares (OLS) regression is a widely used method for fitting linear regression models. While OLS regression has several advantages, it also has some limitations and potential problems. Here are some of the main issues associated with OLS regression:

1. Sensitivity to Outliers: OLS regression is highly sensitive to outliers in the data. Outliers can significantly influence the estimated coefficients, leading to biased and unreliable results. The ordinary least squares method tries to minimize the sum of squared residuals, so even a single extreme outlier can heavily impact the estimated regression line.
2. Violation of Assumptions: OLS regression assumes certain conditions, and violations of these assumptions can lead to inaccurate results. Key assumptions include linearity (the relationship between predictors and the response is linear), independence of errors, constant variance of errors (homoscedasticity), and normality of error terms. If these assumptions are violated, the estimated coefficients may be biased, and statistical inference can be invalid.
3. Multicollinearity: Multicollinearity occurs when predictor variables in the regression model are highly correlated with each other. This situation can cause issues in OLS regression, as it becomes difficult to distinguish the individual effects of correlated predictors. Multicollinearity can lead to unstable and imprecise coefficient estimates and reduce the interpretability of the model.
4. Overfitting or Underfitting: OLS regression may suffer from overfitting or underfitting issues. Overfitting occurs when the model is overly complex and fits the noise in the training data, leading to poor generalization on new data. Underfitting occurs when the model is too simple and fails to capture the underlying patterns in the data. Both cases result in suboptimal predictive performance.
5. Nonlinear Relationships: OLS regression assumes a linear relationship between predictors and the response variable. When the true relationship is nonlinear, OLS regression may provide inadequate predictions and inaccurate coefficient estimates. In such cases, alternative regression techniques like polynomial regression or nonparametric models may be more appropriate.

Even treating the distribution of the regressors as fixed, you can still define a covariance between the regressors and the error term.

If this covariance does not equal zero, then the exogeneity condition [math]\mathbb{E}[\epsilon|x] = 0[/math] is necessarily violated.

People often use “covariance” and “correlation” interchangeable when speaking loosely, because the correlation is just the covariance divided by a normalization, and the two always have the same sign:

[math]\displaystyle \text{cov}(x,\epsilon) = \mathbb{E}\left[(x - \mathbb{E}[x]) (\epsilon - \mathbb{E}[\epsilon]) \right][/math]

[math]\displaystyle \text{corr}(x,\epsilon) = \frac{\text{cov}(x,\epsilon)}{\sigma_x \sigma_\epsilon}[/math]

If one of the variables in question is a constant, then it has a standard deviation of zero and this normalization is not well-defined. But as long as you have multiple values of some regressor [math]x[/math], you can still define [math]\sigma_x[/math], even if the interpretation of this standard deviation is strange. And if you do not have multiple values of [math]x[/math], then your regression is unidentified to begin with.

There are actually quite a few ways to do this, made easier if we write the linear regression model in matrix notation as [math]Y=X\beta+\varepsilon[/math], where:

• [math]Y[/math] is a column vector of length [math]n[/math], representing the values of the outcome variable for each observation;
• [math]X[/math] is an [math]n\times p[/math] data matrix containing the values of the [math]p[/math] predictor variables (including the intercept term, which is just a column of ones) for the [math]n[/math] observations;
• [math]\beta[/math] is a column vector of length [math]p[/math] representing the [math]p[/math] slope coefficients; and
• [math]\varepsilon[/math] is a column vector of length [math]n[/math] representing the error term.

The estimator follows quite naturally if you think about what we’re actually doing, which is to minimize the sum of squared errors. That sum of squared errors is actually [math]\hat{\varepsilon}'\hat{\varepsilon}[/math], where [math]\hat{\varepsilon}=Y-X\hat{\beta}[/math], more commonly referred to as the residuals of the model—what’s leftover once you estimate [math]\beta[/math], plug in [math]X[/math], and subtract that off [math]Y.[/math]

So let’s expand [math]\hat{\varepsilon}'\hat{\varepsilon}[/math] and write it in terms of [math]X[/math], [math]Y[/math], and [math]\hat{\beta}[/math], which is ultimately what we’re trying to solve for:

[math]\begin{align}\qquad\qquad\qquad\hat{\varepsilon}'\hat{\varepsilon} & =(Y-X\hat{\beta})'(Y-X\hat{\beta}) \\ & =Y'Y-Y'X\hat{\beta}-(X\hat{\beta})'Y+\hat{\beta}'X'X\hat{\beta} \\ & =Y'Y-2\hat{\beta}'X'Y+\hat{\beta}'X'X\hat{\beta}\end{align}[/math]

(Don’t worry if you can’t really follow the matrix calculus or algebra so easily; if you squint you can see some analogies with scalar algebra like you’re used to, with [math]A'A[/math] showing up where you’d usually expect [math]a^2[/math] to show up and similarly with left-multiplication by [math]A^{-1}[/math] and dividing by [math]a[/math].) We can then take the derivative of this with respect to [math]\hat{\beta}[/math] and set that equal to zero:

[math]\begin{align}\qquad\qquad\qquad\frac{\text{d}}{\text{d}\hat{\beta}}\hat{\varepsilon}'\hat{\varepsilon} & =-2X'Y+2X'X\hat{\beta} \\ & :=0\end{align}[/math]

and solve for [math]\hat{\beta}[/math]:

[math]\begin{align}\qquad\qquad\qquad 2X'X\hat{\beta} & =2X'Y\hat{\beta} \\ & =(X'X)^{-1}X'Y\end{align}[/math].

Furthermore, we can take the second derivative, find that it’s positive definite for all values of [math]\hat{\beta}[/math], and therefore [math]\hat{\beta}[/math] does indeed minimize (and not maximize) the sum of squared residuals. So [math]\hat{\beta}=(X'X)^{-1}X'Y[/math] is the OLS estimator for [math]\beta[/math].

You should note, at the end here, that several facts about OLS are implied by this estimator. First, you may recall that for [math]\hat{\beta}[/math] to be an unbiased estimator of [math]\beta[/math]—that is, [math]E(\hat{\beta})=\beta[/math]—we require the assumption that [math]X[/math] and [math]\varepsilon[/math] be uncorrelated. In matrix algebra terms, that’s equivalent to assuming [math]X'\varepsilon=0[/math]. Watch what happens when we plug [math]\hat{\beta}[/math] back into the linear regression model:

[math]\begin{align}\qquad\qquad\qquad Y & =X(X'X)^{-1}X'Y+\varepsilon \\ X'Y & = X'X(X'X)^{-1}X'Y+X'\varepsilon \\ X'Y & = X'X(X'X)^{-1}X'Y+X'\varepsilon \\ X'Y & = X'Y+X'\varepsilon \\ \qquad\qquad\qquad X'\varepsilon & =0.\end{align}[/math]

So [math]\hat{\beta}[/math] being unbiased implies [math]X'\varepsilon=0[/math]; put another way, if [math]X[/math] and [math]\varepsilon[/math] are correlated, then it’s not true that [math]\hat{\beta}[/math] is an unbiased estimator of [math]\beta[/math]. Specifically, it suffers from omitted variables bias: you forgot to include a predictor that’s correlated with both [math]X[/math] and [math]Y[/math], so it got buried in the error term. A lot of research in econometrics was really an attempt to justify the simple claim that [math]X'\varepsilon=0[/math].

Second, and related to this, is that many introductory econometrics students mistakenly think that we can test the assumption [math]X'\varepsilon=0[/math] by looking at [math]X'\hat{\varepsilon}[/math]—surely, if [math]\hat{\varepsilon}[/math] is an estimator of [math]\varepsilon[/math], then we just need to look at [math]X'\hat{\varepsilon}[/math] to see if we have omitted variables bias? The answer, unfortunately, is no. Go back to where we differentiated [math]\hat{\varepsilon}'\hat{\varepsilon}[/math] with respect to [math]\hat{\beta}[/math] and set it equal to zero; we can equivalently write

[math]\qquad\qquad\qquad X'(Y-X\hat{\beta})=X'\hat{\varepsilon}=0.[/math]

So if we’re minimizing the sum of squared residuals, it’s necessarily the case that the predictors are uncorrelated with the residuals. Checking whether their covariance is equal to zero tells you nothing.

Finally, the assumption of no perfect multicollinearity between predictor variables is not only necessary for OLS to be unbiased, it’s necessary for the OLS estimate to exist at all. If you have any form of linear dependence between predictor variables—that is, if you can take some linear combination of some variables to get exactly the values of another variable—then [math]X'X[/math] doesn’t have an inverse. It’s the matrix algebra equivalent of dividing by zero.

There are some concerns with acronyms. We know what OLS is - I often substitute CLR Classical Linear Regression. Then there’s General Linear Model GLM which is “a useful framework for comparing how several variables affect different continuous variables … (Rutherford, (2001)). GLM is the foundation for several statistical tests, including ANOVA, ANCOVA and regression analysis.” General Linear Model (GLM): Simple Definition / Overview - Statistics How To

Then your Generalized Least Squares is a different formulation that is supposed to correct better for correlation in the equation error. That often makes OLS inconsistent. Aitken, A. C. (1936) Generalized least squares - Wikipedia

Assumptions of CLRM

1. Model should be linear in parameters.

2. None of the independent variables have linear relationship with any other independent variables. {i.e., no Multicollinearity}

3. None of the independent variables are correlated with mean of error

4. Error term observation are independent to each other OR they are not correlated to each other. {i.e., no autocorrelation}

5. Mean of error term are zero.

6. Error term has a constant variance. {i.e., no Heteroscedasticity}

7. The error term is normally distributed.

If you go through these assumptions, you will get the answer., please see link below to understand the CLRM.

Ordinary least sqares or linear least sqares estimates the parameters in regression model by minimising the sum of sqares residuals

This method draws a line through the data points that minimises the sum of the sqared differences between the observed and the the corresponding fit red values

OL S is to fit closely a function with the data and does minimising the sum of the sqared errors from the data

Basic assumption of OL S is that linear model should produce residuals that halve a mean of zero and constant variance and are not CORRELATED with themselves or other VARIABLES

If assumption holds good then OLS gives the best possible estimates

General linear model GLM /GLS refers to linear regression models for a continuous response variable given CONTINOUS and or categorical predictors

It includes ANOVA and ANCOVA

Unlike ordinary regression, logistic regression is used to estimate a variable’s effect on a binary dependent variable (true or false, male or female, formal or informal sector, etc.). Suppose we want to explore the effect of a weekly number of hours of sleep on the likelihood of a child moving up a grade. Our explanatory variable, the number of hours of sleep, is a continuous variable between 0 and 168 so it can be 15, 26.5, 47.3, 68, 100, and so on. Meanwhile, our dependent variable is a probability that only takes the value 0 or 1.

Our regression equation becomes,

I will now explain why this makes sense.

First, if we use a linear regression equation,

This means we regress a binary variable (only 0 or 1) on a continuous variable. Meanwhile, we have an unlimited number on the right-hand side of the equation. The value can be more than 1, less than 0, or somewhere in between. So, this doesn’t make sense.

Then, if we use the regression equation

This equation is also problematic because ln(0) is equal to -∞ and ln(1) is equal to 1. We only have values between -∞ and 1 for the predictor which can be up to +∞. So, this equation cannot be used either.

The correct equation is

Because if we solve this equation, we will get (assume the error is zero),

Suppose β1 is a positive number. Then with hi close to ∞, we will get p close to 1. With hi close to -∞, then p will go to 0. However, if we plug in different values of hi then we will get a value of p between 0 and 1. Therefore, p is not only equal to 0 or 1, but it is 0 to 1. And if we draw a logistic function curve, it will be S-shaped from 0 to 1.

When a student has more hours of sleep, he or she is more likely to be promoted. The curve above also shows the difference between logistic regression and linear regression. In linear regression, we fit the line using least squares, which is the line that minimizes the sum of the squares of these residuals followed by calculating R-squared. Meanwhile, logistic regression does not calculate R-squared, but uses “maximum likelihood”.

The regressors may still be endogenous (anti-exogeneous), [math]([/math]i.e., [math]E\left[\bar{\epsilon} \ | \ \mathbf{X}\right]\ne 0)[/math] and in which the range of permissible values for the regressor [math]x_i[/math] in [math]\mathbf{X}[/math] below, are very small intervals [math]I_{b_i}[/math], [math]([/math]i.e. [math]\mu\left(I_{b_i}\right)\sim 0)[/math] such that [math]b_i[/math] is the midpoint in [math]I_{b_i}[/math]. The regressors may be considered to be approximately constants [math]b_i[/math]. In this case, the estimates from an [math]n[/math]-ordered OLS regression model [math](1)[/math] below are invalid and should be replaced with a model with instrumental variables.

[math]\mathbf{y}=\mathbf{X}\beta+\epsilon\tag{1}[/math]

That said, if a subset of regressors [math]x_i, i \in I\subset [1,\ldots,n][/math], in [math]\mathbf{X}[/math], are assumed to really be fixed, the OLS model is changed such that those fixed regressors can be replaced with fixed values and assimilated with the default [math]1[/math] value in [math]\mathbf{X}[/math] for [math]\beta_0[/math], to represent the new constant coefficient,

[math]\beta_0^*=\beta_0+\displaystyle\sum_{i\in I} b_i \beta_i,\tag{2}[/math]

where [math]b_i[/math] is the fixed value for [math]x_i[/math], for [math]i\in I[/math], and the model order in [math](1)[/math] is reduced by [math]|I|[/math]. If [math]I=[1,2,\ldots,n][/math], then [math](1)[/math] is a constant model fit,

[math]\mathbf{y}=\beta_0+\epsilon\tag{3}[/math]

and the OLS regression degenerates to [math]\hat{\beta}_0=\bar{y}[/math]. The concept of statistically correlating a random error [math]\epsilon[/math] with a constant, loses meaning.

I’m going to assume that you are given a dataset and when you ran the regression (OLS), and checked for heteroscedasticity, the null of no het was rejected immediately (P-value <0.05).

Here is where it gets interesting. If you were to continue using the normal OLS where your objective function is to minimize the sum of squared errors than your t stat and f stat will be invalid as they use standard errors for computation.

The reason why you switch to robust regression is to deal with this issue of non-constant variance in your residuals. However this wouldn’t change the location of your mean (beta) but would correct the standard errors. Prior to this, your prediction/confidence intervals will not factor this and hence can result in overconfidence in your results. By incorporating either HAC estimators or Whites adjustment, your prediction/confidence intervals will expand but your resulting vector estimates will be correct. Look up “HAC estimators” if you want to study this in more detail.

If you are doing classical time series analysis, than most likely your data will also suffer from autocorrelation where the covariance between et is dependent on et-s. Here also you might want to correct your OLS for robust standard errors.

Hope this all makes sense.

Please let me know if you have any follow up questions.

Good luck.

The effect of a group (two or more) of independent variables being substantially correlated (“Imperfect Multicollinearity”) is that the individual correlation coefficients can’t be trusted. Observational variation can cause a major shift in the values - even rounding error can have unexpectedly large effects on the results. This can also be seen in the ANOVA presentation of the regression analysis because the order of the variables will cause major changes in their how much variation they account for and their statistical significance.

One way out of this is perform a Principal Components Analysis on this group of variables and use the new variables (but fewer of them.)

OLS regression model - ordinary least squares regression. It is just a normal regression model, but the estimation is through least square method, not ML (maximum likelihood) or any other methods. Least square estimation is the most common method, at undergrad level, most students use OLS method without knowing it. As you have more data (collecting thousands of observations), your matrix is getting larger, then your estimation is through invert a matrix becomes impossible, OLS usefulness diminishes as data gets larger.

The model is still the same as below:

y = beta0+ beta1 * X1 + beta2 * X2 + … betap * Xp + error

X1, X2, …,Xp are independent variables (also known as attributes),

p - number of predictors,

An autoregressive model (AR) is when a value from a time series is compared to previous values from that same time series and a regression model is used to fit the data. The model is used to describe certain time-varying processes in nature, economics, etc. It specifies that the variable depends linearly on its own previous values and on a stochastic term (random term):

The order of an AR model is the number of immediately preceding values in the series that are used to predict the value at the present time. So, the preceding model is a first-order autoregression, written as AR(1).

The model is a second-order AR, written as AR(2), if the value at time t is predicted from the values at times t−1 and t−2. More generally, a kth-order AR, written as AR(k), is a multiple linear regression in which the value of the series at any time t is a (linear) function of the values at times t−1,t−2,…,t−k

Here’s a great article about time series forecasting : http://www.ssc.upenn.edu/~fdiebold/Teaching104/Ch14_slides.pdf

.

For the implementation of OLS regression in R, we use – Data (CSV)

So, let’s start with the steps with our first R linear regression model.

Step 1: First, we import the important library that we will be using in our code.

> library(caTools) 

Output:

Step 2: Now, we read our data that is present in the .csv format (CSV stands for Comma Separated Values).

> data = read.csv("/home/admin1/Desktop/Data/hou_all.csv") 

Output:

Step 3: Now, we will display the compact structure of our data and its variables with the help of str() function.

> str(data) 

Output:

Step 4: Then to get a brief idea about our data, we will output the first 6 data values using the head() function.

> head(data) 

Output:

Step 5: Now, in order to have an understanding of the various statistical features of our labels like mean, median, 1st Quartile value etc., we use the summary() function.

> summary(data) 

Output:

Step 6: Now, we will take our first step towards building our linear model. Firstly, we initiate the set.seed() function with the value of 125. In R, set.seed() allows you to randomly generate numbers for performing simulation and modeling.

> set.seed(125) 

Output:

Step 7: The next important step is to divide our data into training data and test data. We set the percentage of data division to 75%, meaning that 75% of our data will be training data and the rest 25% will be the test data.

> data_split = sample.split(data, SplitRatio = 0.75) 
> train <- subset(data, data_split == TRUE) 
> test <-subset(data, data_split == FALSE) 

Output:

Step 8: Now that our data has been split into training and test set, we implement our linear modeling model as follows:

model <- lm(X1.1 ~ X0.00632 + X6.575 + X15.3 + X24, data = train) #DataFlair 

Output:

Lastly, we display the summary of our model using the same summary() function that we had implemented above.

> summary(model) 

Output:

And, that’s it! You have implemented your first OLS regression model in R using linear modeling!

The constant term indicates where the line crosses the y-axis when all the x1, x2, x3, etc. are zero. Another way to think about it is how much to move the curve up or down. In the image below we observe three lines that follow the following functions:

Y_red = x^2 + 5

Y_blue = x^2 + 0

Y_green = x^2 – 5

Note that the shape of the three lines is the same, but the location in height is different, and controlled by the constant.

Sorry, but most of the answers to this question seem to confuse multivariate regression with multiple regression.

I know there is a lot of confusion about how they differ, but wrapping our head around the right terminology might actually make it easier to understand the different models are and how they relate to each other.

In general, the terms multivariate statistics and multivariate models is not used to indicate any statistical model using more than one variable. The term multivariate is reserved to describe models of more than one variable. That is models where the “thing being modelled” has many variables, not “the things predicting/explaining” the thing being modelled.

This might sound cryptic, but it relates to a more general misunderstanding of statistics as being more about the structural part of the model than about the error term/random part of the variable(s) studied. In a multivariate model it is the random part of the model that is multivariate. That means that the error term varies varies along all the different dimensions that the dependent variables vary in.

Multivariate regression is part of multivariate statistics, concerned with models with more than one outcome variable. So a multivariate regression model refers to regression models with at least two dependent/outcome variables which can be predicted by one or more independent variables.

An example of a multivariate regression model could be a model of how students answered on a range of different questions in a test (many dependent variables), dependent on how much they each studied, gender, age etc (many independent variables).

Multivariate regression models are also called general linear models or even multivariate general linear models. That is why the wikipedia page on Multivariate linear regression models redirects to General linear models.

Multivariate linear regression - Wikipedia

Multivariate statistics - Wikipedia

Here is an example of an introductory book to multivariate regression that explicitly makes the connection.

Amazon.com: Multivariate General Linear Models (Quantitative Applications in the Social Sciences) (9781412972499): Richard F. Haase: Books

A multiple regression model, on the other hand, refers to regression models with only one dependent/outcome variable and many independent/predictor variables. I believe that is what most people are thinking of when they talk about linear regression.

The confusion between multiple and multivariate models is not helped by the fact that 1) bivariate models refer to models with one independent and one dependent variable, and that 2) many introductory stats courses for non statisticians (at least in my experience) refer to any statistic based on more than one or maybe two variables as an example of multivariate statistics.