yes. and not just XGBoost, but any decision tree based classifier or ensemble of classifiers are not influenced by outliers, if you mean outliers that are exceptionally large or exceptionally small values (the most common meaning of outliers). So single trees, (CART, C5, CHAID, etc.), ensembles of trees (Random Forests, Gradient Boosting Machines, Bagging, Boosting) are all robust to outliers. Outliers literally don’t matter to trees because what makes an outlier an outlier (using the common meaning) is distance from other data points…outliers are far away, most often measured in Euclidean Distance. Trees do not care about distance. They only care if the data is on one side or the other of a split. It doesn’t matter at all how far from the split the data point is.

As an aside, it’s possible have an outlier (an unusual or atypical value) that is in the middle of a distribution. I’ve never seen this written up in the literature (let me know if you have a good reference) but I’ve dealt with this phenomenon in the past. These outliers are harder to detect because you have to know more about the multi-modal distribution of the variable to detect the unusual value. For example, if you have a bi-modal distribution, there may be a few values and only a few between the modes (think of a composition of two Normal distributions separated from one another). Trees would still be robust to these outliers.

Well, outliers can be identified just by plotting the data, that is simplest. The observation that doesn’t match with rest of observation would be outlier

The red dot is outlier

The more technical way is boxplot. Several softwares can make boxplot of data. In a boxplot, the data beyond whiskers may be treated as outlier

How to handle outlier? Well, this is a separate science and there is no unique solution to this. Sometimes the log transformation may work, sometime, you need to trim the data and sometime, you don’t need any treatment of the outlier

The question isn’t entirely pinned down, so it’s hard to know how to answer it rigorously. That said, they are roughly inverse to one another: a model that is not robust to outliers at all is very likely overfitting the data.

Consider a common example: estimating central tendency by the median, which is far more robust to outliers than the mean. One might argue that this underfits the data, since a true, extreme outlier can help invalidate your model (e.g., you have assumed the data are normal, and therefore have thin tails, making such an outlier nearly impossible).

But it’s not quite so simple. Underparameterized models — like assuming a parsimoniously parameterized distribution for complex univariate data — can also be non-robust while not overfitting the data. Adding an extreme outlier to a large set of otherwise normally-distributed data could have an enormous impact on both the mean and variance, but not really “overfit” the data, since it is now explaining all the data, including the outlier, badly. It’s basically misspecified.

Nonparametric models can be specifically designed to “detect” outliers and put them, so to speak, in a separate bucket, effectively making the model for the rest of the data robust. They would only grow in complexity (i.e., number of parameters) if the data warrant it, but the analyst could still say something like “the vast majority of the data are captured well by a normal (or Poisson, etc.) model, but we also have detected some anomalies.” I don’t think anyone would view that as “overfitting” the data, in the way that, say, polynomial regression might.

In machine learning, regularization is a standard way to help avoid overfitting, and basically enacts a penalty for doing so. It’s a big topic, but hopefully the description above helps sketch a broad relationship between the two.

I guess one big reason is that they do a slice on the data, and then after that slice, it doesn't matter how big of a value you have. If you had five data points, and one of their features looked like [math] \{ 1, 2, 3, 4, 1000000\} [/math], you might choose a split point at x = 2.5. At that point, 3,4, and a million all go into the same bucket, and their values are treated the same way. You could replace one million with something orders of magnitude bigger and it wouldn't matter, or you could change its value to 5 and it wouldn't matter. This restricts how much influence the outlying point can have. Contrast with linear regression, where the bigger that point gets, the more influence it will have on the entire model.

I don't know if this is a common terminology, but the tree based methods are kind of like [math]L^0 [/math] flavored, which is the most robust you can get.

The best approach is to take a course on data cleansing.

Instead of asking people, most who have no clue, you’ll actually learn how to handle them.

I know, some novel shit I tell you.

Here’s some insight for you.

Here’s the graph of some popular loss functions:

Here, the blue curve is hinge loss, red curve is logistic loss, and green curve is least squares loss.

The x-axis corresponds to [math]y f(x)[/math], that is, the product of the true label and the predicted label. Ideally, we want these to be both +1 or both -1, so that when the product is 1, there is no penalty. As you deviate from 1, there are penalties. Two things to observe here w.r.t logistic loss and square loss:

• Square loss diverges to infinity much faster as [math]y f(x)[/math] goes below zero. This is the reason it is less robust to outliers compared to the logistic loss. As you can guess, hinge loss is even better. (More details here — Prasoon Goyal's answer to When does Logistic Regression perform poorly and Support Vector Machine (SVM) should be preferred?)
• Square loss penalizes points even if they are correctly classified. So if the true label [math]y[/math] is 1 and the prediction [math]f(x)[/math] is 2, you still pay a price (although this does not directly contribute to sensitivity to outliers).

(Image source: What are the impacts of choosing different loss functions in classification to approximate 0-1 loss)

Usually, supervised learning algorithm finds an estimate which minimizes the cost function. Linear regression uses square loss function and logistic regression uses inverse logistic loss function (cost function of logistic regression)

[math]yf(x) [/math]in the x-axis is nothing but product of actual label (y)and predicted label[math](f(x))[/math].

For example: Decision boundary by linear regression (Square loss function)

Due to some of the outlier observations in the second graph, linear regression gives a decision boundary which classifies the labels poorly. In order to reduce the square loss, it chooses an estimate at the cost of predicting labels incorrectly. In an other hand logistic cost function doesn’t penalise the estimate for outlier observation.

Overfitting happens when a model trains too long, failing to generalize to new data. Also, when the model becomes complex, it can learn from ‘noise’ in data and overfit, leading to poor performance.

My take on the question with that background: Models robust to outliers can be affected by outliers in the dataset when they get too complex.

For example, a decision tree is (robust to outliers) but can get to split outliers if it can't split any data points and overfit. It’s recommended to prune it to lessen any outlier effect.

Always test approaches to find out what gives you the best results, and I’d suggest you use the deepchecks since it offers a faster way to validate your data and models.

[Segment performance code sample]

from deepchecks.tabular.checks.performance import SegmentPerformance 
SegmentPerformance(feature_1='workclass', feature_2='hours-per-week').run(validation_ds, model) 

I hope this helps.

Outlier detection is a crucial step in data analysis and machine learning to identify observations that deviate significantly from the rest of the data. Once outliers are detected, several techniques can be employed for their treatment. Here's an overview:

Outlier Detection Techniques:

1. Z-Score or Standard Score: Detection Method: Calculate the Z-score for each data point and identify those with a Z-score beyond a certain threshold. Treatment: Remove or adjust data points with high Z-scores.
2. Interquartile Range (IQR): Detection Method: Define a range based on the interquartile range and identify outliers outside this range. Treatment: Trim or winsorize (capping extreme values) the outliers.
3. Density-Based Methods (e.g., DBSCAN):Detection Method: Identify regions with lower data point density as potential outliers. Treatment: Remove or adjust data points in low-density regions.
4. Isolation Forest: Detection Method: Construct trees isolating instances, and outliers are expected to have shorter paths. Treatment: Remove instances isolated in the forest.
5. Local Outlier Factor (LOF):Detection Method: Compares the local density of a data point with that of its neighbors. Treatment: Adjust or remove points with significantly lower density.

Outlier Treatment Techniques:

1. Removal: Approach: Simply exclude the outlier from the dataset. Consideration: May lead to data loss and impact statistical analysis.
2. Transformation: Approach: Apply mathematical transformations (e.g., log transformations) to reduce the impact of outliers. Consideration: The transformed data may better adhere to assumptions of statistical methods.
3. Imputation: Approach: Replace outliers with estimated values based on the rest of the data. Consideration: Imputation methods should be chosen carefully to maintain data integrity.
4. Capping/Clipping: Approach: Set a threshold beyond which values are capped or clipped. Consideration: Reduces the impact of extreme values without complete removal.
5. Winsorizing: Approach: Replace extreme values with values at a specified percentile. Consideration: A less harsh alternative to trimming.
6. Model-Based Correction: Approach: Use statistical or machine learning models to predict values for outliers. Consideration: Requires a model that is not influenced by outliers.
7. Separate Analysis: Approach: Analyze the data both with and without outliers to understand their impact. Consideration: Provides insights into the sensitivity of results to outliers.

The choice of outlier detection and treatment techniques depends on the characteristics of the data and the goals of the analysis. It's often valuable to explore multiple methods and assess their impact on the overall analysis.

Q: How do outliers affect regression?

It depends on the type of regression. In the most widely taught form, you are minimizing the sum of squared errors, but this is equivalent to minimizing the average squared error, and like all averages, a single huge value can dominate the calculation. So the regression can be overwhelmed by one or two outliers.

There is an alternative measure of central tendency, the median, that is immune to outliers. It makes no difference how small the smallest value is; or how large the largest value, the median is just the middle value when everything is sorted.

Rousseeuw introduced Least Median of Squares Regression in 1984; you can read his paper here: http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/LeastMedianOfSquares.pdf

It takes longer, but it also can find the line that fits the majority of points, and excludes outliers.

In general this is part of what is called “robust” statistics, a technical term meaning statistics with some immunity to outliers. It is quite useful if you analyze data that is often subject to error, or occasionally has natural real outliers. For example, if you are in economics and trying to relate some behaviors with income levels, sometimes people have extreme income that you don’t necessarily want to overwhelm your data.

The answer to this question lies in the functionality of least square method.

The trouble with outliers in the least-squares method is that the Least-squares method only knows data in terms of their mean and their squared differences from the mean. Outliers will distort (either amplify or radically diminish) means in the first place. Then, in the second place, squaring these differences will only accentuate the distortion.
So presence of outliers will make a huge impact on LS method.

Now, for logistic regression, decision boundary takes into consideration only the points that are closer to it, hence effect of outliers on the decision boundary is very less. This doesn’t mean a outlier can not put a significant effect on boundary.

Hope this helps.

Outliers are unusual values that one can see in a dataset. They can be caused by measurement or execution errors. Usually when a data point significantly varies from the other data point in a dataset then it is called as an outlier. Whereas determining an outlier might altogether be subjective based on the extent of understanding the collected information.

E.g., One can consider different age groups. Here I have taken ages between 0-11, 18-28, 65-74, or maybe 90-100.

But an age of a person will never be called 298. 298. As it is not valid. But a proper number would be like 32.

This is an example but there are several more examples that one can choose to. It is called a univariate which means only one variable i.e. age is considered here.

Now let’s dive deeper and understand the multiple variables or multivariate data.

Detecting outliers and anomalies is very important as they are very useful while applying statistical techniques or for training the Machine Learning (ML) algorithms.

There are quite a few ways or techniques to detect and remove outliers. Below I have mentioned a few. Hope it will be useful for you.

Ways for outlier detection and removal

The analysis for outlier detection is referred to as outlier mining. There are many ways to detect outliers, and the removal process is the data frame. For removing the outlier, one must follow the same process of removing an entry from the dataset with the help of its exact position in the dataset because in all the below mentioned techniques of detecting the outliers end result is the list of all those data items that satisfy the outlier definition according to the method one chooses to use.

1) Python library – NumPy

If you’re a Python user, you’d probably be comfortable using this method. It could be a piece of cake for outlier detection. Here’s an example:

Image source: GitHub

This could be the code you’ll be installing NumPy and running it on a dataset, provided the dataset is already imported in Python. The result that returns above will be your outliers.

2) Interquartile Range (IQR)

Interquartile Range (IQR) method is mainly to measure variability. This is done by dividing data into quartiles. These data are split into four equal parts and in ascending order. Q1, Q2, Q3 are also called the first, second, and third quartiles are values that separate the four equal parts.

Now IQR is the range that falls between the first and the third quartile i.e. Q1 and Q3.

So, IQR=Q3-Q1.

Now any data point that falls above Q3 + 1.5 and below Q1 + 1.5 IQR are detected as outliers.

3) Numeric Outlier

Numeric Outlier detects and treats the outliers for each of the selected columns individually by means of the interquartile range (IQR).

To detect the outliers for a given column, the first and third quartile (Q1, Q3) is computed. An observation is flagged an outlier if it lies outside the range R = [Q1 - k(IQR), Q3 + k(IQR)] with IQR = Q3 - Q1 and k >= 0. Setting k = 1.5 the smallest value in R corresponds, mainly, to the lower end of a boxplot's whisker and the largest value to its upper end.

To offer grouping information this allows for detecting outliers only within their respective groups. If an observation is flagged as an outlier, one can either replace it with some other value or remove/retain the corresponding row. Whereas, the missing values contained in the data will be ignored, i.e., they will neither be used for the outlier computation nor will they be flagged as an outlier.

4) Z-Score treatment

Z- Score is also called a standard score. It is the signed number of standard deviations by which the value of an observation or data point is above the mean value of what is being observed or measured.

This value/score helps to understand how far is the data point from the mean. And after setting up a threshold value one can utilize z-score values of data points to define the outliers.

Zscore = (data_point -mean) / std. deviation

The intuition behind Z-score is to describe any data point by finding its relationship with the Standard Deviation and Mean of the group of data points. Z-score is finding the distribution of data where the mean is 0 and the standard deviation is 1 i.e. normal distribution.

While calculating the Z-score we re-scale and center the data and look for data points that are too far from zero. These data points which are way too far from zero will be treated as outliers. In most of the cases, a threshold of 3 or -3 is used i.e., if the Z-score value is greater than or less than 3 or -3 respectively, that data point will be identified as an outlier. One can also use the Z-score function defined in the SciPy library to detect the outliers.

According to my observations, the easiest way to remove outliers is by deleting observations as such when the value entered is due to a data entry error or data processing error or when the value observations are small in numbers. Other methods include imputing and transforming and binning values.

I hope this information will help you. All the best!

Outliers are anomalous values in the data. Outliers tend to increase the estimate of sample variance, thus decreasing the calculated F statistic for the ANOVA and lowering the chance of rejecting the null hypothesis. Run ANOVA on the entire data. Remove outlier(s) and rerun the ANOVA. If the results are the same then you can report the analysis on the full data and report that the outliers did not influence the results. This study finds evidence that the estimates in ANOVA are sensitive to outliers, i.e. that the procedure is not robust. Samples with a larger portion of extreme outliers have a higher type-I error probability than the expected level. An outlier is an observation that lies an abnormal distance from other values in a random sample from a population. There is, of course, a degree of ambiguity. Qualifying a data point as an anomaly leaves it up to the analyst or model to determine what is abnormal—and what to do with such data points. An ANOVA is quite robust against violations of the normality assumption, which means the Type 1 error rate remains close to the alpha level specified in the test. Violations of the homogeneity of variances assumption can be more impactful, especially when sample sizes are unequal between conditions.

Tree based methods divide the predictor space, that is, the set of possible values for X1, X2,… Xp ,into J distinct and non-overlapping regions, R1, R2….. RJ. In theory, the regions could have any shape. However, we choose to divide the predictor space into high-dimensional rectangles, or boxes, for simplicity and for ease of interpretation of the resulting predictive model

The goal is to find boxes R1,R2, ….. RJ that minimize the Residual sum of Squares (RSS) , given by

Unfortunately, it is computationally infeasible to consider every possible partition of the feature space into J boxes. For this reason, we take a top-down, greedy approach that is known as recursive binary splitting. The approach is top-down because it begins at the top of the tree and then successively splits the predictor space; each split is indicated via two new branches further down on the tree.

It is greedy because at each step of the tree-building process, the best split is made at that particular step, rather than looking ahead and picking a split that will lead to a better tree in some future step.

We first select the predictor Xj and the cutpoint s such that splitting the predictor space into the regions {X|Xj < s } leads to the greatest possible reduction in RSS.

Next, we repeat the process, looking for the best predictor and best cutpoint in order to split the data further so as to minimize the RSS within each of the resulting regions.

However, this time, instead of splitting the entire predictor space, we split one of the two previously identified regions. We now have three regions. Again, we look to split one of these three regions further,so as to minimize the RSS. The process continues until a stopping criterion is reached; for instance, we may continue until no region contains more than five observations.

Example :-

Since, extreme values or outliers, never cause much reduction in RSS, they are never involved in split.

Hence, tree based methods are insensitive to outliers.

The essential problem with outliers is their “undue” (whatever that means), influence on summary statistics you are calculating.

One approach to dealing with this is to calculate your summary statistic and eliminate a fixed percentage of the most extreme data-points as judged by the statistic and then recalculate the statistic. If the same extreme datapoints would be excluded using the new version of the statistic, you’re done. If not, modify the excluded data points and repeat.

Here are two articles I wrote years ago doing this for correlation analysis:

If you are not familiar with covariance ellipses, read this:

I am not advocating this because a datapoint is either wrong - mis-measured, instrument failure, etc., or it is real, so why are you excluding it?

The best way isn’t taught much in university courses. Namely to “teach” what we already know, before we try to “learn” from our data. We need hybrid methods.

Hundreds of examples could be offered here. Two should make the point:

• In a live Big Data demonstration of predictions of lung diseases, clustering and correlation showed cancer caused people to smoke. The speaker calmly swapped the cause/effect arrow because “we all know” that’s right. But he let the data drive all the other causal indicators. Of course, what was needed was some expert advice on any mistakes or missing realities.
• A retailer’s data science team, looking at seasonal stocking patterns, saw a big demand for plywood in a short seasonal window, in one particular city, and recommended positioning a large stock for the next season’s sales. The inventory managers patiently explained the city in question had been hit by hurricanes on the same week two years in a row. The odds of it repeating were very low. What the retailer did (already) was to stock plywood in regional distribution centers for hurricane season, and move the inventory into the predicted path of named storms.

So the lessons are:

• Ask the experts what rules, relationships and limitations are known. Build those in, or at least check for them. This will save you the embarrassment of thinking you have noticed faster than speed of light travel times, or other silly things.
• When a pattern or prediction seems to appear, ask an expert why you are seeing it BEFORE you offer formal recommendations. This will save you the experience of helping the experts “prove” your analysis was a waste of time and money.

in the real world we never have data as clean as a Kagle competition or your homework assignment, and there are always factors at play which aren’t fully explained by your data set. Never and always are strong words but they seem to be true in this case.

Mean value of data changes drastically with the presence of outliers.

As Standard Deviation is calculated using mean value,

Hence, Standard Deviation is also affected by Outliers very badly.

The effects of outliers in a dataset can result in longer training times, skewed results, and ultimately less accurate models.

Overall, outliers cause bias, reduce statistical test power, and negatively influence predictions.

Some models, such as regression-based models, are sensitive to outliers, while some are robust to outliers, such as tree-based models. Robust models usually handle outliers automatically, but overfitting can occur when the tree fits all the training data set samples. For decision trees, pruning helps overcome overfitting.

Usually, there’s never one way to handle outliers in a dataset. Seasoned data scientists and ML engineers study the dataset and understand features during data preparation before building models. If outliers are likely to affect your training, it's best to remove them. Sometimes, you can ignore them if your model selection is robust to outliers.

The best way is to identify/test your data for potential outliers and move forward on how to handle them. I use deepchecks outlier detection code/check.

import pandas as pd 
from sklearn.datasets import load_iris 
 
from deepchecks.tabular import Dataset 
from deepchecks.tabular.checks.integrity.outlier_sample_detection import OutlierSampleDetection 

I hope this helps.

I have thrown out the outliers when doing regressions on rare occasions. I have also used binaries in MLS to drop the outliers. Again very rarely. Another option is to use median in place of mean for normalizing the distribution in probability analysis.

It is generally not a good idea to screw around with the data. Strange events occur naturally. “Bad Data” some times occurs such as an instrument failure or data transcription error. Those are the ones to put in jail with a footnote.
The real trick is to understand WHY the values are outliers. If you can’t write an explanation in a footnote, don’t ignore the data point. It is also a good idea to quantify just how much the outliers are affecting your analysis.

In doing electric peak load forecasting in Florida I ran across one winter out of 20 that was especially cold. I agonized over that data point every year. The data was accurate, but that one winter had enough impact on the forecast to call for new generation five years earlier than without the outlier.

MISSING VALUES:

There is no golden solution for dealing with missing values. It all depends on the situation. I ll mention few.

1. If if a feature has too many missing values then then drop the whole feature.
2. If the feature is too important to drop then introduce another binary feature as isnull of this feature and impute the null values of the existing feature with median/mean.
3. If there are very few missing values in a feature and removing those rows doesn't hurt the sample size then remove the rows.
4. If removing rows with missing values in either of the feature reduce the sample size drastically then go for imputation. There are multiple ways for that.

• impute with mean/median of the column.
• mean/median of that column for N nearest neighbors
• If if it's a time series data set then use Marcov Chain to predict the missing values
• If each row is a time series and your algorithm doesn't demand the rows to be of same size then leave it as is. One example would be dynamic time warping distance between time series.

There are many other ways to deal with this. I wrote a few from my tiny experience if data pre-processing. Come up with your creative solution once you ate stuck with missing values.

OUTLIERS:

Straight way drop those; I hate those!!! Now it depends on you u how you decide whether a data point is outlier or not. The most common way is to calculate the standard deviation. Never try to impute the outliers with some other values. That will harm your analysis. Also, recalculate the statistics after you remove the outliers.

If you’re dealing with observations that should be independent and identically distributed, then there are three good general methods:

• One-class support vector machines
• Isolation forests
• Local outlier factors

In every case, the method assigns a score to each observation measuring how anomalous it is, and it’s up to the user to decide which points are outliers. That’s not usually a completely trivial problem.

If your data has time series structure, then the problem gets more complicated. You have to worry about trends and seasonality and nonstationarity in general. There are a few methods out there, but I’m not aware of any small number of methods that generally work well.

If your data has a spatial dependence structure, you’re on your own here.

This question seems to be broad. It depends on what problem you are trying to solve and the current state of your model. Something like transfer learning could give you a high accuracy only by training on 100 examples, and the improvement would not be significant if you fine-tune it on million examples. Also, you need to understand if your model seems to get stuck in local minima, changing hyper-parameter and tuning them with grid search can be one of the ways you can improve the performance of your predictive model. Lastly, depending on the problem you are trying to solve accuracy need not be the only performance metric you track to analyze the model performance.