How To Find Outliers In A Data Set

Outliers are data points that are far from other data points. In other words, they're unusual values in a dataset. Outliers are problematic for many statistical analyses because they can cause tests to either miss significant findings or misconstrue real results.

Unfortunately, at that place are no strict statistical rules for definitively identifying outliers. Finding outliers depends on discipline-area knowledge and an understanding of the data drove process. While in that location is no solid mathematical definition, at that place are guidelines and statistical tests you lot can employ to find outlier candidates.

In this post, I'll explain what outliers are and why they are problematic, and present various methods for finding them. Additionally, I close this mail by comparison the different techniques for identifying outliers and share my preferred approach.

Outliers and Their Touch

Outliers are a unproblematic concept—they are values that are notably unlike from other information points, and they can cause problems in statistical procedures.

To demonstrate how much a single outlier can touch the results, permit's examine the properties of an case dataset. It contains xv height measurements of human males. 1 of those values is an outlier. The table below shows the mean height and standard departure with and without the outlier.

Throughout this post, I'll exist using this example CSV dataset: Outliers.

With Outlier	Without Outlier	Difference
two.4m (7' 10.5")	1.8m (5' ten.8")	0.6m (~2 feet)
ii.3m (7' 6")	0.14m (five.v inches)	2.16m (~7 feet)

From the tabular array, it's easy to encounter how a single outlier can distort reality. A single value changes the mean height by 0.6m (two feet) and the standard divergence by a whopping 2.16m (7 feet)! Hypothesis tests that use the mean with the outlier are off the mark. And, the much larger standard divergence will severely reduce statistical power!

Before performing statistical analyses, y'all should identify potential outliers. That's the bailiwick of this post. In the adjacent post, we'll move on to figuring out what to practise with them.

There are a variety of ways to find outliers. All these methods employ dissimilar approaches for finding values that are unusual compared to the residual of the dataset. I'll showtime with visual assessments and then move onto more analytical assessments.

Permit'southward find that outlier! I've got 5 methods for you to try.

Sorting Your Datasheet to Find Outliers

Sorting your datasheet is a simple only effective way to highlight unusual values. Merely sort your data sheet for each variable and then expect for unusually high or low values.

For example, I've sorted the example dataset in ascending order, equally shown below. The highest value is conspicuously different than the others. While this approach doesn't quantify the outlier's degree of unusualness, I like it because, at a glance, y'all'll notice the unusually high or low values.

Graphing Your Information to Identify Outliers

Boxplots, histograms, and scatterplots can highlight outliers.

Boxplots display asterisks or other symbols on the graph to indicate explicitly when datasets comprise outliers. These graphs utilise the interquartile method with fences to find outliers, which I explain subsequently. The boxplot below displays our case dataset. It's clear that the outlier is quite unlike than the typical data value.

You can likewise use boxplots to find outliers when you take groups in your information. The boxplot below shows a different dataset that has an outlier in the Method 2 group. Click here to learn more about boxplots.

Histograms also emphasize the existence of outliers. Look for isolated confined, as shown below. Our outlier is the bar far to the right. The graph crams the legitimate information points on the far left.

Click here to acquire more about histograms.

Most of the outliers I discuss in this post are univariate outliers. Nosotros look at a data distribution for a single variable and detect values that fall outside the distribution. However, yous can use a scatterplot to find outliers in a multivariate setting.

In the graph below, nosotros're looking at two variables, Input and Output. The scatterplot with regression line shows how most of the points follow the fitted line for the model. Yet, the circled point does not fit the model well.

Interestingly, the Input value (~14) for this observation isn't unusual at all because the other Input values range from 10 through 20 on the Ten-axis. Also, notice how the Output value (~50) is similarly within the range of values on the Y-centrality (10 – threescore). Neither the Input nor the Output values themselves are unusual in this dataset. Instead, it's an outlier considering it doesn't fit the model.

This type of outlier tin be a problem in regression analysis. Given the multifaceted nature of multivariate regression, there are numerous types of outliers in that realm. In my ebook nearly regression analysis, I item diverse methods and tests for identifying outliers in a multivariate context.

For the remainder of this post, we'll focus on univariate outliers.

Using Z-scores to Detect Outliers

Z-scores tin quantify the unusualness of an ascertainment when your information follow the normal distribution. Z-scores are the number of standard deviations higher up and beneath the mean that each value falls. For example, a Z-score of two indicates that an observation is two standard deviations above the boilerplate while a Z-score of -2 signifies information technology is ii standard deviations below the mean. A Z-score of zero represents a value that equals the hateful.

To calculate the Z-score for an observation, accept the raw measurement, subtract the mean, and divide past the standard deviation. Mathematically, the formula for that process is the following:

The further away an observation's Z-score is from zero, the more unusual information technology is. A standard cut-off value for finding outliers are Z-scores of +/-3 or further from zero. The probability distribution below displays the distribution of Z-scores in a standard normal distribution. Z-scores beyond +/- 3 are and so extreme you can barely come across the shading under the curve.

In a population that follows the normal distribution, Z-score values more than extreme than +/- 3 take a probability of 0.0027 (two * 0.00135), which is about one in 370 observations. Nevertheless, if your information don't follow the normal distribution, this approach might non be accurate.

Z-scores and Our Example Dataset

In our example dataset below, I brandish the values in the example dataset along with the Z-scores. This approach identifies the same observation as being an outlier.

Note that Z-scores tin can exist misleading with small datasets because the maximum Z-score is limited to (north−i) / √ n.*

Indeed, our Z-score of ~three.6 is right near the maximum value for a sample size of 15. Sample sizes of 10 or fewer observations cannot accept Z-scores that exceed a cutoff value of +/-three.

Likewise, notation that the outlier's presence throws off the Z-scores because it inflates the mean and standard deviation as we saw earlier. Find how all the Z-scores are negative except the outlier'due south value. If we calculated Z-scores without the outlier, they'd exist different! Be aware that if your dataset contains outliers, Z-values are biased such that they announced to be less extreme (i.e., closer to zero).

For more information about z-scores, read my post, Z-score: Definition, Formula, and Uses.

The z-score cutoff value is based on the empirical rule. For more information, read my post, Empirical Dominion: Definition, Formula, and Uses.

Related posts: Normal Distribution and Agreement Probability Distributions

Using the Interquartile Range to Create Outlier Fences

You can use the interquartile range (IQR), several quartile values, and an adjustment cistron to calculate boundaries for what constitutes minor and major outliers. Minor and major denote the unusualness of the outlier relative to the overall distribution of values. Major outliers are more extreme. Analysts also refer to these categorizations as mild and extreme outliers.

The IQR is the center 50% of the dataset. It'southward the range of values betwixt the third quartile and the first quartile (Q3 – Q1). We can take the IQR, Q1, and Q3 values to calculate the following outlier fences for our dataset: lower outer, lower inner, upper inner, and upper outer. These fences determine whether data points are outliers and whether they are mild or farthermost.

Values that fall within the two inner fences are not outliers. Let's run into how this method works using our instance dataset.

Click here to learn more about interquartile ranges and percentiles.

Calculating the Outlier Fences Using the Interquartile Range

Using statistical software, I can determine the interquartile range along with the Q1 and Q3 values for our example dataset. We'll need these values to calculate the "fences" for identifying minor and major outliers. The output beneath indicates that our Q1 value is 1.714 and the Q3 value is 1.936. Our IQR is 1.936 – one.714 = 0.222.

To calculate the outlier fences, do the following:

Take your IQR and multiply it by 1.v and 3. We'll use these values to obtain the inner and outer fences. For our case, the IQR equals 0.222. Consequently, 0.222 * 1.5 = 0.333 and 0.222 * 3 = 0.666. Nosotros'll use 0.333 and 0.666 in the post-obit steps.
Calculate the inner and outer lower fences. Accept the Q1 value and subtract the two values from step 1. The 2 results are the lower inner and outer outlier fences. For our example, Q1 is 1.714. And so, the lower inner fence = 1.714 – 0.333 = 1.381 and the lower outer argue = 1.714 – 0.666 = 1.048.
Summate the inner and outer upper fences. Have the Q3 value and add together the 2 values from step 1. The 2 results are the upper inner and upper outlier fences. For our example, Q3 is 1.936. So, the upper inner argue = i.936 + 0.333 = 2.269 and the upper outer argue = 1.936 + 0.666 = 2.602.

Using the Outlier Fences with Our Example Dataset

For our example dataset, the values for these fences are 1.048, 1.381, two.269, and ii.602. Almost all of our data should fall between the inner fences, which are 1.381 and two.269. At this bespeak, nosotros look at our information values and decide whether any qualify as being major or minor outliers. 14 out of the 15 information points autumn inside the inner fences—they are non outliers. The fifteen^thursday information indicate falls outside the upper outer contend—it's a major or extreme outlier.

The IQR method is helpful because information technology uses percentiles, which do non depend on a specific distribution. Additionally, percentiles are relatively robust to the presence of outliers compared to the other quantitative methods.

Boxplots employ the IQR method to decide the inner fences. Typically, I'll use boxplots rather than computing the fences myself when I want to employ this arroyo. Of the quantitative approaches in this post, this is my preferred method. The interquartile range is robust to outliers, which is clearly a crucial holding when you're looking for outliers!

Related post: What are Robust Statistics?

Finding Outliers with Hypothesis Tests

You tin can utilise hypothesis tests to find outliers. Many outlier tests exist, but I'll focus on one to illustrate how they piece of work. In this post, I demonstrate Grubbs' test, which tests the following hypotheses:

Null: All values in the sample were drawn from a unmarried population that follows the same normal distribution.
Culling: One value in the sample was non drawn from the same normally distributed population as the other values.

If the p-value for this exam is less than your significance level, you can turn down the null and conclude that one of the values is an outlier. The analysis identifies the value in question.

Let's perform this hypothesis test using our sample dataset. Grubbs' test assumes your data are drawn from a usually distributed population, and it can notice only one outlier. If you suspect y'all have additional outliers, employ a different test.

Grubbs' outlier test produced a p-value of 0.000. Because it is less than our significance level, we can conclude that our dataset contains an outlier. The output indicates it is the high value we found earlier.

If you use Grubbs' examination and find an outlier, don't remove that outlier and perform the assay again. That process can crusade you to remove values that are not outliers.

Challenges of Using Outlier Hypothesis Tests: Masking and Swamping

When performing an outlier test, y'all either need to choose a procedure based on the number of outliers or specify the number of outliers for a exam. Grubbs' test checks for only one outlier. Still, other procedures, such as the Tietjen-Moore Examination, require you to specify the number of outliers. That's difficult to exercise correctly! Later all, yous're performing the test to find outliers! Masking and swamping are two problems that tin can occur when you specify the wrong number of outliers in a dataset.

Masking occurs when you specify too few outliers. The additional outliers that exist can affect the examination so that information technology detects no outliers. For example, if you specify i outlier when there are two, the examination can miss both outliers.

Conversely, swamping occurs when you specify too many outliers. In this instance, the test identifies too many data points every bit being outliers. For example, if you specify two outliers when there is only one, the examination might determine that there are two outliers.

Because of these issues, I'm not a large fan of outlier tests. More than on this in the next section!

My Philosophy nearly Finding Outliers

As y'all saw, at that place are many means to identify outliers. My philosophy is that you must use your in-depth knowledge virtually all the variables when analyzing data. Part of this knowledge is knowing what values are typical, unusual, and impossible.

I find that when you have this in-depth knowledge, it's best to use the more than straightforward, visual methods. At a glance, data points that are potential outliers volition pop out under your knowledgeable gaze. Consequently, I'll ofttimes use boxplots, histograms, and good old-fashioned data sorting! These simple tools provide enough information for me to find unusual information points for farther investigation.

Typically, I don't use Z-scores and hypothesis tests to find outliers considering of their various complications. Using outlier tests tin can be challenging because they usually assume your data follow the normal distribution, and then in that location's masking and swamping. Additionally, the existence of outliers makes Z-scores less extreme. It'due south ironic, only these methods for identifying outliers are actually sensitive to the presence of outliers! Fortunately, as long every bit researchers use a simple method to display unusual values, a knowledgeable analyst is probable to know which values demand further investigation.

In my view, the more formal statistical tests and calculations are overkill because they can't definitively identify outliers. Ultimately, analysts must investigate unusual values and use their expertise to decide whether they are legitimate data points. Statistical procedures don't know the bailiwick matter or the data drove process and can't brand the final determination. You lot should not include or exclude an observation based entirely on the results of a hypothesis examination or statistical measure.

At this stage of the analysis, we're only identifying potential outliers for farther investigation. Information technology's merely the offset footstep in handling them. If we err, we want to err on the side of investigating too many values rather than too few.

In my next post, I'll explain what you're looking for when investigating outliers and how that helps yous determine whether to remove them from your dataset. Not all outliers are bad and some should not exist deleted. In fact, outliers tin can be very informative almost the subject-expanse and data drove process. Information technology'south of import to understand how outliers occur and whether they might happen once again as a normal part of the process or study area.

Read my Guidelines for Removing and Handling Outliers.

If you're learning nearly hypothesis testing and similar the approach I utilise in my blog, cheque out my eBook!