# Outliers

** Published:**

This post covers detecting outliers.

# Outliers

https://towardsdatascience.com/5-ways-to-detect-outliers-that-every-data-scientist-should-know-python-code-70a54335a623

https://medium.datadriveninvestor.com/finding-outliers-in-dataset-using-python-efc3fce6ce32

```
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats
```

## Data

```
# multiply and add by random numbers to get some real values
data = np.random.randn(50000) * 20 + 20
data
```

## Method 1 - Scatter Plot

```
plt.scatter(x=range(len(data)), y=data);
```

## Method 2 — Standard Deviation

If a data distribution is approximately normal then about 68% of the data values lie within one standard deviation of the mean and about 95% are within two standard deviations, and about 99.7% lie within three standard deviations

https://miro.medium.com/max/1400/1*rV7rq7F_uB5gwjzzGJ9VqA.png

```
def outliers_SD(data):
#define a list to accumlate anomalies
anomalies = []
# Set upper and lower limit to 3 standard deviation
data_std = np.std(data)
data_mean = np.mean(data)
lower_limit = data_mean - data_std * 3
upper_limit = data_mean + data_std * 3
# Generate outliers
for outlier in data:
if outlier > upper_limit or outlier < lower_limit:
anomalies.append(outlier)
return anomalies
outliers = outliers_SD(data)
plt.scatter(range(len(outliers)), outliers);
```

## Method 3 — Boxplots

https://miro.medium.com/max/1280/1*AU07MCIdvUnjskY1XH9auw.png |

https://miro.medium.com/max/1400/1*J5Xm0X-phCJJ-DKZMZ_88w.png |

```
sns.boxplot(data=data);
```

## Method 4 - Using Z score

Z-score is finding the distribution of data where mean is 0 and standard deviation is 1 i.e. normal distribution.

Re-scale and center the data and look for data points which are too far from zero. These data points which are way too far from zero will be treated as the outliers.

In most of the cases a threshold of 3 or -3 is used i.e if the Z-score value is greater than or less than 3 or -3 respectively, that data point will be identified as outliers.

$ Z~score = \frac{(Observation — Mean)}{Standard Deviation} $

$ z = \frac{X - \mu}{\sigma} $

```
def outliers_zscore(data, threshold=3):
outliers=[]
mean = np.mean(data)
std = np.std(data)
for x in data:
z = (x - mean)/std
if np.abs(z) > threshold:
outliers.append(x)
return outliers
outliers = outliers_zscore(data)
plt.scatter(range(len(outliers)), outliers);
```

```
def outliers_zscore(data, threshold=3):
z = stats.zscore(data)
z = np.abs(z)
outliers_idx = np.where(z > 3)
outliers = data[outliers_idx]
return outliers
outliers = outliers_zscore(data)
plt.scatter(range(len(outliers)), outliers);
```