21 Anomaly Detection Interview Questions For Data Analysts & ML Engineers

Anomaly detection (or outlier detection) is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data.

• Since anomalies are rare events, making it very difficult to label them with high accuracy, swamping is the phenomenon of labeling normal events as anomalies.
• When clustering algorithms are used, the data points belonging to different clusters get merged into one cluster, if the number of segments in the dataset is not known, this causes the outlier cluster to be merged to a cluster with normal data points. This causes the outliers to not be detected. This is defined as masking.

Uniform Distribution

• A uniform distribution looks like the following:

• When data is distributed uniformly over a finite range, the mean and standard deviation merely characterize the range of values.
• One possible indication of anomalous behavior could be that a small neighborhood contains substantially fewer or more data points than expected from a uniform distribution.

Normal Distribution

• A normal distribution looks like the following:

• A normal distribution follows the empirical rule, which states that 68%, 95%, and 99.7% of the values lie within one, two, and three standard deviations of the mean, respectively.
• About 0.1% of the points are more than $3 \sigma$ (three standard deviations) away from the mean, and only about $5 \times 10^{-8}$% of the points are more than six standard deviations away from the mean.
• Hence, a threshold (such as $3 \times \sigma$) is chosen, and points beyond that distance from the mean are declared to be anomalous.

Normalization rescales the values into a range of [0,1]. This might be useful in some cases where all parameters need to have the same positive scale. However, the outliers from the data set are lost.

Standardization rescales data to have a mean ($\mu$) of 0 and standard deviation ($\sigma$) of 1 (unit variance).

For most applications standardization is recommended.

• The goal of anomaly detection is to identify cases that are unusual within data that is seemingly comparable hence anomaly detection can be used effectively as a tool for risk mitigation and fraud detection.
• When preparing datasets for machine learning models, it is really important to detect all the outliers and either get rid of them or analyze them to know why you had them there in the first place.

• Using k-Means for spotting outliers is not good in a multivariate dataset because it is not built for that purpose. It will always give a solution that minimizes the total within-cluster sum of squares, and the outliers will not necessarily define their own cluster.
• The figure below shows that the outlying value is never recovered as such, it always belongs to one of the clusters.

• For Logistic Regression, outliers can have an unusually large effect on the estimate of logistic regression coefficients. It will find a linear boundary if it exists to accommodate the outliers. To solve the problem of outliers, sometimes a sigmoid function is used in logistic regression.

• For SVM, outliers can make the decision boundary deviate severely from the optimal hyperplane. One way for SVM to get around the problem is to intrduce slack variables. There is a penalty involved with using slack variables, and how SVM handles outliers depends on how this penalty is imposed.

In statistics, if a data distribution is approximately normal then about 68% of the data values lie within one standard deviation of the mean and about 95% are within two standard deviations, and about 99.7% lie within three standard deviations:

Therefore, if you have any data point that is more than 3 times the standard deviation, then those points are very likely to be anomalous or outliers.

• The training data contains outliers which are defined as observations that are far from the others. Outlier detection estimators thus try to fit the regions where the training data is the most concentrated, ignoring the deviant observations.
• The training data is not polluted by outliers and we are interested in detecting whether a new observation is an outlier. In this context an outlier is also called a novelty.

Outlier detection and novelty detection are both used for anomaly detection, where one is interested in detecting abnormal or unusual observations. Outlier detection is then also known as unsupervised anomaly detection and novelty detection as semi-supervised anomaly detection.

In the context of outlier detection, the outliers/anomalies cannot form a dense cluster as available estimators assume that the outliers/anomalies are located in low density regions. On the contrary, in the context of novelty detection, novelties/anomalies can form a dense cluster as long as they are in a low density region of the training data, considered as normal in this context.

Naturally, SVM is used in solving multi-class classification problems.

However, SVM is also increasingly being used in one class problem, where all data belong to a single class. In this case, the algorithm is trained to learn what is “normal”, so that when a new data is shown the algorithm can identify whether it should belong to the group or not. If not, the new data is labeled as out of ordinary or anomaly.

When modeling one class, the algorithm captures the density of the majority class and classifies examples on the extremes of the density function as outliers. This modification of SVM is referred to as One-Class SVM.

Anomalies can be broadly categorized as:

1. Point (global) anomalies: A single instance of data is anomalous if it's too far off from the rest.

   Business use case: Detecting credit card fraud based on "amount spent."

2. Contextual (conditional) anomalies: The abnormality is context specific. This type of anomaly is common in time-series data.

   Business use case: Spending $100 on food every day during the holiday season is normal, but may be odd otherwise.

3. Collective anomalies: When a subset of data points within a set is anomalous to the entire dataset, those values are called collective outliers. In this category, individual values aren’t anomalous globally or contextually. You start to see these types of outliers when examining distinct time series together. Individual behavior may not deviate from the normal range in a specific time series dataset. But when combined with another time series dataset, more significant anomalies become clear.

   Business use case: Someone is trying to copy data form a remote machine to a local host unexpectedly, an anomaly that would be flagged as a potential cyber attack.

There are three categories of outlier detection, namely, supervised, semi-supervised, and unsupervised:

• Supervised: Requires fully labeled training and testing datasets. An ordinary classifier is trained first and applied afterward.
• Semi-supervised: Uses training and test datasets, whereas training data only consists of normal data without any outliers. A model of the normal class is learned and outliers can be detected afterward by deviating from that model.
• Unsupervised: Does not require any labels; there is no distinction between a training and a test dataset Data is scored solely based on intrinsic properties of the dataset.

And three fundamental approaches to detect anomalies are based on:

• By Density - Normal points occur in dense regions, while anomalies occur in sparse regions
• By Distance - Normal point is close to its neighbors and anomaly is far from its neighbors
• By Isolation - The term isolation means ‘separating an instance from the rest of the instances’. Since anomalies are ‘few and different’ and therefore they are more susceptible to isolation.

• Out of distribution (OOD) data refers to data that was collected at a different time, and possibly under different conditions or in a different environment than the data collected to create the model. It can be said that the data is from a different distribution.
• After the out of distribution data is collected, the model can perform either Novelty detection or Anomaly detection.
• Novelty data is the data that is in-distribution. Novelty detection checks whether the new data is in-distribution or not.
• Anomaly detection is used to test the data to see if it is different than what the model was trained on.