27 SVM Interview Questions (ANSWERED) To Master Before ML & Data Science Interview

Support vector machines (SVMs) are a set of supervised learning methods used for classification, regression and outliers detection. The objective of the support vector machine algorithm is to find a hyperplane in an N-dimensional space(N — the number of features) that distinctly classifies the data points.

Support vector machines focus only on the points that are the most difficult to tell apart, whereas other classifiers pay attention to all of the points.

The intuition behind the support vector machine approach is that if a classifier is good at the most challenging comparisons (the points in B and A that are closest to each other), then the classifier will be even better at the easy comparisons (comparing points in B and A that are far away from each other).

• For a hard-margin SVM all of the support vectors lie exactly on the margin.
• Regardless of the number of dimensions or size of the data set, the number of support vectors could be as little as 2.

• Guaranteed Optimality: Owing to the nature of Convex Optimization, the solution will always be global minimum not a local minimum.
• Abundance of Implementations: We can access it conveniently, be it from Python or Matlab.
• SVM can be used for linearly separable as well as non-linearly separable data. Linearly separable data is the hard margin whereas non-linearly separable data poses a soft margin.
• SVMs provide compliance to the semi-supervised learning models. It can be used in areas where the data is labeled as well as unlabeled. It only requires a condition to the minimization problem which is known as the Transductive SVM.
• Feature Mapping used to be quite a load on the computational complexity of the overall training performance of the model. However, with the help of Kernel Trick, SVM can carry out the feature mapping using a simple dot product.

• Hard-Margin SVMs have linearly separable training data. No data points are allowed in the margin areas. This type of linear classification is known as Hard margin classification.
• Soft-Margin SVMs have training data that are not linearly separable. Margin violation means choosing a hyperplane, which can allow some data points to stay either in between the margin area or on the incorrect side of the hyperplane.

• Hard-Margin SVMs are quite sensitive to outliers.
• Soft-Margin SVMs try to find the best balance between keeping the margin as large as possible and limiting the margin violations.

• Support vectors are the data points nearest to the hyperplane, the points of a data set that, if removed, would alter the position of the dividing hyperplane.
• Using these support vectors, we maximize the margin of the classifier.
• For computing predictions, only the support vectors are used.

• Data is rarely as clean that hyperplane is a line that linearly separates and classifies a set of data. In order to classify a dataset, it’s necessary to move away from a 2D view of the data to a 3D view.
• This ‘lifting’ of the data points represents the mapping of data into a higher dimension. This is known as kernelling.

• In this example, the picture on the left shows our original data points. In 1-dimension, this data is not linearly separable, but after applying the transformation ϕ(x) = x² and adding this second dimension to our feature space, the classes become linearly separable.

When it comes to classification problems, the goal is to establish a decision boundary that maximizes the margin between the classes. However, in the real world, this task can become difficult when we have to treat with non-linearly separable data. One approach to solve this problem is to perform a data transformation process, in which we map all the data points to a higher dimension find the boundary and make the classification.

That sounds alright, however, when there are more and more dimensions, computations within that space become more and more expensive. In such cases, the kernel trick allows us to operate in the original feature space without computing the coordinates of the data in a higher-dimensional space and therefore offers a more efficient and less expensive way to transform data into higher dimensions.

There exist different kernel functions, such as:

• linear,
• nonlinear,
• polynomial,
• radial basis function (RBF), and
• sigmoid.

Each one of them can be suitable for a particular problem depending on the data.

• Linear SVM is considered parametric because it can only produce linear boundaries.

• kNN is a nonparametric algorithm because it avoids a priori assumptions about the shape of the class boundary and can thus adapt more closely to nonlinear boundaries as the amount of training data increases.

• kNN has higher variance than linear SVM but it has the advantage of producing classification fits that adapt to any boundary. Even though the true class boundary is unknown in most real-world applications, kNN has been shown to approach the theoretically optimal classification boundary as the training set increases to massive data.

• Because kNN does not impose any structure on the boundary, it can create class boundaries that may be less interpretable than those of linear SVM.

• For Logistic Regression, outliers can have an unusually large effect on the estimate of logistic regression coefficients. It will find a linear boundary if it exists to accommodate the outliers. To solve the problem of outliers, sometimes a sigmoid function is used in logistic regression.

• For SVM, outliers can make the decision boundary deviate severely from the optimal hyperplane. One way for SVM to get around the problem is to intrduce slack variables. There is a penalty involved with using slack variables, and how SVM handles outliers depends on how this penalty is imposed.

In its simplest form, In SVM we look for a decision boundary such as we can expand it as wide as possible to separate the classes.

Let's take, for example, a simple two-dimensional classification problem: we want to find a line such that points on one side are Class A and points on the other side are Class B. To solve this problem, we may try with different lines and evaluate each one:

Note that all these lines can separate the two classes (chimps and humans), but some lines are probably better classifier lines than others.

• The blue one is heavily dependent on height, so slightly taller chimps or slightly shorter humans may be classified incorrectly.
• The red line is horizontal and therefore entirely dependent on weight. Extra-heavy chimps (say 62 kg.) will be incorrectly classified as human.
• The green one seems to take both factors into account.

SVM will take each guess and try to widen it. The line that can be widened the most before it touches a data point is then considered the best classifier.

Out of our choices, the green one can expand the most without touching a data point. So if an SVM algorithm could only choose from these three lines, it would choose the green line from the first picture as the decision boundary.

For a N-dimensional problem the previous intuition holds.

• The polynomial kernel is a kernel function commonly used with SVMs and other kernelized models, that represents the similarity of vectors in a feature space over polynomials of the original variables, allowing learning of non-linear models.
• The polynomial kernel looks not only at the given features of input samples to determine their similarity, but also combinations of these. In regression analysis, these combinations of features are called interaction features.
• For degree d polynomials, the polynomial kernel is defined as:

where x and y are vectors in the input space.

• Polynomial kernels are used for Natural Language Processing

• Parametric: SVM and neural networks are both parametric but for different reasons.
  • For SVM the typical parameters are; soft-margin parameter (C), parameter of the kernel function (gamma).
  • Neural networks also have parameters but it is a lot more than SVM. Some NN parameters are the number of layers and their size, number of training epochs, and the learning rate.

• Embedding Non-Linearity: Both the methods can embed non-linear functions.

  • SVM does this through the usage of kernel method.
  • Neural Networks embed non-linearity using non-linear activation functions.

• Comparable Accuracy:

  • If both SVM and Neural Networks are trained in the same dataset, given the same training time, and the same computation power they have comparable accuracy.
  • If neural networks are given as much computation power and training time as possible then it outperforms SVMs.

When we talk about loss function, what we really mean is a training objective that we want to minimize.

• Hinge loss is a loss function used for training classifiers. Hinge loss is most notably used for soft-margin SVMs. The hinge loss penalizes the SVM model for inaccurate predictions (misclassifications).

• Hinge loss $l(y, \hat{y})$ is defined to be $\max(0, 1- y\hat{y})$ where $y \in \{1, -1\}$.

  We want to use $\hat{y}$ to estimate $y$.

Let's try to understand $y\hat{y}$.

• If $y=1$, we would want $\hat{y}$ to be as positive as possible, in particular, if $\hat{y} \ge 1$, we are happy and the hinge loss would be evaluated to zero. If $y<1$, we would want to penalize our prediction.

• On the other hand, if $y=-1$, we would want $\hat{y}$ to be as negative as possible, in particular, if $\hat{y} \le -1$, we are happy and the hinge loss would be evaluated to zero. If $y>-1$, we would want to penalize our prediction.

These two conditions can be combined compactly, if the model is doing well, we would want $y\hat{y} \ge 1$ and we want to penalize our model otherwise.

SVM is a model for classification. For regression, you use support vector regression (SVR). The idea of SVR is closely related to that of SVM:

• In SVM, you want to find a separating hyperplane such that all points are at a certain distance from this plane. If there are points that are too close to the separating hyperplane, there is a penalty [the hinge loss]:

• In SVR, you want to find a function such that all points are within a certain distance from this function. Unlike other Regression models that try to minimize the error between the real and predicted value, the SVR tries to fit the best line within a threshold value (if points are outside this distance [“ 𝜖 -tube”], there is a penalty):

SVM is not a good approach if:

• Data is hardly given as feature vectors. In reality, more often than not, data might be images, videos, speech, text/language, music, and so on. How do you engineer the features in order to feed an SVM?
• If your data is non-linear and you choose a linear kernel to model it.
• If kernel and its parameters are not properly estimated/chosen.
• The quadratic programming problem is computationally intensive and with the increase in training data, it can be very slow on normal machines.
• SVM is mainly developed for binary classification problems. For handling multi-class problems, either you perform a 1-against-all strategy or do other tricks like optimizing classes together
• Your data is imbalanced.
• SVM can only be feasibly trained with a few 10’s of thousands of samples at most
• There’s a “sweet spot” for an RBF SVM
  • 500-10k samples
  • 300–10k features
  • all features are expected to have similar information content / scale

The choice depends very much on what data you have and what is your purpose:

• Random forest is suited for multiclass problems.
• SVM is suited for two-class problems. If the problem is multiclass, then the problem needs to be reduced to multiple binary classification problems. Usually this consists of building binary classifiers which distinguish:
  • (i) between one of the labels and the rest (one-versus-all) or
  • (ii) between every pair of classes (one-versus-one).

• Random forest works well with both numerical or categorical features. Features need not be scaled either. It is generally okay to keep the dataset without processing it in random forest.
• SVM tries to maximize the distance between different points. It is up to the designer to define what the distance represents. As a consequence, one-hot-encoding, and scaling data is very important in SVM.

• For a classification problem Random Forest gives you the probability of belonging to the class.
• SVM gives you the distance to the boundary, you still need to convert it to probability somehow if you need probability.
• If you have data with 𝑛 n points and 𝑚 m features, an intermediate step in SVM is constructing an 𝑛×𝑛 matrix (think about memory requirements for storage) by calculating 𝑛² dot products (computational complexity). Therefore, as a rule of thumb, SVM is hardly scalable beyond 10⁵ points.

While designing an SVM classifier, the designer should select:

• The Regularization Weight - It helps to include the problematic points into the equation and weight them using the C-Parameter. This parameter is a tradeoff between maximizing the margin and minimizing the error.
• The Type of Kernel Function - The kernel functions return the inner product between two points in a suitable feature space. Thus by defining a notion of similarity, with a little computational cost even in very high-dimensional spaces. Kernel function examples are:
  • linear,
  • nonlinear,
  • polynomial,
  • Gaussian kernel,
  • Radial basis function (RBF),
  • sigmoid
• The Parameters of the Kernel Function - like Gamma for Gaussian Radial Basis function kernel.

These values should be selected by hand-tuning or by cross-validation.