Top 21 Scikit-learn Interview Questions (SOLVED with CODE) ML Devs Must Know

Data leakage occurs when information that would not be available at prediction time is used when building the model. This results in overly optimistic performance estimates. To avoid this problem we should:

• Always split the data into train and test subsets first, particularly before any preprocessing steps.

• Never include test data when using the fit and fit_transform methods. Conversely, the transform method should be used on both train and test subsets as the same preprocessing should be applied to all the data. This can be achieved by using fit_transform on the train subset and transform on the test subset.

• Using scikit-learn pipeline to ensure that the appropriate method is performed on the correct data subset.

A machine learning pipeline should contain a sequence of steps involved in training a machine learning model. The pipeline can involve pre-processing, feature selection, classification/regression, and post-processing.

For example, a basic pipeline for a classification model contains:

1. Scaler: For pre-processing data, i.e., transform the data to zero mean and unit variance using the StandardScaler().
2. Feature selector: Use VarianceThreshold() for discarding features whose variance is less than a certain defined threshold.
3. Model: For example KNeighborsClassifier(), which implements the k-nearest neighbor classifier and selects the class of the majority k points, which are closest to the test example.

...
pipe = Pipeline([
 ('scaler', StandardScaler()),
 ('selector', VarianceThreshold()),
 ('classifier', KNeighborsClassifier())
])

It says, scale first, select features second and classify in the end. The last step is call fit() method of the pipe object on our training data and get the training and test scores.

pipe.fit(X_train, y_train)

They effectively try to achieve the same result but the methodology used by each technique varies a little:

• RFE removes least significant features over iterations. So basically it first removes a few features which are not important and then fits and removes again and fits. It repeats this iteration until it reaches a suitable number of features.

• SelectFromModel is a little less robust as it just removes less important features based on a threshold given as a parameter. There is no iteration involved.

• With Normalize each sample (i.e. each row of the data matrix) with at least one non-zero component is rescaled independently of other samples so that its norm (l1 or l2) equals one.

• On another hand, StandardScaler standardizes features by removing the mean and scaling to unit variance.

• In other words Normalizer acts row-wise and StandardScaler column-wise. Normalizer does not remove the mean and scale by deviation but scales the whole row to unit norm.

• Now, regarding the question when to use each one, whether input variables require scaling depends on the specifics of your problem and of each variable:

  • You may have a sequence of quantities as inputs, such as prices or temperatures. If the distribution of the quantity is normal, then it should be standardized, otherwise, the data should be normalized. This applies if the range of quantity values is large (10s, 100s, etc.) or small (0.01, 0.0001). If the quantity values are small (near 0-1) and the distribution is limited (e.g. standard deviation near 1), then perhaps you can get away with no scaling of the data.

  • Predictive modelling problems can be complex, and it may not be clear how to best scale input data. The way to proceed is to explore modeling with the raw data, standardized data, and normalized data and see if there is a beneficial difference in the performance of the resulting model.

Given an initial model, as soon you call model.fit(features_train, label_train) the model starts training using the features and labels that you have passed. If you will again call model.fit(features_train2, label_train2) it will start training again using passed data and will remove the previous results. The model will reset the following inside model:

• Weights
• Fitted Coefficients
• Bias, etc.

To overcome this issue we can use partial_fit() method as well if we want our previous calculated stuff to stay and additionally train using the next data.

The usual way to compute the feature importance values of a single tree is as follows:

1. You initialize an array feature_importances of all zeros with size n_features.

2. You traverse the tree: for each internal node that splits on feature i you compute the error reduction of that node multiplied by the number of samples that was routed to the node and add this quantity to feature_importances[i].

The error reduction depends on the impurity criterion that you use (in Scikit-learn the criterion available are gini and entropy). Its the impurity of the set of examples that gets routed to the internal node minus the sum of the impurities of the two partitions created by the split.

Its important that these values are relative to a specific dataset (both error reduction and the number of samples are dataset specific) thus these values cannot be compared between different datasets.

• In PCA the principal components are represented by the eigenvectors which are stored in the attribute components_.
• The eigenvalues represent the variance in the direction of the eigenvector. We can get them through the attribute explained_variance_.

from sklearn.decomposition import PCA
import numpy as np

data = np.array([[2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3.0], [2.3, 2.7], [2, 1.6], [1, 1.1], [1.5, 1.6], [1.1, 0.9]])
pca = PCA()
pca.fit(data)

# eigenvectors
print(pca.components_)
# eigenvalues
print(pca.explained_variance_)

If the data contains many outliers, scaling using the mean and variance of the data is likely to not work very well. In these cases, we can use RobustScaler as a drop-in replacement instead.

RobustScaler removes the median and scales the data according to the quantile range (defaults to IQR: Interquartile Range). The IQR is the range between the 1st quartile (25th quantile) and the 3rd quartile (75th quantile).

Centering and scaling happen independently on each feature by computing the relevant statistics on the samples in the training set. Median and interquartile range are then stored to be used on later data using the transform method.

In Ridge Regression, the regularization parameter is typically represented as alpha in scikit-learn when calling RidgeRegression. One way to tune and choose the best alpha parameter is through cross-validation through the object RidgeCV, which stands for ridge cross-validation.

RigdeCV accepts an array of alpha values to try and then will train the model for all samples except one. It'll then evaluate the error in predicting this one test case. For example,

from sklearn.datasets import make_regression
from sklearn.linear_model import RidgeCV

reg_data, reg_target = make_regression(n_samples=100, n_features=2,
effective_rank=1, noise=10)
rcv = RidgeCV(alphas=np.array([.1, .2, .3, .4]))
rcv.fit(reg_data, reg_target)
rcv.fit(reg_data, reg_target)

After we fit the regression, the alpha attribute will be the best alpha choice:

>>> rcv.alpha_
0.10000000000000001

We can use LabelEncoder to encode a Pandas DataFrame of string or numerical labels. If the dataframe has many columns (50+, for example) creating a LabelEncoder for each feature is not efficient. In scikit-learn, the recommended way is to encode all the features is:

OneHotEncoder().fit_transform(df)

On other hand, if we are interested in applying OneHotEncoder only to certain columns then we can use ColumnTransformer.

• Though they have similar goals (i.e. placing some restrictions to the model so that it doesn't grow very complex and overfit), max_depth isn't equivalent to pruning.
• The way pruning usually works is that go back through the tree and replace branches that do not help with leaf nodes.
• However, scikit-learn does pruning based on minimal cost-complexity pruning: the subtree with the largest cost complexity that is smaller than the ccp_alpha parameter will be chosen.

There are three main centers of this computational cost; they are:

• Training machine learning models.
• Evaluating machine learning models.
• Hyperparameter tuning machine learning models.

Scikit-learn provides the capability of harnessing multiple cores of your computer, dramatically speeding up computationally expensive operations via the n_jobs argument, which specifies the number of cores to use for key machine learning tasks.

Common values are:

• n_jobs = None: Use a single core or the default configured by your backend library.
• n_jobs = 4: Use the specified number of cores, in this case 4.
• n_jobs = -1: Use all available cores.

However, this feature is only available for algorithms that support training in parallel. For example, to train and evaluate the RandomForestClassifier model using multiple cores we write:

from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import RandomForestClassifier
# define dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=3)
# define the model
model = RandomForestClassifier(n_estimators=100, n_jobs=8)
# define the evaluation procedure
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=4)
# evaluate the model
n_scores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=4)

Running the example evaluates the model using four cores, and each model is trained using four different cores. This will bring as a result an improvement over computational cost, but the performance of the model may be affected.

Furthermore, one way to perform multi-core hyperparameter tuning is:

from sklearn.datasets import make_classification
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
# define dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=3)
# define the model
model = RandomForestClassifier(n_estimators=100, n_jobs=4)
# define the evaluation procedure
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# define grid
grid = dict()
grid['max_features'] = [1, 2, 3, 4, 5]
# define grid search
search = GridSearchCV(model, grid, n_jobs=4, cv=cv)

However, when we perform hyperparameter tuning, it is probably better to make the search multi-core and leave the model training and evaluation single core. So as usual, we should iterate over different strategies to obtain the best performance of the model.

• cross_val_score calculates the score for each CV split and returns an array of scores of the estimator for each run of the cross validation.
• cross_validate it allows specifying multiple metrics for evaluation, i.e. calculate one or more scores and timings for each CV split and it returns dict containing training scores, fit-times and score-times in addition to the test score.

• Every Scikit-Learn's transform's fit() calculates the model parameters (e.g. μ and σ in case of StandardScaler) and saves them as an internal object's state.
• Afterwards, you can call its transform() method to apply the transformation to any particular set of examples.
• fit_transform() joins these two steps and is used for the initial fitting of parameters on the training set x, while also returning the transformed x'. Internally, the transformer object just calls first fit() and then transform() on the same data.

In practice, we need to have separate training and testing dataset and that is where having a separate fit() and transform() method helps. We apply fit() on the training dataset and use the transform() method on both -the training dataset and the test dataset-. Thus the training, as well as the test dataset, are then transformed (scaled) using the model parameters that were learned on applying the fit() method to the training dataset.

So for x training set, we do fit_transform because we need to compute model parameters, and then use it to autoscale the data. For x test set, well, we already have the model parameters, so we only do the transform part.

When considering OneHotEncoder (OHE) and LabelEncoder, we must try and understand what model you are trying to build. Namely, the two categories of the model we will be considering are:

1. Tree-Based Models: Gradient Boosted Decision Trees and Random Forests.
2. Non-Tree Based Models: Linear, kNN or Neural Network-based.

Let's consider when to apply OHE and when to apply Label Encoding while building tree-based models.

We apply OHE when:

• The categorical feature is not ordinal (dog,cat,mouse).
• The values that are close to each other in the label encoding correspond to target values that aren't close (non - linear data).

We apply Label encoding when:

• The categorical feature is ordinal (Jr. kg, Sr. kg, Primary school, high school, etc).
• We can come up with a label encoder that assigns close labels to similar categories: This leads to fewer splits in the tress hence reducing the execution time.
• The number of categorical features in the dataset is huge: One-hot encoding a categorical feature with huge number of values can lead to (1) high memory consumption and (2) the case when non-categorical features are rarely used by the model.

Let's consider now when to apply OHE and when to apply Label Encoding while building non tree-based models.

• To apply Label encoding, the dependence between feature and target must be linear in order for Label Encoding to be utilized effectively.

• Similarly, in case the dependence is non-linear, you might want to use OHE for the same.

• weights = 'distance' is in contrast to the default which is weights = 'uniform'.

• When weights are uniform, a simple majority vote of the nearest neighbors are used to assign cluster membership.

• When weights are distance weighted, the voting is proportional to the distance value. Nearby points will have a greater influence than more distance points (even if the counts of different groups are similar). Therefore distance weighting is very useful when we have sparse data.

The PCA object is very useful but has certain limitations for large datasets. The biggest limitation is that PCA only supports batch processing, which means all of the data to be processed must fit in the main memory.

A better alternative to use with large dataset is IncrementalPCA, this object uses a different form of processing and allows for partial computations which almost exactly match the results of PCA while processing the data in a minibatch fashion.

IncrementalPCA has the parameter batch_size to specify the number of samples to use for each batch and only stores estimates of component and noise variances, in order to update explained_variance_ratio_ incrementally. The memory usage depends on the number of samples per batch, rather than the number of samples to be processed in the dataset.

Therefore depending on the size of the input data, this algorithm can be much more memory efficient than a PCA, and allows sparse input.