17 Principal Component Analysis (PCA) Interview Questions (ANSWERED)

Feature selection refers to choosing a subset of the features from the complete set of features.

In PCA, we obtain Principal Components axis, this is a linear combination of all the original set of feature variables which defines a new set of axes that explain most of the variations in the data.

Therefore while PCA performs well in many practical settings, it does not result in the development of a model that relies upon a small set of the original features and so for this reason, PCA is not a feature selection technique.

Principal Component Analysis (PCA) is an unsupervised, non-parametric statistical technique primarily used for dimensionality reduction in machine learning.

Principal component analysis is a useful technique when dealing with large datasets. In some fields, (bioinformatics, internet marketing, etc) we end up collecting data which has many thousands or tens of thousands of dimensions. Manipulating the data in this form is not desirable, because of practical considerations like memory and CPU time. However, we can't just arbitrarily ignore dimensions either. We might lose some of the information we are trying to capture!

Principal component analysis is a common method used to manage this tradeoff. The idea is that we can somehow select the 'most important' directions, and keep those, while throwing away the ones that contribute mostly noise.

For example, this picture shows a 2D dataset being mapped to one dimension:

Note that the dimension chosen was not one of the original two: in general, it won't be, because that would mean your variables were uncorrelated to begin with.
We can also see that the direction of the principal component is the one that maximizes the variance of the projected data. This is what we mean by 'keeping as much information as possible.'

In Principal Component Analysis (PCA) we look to summarize a large set of correlated variables (basically a high dimensional data) into a smaller number of representative variables, called the principal components, that explains most of the variability in the original set.

The first principal component axis is selected in a way such that it explains most of the variation in the data and is closest to all n observations.

• The Principal Component Analysis (PCA) is the process of computing principal components and using them to perform a change of basis on the data.
• The Principal Component of a collection of points in a real coordinate space are a sequence of p unit vectors, where the i-th vector is the direction of a line that best fits the data while being orthogonal to the i - 1 vectors. The best-fitting line is defined as the line that minimizes the average squared distance from the points to the line.
• PCA is commonly used in dimensionality reduction by projecting each data point onto only the first few principal components to obtain lower-dimensional data while preserving as much of the data's variation as possible.

• In PCA the principal components are represented by the eigenvectors which are stored in the attribute components_.
• The eigenvalues represent the variance in the direction of the eigenvector. We can get them through the attribute explained_variance_.

from sklearn.decomposition import PCA
import numpy as np

data = np.array([[2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3.0], [2.3, 2.7], [2, 1.6], [1, 1.1], [1.5, 1.6], [1.1, 0.9]])
pca = PCA()
pca.fit(data)

# eigenvectors
print(pca.components_)
# eigenvalues
print(pca.explained_variance_)

PCA is an operation applied to a dataset, represented by an n x m matrix A that results in a projection of A which we will call B. This operation is calculated using the tools of linear algebra as follows:

1. Calculate the mean values of each column, M.
2. Calculate the centered matrix C = A - M, as its name suggests this matrix centers the values in each column of A by subtracting the mean column value M.
3. Calculate the covariance matrix V of the centered matrix C, with this, we obtain a generalized and unnormalized version of correlation across multiple columns which provide information about the linear relationship between them.
4. Calculate the eigendecomposition of the covariance matrix. This results in a list of eigenvalues and a list of eigenvectors. The eigenvectors represent the directions or components for the reduced subspace and the eigenvalues represent the magnitudes for the directions.
5. Sort the eigenvectors by the eigenvalues in descending order to provide a ranking of the components or axes of the new subspace for A.

6. Select the k largest vectors who correspond to the k largest eigenvalues to form the matrix B. The k value varies depending on the problem, but it's generally fewer than m.

By now we have obtained the principal components of the dataset A, they represent the directions of the data that explain a maximal amount of variance, that is to say, the lines that capture most information of the data.

7. The last step is to project A into B via matrix multiplication P = Bᵗ⋅A, with this, we reorient the data from the original axes to the ones represented by the principal components and reduce the dimensions of A.

• LLE goes beyond the density modeling of the PCA. Density models do not provide a consistent set of global coordinates which embed the observations across the entire manifold. These can only detect linear patterns in the data and does a bad job detecting the curved pattern. LLE is able to detect the curved path of the dataset as shown below:

• LLE is more efficient compared to other algorithms in terms of its computation space and time because it tends to accumulate sparse matrices.

• Where PCA calculates the vector or direction that maximizes the variance so loses the least amount of information during the projection,
• Random Projection simply picks any vector and then performs the projection.

This works very well in high-dimensional spaces and is very computationally efficient.

The basic premise is that it is possible to reduce the number of dimensions in a dataset by multiplying the dataset by a random matrix.

The theoretical underpinning is something called the Johnson-Lindenstrauss lemma:

“A dataset of N points in high-dimensional Euclideanspace can be mapped down to a space in much lower dimension in a way that preserves the distance between the points to a large degree.”

• PCA:

  • It is a mathematical technique based on linear algebra.
  • Tries to preserve global structure/Shape.
  • It does not take into consideration the distance between data points but the direction of maximum variability.
  • Is a linear technique: reduce the dimension of the data when the linear correlations are strong.

• t-SNE:

  • Is based on probabilities.
  • Tries to preserve the local structure by taking into account the distance between the point.
  • Is a non-linear technique so it can interpret the complex polynomial relationships between features.

• Scaling is important because of the way that PCA is calculated. It is calculated via the Singular Value Decomposition, which finds linear subspaces which best represent the data in the squared sense.
• Scaling is important because singular value decomposition approximates in the sum of squares sense. So, if one variable is on a different scale than another, it will dominate the PCA procedure.
• Linear subspaces is an important topic of study in linear algebra, but the most important consequence of a linear subspace for PCA is that it should go through the origin. For example, if the GDP is being measured then most of the data will live very far from the origin, and be poorly approximated by any linear space, so it is important to center the data. Centering the data guarantees that it exists near the origin. A centered dataset is shown below:

The PCA object is very useful but has certain limitations for large datasets. The biggest limitation is that PCA only supports batch processing, which means all of the data to be processed must fit in the main memory.

A better alternative to use with large dataset is IncrementalPCA, this object uses a different form of processing and allows for partial computations which almost exactly match the results of PCA while processing the data in a minibatch fashion.

IncrementalPCA has the parameter batch_size to specify the number of samples to use for each batch and only stores estimates of component and noise variances, in order to update explained_variance_ratio_ incrementally. The memory usage depends on the number of samples per batch, rather than the number of samples to be processed in the dataset.

Therefore depending on the size of the input data, this algorithm can be much more memory efficient than a PCA, and allows sparse input.