17 Gradient Descent Interview Questions Every Data Analyst And ML Engineer Must Know

Gradient descent is an optimization algorithm that’s used when training a machine learning model and is based on a convex function and tweaks its parameters iteratively to minimize a given function to its local minimum (that is, slope = 0).

For a start, we have to select a random bias and weights, and then iterate over the slope function to get a slope of 0.

The way we change update the value of the bias and weights is through a variable called the learning rate. We have to be wise on the learning rate because choosing:

• A small leaning rate may lead to the model to take some time to learn
• A large learning rate will make the model converge as our pointer will shoot and we’ll not be able to get to minima.

• A Cost Function is something we want to minimize. For example, our cost function might be the sum of squared errors over the training set.
• Gradient Descent is a method for finding the minimum of a function of multiple variables.

• A Gradient Descent is a type of optimization algorithm used to find the local minimum of a differentiable function.
• The main idea behind the gradient descent is to take steps in the negative direction of the gradient. This will lead to the steepest descent and eventually it will lead to the minimum point.
• It is shown as an equation by:

Where:

• a is the point.
• $\gamma$ is the step size.
• F(x) is the multi-variable function.

The applicability of batch or stochastic gradient descent depends on the error manifold expected.

• Batch gradient descent computes the gradient using the whole dataset. This is great for convex, or relatively smooth error manifolds. In this case, we move somewhat directly towards an optimum solution, either local or global. Additionally, batch gradient descent, given an annealed learning rate, will eventually find the minimum located in its basin of attraction.
• Stochastic gradient descent (SGD) computes the gradient using a single sample. Most applications of SGD use a minibatch of several samples. SGD works better than batch gradient descent for error manifolds that have lots of local maxima/minima. In this case, the somewhat noisier gradient calculated using the reduced number of samples tends to jerk the model out of local minima into a region that hopefully is more optimal. Single samples are noisy, while mini-batches tend to average a little of the noise out. Thus, the amount of jerk is reduced when using mini-batches. A good balance is struck when the minibatch size is small enough to avoid some of the poor local minima but large enough that it doesn't avoid the global minima or better-performing local minima.

One benefit of SGD is that it's computationally a whole lot faster. Large datasets often can't be held in RAM, which makes vectorization much less efficient. Rather, each sample or batch of samples must be loaded, worked with, the results stored, and so on. Minibatch SGD, on the other hand, is usually intentionally made small enough to be computationally tractable. Usually, this computational advantage is leveraged by performing many more iterations of SGD, making many more steps than conventional batch gradient descent. This usually results in a model that is very close to that which would be found via batch gradient descent, or better.

• For a Batch Gradient Descent the whole training data is taken into consideration to take a single step.
• For a Stochastic Gradient Descent only one data is taken to take a single step.
• For a Mini-batch Gradient Descent a batch of a fixed number of data is taken and the descent is calculated.

• Batch Gradient Descent is good for convex or relatively smooth error manifolds.
• Stochastic Gradient Descent has a fluctuating cost function but in the long run the cost function will decrease with fluctuations.
• Mini-batch Gradient Descent also has a fluctuating cost function because the error is being averaged over a batch.

The Gradient Descent works by starting with random values for each coefficient in the linear regression model.

• After this, the sum of the squared errors is calculated for each pair of input and output values (loss function), using a learning rate as a scale factor.
• For each iteration, the coefficients are updated in the direction towards minimizing the error,
• then we keep repeating the iteration process until a minimum sum squared error is achieved or no further improvement is possible.

To answer the given question, let’s first understand the difference between these two techniques.

In Gradient descent:

• We need to perform hyper-parameter tuning for the learning parameter.
• We need to iterate.
• We have a time complexity of O(kn²).

Meanwhile, in Ordinary Least Squares:

• There is no need for any hyperparameter.
• No iteration is needed.
• We have a time complexity of O(n³).

Clearly, if we have large training data, ordinary least squares is not preferred for use due to very high time complexity but for small values of n, ordinary least squares is faster than gradient descent so it would be preferred the classical approach.

• Mean absolute error (MAE): calculates the absolute difference between actual and predicted values. It can be used when we want that our model be robust to outliers, but this metric has the disadvantage of not being differentiable so we can't use it if we want to apply optimizers like Gradient descent.

• Mean squared error (MSE): calculates the squared difference between actual and predicted value. We can use this metric if we want to give bigger penalization to outliers and apply optimizers who require differentiation. MSE is a differentiable function that makes it easy to perform mathematical operations in comparison to a non-differentiable function like MAE.

• Root mean squared error (RMSE): This is simply the square root of mean squared error. This metric is not so robust to outliers as the mean absolute error but it has the advantage to be differentiable so we can use it if we want to apply gradient descent to minimize losses.

When to use one depends on your loss function:

• When to use MAE: If being off by ten is just twice as bad as being off by 5. it is better to use the MAE if you don't want your performance metric to be overly sensitive to outliers.
• When to use RMSE: In many circumstances, it makes sense to give more weight to points further away from the mean - that is, being off by 10 is more than twice as bad as being off by 5. In such cases, RMSE is a more appropriate measure of error.

There are three types of gradient descent methods based on the amount of data used to calculate the gradient:

• Batch gradient descent - In batch gradient descent, to calculate the gradient of the cost function, we calculate the error for each example in the training dataset and then take the sum. The model is updated only after all examples have been evaluated.
• Stochastic gradient descent - In SGD, we use one training sample at each iteration instead of using the whole dataset to sum all for every step, that is — SGD performs a parameter update for each observation. So instead of looping over each observation, it just needs one to perform the parameter update. In SGD, before for-looping, we need to randomly shuffle the training examples.
• Mini-batch gradient descent - Mini-batch gradient descent uses n data points (instead of one sample in SGD) at each iteration.

A few types of stopping conditions are as follows:

• maxit: this is a predetermined maximum number of iterations. Another similar alternative is a maximum number of seconds before timing out. If all you need is an approximate solution, this can be a very reasonable solution.
• abstol: i.e., stop when the function gets "close enough" to zero.
• reltol: this is to stop when the improvement drops below a threshold.

Other possible options for finding lower minima in a reasonable amount of time could include:

• Stochastic gradient descent, which only requires estimating the gradients for a small portion of your data at a time (e.g. one data point for "pure" SGD, or small mini-batches).
• More advanced optimization functions (e.g. Newton-type methods or Conjugate Gradient), which use information about the curvature of your objective function to help you point in better directions and take better step sizes as you move downhill.
• A momentum term in your update rule, so that your optimizer does a better job of rolling downhill rather than bounding off canyon walls in your objective function.