35 Artificial Neural Network (ANN) Interview Questions (EXPLAINED) for Data Scientists

• Artificial neural networks are modelled from biological neurons.
• The connections of the biological neuron are modeled as weights.
  • A positive weight reflects an excitatory connection, while negative values mean inhibitory connections.
  • All inputs are modified by a weight and summed. This activity is referred to as a linear combination.
• Finally, an activation function controls the amplitude of the output. For example, an acceptable range of output is usually between 0 and 1, or it could be −1 and 1.

• Neural Networks are used in unsupervised learning to learn better representations of the input data.
• Neural networks can learn a mapping from document to real-valued vector in such a way that resulting vectors are similar for documents with similar content. This can be achieved using autoencoders which is a model that is trained to reconstruct the original vector from a smaller representation with reconstruction error as a cost function.
• There are neural networks that are specifically designed for clustering as well. The most widely known is the self-organizing maps (SOM).

• The neural networks get the optimal weights and bias values through an Error Gradient.
• To decide whether to increase or decrease the current weights and bias, it needs to be compared to the optimal value. This is found by the gradients of error with respect to weights and bias:

• The gradient value is calculated from a selected algorithm called backpropagation.
• An optimization algorithm utilizes the gradient to improve the weight values and bias.

• An activation function applies a step rule (convert the numerical output into +1 or -1) to check if the output of the weighting function is greater than zero or not.
• An activation function of a node defines the output of the node given an input or set of inputs to the node.

• Activation functions can be divided into three categories:
  • Ridge functions
  • Radial functions, and
  • Fold functions.

• A type of ridge function called Rectified Linear Function (ReLU) is shown below:

• A type of radial function called Gaussian Function is shown below:

• A fold function perform an aggregation over the inputs, such as taking the mean, minimum or maximum.

• When researchers started to create large artificial neural networks, they started to use the word deep to refer to them.
• As the term deep learning started to be used, it is generally understood that it stands for artificial neural networks which are deep as opposed to shallow artificial neural networks.
• Deep Artificial Neural Networks and Deep Learning are generally the same thing and mostly used interchangeably.

• Forward propagation is the input data that is fed in the forward direction through the network. Each hidden layer accepts the input data, processes it as per the activation function, and passes it to the successive layer.
• Back propagation is the practice of fine-tuning the weights of the neural network based on the error rate obtained from the previous epoch. Proper tuning of the weights ensures low error rates, making the model more reliable.

• In a single layer Neural Network, a set of inputs are directly mapped to an output by using a generalized variation of a linear function. This is called a perceptron.

• A perceptron may be designed to give a binary class output. Historical cases for which the class variable is observed, and other cases in which the class variable has not been observed needs to be predicted.

The initialization step can be critical to the model's performance, and it requires the right method.

• Initializing the weights to zero leads the network to learn zero output which makes the network not learn anything.
• Initializing the weights to be too large causes the network to experience exploding gradients.
• Initializing the weights to be too small causes the network to experience vanishing gradients.

To find the perfect initialization, there are a few rules of thumb to follow:

• The mean of activations should be zero.
• The variance of activations should stay the same across every layer.

• Which scaling method works best depends on the problem, because different problems have different optimization surfaces.
• A very general strategy is to experiment: test how well the model works with alternative methods. This can be expensive, though, since the scaling will interact with other model configuration choices, such as the learning rate, effectively meaning that you'll be testing all model configurations for all scaling choices. This can be tedious, so it's typical to pick a simple method that works "well enough" for some problem and focus on more interesting considerations.

• It should be kept in mind that scaling using the min and max can be extremely sensitive to outliers: if there is even one value orders of magnitude larger or smaller than the rest of the data, then the denominator is very large. As a result, scaling will clump the rest of the data in a narrow segment of the [0,1] or [−1,1] interval, so the range used by most of the data is much narrower.
• A single large outlier will strongly influence the denominator of the scaling even for z scales, but the larger the sample size, the less and less that influence is present. On the other hand, methods using the max and min will always be strongly influenced by a single outlier.

• Convolutional neural networks (CNN) is mostly used in image classification but it can also be used for NLP.
• For NLP tasks the sentences are represented as matrices. The row of the matrix consists of a token (or a character).
• The filters of the CNN can be made to slide over the row of the matrix.
• The height may vary, but sliding the windows over 2-5 words is typical.

• Being able to learn from one or a few examples is called one-shot learning. This is attainable in human beings but it is not yet attainable in artificial neural networks.
• The connectivity of neurons in the human brain is relatively sparse and has been evolving for millions of years.
• Much like how human beings learn by fine-tuning their encodings over a lifetime and pass those encodings through generations, transfer learning has been a subject of research for artificial neural networks.
• Being able to transfer learn between neural network models will prove to be useful when learning new models. Then the act of learning new images to classify will just be fine-tuning some parameters in the ANN.

• A very strong correlation between the new feature and an existing feature is a fairly good sign that the new feature provides little new information.

• A low correlation between the new feature and existing features is likely preferable.

• A strong linear correlation between the new feature and the predicted variable is a good sign that a new feature will be valuable, but the absence of a high correlation is not necessarily a sign of a poor feature, because neural networks are not restricted to linear combinations of variables.

• If the new feature was manually constructed from a combination of existing features, consider leaving it out. The beauty of neural networks is that little feature engineering and preprocessing are required - features are instead learned by intermediate layers.

• Whenever possible, prefer learning features to engineering them.

A model with too little capacity cannot learn the problem, but a model with too much capacity will overfit the data. To prevent overfitting the data some things that can be done are:

• Training the model in more data. Giving the model an infinite number of examples will plateau the model in terms of what the capacity of the network is capable of learning.
• Changing the network structure (number of weights). Pruning by removing the nodes in the network will counteract the overfitting.
• Changing the network parameters (values of weights). Larger weights cause sharp transitions in the activation functions and cause large changes in output for small changes in inputs. This method to keep weights small is called Regularization.

• Learning Rate: Setting the learning rate low will cause the learning process to slow down but converge smoothly. If the learning rate is high then the learning process will speed up but it might never converge.
• Momentum: Momentum helps know the direction of the next step with the knowledge of the previous step. This will help prevent oscillations. Typically, momentum is chosen between 0.5 and 0.9.
• Number of epochs: Increasing the number of epochs will cause the validation accuracy to decrease while increasing the training accuracy, this is due overfitting.
• Batch size: This is the number of subsamples given to the network before which the parameter update happens. Typically, batch size is kept as 32, while different numbers such as 64, 128, 256 should also be tried.

• A multilayer perceptron is a type of neural network which has many layers of perceptron stacked on top of each other.
• Mathematically, multilayer perceptron are capable of learning any mapping function and have been proven to be a universal approximation algorithm.
• Single layer perceptron only learn linear patterns, while multilayer perceptron can learn complex relationships. This predictive capability comes from the multi-layered structure of the network, so that the features can be combined into higher-order features.

• Neural networks require too much data to train. A classification network may require thousands of examples in a single class for it to identify it in unseen data. Due to this, sometimes it is not feasible to create ANN models for fringe applications.
• Neural networks are not interpretable. The user needs to input data into the network and it outputs the required output, but the work that goes into processing the input and giving an output is not understandable to human beings.
• The power required to train the neural network is extremely high compared to the amount of power that a human brain uses (around 20 Watts) to do almost the same things such as image classification.

• SVM has a number of parameters that increases linearly with the linear increase in the size of the input. All the parameters of the SVM interacts with one another.

• Neural Networks have as many layers as possible, but due to the complex interaction between the model's parameters, NN always has a higher complexity when compared with SVM with a similar number of parameters.

• SVMs use only a subset of data for training. They can reliably define the decision boundary solely on the basis of the support vectors. As a result, they do not require much data for training.

• Neural Networks get trained based on the batches of data fed into it. This means that the specific decision boundary that the neural network learns is highly dependent on the order in which the batches of data are presented to it. This, in turn, requires processing the whole training dataset; or otherwise, the network may perform extremely poorly.

• Parametric: SVM and neural networks are both parametric but for different reasons.
  • For SVM the typical parameters are; soft-margin parameter (C), parameter of the kernel function (gamma).
  • Neural networks also have parameters but it is a lot more than SVM. Some NN parameters are the number of layers and their size, number of training epochs, and the learning rate.

• Embedding Non-Linearity: Both the methods can embed non-linear functions.

  • SVM does this through the usage of kernel method.
  • Neural Networks embed non-linearity using non-linear activation functions.

• Comparable Accuracy:

  • If both SVM and Neural Networks are trained in the same dataset, given the same training time, and the same computation power they have comparable accuracy.
  • If neural networks are given as much computation power and training time as possible then it outperforms SVMs.

• The RNN is a stateful neural network, which means that it not only retains information from the previous layer but also from the previous pass. Thus, this neuron is said to have connections between passes, and through time.
• For the RNN the order of the input matters due to being stateful. The same words with different orders will yield different outputs.
• RNN can be used for unsegmented, connected applications such as handwriting recognition or speech recognition.

A 1x1 convolution simply maps an input pixel with all its channels to an output pixel, not looking at anything around itself. It is often used to reduce the number of depth channels since it is often very slow to multiply volumes with extremely large depths.

input (256 depth) -> 1x1 convolution (64 depth) -> 4x4 convolution (256 depth)

input (256 depth) -> 4x4 convolution (256 depth)

The bottom one is about ~3.7x slower.

Theoretically, the neural network can choose which input colors to look at using this, instead of brute force-multiplying everything.

• Networks with more layers (Depth) tend to require fewer units per layer because the composition functions created by successive layers make the neural network more powerful.
• The number of units in each layer of a deep network can be decreased to such an extent that when compared to a shallower network (more Breadth), the deep network has far fewer parameters even when added up over the greater number of layers.
• In neural networks with many layers the loss derivatives with respect to the weights in different layers of the network tend to have vastly different magnitudes, which causes challenges in properly choosing step sizes. This can lead to vanishing and exploding gradient problems.

• The vanishing gradient problem is encountered in artificial neural networks with gradient-based learning methods and backpropagation.
• In these learning methods, each of the neural networks weights receives an update proportional to the partial derivative of the error function with respect to the current weight in each iteration of training. Sometimes when gradients become vanishingly small, this prevents the weight to change value.
• If the neural network has many hidden layers, the gradients in the earlier layers will become very low as we multiply the derivatives of each layer. As a result, learning in the earlier layers becomes very slow.
• This can cause the neural network to stop learning.
• This problem of vanishing gradient descent happens when training neural networks with many layers because the gradient diminishes dramatically as it propagates backwards through the network.

• Many fixes and workarounds have been proposed and investigated to fix the vanishing gradient problem, such as

  • alternate weight initialization schemes,
  • unsupervised pre-training,
  • layer-wise training, and
  • variations on gradient descent.

  Perhaps the most common change is the use of the rectified linear activation function that has become the new default, instead of the hyperbolic tangent activation function that was the default through the late 1990s and 2000s.

• Backpropagation is an algorithm that computes the gradient of the loss function with respect to the weights of the network for a single input-output example.
• Backpropagation only refers to the algorithm used to compute the gradient, it does not include how the gradient is used.
• The formula for updating the weights and biases after calculating the gradient (the partial differentiation in the equation below) is shown below:

$\theta^{t+1} = \theta^t - \alpha \frac{\partial E(X, \theta^t)}{\partial \theta}$

Where:

• $E(X, \theta)$ is the error function.
• $w_{ij}^k$ is the weights.
• $b_i^k$ is the biases.
• $\alpha$ is the learning rate.
• $\theta^t$ is the parameters of the neural network (weights and biases) at iteration t in gradient descent.

Normalizing the data to obtain a mean as close to 0 as possible is always a good thing to do to the input due to the following reasons:

• It improves the numerical stability of the model. This is because the data determines the update rate and a larger value will cause the gradient descent algorithm to overshoot the minimum and hence cause oscillations.
• It improves the training process. When different features have different ranges, then the learning rate depends on the feature with the largest range. When the input is standardized, the training process speeds up.
• Standardization gives equal importance to every feature. In most of the networks, 'good' and 'bad' features can not be differentiated, so it is better to consider every feature equal by standardization.
• In neural networks, empirical evidence show that standardization improves the accuracy of the model.