17 Recurrent Neural Network (RNN) Interview Questions For Data Scientists and ML Engineers

Convolutional neural nets apply a convolution to the data before using it in fully connected layers.

• They are best used in cases where you want positional invariance, that is to say, you want features to be captured regardless of where they are in the input sample.

• Think of a picture with all sorts of animals in it. If you apply a convolutional neural net to classify whether there is a cat in the picture, it will identify the cat no matter what position in the picture the cat is (at the top, the bottom, left or right). This is very useful for image classification.

Recurrent neural nets are neural networks that keep state between input samples. They remember previous input samples and use those to help classify the current input sample.

• They are most useful when the order of your data is important. So for instance in speech (previous words do help identify the current word), video (frames are ordered) and also text processing.

• Generally speaking, problems related to time-series data (data with a timestamp on them) are good candidates to be solved well with recurrent neural nets.

An RNN layer must have three-dimensional inputs:

• The first dimension is the batch dimension (its size is the batch size.
• The second dimension represents the time (its size is the number of time steps),
• And the third dimension holds the inputs at each time step (its size is the number of input features per time step).

For example, if you want to process a batch containing 5 time series of 10 time steps each, with 2 values per time step (e.g., the temperature and the wind speed), the shape will be [5, 10, 2].

The outputs are also three-dimensional, with the same first two dimensions, but the last dimension is equal to the number of neurons. For example, if an RNN layer with 32 neurons processes the batch we just discussed, the output will have a shape of [5, 10, 32].

The two main difficulties when training RNNs are unstable gradients (exploding or vanishing) and a very limited short-term memory. These problems both get worse when dealing with long sequences.

To alleviate the unstable gradients problem, we can:

• Use a smaller learning rate.

• Use a saturating activation function such as the hyperbolic tangent (which is the default), and possibly use gradient clipping, Layer Normalization, or dropout at each time step.

To tackle the limited short-term memory problem, we can use a Long Short-Term Memory layer or a Gated recurrent unit layer.

• The RNN is a stateful neural network, which means that it not only retains information from the previous layer but also from the previous pass. Thus, this neuron is said to have connections between passes, and through time.
• For the RNN the order of the input matters due to being stateful. The same words with different orders will yield different outputs.
• RNN can be used for unsegmented, connected applications such as handwriting recognition or speech recognition.

There are different types of recurrent neural networks with varying architectures. Some examples are:

• One to one: Here there is a single (xt,yt) pair. Traditional neural networks employ a one to one architecture.

• One to many: In one to many networks, a single input at xt can produce multiple outputs, e.g., yt0, yt1, yt2 Music generation is an example area, where one to many networks are employed.

• Many to one: In this case, many inputs from different time steps produce a single output. For example, xt, xt+1, xt+2 can produce a single output. Such networks are employed in sentiment analysis or emotion detection, where the class label depends upon a sequence of words.

• Many to many: There are many possibilities for many to many. An example is shown below, where two inputs produce three outputs. Many to many networks are applied in machine translation, e.g, English to French or vice versa translation systems.

• In a stateless RNNs, on each training iteration the model starts with a hidden state full of zeros, then it updates this state at each time step, and after the last time step, it throws it away, as it is not needed anymore.
• A stateful RNNs preserve this final state after processing one training batch and use it as the initial state for the next training batch, this way the model can learn long-term patterns.
• Given the inner work of stateless RNNs, they can only capture patterns whose length is less than, or equal to, the size of the windows the RNN is trained. Conversely, stateful RNNs can capture longer-term patterns.
• However, implementing a stateful RNN is much harder, especially preparing the dataset properly.
• Moreover, stateful RNNs do not always work better, in part because consecutive batches are not independent and identically distributed (IID) and Gradient Descent is not fond of non-IID datasets.

The main difference is in how the input data is taken in by the model.

• Traditional feedforward neural networks take in a fixed amount of input data all at the same time and produce a fixed amount of output each time.

• Recurrent neural networks do not consume all the input data at once. Instead, they take them in one at a time and in a sequence:

  • At each step, the RNN does a series of calculations before producing an output. The output, known as the hidden state, is then combined with the next input in the sequence to produce another output.
  • It may seem that a different RNN cell is being used at each time step in the network, but the underlying principle of Recurrent Neural Networks is that the RNN cell is actually the exact same one and reused throughout.
  • This process continues until the model is programmed to finish or the input sequence ends.

In the image below there is a comparison of Recurrent Neural Networks (on the left) and Feedforward Neural Networks (on the right).

• Recurrent Neural Networks:

  • It is used for sequential inputs where the time factor is the main differentiating factor between the elements of the sequence, that's why it's commonly used in time-series.
  • When we unfold the network, at each time step, it accepts the user input at that time step and the output of the hidden layer that was computed at the previous time step.
  • The weights are shared (and dimensionality remains constant) along the length of the sequence.

• Recursive Neural Networks:

  • Is more like a hierarchical network where there is really no time aspect to the input sequence but the input has to be processed hierarchically in a tree fashion.
  • Is very used in NLP, where the way to learn a parse tree of a sentence is recursively taking the output of the operation performed on a smaller chunk of the text.
  • Here, the weights are shared (and dimensionality remains constant) at every node.

• Multilayer Perceptrons, or MLPs for short are the classical type of neural network. They are very flexible and can be used generally to learn a mapping from inputs to outputs, however, they are perhaps more suited to classification and regression problems.

• Convolutional Neural Networks, or CNNs, were developed and are best used for image classification. But they can also be used generally with data that has a spatial structure, such as a sequence of words, and can be used for document classification.

• Recurrent Neural Network or RNNs, was developed for sequence prediction and is well suited for problems that have a sequence of input observations or a sequence of output observations. They are suitable for text data, audio data, and similar applications.

• In an RNN, the output of the previous state is passed as an input to the current state.
• There is a temporal relationship in the way in which input is processed in an RNN. It can understand how the current state was achieved based on the previous values, i.e. value at time-step t is a result of value at time-steps t-1, t-2, and so on.
• In DNN, there is no temporal relationship in the way input is processed. Values at time-steps t, t-1, t-2,... are all treated distinctly and not as a continuation of the previous time-step values.

A plain sequence-to-sequence RNN would start translating a sentence immediately after reading the first word of a sentence, while an Encoder-Decoder RNN will first read the whole sentence and then translate it.

In general, if you translate a sentence one word at a time, the result will be terrible. For example, the french sentence "Je vous en prie" means "You are welcome" but if you translate it one word at a time using plain sequence-to-sequence RNN, you get "I you in pray" which it does not have sense. So in automatic translation cases is much better to use Encoder-Decoder RNNs to read the whole sentence first and then translate it.