27 Keras Interview Questions (ANSWERED) for ML Engineers and Data Scientists

All layers & models have a layer.trainable boolean attribute. To freeze them, we just iterate over the layers and set the attribute to False:

# Loop for freezing all layers 
for layer in model.layers:
        layer.trainable = False

# Checking
print(layer.trainable)
print(layer2.trainable)

Output:

False
False

• The Sequential model: the most approachable API, it's basically a Python list. As such, it’s limited to simple stacks of layers.
• The Functional API, which focuses on graph-like model architectures. It represents a nice mid-point between usability and flexibility, and as such, it’s the most commonly used model-building API.
• Model subclassing: a low-level option where you write everything yourself from scratch. This is ideal if you want full control over every little thing. However, you won’t get access to many built-in Keras features, and you will be more at risk of making mistakes.

The neural network needs to start with some weights and then iteratively update them to better values. The term kernel_initializer is a fancy term for which statistical distribution or function to use for initializing the weights. In the case of some statistical distribution defined, Keras will generate numbers from that statistical distribution and use them as starting weights.

For example, in the code model.add(Dense(13, input_dim=13, kernel_initializer='normal', activation='relu')) normal distribution will be used to initialise weights. You can use other functions and distributions like Glorot Normal, Glorot uniform, etc.

Those methods perform the steps that we need to do when we want to train a model:

1. First, we want to decide on a model architecture, this is the number of hidden layers and activation functions, and that's what compile does.
2. Secondly, the fit train our model to get all the parameters to the correct value to map our inputs to our outputs.
3. Lastly, we will want to use this model to do some feed-forward passes to predict novel inputs, that's what the predict method does.

keras.layers.flatten() converts the multi-dimensional arrays into flattened one-dimensional arrays or single-dimensional arrays. In other words, it takes all the elements in the original tensor (multi-dimensional array) and puts them into a single-dimensional array.

From the image above we see that the flatten operation on a tensor reshapes the tensor to have the shape that is equal to the number of elements contained in tensor non including the batch dimension.

A callback is an object (a class instance implementing specific methods) that is passed to the model in the call to fit() and that is called by the model at various points during training. It has access to all the available data about the state of the model and its performance, and it can take action: interrupt training, save a model, load a different weight set, or otherwise alter the state of the model.

Callbacks are useful to:

• Write TensorBoard logs after every batch of training to monitor your metrics
• Periodically save the model to disk.
• Do early stopping when a monitored metric has stopped improving.
• Get a view on internal states and statistics of a model during training, etc.

• The model.evaluate() function first predicts the output for the given input and then computes the metrics function specified in the model.compile, and based on y_true and y_pred it returns the computed metric value as the output for every batch.

• The model.predict() function will give you the actual predictions for all samples in a batch, for all batches.

• So even if you use the same data, the differences between the outputs will be there because the value of a loss function will be almost always different than the predicted values.

• fit() is for training the model with the given inputs (and corresponding training labels).
• evaluate() is for evaluating the already trained model using the validation (or test) data and the corresponding labels. Returns the loss value and metrics values for the model.

The difference lies in when you pass as x data that is larger than one batch.

• predict will go through all the data, batch by batch, predicting labels. It thus internally does the splitting in batches and feeding one batch at a time.
• predict_on_batch, on the other hand, assumes that the data you pass in is exactly one batch and thus feeds it to the network. It won't try to split it (which, depending on your setup, might prove problematic for your GPU memory if the array is very big).

• The units is the amount of neurons, or cells that the layer has inside it.
• The shapes are consequences of the model's configuration and are tuples representing how many elements an array or tensor has in each dimension. What flows between layers are tensors. Tensors can be seen as matrices with shapes. In Keras, the input layer itself is not a layer, but a tensor and its shape it's the input shape. It's that starting tensor you send to the first hidden layer. This tensor must have the same shape as the training data.
• Given the input shape, all other shapes are results of layers calculations: The units of each layer will define the output shape (the shape of the tensor that is produced by the layer and that will be the input of the next layer).
• Furthermore, each type of Layer works in a particular way. For example,
  • Dense layers have output shape based on units
  • Convolutional layers have output shapes based on filters, etc.

You can implement a custom metric in two ways:

1. As simple callables, in which the callable has signature metric_fn(y_true, y_pred) that returns an array of losses (one of sample in the input batch). y_true and y_pred must be tensors so in order to correctly calculate the metric you need to use keras.backend functionality. For example:

   import keras.backend as K
   
   def mean_pred(y_true, y_pred):
      return K.mean(y_pred)
   
   model.compile(optimizer='sgd',
            loss='binary_crossentropy',
            metrics=['accuracy', mean_pred])

2. As subclasses of Metric, in which you have to override the update_state, result, and reset_state functions, where:

   • update_state() does all the updates to state variables and calculates the metric,
   • result() returns the value for the metric from state variables.
   • reset_state() sets the metric value at the beginning of each epoch to a predefined constant.

You may use the Lambda callback and get_weights() method from the Sequential model to hand save the weights in a dictionary.

# Generate Callback
weights_dict = {}
weight_callback = tf.keras.callbacks.LambdaCallback \
( on_epoch_end=lambda epoch, logs:  weights_dict.update({epoch:model.get_weights()}))


history = model.fit(xs, ys, batch_size=batch_size, epochs=epochs, 
                    callbacks=[weight_callback],verbose=0)

# Retrive weights
for epoch,weights in weights_dict.items():
  print(f"Epoch #{epoch}")
  print(f"Weights: {weights[0]}, bias: {weights[1]} \n")

Output:

Epoch #0
Weights: [[0.5750251]], bias: [0.00499998] 

Epoch #1
Weights: [[0.57998466]], bias: [0.00999998] 

Epoch #2
Weights: [[0.58493966]], bias: [0.01499997] 

Epoch #3
Weights: [[0.5898572]], bias: [0.01999997] 

Epoch #4
Weights: [[0.5948121]], bias: [0.02499996] 

Epoch #5
Weights: [[0.5997918]], bias: [0.02999996] 

Epoch #6
Weights: [[0.6047719]], bias: [0.03499996] 

Epoch #7
Weights: [[0.6097574]], bias: [0.03999995] 

Epoch #8
Weights: [[0.61469406]], bias: [0.04499994] 

Epoch #9
Weights: [[0.6196337]], bias: [0.04999993] 

Consider:

from keras.layers import Input, Dense
from keras.models import Model
from keras.utils.vis_utils import plot_model

A1 = Input(shape=(30,),name='A1')
A2 = Dense(8, activation='relu',name='A2')(A1)
A3 = Dense(30, activation='relu',name='A3')(A2)

B2 = Dense(40, activation='relu',name='B2')(A2)
B3 = Dense(30, activation='relu',name='B3')(B2)

merged = Model(inputs=[A1],outputs=[A3,B3])
plot_model(merged,to_file='demo.png',show_shapes=True)

Output structure:

Given the regression equation $y=Wx+b$ where $x$ is the input, $W$ the weights matrix, and $b$ the bias, then

• Kernel Regularizer: Tries to reduce the weights $W$ (excluding bias).
• Bias Regularizer: Tries to reduce the bias $b$.
• Activity Regularizer: Tries to reduce the layer's output $y$, thus will reduce the weights and adjust bias so $Wx+b$ is smallest.

Usually, if you have no prior on the distribution that you wish to model, you would only use the kernel regularizer, since a large enough network can still model your function even if the regularization on the weights are big. We could also apply kernel_regularizer to penalize the weights which are very large causing the network to overfit, after applying kernel_regularizer the weights will become smaller.

On other hand, if you want the output function to pass through (or have an intercept closer to) the origin, you can use the bias regularizer. But if you want the output to be smaller (or closer to 0), you can use the activity regularizer, using this we could also reduce overfitting.

Transfer learning consists of taking features learned on one problem and leveraging them on a new, similar problem. For instance, features from a model that has learned to identify raccoons may be useful to kick-start a model meant to identify tanukis.

The typical transfer-learning workflow in Keras is:

1. Instantiate a base model and load pre-trained weights into it.

base_model = keras.applications.Xception(
    weights='imagenet',  # Load weights pre-trained on ImageNet.
    input_shape=(150, 150, 3),
    include_top=False)  # Do not include the ImageNet classifier at the top.

2. Freeze all layers in the base model by setting trainable = False to avoid updating during training.

base_model.trainable = False

3. Create a new model on top of the output of one (or several) layers from the base model.

inputs = keras.Input(shape=(150, 150, 3))
x = base_model(inputs, training=False)
# Convert features of shape `base_model.output_shape[1:]` to vectors
x = keras.layers.GlobalAveragePooling2D()(x)
# A Dense classifier with a single unit (binary classification)
outputs = keras.layers.Dense(1)(x)
model = keras.Model(inputs, outputs)

4. Train your new model on your new dataset.

model.compile(optimizer=keras.optimizers.Adam(),
              loss=keras.losses.BinaryCrossentropy(from_logits=True),
              metrics=[keras.metrics.BinaryAccuracy()])
model.fit(new_dataset, epochs=20, callbacks=..., validation_data=...)

5. Once the model has converged on the new data, you can try to unfreeze all or part of the base model and retrain the whole model end-to-end with a very low learning rate. This is an optional last step that can potentially give you incremental improvements, but it could also potentially lead to quick overfitting.

# Unfreeze the base model
base_model.trainable = True

# Recompile the model after you make any changes to the `trainable` attribute
model.compile(optimizer=keras.optimizers.Adam(1e-5),  
              loss=keras.losses.BinaryCrossentropy(from_logits=True),
              metrics=[keras.metrics.BinaryAccuracy()])

# Train end-to-end
model.fit(new_dataset, epochs=10, callbacks=..., validation_data=...)

The Sequential model API is a way of creating deep learning models where an instance of the Sequential class is created and model layers are created and added to it. This is great for developing deep learning models in most situations, but it also has some limitations: For example, it is not straightforward to define models that may have multiple different input sources, produce multiple output destinations or models that re-use layers.

The Keras functional API provides a more flexible way for defining models. It specifically allows you to define multiple input or output models as well as models that share layers. More than that, it allows you to define ad hoc acyclic network graphs.

In the functional API, the models are defined by creating instances of layers and connecting them directly to each other in pairs, then defining a Model that specifies the layers to act as the input and output to the model. As a result, creating complex networks such as siamese networks or residual networks becomes possible.

• LSTM is a recurrent layer.
• LSTMCell is an object (which happens to be a layer too) used by the LSTM layer that contains the calculation logic for one step.

A recurrent layer contains a cell object:

• The cell contains the core code for the calculations of each step,
• while the recurrent layer commands the cell and performs the actual recurrent calculations.

Usually, people use LSTM layers in their code, or they use RNN layers containing LSTMCell. Both things are almost the same. An LSTM layer is an RNN layer using an LSTMCell.

Data Generators is one of the most useful features of the Keras API. Consider a scenario where you have lots of data, so much that you cannot have all of it at once in the RAM.

Well, the solution to this can be loading the mini-batches fed to the model dynamically. This is exactly what data generators do and why they are useful. They can generate the model input dynamically thus forming a pipeline from the storage to the RAM to load the data as and when it is required. Another advantage of this pipeline is, one can easily apply preprocessing routines on these mini-batches of data as they are prepared to feed the model.

• BatchNormalization (BN) is just another layer, so you can use it as such to create your desired network architecture.
• The general use case is to use BN is between the linear and non-linear layers in the network, because it normalizes the input to your activation function so that you're centered in the linear section of the activation function (such as Sigmoid).