30 TensorFlow Interview Questions (ANSWERED) for ML Devs & Data Science

• In TensorFlow 1.x, the easiest way to evaluate the actual value of a Tensor object is to pass it to the Session.run() method, or call Tensor.eval() when you have a default session (i.e. in a with tf.Session() block).

  matrix1 = tf.constant([[3., 3.]])
  matrix2 = tf.constant([[2.],[2.]])
  product = tf.matmul(matrix1, matrix2)
  
  with tf.Session() as sess:     
           print(sess.run(product))
           print (product.eval())
  [[12.]]
  [[12.]]

• In Tensorflow 2.x (or in Eager mode environment) you can call numpy() method:

  import tensorflow as tf
  
  matrix1 = tf.constant([[3., 3.0]])
  matrix2 = tf.constant([[2.0],[2.0]])
  product = tf.matmul(matrix1, matrix2)
  
  print(product.numpy()) 
  [[12.]]

1. Create required model

   model = tf.keras.models.Sequential()
   
    # Create model
    model.add(tf.keras.layers.Flatten(input_shape=(28, 28)))
    model.add(tf.keras.layers.Dense(512, activation='relu'))
    model.add(tf.keras.layers.Dropout(0.2))
    model.add(tf.keras.layers.Dense(10, activation='softmax'))
   
    model.summary() 

   Output:

   Model: "sequential"
   _________________________________________________________________
   Layer (type)                Output Shape              Param #   
   =================================================================
   flatten (Flatten)           (None, 784)               0         
                                                                 
   dense (Dense)               (None, 512)               401920    
                                                                 
   dropout (Dropout)           (None, 512)               0         
                                                                 
   dense_1 (Dense)             (None, 10)                5130      
                                                                 
   =================================================================
   Total params: 407,050
   Trainable params: 407,050
   Non-trainable params: 0
   _________________________________________________________________

2. Configure TensorBoard

   # 2. Configure Tensorboard
   log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
   tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

3. Compile and train model

   # Compile model
   model.compile(optimizer='sgd', 
     loss='sparse_categorical_crossentropy',
     metrics=['accuracy'])
   
   # train model
   model.fit(x=X_train, 
             y=y_train, 
             epochs=10, 
             validation_data=(X_test, y_test), 
             callbacks=[tensorboard_callback])

   Output:

   Epoch 1/10
   1875/1875 [==============================] - 9s 5ms/step - loss: 0.6415 - accuracy: 0.8337 - val_loss: 0.3472 - val_accuracy: 0.9066
   Epoch 2/10
   1875/1875 [==============================] - 9s 5ms/step - loss: 0.3460 - accuracy: 0.9028 - val_loss: 0.2817 - val_accuracy: 0.9247
   Epoch 3/10
   1875/1875 [==============================] - 8s 4ms/step - loss: 0.2936 - accuracy: 0.9180 - val_loss: 0.2487 - val_accuracy: 0.9309
   Epoch 4/10
   1875/1875 [==============================] - 8s 4ms/step - loss: 0.2610 - accuracy: 0.9259 - val_loss: 0.2245 - val_accuracy: 0.9392
   Epoch 5/10
   1875/1875 [==============================] - 9s 5ms/step - loss: 0.2346 - accuracy: 0.9337 - val_loss: 0.2038 - val_accuracy: 0.9435
   Epoch 6/10
   1875/1875 [==============================] - 12s 6ms/step - loss: 0.2132 - accuracy: 0.9400 - val_loss: 0.1894 - val_accuracy: 0.9468
   Epoch 7/10
   1875/1875 [==============================] - 12s 6ms/step - loss: 0.1968 - accuracy: 0.9458 - val_loss: 0.1749 - val_accuracy: 0.9509
   Epoch 8/10
   1875/1875 [==============================] - 10s 5ms/step - loss: 0.1843 - accuracy: 0.9485 - val_loss: 0.1639 - val_accuracy: 0.9540
   Epoch 9/10
   1875/1875 [==============================] - 11s 6ms/step - loss: 0.1708 - accuracy: 0.9524 - val_loss: 0.1544 - val_accuracy: 0.9557
   Epoch 10/10
   1875/1875 [==============================] - 11s 6ms/step - loss: 0.1606 - accuracy: 0.9557 - val_loss: 0.1453 - val_accuracy: 0.9588
   <keras.callbacks.History at 0x7f2f007d7750>

4. Show TensorBoard

   %load_ext tensorboard # (On Jupyter notebook or Colab)
   %tensorboard --logdir logs/fit

5. Show model losses and final architecture.

• The Scalars tab shows changes in the loss and metrics over the epochs.

• The Graphs tab shows the model's layers. We can use this to check if the architecture of the model looks as intended. To see the model select the keras tag. For this example, you’ll see a collapsed Sequential node Double-click the node to see the model’s structure:

In TensorFlow, there is no native .dot_product method. However, a dot product between two vectors is just element-wise multiply summed, so the following example works:

import tensorflow as tf

a = tf.constant([[1,2,3],[4,5,6]])
b = tf.constant([[2,3,4],[5,6,7]])

c = tf.reduce_sum( tf.multiply( a, b ), 1)
print(c.numpy())
# [20 92]

Taking the diagonal of tf.tensordot also does what you want, if you set the axes to [[1], [1]]:

d = tf.linalg.diag_part(tf.tensordot( a, b, axes=[[1],[1]]))
print(d.numpy())
# [20 92]

1. Print initial dataframe

   import tensorflow as tf  
   tf_dataset = tf.data.Dataset.from_tensor_slices(dataset) 
   
   for i in tf_dataset:
       print(i.numpy())

   Output:

   12
   15
   67
   -56
   78
   90
   25
   -890
   -45
   67
   90
   45
   34
   -100
   300

2. Make data pipeline and print final dataset

   tf_dataset_new = tf_dataset.filter(lambda x: x>0).map(lambda a: a*2).shuffle(2)
   
   for i in tf_dataset_new:
     print(i.numpy())

   Output: (it could vary given randomness effects)

    30
    134
    24
    180
    50
    156
    134
    180
    90
    600
    68

To save a model's architecture, weights, and training configuration in a single file/folder we can use two different file formats: SavedModel and HDF5.

• SavedModel: Models saved in this format can be restored using tf.keras.models.load_model and are compatible with TensorFlow Serving.

  # Create and train a new model instance.
  model = create_model()
  model.fit(train_images, train_labels, epochs=5)
  
  # Save the entire model as a SavedModel.
  !mkdir -p saved_model
  model.save('saved_model/my_model')

  The SavedModel format is a directory containing a protobuf binary and a TensorFlow checkpoint.

• HDF5 format: it's a save format using the HDF5 standard.

    # Create and train a new model instance.
    model = create_model()
    model.fit(train_images, train_labels, epochs=5)
  
    # Save the entire model to a HDF5 file.
    model.save('my_model.h5')

Merging refers to merging multiple tensors into one tensor in a certain dimension. There are two kinds of Merging-concatenation and stacking:

• tf.concat directly merges data on existing dimensions and does not create a new dimension. The only constraint is that the length of the non-concatenated dimensions must be the same.

• tf.stack create a new dimension when merging data, it uses the parameter axis to specify where to insert the new dimension. When an axis is a positive number, it will insert a new dimension in front of axis. When the axis is negative, it will insert a new dimension in the next position. tf.stack also needs to meet the merging conditions of tensor stacking: it requires all merged tensors to have the same shape before they are merged.

import tensorflow as tf
tf.random.set_seed(42)

a = tf.random.normal([1,2,3])
b = tf.random.normal([2,2,3])
c = tf.concat([a,b], axis=0)
print("c:", c)
# c: tf.Tensor(
# [[ 0.73423964  0.99546355  0.3393789 ]
#  [-0.8285848   1.3671659   1.2136413 ]]
#
# [[-0.88899225  0.1333008  -0.48107582]
#  [ 0.36787292 -0.802009   -0.405421  ]]
#
# [[-0.87827003  0.4933785  -0.20411207]
#  [ 0.7755407  -1.7382799  -0.10844299]]], shape=(3, 2, 3), dtype=float32)

a = tf.random.normal([2,3])
b = tf.random.normal([2,3])
d = tf.stack([a,b], axis = -1)
print("d:", d)
# d: tf.Tensor(
# [[[ 0.20421563  1.5475597 ]
#  [-0.26615778  1.1176629 ]
#  [-0.76890045 -0.9775718 ]]
#
# [[-0.36246812 -0.22080888]
#  [ 0.9352817  -1.5796733 ]
#  [-0.9569862  -0.34479442]]], shape=(2, 3, 2), dtype=float32)

1. Convert numpy array to tensor with tf.convert_to_tensor()
2. Use tf.unique() function to return the unique elements and its index.

import numpy as np
import tensorflow as tf

_x = np.array([1, 2, 6, 4, 2, 3, 2, 4, 1,5 ,6,8, 33 ])
x = tf.convert_to_tensor(_x)

out, indices = tf.unique(x)
print(out.numpy())
print(indices.numpy())
# [ 1  2  6  4  3  5  8 33]
# [0 1 2 3 1 4 1 3 0 5 2 6 7]

The indices have the same size as _x and contain the index of each value of _x in the unique out tensor.

If the initializer is None, the default initializer passed in the variable scope will be used. If that one is None too, a glorot_uniform_initializer will be used. The glorot_uniform_initializer function, initializes values from a uniform distribution.

• from_tensors combines the input and returns a dataset with a single element; can be used to construct a larger dataset from several small datasets, i.e., the size (length) of the final dataset becomes larger;

  >>> t = tf.constant([[1, 2], [3, 4]])
  >>> ds = tf.data.Dataset.from_tensors(t)
  >>> [x for x in ds]
      [<tf.Tensor: shape=(2, 2), dtype=int32, numpy=
      array([[1, 2],
              [3, 4]], dtype=int32)>]

• from_tensor_slices create a dataset with a separate element for each row of the input tensor; can be used to combine different elements into one dataset, e.g., combine features and labels into one dataset. That is, the dataset becomes "wider".

  >>> t = tf.constant([[1, 2], [3, 4]])
  >>> ds = tf.data.Dataset.from_tensor_slices(t)
  >>> [x for x in ds]
      [<tf.Tensor: shape=(2,), dtype=int32, numpy=array([1, 2], dtype=int32)>,
      <tf.Tensor: shape=(2,), dtype=int32, numpy=array([3, 4], dtype=int32)>]

1. Create model as required:

   cnn_model = ks.models.Sequential()
   
   # First layer
   cnn_model.add(ks.layers.Conv2D(50, (3, 3), activation='relu', input_shape=(28, 28, 1), name='Conv2D_layer'))
   # Second layer
   cnn_model.add(ks.layers.MaxPooling2D((2, 2), name='Maxpooling_2D'))
   # Third layer
   cnn_model.add(ks.layers.Flatten(name='Flatten'))
   cnn_model.add(ks.layers.Dense(50, activation='relu',name='Hidden_layer'))
   # Output layer
   cnn_model.add(ks.layers.Dense(10, activation='softmax',name='Output_layer'))
   
   cnn_model.summary()

   Output:

   Model: "sequential"
   _________________________________________________________________
   Layer (type)                Output Shape              Param #   
   =================================================================
   Conv2D_layer (Conv2D)       (None, 26, 26, 50)        500       
                                                                   
   Maxpooling_2D (MaxPooling2D  (None, 13, 13, 50)       0         
   )                                                               
                                                                   
   Flatten (Flatten)           (None, 8450)              0         
                                                                   
   Hidden_layer (Dense)        (None, 50)                422550    
                                                                   
   Output_layer (Dense)        (None, 10)                510       
                                                                   
   =================================================================
   Total params: 423,560
   Trainable params: 423,560
   Non-trainable params: 0
   _________________________________________________________________

2. Compile and train model:

    cnn_model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
    cnn_model.fit(training_images, training_labels, epochs=10)

   Output:

    Epoch 1/10
    1875/1875 [==============================] - 40s 21ms/step - loss: 0.3958 - accuracy: 0.8603
    Epoch 2/10
    1875/1875 [==============================] - 38s 20ms/step - loss: 0.2718 - accuracy: 0.9025
    Epoch 3/10
    1875/1875 [==============================] - 38s 20ms/step - loss: 0.2306 - accuracy: 0.9156
    Epoch 4/10
    1875/1875 [==============================] - 38s 20ms/step - loss: 0.2007 - accuracy: 0.9258
    Epoch 5/10
    1875/1875 [==============================] - 38s 20ms/step - loss: 0.1745 - accuracy: 0.9352
    Epoch 6/10
    1875/1875 [==============================] - 38s 20ms/step - loss: 0.1539 - accuracy: 0.9430
    Epoch 7/10
    1875/1875 [==============================] - 38s 20ms/step - loss: 0.1370 - accuracy: 0.9494
    Epoch 8/10
    1875/1875 [==============================] - 38s 20ms/step - loss: 0.1196 - accuracy: 0.9561
    Epoch 9/10
    1875/1875 [==============================] - 38s 20ms/step - loss: 0.1051 - accuracy: 0.9611
    Epoch 10/10
    1875/1875 [==============================] - 38s 20ms/step - loss: 0.0932 - accuracy: 0.9658
    <keras.callbacks.History at 0x7fc5c8ff2650>

3. Model evaluation:

   # Training evaluation
   training_loss, training_accuracy = cnn_model.evaluate(training_images, training_labels)
   print('Training Accuracy {}'.format(round(float(training_accuracy), 2)))
   
   # Test evaluation
   test_loss, test_accuracy = cnn_model.evaluate(test_images, test_labels)
   print('Test Accuracy {}'.format(round(float(test_accuracy), 2)))

   Output:

   1875/1875 [==============================] - 13s 7ms/step - loss: 0.0824 - accuracy: 0.9692
   Training Accuracy 0.97
   313/313 [==============================] - 2s 7ms/step - loss: 0.3182 - accuracy: 0.9095
   Test Accuracy 0.91

In TensorFlow, tf.GradientTape is an API for automatic differentiation, it computes the gradient of computation with respect to its input variables. Tensorflow records all operations executed inside the context of a tf.GradientTape onto a tape.

import tensorflow as tf

x = tf.constant(3.0)
with tf.GradientTape(persistent=True) as t:
    t.watch(x)               # Ensures that `tensor` is being traced by this tape.
    y = x * x
    z = y * y
dz_dx = t.gradient(z, x)  # 108.0 (4*x^3 at x = 3)
dy_dx = t.gradient(y, x)  # 6.0
print("dz/dx=", dz_dx.numpy())
print("dy/dx=", dy_dx.numpy())
del t  # Drop the reference to the tape

Output:

dz/dx= 108.0
dy/dx= 6.0

We can use the statement tf.function to convert a Python function to an equivalent graph either as a direct call or as a decorator, but in practice, getting tf.function or a graph to work correctly can be tricky. By default, the code it's executed as a graph, so if we want to reproduce an exact Python function, we need to enable eager execution first (available in TensorFlow 2.7).

tf.config.run_functions_eagerly(True)
tf_simple_relu = tf.function(simple_relu)

print("Example 1: ", tf_simple_relu(tf.constant(1)).numpy())
print("Example 2: ", tf_simple_relu(tf.constant(-1)))

tf.config.run_functions_eagerly(False)

There are two ways you could be using preprocessing layers:

1. Make them part of the model, like this:

   inputs = keras.Input(shape=input_shape)
   x = preprocessing_layer(inputs)
   outputs = rest_of_the_model(x)
   model = keras.Model(inputs, outputs)

   With this option, preprocessing will happen on device, synchronously with the rest of the model execution, meaning that it will benefit from GPU acceleration. If you're training on GPU, this is the best option for the Normalization layer, and for all image preprocessing and data augmentation layers.

2. Apply the layer to the tf.data.Dataset, to obtain a dataset that yields batches of preprocessed data, like this:

   dataset = dataset.map(lambda x, y: (preprocessing_layer(x), y))

   With this option, your preprocessing will happen on CPU, asynchronously, and will be buffered before going into the model. In addition, if you prefetch the dataset, the preprocessing will happen efficiently in parallel with training.

In TensorFlow 2.x you can use GradientTape to achieve this. GradientTape records the gradients of any computation that happens in the context of that. Below is an example of how we might do that.

import tensorflow as tf

# Here goes the neural network weights as tf.Variable
x = tf.Variable(3.0)

# TensorFlow operations executed within the context of
# a GradientTape are  recorded for differentiation 
with tf.GradientTape() as tape:
  # Doing the computation in the context of the gradient tape
  # For example computing loss
  y = x ** 2 

# Getting the gradient of network weights w.r.t. loss
dy_dx = tape.gradient(y, x) 
print(dy_dx)  # Returns 6

• The tf.data.Dataset.cache transformation can cache a dataset, either in memory or on local storage. This will save some operations (like file opening and data reading) from being executed during each epoch. The next epochs will reuse the data cached by the cache transformation.

• Prefetch overlaps the preprocessing and model execution of a training step. While the model is executing training step s, the input pipeline is reading the data for step s+1. Doing so reduces the step time to the maximum (as opposed to the sum) of the training and the time it takes to extract the data.

x has a shape of (2, 3) (two rows and three columns):

1 1 1
1 1 1

• reduce_sum computes the sum of elements across specified dimensions of a tensor and by this, it reduces the input tensor.

• By doing tf.reduce_sum(x, 0) the tensor is reduced along the first dimension (rows), so the result is [1, 1, 1] + [1, 1, 1] = [2, 2, 2].

• By doing tf.reduce_sum(x, 1) the tensor is reduced along the second dimension (columns), so the result is [1, 1] + [1, 1] + [1, 1] = [3, 3].

• By doing tf.reduce_sum(x, [0, 1]) the tensor is reduced along BOTH dimensions (rows and columns), so the result is 1 + 1 + 1 + 1 + 1 + 1 = 6 or, equivalently, [1, 1, 1] + [1, 1, 1] = [2, 2, 2], and then 2 + 2 + 2 = 6 (reduce along rows, then reduce the resulted array).

import tensorflow as tf
x = tf.constant([[1, 1, 1], [1, 1, 1]])

print(tf.reduce_sum(x, 0) )
# tf.Tensor([2 2 2], shape=(3,), dtype=int32)

print(tf.reduce_sum(x, 1)  )
# tf.Tensor([3 3], shape=(2,), dtype=int32)

print(tf.reduce_sum(x, [0, 1]))
# tf.Tensor(6, shape=(), dtype=int32)

Imagine, you have a dataset: [1, 2, 3, 4, 5, 6], then:

• dataset.shuffle(buffer_size=n) will allocate a buffer of size n for picking random entries. This buffer will be connected to the source dataset; if n = 3, we could imagine it like this:

  Random buffer
    |
    |   Source dataset where all other elements live
    |         |
    ↓         ↓
  [1,2,3] <= [4,5,6]

  Let's assume that entry 2 was taken from the random buffer. Free space is filled by the next element from the source buffer, that is 4:

  2 <= [1,3,4] <= [5,6]

  We continue reading till nothing is left:

  1 <= [3,4,5] <= [6]
  5 <= [3,4,6] <= []
  3 <= [4,6]   <= []
  6 <= [4]      <= []
  4 <= []      <= []

• dataset.repeat(count=0) will repeat the dataset count number of times. In the present example, as soon as all the entries are read from the dataset and you try to read the next element, the dataset will throw an error. That's where dataset.repeat() comes into play. It will re-initialize the dataset, making it again like this:

  [1,2,3] <= [4,5,6]

• dataset.batch(batch_size) will take first batch_size entries and make a batch out of them. So, batch size of 3 for our example dataset will produce two batch records:

  [2,1,5]
  [3,6,4]

A Feature Cross is a synthetic feature formed by multiplying (crossing) two or more features. Crossing combinations of features can provide predictive abilities beyond what those features can provide individually.

If you suspect that two (or more) features are more meaningful when used jointly, then you can create a crossed column. A common use case for crossed columns is to cross latitude and longitude into a single categorical feature: you start by bucketing the latitude and longitude, for example into 20 buckets each, then you cross these bucketed features into a location column. This will create a 20×20 grid over some city, and each cell in the grid will correspond to one category:

import tensorflow as tf

latitude = tf.feature_column.numeric_column("latitude")
longitude = tf.feature_column.numeric_column("longitude")
bucketized_latitude = tf.feature_column.bucketized_column(latitude, 
  boundaries=list(np.linspace(32., 42., 20 - 1)))
bucketized_longitude = tf.feature_column.bucketized_column(longitude, 
  boundaries=list(np.linspace(-125., -114., 20 - 1)))
location = tf.feature_column.crossed_column( 
  [bucketized_latitude, bucketized_longitude], hash_bucket_size=1000)

Note that crossed_column does not build the full table of all possible combinations (which could be very large). Instead, it is backed by a hashed_column, so you can choose how large the table is.

The main advantage is in domains where you can't fit all of your data into memory. However, there are performance improvements even in cases where all the data fit into memory. Two reasons contribute to this are because in tensorflow.data.Dataset there are two relevant methods:

• One is cache(), where some operations (like file opening and data reading) will be cached and performed only in the first epoch. This will save such operations from being executed during each epoch reducing execution time.

• Another one is prefetch(): while the model is being trained on a batch in the GPU, the CPU loads and prepares the next batch. This can help save a lot of time.

Some other capabilities are allowing for the vectorization of user-defined functions (e.g. for data augmentation) and their parallelization.

• tf.get_variable gets an existing variable with specified parameters from the graph, and if it doesn't exist, creates a new one. This feature it will make it way easier to refactor your code if you need to share variables at any time, e.g. in a multi-gpu setting.
• tf.Variable will always create a new variable and requires that an initial value to be specified. Is said to be is lower-level: at some point tf.get_variable() did not exist so some code still uses the low-level way.

Let's begin with a short introduction to variable sharing: it is a mechanism in TensorFlow that allows for sharing variables accessed in different parts of the code without passing references to the variable around.

The method tf.get_variable can be used with the name of the variable as the argument to either create a new variable with such name or retrieve the one that was created before. This is different from using the tf.Variable constructor which will create a new variable every time it is called (and potentially add a suffix to the variable name if a variable with such name already exists).

It is for the purpose of the variable sharing mechanism that a separate type of scope (variable scope) was introduced. As a result, we end up having two different types of scopes:

• name scope, created using tf.name_scope.
• variable scope, created using tf.variable_scope.

Both scopes have the same effect on all operations as well as variables created using tf.Variable, i.e., the scope will be added as a prefix to the operation or variable name. However, their difference relies in that name scope is ignored by tf.get_variable.

In other words, tf.variable_scope() adds a prefix to the names of all ops and variables (no matter if you create them with tf.get_variable or tf.Variable), but tf.name_scope() ignores variables created with tf.get_variable() because it assumes that you know which variable and in which scope you wanted to use.

Often, you don’t want to feed a number directly into the model, but instead, split its value into different categories based on numerical ranges. For example, consider raw data that represents the year a house was built. Instead of representing that year as a scalar numeric column, we could split the year into the following four buckets:

The model will represent the buckets as follows:

Notice that the categorization splits a single input number into a four-element vector. Therefore, the model now can learn four individual weights rather than just one; four weights create a richer model than one weight. More importantly, bucketing enables the model to clearly distinguish between different year categories since only one of the elements is set (1) and the other three elements are cleared (0).

For example, when we just use a single number (a year) as input, a linear model can only learn a linear relationship. So, bucketing provides the model with additional flexibility that the model can use to learn.

The following code demonstrates how to create a bucketed feature:

import pandas as pd
import tensorflow as tf
from tensorflow import feature_column
from tensorflow.keras import layers

data = {'years': [1955,1921,1963,1988,1974,1954,1995,1941,1984,1952]}
df = pd.DataFrame(data)

years = feature_column.numeric_column("years")
marks_buckets = feature_column.bucketized_column(years, boundaries=[1930,1940,1950,1960,1970,1980,1990])

# Show marks_buckets
feature_layer = layers.DenseFeatures(marks_buckets)
print(feature_layer(data).numpy())

[[0. 0. 0. 1. 0. 0. 0. 0.]
 [1. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 1. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 1. 0.]
 [0. 0. 0. 0. 0. 1. 0. 0.]
 [0. 0. 0. 1. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 1.]
 [0. 0. 1. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 1. 0.]
 [0. 0. 0. 1. 0. 0. 0. 0.]]