30 PyTorch Interview Questions (ANSWERED) To Beat Your Next Machine Learning Interview

PyTorch is an open-source machine learning library used for developing and training neural network-based deep learning models which is a type of machine learning that’s commonly used in applications like image recognition and language processing. It is primarily developed by Facebook’s AI research group. PyTorch can be used with Python as well as a C++.

PyTorch is distinctive for its excellent support for GPUs and its use of reverse-mode auto-differentiation, which enables computation graphs to be modified on the fly. This makes it a popular choice for fast experimentation and prototyping.

PyTorch adopted a Chainer innovation called reverse-mode automatic differentiation. Essentially, it’s like a tape recorder that records completed operations and then replays backward to compute gradients. This makes PyTorch relatively simple to debug and well-adapted to certain applications such as dynamic neural networks. It’s popular for prototyping because every iteration can be different.

1. Dynamic vs Static: Though both PyTorch and TensorFlow work on tensors, the primary difference between PyTorch and Tensorflow is that while PyTorch uses dynamic computation graphs, TensorFlow uses static computation graphs. That being said, with the release of TensorFlow 2.0 there has been a major shift towards eager execution, and away from static graph computation. Eager execution in TensorFlow 2.0 evaluates operations immediately, without building graphs.
2. Data Parallelism: PyTorch uses asynchronous execution of Python to implement data parallelism, but with TensorFlow this is not the case. With TensorFlow you need to manually configure every operation for data parallelism.
3. Visualization Support: TensorFlow has a very good visualization library called TensorBoard. This visualization support helps developers to track the model training process nicely. PyTorch initially had a visualization library called Visdom, but has since provided full support for TensorBoard as well. PyTorch users can utilize TensorBoard to log PyTorch models and metrics within the TensorBoard UI. Scalars, images, histograms, graphs, and embedding visualizations are all supported for PyTorch models and tensors.
4. Model Deployment: TensorFlow has great support for deploying models using a framework called TensorFlow serving. It is a framework that uses REST Client API for using the model for prediction once deployed. On the other hand, PyTorch does not provide a framework like serving to deploy models onto the web using REST Client.

The derivatives of the function are calculated with the help of the gradient. There are four simple steps through which we can calculate derivative easily:

1. Initialize the function for which we will calculate the derivatives.
2. Set the value of the variable which is used in the function with requires_grad=True to store the gradient in the grad attribute.
3. Compute the derivative of the function by using the backward() method.
4. Print the value of the derivative using the grad attribute.

# Step 1
y = 4 * x + 3
# Step 2
x = torch.autograd.Variable(torch.Tensor([1.0]),requires_grad=True)
# Step 3
y.backward()
# Step 4 
print(x.grad)

The basic steps to define a neural network in PyTorch are:

1. Define the architecture and initialize the neural network: this step involves defining a class that extends the Module class. On it, we define the model elements such as linear layers, regularization layers, activation functions, etc.

2. Define the feed-forward algorithm : the feed-forward algorithm would be encapsulated by the forward function whose goal is to pass the input data into the computation graph (i.e. our neural network).

3. As a last step, you can test your model by instantiating it and passing data through it.

# Step 1: Model definition
class Net(nn.Module):
    # Define model elements
    def __init__(self, n_inputs):
        super(Net, self).__init__()
        self.layer = Linear(n_inputs, 1)
        self.activation = Sigmoid()

    # Step 2: Forward propagate input
    def forward(self, X):
        X = self.layer(X)
        X = self.activation(X)
        return X

# Step 3: Test model
my_nn = Net()
result = my_nn(some_data)
print (result)

The five different components of PyTorch:

• Tensors: Tensors are very homogeneous to the Numpy array and it is also multi-dimensional. The Tensors are accessible in PyTorch as a torch. Some examples are torch.CharTen, torch.IntTesnor, torch.FloatTensor, etc.
• Variable: A variable works as a wrapper around the Tensor to clutch the gradient. You can find variables under the torch.autograd in the form of a torch.autograd.Variable.
• Parameters: The work of a Parameter is to wrap the variable and we use it when the Tensors of a module do not possess a gradient. We can find parameters under the torch.nn in the form of torch.nn.Parameter.
• Functions: Functions do not possess any memory and their work is to transform the operations. Some examples of function are torch.sum, torch.log, etc. Functions are implemented using torch.nn.functional.
• Modules: Modules are the base class of all neural networks and they also can contain different functions, modules, and parameters. It is efficient in storing learnable weights and states. Modules can be applied as torch.nn.Linear, torch.nn.Conv2d, etc.

Tensors are a core PyTorch data type, similar to a multidimensional array, used to store and manipulate the inputs and outputs of a model, as well as the model’s parameters. Tensors are similar to NumPy’s ndarrays, except that tensors can run on GPUs to accelerate computing.

Although completely reproducible results are not guaranteed across PyTorch releases, there are some steps you can take to limit the number of sources of nondeterministic behavior and increase reproducibility, such as:

• Controlling the sources of randomness by setting a Random number generator (RNG): One can use torch.manual_seed() at the beginning of an application to seed the RNG for all devices (both CPU and CUDA). This allows to generate the same series of random numbers each time the application is run in the same environment.

• Avoiding nondeterministic algorithms for some operations: using the function torch.use_deterministic_algorithms() lets you configure PyTorch to use deterministic algorithms instead of nondeterministic ones where available, and to throw an error if an operation is known to be nondeterministic (and without a deterministic alternative).

There are multiple methods to reshape tensor dimensions in PyTorch, some of which are:

• torch.reshape(input, shape): Reshapes input to shape (if compatible).
• torch.Tensor.view(shape): Returns a view of the original tensor in a different shape but shares the same data as the original tensor.
• torch.permute(input, dims): Returns a view of the original input with its dimensions rearranged to dims.

Tensor and matrix multiplications are essential for neural networks. Quite often, we have an input vector, which is transformed using a learned weight matrix. Depending on the best use case, there are multiple ways and functions to perform matrix multiplication, some of which are:

• torch.matmul: Performs the matrix product over two tensors, where the specific behavior depends on the dimensions. If both inputs are matrices (2-dimensional tensors), it performs the standard matrix product. For higher dimensional inputs, the function supports broadcasting.

• torch.mm: Performs the matrix product over two matrices, but doesn’t support broadcasting.

• torch.bmm: Performs the matrix product with a support batch dimension. For example, if the first tensor T is of shape (b ⨯ n ⨯ m), and the second tensor (b ⨯ m ⨯ p), the output O is of shape (b ⨯ n ⨯ p), and has been calculated by performing b matrix multiplications of the submatrices of T and R.

In a typical training loop, the following operations are implemented for each batch during each epoch:

1. Gets a batch of training data.
2. Zeros the optimizer’s gradients.
3. Performs an inference - that is, gets predictions from the model for an input batch
4. Calculates the loss for that set of predictions vs. the labels on the dataset
5. Calculates the backward gradients over the learning weights - that is, adjust the model’s learning weights based on the observed gradients for this batch, according to the optimization algorithm we chose.
6. Store the loss for each training step to be used later for reporting.

running_loss = 0.

for i, data in enumerate(training_batch_set):

    # 1. Every data instance is an input + label pair
    inputs, labels = data

    # 2. Zero your gradients for every batch!
    optimizer.zero_grad()

    # 3. Make predictions for this batch
    outputs = model(inputs)

    # 4. Compute the loss and its gradients
    loss = loss_fn(outputs, labels)
    loss.backward()

    # 5. Adjust learning weights
    optimizer.step()

    # 6. Gather data and for report
    running_loss += loss.item()

The most common errors when working with PyTorch are:

• Shape errors: occur when you're trying to operate on matrices/tensors with shapes that don't line up. For example, your data's shape is [1, 28, 28] but your first layer takes an input of [10]. Depending on the context, one way to solve it is by reshaping or transposing the tensor such as the shapes line up with the desired operation.

• Device errors: occur when your model/data are on different devices, such as when you've sent your model to the target GPU device but your data is still on the CPU. So the straightforward way to solve this issue is sending to model/data to the properly targeted device, this can be done with the method .to().

• Datatype errors: occurs when the data is one datatype (e.g. torch.float32), and you're trying to perform requires another datatype (e.g. torch.int64). To solve this, we just adjust the tensor properly with torch.Tensor.type(dtype=None) where the dtype parameter is the datatype you'd like to use.

• In PyTorch, the forward() method is used to compute the forward pass of the neural network model which defines how your model is going to be run, from input to output.

• The backward function computes the gradient of the current Tensors with respect to some scalar value. In the neural network model context, backward() computes the gradients of the model's parameters with respect to a given loss function.

In summary, the forward() method specifies how the model runs, while the backward() method calculates gradients that are used to optimize the model's parameters.

1. Define the layers to be frozen: Here we can use model.state_dict() to get the key information of all parameters and we can print it out to help us figure out which layers we want to freeze, or we can identify the parameter by name with model.named_parameters().

2. After identifying the layers, we simply set the requires_grad to False in such parameters to freeze them.

For example, suppose we want to freeze the layer containing the name of "fc1". Then the implementation would be

for name, param in model.named_parameters():
    if param.requires_grad and 'fc1' in name:
        param.requires_grad = False

One can change the learning rate as the training progress using the learning rate schedules which adjusts the learning rate according to some specified rule.

One of the most popular learning rate schedulers is a step decay (StepLR) where the learning rate is reduced by some percentage after a set number of training epochs. The steps to implement this change in the learning rate are:

1. Define the rules for changing the learning rate, i.e. define in how many epochs (step_size) the change should occur and in what percentage (gamma).

2. In the training loop, place the scheduler after updating model weights, in such a way, the new learning rate will take effect in the next iteration.

# Step 1
scheduler = StepLR(optimizer, step_size=5, gamma=0.1)


for epoch in range(20):
    for input, target in dataset:
        optimizer.zero_grad()
        output = model(input)
        loss = loss_fn(output, target)
        loss.backward()
        optimizer.step()
    # Step 2
    scheduler.step()

In PyTorch, custom loss functions can be implemented by creating a subclass of the nn.Module class and overriding the forward method. The forward method takes as input the predicted output and the actual output and returns the value of the loss.

import torch
import torch.nn as nn

# define your custom loss function
class CustomLoss(nn.Module):
    def __init__(self):
        super(MyLoss, self).__init__()
    
    def forward(self, output, target):
        # compute your custom loss function
        loss = ... # compute the loss
        
        return loss

Now, to use the custom loss function, we need to instantiate it and pass it as the argument to the criterion parameter of the optimizer in the training loop.

# Initialize your model and loss function
model = ...
optimizer = ...
criterion = CustomLoss()

# Training loop
for epoch in range(num_epochs):

        optimizer.zero_grad()
        outputs = model(data)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

# Rest of the code to evaluate the model goes here...	

• The first thing to do is to verify that you have the required CUDA libraries and NVIDIA drivers, and that you have an available GPU to work with. One can check such requirements with torch.cuda.is_available().

• Assuming a positive response to previous requirements, the next thing to do is move the existing tensors to the GPU, this can be done in either three ways:

  • Moving tensors with the to() function: Every Tensor in PyTorch has a to() member function. Its job is to put the tensor to the specified device. The options for device are: cpu and cuda:0 for putting it on GPU number 0. Similarly, if your system has multiple GPUs, the number would be the GPU you want to put tensors on.

  • Moving tensors with the cuda() function: Another way to put tensors on GPUs is to call cuda(n) a function on them where n is the index of the GPU.

  • Moving the NN model with to() function: The torch.nn.Module class also has the function to() which put the entire network on a particular device just to call the function on the nn.Modue.

• Another way to is to automatically the tensors to the GPU device with torch.cuda.set_device(n) with n as the index of the GPU. This eliminates the necessity of passing one by one all tensors to the GPU.

Graphs are data structures consisting of connected nodes (called vertices) and edges. Every modern framework for deep learning is based on the concept of graphs, where Neural Networks are represented as a graph structure of computations. PyTorch keeps a record of tensors and executed operations in a directed acyclic graph (DAG) consisting of Function objects. In this DAG, leaves are the input tensors, roots are the output tensors.

In many popular frameworks, including TensorFlow, the computation graph is a static object. PyTorch is based on dynamic computation graphs, where the computation graph is built and rebuilt at runtime, with the same code that performs the computations for the forward pass also creating the data structure needed for backpropagation. PyTorch is the first define-by-run deep learning framework that matches the capabilities and performance of static graph frameworks like TensorFlow, making it a good fit for everything from standard convolutional networks to recurrent neural networks.

• The wrapper “with torch.no_grad()” temporarily set the attribute reguireds_grad of tensor False and deactivates the Autograd engine which computes the gradients with respect to parameters.

• This wrapper is recommended to use in the test phase as we don’t need gradients in the test step since the parameter updates have been done in the training step. Using torch.no_grad() in the test and validation phase yields the faster inference(speed up computation) and reduced memory usage(which allows us to use a larger size of batch).

The attribute requires_grad tracks whether or not a tensor should be added to the computational graph generated during runtime in a forward pass in PyTorch. In practice, this translates to the tracking of whether or not the gradients to the tensor need to be computed.

This attribute is useful in cases where you might need to freeze/unfreeze some parts of your neural network and avoid/let some of the parameters be optimized during the training. The "requires_grad" argument provides an easy way to include or exclude your network's parameters in the training phase.

A hook is basically a function that is executed when either forward or backward is called. You can register a hook on a Tensor or a nn.Module. In Tensors hooks are executed after backward is called and it returns a custom operation based on the tensor's grad. Some uses of the functionality of hooks in tensors are:

• Debugging and Logging: You can print the value of the gradient for debugging and you can also log them. This is especially useful with tensor variables that are the result of a differentiable operation(non-leaf variables) whose gradients are freed up unless you call retain_grad() upon them. Doing the latter can lead to increased memory retention. Hooks provide a much cleaner way to aggregate these values.

• Update gradients: Hooks allow you to modify gradients during the backward pass. Without hooks you can still access the grad variable of a tensor in a network, but you can only access it after the entire backward pass has been done. So depending on if it's needed, a hook can be implemented to modify gradients.