23 Computer Vision Interview Questions (ANSWERED) To Nail on ML Interview

We can use Intersection over Union (IoU): an evaluation metric used to measure the accuracy of an object detector on a particular dataset. Any algorithm that provides predicted bounding boxes as output can be evaluated using IoU. R-CNN, Faster R-CNN, YOLO are examples of such models

In order to evaluate the prediction we must have:

• The ground-truth bounding boxes (i.e., the hand-labeled bounding boxes from the testing set that specify wherein the image our object is).
• The predicted bounding boxes from our model.

For example, in the image below the predicted bounding box is drawn in red while the ground-truth bounding box is drawn in green. The performance is the Intersection over Union between these bounding boxes.

Examining this equation you can see that Intersection over Union is simply a ratio:

• If IoU is close to 1 then we can say that our model perfectly anticipated the object.
• If IoU is close to 0 or is 0 then we can say that our model didn’t predict object coordinates at all.
• An IoU greater than 0.5 is normally considered a good prediction.

Many vision applications follow the flow of acquiring images and data, processing that data, performing some analysis and recognition steps, and then finally making a prediction based on the extracted information:

Let's illustrate this pipeline with an example:

1. Input image: a computer receives a visual input from an imaging device like a camera. This is typically captured as an image or a sequence of images forming a video.
2. Pre-processing: Each image is sent through some pre-processing steps whose purpose is to standardize each image. Common preprocessing steps include resizing an image, blurring, rotating, changing its shape, or transforming the image from one color to another. Only by standardizing each image, can you then compare them and further analyze them in the same way.
3. Next, comes Feature extraction. Features are what help us define certain objects, and they’re usually information about object shape or color. For example, some features that distinguish the shape of a motorcycle’s wheel, headlights, mudguards, and so on. The output of this process is a features vector, which is a list of unique shapes that identify the object.
4. Finally, these features are fed into an ML model, in the example, a classification model. This step looks at the features vector from the previous step and predicts the class of the image. As you can see in the above example, the dog has the lowest probability of 1% whereas there’s an 85% probability that this is a motorcycle.

To improve the accuracy, we may need to do more of step 1 (acquire more training images) or step 2 (more processing to remove noise) or step 3 (extract better features) or step 4 (change the classifier algorithm and tune some hyperparameters or even more training time). Many different approaches can improve the performance of the model. They all lie in one or more of the pipeline steps.

• Semantic Segmentation is a technique that detects, for each pixel, the object category to which it belongs to. All object categories (labels) must be known to the model.
• Instance Segmentation, is similar to Semantic Segmentation, but dives a bit deeper. It identifies for each pixel, the object instance it belongs to. The main difference is that differentiates two objects with the same labels in comparison to semantic segmentation. For example:

• In the second image where Semantic Segmentation is applied, the category (chair) is one of the outputs, all chairs are colored the same.

• In the third image, the Instance Segmentation, goes a step further and separates the instances (the chairs) from one another apart from identifying the category (chair) in the first step.

• In the beginning, neural networks scoop up all the features available and give them random weights.
• During the training process, the neural network adjusts these weights to reflect their importance and how they should impact the output prediction.
• The patterns within the image with the highest appearance frequency will have higher weights and are considered more useful features.
• Features with the lowest weights will have very little impact on the output.

• In the image, if W1 > W2 during the training process, X1 then will be considered a more useful feature than X2, and so on.

Image Registration is the process of transforming different sets of data into one coordinate system. The idea is to align two or more images of the same scene. Data may be multiple photographs, data from different sensors, times, depths, or viewpoints.

Usually, one of the images is referred to as the moving or source, and the others are referred to as the target, fixed or sensed images. Image registration involves spatially transforming the source/moving image(s) to align with the target image. The reference frame in the target image is stationary, while the other datasets are transformed to match the target.

This technique is often used in medical and satellite imagery to align images from different camera sources. Digital cameras use image registration to align and connect adjacent images into a single panoramic image.

The main idea to detect edges in an image is by detecting Discontinuities in Brightness. A change in brightness results from discontinuity in depth, orientation, illumination, or corners. Those discontinuities are calculated following edge detection methods that can be based on the first or second-order:

where f is the image function. For example, in an image where the intensity of the image changes from dark to bright around the midway point, the edge of the image is at the middle point:

• The first derivative (the intensity gradient) goes up and then down at the midway point, so the edge detection can be calculated by looking at the maximum value of the first derivative.

• However, the problem with the first derivative method is, depending on the input function, the maximum value can change, so the threshold value of the maximum value cannot be predetermined. However, the second derivative, as shown, always goes through zero points at the edges.

• Sobel and Canny are the first-order edge detection methods, while the second-order method is a Laplacian edge detector.

In general, converting color images to grayscale reduces Computation Complexity. For example, suppose you want to convert your colored images to grayscale because, for many objects, color is not necessary to recognize and interpret an image. Grayscale can be good enough for recognizing certain objects. Since color images contain more information than black-and-white images, they can add unnecessary complexity and take up more space in memory.

However, converting an image to grayscale might not be a good decision for some problems. There are several applications for which color is very important: for example, building a diagnostic system to identify red skin rashes in medical images. This application relies heavily on the intensity of the red color in the skin. Removing colors from the image will make it harder to solve this problem. In general, color images provide very helpful information in many medical applications.

Another example of the importance of color in images is lane-detection applications in a self-driving car, where the car has to identify the difference between yellow and white lines because they are treated differently. Grayscale images do not provide enough information to distinguish between the yellow and white lines.

The rule of thumb to identify the importance of colors in your problem is to look at the image with the human eye. If you are able to identify the object you are looking for in a grey image, then you probably have enough information to feed to your model. If not, then you definitely need more information (colors) for your model.

The main idea is to divide the image into regions or specific areas and then classify each one of them. For this, a rectangular box slides through the entire image. The box is of fixed length and width with a stride to move over the entire image. Consider the following example in which we want to detect a vacuum cleaner:

• We use a sliding window at each part of the image. The red box is sliding over the entire image of the vacuum cleaner: From left to right and then vertically.
• Then, for each of these regions cropped, we can classify whether this region contains an object that interests us or not.
• Finally, we increase the size of the sliding window and continue the process.
• So, the output and the classification are dependent on the size of the window.
• As we can see from the image below, different parts of the image are becoming the point of observation, so this is a computationally expensive technique and will be slow to implement as we are classifying all the regions in an image.

• Gaussian Blur (also known as Gaussian smoothing): is the result of blurring an image by a Gaussian function. It is a widely used effect in graphics software, typically to reduce image noise and reduce detail.

• Mean Filter: is a simple sliding window that replaces the center value with the average of all pixel values in the window. This process is repeated for all pixel values in the image. By doing this, it replaces pixels that are unrepresentative of their surroundings. The window or kernel is usually a square but it can be of any shape.

• Median Filter: is similar to the mean filter, but here do not replace the pixel value of the image with the mean of all neighbouring pixel values; we replace it with the median value. Median filtering is done by, first sorting all the pixel values from the surrounding neighbourhood into numerical order and then replacing the pixel being considered with the middle pixel value.

• Bilateral Filter: uses Gaussian Filter with an additional one more multiplicative component which is a function of pixel intensity difference. It ensures that only pixel intensity similar to that of the central pixel is included in computing the blurred intensity value. This filter preserves edges.

• Gaussian Image Processing: Gaussian blur which is also known as Gaussian smoothing, is the result of blurring an image by a Gaussian function. It is used to reduce image noise and reduce details. The visual effect of this blurring technique is similar to looking at an image through a translucent screen.

• Edge Detection: is used to find the boundaries of objects within images. It works by detecting discontinuities in brightness. This could be very beneficial in extracting useful information from the image because most of the shape information is enclosed in the edges. The most common edge detection algorithm is Sobel edge detection, made up of 3x3 convolutional kernels.

• Fourier Transform: breaks down an image into sine and cosine components. is used if we want to access the geometric characteristics of a spatial domain image. Because the image in the Fourier domain is decomposed into its sinusoidal components, it is easy to examine or process certain frequencies of the image, thus influencing the geometric structure in the spatial domain. It has multiple applications like image reconstruction, image compression, or image filtering.

• Wavelet Image Processing: It's an extension of the Fourier transform algorithm designed in such a way that we get good frequency resolution for low-frequency components. Below is the 2D wavelet transform example:

Thresholding is used to create a binary image from a grayscale image. It is the simplest way to segment objects from a background. Some thresholding methods are:

• Simple Thresholding: Here, the matter is straightforward. For every pixel, the same arbitrary threshold value is applied. If the pixel value is smaller than the threshold, it is set to 0, otherwise, it is set to a maximum value.

• Adaptive Thresholding: In simple thresholding, one global value is used as a threshold. But this might not be good in all cases, e.g. if an image has different lighting conditions in different areas. In that case, adaptive thresholding can help. Here, the algorithm determines the threshold for a pixel-based on a small region around it. So we get different thresholds for different regions of the same image which gives better results for images with varying illumination.

• Otsu's Binarization: avoids having to choose an arbitrary threshold value by determining it automatically. For example, consider an image with only two distinct image values (bimodal image), then the histogram would only consist of two peaks. A good threshold would be in the middle of those two values and this is what Otsu's Binarization does: determines an optimal global threshold value from the image histogram.

Morphological Transformations are some simple operations based on the image shape. It is normally performed on binary images. It needs two inputs, one is our original image, and the second one is called structuring element or kernel which decides the nature of the operation. Two basic morphological operators are Erosion and Dilation. Then its variant forms like Opening, Closing, etc also come into play.

Let's take for example the following input image:

• Erosion: The basic idea of erosion is just like soil erosion only, it erodes the boundaries of the foreground object, so in erosion we remove pixels on object boundaries. The way this works is with a kernel sliding through an image (as in 2D convolution). A pixel in the original image (either 1 or 0) will be considered 1 only if all the pixels under the kernel are 1, otherwise, it is eroded (made to zero). So what happens is that all the pixels near the boundary will be discarded depending on the size of the kernel.

• Dilation: It is just the opposite of erosion, i.e. adds pixels to the boundaries of objects in an image. Here, a pixel element is '1' if at least one pixel under the kernel is '1'. So it increases the image or size of the foreground object increases.

• Opening is just another name for erosion followed by dilation. Normally, in cases like noise removal, this method is used. Because, erosion removes white noises, but also shrinks our objects. So we dilate it. Since noise is gone, they won't come back, but our object area increases. It is also useful in joining broken parts of an object.

• Closing is the reverse of Opening: Dilation followed by Erosion. It is useful in closing small holes inside the foreground objects, or small black points on the object.

• Feature Detection selects regions of an image that have unique content, such as corners or blobs. They are helpful to find points of interest for further processing. These points do not necessarily correspond to physical structures, such as the corners of a table. The key to feature detection is to find features that remain locally invariant so that you can detect them even in the presence of rotation or scale change.

• Feature Extraction involves computing a descriptor, which is typically done on regions centered around detected features. Descriptors rely on image processing to transform a local pixel neighbourhood into a compact vector representation. This new representation permits comparison between neighbourhoods regardless of changes in scale or orientation.

• So, Feature Detection allows us to obtain regions that are locally invariant to rotation or scale changes while Feature Extraction allows us to compare features regardless of such transformations.

When we face this problem we could do Image Augmentation, which is a process of creating new training examples from the existing ones. To make a new sample, you slightly change the original image. For instance, you could make a new image a little brighter; you could cut a piece from the original image; you could make a new image by mirroring the original one, etc. Here are some examples of transformations of the original image that will create a new training sample.

By applying those transformations to the original training dataset, you could create an almost infinite amount of new training samples.

This technique also helps to fight overfitting and improve the performance of deep neural networks for more computer vision tasks such as classification, segmentation, and object detection.

• Linear Filters remove noise by convolving the original image with a mask that represents a low-pass filter or smoothing operation. The output of a linear operation due to the sum of two inputs is the same as performing the operation on the inputs individually and then summing the results. These filters also tend to blur the sharp edgesand destroy the lines and other fine details of the image. Linear methods are fast but they do not preserve the details of the image.

• Non-linear Filters are filters whose outputs are not a linear function of their inputs. They preserve the details of the image and have many applications, especially removal of certain types of noise that are not additive (like Gaussian or Salt-Pepper type of noises). Non-linear filters are considerably harder to use and design than linear ones. The image below is an example of the median filter, a type of non-linear filter.

Sampling:

• Is the process that will determine the size of the picture to be obtained. It deals with the number of pixels of the digital picture, that is, the matrix size.
• Is a spatial process, because it deals with the size of the picture, completely independent of the content of the picture or the values it will have.

• In other words, sampling deals with the coordinate of an analogue image. For example, in case of equation y = sin(x), sampling is done on x variable.

• When looking at the image above, we can see there are some random variations in the signal caused by noise. In sampling, we reduce this noise by taking samples. The more samples we take, the quality of the image would be better.

Quantization:

• Is the process that will determine the value of each pixel in the image.
• It can be seen as the opposite of sampling because it is done on the “y-axis” while sampling is done on the “x-axis”. This is, the amplitude values of the image are digitized.
• The idea is to transform a real-valued sampled image into one taking only a finite number of distinct values. In simple words, when you are quantizing an image, you are dividing the amplitude of a signal into quanta(partitions).

• In the image shown below, quantization assign levels to the values generated by the sampling process. These vertically ranging values have been quantized into 5 different levels or partitions. Ranging from 0 black to 4 white. This level could vary according to the type of image you want.

A Homography is a type of projective transformation in which we take advantage of projections to relate two images. They are studied by examining the fact that cameras pick up different images depending on their position and orientation. In essence, a homography is a transformation between two images of the same scene, but from a different perspective. There are two only cases for which homography applies (both cases assume that the world view can be modelled by plane):

• Images are captured by the same camera but at a different angle (the world is now essentially a plane).

• Two cameras are viewing the same plane from a different location.

If we have a case in which whether of those criteria are met, we can use this technique and the result will allow us to better understand how images change when we look at them from a different perspective. They are a powerful tool to project certain images onto the environment and combine multiple images to create a larger panorama.