27 Advanced NumPy Interview Questions (ANSWERED) To Crush Your Data Science Interview

NumPy is the fundamental package for scientific computing in Python. It is a Python library that provides:

• a multidimensional array object,
• various derived objects (such as masked arrays and matrices), and
• an assortment of routines for fast operations on arrays, including
  • mathematical,
  • logical,
  • shape manipulation,
  • sorting,
  • selecting,
  • I/O,
  • discrete Fourier transforms,
  • basic linear algebra,
  • basic statistical operations,
  • random simulation and much more.

apply_along_axis applies the supplied function along with 1D slices of the input array, with the slices taken along the axis you specify.

Consider:

# Input
np.random.seed(100)
a = np.random.randint(1,10, [5,3])
a

# Solution
np.apply_along_axis(lambda x: np.min(x)/np.max(x), arr=a, axis=1)
#> array([ 0.44444444,  0.125     ,  0.5       ,  1.        ,  0.11111111])

At the core of the NumPy package, is the ndarray object. This encapsulates n-dimensional arrays of homogeneous data types, with many operations being performed in compiled code for performance. There are several important differences between NumPy arrays and the standard Python sequences:

• NumPy arrays have a fixed size at creation, unlike Python lists (which can grow dynamically). Changing the size of an ndarray will create a new array and delete the original.
• The elements in a NumPy array are all required to be of the same data type, and thus will be the same size in memory. The exception: one can have arrays of (Python, including NumPy) objects, thereby allowing for arrays of different-sized elements.
• NumPy arrays facilitate advanced mathematical and other types of operations on large numbers of data. Typically, such operations are executed more efficiently and with less code than is possible using Python’s built-in sequences.
• A growing plethora of scientific and mathematical Python-based packages are using NumPy arrays; though these typically support Python-sequence input, they convert such input to NumPy arrays prior to processing, and they often output NumPy arrays. In other words, in order to efficiently use much (perhaps even most) of today’s scientific/mathematical Python-based software, just knowing how to use Python’s built-in sequence types is insufficient - one also needs to know how to use NumPy arrays.

You can use a list comprehension:

indices = [i for i, x in enumerate(my_list) if x == "whatever"]

The iterator enumerate(my_list) yields pairs (index, item) for each item in the list. Using i, x as loop variable target unpacks these pairs into the index i and the list item x. We filter down to all x that match our criterion and select the indices i of these elements.

And for arrays we can use np.where:

import numpy as np
values = np.array([1,2,3,1,2,4,5,6,3,2,1])
searchval = 3
ii = np.where(values == searchval)[0]
# returns:
# ii ==>array([2, 8])

This can be significantly faster for lists (arrays) with a large number of elements vs some of the other solutions.

Use to_numpy(), which is defined on Index, Series, and DataFrame objects.

Consider:

# Setup
df = pd.DataFrame(data={'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]}, 
                  index=['a', 'b', 'c'])

# Convert the entire DataFrame
df.to_numpy()
# array([[1, 4, 7],
#        [2, 5, 8],
#        [3, 6, 9]])

# Convert specific columns
df[['A', 'C']].to_numpy()
# array([[1, 7],
#        [2, 8],
#        [3, 9]])

Other solutions you shall avoid:

• DataFrame.values has inconsistent behaviour. With .values it was unclear whether the returned value would be the actual array, some transformation of it, or one of pandas custom arrays (like Categorical). For example, with PeriodIndex, .values generates a new ndarray of period objects each time.
• DataFrame.get_values() is simply a wrapper around DataFrame.values, so everything said above applies.
• DataFrame.as_matrix() is deprecated now, do NOT use!

numpy.array is just a convenience function to create an ndarray; it is not a class itself.

You can also create an array using numpy.ndarray, but it is not the recommended way.

If you want to create an array from ndarray class you can do it in 2 ways as quoted:

1. Using array(), zeros() or empty() methods: Arrays should be constructed using array, zeros or empty (refer to the See Also section below). The parameters are given here refer to a low-level method (ndarray(…)) for instantiating an array.

2. From ndarray class directly: There are two modes of creating an array using __new__: If buffer is None, then only shape, dtype, and order are used. If buffer is an object exposing the buffer interface, then all keywords are interpreted.

Your array a defines the columns of the nonzero elements in the output array. You need to also define the rows and then use fancy indexing:

>>> a = np.array([1, 0, 3])
>>> b = np.zeros((a.size, a.max()+1))
>>> b[np.arange(a.size),a] = 1
>>> b
array([[ 0.,  1.,  0.,  0.],
       [ 1.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  1.]])

Vectorization takes one elemental operation and applies to all the elements in that array for you. Underneath, the implementatiosn are in C, hence providing substantial speed gains. NumPy already has a large number of operations vectorized, for eg: all arithmetic operators, logical operators, etc. Numpy also provides a way for you to vectorize your function. All you need to do is:

• Write a function to do the operation you want to do, taking the elements of the array as arguments.
• Vectorize the function.
• Provide the arrays as inputs to this vectorized function.
• Done

# Creating your own Vectorized function
# Note that the following function would already be vectorized since `+` and `**` operations are vectorized
>>> def add_square(a, b):
...     return (a + b)**2
...
>>> a_array = np.array([1,2,3])
>>> b_array = np.array([4,5,6])
>>> vadd_square = np.vectorize(add_square)
>>> vadd_square(a_array, b_array)
array([25, 49, 81])

Consider:

# Bin petal length 
petal_length_bin = np.digitize(iris[:, 2].astype('float'), [0, 3, 5, 10])

# Map it to respective category
label_map = {1: 'small', 2: 'medium', 3: 'large', 4: np.nan}
petal_length_cat = [label_map[x] for x in petal_length_bin]

# View
petal_length_cat[:4]
<#> ['small', 'small', 'small', 'small']

Consider:

a = np.array([1, 3, 7, 1, 2, 6, 0, 1])
doublediff = np.diff(np.sign(np.diff(a)))
peak_locations = np.where(doublediff == -2)[0] + 1
peak_locations
# array([2, 5])

arr = np.array(some_sequence)
reversed_arr = arr[::-1]

arr[::-1] just returns a reversed view. You can then change the original array, and the view will update to reflect the changes.

Regarding performance, it's a constant-time operation O(1). It's as fast as you can get, doesn't depend on the number of items in the array, and doesn't take long as the array grows.

• Memory efficiency. NumPy's arrays are more compact than Python lists. Access to reading and writing items is also faster with NumPy.

  More detailed: A Python list is an array of pointers to Python objects, at least 4 bytes per pointer plus 16 bytes for even the smallest Python object (4 for type pointer, 4 for reference count, 4 for value -- and the memory allocators rounds up to 16). A NumPy array is an array of uniform values - single-precision numbers take 4 bytes each, double-precision ones, 8 bytes.

• Convenience. You get a lot of vector and matrix operations for free, which sometimes allows one to avoid unnecessary work. And they are also efficiently implemented.

• Speed. A simple test on doing a sum over a list and a NumPy array, showing that the sum on the NumPy array is 10x faster.
• Functionality. You get a lot built-in with NumPy, FFTs, convolutions, fast searching, basic statistics, linear algebra, histograms, etc.

Numpy matrices are strictly 2-dimensional, while NumPy arrays (ndarrays) are N-dimensional. Matrix objects are a subclass of ndarray, so they inherit all the attributes and methods of ndarrays.

The main advantage of NumPy matrices is that they provide a convenient notation for matrix multiplication

In short, use arrays.

As per the official documents, it's no anymore advisable to use matrix class since it will be removed in the future.

• They are the standard vector/matrix/tensor type of NumPy. Many NumPy functions return arrays, not matrices.
• There is a clear distinction between element-wise operations and linear algebra operations.
• You can have standard vectors or row/column vectors if you like.

The only disadvantage of using the array type is that you will have to use dot instead of * to multiply (reduce) two tensors (scalar product, matrix-vector multiplication, etc.).

• np.mean always computes arithmetic mean, and has some additional options for input and output (e.g. what datatypes to use, where to place the result).

• np.average can compute a weighted average if the weights parameter is supplied.

From wiki:

The weighted arithmetic mean is similar to an ordinary arithmetic mean (the most common type of average), except that instead of each of the data points contributing equally to the final average, some data points contribute more than others.

einsum stands for Einstein’s summation. Using this function you can implement a lot of in-built functions involving summation.

Imagine that we have two multi-dimensional arrays, A and B. Now let's suppose we want to:

• multiply A with B in a particular way to create new array of products; and then maybe
• sum this new array along particular axes; and then maybe
• transpose the axes of the new array in a particular order.

There's a good chance that einsum will help us do this faster and more memory-efficiently than combinations of the NumPy functions like multiply, sum and transpose will allow.

Consider:

A = np.array([0, 1, 2])

B = np.array([[ 0,  1,  2,  3],
              [ 4,  5,  6,  7],
              [ 8,  9, 10, 11]])

We will multiply A and B element-wise and then sum along the rows of the new array. In "normal" NumPy we'd write:

# Normal Python
>>> (A[:, np.newaxis] * B).sum(axis=1)
array([ 0, 22, 76])

# einsum
>>> np.einsum('i,ij->i', A, B)
array([ 0, 22, 76])

Here is what happens:

• A has one axis; we've labelled it i. And B has two axes; we've labelled axis 0 as i and axis 1 as j.
• By repeating the label i in both input arrays, we are telling einsum that these two axes should be multiplied together. In other words, we're multiplying array A with each column of array B, just like A[:, np.newaxis] * B does.
• Notice that j does not appear as a label in our desired output; we've just used i (we want to end up with a 1D array). By omitting the label, we're telling einsum to sum along this axis. In other words, we're summing the rows of the products, just like .sum(axis=1) does.

• Vectorization. When looping over an array or any data structure in Python, there’s a lot of overhead involved. Vectorized operations in NumPy delegate the looping internally to highly optimized C and Fortran functions, making for a cleaner and faster Python code.
• Broadcasting is the name given to the method that NumPy uses to allow array arithmetic between arrays with a different shape or size. Broadcasting solves the problem of arithmetic between arrays of differing shapes by in effect replicating the smaller array along the last mismatched dimension. Subject to certain constraints, the smaller array is broadcast across the larger array so that they have compatible shapes.

# scalar and one-dimensional
from numpy import array
a = array([1, 2, 3])
print(a)
b = 2
print(b)
c = a + b
print(c)

The current API is that:

• flatten always returns a copy. Can only be called for true numpy arrays.
• ravel returns a view of the original array whenever possible. This isn't visible in the printed output, but if you modify the array returned by ravel, it may modify the entries in the original array. If you modify the entries in an array returned from flatten this will never happen. ravel will often be faster since no memory is copied, but you have to be more careful about modifying the array it returns. ravel can be called on any object that can successfully be parsed.

• test[:,0] - gives you a row vector.

>>> test[:,0]
array([1, 3, 5])

If you just want to loop over it, it's fine, but if you want to hstack with some other array with dimension 3xN, you will have

ValueError: all the input arrays must have the same number of dimensions

• test[:,[0]] - gives you a column vector:

>>> test[:,[0]]
array([[1],
       [3],
       [5]])

As a result, you can do concatenate or hstack operation:

>>> np.hstack((test, test[:,[0]]))
array([[1, 2, 1],
       [3, 4, 3],
       [5, 6, 5]])

A direct method of applying the lambda function is always the fastest and simplest way to map a function over Numpy arrays:

import numpy as np
x = np.array([1, 2, 3, 4, 5])
f = lambda x: x ** 2
squares = f(x)

Another way is to use numpy.vectorize:

import numpy as np
x = np.array([1, 2, 3, 4, 5])
squarer = lambda t: t ** 2
vfunc = np.vectorize(squarer)
vfunc(x)
# Output : array([ 1,  4,  9, 16, 25])

Note: The vectorize function is provided primarily for convenience, not for performance. The implementation is essentially a for loop.

The purpose of meshgrid is to create a rectangular grid out of an array of x values and an array of y values:

xx, yy = np.meshgrid(x, y)

Consider:

xx, yy = np.meshgrid(xvalues, yvalues)
plt.plot(xx, yy, marker='.', color='k', linestyle='none')

A simple way to achieve this is by using np.convolve. The idea behind this is to leverage the way the discrete convolution is computed and use it to return a rolling mean. This can be done by convolving with a sequence of np.ones of a length equal to the sliding window length we want.

In order to do so we could define the following function:

def moving_average(x, w):
    return np.convolve(x, np.ones(w), 'valid') / w

This function will be taking the convolution of the sequence x and a sequence of ones of length w. Note that the chosen mode is valid so that the convolution product is only given for points where the sequences overlap completely.

Some examples:

x = np.array([5,3,8,10,2,1,5,1,0,2])

For a moving average with a window of length 2 we would have:

moving_average(x, 2)
# array([4. , 5.5, 9. , 6. , 1.5, 3. , 3. , 0.5, 1. ])

And for a window of length 4:

moving_average(x, 4)
# array([6.5 , 5.75, 5.25, 4.5 , 2.25, 1.75, 2.  ])

If you just want a straightforward non-weighted moving average, you can easily implement it with np.cumsum:

def moving_average(a, n=3) :
    ret = np.cumsum(a, dtype=float)
    ret[n:] = ret[n:] - ret[:-n]
    return ret[n - 1:] / n

>>> a = np.arange(20)
>>> moving_average(a)
array([  1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,  11.,
        12.,  13.,  14.,  15.,  16.,  17.,  18.])
>>> moving_average(a, n=4)
array([  1.5,   2.5,   3.5,   4.5,   5.5,   6.5,   7.5,   8.5,   9.5,
        10.5,  11.5,  12.5,  13.5,  14.5,  15.5,  16.5,  17.5])