54 Data Analyst Interview Questions (ANSWERED with PDF) to Crack Your ML & DS Interview

Preprocessing describes the process of cleaning and converting a 'raw' (i.e. unprocessed) dataset into a clean dataset.

Possible preprocessing steps are:

• Rescaling attributes with different scales
• Standardizing the dataset
• Encoding categorical attributes with integer values
• Handling missing data
• Removing duplicated data points
• Removing outliers/ handle noisy data
• Discretize the data
• Split the dataset into a training and test set

• Feature engineering allows us to create new features from the ones we already have in order to help the machine learning model make more effective and accurate predictions. Some of the tasks that imply feature engineering are:
  • Filling missing values within a variable.
  • Encoding categorical variables into numbers.
  • Variable transformation.
  • Creating or extracting new features from the ones available in the dataset.

• Feature selection, on the other hand, allows us to select features from the feature pool (including any newly-engineered ones) that will help machine learning models make predictions on target variables more efficiently. For this, two common methods used are wrapper and filter methods. These methods are almost always supervised and are evaluated based on the performance of a resulting model on a holdout dataset.

In a typical machine learning pipeline, we perform feature selection after completing feature engineering.

The workflow of a typical data mining application contains the following phases:

1. Data collection: Data collection may require the use of specialized hardware such as a sensor network, manual labor such as the collection of user surveys, or software tools such as a Web document crawling engine to collect documents. After the collection phase, the data are often stored in a database, or, more generally, a data warehouse for processing.
2. Feature extraction and data cleaning: When data is collected, they are often not in a form that is suitable for processing. In many cases, different types of data may be arbitrarily mixed in a free-form document. To make the data suitable for processing, it is essential to transform them into a format that is friendly to data mining algorithms, such as multidimensional, time series, or semi-structured format. The multi-dimensional formal is the most common one, in which different fields of the data correspond to the different measured properties that are referred to as features, attributes, or dimensions.
3. Analytical processing and algorithms: The final part of the mining process is to design effective analytical methods from the processed data. It may not be possible to directly use a standard data mining problem for the application at hand.

• If we remove redundancies from the dataset, the dataset is smaller but the information a model can obtain by being trained on it won't decrease by the same amount.
• The easiest example of such redundancy is a similar-looking image. But we’re looking for more than just similar images, we look for similar feature activations or semantic redundancies.

DataFrame.iterrows is a generator which yields both the index and row (as a Series):

import pandas as pd

df = pd.DataFrame({'c1': [10, 11, 12], 'c2': [100, 110, 120]})

for index, row in df.iterrows():
    print(row['c1'], row['c2'])

10 100
11 110
12 120

• We can talk about the error of a single measurement, but bias is the average of errors of many repeated measurements,
• Bias is a statistical property of the error of a measuring technique,
• Sometimes the term "bias error" is used as opposed to "root-mean-square error".

The solution is to join the table to itself when the dates differ by one day (DATEDIFF() function) and make sure that the temperature is higher than the previous date.

SELECT W1.ID
FROM WEATHER W1 INNER JOIN WEATHER W2 ON DATEDIFF(W1.RecordDate, W2.RecordDate) = 1
WHERE W1.Temperature > W2.Temperature

• Yes, it should be pre-processed in the same way. The classifier or any model will be trained to classify data that is cleaned, that is the form of data that it expects as input.
• There are exceptions to this principle, though. In certain applications, it is common to perturb (add noise to) replicate training data. This is done to make the classifier output smoother and less jittery when exposed to new data. Test data is usually not perturbed in the same way.

Data mining works by using various algorithms and techniques to turn large volumes of data into useful information. Here are some of the most common ones:

• Association rules: An association rule is a rule-based method for finding relationships between variables in a given dataset. These methods are frequently used for market basket analysis, allowing companies to better understand relationships between different products.
• Neural networks: Neural networks process training data by mimicking the interconnectivity of the human brain through layers of nodes. Each node is made up of inputs, weights, a bias (or threshold), and an output. If that output value exceeds a given threshold, it “fires” or activates the node, passing data to the next layer in the network. Neural networks learn this mapping function through supervised learning, adjusting based on the loss function through the process of gradient descent. When the cost function is at or near zero, we can be confident in the model’s accuracy to yield the correct answer.
• Decision tree: This data mining technique uses classification or regression methods to classify or predict potential outcomes based on a set of decisions. It uses a tree-like visual to represent the potential outcomes of these decisions.
• K-nearest neighbor: It is also known as the KNN algorithm. It is a non-parametric algorithm that classifies data points based on their proximity and association to other available data. This algorithm assumes that similar data points can be found near each other. As a result, it seeks to calculate the distance between data points, usually through Euclidean distance, and then it assigns a category based on the most frequent category or average.

• Cross-validation is a method of assessing how the results of a statistical analysis will generalize on an independent dataset,
• It can be used in machine learning tasks to evaluate the predictive capability of the model,
• It also helps us to avoid overfitting and underfitting,

• A common way to cross-validate is to divide the dataset into training, validation, and testing where:

  • Training dataset is a dataset of known data on which the training is run.
  • Validation dataset is the dataset that is unknown against which the model is tested. The validation dataset is used after each epoch of learning to gauge the improvement of the model.
  • Testing dataset is also an unknown dataset that is used to test the model. The testing dataset is used to measure the performance of the model after it has finished learning.

A measure of central tendency is a single value that attempts to describe a set of data by identifying the central position within that set of data. In the simplest of terms, it attempts to find a single value that best represents an entire distribution of scores.

Mean, Median and Mode are average values or central tendency of a numerical data set.

The normal distribution is the most important probability distribution in statistics because it fits many natural phenomena. For example, heights, blood pressure, measurement error, and IQ scores follow the normal distribution. It is also known as the Gaussian distribution and the bell curve.

Normal distributions have the following features:

• symmetric bell shape
• mean and median and mode are equal; both located at the center of the distribution
• ≈68% of the data falls within 1 standard deviation of the mean
• ≈95% of the data falls within 2 standard deviations of the mean
• ≈99.7% of the data falls within 3 standard deviations of the mean

Normalization is basically to design a database schema such that duplicate and redundant data is avoided. If the same information is repeated in multiple places in the database, there is the risk that it is updated in one place but not the other, leading to data corruption.

There is a number of normalization levels from 1. normal form through 5. normal form. Each normal form describes how to get rid of some specific problem.

By having a database with normalization errors, you open the risk of getting invalid or corrupt data into the database. Since data "lives forever" it is very hard to get rid of corrupt data when first it has entered the database.

• The 68-95-99.7 rule, is also known as the Empirical Rule.
• It is a shorthand used to remember the percentage of values that lie within an interval estimate in a Normal Distribution.
• It is shown below:

The rule states that 68%, 95%, and 99.7% of the values lie within one, two, and three standard deviations of the mean, respectively.

• Scaling simply means $f(x) = cx, c \in R$, this is, multiplying your observations by a constant c which changes the scale (for example from nanometers to kilometers).
• Normalization can either mean applying a transformation so that the transformed data is roughly normally distributed, but it can also simply mean putting different variables on a common scale. In normalization, the observations x are transformed into f(x) (where f is a measurable, typically continuous, function) much that they look normally distributed.

• Data definition language (DDL) commands are the commands which are used to define the database. CREATE, ALTER, DROP and TRUNCATE are some common DDL commands.

• Data manipulation language (DML) commands are commands which are used for manipulation or modification of data. INSERT, UPDATE and DELETE are some common DML commands.

Simply put:

• Feature Selection: you select a subset of the original feature set; while
• Feature Extraction: you build a new set of features from the original feature set.

Examples of feature extraction:

• extraction of contours in images,
• extraction of digrams from a text,
• extraction of phonemes from a recording of spoken text, etc.

Feature extraction involves a transformation of the features, which often is not reversible because some information is lost in the process of dimensionality reduction.

In general, it is safer to work on copies than on original DataFrames, except when you know that you won't be needing the original anymore and want to proceed with the manipulated version.

This is because in Pandas, indexing a DataFrame returns a reference to the initial DataFrame. Thus, changing the subset will change the initial DataFrame. Thus, you'd want to use the copy if you want to make sure the initial DataFrame shouldn't change.

Normally, you would still have some use for the original data frame to compare with the manipulated version, etc. Therefore, depending on the case it's a good practice to work on copies and merge at the end.

It's not recommended to perform K-NN on large datasets, given that the computational and memory cost can increase. To understand the reason why we should remember how the K-NN algorithm works:

1. Starts by calculating the distances to all vectors in a training set and store them.
2. Then, it sorts the calculated distances.
3. Then, we store the K nearest vectors.
4. And finally, calculate the most frequent class displayed by K nearest vectors.

So implement K-NN on a large dataset it is not only a bad decision to store a large amount of data but it is also computationally costly to keep calculating and sorting all the values. For that reason, K-NN is not recommended and another classification algorithm like Naive Bayes or SVM is preferred in such cases.

• Atomicity: It ensures all-or-none rule for database modifications.
• Consistency: Data values are consistent across the database.
• Isolation: Two transactions are said to be independent of one another.
• Durability: Data is not lost even at the time of server failure.

• INNER JOIN is ANSI syntax that you should use. INNER JOIN helps human readability, and that's a top priority. It can also be easily replaced with an OUTER JOIN whenever a need arises.

• Implicit joins (with multiple FROM tables) become much much more confusing, hard to read, and hard to maintain once you need to start adding more tables to your query. The old syntax, with just listing the tables, and using the WHERE clause to specify the join criteria, is being deprecated in most modern databases.

To answer this question we can use the standardized variable z, defined as

which measures the deviation of X from the mean, in terms of standard deviation s.

Therefore, the standardized scores are

Thus, the mathematics exam score was 0.67 standard deviations better than the class average; the biology score was 0.7 standard deviations better than the class average, and the physics score was only 0.5 standard deviations better than the class average. Therefore, even though the actual score on the physics exam was the lowest of the three exam scores, relative to the distribution of all class exam scores, the physics exam score was the highest relative grade.

Simply group on both of the columns:

SELECT
    name, email, COUNT(*) as CountOf
FROM
    users
GROUP BY
    name, email
HAVING 
    COUNT(*) > 1

The whole point of having an index is to speed up search queries by essentially cutting down the number of records/rows in a table that need to be examined. An index is a data structure (most commonly a B- tree) that stores the values for a specific column in a table.

• If the proportion of missing on X is not low (for example, > 10%), you can create two variables: $X_1 = X$ if X is not missing, = 0 if X is missing. $X_2 = 0$ is X is not missing, = 1 if X is missing.
• Then fit the model with $X_1, X_2$ as covariates (other covariates have no change and remain in the model).
• The meaning of coefficient of $X_1$ is the same, the change of response variable when X increase by 1 unit. The meaning of coefficient of $X_2$ is the average change of response variable comparing X is missing and X = 0.
• If the proportion of X missing is low, deleting them from the analysis is a good approach.

• There is a measure of similarity in data mining and machine learning, and it falls under Metric Learning.

• It aims to automatically construct task-specific distance metrics from (weakly) supervised data, in a machine learning manner. The learned distance metric can then be used to perform various tasks. So,e distance metrics examples:

  • Euclidean distance
  • Mahalanobis distance
  • Pearson correlation
  • Cosine similarity
  • Jaccard similarity

• Many distance-based data mining applications lose their effectiveness as the dimensionality of the data increases.
• For example, a distance-based clustering algorithm may group unrelated data points because the distance function may poorly reflect the intrinsic semantic distances between data points with increasing dimensionality.
• As a result, distance-based models of clustering, classification, and outlier detection are often qualitatively ineffective. This phenomenon is referred to as the curse of dimensionality.

Iteration in Pandas is an anti-pattern and is something you should only do when you have exhausted every other option. Iterating through pandas objects is generally slow. In many cases, iterating manually over the rows is not needed. You should not use any function with "iter" in its name for more than a few thousand rows or you will have to get used to a lot of waiting.

Do you want to print a DataFrame? Use DataFrame.to_string().

Do you want to compute something? In that case, search for methods in this order (list modified from here):

1. Vectorization
2. Cython routines
3. List Comprehensions (vanilla for loop)
4. DataFrame.apply(): i)  Reductions that can be performed in Cython, ii) Iteration in Python space
5. DataFrame.itertuples() and iteritems()
6. DataFrame.iterrows()

iterrows and itertuples (both receiving many votes in answers to this question) should be used in very rare circumstances, such as generating row objects/nametuples for sequential processing, which is really the only thing these functions are useful for.

There are basic things you can do with any set of data:

1. Validate values (String length tolerance, data type, formatting masks, required field presence, etc.)
2. Range correctness (Does this seemingly correct data fall within expected ranges of values)
3. Preliminary processing (If I attempt to analyze this data, can I perform the basics without running into errors)
4. Preliminary reporting (run a report against a data set and ensure that it passes a sanity test)
5. Defining null vs. empty vs. zero vs. False for any given column of data
6. Identifying data that is out of place (numeric values dramatically different than other values in a data set, string values that look like they might be misspelled, etc.)
7. Eliminating or correcting obviously errant data
8. Do not assume that your data was collected by perfect processes /humans.
9. Do try to understand the limits of your data providers.
10. Look at individual patterns/values and try to determine if they are logical (easy for movement / geographic data)

Understanding data to identify errors is a whole different ball game, and it is very important.

For instance, you can have a rule that says a serial number must be present in a given data set and that serial number must be alphanumeric with a maximum string length of 255 and a minimum string length of 5.

The way I avoid problems is by leveraging the people around me. For small data sets, I can ask someone to review the data in it's entirety. For large ones, pulling a set of random samples and asking someone to do a sanity check on the data is more appropriate.

Further, questioning the source of the data and how well that data source can be trusted is imperative. I often have multiple conflicting sources of data and we create rules to determine the "source of truth". Sometimes one data set has great data in a given aspect, but other data sets are stronger in other areas.

1. Handling missing entries: Many entries in the data may remain unspecified because of weakness in data collection or the inherent nature of the data. Such missing entries may need to be estimated. The process of estimating missing entries is also referred to as imputation.
2. Handling incorrect entries: In cases where the same information is available from multiple sources, inconsistencies may be detected. Such inconsistencies can be removed as a part of the analytical process. Another method for detecting incorrect entries is to use domain-specific knowledge about what is already known about the data.
3. Scaling and normalization: The data may often be expressed in very different scales. This may result in some features being inadvertently weighted too much so that the other features are implicitly ignored. Therefore, it is important to normalize the different features.

1. Missing at Random (MAR): Missing at random means that the propensity for a data point to be missing is not related to the missing data, but is related to some of the observed data
2. Missing Completely at Random (MCAR): The fact that a certain value is missing has nothing to do with its hypothetical value and with the values of other variables.
3. Missing not at Random (MNAR): Two possible reasons are that the missing value depends on the hypothetical value (e.g. People with high salaries generally do not want to reveal their incomes in surveys) or a missing value is dependent on some other variable’s value.

In the first two cases, it is safe to remove the data with missing values depending upon their occurrences, while in the third case removing observations with missing values can produce a bias in the model.

Statistical tests work by calculating a test statistic – a number that describes how much the relationship between variables in your test differs from the null hypothesis of no relationship.

It then calculates a p-value (probability value). The p-value estimates how likely it is that you would see the difference described by the test statistic if the null hypothesis of no relationship were true.

If the value of the test statistic is more extreme than the statistic calculated from the null hypothesis, then you can infer a statistically significant relationship between the predictor and outcome variables.

If the value of the test statistic is less extreme than the one calculated from the null hypothesis, then you can infer no statistically significant relationship between the predictor and outcome variables.

It is the process of improving the performance of the database by adding redundant data.

In general, the normalization of a vector v with respect to a norm || . || is given by $y = \frac{v}{||v||}$. This new vector y has the properties:

1. It has norm one, meaning that $||y|| = 1$.
2. It has the same direction as the original vector v, meaning that v is proportionate to y.
   • The term $l_1$ normalization just means that the norm being used is the $l_1$ norm $||v||_1 = \sum_{i=1}^n |v_i|$.
   • In general, $l_1$ normalization does not make a vector into a probability mass function because the normalized vector can have negative entries.

Short answer:

• The training set is used to fit the models;
• the validation set is used to estimate prediction error for model selection;
• the test set is used for assessment of the generalization error of the final chosen model.

When you have a large data set, it's recommended to split it into 3 parts:

• Training set (60% of the original data set): This is used to build up our prediction algorithm. Our algorithm tries to tune itself to the quirks of the training data sets. In this phase we usually create multiple algorithms in order to compare their performances during the Cross-Validation Phase.

• Cross-Validation set (20% of the original data set): This data set is used to compare the performances of the prediction algorithms that were created based on the training set. We choose the algorithm that has the best performance.

• Test set (20% of the original data set): Now we have chosen our preferred prediction algorithm but we don't know yet how it's going to perform on completely unseen real-world data. So, we apply our chosen prediction algorithm on our test set in order to see how it's going to perform in the wild so we can have an idea about our algorithm's performance on unseen data.

An inner join retrieve the matched rows only.

Whereas an outer join retrieve the matched rows from one table and all rows in other table ....the result depends on which one you are using:

• Left: Matched rows in the right table and all rows in the left table
• Right: Matched rows in the left table and all rows in the right table or
• Full: All rows in all tables. It doesn't matter if there is a match or not

Inner Join

Retrieve the matched rows only, that is, A intersect B.

SELECT *
FROM dbo.Students S
INNER JOIN dbo.Advisors A
    ON S.Advisor_ID = A.Advisor_ID

Left Outer Join

Select all records from the first table, and any records in the second table that match the joined keys.

SELECT *
FROM dbo.Students S
LEFT JOIN dbo.Advisors A
    ON S.Advisor_ID = A.Advisor_ID

Full Outer Join

Select all records from the second table, and any records in the first table that match the joined keys.

SELECT *
FROM dbo.Students S
FULL JOIN dbo.Advisors A
    ON S.Advisor_ID = A.Advisor_ID

• Bagging:

  • Is a parallel ensemble method: the base learners are created independently for each other and are trained simultaneously.
  • Is based on the bootstrap aggregate: it aggregates several sampling subsets of the original dataset to train different learners chosen randomly with replacement.
  • Is focused on decreases variance, not bias, and solves over-fitting issues in a model.

• Boosting

  • Is a sequential ensemble method: the base learners depend on each other, so each learner influences the next one and general paternal behavior can be deduced.
  • Each base learner in the sequence is fitted giving more importance to observations in the dataset that were badly handled by the previous learners in the sequence. Intuitively, each new model focuses its efforts on the most difficult observations to fit up to now, so that we obtain, at the end of the process, a strong learner.
  • Is mainly focused on reducing bias, not variance, and can increase the overfitting in some cases.

Primary Key:

• There can only be one primary key in a table
• In some DBMS it cannot be NULL - e.g. MySQL adds NOT NULL
• Primary Key is a unique key identifier of the record
• Primary key can be created on multiple columns (composite primary key)

Unique Key:

• Can be more than one unique key in one table
• Unique key can have NULL values
• It can be a candidate key
• Unique key can be NULL ; multiple rows can have NULL values and therefore may not be considered "unique"

Difference between Primary Key and Unique key

• 1. Behavior: Primary Key is used to identify a row (record) in a table, whereas Unique-key is to prevent duplicate values in a column (with the exception of a null entry).
• 2. Indexing: By default SQL-engine creates Clustered Index on primary-key if not exists and Non-Clustered Index on Unique-key.
• 3. Nullability: Primary key does not include Null values, whereas Unique-key can.
• 4. Existence: A table can have at most one primary key, but can have multiple Unique-key.
• 5. Modifiability: You can’t change or delete primary values, but Unique-key values can.

• Identifying Outliers with Visualization
  • Box Plot
  • Histogram
  • Scatterplot
• Identifying Outliers with Skewness - Several machine learning algorithms make the assumption that the data follow a normal (or Gaussian) distribution. This is easy to check with the skewness value, which explains the extent to which the data is normally distributed. Ideally, the skewness value should be between -1 and +1, and any major deviation from this range indicates the presence of extreme values.

print(df['Income'].skew())
df['Income'].describe()

• Identifying Outliers with Interquartile Range (IQR) - The interquartile range (IQR) is a measure of statistical dispersion and is calculated as the difference between the 75th and 25th percentiles.

Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3 - Q1
print(IQR)

it makes no sense to remove two correlated variables unless correlation = 1 or -1, in which case one of the variables is redundant.

You do not want to remove all correlated variables. It is only when the correlation is so strong that they do not convey extra information (see above). This is both a function of the strength of correlation, how much data you have, and whether any small difference between correlated variables tells you something about the outcome, after all.

The first two you can tell before you do any model, the final one not. So, it may be very reasonable to remove variables based on the combination of the first two considerations (i.e. even if the extra variables may in principle contain some useful information, you would not be able to tell given the strength of correlation and how much data you have) before you do any modeling/feature engineering. The final point can really only be assessed after doing some modeling.