27 Reinforcement Learning Interview Questions (ANSWERED) for Machine Learning Engineers

Reinforcement learning (RL) is a subset of machine learning that allows an AI-driven system (sometimes referred to as an agent) to learn through trial and error using feedback from its actions. This feedback is either negative or positive, signaled as punishment or reward with, of course, the aim of maximizing the reward function.

In terms of learning methods, RL is similar to supervised learning only in that it uses mapping between input and output, but that is the only thing they have in common. Whereas in supervised learning, the feedback contains the correct set of actions for the agent to follow. In RL there is no such answer key. The agent decides what to do itself to perform the task correctly.

Compared with unsupervised learning, RL has different goals. The goal of unsupervised learning is to find similarities or differences between data points. RL's goal is to find the most suitable action model to maximize total cumulative reward for the RL agent. With no training dataset, the RL problem is solved by the agent's own actions with input from the environment.

The problem of State Representation in Reinforcement Learning (RL) is similar to problems of feature representation, feature selection and feature engineering in Supervised or Unsupervised Learning.

A common approach to modelling complex problems is Discretization. At a basic level, this is splitting a complex and continuous space into a grid. Then you can use any of the classic RL techniques that are designed for discrete, linear, spaces.

Using tabular learning algorithms is another good approach to define states given that they have reasonable theoretical guarantees of convergence, which means if you can simplify your problem so that it has, say, less than a few million states, then this is worth trying.

Most interesting control problems will not fit into that number of states, even if you discretize them. This is due to the curse of dimensionality. For those problems, you will typically represent your state as a vector of different features - e.g. for a robot learning to walk, various positions, angles, velocities of mechanical parts. As with supervised learning, you may want to treat these for use with a specific learning process. For instance, typically you will want them all to be numeric, and if you want to use a neural network you should also normalize them to a standard range (e.g. -1 to 1).

• Dynamic Programming (DP): When the model is fully known, following Bellman equations, we can use DP to iteratively evaluate value functions and improve policy.

• Monte-Carlo (MC)Methods: It learns from episodes of raw experience without modeling the environmental dynamics and computes the observed mean return as an approximation of the expected return. One important thing here is that the episodes must be complete, which means that all the episodes must eventually terminate.

• Temporal-Difference (TD) Learning: Similar to Monte-Carlo methods, TD Learning is model-free and learns from episodes of experience. However, TD learning can learn from incomplete episodes and hence we don’t need to track the episode up to termination.

• Policy Gradient: All previous methods aim to learn the state/action-value function and then to select actions accordingly. Policy Gradient methods instead learn the policy function directly with respect to some parameter θ, so here we aim to find the best θ that produces the highest return.

• Evolution Strategies (ES): It learns the optimal solution by imitating Darwin's theory of the evolution of species by natural selection. Two prerequisites for applying ES: (i) our solutions can freely interact with the environment and see whether they can solve the problem; (ii) we are able to compute a fitness score of how good each solution is. We don’t have to know the environment configuration to solve the problem.

A typical reinforcement learning (RL) problem have some basics elements such as:

• An Environment: Physical world in which the agent operates.
• State: Current situation of the agent.
• Reward: Feedback from the environment.
• Policy: Method to map agent’s state to actions.

But we can think the policy like an agent's strategy. For example, imagine a world where a robot (agent) moves across the room and the task is to get to the target point (x, y), where it gets a reward. Here:

• A room is an environment.
• Robot's current position is a state.
• A policy is what an agent (the robot) does to accomplish this task. The robots have a few options:
  • Policy #1: dumb robots just wander around randomly until they accidentally end up in the right place.
  • Policy #2: other robots may, for some reason, learn to go along the walls most of the route.
  • Policy #3: smart robots plan the route in their "head" and go straight to the goal.

Obviously, some policies are better than others, and there are multiple ways to assess them, but the goal of RL is to learn the best policy. In the example, the best policy would be option 3. In such terms, the policy is then what defines the learning agent's way of behaving at a given time and is typically used by the agent to decide what action should be performed.

1. First, the agent interacts with the environment by performing an action.
2. The agent performs an action and moves from one state to another.
3. Then the agent will receive a reward based on the action it performed.
4. Based on the reward, the agent will understand whether the action is good or bad.
5. If the action was good, that is, if the agent received a positive reward, then the agent will prefer performing that action, else the agent will try performing other actions that can result in a positive reward. So reinforcement learning is basically a trial-and-error learning process.

• A state is Markov if and only if:

  $P[S_{t+1}|S_t]=P[S_{t+1}|S_1,...,S_t]$

  This equation means that the current state of the agent only depends on the previous state and not on any state prior to that.

• This Markov Decision Process is used in Reinforcement Learning.

To understand the difference between On-Policy Learning and Off-Policy Learning let us first take a look at two terms before moving further.

• Target Policy: It is the policy that an agent is trying to learn i.e agent is learning value function for this policy.
• Behavior Policy: It is the policy that is being used by an agent for action select i.e agent follows this policy to interact with the environment.

Now, On-Policy Learning :

• We evaluate and improve the same policy which is being used to select actions. That means we will try to evaluate and improve the same policy that the agent is already using for action selection. In short , [Target Policy == Behavior Policy]. Some examples of On-Policy algorithms are Policy Iteration, Value Iteration, Monte Carlo for On-Policy, Sarsa, etc.

In Off-Policy Learning:

• We evaluate and improve a policy that is different from the policy that is used for action selection. In short, [Target Policy != Behavior Policy]. Some examples of Off-Policy learning algorithms are Q learning, expected sarsa(can act in both ways), etc.

• A Reward is a number returned at a certain step of the Markov Decision Process. If you arrange things in sequence over a whole time step s,a,r,s' for state, action, reward, next state, then the reward r is allowed to depend on all three of s,a,s', and it can also be from a random distribution of real numbers or just a single number.

• State values are a way to measure longer-term benefits of being in a state, they are also called the expected return for an agent starting from that state and following a particular policy.

• Therefore, we can see state values composed of many rewards weighted by their probability of occurring in the future.

The discount factor, 𝛾, is a real value ∈ [0, 1], cares for the rewards agent achieved in the past, present, and future. In different words, it relates the rewards to the time-domain. Let’s explore the two following cases:

• When we set the discount factor 𝛾 = 1, it implies that we consider all the future rewards. Then the agent will learn forever, looking for all the future rewards, which may lead to infinity.
• When we set the discount factor 𝛾 = 0, it implies that we consider only the immediate reward and not the reward obtained from the future time steps. Then the agent will never learn as it will consider only the immediate reward.

Therefore the optimal value of the discount factor lies between 0.2 and 0.8 it set the importance to immediate and future rewards depending on the tasks. In some tasks, future rewards are more desirable than immediate rewards, and vice versa.

For example, in a chess game, the goal is to defeat the opponent's king. If we give more importance to the immediate reward, which is acquired by actions such as our pawn defeating any opposing chessman, then the agent will learn to perform this sub-goal instead of learning the actual goal. So, in this case, we give greater importance to future rewards than the immediate reward, whereas in some cases, we prefer immediate rewards over future rewards. For example, you may prefer to eat a fresh chocolate chip today and not in 13 days later.

The ϵ-greedy policy is a policy that chooses the best action (i.e. the action associated with the highest value) with probability 1−ϵ ∈[0,1] and a random action with probability ϵ. The problem with ϵ-greedy is that when it chooses the random actions (i.e. with probability ϵ), it chooses them uniformly (i.e. it considers all actions equally good), even though certain actions (even excluding the currently best one) are better than others.

One solution is to this is to use Softmax Action Selection Rules, here we vary the action probabilities as a graded function of estimated value. In this way, the greedy action is still given the highest selection probability, but all the others are ranked and weighted according to their value estimates. The most common softmax method uses a Gibbs, or Boltzmann, distribution.

Monte Carlo is a model-free method, and so it doesn't require the model dynamics of the environment to compute the value and Q function in order to find the optimal policy. Instead, the Monte Carlo method computes the value function and Q function by just taking the average return of the state and the average return of the state-action pair, respectively.

But one issue with the Monte Carlo method is that it is applicable only to episodic tasks: In the Monte Carlo method, we compute the value of the state by taking the average return of the state and the return is the sum of rewards of the episode. But when there is no episode, that is, if our task is a continuous task (non-episodic task), then we cannot apply the Monte Carlo method.

An Epsilon-Greedy Policy allows the agent to decide according to a certain threshold, between an action that maximizes a Q-value or over a random action that it may maximize the Q-value.

For example, say there are many routes from our work to home and we have explored only two routes so far. Thus, to reach home, we can select the route that takes us home most quickly out of the two routes we have explored (this is our Q-value). However, there are still many other routes that we have not explored yet that might be even better than our current optimal route. The question is whether we should explore new routes (exploration) or whether we should always use our current optimal route (exploitation).

In such context, we introduce a policy called the epsilon-greedy policy:

• With a probability epsilon, we explore different actions of ways to go home from work (exploration).
• With a probability 1-epsilon, we choose an action that has the maximum Q value, that is, the route that takes us to home in the quickest way (exploitation).

Now, before selecting an action, a random number r in the range of [0,1] is selected. If that r is larger than epsilon, we use the well-known route that will take us home more quickly; but if r < epsilon, a random action is selected and we explore other routes. If we follow these rules then we have implemented an epsilon-greedy policy.

• Supervised Learning analyses the training data and produces a generalized formula.
• In Reinforcement Learning basic reinforcement is defined in the model Markov’s Decision process.

• In Supervised Learning, each example will have input objects and output with desired values.
• In Reinforcement Learning, the agent interacts with the environment in discrete steps, and receives a reward for ever observation. The goal is to collect as many rewards as possible to make more observations.

Let's recap the definition of the Value Function. The value function or the value of the state s can be defined as the expected return the agent would obtain starting from the state s and following the policy 𝜋. It can be expressed as:

In order to approximate the value of the state using the Monte Carlo method, we do the following: 1. Sample N episodes (trajectories) following the given policy 𝜋. Our approximation will be better when N is higher. 2. Compute the value function as the average return of a state across the sample episodes.

In a nutshell, in the Monte Carlo prediction method, we generate some N episodes using the given policy and then we compute the value function as the average return of the state across these N episodes, instead of taking the expected return.

• Typically the gamma (γ) parameter is viewed as part of the problem, not of the algorithm. A reinforcement learning algorithm tries for each state to optimize the cumulative discounted reward:

  $r_1 + \gamma \cdot r_2 + \gamma^2 \cdot r_3 + \gamma^3 \cdot r_4 \cdots$

  Where r_n is the reward received at time step n from the current state. So, for one choice of gamma the algorithm may optimize one thing, and for another choice, it will optimize something else. However, when you have defined a certain high-level goal, there still often remains a modeling choice, as many different gamma's might satisfy the requirements of the goal.

• In general, most algorithms learn faster when they don't have to look too far into the future. So, it sometimes helps the performance to set gamma relatively low. A general rule of thumb might be: determine the lowest gamma min_gamma that still satisfies your high-level goal, and then set the gamma to gamma = (min_gamma + 1)/2. (You don't want to use gamma = min_gamma itself, since then some suboptimal goal will be deemed virtually as good as the desired goal.) Another useful rule of thumb: for many problems a gamma of 0.9 or 0.95 is fine. However, always think about what such a gamma means for the goal you are optimizing when combined with your reward function.

• The lambda parameter determines how much you bootstrap on earlier learned value versus using the current Monte Carlo roll-out. This implies a trade-off between more bias (low lambda) and more variance (high lambda). In many cases, setting lambda equal to zero is already a fine algorithm, but setting lambda somewhat higher helps speed up things. Here, you do not have to worry about what you are optimizing: the goal is unrelated to lambda and this parameter only helps to speed up learning. In other words, lambda is completely part of the algorithm and not of the problem.

• A general rule of thumb is to use a lambda equal to 0.9. However, it might be good just to try a few settings (e.g., 0, 0.5, 0.8, 0.9, 0.95 and 1.0) and plot the learning curves. Then, you can pick whichever seems to be learning the fastest.

Temporal Difference (TD) is the combination of both Monte Carlo (MC) and Dynamic Programming (DP) ideas. In this sense, like Monte Carlo methods, TD methods can learn directly from the experiences without the model of the environment, but on other hand, there are inherent advantages of TD-learning over Monte Carlo methods.

• In MC methods:

  • We must wait until the end of the episode before the return is known.
  • We have high variance and low bias.
  • We don't exploit the Markov property.

• In Temporal Difference learning:

  • We can learn online after every step and do not need to wait until the end of the episode.
  • We have low variance and some decent bias.
  • We exploit the Markov property.

The Markov property state that the future is independent of the past given the present, so in this sense temporal difference could be applied in environments that satisfy such assumption.

• Reinforcement learning is a type of machine learning paradigm where the model take action to maximize the notion of cumulative reward much like living beings do.
• Learning how to play games, and self-driving cars are all modeled as a reinforcement learning problem.
• If the problem to be modeled is a game, then the screen is taken as input. The algorithm takes the pixels as input and passes it through multiple layers of convolutional neural networks to give an output for the next steps to take. The outcome of the steps taken by the model serves as the positive or negative reinforcement.

We can categorize the environment into different types:

• Deterministic and Stochastic environments: In a deterministic environment, we are certain that when an agent performs action a in state s, then it always reaches state 𝑠′. For example, let's consider a grid world environment. Say the agent is in state A, and when it moves down from state A, it always reaches state D. Hence the environment is called a deterministic environment. On other hand, in a stochastic environment instead we don't have certainly but a probability distribution over the agent's actions. Taking the grid world environment example, let's say our agent is in state A; now if it moves down from state A, then the agent doesn't always reach state D. Instead, it reaches state D 70% of the time and state B 30% of the time.

• Discrete and Continuous environments: A discrete environment is one where the environment's action space is discrete. For instance, in the grid world environment, we have a discrete action space, which consists of the actions up, down, left, right. A continuous environment is one where the environment's action space is continuous. For instance, suppose we are training an agent to drive a car, then our action space will be continuous, with several continuous actions such as changing the car's speed, the number of degrees the agent needs to rotate the wheel, and so on.

• Episodic and Non-episodic environments: In an episodic environment, an agent's current action will not affect future actions and in a non-episodic (also called sequential) environment, an agent's current action will affect future actions. For example, a chessboard is a sequential environment since the agent's current action will affect future actions in a chess match.

• Finally, we have Single and Multi-agent environments: When our environment consists of only a single-agent, then it is called a single-agent environment, and when we have multiple agents, then it is called a multi-agent environment.

1. Objective Function

   • In Q-Learning we learn a Q-function that satisfies the Bellman (Optimality) Equation. This is most often achieved by minimizing the Mean Squared Bellman Error (MSBE) as the loss function. The Q-function is then used to obtain a policy (e.g. by greedily selecting the action with maximum value).
   • Policy Gradient methods directly try to maximize the expected return by taking small steps in the direction of the policy gradient. The policy gradient is the derivative of the expected return with respect to the policy parameters.

2. On vs. Off-Policy

   • The Policy Gradient is derived as an expectation over trajectories (s1,a1,r1,s2,a2,...,rn), which is estimated by a sample mean. To get an unbiased estimate of the gradient, the trajectories have to be sampled from the current policy. Thus, policy gradient methods are on-policy methods.
   • Q-Learning only makes sure to satisfy the Bellman-Equation. This equation has to hold true for all transitions. Therefore, Q-learning can also use experiences collected from previous policies and is off-policy.

3. Stability and Sample Efficiency

   • By directly optimizing the return and thus the actual performance on a given task, Policy Gradient methods tend to more stably converge to a good behavior. Indeed being on-policy, makes them very sample inefficient.
   • Q-learning find a function that is guaranteed to satisfy the Bellman-Equation, but this does not guarantee to result in near-optimal behavior. Several tricks are used to improve convergence and in this case, Q-learning is more sample efficient.

Let's illustrate the difference with an example. In the preceding grid world environment, the goal of the agent is to reach state I starting from state A without visiting the shaded states. In each of the states, the agent can perform any of the four actions: up, down, left, and right to achieve the goal.

• A Deterministic Policy tells the agent to perform one particular action in a state. Thus, the deterministic policy maps the state to one particular action and is often denoted by 𝜇. Formally, given a state s at a time t, a deterministic policy tells the agent to perform one particular action a. It can be expressed as:

  $a_t = \mu(s_t)$

  In the example, given state A, the deterministic policy 𝜇 tells the agent to perform the action down. This can be expressed as:

  $\mu(A) = Down$

  Thus, according to the deterministic policy, whenever the agent visits state A, it performs the action down.

• A Stochastic Policy does not map a state directly to one particular action; instead, it maps the state to a probability distribution over an action space. So instead of performing the same action every time the agent visits the state, the agent performs different actions each time based on a probability distribution returned by the stochastic policy.

  The stochastic policy is often denoted by 𝜋. Say we have a state s and action a at a time t, then we can express the stochastic policy as:

  $a_t \sim \pi(s_t)$

  In the example, given a state A, suppose the stochastic policy returns the probability distribution over the action space as [0.10, 0.70, 0.10, 0.10]. Now, whenever the agent visits state A, the agent selects up 10% of the time, down 70% of the time, left 10% of the time, and right 10% of the time.

When the environment consists of a large number of states and actions, it will be very expensive to compute the Q value of all possible state-action pairs in an exhaustive fashion. So, instead of computing Q values in this way, can we approximate them using any function approximator, such as a neural network.

In a Deep Q-Network we can parameterize our Q function by a parameter 𝜃 and compute the Q value where the parameter 𝜃 is just the parameter of our neural network. So, we just feed the state of the environment to a neural network and it will return the Q value of all possible actions in that state.

Once we obtain the Q values, then we can select the best action as the one that has the maximum Q value.

In Value-based methods, we improve the Q function iteratively, and once we have the optimal Q function, then we extract optimal policy by selecting the action in each state that has the maximum Q value.

One of the disadvantages of the value-based method is that it is suitable only for discrete environments (environments with a discrete action space), and we cannot apply value-based methods in continuous environments (environments with a continuous action space).

For example, say we are training an agent to drive a car and say we have one continuous action in our action space. Let the action be the speed of the car and the value of the speed of the car ranges from 0 to 150 kmph.

In this case, we can discretize the continuous actions into speed (0 to 10) as action 1, speed (10 to 20) as action 2, and so on. After discretization, we can compute the Q value of all possible state-action pairs. However, discretization is not always desirable. We might lose several important features and we might end up in an action space with a huge set of actions.

Most real-world problems have continuous action space, say, a self-driving car, or a robot learning to walk and more. Apart from having a continuous action space they also have a high dimension. Thus, using the Deep Q Network and other value-based methods cannot deal with the continuous action space effectively.

So, we use the Policy-based methods. With policy-based methods, we don't need to compute the Q function (Q values) to find the optimal policy; instead, it finds the optimal policy by parameterizing the policy using some parameter 𝜃. The basic idea is to find the best θ that produces the highest return.

In addition to that, Most policy-based methods use a stochastic policy. We know that with a stochastic policy, we select actions based on the probability distribution over the action space, which allows the agent to explore different actions instead of performing the same action every time. Thus, policy-based methods take care of the exploration-exploitation trade-off implicitly by using a stochastic policy.