Markovljev proces odlučivanja

In mathematics, a Markov decision process (MDP) is a discrete-time stochastic control process. It provides a mathematical framework for modeling decision making in situations where outcomes are partly random and partly under the control of a decision maker. MDPs are useful for studying optimization problems solved via dynamic programming. MDPs were known at least as early as the 1950s; a core body of research on Markov decision processes resulted from Ronald Howard's 1960 book, Dynamic Programming and Markov Processes. They are used in many disciplines, including robotics, automatic control, economics and manufacturing. The name of MDPs comes from the Russian mathematician Andrey Markov as they are an extension of Markov chains.

At each time step, the process is in some state $s$, and the decision maker may choose any action $a$ that is available in state $s$. The process responds at the next time step by randomly moving into a new state $s'$, and giving the decision maker a corresponding reward $R_{a}(s,s')$.

The probability that the process moves into its new state $s'$ is influenced by the chosen action. Specifically, it is given by the state transition function $P_{a}(s,s')$. Thus, the next state $s'$ depends on the current state $s$ and the decision maker's action $a$. But given $s$ and $a$, it is conditionally independent of all previous states and actions; in other words, the state transitions of an MDP satisfy the Markov property.

Markov decision processes are an extension of Markov chains; the difference is the addition of actions (allowing choice) and rewards (giving motivation). Conversely, if only one action exists for each state (e.g. "wait") and all rewards are the same (e.g. "zero"), a Markov decision process reduces to a Markov chain.

Definition

Example of a simple MDP with three states (green circles) and two actions (orange circles), with two rewards (orange arrows)

A Markov decision process is a 4-tuple $(S,A,P_{a},R_{a})$, where:

• $S$ is a set of states called the state space,
• $A$ is a set of actions called the action space (alternatively, $A_{s}$ is the set of actions available from state $s$),
• $P_{a}(s,s')=\Pr(s_{t+1}=s'\mid s_{t}=s,a_{t}=a)$ is the probability that action $a$ in state $s$ at time $t$ will lead to state $s'$ at time $t+1$,
• $R_{a}(s,s')$ is the immediate reward (or expected immediate reward) received after transitioning from state $s$ to state $s'$, due to action $a$

The state and action spaces may be finite or infinite, for example the set of real numbers. Some processes with countably infinite state and action spaces can be reduced to ones with finite state and action spaces.

A policy function $\pi$ is a (potentially probabilistic) mapping from state space ($S$) to action space ($A$).

Optimization objective

The goal in a Markov decision process is to find a good "policy" for the decision maker: a function $\pi$ that specifies the action $\pi (s)$ that the decision maker will choose when in state $s$. Once a Markov decision process is combined with a policy in this way, this fixes the action for each state and the resulting combination behaves like a Markov chain (since the action chosen in state $s$ is completely determined by $\pi (s)$ and $\Pr(s_{t+1}=s'\mid s_{t}=s,a_{t}=a)$ reduces to $\Pr(s_{t+1}=s'\mid s_{t}=s)$, a Markov transition matrix).

The objective is to choose a policy $\pi$ that will maximize some cumulative function of the random rewards, typically the expected discounted sum over a potentially infinite horizon:

$E\left[\sum _{t=0}^{\infty }{\gamma ^{t}R_{a_{t}}(s_{t},s_{t+1})}\right]$ (where we choose $a_{t}=\pi (s_{t})$, i.e. actions given by the policy). And the expectation is taken over $s_{t+1}\sim P_{a_{t}}(s_{t},s_{t+1})$

where $\ \gamma \$ is the discount factor satisfying $0\leq \ \gamma \ \leq \ 1$, which is usually close to 1 (for example, $\gamma =1/(1+r)$ for some discount rate r). A lower discount factor motivates the decision maker to favor taking actions early, rather than postpone them indefinitely.

A policy that maximizes the function above is called an optimal policy and is usually denoted $\pi ^{*}$. A particular MDP may have multiple distinct optimal policies. Because of the Markov property, it can be shown that the optimal policy is a function of the current state, as assumed above.

Simulator models

In many cases, it is difficult to represent the transition probability distributions, $P_{a}(s,s')$, explicitly. In such cases, a simulator can be used to model the MDP implicitly by providing samples from the transition distributions. One common form of implicit MDP model is an episodic environment simulator that can be started from an initial state and yields a subsequent state and reward every time it receives an action input. In this manner, trajectories of states, actions, and rewards, often called episodes may be produced.

Another form of simulator is a generative model, a single step simulator that can generate samples of the next state and reward given any state and action. (Note that this is a different meaning from the term generative model in the context of statistical classification.) In algorithms that are expressed using pseudocode, $G$ is often used to represent a generative model. For example the expression $s',r\gets G(s,a)$ might denote the action of sampling from the generative model where $s$ and $a$ are the current state and action, and $s'$ and $r$ are the new state and reward. Compared to an episodic simulator, a generative model has the advantage that it can yield data from any state, not only those encountered in a trajectory.

These model classes form a hierarchy of information content: an explicit model trivially yields a generative model through sampling from the distributions, and repeated application of a generative model yields an episodic simulator. In the opposite direction, it is only possible to learn approximate models through regression. The type of model available for a particular MDP plays a significant role in determining which solution algorithms are appropriate. For example, the dynamic programming algorithms described in the next section require an explicit model, and Monte Carlo tree search requires a generative model (or an episodic simulator that can be copied at any state), whereas most reinforcement learning algorithms require only an episodic simulator.

Literatura

• Bellman., R. E. (2003) [1957]. Dynamic Programming (Dover paperback изд.). Princeton, NJ: Princeton University Press. ISBN 978-0-486-42809-3. 
• Bertsekas, D. (1995). Dynamic Programming and Optimal Control. 2. MA: Athena. 
• Derman, C. (1970). Finite state Markovian decision processes. Academic Press. 
• Feinberg, E.A.; Shwartz, A., ур. (2002). Handbook of Markov Decision Processes. Boston, MA: Kluwer. ISBN 9781461508052. 
• Guo, X.; Hernández-Lerma, O. (2009). Continuous-Time Markov Decision Processes. Stochastic Modelling and Applied Probability. Springer. ISBN 9783642025464. 
• Meyn, S. P. (2007). Control Techniques for Complex Networks. Cambridge University Press. ISBN 978-0-521-88441-9. Архивирано из оригинала 19. 6. 2010. г. CS1 одржавање: Формат датума (веза) Appendix contains abridged „Meyn & Tweedie”. Архивирано из оригинала 18. 12. 2012. г. CS1 одржавање: Формат датума (веза)
• Puterman., M. L. (1994). Markov Decision Processes. Wiley. 
• Ross, S. M. (1983). Introduction to stochastic dynamic programming (PDF). Academic press. 
• Sutton, R. S.; Barto, A. G. (2017). Reinforcement Learning: An Introduction. Cambridge, MA: The MIT Press. 
• Tijms., H.C. (2003). A First Course in Stochastic Models. Wiley. ISBN 9780470864289. 

Spoljašnje veze

• Learning to Solve Markovian Decision Processes by Satinder P. Singh