Regime Switching Models and the Mental Accounting Framework

Contents

1 Introduction

2 Literature Review
2.1 The Mental Accounting Framework
2.1.1 Markowitz’s MVPT vs. MA framework
2.1.2 Mean-Variance Optimization of Mental Accounts
2.2 Dynamic Investment Management
2.2.1 Components of a Stochastic DP model
2.2.2 Abstraction of the stochastic programming approach
2.2.3 Scenario Generation
2.3 Regime Switching and Markov Chain Models
2.3.1 Components of a Hidden Markov Model
2.3.2 The Three Basic Problems for HMMs
2.3.3 Model Selection
2.4 Gaussian Mixture Models
2.4.1 Definition of Gaussian Mixtures
2.4.2 Gaussian Mixture Models in Stochastic Programming
2.4.3 Moments of a Gaussian Mixture Distribution

3 Mental Accounting with Regime Switching
3.1 Stochastic Programming Problem Formulation
3.1.1 Objective Function
3.1.2 Constraints
3.1.3 Definition of Stages
3.2 Scenario Generation
3.3 Decision Policy
3.4 Backtesting
3.5 Hypotheses

4 Market Data and Methodology
4.1 Market Data
4.2 Methodology
4.3 Missing Data

5 Results
5.1 HMM Calibration
5.1.1 Model Selection
5.1.2 Analysis of the selected model
5.2 Scenario Generation
5.3 Asset Allocation
5.4 Backtesting

6 Summary and Conclusion
A The Original Markowitz Problem
B Solution of the MVPT Problem
C Proof of the Optimality Theorem
D Results

References

List of Figures

2.2 The MVPT Efficient Frontier and Mental Accounting Subportfolios

2.3 The Efficient Frontiers in MA. Each frontier corresponds to a specific threshold return H.

2.4 Stages in a DP problem

2.5 The state space in a DP model

2.6 The set of all states that can be reached from state x in one transition after making decision dx.

3.3 Sample equity curve for three MVPT optimized portfolios.

4.3 Equity Curves for Assets.

5.2 Plot of the Viterbi sequence for the whole data set

5.3 Asset return distributions in different states.

5.4 Expected asset return distribution..

5.5 MVPT Efficient Frontier

5.7 MA Efficient Frontier.

5.9 RoMAD in Subportfolio 2.

5.10 Maximum drawdown MAD in Subportfolio 2

5.11 Standard deviation of returns σ in Subportfolio

5.12 Value at Risk VaR in Subportfolio 2

5.13 Equity Curves for hypothesis Q18.

5.14 Asset Allocation for hypothesis Q18 in MA & MVPT Subportfolio 2

5.15 Implied γ for hypothesis Q18 in MA Subportfolio 2.

5.16 Equity Curves in hypothesis Q24

D.1 Frequency of states in all hypotheses

D.2 Annualized returns in Subportfolio 1

D.3 Annualized returns in Subportfolio 2

D.4 Annualized returns in Subportfolio 3

D.5 Excess annualized returns in Subportfolio

D.6 Excess annualized returns in Subportfolio

D.7 Excess annualized returns in Subportfolio

D.8 Maximum drawdown MAD in Subportfolio 1

D.9 Maximum drawdown MAD in Subportfolio 2

D.10 Maximum drawdown MAD in Subportfolio 3

D.11 RoMAD in Subportfolio 1.

D.12 RoMAD in Subportfolio 2.

D.13 RoMAD in Subportfolio 3.

D.14 Standard deviation of returns σ in Subportfolio 1

D.15 Standard deviation of returns σ in Subportfolio 2

D.16 Standard deviation of returns σ in Subportfolio 3

D.17 Value at risk VaR in Subportfolio 1

D.18 Value at risk VaR in Subportfolio 2

D.19 Value at risk VaR in Subportfolio 3

List of Tables

2.1 Example of risk aversion specification under MVPT and MA.

3.1 Overview of stage lengths used to test the models.

3.2 Risk aversion specification under MVPT and MA

3.4 Hypotheses

3.5 Number of rolling subsets of length Π for all hypothesis

4.1 Data Sources.

4.2 Data Statistics

5.1 Log-likelihood and information criteria for calibrated HMMs.

5.6 MVPT Portfolio Weights

5.8 MA Portfolio Weights.

Chapter 1 Introduction

The pioneering work in modern portfolio of Harry Markowitz in the 1950s [1], for which he was rewarded the Nobel Memorial Prize in Economic Sciences in 1990, has been mainstream until the emerge of behavioral finance theory. It attempts to maximize the expected return of a portfolio for a given amount of risk, or equivalently minimize risk for a given level of expected return by allocating assets in an optimal way. To achieve this goal, Markowitz’s mean-variance portfolio theory (MVPT) mathematically formulated the concept of diversification, the selection of a collection of assets that has a collectively lower risk than any individual asset. However, the assumptions of MVPT are quite strong. For example, asset returns are assumed to be (jointly) normally distributed, risk is defined as the standard deviation of return and all investors are assumed to be rational, risk-averse and try to optimize all of their asset as one portfolio. Despite those limitations, MVPT has been an attractive model for a long time due to its logic and practical application. However, in recent years the basic assumptions of MVPT have been widely challenged, especially by behavioral finance theory.

Some of the main critique points of behavioral finance theory are the empirically observed fat tails in return distributions, as well as the fact that investors are sometimes irrational and occasionally even exhibit risk-seeking behavior.

A model that tries to overcome some of these fallacies is the Mental Accounting (MA) framework developed by Das et al. in 2010 [2]. It is especially appealing because the mathematical equivalence of MA and MVPT can be shown. A big problem in the practical use of MVPT is the specification of the investor’s risk aversion coefficient. There is vast literature by financial institutions establishing countless surveys to estimate this parameter required for the asset allocation optimization. MA solves this problem by introducing a required threshold return the investor would like to achieve with his portfolio and a probability of failing to reach this threshold that is acceptable to him. This specification of risk aversion is a lot more intuitive for an investor and removes the bias of surveys. Both the MVPT and the MA model assume stationary and normally dis- tributed asset returns, which is in contrast to the well established fact that return distributions exhibit fat tails and that correlations with other assets change over time [3-5]. To extend these models from their stationary setting to a dynamic one that allows for a more flexible return distribution specification, this thesis employs dynamic programming. Dynamic programming is a theory concerned with problems where an individual makes a sequence of decisions over a number of stages in time and hence acknowledges the dynamic nature of financial markets by design. It was first formalized by Richard Bellman in the late 1940s [6] and has since been applied successfully to various fields, such as resource allocation [7, 8], shortest path problems [9-13], asset acquisition [14, 15] and information collection [16, 17]. Its application to asset allocation, however, has been rather limited so far.

An essential part of a dynamic program is scenario generation for the asset returns. This thesis uses Regime Switching models, and more specifically Hidden Markov Models and Gaussian Mixture Models, to generate time-varying scenarios for asset returns. The main theoretical justification for the use of Markov processes in general and of the normal distribution in particular stems from the Efficient Market Hypothesis (EMH). The EMH is the result of empirical investigations by Eugene Fama in the 1960s [18, 19] and was widely accepted until the 1990s, when behavioral finance theory became mainstream. It is concerned with the efficiency of information transmission in financial markets. An efficient market would imply that information is priced into the market immediately as it becomes available. There are several forms of the EMH, ranging from the weakest to the strongest form. At the one extreme, the weak form EMH postulates that all past prices and volume information are fully reflected in the current market price and a trader cannot generate above average risk-adjusted returns using technical analysis or other tools. At the opposite extreme, the strong form of the EMH states that prices reflect all public and private information, implying that traders cannot profit even if they possess insider knowledge. In the middle of these two extremes lies the semi-strong form of the EMH, according to which a trader cannot outperform the market by using public information but might be able to profit from private information. Empirical studies commonly support the weak form of the EMH, but are often less specific about the other forms. By ruling out the possibility of designing profitable trading strategies based on past prices and volume, the weak form EMH implies that asset returns are conditionally independently distributed. This is another way of postulating that today’s returns have no effect on tomorrow’s returns, which renders the exact definition of a Markov process as a “memoryless” process. The main goal of this thesis is to combine all of these concepts into a unified framework, evaluate its feasibility and performance, and to perform an analysis of the most common pitfalls and practical considerations. This unified framework uses both the MVPT and MA approach for asset allocation, but at the same time allows for dynamic and fat-tailed distributions of asset returns. It is implemented in approximately 1200 lines of efficient MATLAB code, which is publicly available at https://github.com/FelixAndresen/RSMentalAccounting. The application is programmed in a way that it is readily expandable to a larger number of assets and other investment approaches. The framework is also independent on the choice of assets, which is why the choice of assets for the illustration of the thesis findings was based on the availability of data.

The thesis is structured as follows. Chapter 2 reviews the current literature and theoretical concepts. More specifically, Chapter 2.1 introduces the Mental Accounting framework and the connections and differences to Markowitz’s Mean Variance Portfolio Theory. In Chapter 2.2 the most important concepts of dynamic investment management and stochastic programming are introduced. Chapter 2.3 discusses the regime switching models used to generate scenarios for the stochastic programming approach and the important topic of model selection. Chapter 2.4 gives an overview of Gaussian Mixture Models which present a tool to create the expected distribution used to optimize the asset allocation. In Chapter 3 all the pieces from the theoretical parts are brought together to formulate the dynamic programming models and the hypotheses to be tested in the thesis. The market data used to carry out the analysis is discussed in Chapter 4, as well as some necessary methodology on how to calculate, aggregate and interpret asset returns.

Chapter 5 presents exemplary and illustrative results, and discusses the strengths and weaknesses of the MVPT and MA investment approaches. The thesis closes with a summary and conclusion in Chapter 6.

Chapter 2 Literature Review

2.1 The Mental Accounting Framework

Markowitz’s mean-variance portfolio theory (MVPT) [1] has been an attractive model due to its logic and practical application. Choosing portfolios based on their overall expected return and risk seems logical and mean-variance optimiza- tion is a quick tool to draw the efficient frontier. The mean-variance efficient frontier represents an investor’s production function, while the corresponding consumption function depends on the utility of expected return and and risk.

However, expected returns and risk are only stations on the way to an investor’s ultimate portfolio consumption goals. Such goals could be a secure retirement, college education for the children, or simply getting rich. Neither does MVPT help in the creation of a portfolio that is best at satisfying any of these goals, nor does it help identifying the investor’s attitude towards risk.

These ultimate portfolio consumption goals are central to Shefrin and Statman’s behavioral portfolio theory (BPT) [20]. In contrast to MVPT, BPT investors do not consider their portfolios as a whole, but as collections of mental accounting subportfolios. Every subportfolio is associated with a goal, which in turn has a threshold level of return. The probability of failing to reach this threshold level of return is the risk of the subportfolio. A subportfolio with the same expected return dominates another subportfolio when it a exhibits a lower probability of failing to reach the required threshold. For each mental account, the trade-off between expected return and the risk can be visualized in an efficient frontier and investors usually choose subportfolios from this efficient frontier set. It is important to point out that MVPT investors are by assumption always risk averse, while it can be optimal for BPT investors to be risk seeking in certain circumstances.

Features of the Mental Accounting framework (MA) developed by Das et al. [2] include a mental accounting structure of portfolios, a definition of risk as the probability of failing to reach the threshold level in each mental account, and attitudes toward risk that vary by account. However, MA does not include the

BPT possibility of investors who are risk-seeking in certain mental accounts. MA is based on two assumptions: First, investors are better at stating their goal thresholds and probabilities of reaching thresholds in MA (the consumption view) than specifying their risk-aversion coefficients in MVPT (the production view). Second, investors are better at stating thresholds and probabilities for subportfolios (e.g., retirement, education, speculation, etc.) than for an aggregate portfolio. Reference [2] show that better problem specification leads to superior portfolios and that the MA framework results in no loss of MVPT efficiency when short selling is permitted. With a short selling constraint, subportfolio optimization results in a few basis points loss in efficiency relative to an aggregate portfolio optimization. This loss was shown to be small compared to the loss from inaccurate risk aversion specification and declining as investors become increasingly risk averse.

2.1.1 Markowitz’s MVPT vs. MA framework

Das et al. established that "portfolio optimization over two moment distributions where wealth is maximized subject to reaching a threshold level of return with a given level of probability (i.e., the MA problem) is mathematically equivalent to MVPT optimization" [2]. These two problems are outlined below:

Mean Variance Portfolio Theory MVPT minimizes the variance of a portfolio σ[2] = minw wTΣw, subject to achieving a specified level of expected return E = wTµ and being fully invested (i.e. wT1 = 1), where w ∈ R n is a vector of portfolio weights for n assets, Σ ∈ R n×n is the covariance matrix of returns of the assets and µ ∈ R n is the vector of n expected returns. Further, 1 denotes a vector consisting of ones in each component, 1 = (1, 1, . . . , 1)T ∈ R n. Carrying out the optimization for varying E produces a set of all mean-variance efficient portfolios {w(E)}, which traces out an efficient frontier. This is displayed in Figure 2.2 using the numerical example presented in Ref. [2]. It is given by the following mean vector and covariance matrix of asset returns,

illustration not visible in this excerpt

The investor’s risk aversion in the different subportfolios are shown in Table 2.1.

illustration not visible in this excerpt

Table 2.1: Example of risk aversion specification under MVPT and MA.

Mental Accounting

MA considers a threshold return H and aims at maximizing the expected return E = maxw wTµ, subject to a specified maximum probability of failing to reach the threshold (i.e., Prob[r(p) < H] < α, where r(p) denotes the return of the portfolio and α is the maximum probability of failing to reach the threshold). To illustrate how investor goals can be subdivided into subportfolio goals in MA, consider an investor who would like his retirement portfolio with current value of P0 to accumulate to a threshold value of PT after T years. This implies a threshold return per year of (PT /P0)[1]/T − 1 ≡ H and a probability of failing to reach this threshold denoted α. By keeping H fixed and solving the problem for different levels of α, the corresponding maximized expected return levels (wTµ) are obtained. Plotting expected returns against α for fixed H results in the MA portfolio frontier, as displayed in Figure 2.3. Here every frontier corresponds to a specific threshold return

illustration not visible in this excerpt

Figure 2.2: The MVPT Efficient Frontier and Mental Accounting Subportfolios.

illustration not visible in this excerpt

Figure 2.3: The Efficient Frontiers in MA. Each frontier corresponds to a specific threshold return H.

Moreover, Ref. [2] demonstrates the mathematical equivalence of MVPT and MA, and provides a linkage to the concept of Value at Risk (VaR). This equivalence has three consequences:

1. MA optimal portfolios always lie on the MVPT efficient frontier.
2. The constraint of each MA problem specifies a mapping onto an “implied” risk-aversion coefficient in the MVPT problem.
3. Many MA portfolios may map onto a single MVPT efficient portfolio.

The following section shows how to map a MA portfolio into a MVPT efficient portfolio.

2.1.2 Mean-Variance Optimization of Mental Accounts

The general definition of the MVPT problem is given by

Definition 2.1.1 (General Formulation of the MVPT problem). Minimize

illustration not visible in this excerpt

subject to the constraint

illustration not visible in this excerpt

and the full investment constraint

illustration not visible in this excerpt

Here the risk-aversion coefficient is denoted as γ, Var(p) stands for the variance of the return of portfolio p, and the expected return on the portfolio E[r(p)] is equal to a level E. Please refer to Appendix A for a closed form solution to this problem. Each solution corresponds to a portfolio on the efficient frontier.

To illustrate the connection between MVPT and MA more explicitly, the following alternative formulation of the Markowitz problem is used.

Proposition 2.1.2 (Alternative Formulation of the MVPT problem). The following optimization problem is equivalent to the one in Definition 2.1.1. Maximize

illustration not visible in this excerpt

subject to the full investment constraint

illustration not visible in this excerpt

for different values of risk aversion γ > 0. Investors choose portfolio weights w = (w1,w2,...,wn)T for n assets, which have a mean return vector µ ∈ R n and a return covariance matrix Σ ∈ R n×n. The full statement of the MVPT problem can then be formulated as follows:

illustration not visible in this excerpt

subject to the full investment constraint

illustration not visible in this excerpt

The derivation of this optimization problem in closed form can be found in Appendix B, and the solution is given by

illustration not visible in this excerpt

This formulation uses the the risk-aversion coefficient γ > 0 to determine the efficient frontier, while the general formulation uses the expected return E. The dif- ficult part is not the charting of the mean-variance efficient frontier, but stating an investor’s risk aversion γ quantitatively. Financial planning firms have developed extensive questionnaires for this task, but the results often remain unsatisfactory.

Das et al. argue that risk attitudes are easier to specify for subportfolios than for an aggregate portfolio and that it is easier for investors to state their threshold levels for each subportfolio and maximum probabilities of failing to reach them than their risk-aversion coefficients. This means that an MA investor specifies that the return on one of his subportfolios should not fall below a level H with probability α, Prob[r(p) < H] ≤ α. (2.1.5)

illustration not visible in this excerpt

In their paper, Das et al. further assume that portfolio returns are normally distributed, which means that an investor tries to choose the best portfolio in the mean-variance space in order to satisfy the following inequality constraint,

illustration not visible in this excerpt

Here Φ(·) denotes the cumulative standard normal distribution function. They impose normality for convenience because “it is a common practical choice” [2]. This assumption will be relaxed in this thesis. Substituting the optimal weights given by Eq. (2.1.4) into Eq. (2.1.6), the resulting equation can be solved for the “implied” risk aversion γ of the investor for this particular mental account. Since the constraint (2.1.6) becomes an equality when optimality is achieved, the solution to the investor’s implied risk aversion γ and the optimal weights w(γ) are given by

illustration not visible in this excerpt

where

illustration not visible in this excerpt

By substituting the latter equation into Eq. (2.1.7), the implied risk aversion γ is easily obtained. Note that the optimal portfolio weights w(γ) are nonlinear in the risk-aversion coefficient.

By stating the MA preferences for each subportfolio through the parame- ters (H, α), an investor is implicitly stating his risk preference γ. This allows specification of the implied risk aversion for each mental account as a mapping function γ(µ, Σ; H, α). Then the the standard MVPT problem can be solved using Eq. (2.1.4).

2.2 Dynamic Investment Management

In environments where earlier decisions affect the feasibility and performance of later decisions, the sole optimization of the immediate impact is usually sub- optimal for the overall process. In these types of multi-stage decision problems one tries to find optimal strategies by considering current and future decisions simultaneously, a task which is usually accomplished through the use of dynamic programming (DP). Dynamic programming models and methods are based on Bellman’s Principle of Optimality [6], which states that in order to attain opti- mality in a sequential decision process, all the remaining decisions after reaching a particular state must be optimal with respect to that state. Put differently, a strategy that makes a suboptimal decision in any one of the intermediate stages cannot be optimal for the overall problem. This allows the formulation of recursive relationships between the optimal strategies of successive decision stages which form the backbone of DP algorithms. DP has been applied to a large variety of problems ranging from resource allocation [7, 8], shortest path problems [9-13], asset acquisition [14, 15] up to information collection [16, 17]. They can also be found throughout the financial literature. Some of the best known and most common examples are the tree or lattice models (binomial, trinomial, etc.), which are often used to describe the evolution of security prices, interest rates, volatilities, etc., and the corresponding pricing and hedging schemes [21, 22]. However, their application to asset allocation has been limited so far.

Elements of DP models include decision stages, a set of possible states in each stage, transitions from states in one stage to states in the next, value functions that measure the best possible objective values that can be achieved in each state, and the recursive relationships between value functions of different states. The decision-maker needs to specify the decision he would make in order to reach that state, and the collection of all decisions associated with all states forms the policy or strategy of the decision-maker. Transitions from the states of a given stage to another state of the next may happen as a result of the actions of a decision-maker, or as a result of random external events, or as a combination of the two. The DP problem is called deterministic if a decision at a particular state uniquely determines the transition state. If there are probabilistic events affecting the transition state, the DP problem is called stochastic.

Stochastic programming models are commonly used to solve stochastic DP problems. They are able to solve optimization problems involving uncertainty, which is a necessity since real world problems almost invariably include parame- ters which are unknown at the time a decision should be made. A straightforward approach to solve this class of problems would be to allow the unknown parame- ters to take on some given set of allowed values, seek a solution that is feasible for all possible parameter choices, and optimizes a given objective function. Stochastic programming models are similar in style, but try to take advantage of the fact that probability distributions governing the parameters are known or can be estimated. Such models usually work well in settings where decisions are made repeatedly in essentially the same circumstances with the objective of finding a decision that performs well on average. An example for such a setting would be the design of routes for daily deliveries with random demand. Using collected data one can estimate probability distributions (e.g. of demand) and find a policy that is feasible for (almost) all possible parameter realizations and optimizes the expectation of the objective function. An illustrated solution to such a problem can be found in Ref. [23].

2.2.1 Components of a Stochastic DP model

In the following section an abstraction of the stochastic programming approach and its assumptions is presented, which closely follows Ziemba and Vickson [24]. The basic components of a DP model are:

1. Stages

An individual makes an investment decision at the beginning of each stage t ∈ τ = {1,2,...}. A termination time for the process is not set in this framework, since it should be able to solve problems set on both finite or infinite horizons. An illustration of the stages of a dynamic programming problem is shown in Figure 2.4.

2. States and the Markovian Regeneration Property

Ωt ∈ R k denotes the state space at time t and is a subset of a k-dimensional Euclidian space representing the set of all possible states that might occur

illustration not visible in this excerpt

Figure 2.4: Stages in a DP problem.

at stage t. The element ω ∈ Ωt is the state at stage t. Ω = {(ω, t)|ω ∈ Ωt, t ∈ T} is the space-time set and x ∈ Ω is called a state. An illustration of the state space in a dynamic programming problem is shown in Figure 2.5. States x ∈ Ω satisfy the Markovian regeneration property:

Proposition 2.2.1 (Markovian regeneration property). A state is a regener- ation point for the process in the sense that costs and rewards associated with current and future decisions only depend on current states but not on prior states.

The Markovian regeneration property is necessary to use Bellman’s Principle of Optimality [6]. Note, however, that many seemingly non-Markovian models can be mapped onto a Markov model by making further assumptions.

3. Decisions and policies

Let Dx denote the set of all decisions available in state x ∈ Ω. With Xt ∈ Ωt being the stochastic process tracking the state of the process at time t, the trajectory of the state process {Xt|t ∈ τ} evolves through time in response to to individual decisions dx ∈ Dx made at state x. When the process Xt is in state x = (ω, t), the selected decision dx determines a (possibly defective) probability distribution function P(·; x, dx ) on Ωt, where

illustration not visible in this excerpt

Figure 2.5: The state space in a DP model.

illustration not visible in this excerpt

is called the probability transition function. A defective probability dis- tribution is a distribution with total probability mass of less than 1. The expression 1 − limz→∞ P(z; x, dx) gives the probability that the process ter- minates at states x, which is equivalent to a bankruptcy in an investment model.

A policy δ is an ordered collection of one decision for each state in Ω. The collection of all policies is denoted by the policy space Δ. Hence the policy space Δ is the Cartesian product of all decision sets

illustration not visible in this excerpt

Here, δx denotes the element of δ related to the decision made in state x. It is important to point out that the process {Xt|t ∈ τ} in response to a policy δ is Markovian.

4. Cost or Reward Function

In a DP model the objective of a decision-maker is either to minimize a given cost function, or maximize a given reward function. This section focuses on the latter, but is easily applicable to the first objective by changing signs. The reward function in response to a policy δ given the space-time process started in state x at time 0 is denoted by Vδ(x). The optimal return is then defined as

illustration not visible in this excerpt

assuming that the maximum is attained by a policy δ∗ ∈ Δ such that

illustration not visible in this excerpt

The structure of the process is assumed to fulfill the monotonicity assump- tion:

Assumption 2.2.2 (Monotonicity Assumption). Let δ, γ ∈ Δ be two policies such that for some state x, δx = γx and Vδ(y) ≥ Vδ(x), ∀y ∈ Ωx. Then Vδ(x) ≥ Vγ(x).

This assumption states that if two policies δ and γ have the same decision in a state x and the reward generated by δ is no less than the reward generated by γ for each state γ ∈ Ωx, then the reward generated by policy δ at x is no less than the reward generated by γ at x.

5. State Transitions

Let T(x, dx) be the set of all states that can be reached from state x in one transition after making decision dx. State z ∈ T(x, dx ) is formally defined if and only if

illustration not visible in this excerpt

which means that the neighborhood of z has a positive probability mass. In a deterministic DP the set T(x, dx ) will contain exactly one element because the model is designed to move from one state to another state with certainty, given a decision dx. In a stochastic DP the set T(x, dx ) will contain multiple states. An illustration of the state transition and the set of all possible states is given in Figure 2.6. The evolution process is also required to be terminating, which means that the process is completed in a fixed and finite number of transitions nx for any initial state x, where nx may depend on the state x. Formally:

Assumption 2.2.3 (Termination Assumption).

For each x ∈ Ω and dx ∈ Dx, the defect 1 − P(∞; x, Dx ) is either 0 or 1. Moreover, for each x ∈ Ω there exists nx < ∞ such that Tnx (x) = ∅.

This implies that for each x ∈ Ω, nx is the smallest positive integer such that Tnx (x) = ∅, ∀n ≥ nx.

illustration not visible in this excerpt

Figure 2.6: The set of all states that can be reached from state x in one transition after making decision dx.

[...]