A Nash-Type Fictitious Game Framework to Time-Inconsistent Stochastic Control Problems
Yuan-Hua Ni, Binbin Si, Xinzhen Zhang
TL;DR
This paper introduces a Nash-type fictitious game framework to address time-inconsistent stochastic control problems, providing a new auxiliary-variable-based mechanism for deriving time-consistent policies and extending to nonzero-sum stochastic games.
Contribution
The paper proposes a novel fictitious-game framework for time-inconsistent control, characterizes equilibrium policies via Riccati equations, and applies it to portfolio selection.
Findings
Framework effectively derives time-consistent policies.
Necessary and sufficient conditions for equilibrium existence.
Numerical simulations demonstrate the framework's efficiency.
Abstract
In this paper, a Nash-type fictitious game framework is introduced to handle a time-inconsistent linear-quadratic optimal control. The Nash-type game in this framework is called fictitious as it is between the decision maker (called real player) and an auxiliary control variable (called fictitious player) with the real player and fictitious player looking for time-consistent policy and precomitted optimal policy, respectively. Namely, the fictitious-game framework is actually an auxiliary-variable-based mechanism where the fictitious player is our particular design. Noting that the real player's cost functional is revised in accordance with that of fictitious player, the equilibrium policy of real player is called an open-loop self-coordination control of original linear-quadratic problem. As a generalization, a time-inconsistent nonzero-sum stochastic linear-quadratic dynamic game is…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsRisk and Portfolio Optimization · Optimization and Variational Analysis · Advanced Control Systems Optimization
A Nash-Type Fictitious Game Framework to Time-Inconsistent Stochastic Control Problems††thanks: This work is supported in part by the National Natural Science Foundation of China (61773222, 11871369, 61973172, 62173191).
Yuan-Hua Ni Binbin Si Xinzhen Zhang College of Artificial Intelligence, Nankai University, Tianjin 300350, P.R. China. Email: [email protected]. College of Artificial Intelligence, Nankai University, Tianjin 300350, P.R. China. Email: [email protected] of Mathematics, Tianjin University, Tianjin 300352, P.R. China. Email: [email protected].
Abstract: In this paper, a Nash-type fictitious game framework is introduced to handle time-inconsistent linear-quadratic optimal control problems. The Nash-type game in this framework is called fictitious as it is between the decision maker (called real player) and an auxiliary control variable (called fictitious player) with the real player and fictitious player looking for time-consistent policy and precomitted optimal policy, respectively. Namely, the fictitious-game framework is actually an auxiliary-variable-based mechanism where the fictitious player is our particular design. Noting that the real player’s cost functional is revised in accordance with that of fictitious player, the equilibrium policy of real player is called an open-loop self-coordination control of original linear-quadratic problem. As a generalization, a time-inconsistent nonzero-sum stochastic linear-quadratic dynamic game is investigated, where one player is to look for precommitted optimal policy and the other player is to search time-consistent policy. Necessary and sufficient conditions are presented to ensure the existence of open-loop equilibrium of the nonzero-sum game, which resort to a set of Riccati-like equations and linear equations. By applying the developed theory of nonzero-sum game, open-loop self-coordination control of the linear-quadratic optimal control is fully characterized, and multi-period mean-variance portfolio selection is also investigated. Finally, numerical simulations are presented, which show the efficiency of the proposed fictitious-game framework.
Key words: time inconsistency, stochastic linear-quadratic problem, dynamic game, precommitted policy, time-consistent policy
1 Introduction
Dynamic programming is a fundamental and powerful approach to solving optimal control problems; the basic idea is to consider a family of problems with different initial times and states, and to establish relationships among these problems. Bellman’s principle of optimality is the core of this approach which states using Bellman’s words [5] as “An optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision.” This property is termed as the time consistency of optimal control. However, in many practical situations, the time-consistency fails quite often and the corresponding optimal control problems are time-inconsistent.
Actually, there are several factors that ruin the time consistency of optimal control. The first one is the nonlinear terms of conditional expectations of state/control that appear in the objective functional. As there is no nonlinear version of the tower property of conditional expectation, the controller at different time instants is facing with different objectives, which are not consistent with the global objective. Therefore, the time inconsistency comes from the conflicts between global optimal control on the lifetime horizon and local optimal control on the tail time horizon. Such a kind of time-inconsistent problems are classified as mean-field optimal controls, which have gained considerable attention during the last few years [33, 51]. Another factor is the non-exponential discounting in objective functional, which does not possess the property of group or separability any longer. Though exponential discounting is of great importance to model people’s time preference [38], empirical researches over the last half century have documented the inadequacy of constant discount rate. Among others, hyperbolic discounting is a known anomaly and is often used to describe the case with a declining discount rate [18]. Further, it is hypothesized [42] that people are born with tendency to overvalue current consumption and that more discounting occurs between the present and the near future than between periods in the more distant future, namely, the discounting function in the objective functional is non-exponential. The aforementioned two factors that ruin the time consistency reflect people’s risk preferences and time preference in some nontraditional ways, both of which are of the phenomenon of “changing tastes” in intertemporal choices [4], namely, today’s preference conflicts with tomorrow’s preference.
1.1 Existing methodologies on handling the time inconsistency
To handle the time inconsistency, there are several different approaches in existing literature and a rule of selecting the preferred solution is called as a choice mechanism [4]. The first one is the precommitment choice for which the initial policy is implemented on the lifetime horizon. This approach neglects the time inconsistency, and the optimal policy is optimal only when viewed at the initial time. The second mechanism is naive choice or myopic choice: at each time instant a naive agent embarks on the option that currently seems best, namely, this agent sticks to the local objective and completely ignores the global interest. However, the naive policy makes no sense of optimality, and simple example [4] shows that it might be the worst one of all the policies viewed from the initial time instant.
1.1.1 Strotz’s time-consistent solution
Another mechanism is sophisticated/time-consistent choice proposed by Strotz [42]. In the viewpoint of Strotz, the decision maker at different time instants is regarded as different selves, and the time inconsistency suggests a conflict between different these selves. At any time instant the current self takes account of future selves’ decisions, and the equilibrium of this intertemporal game is called a sophisticated policy, or a time-consistent policy. Inspired by Strotz’s idea, hundreds of works have sought to tackle practical problems in economics and finance; see, for example, [11, 12, 16, 17, 21, 26, 28, 32, 36] and the references therein. Moreover, accompanying the appearance of time-inconsistent mean-field optimal control, recent years have witnessed the rapid progresses of extending Strotz’s idea in the theoretical control community [7, 8, 23, 24, 31, 33, 37, 44, 45, 46, 47, 48, 49, 50, 51]; in particular, the open-loop time-consistent control, feedback time-consistent strategy and mixed time-consistent solution are elaborately studied. It should be noted that Strotz’s solution is essentially a closed-loop time-consistent strategy. So far, the sophisticated policy and precommitted policy are two extreme solutions; namely, the sophisticated policy recovers the time consistency and ignores the global optimality, while the precomitted solution does not care about the time consistency and just pays attention to the global optimality on the lifetime horizon.
1.1.2 Self-control of Thaler and Shefrin
Different from Strotz’s formulation and to balance the global optimality and time consistency, the economists Thaler and Shefrin [43] introduce a two-dimensional self-control model: the individual at any instant in time is assumed to be both a farsighted planner and a myopic doer. This division into conflicting subselves is how psychologists think about self-control, and the notion of self-control is paradoxical without it. The doer at each moment in time exists only for one period and is completely selfish, or myopic [43], namely, the objective of each doer is independent of past and future variables that are concerned. On the contrary, the planner is concerned with the lifetime objective, which is derived from the objectives of all the doers. Interestingly, pointed out by [43] and due to the myopic nature of the doers, the conflict between the planner and doers is fundamentally similar to the agency relationship between the employer/principal and employees/agents of a firm. In fact, this two-self model to understand the savings behavior of individuals and households is one of four Thaler’s contributions in behavioral economics to win 2017 Nobel Prize [1]. For the recent progresses on self-control, we are referred to [6, 13, 14, 19, 20, 22, 35] and references therein.
Actually, the idea of two-self model can be traced back to the work of Adam Smith [41] in 1759; and [43] is the first systematic and formal treatment of a two-self economic man, which integrates economics with psychology. A key feature of this planner-doer modelling is that the planner is also allowed to bear some influence on doers’ behavior. For this, the doers are given the discretion to either modify their preferences or alter the incentives (rewards, punishments, etc). Specifically, through incorporating the costly control of a “preference modification parameter” (selected by the planner) into the doers’ utilities, the behaviors of the planner and doers can be mutually influenced, and the planner’s utility is simultaneously modified. By finding the equilibrium of this intrapersonal game, a balance between the lifetime objective and myopic objectives is achieved. Noting that the planner does not actually consume, the policy of this game selected by the doers is the one that is executed by the individual.
1.1.3 Two-tier game framework of Cui, Li and Shi
Note that the doers’ local objectives of standard self-control schemes are fully myopic, namely, there are no conflicts among different local objectives. To meet the time-inconsistent stochastic decision problems with conflicting non-myopic local objectives and non-expectation operators, Cui, Li and Shi propose a two-tier planner-doer game framework [13] to reconcile the global and local interests, where a sequential game among the doers is involved that significantly extends existing self-control schemes [19, 20, 22, 35, 43]. Through commitment by punishment, the proposed mechanism of [13] revises the original preference of each individual doer by adding a penalty term, while the expected total penalty in turn modifies the planner’s preference. Then, both a sequential game among the doers in the lower tier and a leader-follower game between the planner and doers in the upper tier are constructed. Given any planned policy, the best-response policies of the doers form a Nash equilibrium of the low tier game and are time-consistent on the lifetime horizon. Therefore, the two-tier game is indeed a game between precommitted policy and time-consistent policy, and the doers’ equilibrium time-consistent policy of this game is called a self-coordination policy of the original time-inconsistent decision problem.
Furthermore, the proposed self-coordination mechanism is applied to the dynamic mean-variance portfolio selection, and an explicit self-coordination policy is obtained together with a detailed sensitivity analysis [13]; this enable investors to understand the trade-off between global and local interests and coordinate among various selves. To the best of the authors and within the realm of dynamic games, the work [13] is the first to study the game between precommitted policy and time-consistent policy, which clearly merits further investigations. The followings are some of questions that might be studied.
- i).
There exist several notions on time-consistent equilibrium solution in existing literature, such as the open-loop equilibrium control, closed-loop equilibrium strategy and mixed equivalent equations [23, 31, 51]. Note that the best-response time-consistent policy of the doers is closed-loop. So, it is desirable to study other types of best-response time-consistent policy.
- ii).
If we go beyond, the game between the planner and doers can be settled within more general framework of games between precommitted policy and time-consistent policy, whose study must enrich the game theory. In particular, it is desirable to study the Nash-type games, namely, the precommitted policy and time-consistent policy have equal status.
- iii).
Linear-quadratic (LQ, for short) optimal control is pioneered by Kalman in 1960s and is now a classical yet fundamental problem in control theory. Though general time-inconsistent decision problems has been considered in [13, 14], the pretty structure of LQ problem has not been fully explored within the games between the precommitted policy and time-consistent policy.
1.2 Nash-type fictitious game framework
1.2.1 The framework
In this paper, a time-inconsistent LQ optimal control is studied via a method of Nash-type fictitious game. Specifically, consider the system
[TABLE]
where , , and , are deterministic. Letting , the noise process is assumed to be a vector-valued martingale difference sequence defined on a probability space with
[TABLE]
where , are assumed to be deterministic. in (1.4) is the conditional mathematical expectation with respect to , and is understood as . Let and be defined as
[TABLE]
and of (1.3) belongs to . The objective functional is
[TABLE]
where , are deterministic symmetric matrices of appropriate dimensions. Then, the LQ problem is stated as follows.
Problem (LQ). Letting and , find a such that
[TABLE]
Problem (LQ) is time-inconsistent as the objective functional (1.2.1) contains nonlinear terms of conditional expectation and the weighting matrices of (1.2.1) depend on the initial time. of (1.8) is called an open-loop precommitted optimal control, or simply precommitted solution/policy, for the initial pair , which totally adheres to the global interest on the lifetime horizon . Noting that neglects the time inconsistency, the following notion yet pays attention to the open-loop time-consistent solution of Problem (LQ).
Definition 1.1**.**
A control is called an open-loop time-consistent equilibrium control of Problem (LQ) for the initial pair , if for any and any ,
[TABLE]
Here, and are the restrictions of on and , respectively; and is given by
[TABLE]
Since Strotz’s work, time-consistent solutions have gained much attention in the areas of economics, finance etc. It should be noted that time-consistent policy and precommitted optimal policy are two extreme and irreconcilable solutions for time-inconsistent optimal control. More precisely, time-consistent policy recovers the time consistency and ignores the global optimality, while the precomitted solution does not care about the time consistency and just pays attention to the global optimality on the lifetime horizon. In this paper, we will extend the notion of time-consistent solution in order to strike a balance between the time consistency and global optimality. The main idea is to introduce an auxiliary fictitious player that plays games with the decision maker in Definition 1.1 (this is called a real player throughout this paper).
The Nash-type fictitious game framework of this paper is divided into the following three steps.
Step 1. Introduce an auxiliary control variable , which is called a fictitious player throughout this paper. The cost functional of is with the internal state
[TABLE]
namely,
[TABLE]
Different from the real player ( of Definition 1.1), the fictitious player would like to find a precommitted optimal solution.
Step 2. Modify the cost functionals of and to
[TABLE]
with being symmetric and . Here, the inner states in and are given, respectively, by (1.12) and (1.15).
Step 3. Solve the fictitious game:
- Problem (LQ)g. Find such that
[TABLE]
hold with computed via
[TABLE]
Here, is called an open-loop self-coordination control of Problem (LQ) for the initial pair and .
Remark 1.2**.**
* and are called the punishment direction and punishment intensity, respectively. Note that modifying the objective functionals (1.16) (1.17) of Step 2 is similar to that of [13]. Actually, it is [13] that motivates the study of this paper, and the term “self-coordination policy” is introduced firstly by [13] that is of closed loop indeed. As the formulation of this paper is somewhat similar to that of [13], the “self-coordination” is borrowed here and the equilibrium policy of real player is called an open-loop self-coordination control of Problem (LQ). Furthermore, Problem (LQ)g is called a fictitious game as it is a game between a real player and a fictitious player.*
1.2.2 Motivation
The Nash-type fictitious game framework can be viewed as an auxiliary-variable method. The of (1.3) is the real controller and its policy is the one that is actually performed, and the fictitious player is an auxiliary control variable with system equation (1.15). On the fictitious game of Problem (LQ)g, the real player’s policy is obtained that is called an open-loop self-coordination control of Problem (LQ). Namely, all the derivations from (1.15) to (1.22) are our particular design in order to obtain the open-loop self-coordination control. Furthermore, the method of fictitious game differs from the two-tier planner-doer game [13] in the following two points. Firstly, the two-tier game framework is a self-control scheme, where the decision maker at any instant is assumed to have conflicting subselves—planner and doer. On the contrary, there is only one real controller in the fictitious game framework, and the fictitious player is an auxiliary variable. Secondly, the game of [13] is of lead-follower type, where the planner is a leader and the doers are the followers. In contrast, the game between real player and fictitious player of this paper is Nash-type, namely, the real player and fictitious player have equal status.
Auxiliary-variable method is somewhat usual in controller design [15, 29]; in this case, some internal auxiliary variables are introduced by which the concerned controllers can be constructed. In the fictitious game framework, the fictitious player is an internal auxiliary control variable, and through the fictitious game the real player’s policy is obtained. Furthermore, the best-response policies of real player and fictitious player are time-consistent and globally optimal, respectively. In other words, the fictitious game is indeed a game between a real time-consistent policy and a fictitious precommitted policy, which shares some similarity with that of [13]. To the best of the authors, the work [13] is the first to study the game between time-consistent policy and precommitted policy, where the conflicting subselves have hierarchical status. Under the idea of introducing auxiliary control variable, it is possible to conduct a fictitious Nash-type game between time-consistent policy and precommitted policy. Similarly to standard Nash equilibrium, the Nash-type equilibrium of fictitious game is also non-cooperative, namely, if any one player stays at its equilibrium policy, the other player’s equilibrium policy is optimal in the sense of (1.18) or (1.19).
Furthermore, on the status of precommitted policy and time-consistent policy in the fictitious game, let us go back to the pioneer work [42] too. Though consistent planning is proposed firstly by [42], it seems that no prejudice on precommitted policy is found in [42]. If the conflicts between the global interest and local interests are recognized, it “may be solved either by (a) a strategy of precommitment, or (b) a strategy of consistent planning”; this is noted in the Summary of [42]. More interestingly, in the very recent year, Caliendo and Findley [9] present some positive results about that the precommitted policy acts better than time-consistent policy: “In some prominent, well-studied examples with intertemporal tradeoffs (like the choice between investing in a project now or later, doing an unpleasant task now or procrastinating it until later, and eating a cake), we find that the commitment allocation can multiself Pareto dominate the non-cooperative equilibrium allocation if the number of time-dated selves exceeds a low threshold.” Here, the commitment allocation and non-cooperative equilibrium allocation are the precommitted policy and time-consistent policy, respectively. Besides, commitment mechanism is widely accepted in the fields such as decision science, economics and finance. To mention a few, see, for example, [3, 10, 25, 27] and references therein. Namely, though nowadays consistent planning has gained much attention in the control community, the above mentioned sample of papers provides many positive evidences of studying precommittmed policy, which ought not to be ignored for its practical values in some situations. Therefore, for the general study, it is reasonable to treat the precommitted policy and time-consistent policy equally; this is the case of [42]. Hence, to balance the global optimality and time consistency, an alternative way to that of [13] is conducting a Nash-type game between time-consistent policy and precommitment policy. This is another motivation of the fictitious game of this paper.
Additionally, concerned with standard nonzero sum games, Nash solution and Stackelberg solution are two standard noncooperative equilibrium solutions in the sense that no player can achieve an improvement if she attempts to deviate from her strategies. Nash solution ensures simultaneously that at the same time each player will not benefit from changing their strategy, and Stackelberg solution is in a sequential manner. Though it is established in [40] that the leader in the Stackelberg solution is at least as good and possibly lower cost than in the Nash solution, yet nothing can be said about the follower who may or may not do better than the Nash solution [39]. Hence, for the self-coordination policy of [13] and open-loop self-coordination control of this paper, generally it is hard to say that one acts better than the other one. Interestingly, according to the examples of Section 4, the scheme of [13] looks for self-coordination policy between open-loop precommitted optimal control and open-loop time-inconsistent equilibrium control, while to some extent this paper goes beyond open-loop precommitted optimal control and open-loop time-consistent equilibrium control. Therefore, our formulation adds a new dimension to handle time-inconsistent optimal control problems.
1.2.3 Generalization
Letting , we have
[TABLE]
Noting , the inner states of and can be replaced by . Hence, (1.17) (1.16) are expressed as
[TABLE]
and
[TABLE]
with . By the above notations, the game finding the open-loop self-coordination control of Problem (LQ) is sublimed to solve a generalized time-inconsistent nonzero-sum LQ dynamic game (Problem (GLQ) below). Namely, finding the open-loop self-coordination control is a motivation to study the generalized time-inconsistent nonzero-sum LQ dynamic game.
Specifically, consider the system
[TABLE]
Here, , and with are the state process and control processes, respectively; the system matrices , , of (1.76) are deterministic. In (1.76), belongs to , which is defined as
[TABLE]
The cost functionals associated with (1.76) are
[TABLE]
and
[TABLE]
where
[TABLE]
The weighting matrices in (1.84) (1.99) are deterministic matrices; and are symmetric. Let
[TABLE]
As the weighting matrices depend on the initial time and the nonlinear terms of conditional expectation appear in the cost functionals, the considered dynamic optimization problem associated with (1.76) (1.84) (1.99) will be time-inconsistent.
Problem (GLQ). For the initial pair , find a pair such that
[TABLE]
where
[TABLE]
above is called an open-loop equilibrium of Problem (GLQ). By the inequalities (1.112) (1.113), the best-response policies and are precommitted and time-consistent, respectively. Specifically, in (1.112) is compared with all other elements in which is global optimal on , and in contrast of (1.113) is called an equilibrium control and is local optimal in the sense that at any time instant it is optimal pointwisely provided that future equilibrium policies are given. In Section 2, Problem (GLQ) is firstly studied and the obtained results are applied directly to Problem (LQ) to obtain the open-loop self-coordination control.
1.3 Contents and findings
By discrete-time convex variation, the stationary conditions and convex conditions are obtained, which together ensure the existence of open-loop equilibrium of Problem (GLQ). Then, several sets of Riccati-like equations and linear equations are introduced, by which the stationary conditions and convex conditions are equivalently characterized. Interestingly, the Riccati-like equations (2.83) and linear equations (2.89) characterizing the convex conditions have nothing to do with the Riccati-like equations (2.41) (2.62) that are for the stationary conditions. To the best of the authors, the result, b)-c) of Theorem 2.2 that characterizes the convexity (Proposition 5.2), is the first one for mean-field LQ problems, which is proved by using a technique of control shifting. Furthermore, sufficient conditions (2.139) in terms of Riccati-like equations and linear equations are presented to characterize the open-loop equilibrium of Problem (GLQ), which can be easily checked. Moreover, the uniqueness of open-loop equilibrium is also studied. By applying the developed theory, open-loop self-coordination control of Problem (LQ) is obtained.
As an example, dynamic multi-period mean-variance portfolio selection is investigated, which itself is of much interest. By introducing a martingale difference sequence, the wealth equation becomes a special case of general linear stochastic system with multiplicative noises. Therefore, the developed theory of Problem (LQ) can be applied directly to this portfolio selection problem. Firstly, the result that is parallel to the general theory is presented on the open-loop self-coordination control, which resort to a pair of Riccati-like equations with special structure. When the punishment matrices become zero, the open-loop self-coordination control should be an open-loop time-consistent equilibrium control, and further the obtained Riccati-like equations should reduce to and coincide with the ones for open-loop time-consistent equilibrium control. This is validated in Theorem 3.2 indeed. Moreover, a term “sequently generic” is introduced, and one shows that given any nonnegative definite punishment direction and any initial pair, Problem (MV) admits unique open-loop self-coordination control for sequently generic punishment intensity.
Section 4 presents two examples to validate the theory developed. Numerical simulations indicate a large body of diversity and the following points are manifested. When punishment intensity is small enough, open-loop self-coordination control perform between open-loop precommitted optimal control and open-loop time-consistent equilibrium control; at some late instants, open-loop self-coordination control outperforms open-loop precommitted optimal control and open-loop time-consistent equilibrium control. In particular, Example 4.2 shows: the scheme of [13] looks for self-coordination policy between open-loop precommitted optimal control and open-loop time-inconsistent equilibrium control, while to some extent this paper goes beyond open-loop precommitted optimal control and open-loop time-consistent equilibrium control. Therefore, our formulation might be viewed as the supplement to that of [13]. Furthermore, on the above technical contents, we have additional comments. Firstly, open-loop self-coordination control relives the open-loop time-inconsistent equilibrium control if one lets the punishment matrices be zero. On the other hand, by adjusting the punishment matrices, open-loop self-coordination control provides many alternatives to handle the time inconsistency. Namely, the necessity to study open-loop self-coordination controls is indicated.
Notation. For a matrix , , and denote the transpose, the Moore-Penrose inverse and the range, respectively, of . Mentioned above, , and for . The spaces and are given in (1.77) (1.111); and and are similarly defined. If a matrix is nonnegative definite or positive definite, it will be denoted as and .
2 Main results
This section presents the main results of this paper, whose proofs are given in Section 5. As looking for open-loop self-coordination control is a special case of solving Problem (GLQ), the results of this section are stated mostly for Problem (GLQ).
Theorem 2.1**.**
For the initial pair , the following statements are equivalent.
- i)
Problem (GLQ) admits an open-loop equilibrium.
- ii)
There exists a such that the stationary conditions
[TABLE]
and the convex conditions
[TABLE]
are satisfied. Here, are computed via the backward stochastic difference equations (BSEs, for short)
[TABLE]
[TABLE]
with
[TABLE]
* of (2.8) are*
[TABLE]
and
[TABLE]
with given by the following stochastic difference equations (SEs, for short)
[TABLE]
and
[TABLE]
Under i) or ii), of ii) is an open-loop equilibrium of Problem (GLQ).
To characterize the stationary conditions (2.5), introduce the Riccati-like equations:
[TABLE]
[TABLE]
where
[TABLE]
and
[TABLE]
Furthermore, the following Riccati equations
[TABLE]
and linear equations
[TABLE]
are introduced to characterize the convex conditions (2.8) with
[TABLE]
Throughout the paper and for a matrix , and denote the range and kernel of , respectively.
Theorem 2.2**.**
For the initial pair , the following statements are equivalent.
- i)
Problem (GLQ) admits an open-loop equilibrium.
- ii)
The following assertions hold.
- a)
The conditions
[TABLE]
are satisfied, where
[TABLE]
*and *
[TABLE]
- b)
The solutions of (2.83) (2.89) have the property , and , .
- c)
For any , the conditions
[TABLE]
are satisfied, where is given by
[TABLE]
with
[TABLE]
Under i) or ii), the open-loop equilibrium of Problem (GLQ) can be selected as
[TABLE]
with given in (2.115).
Remark 2.3**.**
The condition a) in Theorem 2.2 is characterizing the stationary conditions (2.5), and b)-c) is equivalent to the convex conditions (2.8). To the best of the authors, b)-c) is the first result about equivalently characterizing the convexity of mean-field LQ problems.
The following result is straightforward by following Theorem 2.2.
Theorem 2.4**.**
If the conditions
[TABLE]
are satisfied, then for any initial pair , Problem (GLQ) admits an open-loop equilibrium that is given in (2.135).
Theorem 2.5**.**
If conditions in (2.139) are satisfied and are nonsigular , then Problem (GLQ) admits unique open-loop equilibrium
[TABLE]
with
[TABLE]
If all the weighting matrices in (1.84) (1.99) do not depend on the initial times, this corresponds to a special case of Problem (GLQ), which is denoted as Problem (sGLQ) below. For Problem (sGLQ), the corresponding of (2.41) (2.62) (2.83) (2.89) are also independent of the initial times, and are denoted, respectively, by . Furthermore, matrices in (2.73) (2.79) (2.79) (2.95) (2.122) do not depend on the initial times too. For example, (2.62) (2.79) become
[TABLE]
and
[TABLE]
Note that looking for open-loop self-coordination control of Problem (LQ) is a special case of solving Problem (GLQ). Now consider Problem (LQ). Using the notations of (1.84) (1.99), the weighting matrices of (1.55) (1.73) are
[TABLE]
and
[TABLE]
Combining (1.55) and (1.73), we can get results that are parallel to Theorem 2.2, Theorem 2.4 and Theorem 2.5 to obtain the open-loop self-coordination control of Problem (LQ). Due to space limitations, the results are not presented here.
3 Multi-period mean-variance portfolio selection
In this section, we find the open-loop self-coordination control of multi-period mean-variance portfolio selection, which is a special example of Problem (LQ). Consider a capital market consisting of one riskless asset and risky assets over a finite time horizon . Let be a given deterministic return of the riskless asset at time period and the vector of random returns of the risky assets at period . We assume that vectors , are statistically independent and the only information known about the random return vector is its first two moments: its mean and its covariance . Clearly, is nonnegative definite, i.e., .
Let be the wealth of the investor at the beginning of the -th period, and let be the amount invested in the -th risky asset at period , . Then, is the amount invested in the riskless asset at period , and the wealth at the beginning of the -th period [30] is given by
[TABLE]
where is the excess return vector of risky assets [30] defined as . In this section, we consider the case where short-selling of stocks is allowed, i.e., , take values in . This leads to a multi-period mean-variance portfolio selection formulation. For this problem, we let , .
To proceed, (3.1) is transformed into a linear system with multiplicative noises such that the general theory of above section can work. Precisely, define
[TABLE]
where the -th entry of is 1. Then, is a martingale difference sequence as , are statistically independent. Furthermore,
[TABLE]
and (3.1) becomes
[TABLE]
Then, a time-inconsistent version of multi-period mean-variance problem [30] is formulated in the following.
Problem (MV). For and , find a such that
[TABLE]
Here,
[TABLE]
with the trade-off parameter between the mean and variance of the terminal wealth.
In what follows, we look for the open-loop self-coordination control of Problem (MV). From the formulation of (1.55) (1.73) and using similar notations, introduce the following objective functionals and system dynamics:
[TABLE]
and
[TABLE]
with
[TABLE]
Here, . We then have the following two results, whose proofs are given in Section 5.
Theorem 3.1**.**
Given , let the conditions
[TABLE]
be satisfied, where
[TABLE]
[TABLE]
and
[TABLE]
with and being the -th and -th entries of , respectively. Then, for any , Problem (MV) admits an open-loop self-coordination control for the initial pair and , which is selected as
[TABLE]
Here, is the first column block of , i.e.,
[TABLE]
and
[TABLE]
When the punishment matrices become zero, the open-loop self-coordination control should be an open-loop time-consistent equilibrium control, and further the obtained Riccati-like equations (3.77) should reduce to and coincide with the ones for open-loop time-consistent equilibrium control. This is validated in Theorem 3.2 below.
Theorem 3.2**.**
The following statements hold.
- i)
Let . Then, with
[TABLE]
- ii)
Let and \mathbb{E}\Theta_{k}\in\mbox{Ran}\big{[}\mbox{Cov}(\Theta_{k})\big{]},k\in\mathbb{T}. Then, for any initial pair Problem (MV) admits an open-loop self-coordination control, which is an open-loop time-consistent equilibrium control.
- iii)
Let \mathbb{E}\Theta_{k}\in\mbox{Ran}\big{[}\mbox{Cov}(\Theta_{k})\big{]},k\in\mathbb{T}. Define \Xi_{k}=\big{\{}\Phi|\Phi=a_{1}\mbox{Cov}(\Theta_{k})+a_{2}\mathbb{E}\Theta_{k}(\mathbb{E}\Theta_{k})^{T},a_{1},a_{2}\geq 0\big{\}},k\in\mathbb{T} and let . Then, for any initial pair Problem (MV) admits an open-loop self-coordination control.
Given any , denote
[TABLE]
The polynomial is at most of order . Let and with . Note that is related to the values of , . Clearly, for , and the Lebesgue measure . In this case, we call that is nonsingular for sequently generic .
To be precise, a property is parameterized by , and this property is called to hold for sequently generic if this property is satisfied for any with ,…, and ; here, for , and is related to the values of .
Theorem 3.3**.**
Give any . Then, for sequently generic and any initial pair , Problem (MV) admits unique open-loop self-coordination controls.
Remark 3.4**.**
Clearly, \mathbb{E}\Theta_{k}\in\mbox{Ran}\big{[}\mbox{Cov}(\Theta_{k})\big{]},k\in\mathbb{T} holds if . Note that is a common assumption in multi-period mean-variance portfolio selection [11] [13] [30]. Moreover, letting , we recover the results on open-loop time-consistent control of multi-period mean-variance portfolio selection [11] [32] [34].
4 Examples
In this section, two examples are presented to validate the theory developed above.
4.1 Two examples
Example 4.1**.**
Consider a discrete-time stochastic LQ problem, whose system dynamics and cost functional are given, respectively, by
[TABLE]
and
[TABLE]
where
[TABLE]
and here is a scalar martingale difference with constant second-order conditional moment . Letting the punishment matrices be
[TABLE]
find the open-loop self-coordination control for the initial pair with .
Example 4.2**.**
Consider a multi-period mean-variance portfolio selection problem. A capital market consists of one riskless asset and three risky assets over a finite time horizon , and the parameters of the model are as follows
[TABLE]
and the covariance of is
[TABLE]
For objective functional of the form (3.9), let the punishment matrices be
[TABLE]
with the identical matrix of order 3. Find the open-loop self-coordination control for the initial pair .
4.2 Findings
For Example 4.1 and Example 4.2, we have a basket of policy candidates to handle the time inconsistency, namely, precommitted optimal control, open-loop time-consistent equilibrium control, open-loop self-coordination control, and self-coordination policy of [13]. Among these candidates, a question arises naturally: Which one should we select to handle the time inconsistency?
The answer depends on whether or not we have the discretion to reconsider the problems in the future. If we are not allowed to reconsider the problems in the future, precommitted optimal control is our unique selection. On the other hand, if we are given the discretion to reconsider the problems at any or some of intermediate time points, it might be reasonable to select the one as our policy, which outperforms the others at that instant (where we lastly reconsider the problem).
In this section, the expected objective functionals will be calculated for intermediate time instants and different policy candidates. For a specific , let be the open-loop self-coordination controls of Example 4.1 and Example 4.2, respectively, and
[TABLE]
and
[TABLE]
are the corresponding expected objective functionals at with here. In (4.13)-(4.14), the internal states are computed via
[TABLE]
and
[TABLE]
Let be the precommitted optimal control of Example 4.1 and
[TABLE]
with
[TABLE]
Then, the following facts are straightforward:
When , the open-loop self-coordination control becomes the open-loop time-consistent control, namely,
[TABLE]
where denotes the expected objective functional under open-loop time-consistent control.
- 2.
It holds that
[TABLE]
Here, is a version of of fictitious game (1.18) (1.19) in Problem (LQ)g that corresponds to Example 4.1 and ; and is computed via
[TABLE]
Furthermore, similar facts hold for of Example 4.2, which are the expected objective functionals under precommitted optimal control and open-loop time-consistent equilibrium control, respectively; denote the expected objective functionals under self-coordination policy of [13] as
[TABLE]
where
[TABLE]
and with given in Theorem 3.1 of [13].
For Example 4.1 and Example 4.2, we have conducted simulations with valued in
[TABLE]
The following tables present the minima and minimizers of over .
[TABLE]
Table 1. values of and , .
[TABLE]
Table 2. values of and , .
[TABLE]
Table 3. values of and , .
In what follows, 16 pictures are presented to show the curves of expected objective functionals. Several points need to be specialized:
Figures 1 is for Example 4.1 and Figures 2-4 correspond to Example 4.2.
- 2)
Figure 4 shows the curves of .
- 3)
In Figures 1-3, local curves are obtained by shrinking the interval scale of time.
[TABLE]
Figure 1: Curves and local curves of , .
[TABLE]
[TABLE]
Figure 2: Curves and local curves of , .
[TABLE]
[TABLE]
Figure 3: Curves and local curves of , .
Figure 4: Curves of , .
Above figures indicate a large body of diversity and the following cases are manifested:
At some early instants, open-loop precommitted optimal control outperforms (with smaller expected objective functional) open-loop time-inconsistent equilibrium control; and at late instants, open-loop time-inconsistent equilibrium control outperforms open-loop precommitted optimal control: Figure 1.
- 2)
At some late instants, open-loop self-coordination control outperforms open-loop precommitted optimal control and open-loop time-consistent equilibrium control: Figures 1 (), Figure 2 ().
- 3)
When is small enough, open-loop self-coordination control performs between open-loop precommitted optimal control and open-loop time-consistent equilibrium control: Figure 2 ().
- 4)
Comparison with [13]: Example 4.2
- a)
Self-coordination policy of [13] performs mostly between open-loop precommitted optimal control and open-loop time-consistent equilibrium control (Figure 3), though the minima of are slightly smaller than those of open-loop precommitted optimal control.
- At , open-loop precommitted optimal control outperforms all the self-coordination policies of [13] over : subfigures 3, 4 of Figure 3.
- b)
Concerned with the minima of expected objective functionals over , open-loop self-coordination control of this paper outperform self-coordination policy [13] at : Figure 4.
Namely, we can select open-loop self-coordination controls such that at they outperform self-coordination policy of [13]; see the line () of Figure 2, and compare with those of Figure 3.
This, yet, is at a price that self-coordination policy of [13] outperforms open-loop self-coordination control at .
- c)
To summarize and for the particular example (Example 4.2), self-coordination policy of [13] performs mostly between open-loop precommitted optimal control and open-loop time-inconsistent equilibrium control (this is also indicated in Page 102 of [13]); at a price at , particular open-loop self-coordination controls can be selected such that at they outperform self-coordination policy of [13], and also outperform open-loop precommitted optimal control and open-loop time-consistent equilibrium control.
Namely, the scheme of [13] looks for self-coordination policy between open-loop precommitted optimal control and open-loop time-inconsistent equilibrium control, while to some extent this paper goes beyond open-loop precommitted optimal control and open-loop time-consistent equilibrium control. Therefore, our formulation might be viewed as some supplement to that of [13], and adds a new dimension to handle time-inconsistent optimal control problems.
The last words of this section go to the necessity to study open-loop self-coordination control.
5 Proofs
5.1 Proof of Theorem 2.1
The proof is based on the method of discrete-time convex variation.
i)ii). Let be an open-loop equilibrium. For and , let satisfy the SE
[TABLE]
From (1.116) and (5.4), we have
[TABLE]
Denote by , then satisfies (2.24). Obviously, . Then, we obtain
[TABLE]
Noting (2.12) and (2.24), we have
[TABLE]
Then, (5.1) becomes
[TABLE]
As (5.27) holds for any and any , we must have
[TABLE]
and
[TABLE]
which implies the first equation of (2.5).
On the other hand, for any and , let satisfy the SE,
[TABLE]
From (1.116) and (5.33), we have
[TABLE]
Denote by , then satisfies (2.28), and . Therefore, it holds that
[TABLE]
From (2.18) and (2.28), we have
[TABLE]
Hence, (5.1) becomes
[TABLE]
which holds for any and any . Therefore,
[TABLE]
and the second equation of (2.5) holds.
ii)i). By reversing the proof of i)ii), we can obtain the conclusion.
5.2 Proof of Theorem 2.2
Proposition 5.1**.**
The following statements are equivalent.
i). There exists a such that the stationary conditions of (2.5) hold.
ii). a) of Theorem 2.2 is satisfied.
Under the condition ii), the backward states of (2.12) and (2.18) have the following expressions
[TABLE]
Proof. i)ii). From (2.5), it holds that
[TABLE]
and
[TABLE]
Let us first consider the case . We have
[TABLE]
Then, the first equations of (5.43)and (5.48) become
[TABLE]
and
[TABLE]
Furthermore,
[TABLE]
Therefore, the second equations of (5.43) and (5.48) become
[TABLE]
and
[TABLE]
With the notations of (2.73) (2.79), we have from above equations
[TABLE]
and
[TABLE]
Therefore, by a property of Moore-Penrose inverse (Lemma 3.1 of [2]), (2.98) (2.101) hold for and we can select
[TABLE]
and
[TABLE]
Hence, (2.135) holds for . Substituting into (2.12) (2.18), we have
[TABLE]
For and by mimic the derivations between (5.51) and (5.62), we have
[TABLE]
and
[TABLE]
Therefore, (2.98) (2.101) hold for and we can select
[TABLE]
and
[TABLE]
Hence, Hence, (2.135) holds for . Substituting into (2.12) (2.18), we have
[TABLE]
By repeating the above procedure and the arguments of induction, we can get the expressions of the backward states and .
ii)i). Due to the property of Moore-Penrose inverse and by reversing the proof of i)ii), we can obtain the conclusion.
We now study the convex condition (2.8). By adding to and subtracting
[TABLE]
from (with ), we have
[TABLE]
Similarly,
[TABLE]
As , are symmetric, there exist orthogonal matrices , such that
[TABLE]
In the above, , , are diagonal matrices, whose diagonal elements are the nonzero eigenvalues of ,, respectively. Let , . Then, we have
[TABLE]
Moreover, , can be decomposed as , , respectively, where the lines of , form the bases of and , respectively. Let
[TABLE]
Hence, we have
[TABLE]
Note that the space spanned by lines of is . Let
[TABLE]
Proposition 5.2**.**
The following statements are equivalent.
- i)
The convex conditions of (2.8) hold.
- ii)
b) and c) of Theorem 2.2 are satisfied.
Proof. i)ii). Note that is convex. If , we have , and
[TABLE]
Introduce a set
[TABLE]
For , let be the lines of , then form a basis of . For any and , there exit such that . Then,
[TABLE]
For , is -measurable and , this implies that is -measurable and . Therefore, .
Introduce a bounded linear operator from to :
[TABLE]
We now prove that is a surjection. In fact, for any , we have and let
[TABLE]
and
[TABLE]
Note that in (5.98) is in . As , from (5.98) we have
[TABLE]
Hence, is a surjection defined from to . From this, (5.95) and the procedure of contradiction, we have . This further implies .
Let
[TABLE]
Note further that is convex. If , from (5.2) we have
[TABLE]
Introduce a set
[TABLE]
For , let denote the lines of . For any and , there exit such that . Then,
[TABLE]
Therefore, is a deterministic space. Furthermore, introduce a bounded linear operator from to :
[TABLE]
We now prove that is a surjection. In fact, for any , let
[TABLE]
and
[TABLE]
Note that in (5.103) is in . As , from (5.103) we have
[TABLE]
Hence is a surjection defined from to . From this, (5.100) and the procedure of contradiction, we have . This further implies . Furthermore, from (5.88), it is easy to get .
We now prove c) of Theorem 2.2. Note that
[TABLE]
Then,
[TABLE]
In the above, we must have
[TABLE]
Otherwise, assume there exist and such that
[TABLE]
with and its probability . If \mathbb{E}_{t}\big{(}|a_{1}|^{2}+|a_{2}|^{2}\big{)}(\omega)=0, a.s., we must have
[TABLE]
which is impossible. Hence, there exists with such that \mathbb{E}_{t}\big{(}|a_{1}|^{2}+|a_{2}|^{2}\big{)}(\omega)>0, for . Introduce a new control
[TABLE]
where
[TABLE]
with be the indicator function. Then, under (5.112) and for , we have
[TABLE]
where
[TABLE]
As , (5.2) contradicts the convex condition (5.2). Hence, we have (5.107), and (2.124) (2.124) follow.
ii)i). From (5.2) and b), c) of Theorem 2.2, we have and for any . Then, we need only to show
[TABLE]
In fact, for any , let
[TABLE]
where
[TABLE]
Thus . Hence, (5.118) holds, which together with implies
[TABLE]
This completes the proof.
Proof of Theorem 2.2. This follows from Theorem 2.1, Proposition 5.1 and Proposition 5.2.
Proof of Theorem 2.5. Following (5.67)-(5.84) and by deduction, we have that Problem (GLQ) admits a unique open-loop equilibrium, as are nonsigular . The expression of open-loop equilibrium follows easily form Theorem 2.2. This completes the proof.
5.3 Proof of Theorem 3.1
Applying the general theory of Section 2 to Problem (MV), (2.41)-(2.79) (with ) becomes to
[TABLE]
[TABLE]
where
[TABLE]
and
[TABLE]
with .
Noting , simple calculations show
[TABLE]
and
[TABLE]
for some , which implies the form of :
[TABLE]
Therefore,
[TABLE]
Hence,
[TABLE]
Moreover,
[TABLE]
Hence, we have (3.58) and (3.77). Furthermore, for Problem (MV) and under the parameters (3.25)-(3.46), (2.83)-(2.95) becomes to
[TABLE]
and
[TABLE]
with
[TABLE]
Introduce a new optimal control problem with the system dynamics
[TABLE]
and the objective functional
[TABLE]
that is to be minimized within . Here, the parameters in (5.150)-(5.151) are from (3.25)-(3.46). Clearly, this is a special example of the static mean-field LQ optimal control problem that is considered in [34]. As , we have from Theorem 4.3 of [34] that
[TABLE]
As , we have . This completes the proof by following Theorem 2.2 and using the notations of (3.58)-(3.77).
5.4 Proof of Theorem 3.2
i). Let be the -th entry of , then
[TABLE]
with
[TABLE]
Therefore, for ,
[TABLE]
with \big{(}\overline{\mathbf{W}}_{k}^{\dagger}\big{)}^{(11)},\big{(}\overline{\mathbf{W}}_{k}^{\dagger}\big{)}^{(21)} being the -th and -th blocks of . Therefore, (3.87) holds with the property . Also, as , we have .
Note that . In this case,
[TABLE]
Checking the definition of Moore-Penrose inverse, we have
[TABLE]
where
[TABLE]
From (5.163), (5.168), and , it holds that . Hence, we have proved that has the following form:
[TABLE]
From (5.134), we have
[TABLE]
By (50) of [34], we know 1-(\mathbb{E}\Theta_{k})^{T}\big{[}\mathbb{E}\big{(}\Theta_{k}\Theta_{k}^{T}\big{)}\big{]}^{\dagger}\mathbb{E}\Theta_{k}>0. Therefore, for .
ii). In this case, is of the following form
[TABLE]
As \mathbb{E}\Theta_{k}\in\mbox{Ran}\big{[}\mbox{Cov}(\Theta_{k})\big{]},k\in\mathbb{T}, the condition (3.47) is satisfied. Hence, for any initial pair Problem (MV) admit open-loop self-coordination controls of precommitted self and of sophisticated selves, which coincide with the open-loop precommitted optimal control and open-loop time-consistent equilibrium control, respectively.
iii). Let \Xi_{k}^{c}=\Big{\{}\zeta\,\big{|}\,\mbox{Cov}(\Theta_{k})\zeta=\mathbb{E}\Theta_{k}\Big{\}}\neq\emptyset. For given , there exists such that
[TABLE]
Note that (3.47) is equivalent to
[TABLE]
and that
[TABLE]
For , we have and for some . Letting , the equation
[TABLE]
becomes to finding such that
[TABLE]
holds for given . By some calculations, the determinant of coefficient matrix of (5.183) is
[TABLE]
If or , and of (5.183) can be both selected to be 0. For and , we have following derivation. As , let and (5.184) becomes
[TABLE]
As , we must have . Therefore, there exists some such that and (5.183) is solvable.
Therefore, by selecting with (5.176) we can have and similarly can be proved. This completes the proof.
6 Conclusion
For a time-inconsistent LQ optimal control, a Nash-type fictitious game framework is introduced, where the game is between the decision maker and an auxiliary control variable. The decision maker and auxiliary control variable are called real player and fictitious player, which look for time-consistent policy and precommitted optimal policy, respectively. The equilibrium policy of real player is called an open-loop self-coordination control of the LQ problem. As a generalization, a time-inconsistent nonzero-sum stochastic LQ dynamic game is studied, for which one player is to find the precommitted policy and the other player is to find the time-consistent policy. Necessary and sufficient conditions are derived to characterize the open-loop equilibrium of this nonzero-sum stochastic LQ dynamic game via Riccati-like equations, and as a byproduct, result to ensure the existence of open-loop self-coordination control of the original LQ optimal control is also obtained. To test the general theory, the mean-variance portfolio selection is investigated. For future research, the closed-loop self-coordination should be investigated.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] Richard H. Thaler: integrating Economics with Psychology, The Committee for the Prize in Econo -mic Sciences in Memory of Alfred Nobel , https://www.nobelprize.org/uploads/2018/06/advanced-economicsciences 2017.pdf.
- 2[2] M. Ait Rami, X. Chen, and X.Y. Zhou, Discrete-time indefinite LQ control with state and control dependent noises, Journal of Global Optimization , 2002, vol.23, pp.245-265.
- 3[3] D. Ariely and K. Wertenbroch, Procrastination, deadlines, and performance: self-control by precommitment, Psychological Science , 2002, vol.13, no.3, pp.219-224.
- 4[4] L.V. Auer, Dynamic preferences, choice mechanisms, and welfare , Lecture Notes in Economics and Mathematical Systems, vol.462, Springer, 1998.
- 5[5] R. Bellman, Dynamic programming , Princeton Univ. Press, Princeton, New Jersey, 1957.
- 6[6] R. B e ´ ´ e \acute{\mbox{e}} nabou and M. Pycia, Dynamic inconsistency and self-control: a planner-doer interpretation, Economics Letter , 2002, vol.77, pp.419-424.
- 7[7] T. Bjork and A. Murgoci, A theory of Markovian time-inconsistent stochastic control in discrete time, Finance and Stochastics , 2014, vol.18, pp.545-592.
- 8[8] T. Bjork, M. Khapko, and A. Murgoci, On time-inconsistent stochastic control in continuous time, Finance and Stochastics , 2017, vol.21, pp.331-360.