Article
Full entry |
PDF
(0.4 MB)
Feedback
PDF
(0.4 MB)
Feedback
Keywords:
semi-Markov decision processes; first passage time; unbounded reward rate; minimal variance; mean-variance optimal policy
semi-Markov decision processes; first passage time; unbounded reward rate; minimal variance; mean-variance optimal policy
Summary:
This paper deals with a first passage mean-variance problem for semi-Markov decision processes in Borel spaces. The goal is to minimize the variance of a total discounted reward up to the system's first entry to some target set, where the optimization is over a class of policies with a prescribed expected first passage reward. The reward rates are assumed to be possibly unbounded, while the discount factor may vary with states of the system and controls. We first develop some suitable conditions for the existence of first passage mean-variance optimal policies and provide a policy improvement algorithm for computing an optimal policy. Then, two examples are included to illustrate our results. At last, we show how the results here are reduced to the cases of discrete-time Markov decision processes and continuous-time Markov decision processes.
References:
[1] Berument, H., Kilinc, Z., Ozlale, U.: The effects of different inflation risk premiums on interest rate spreads. Phys. A 333 (2004), 317-324. DOI 10.1016/j.physa.2003.10.039 | MR 2100223
[2] Baykal-Gürsoy, M., Gürsoy, K.: Semi-Markov decision processes: nonstandard criteria. Probab. Engrg. Inform. Sci. 21 (2007), 635-657. DOI 10.1017/S026996480700037X | MR 2357126
[3] Bäuerle, N., Rieder, U.: Markov decision processes with applications to finance. In: Universitext, Springer, Heidelberg 2011. DOI 10.1007/978-3-642-18324-9 | MR 2808878 | Zbl 1236.90004
[4] Collins, E.: Finite-horizon variance penalised Markov decision processes. OR Spektrum 19 (1997), 35-39. DOI 10.1007/s002910050017 | MR 1464393 | Zbl 0894.90161
[5] Costa, O. L. V., Maiali, A. C., Pinto, A. de C.: Sampled control for mean-variance hedging in a jump diffusion financial market. IEEE Trans. Automat. Control 55 (2010), 1704-1709. DOI 10.1109/tac.2010.2046923 | MR 2675836
[6] Filar, J. A., Kallenberg, L. C. M., Lee, H. M.: Variance-penalized Markov decision processes. Math. Oper. Res. 14 (1989), 147-161. DOI 10.1287/moor.14.1.147 | MR 0984562 | Zbl 0676.90096
[7] Fu, C. P., Lari-Lavassani, A., Li, X.: Dynamic mean-variance portfolio selection with borrowing constraint. European J. Oper. Res. 200 (2010), 312-319. DOI 10.1016/j.ejor.2009.01.005 | MR 2561109 | Zbl 1183.91192
[8] Guo, X. P., Hernández-Lerma, O.: Continuous-Time Markov Decision Processes: Theory and Applications. Springer-Verlag, Berlin 2009. DOI 10.1007/978-3-642-02547-1 | MR 2554588 | Zbl 1209.90002
[9] Guo, X. P., Song, X. Y.: Mean-variance criteria for finite continuous-time Markov decision processes. IEEE Trans. Automat. Control 54 (2009), 2151-2157. DOI 10.1109/tac.2009.2023833 | MR 2567941
[10] Guo, X. P., Ye, L. E., Yin, G.: A mean-variance optimization problem for discounted Markov decision processes. European J. Oper. Res. 220 (2012), 423-429. DOI 10.1016/j.ejor.2012.01.051 | MR 2908853 | Zbl 1253.90214
[11] Guo, X. P., Huang, X. X., Zhang, Y.: On the first passage $g$-mean variance optimality for discounted continuous-time Markov decision processes. SIAM J. Control Optim. 53 (2015), 1406-1424. DOI 10.1137/140968872 | MR 3352600 | Zbl 1322.90108
[12] Hu, Q. Y.: Continuous time Markov decision processes with discounted moment criterion. J. Math. Anal. Appl. 203 (1996), 1-12. DOI 10.1006/jmaa.1996.9999 | MR 1412477 | Zbl 0858.90135
[13] Hernández-Lerma, O., Lasserre, J. B.: Further Topics on Discrete-Time Markov Control Processes. Springer-Verlag, New York 1999. DOI 10.1007/978-1-4612-0561-6 | MR 1697198 | Zbl 0928.93002
[14] Hernández-Lerma, O., Vega-Amaya, O., Carrasco, G.: Sample-path optimality and variance-minimization of average cost Markov control processes. SIAM J. Control Optim. 38 (1999), 79-93. DOI 10.1137/S0363012998340673 | MR 1740606 | Zbl 0951.93074
[15] Haberman, S., Sung, J. H.: Optimal pension funding dynamics over infinite control horizon when stochastic rates of return are stationary. Insurance Math. Econom. 36 (2005), 103-116. DOI 10.1016/j.insmatheco.2004.10.006 | MR 2122668 | Zbl 1111.91023
[16] Huang, Y. H., Guo, X. P.: First passage models for denumerable semi-Markov decision processes with nonnegative discounted costs. Acta Math. Appl. Sin. Engl. Ser. 27 (2011), 177-190. DOI 10.1007/s10255-011-0061-2 | MR 2784052 | Zbl 1235.90177
[17] Huang, Y. H., Guo, X. P., Song, X. Y.: Performance analysis for controlled semi-Markov systems with application to maintenance. J. Optim. Theory Appl. 150 (2011), 395-415. DOI 10.1007/s10957-011-9813-7 | MR 2818928 | Zbl 1222.90076
[18] Huang, Y. H., Guo, X. P.: Constrained optimality for first passage criteria in semi-Markov decision processes. Optimization, Control, and Applications of Stochastic Systems, pp. 181-202, Systems Control Found. Appl., Birkhäuser/Springer, New York 2012. DOI 10.1007/978-0-8176-8337-5_11 | MR 2961386
[19] Huang, Y. H., Guo, X. P.: Mean-variance problems for finite horizon semi-Markov decision processes. Appl. Math. Optim. 72 (2015), 233-259. DOI 10.1007/s00245-014-9278-9 | MR 3394396 | Zbl 1343.93100
[20] Jaquette, S. C.: Markov decision processes with a new optimality criterion: continuous time. Ann. Statist. 3 (1975), 547-553. DOI 10.1214/aos/1176343087 | MR 0363493 | Zbl 0321.90051
[21] Kurano, M.: Markov decision processes with a minimum-variance criterion. J. Math. Anal. Appl. 123 (1987), 572-583. DOI 10.1016/0022-247x(87)90332-5 | MR 0883710 | Zbl 0619.90080
[22] Kharroubi, I., Lim, T.: A. Ngoupeyou, Mean-variance hedging on uncertain time horizon in a market with a jump. Appl. Math. Optim. 68 (2013), 413-444. DOI 10.1007/s00245-013-9213-5 | MR 3131502
[23] Lee, M. J., Li, W. J.: Drift and diffusion function specification for short-term interest rates. Econom. Lett. 86 (2005), 339-346. DOI 10.1016/j.econlet.2004.09.002 | MR 2124417 | Zbl 1254.91733
[24] Mandl, P.: On the variance in controlled Markov chains. Kybernetika 7 (1971), 1-12. MR 0286178 | Zbl 0215.25902
[25] Mannor, S., Tsitsiklis, J. N.: Algorithmic aspects of mean-variance optimization in Markov decision processes. European J. Oper. Res. 231 (2013), 645-653. DOI 10.1016/j.ejor.2013.06.019 | MR 3092864 | Zbl 1317.90318
[26] Markowitz, H. M.: Portfolio Selection: Efficient Diversification of Investments. John Wiley and Sons, Inc., New York 1959. MR 0103768
[27] Prieto-Rumeau, T., Hernández-Lerma, O.: Variance minimization and the overtaking optimality approach to continuous-time controlled Markov chains. Math. Methods Oper. Res. 70 (2009), 527-540. DOI 10.1007/s00186-008-0276-z | MR 2558430 | Zbl 1177.93101
[28] Sobel, M. J.: The variance of discounted Markov decision processes. J. Appl. Probab. 19 (1982), 794-802. DOI 10.1017/s0021900200023123 | MR 0675143 | Zbl 0503.90091
[29] White, D. J.: Computational approaches to variance-penalised Markov decision processes. OR Spektrum 14 (1992), 79-83. DOI 10.1007/bf01720350 | MR 1175342 | Zbl 0768.90087
[30] Wu, X., Guo, X. P.: First passage optimality and variance minimisation of Markov decision processes with varying discount factors. J. Appl. Probab. 52 (2015), 441-456. DOI 10.1017/s0021900200012560 | MR 3372085 | Zbl 1327.90374
[31] Zhou, X. Y., Yin, G.: Markowitz's mean-variance portfolio selection with regime switching: a continuous-time model. SIAM J. Control Optim. 42 (2003), 1466-1482. DOI 10.1137/s0363012902405583 | MR 2044805 | Zbl 1175.91169
[32] Zhu, Q. X., Guo, X. P.: Markov decision processes with variance minimization: a new condition and approach. Stoch. Anal. Appl. 25 (2007), 577-592. DOI 10.1080/07362990701282807 | MR 2321898 | Zbl 1152.90646
Search
Partner of
