Article
Full entry |
PDF
(1.9 MB)
Feedback
PDF
(1.9 MB)
Feedback
Keywords:
Markov control process; unbounded costs; discounted asymptotic optimality; density estimator; rate of convergence
Markov control process; unbounded costs; discounted asymptotic optimality; density estimator; rate of convergence
Summary:
We study the adaptive control problem for discrete-time Markov control processes with Borel state and action spaces and possibly unbounded one-stage costs. The processes are given by recurrent equations $x_{t+1}=F(x_t,a_t,\xi _t),\,\,t=0,1,\ldots $ with i.i.d. $\Re ^k$-valued random vectors $\xi _t$ whose density $\rho $ is unknown. Assuming observability of $\xi _t$ we propose the procedure of statistical estimation of $\rho $ that allows us to prove discounted asymptotic optimality of two types of adaptive policies used early for the processes with bounded costs.
References:
[1] Agrawal R.: Minimizing the learning loss in adaptive control of Markov chains under the weak accessibility condition. J. Appl. Probab. 28 (1991), 779–790 DOI 10.2307/3214681 | MR 1133786 | Zbl 0741.60070
[2] Ash R. B.: Real Analysis and Probability. Academic Press, New York 1972 MR 0435320
[3] Cavazos–Cadena R.: Nonparametric adaptive control of discounted stochastic system with compact state space. J. Optim. Theory Appl. 65 (1990), 191–207 DOI 10.1007/BF01102341 | MR 1051545
[4] Dynkin E. B., A A.: Yushkevich: Controlled Markov Processes. Springer–Verlag, New York 1979 MR 0554083
[5] Fernández–Gaucherand E., Arapostathis A., Marcus S. I.: A methodology for the adaptive control of Markov chains under partial state information. In: Proc. of the 1992 Conf. on Information Sci. and Systems, Princeton, New Jersey, pp. 773–775
[6] Fernández–Gaucherand E., Arapostathis A., Marcus S. I.: Analysis of an adaptive control scheme for a partially observed controlled Markov chain. IEEE Trans. Automat. Control 38 (1993), 987–993 DOI 10.1109/9.222316 | MR 1227213 | Zbl 0786.93089
[7] Gordienko E. I.: Adaptive strategies for certain classes of controlled Markov processes. Theory Probab. Appl. 29 (1985), 504–518 Zbl 0577.93067
[8] Gordienko E. I.: Controlled Markov sequences with slowly varying characteristics II. Adaptive optimal strategies. Soviet J. Comput. Systems Sci. 23 (1985), 87–93 MR 0844298 | Zbl 0618.93070
[9] Gordienko E. I., Hernández–Lerma O.: Average cost Markov control processes with weighted norms: value iteration. Appl. Math. 23 (1995), 219–237 MR 1341224 | Zbl 0829.93068
[10] Gordienko E. I., Montes–de–Oca R., Minjárez–Sosa J. A.: Approximation of average cost optimal policies for general Markov decision processes with unbounded costs. Math. Methods Oper. Res. 45 (1997), 2, to appear DOI 10.1007/BF01193864 | MR 1446409 | Zbl 0882.90127
[11] Hasminskii R., Ibragimov I.: On density estimation in the view of Kolmogorov’s ideas in approximation theory. Ann. of Statist. 18 (1990), 999–1010 DOI 10.1214/aos/1176347736 | MR 1062695 | Zbl 0705.62039
[12] Hernández–Lerma O.: Adaptive Markov Control Processes. Springer–Verlag, New York 1989 MR 0995463 | Zbl 0698.90053
[13] Hernández–Lerma O.: Infinite–horizon Markov control processes with undiscounted cost criteria: from average to overtaking optimality. Reporte Interno 165. Departamento de Matemáticas, CINVESTAV-IPN, A.P. 14-740.07000, México, D. F., México (1994). (Submitted for publication)
[14] Hernández–Lerma O., Cavazos–Cadena R.: Density estimation and adaptive control of Markov processes: average and discounted criteria. Acta Appl. Math. 20 (1990), 285–307 DOI 10.1007/BF00049572 | MR 1081591 | Zbl 0717.93066
[15] Hernández–Lerma O., Lasserre J. B.: Discrete–Time Markov Control Processes. Springer–Verlag, New York 1995 Zbl 0928.93002
[16] Hernández–Lerma O., Marcus S. I.: Adaptive control of discounted Markov decision chains. J. Optim. Theory Appl. 46 (1985), 227–235 DOI 10.1007/BF00938426 | MR 0794250 | Zbl 0543.90093
[17] Hernández–Lerma O., Marcus S. I.: Adaptive policies for discrete–time stochastic control system with unknown disturbance distribution. Systems Control Lett. 9 (1987), 307–315 DOI 10.1016/0167-6911(87)90055-7 | MR 0912683
[18] Hinderer K.: Foundations of Non-Stationary Dynamic Programming with Discrete Time Parameter. (Lecture Notes in Operations Research and Mathematical Systems 33.) Springer–Verlag, Berlin – Heidelberg – New York 1970 MR 0267890 | Zbl 0202.18401
[19] Köthe G.: Topological Vector Spaces I. Springer–Verlag, New York 1969 MR 0248498
[20] Kumar P. R., Varaiya P.: Stochastic Systems: Estimation, Identification and Adaptive Control. Prentice–Hall, Englewood Cliffs 1986 Zbl 0706.93057
[21] Lippman S. A.: On dynamic programming with unbounded rewards. Management Sci. 21 (1975), 1225–1233 DOI 10.1287/mnsc.21.11.1225 | MR 0398535 | Zbl 0309.90017
[22] Mandl P.: Estimation and control in Markov chains. Adv. in Appl. Probab. 6 (1974), 40–60 DOI 10.2307/1426206 | MR 0339876 | Zbl 0281.60070
[23] Rieder U.: Measurable selection theorems for optimization problems. Manuscripta Math. 24 (1978), 115–131 DOI 10.1007/BF01168566 | MR 0493590 | Zbl 0385.28005
[24] Schäl M.: Estimation and control in discounted stochastic dynamic programming. Stochastics 20 (1987), 51–71 DOI 10.1080/17442508708833435 | MR 0875814
[25] Stettner L.: On nearly self-optimizing strategies for a discrete–time uniformly ergodic adaptive model. J. Appl. Math. Optim. 27 (1993), 161–177 DOI 10.1007/BF01195980 | MR 1202530 | Zbl 0769.93084
[26] Stettner L.: Ergodic control of Markov process with mixed observation structure. Dissertationes Math. 341 (1995), 1–36 MR 1318335
[27] Nunen J. A. E. E. van, Wessels J.: A note on dynamic programming with unbounded rewards. Management Sci. 24 (1978), 576–580 DOI 10.1287/mnsc.24.5.576
Search
Partner of
