Relation between continuous and discrete time markovian decision problems

An associated discrete (continuous) time Markovian decision problem for any given continuous (discrete) time Markovian decision model is formulated. A relationship between the continuous and associated discrete time discounted returns is obtained. This result is used to show the existence of a deterministic stationary policy that optimizes both discounted return functions. It is also proved that the same policy optimizes the average expected return for both continuous and the associated discrete time processes. The results obtained in this paper can be used to solve the continuous time Markovian decision problem by using the discrete time algorithms or vice versa.     

Solution of continuous-time markovian decision models using infinite linear programming

Infinite-horizon, countable-state, continuous-time Markovian decision models are solved by formulating as a pair of infinite linear-programming problems. Expected discounted and average returns are considered as criterion functions. For both criterion functions, the existence of deterministic optimal stationary policies is established by solving the associated infinite linear-programming problems. Computational procedures for finite state and action sets are discussed by considering associated finite linear-programming problems.