A note on the costly surveillance of a stochastic system

We consider the costly surveillance of a stochastic system with a finite state space and a finite number of actions in each state. There is a positive cost of observing the system and the system earns at a rate depending on the state of the system and the action taken. A policy for controlling such a system specifies the action to be taken and the time to the next observation, both possibly random and depending on the past history of the system. A form of the long range average income is the criterion for comparing different policies. If R ^Δ denotes the class of policies for which the times between successive observations of the system are random variables with cumulative distribution functions on [0, Δ], Δ < ∞, we show that there exists a nonrandomized stationary policy that is optimal in R ^Δ. Furthermore, for sufficiently large Δ, this optimal policy is independent of Δ.