Structured markovian decision problems

Structured finite action-finite state space discounted Markovian decision problems are analyzed. Any problem of a general class is shown to be equivalent to a “separated” problem with decomposable problem structure. A modified policy iteration approach is developed for this decomposable reformulation. Both analytic and computer evaluations of the decomposition algorithm's effectiveness are presented.     

Structured maintenance policies on interior sample paths

We examine the problem of adaptively scheduling perfect observations and preventive replacements for a multi‐state, Markovian deterioration system with silent failures such that total expected discounted cost is minimized. We model this problem as a partially observed Markov decision process and show that the structural properties of the optimal policy hold for certain non‐extreme sample paths. © 2007 Wiley Periodicals, Inc. Naval Research Logistics, 2007     

Solution of continuous-time markovian decision models using infinite linear programming

Infinite-horizon, countable-state, continuous-time Markovian decision models are solved by formulating as a pair of infinite linear-programming problems. Expected discounted and average returns are considered as criterion functions. For both criterion functions, the existence of deterministic optimal stationary policies is established by solving the associated infinite linear-programming problems. Computational procedures for finite state and action sets are discussed by considering associated finite linear-programming problems.     

Denumerable state markovian sequential control processes: On randomizations of optimal policies

We consider a denumerable state Markovian sequential control process. It is well known that when we consider the expected total discounted income as a criterion, there exists a nonrandomized stationary policy that is optimal. It is also well known that when we consider the expected average income as a criterion, an optimal nonrandomized stationary policy exists when a certain system of equations has a solution. The problem considered here is: if there exist two optimal nonrandomized stationary policies, will a randomization of these two policies be optimal? It is shown that in the discounted case the answer is always yes, but in the average income case, the answer is yes only under certain additional conditions.     

Evaluation of costs of ordering policies in large machine repair problems

This article examines a version of the machine repair problem where failures may be irreparable. Since the number of machines in the system keeps decreasing, we impose a fixed state-dependent ordering policy of the type often encountered in inventory models. Although the system is Markovian, the number of states becomes very large. The emphasis of the article, therefore, is on deriving computationally tractable formulas for the steady-state probabilities, the long-run average cost per unit time, and the vector of expected discounted costs. When the state space is so large that exact computations may be infeasible, we propose approximations which are relatively quick and simple to compute and which yield very accurate results for the test problems examined.     

Replacing continuous demand with discrete demand in a competitive location model

Location models commonly represent demand as discrete points rather than as continuously spread over an area. This modeling technique introduces inaccuracies to the objective function and consequently to the optimal location solution. In this article this inaccuracy is investigated by the study of a particular competitive facility location problem. First, the location problem is formulated over a continuous demand area. The optimal location for a new facility that optimizes the objective function is obtained. This optimal location solution is then compared with the optimal location obtained for a discrete set of demand points. Second, a simple approximation approach to the continuous demand formulation is proposed. The location problem can be solved by using the discrete demand algorithm while significantly reducing the inaccuracies. This way the simplicity of the discrete approach is combined with the approximated accuracy of the continuous-demand location solution. Extensive analysis and computations of the test problem are reported. It is recommended that this approximation approach be considered for implementation in other location models. © 1997 John Wiley & Sons, Inc.     

Value management of diagnostic equipment with cancelation,no‐show,and emergency patients

Magnetic resonance imaging and other multifunctional diagnostic facilities, which are considered as scarce resources of hospitals, typically provide services to patients with different medical needs. This article examines the admission policies during the appointment management of such facilities. We consider two categories of patients: regular patients who are scheduled in advance through an appointment system and emergency patients with randomly generated demands during the workday that must be served as soon as possible. According to the actual medical needs of patients, regular patients are segmented into multiple classes with different cancelation rates, no‐show probabilities, unit value contributions, and average service times. Management makes admission decisions on whether or not to accept a service request from a regular patient during the booking horizon to improve the overall value that could be generated during the workday. The decisions should be made by considering the cancelation and no‐show behavior of booked patients as well as the emergency patients that would have to be served because any overtime service would lead to higher costs. We studied the optimal admission decision using a continuous‐time discrete‐state dynamic programming model. Identifying an optimal policy for this discrete model is analytically intractable and numerically inefficient because the state is multidimensional and infinite. We propose to study a deterministic counterpart of the problem (i.e., the fluid control problem) and to develop a time‐based fluid policy that is shown to be asymptotically optimal for large‐scale problems. Furthermore, we propose to adopt a mixed fluid policy that is developed based on the information obtained from the fluid control problem. Numerical experiments demonstrate that this improved policy works effectively for small‐scale problems. © 2016 Wiley Periodicals, Inc. Naval Research Logistics 63: 287–304, 2016     

Optimal lot sizing and inspection policy for an EMQ model with imperfect inspections

An EMQ model with a production process subject to random deterioration is considered. The process can be monitored through inspections, and both the lot size and the inspection schedule are subject to control. The “in-control” periods are assumed to be generally distributed and the inspections are imperfect, i.e., the true state of the process is not necessarily revealed through an inspection. The objective is the joint determination of the lot size and the inspection schedule, minimizing the long-run expected average cost per unit time. Both discrete and continuous cases are examined. A dynamic programming formulation is considered in the case where the inspections can be performed only at discrete times, which is typical for the parts industry. In the continuous case, an optimum inspection schedule is obtained for a given production time and given number of inspections by solving a nonlinear programming problem. A two-dimensional search procedure can be used to find the optimal policy. In the exponential case, the structure of the optimal inspection policy is established using Lagrange's method, and it is shown that the optimal inspection times can be found by solving a nonlinear equation. Numerical studies indicate that the optimal policy performs much better than the optimal policy with periodic inspections considered previously in the literature. The case of perfect inspections is discussed, and an extension of the results obtained previously in the literature is presented. © 1998 John Wiley & Sons, Inc. Naval Research Logistics 45: 165–186, 1998     

Developing an optimal repair-replacement strategy for pallets

A Markovian model is presented for the development of an optimal repair-replacement policy for the situation requiring a decision only at failure. The problem is characterized by the presence of growth which is integrated into the formulation. The model is applied to an actual problem, with data analysis and results given. Substantial savings are indicated.     

Optimal maintenance policies for a safety‐critical system and its deteriorating sensor

We consider the integrated problem of optimally maintaining an imperfect, deteriorating sensor and the safety‐critical system it monitors. The sensor's costless observations of the binary state of the system become less informative over time. A costly full inspection may be conducted to perfectly discern the state of the system, after which the system is replaced if it is in the out‐of‐control state. In addition, a full inspection provides the opportunity to replace the sensor. We formulate the problem of adaptively scheduling full inspections and sensor replacements using a partially observable Markov decision process (POMDP) model. The objective is to minimize the total expected discounted costs associated with system operation, full inspection, system replacement, and sensor replacement. We show that the optimal policy has a threshold structure and demonstrate the value of coordinating system and sensor maintenance via numerical examples. © 2017 Wiley Periodicals, Inc. Naval Research Logistics 64: 399–417, 2017     

Dynamic pricing and replenishment: Optimality,bounds, and asymptotics

In many applications, managers face the problem of replenishing and selling products during a finite time horizon. We investigate the problem of making dynamic and joint decisions on product replenishment and selling in order to improve profit. We consider a backlog scenario in which penalty cost (resulting from fulfillment delay) and accommodation cost (resulting from shortage at the end of the selling horizon) are incurred. Based on continuous‐time and discrete‐state dynamic programming, we study the optimal joint decisions and characterize their structural properties. We establish an upper bound for the optimal expected profit and develop a fluid policy by resorting to the deterministic version of the problem (ie, the fluid problem). The fluid policy is shown to be asymptotically optimal for the original stochastic problem when the problem size is sufficiently large. The static nature of the fluid policy and its lack of flexibility in matching supply with demand motivate us to develop a “target‐inventory” heuristic, which is shown, numerically, to be a significant improvement over the fluid policy. Scenarios with discrete feasible sets and lost‐sales are also discussed in this article.     

An application of yield management for Internet Service Providers

In this paper we study strategies for better utilizing the network capacity of Internet Service Providers (ISPs) when they are faced with stochastic and dynamic arrivals and departures of customers attempting to log‐on or log‐off, respectively. We propose a method in which, depending on the number of modems available, and the arrival and departure rates of different classes of customers, a decision is made whether to accept or reject a log‐on request. The problem is formulated as a continuous time Markov Decision Process for which optimal policies can be readily derived using techniques such as value iteration. This decision maximizes the discounted value to ISPs while improving service levels for higher class customers. The methodology is similar to yield management techniques successfully used in airlines, hotels, etc. However, there are sufficient differences, such as no predefined time horizon or reservations, that make this model interesting to pursue and challenging. This work was completed in collaboration with one of the largest ISPs in Connecticut. The problem is topical, and approaches such as those proposed here are sought by users. © 2001 John Wiley & Sons, Inc., Naval Research Logistics 48:348–362, 2001     

Monotone optimal preventive maintenance policies for stochastically failing equipment

This paper examines various models for maintenance of a machine operating subject to stochastic deterioration. Three alternative models are presented for the deterioration process. For each model, in addition to the replacement decision, the option exists of performing preventive maintenance. The effect of this maintenance is to “slow” the deterioration process. With an appropriate reward structure imposed on the processes, the models are formulated as continuous time Markov decision processes. the optimality criterion being the maximization of expected discounted reward earned over an infinite time horizon. For each model conditions are presented under which the optimal maintenance policy exhibits the following monotonic structure. First, there exists a control limit rule for replacement. That is, there exists a number i* such that if the state of machine deterioration exceeds i* the optimal policy replaces the machine by a new machine. Secondly, prior to replacement the optimal level of preventive maintenance is a nonincreasing function of the state of machine deterioration. The conditions which guarantee this result have a cost/benefit interpretation.     

Suboptimality of equal lot sizes for finite-horizon problems

We consider the problem of determining optimal lot sizes in continuous time for finite-horizon problems with stationary parameters. Using the average cost criterion, earlier researchers concluded that the optimal lot sizes should be equal. Using the conceptually rigorous discounted cash flow analysis, we show that equal lot sizes are optimal only when the finite horizon is an integral multiple of the optimal reorder interval for the infinite-horizon problem or, trivially, when the discount rate is zero. In all other cases, optimal lot sizes are either monotonically increasing or decreasing. Our characterization of the optimal policy is also useful in determining optimal lot sizes. © 1994 John Wiley & Sons, Inc.     

A model for use in the rationing of inventory during lead time

While the traditional solution to the problem of meeting stochastically variable demands for inventory during procurement lead time is through the use of some level of safety stock, several authors have suggested that a decision be made to employ some form of rationing so as to protect certain classes of demands against stockout by restricting issues to other classes. Nahmias and Demmy [10] derived an approximate continuous review model of systems with two demand classes which would permit an inventory manager to calculate the expected fill rates per order cycle for high-priority, low-priority, and total system demands for a variety of parameters. The manager would then choose the rationing policy that most closely approximated his fill-rate objectives. This article describes a periodic review model that permits the manager to establish a discrete time rationing policy during lead time by prescribing a desired service level for high-priority demands. The reserve levels necessary to meet this level of service can then be calculated based upon the assumed probability distributions of high- and low-priority demands over lead time. The derived reserve levels vary with the amount of lead time remaining. Simulation tests of the model indicate they are more effective than the single reserve level policy studied by Nahmias and Demmy.     

Optimal resource extraction with uncertain reserves

We consider the problem of finding a plan that maximizes the expected discounted return when extracting a nonrenewable resource having uncertain reserves. An extraction plan specifies the rate at which the resource is extracted as a function of time until the resource is exhausted or the time horizon is reached. The return per unit of resource extracted may depend on the rate of extraction, time, and the amount of resource previously extracted. We apply a new method called the generalized search optimization technique to find qualitative features of optimal plans and to devise algorithms for the numerical calculation of optimal plans.     

Dynamic server assignment policies for assembly‐type queues with flexible servers

We seek dynamic server assignment policies in finite‐capacity queueing systems with flexible and collaborative servers, which involve an assembly and/or a disassembly operation. The objective is to maximize the steady‐state throughput. We completely characterize the optimal policy for a Markovian system with two servers, two feeder stations, and instantaneous assembly and disassembly operations. This optimal policy allocates one server per station unless one of the stations is blocked, in which case both servers work at the unblocked station. For Markovian systems with three stations and instantaneous assembly and/or disassembly operations, we consider similar policies that move a server away from his/her “primary” station only when that station is blocked or starving. We determine the optimal assignment of each server whose primary station is blocked or starving in systems with three stations and zero buffers, by formulating the problem as a Markov decision process. Using this optimal assignment, we develop heuristic policies for systems with three or more stations and positive buffers, and show by means of a numerical study that these policies provide near‐optimal throughput. Furthermore, our numerical study shows that these policies developed for assembly‐type systems also work well in tandem systems. © 2008 Wiley Periodicals, Inc. Naval Research Logistics, 2008     

Replacement models under additive damage

A production system which generates income is subject to random failure. Upon failure, the system is replaced by a new identical one and the replacement cycles are repeated indefinitely. In our breakdown model, shocks occur to the system in a Poisson stream. Each shock causes a random amount of damage, and these damages accumulate additively. The failure time depends on the accumulated damage in the system. The income from the system and the cost associated with a planned replacement depend on the accumulated damage in the system. An additional cost is incurred at each failure in service. We allow a controller to replace the system at any stopping time T before failure time. We will consider the problem of specifying a replacement rule that is optimal under the following criteria: maximum total long-run average net income per unit time, and maximum total long-run expected discounted net income. Our primary goal is to introduce conditions under which an optimal policy is a control limit policy and to investigate how the optimal policy can be obtained. Examples will be presented to illustrate computational procedures.     

Optimal policies for stochastic clearing systems with time-dependent delay penalties

We study stochastic clearing systems with a discrete-time Markovian input process, and an output mechanism that intermittently and instantaneously clears the system partially or completely. The decision to clear the system depends on both quantities and delays of outstanding inputs. Clearing the system incurs a fixed cost, and outstanding inputs are charged a delay penalty, which is a general increasing function of the quantities and delays of individual inputs. By recording the quantities and delays of outstanding inputs in a sequence, we model the clearing system as a tree-structured Markov decision process over both a finite and infinite horizon. We show that the optimal clearing policies, under realistic conditions, are of the on-off type or the threshold type. Based on the characterization of the optimal policies, we develop efficient algorithms to compute parameters of the optimal policies for such complex clearing systems for the first time. We conduct a numerical analysis on the impact of the nonlinear delay penalty cost function, the comparison of the optimal policy and the classical hybrid policy (ie, quantity and age thresholds), and the impact of the state of the input process. Our experiments demonstrate that (a) the classical linear approximation of the cost function can lead to significant performance differences; (b) the classical hybrid policy may perform poorly (as compared to the optimal policies); and (c) the consideration of the state of the input process makes significant improvement in system performance.     

TTT transforms and age replacements with discounted costs

The ordinary age replacement problem consists of finding an optimal age at which a unit, needed in a continuous production process, should be replaced in order to minimize the average long-run cost per unit time. Bergman introduced a graphical procedure based on the total-time-on-test (TTT) concept for the analysis of the age replacement problem. In this article, that idea is generalized to the situation of discounted costs. We also study a more general age replacement problem in which we have a form of imperfect repair.