Minimizing the false alarm rate in systems with transient abnormality

We consider a stochastic partially observable system that can switch between a normal state and a transient abnormal state before entering a persistent abnormal state. Only the persistent abnormal state requires alarms. The transient and persistent abnormal states may be similar in appearance, which can result in excess false alarms. We propose a partially observable Markov decision process model to minimize the false alarm rate, subject to a given upper bound on the expected alarm delay time. The cost parameter is treated as the Lagrange multiplier, which can be estimated from the bound of the alarm delay. We show that the optimal policy has a control‐limit structure on the probability of persistent abnormality, and derive closed‐form bounds for the control limit and present an algorithm to specify the Lagrange multiplier. We also study a specialized model where the transient and persistent abnormal states have the same observation distribution, in which case an intuitive “watchful‐waiting” policy is optimal. © 2016 Wiley Periodicals, Inc. Naval Research Logistics 63: 320–334, 2016     

Application of Markov renewal theory and semi-Markov decision processes in maintenance modeling and optimization of multi-unit systems

In this paper, a condition-based maintenance model for a multi-unit production system is proposed and analyzed using Markov renewal theory. The units of the system are subject to gradual deterioration, and the gradual deterioration process of each unit is described by a three-state continuous time homogeneous Markov chain with two working states and a failure state. The production rate of the system is influenced by the deterioration process and the demand is constant. The states of the units are observable through regular inspections and the decision to perform maintenance depends on the number of units in each state. The objective is to obtain the steady-state characteristics and the formula for the long-run average cost for the controlled system. The optimal policy is obtained using a dynamic programming algorithm. The result is validated using a semi-Markov decision process formulation and the policy iteration algorithm. Moreover, an analytical expression is obtained for the calculation of the mean time to initiate maintenance using the first passage time theory.     

A non-myopic scheduling method of radar sensors for maneuvering target tracking and radiation control

In decades, the battlefield environment is becoming more and more complex with plenty of electronic equipments. Thus, in order to improve the survivability of radar sensors and satisfy the requirement of maneuvering target tracking with a low probability of intercept, a non-myopic scheduling is proposed to minimize the radiation cost with tracking accuracy constraint. At first, the scheduling problem is formulated as a partially observable Markov decision process (POMDP). Then the tracking accuracy and radiation cost over the future finite time horizon are predicted by the posterior carmér-rao lower bound (PCRLB) and the hidden Markov model filter, respectively. Finally, the proposed scheduling is implemented efficiently by utilizing the branch and bound (B&B) pruning algorithm. Simulation results show that the performance of maneuvering target tracking was improved by the improved interacting multiple model (IMM), and the scheduler time and maximum memory consumption were significant reduced by the present B&B pruning algorithm without losing the optimal solution.     

Model parameter estimation and residual life prediction for a partially observable failing system

We consider a partially observable degrading system subject to condition monitoring and random failure. The system's condition is categorized into one of three states: a healthy state, a warning state, and a failure state. Only the failure state is observable. While the system is operational, vector data that is stochastically related to the system state is obtained through condition monitoring at regular sampling epochs. The state process evolution follows a hidden semi‐Markov model (HSMM) and Erlang distribution is used for modeling the system's sojourn time in each of its operational states. The Expectation‐maximization (EM) algorithm is applied to estimate the state and observation parameters of the HSMM. Explicit formulas for several important quantities for the system residual life estimation such as the conditional reliability function and the mean residual life are derived in terms of the posterior probability that the system is in the warning state. Numerical examples are presented to demonstrate the applicability of the estimation procedure and failure prediction method. A comparison results with hidden Markov modeling are provided to illustrate the effectiveness of the proposed model. © 2015 Wiley Periodicals, Inc. Naval Research Logistics 62: 190–205, 2015     

An optimal structured policy for maintenance of partially observable aircraft engine components

This paper considers the maintenance of aircraft engine components that are subject to stress. We model the deterioration process by means of the cumulative jump process representation of crack growth. However, because in many cases cracks are not easily observable, maintenance decisions must be made on the basis of other information. We incorporate stress information collected via sensors into the scheduling decision process by means of a partially observable Markov decision process model. Using this model, we demonstrate the optimality of structured maintenance policies, which support practical maintenance schedules. © 1998 John Wiley & Sons, Inc. Naval Research Logistics 45: 335–352, 1998     

Optimal replacement of continuously degrading systems in partially observed environments

Motivated by wind energy applications, we consider the problem of optimally replacing a stochastically degrading component that resides and operates in a partially observable environment. The component's rate of degradation is modulated by the stochastic environment process, and the component fails when it is accumulated degradation first reaches a fixed threshold. Assuming periodic inspection of the component, the objective is to minimize the long‐run average cost per unit time of performing preventive and reactive replacements for two distinct cases. The first case examines instantaneous replacements and fixed costs, while the second considers time‐consuming replacements and revenue losses accrued during periods of unavailability. Formulated and solved are mixed state space, partially observable Markov decision process models, both of which reveal the optimality of environment‐dependent threshold policies with respect to the component's cumulative degradation level. Additionally, it is shown that for each degradation value, a threshold policy with respect to the environment belief state is optimal if the environment alternates between two states. The threshold policies are illustrated by way of numerical examples using both synthetic and real wind turbine data. © 2015 Wiley Periodicals, Inc. Naval Research Logistics 62: 395–415, 2015     

Optimal policies for stochastic clearing systems with time-dependent delay penalties

We study stochastic clearing systems with a discrete-time Markovian input process, and an output mechanism that intermittently and instantaneously clears the system partially or completely. The decision to clear the system depends on both quantities and delays of outstanding inputs. Clearing the system incurs a fixed cost, and outstanding inputs are charged a delay penalty, which is a general increasing function of the quantities and delays of individual inputs. By recording the quantities and delays of outstanding inputs in a sequence, we model the clearing system as a tree-structured Markov decision process over both a finite and infinite horizon. We show that the optimal clearing policies, under realistic conditions, are of the on-off type or the threshold type. Based on the characterization of the optimal policies, we develop efficient algorithms to compute parameters of the optimal policies for such complex clearing systems for the first time. We conduct a numerical analysis on the impact of the nonlinear delay penalty cost function, the comparison of the optimal policy and the classical hybrid policy (ie, quantity and age thresholds), and the impact of the state of the input process. Our experiments demonstrate that (a) the classical linear approximation of the cost function can lead to significant performance differences; (b) the classical hybrid policy may perform poorly (as compared to the optimal policies); and (c) the consideration of the state of the input process makes significant improvement in system performance.     

Maintaining systems with heterogeneous spare parts

We consider the problem of optimally maintaining a stochastically degrading, single‐unit system using heterogeneous spares of varying quality. The system's failures are unannounced; therefore, it is inspected periodically to determine its status (functioning or failed). The system continues in operation until it is either preventively or correctively maintained. The available maintenance options include perfect repair, which restores the system to an as‐good‐as‐new condition, and replacement with a randomly selected unit from the supply of heterogeneous spares. The objective is to minimize the total expected discounted maintenance costs over an infinite time horizon. We formulate the problem using a mixed observability Markov decision process (MOMDP) model in which the system's age is observable but its quality must be inferred. We show, under suitable conditions, the monotonicity of the optimal value function in the belief about the system quality and establish conditions under which finite preventive maintenance thresholds exist. A detailed computational study reveals that the optimal policy encourages exploration when the system's quality is uncertain; the policy is more exploitive when the quality is highly certain. The study also demonstrates that substantial cost savings are achieved by utilizing our MOMDP‐based method as compared to more naïve methods of accounting for heterogeneous spares.     

Technical note: A computationally efficient algorithm for undiscounted Markov decision processes with restricted observations

We present a computationally efficient procedure to determine control policies for an infinite horizon Markov Decision process with restricted observations. The optimal policy for the system with restricted observations is a function of the observation process and not the unobservable states of the system. Thus, the policy is stationary with respect to the partitioned state space. The algorithm we propose addresses the undiscounted average cost case. The algorithm combines a local search with a modified version of Howard's (Dynamic programming and Markov processes, MIT Press, Cambridge, MA, 1960) policy iteration method. We demonstrate empirically that the algorithm finds the optimal deterministic policy for over 96% of the problem instances generated. For large scale problem instances, we demonstrate that the average cost associated with the local optimal policy is lower than the average cost associated with an integer rounded policy produced by the algorithm of Serin and Kulkarni Math Methods Oper Res 61 (2005) 311–328. © 2008 Wiley Periodicals, Inc. Naval Research Logistics 2009     

Criteria and approximate methods for path-constrained moving-target search problems

A search is conducted for a target moving in discrete time among a finite number of cells according to a known Markov process. The searcher must choose one cell in which to search in each time period. The set of cells available for search depends upon the cell chosen in the last time period. The problem is to find a search path, i.e., a sequence of search cells, that either maximizes the probability of detection or minimizes the mean number of time periods required for detection. The search problem is modelled as a partially observable Markov decision process and several approximate solutions procedures are proposed. © 1995 John Wiley & Sons, Inc.     

On‐line parameter estimation for a partially observable system subject to random failure

In this paper, we study the on‐line parameter estimation problem for a partially observable system subject to deterioration and random failure. The state of the system evolves according to a continuous time homogeneous Markov process with a finite state space. The system state is not observable, except for the failure state. The information related to the system state is available at discrete times through inspections. A recursive maximum likelihood (RML) algorithm is proposed for the on‐line parameter estimation of the model. The RML algorithm proposed in the paper is considerably faster and easier to apply than other RML algorithms in the literature, because it does not require projection into the constraint domain and calculation of the gradient on the surface of the constraint manifolds. The algorithm is illustrated by an example using real vibration data. © 2006 Wiley Periodicals, Inc. Naval Research Logistics, 2006     

Optimal admission pricing policies for M/Ek/1 queues

This paper extends the Low-Lippman M/M/1 model to the case of Gamma service times. Specifically, we have a queue in which arrivals are Poisson, service time is Gamma-distributed, and the arrival rate to the system is subject to setting an admission fee p. The arrival rate λ(p) is non-increasing in p. We prove that the optimal admission fee p^* is a non-decreasing function of the customer work load on the server. The proof is for an infinite capacity queue and holds for the infinite horizon continuous time Markov decision process. In the special case of exponential service time, we extend the Low-Lippman model to include a state-dependent service rate and service cost structure (for finite or infinite time horizon and queue capacity). Relatively recent dynamic programming techniques are employed throughout the paper. Due to the large class of functions represented by the Gamma family, the extension is of interest and utility.     

Optimal maintenance policies for a safety‐critical system and its deteriorating sensor

We consider the integrated problem of optimally maintaining an imperfect, deteriorating sensor and the safety‐critical system it monitors. The sensor's costless observations of the binary state of the system become less informative over time. A costly full inspection may be conducted to perfectly discern the state of the system, after which the system is replaced if it is in the out‐of‐control state. In addition, a full inspection provides the opportunity to replace the sensor. We formulate the problem of adaptively scheduling full inspections and sensor replacements using a partially observable Markov decision process (POMDP) model. The objective is to minimize the total expected discounted costs associated with system operation, full inspection, system replacement, and sensor replacement. We show that the optimal policy has a threshold structure and demonstrate the value of coordinating system and sensor maintenance via numerical examples. © 2017 Wiley Periodicals, Inc. Naval Research Logistics 64: 399–417, 2017     

Solving partially observable agent‐intruder games with an application to border security problems

In this paper, we introduce partially observable agent‐intruder games (POAIGs). These games model dynamic search games on graphs between security forces (an agent) and an intruder given possible (border) entry points and high value assets that require protection. The agent faces situations with dynamically changing, partially observable information about the state of the intruder and vice versa. The agent may place sensors at selected locations, while the intruder may recruit partners to observe the agent's movement. We formulate the problem as a two‐person zero‐sum game, and develop efficient algorithms to compute each player's optimal strategy. The solution to the game will help the agent choose sensor locations and design patrol routes that can handle imperfect information. First, we prove the existence of ?‐optimal strategies for POAIGs with an infinite time horizon. Second, we introduce a Bayesian approximation algorithm to identify these ?‐optimal strategies using belief functions that incorporate the imperfect information that becomes available during the game. For the solutions of large POAIGs with a finite time horizon, we use a solution method common to extensive form games, namely, the sequence form representation. To illustrate the POAIGs, we present several examples and numerical results.     

The LP/POMDP marriage: Optimization with imperfect information

A new technique for solving large‐scale allocation problems with partially observable states and constrained action and observation resources is introduced. The technique uses a master linear program (LP) to determine allocations among a set of control policies, and uses partially observable Markov decision processes (POMDPs) to determine improving policies using dual prices from the master LP. An application is made to a military problem where aircraft attack targets in a sequence of stages, with information acquired in one stage being used to plan attacks in the next. © 2000 John Wiley & Sons, Inc., Naval Research Logistics 47: 607–619, 2000     

Optimal switchover times between two activities utilizing the same resource

The “gold‐mining” decision problem is concerned with the efficient utilization of a delicate mining equipment working in a number of different mines. Richard Bellman was the first to consider this type of a problem. The solution found by Bellman for the finite‐horizon, continuous‐time version of the problem with two mines is not overly realistic since he assumed that fractional parts of the same mining equipment could be used in different mines and this fraction could change instantaneously. In this paper, we provide some extensions to this model in order to produce more operational and realistic solutions. Our first model is concerned with developing an operational policy where the equipment may be switched from one mine to the other at most once during a finite horizon. In the next extension we incorporate a cost component in the objective function and assume that the horizon length is not fixed but it is the second decision variable. Structural properties of the optimal solutions are obtained using nonlinear programming. Each model and its solution is illustrated with a numerical example. The models developed here may have potential applications in other areas including production of items requiring the same machine or choosing a sequence of activities requiring the same resource. © 2002 Wiley Periodicals, Inc. Naval Research Logistics 49: 186–203, 2002; DOI 10.1002/nav.10008     

Monotone optimal preventive maintenance policies for stochastically failing equipment

This paper examines various models for maintenance of a machine operating subject to stochastic deterioration. Three alternative models are presented for the deterioration process. For each model, in addition to the replacement decision, the option exists of performing preventive maintenance. The effect of this maintenance is to “slow” the deterioration process. With an appropriate reward structure imposed on the processes, the models are formulated as continuous time Markov decision processes. the optimality criterion being the maximization of expected discounted reward earned over an infinite time horizon. For each model conditions are presented under which the optimal maintenance policy exhibits the following monotonic structure. First, there exists a control limit rule for replacement. That is, there exists a number i* such that if the state of machine deterioration exceeds i* the optimal policy replaces the machine by a new machine. Secondly, prior to replacement the optimal level of preventive maintenance is a nonincreasing function of the state of machine deterioration. The conditions which guarantee this result have a cost/benefit interpretation.     

A single-machine replacement model with learning

The replacement or upgrade of productive resources over time is an important decision for a manufacturing organization. The type of technology used in the productive resources determines how effectively the manufacturing operations can support the product and marketing strategy of the organization. Increasing operating costs (cost of maintenance, labor, and depreciation) over time force manufacturing organizations to periodically consider replacement or upgrade of their existing productive resources. We assume that there is a setup cost associated with the replacement of a machine, and that the setup cost is a nonincreasing function of the number of replacements made so far due to learning in setups. The operating cost of a newer machine is assumed to be lower than the operating cost of an older machine in any given period, except perhaps in the first period of operation of the new machine when the cost could be unusually high due to higher initial depreciation. A forward dynamic programming algorithm is developed which can be used to solve finite-horizon problems. We develop procedures to find decision and forecast horizons such that choices made during the decision horizon based only on information over the forecast horizon are also optimal for any longer horizon problem. Thus, we are able to obtain optimal results for what is effectively an infinite-horizon problem while only requiring data over a finite period of time. We present a numerical example to illustrate the decision/forecast horizon procedure, as well as a study of the effects of considering learning in making a series of machine replacement decisions. © 1993 John Wiley & Sons. Inc.     

On the reduction of total‐cost and average‐cost MDPs to discounted MDPs

This article provides conditions under which total‐cost and average‐cost Markov decision processes (MDPs) can be reduced to discounted ones. Results are given for transient total‐cost MDPs with transition rates whose values may be greater than one, as well as for average‐cost MDPs with transition probabilities satisfying the condition that there is a state such that the expected time to reach it is uniformly bounded for all initial states and stationary policies. In particular, these reductions imply sufficient conditions for the validity of optimality equations and the existence of stationary optimal policies for MDPs with undiscounted total cost and average‐cost criteria. When the state and action sets are finite, these reductions lead to linear programming formulations and complexity estimates for MDPs under the aforementioned criteria.© 2017 Wiley Periodicals, Inc. Naval Research Logistics 66:38–56, 2019     

Modeling and analysis of a mixed‐model assembly line with stochastic operation times

We consider a mixed‐model assembly line (MMAL) comprised a set of workstations and a conveyor. The workstations are arranged in a serial configuration. The conveyor moves at a constant speed along the workstations. Initial units belonging to different models are successively fed onto the conveyor, and they are moved by the conveyor to pass through the workstations to gradually generate final products. All assembling tasks are manually performed with operation times to be stochastic. An important performance measure of MMALs is overload times that refer to uncompleted operations for operators within their work zones. This paper establishes a method to analyze the expected overload times for MMALs with stochastic operation times. The operation processes of operators form discrete time nonhomogeneous Markov processes with continuous state spaces. For a given daily production schedule, the expected overload times involve in analyzing the Markov processes for finite horizon. Based on some important properties of the performance measure, we propose an efficient approach for calculating the expected overload times. Numerical computations show that the results are very satisfactory. © 2007 Wiley Periodicals, Inc. Naval Research Logistics, 2007