Markov decision processes with restricted observations: Finite horizon case

In this article we consider a Markov decision process subject to the constraints that result from some observability restrictions. We assume that the state of the Markov process under consideration is unobservable. The states are grouped so that the group that a state belongs to is observable. So, we want to find an optimal decision rule depending on the observable groups instead of the states. This means that the same decision applies to all the states in the same group. We prove that a deterministic optimal policy exists for the finite horizon. An algorithm is developed to compute policies minimizing the total expected discounted cost over a finite horizon. © 1997 John Wiley & Sons, Inc. Naval Research Logistics 44 : 439–456, 1997     

Application of Markov renewal theory and semi-Markov decision processes in maintenance modeling and optimization of multi-unit systems

In this paper, a condition-based maintenance model for a multi-unit production system is proposed and analyzed using Markov renewal theory. The units of the system are subject to gradual deterioration, and the gradual deterioration process of each unit is described by a three-state continuous time homogeneous Markov chain with two working states and a failure state. The production rate of the system is influenced by the deterioration process and the demand is constant. The states of the units are observable through regular inspections and the decision to perform maintenance depends on the number of units in each state. The objective is to obtain the steady-state characteristics and the formula for the long-run average cost for the controlled system. The optimal policy is obtained using a dynamic programming algorithm. The result is validated using a semi-Markov decision process formulation and the policy iteration algorithm. Moreover, an analytical expression is obtained for the calculation of the mean time to initiate maintenance using the first passage time theory.     

Maintaining systems with heterogeneous spare parts

We consider the problem of optimally maintaining a stochastically degrading, single‐unit system using heterogeneous spares of varying quality. The system's failures are unannounced; therefore, it is inspected periodically to determine its status (functioning or failed). The system continues in operation until it is either preventively or correctively maintained. The available maintenance options include perfect repair, which restores the system to an as‐good‐as‐new condition, and replacement with a randomly selected unit from the supply of heterogeneous spares. The objective is to minimize the total expected discounted maintenance costs over an infinite time horizon. We formulate the problem using a mixed observability Markov decision process (MOMDP) model in which the system's age is observable but its quality must be inferred. We show, under suitable conditions, the monotonicity of the optimal value function in the belief about the system quality and establish conditions under which finite preventive maintenance thresholds exist. A detailed computational study reveals that the optimal policy encourages exploration when the system's quality is uncertain; the policy is more exploitive when the quality is highly certain. The study also demonstrates that substantial cost savings are achieved by utilizing our MOMDP‐based method as compared to more naïve methods of accounting for heterogeneous spares.     

油料装备智能维修决策系统研究

在分析油料装备维修管理特点的基础上，提出油料装备智能维修决策系统体系结构，论述模型库与知识库的构建方法，描述维修推理机工作原理。针对油料装备故障诊断信息具有模糊性的特点，建立油料装备故障智能模糊诊断模型，论述故障诊断模糊规则、模糊推理步骤、模糊算子定义及其推理方式。对维修策略决策的主要影响因素进行分析，以使油料装备稳定性及经济性、维修周期达到最佳为维修策略目标，构建油料装备维修策略模型，论述维修策略原则和模型工作流程。油料装备智能维修决策系统的研究，对提高油料装备维修科学化和智能化水平，加强油料装备保障力度具有重要意义。     

Optimal maintenance policies for a safety‐critical system and its deteriorating sensor

We consider the integrated problem of optimally maintaining an imperfect, deteriorating sensor and the safety‐critical system it monitors. The sensor's costless observations of the binary state of the system become less informative over time. A costly full inspection may be conducted to perfectly discern the state of the system, after which the system is replaced if it is in the out‐of‐control state. In addition, a full inspection provides the opportunity to replace the sensor. We formulate the problem of adaptively scheduling full inspections and sensor replacements using a partially observable Markov decision process (POMDP) model. The objective is to minimize the total expected discounted costs associated with system operation, full inspection, system replacement, and sensor replacement. We show that the optimal policy has a threshold structure and demonstrate the value of coordinating system and sensor maintenance via numerical examples. © 2017 Wiley Periodicals, Inc. Naval Research Logistics 64: 399–417, 2017     

Machine maintenance with workload considerations

Machine maintenance is modeled in the setting of a single‐server queue. Machine deterioration corresponds to slower service rates and failure. This leads to higher congestion and an increase in customer holding costs. The decision‐maker decides when to perform maintenance, which may be done pre‐emptively; before catastrophic failures. Similar to classic maintenance control models, the information available to the decision‐maker includes the state of the server. Unlike classic models, the information also includes the number of customers in queue. Considered are both a repair model and a replacement model. In the repair model, with random replacement times, fixed costs are assumed to be constant in the server state. In the replacement model, both constant and variable fixed costs are considered. It is shown in general that the optimal maintenance policies have switching curve structure that is monotone in the server state. However, the switching curve policies for the repair model are not always monotone in the number of customers in the queue. Numerical examples and two heuristics are also presented. © 2007 Wiley Periodicals, Inc. Naval Research Logistics, 2007     

Optimizing usage and maintenance decisions for k‐out‐of‐n systems of moving assets

We consider an integrated usage and maintenance optimization problem for a k‐out‐of‐n system pertaining to a moving asset. The k‐out‐of‐n systems are commonly utilized in practice to increase availability, where n denotes the total number of parallel and identical units and k the number of units required to be active for a functional system. Moving assets such as aircraft, ships, and submarines are subject to different operating modes. Operating modes can dictate not only the number of system units that are needed to be active, but also where the moving asset physically is, and under which environmental conditions it operates. We use the intrinsic age concept to model the degradation process. The intrinsic age is analogous to an intrinsic clock which ticks on a different pace in different operating modes. In our problem setting, the number of active units, degradation rates of active and standby units, maintenance costs, and type of economic dependencies are functions of operating modes. In each operating mode, the decision maker should decide on the set of units to activate (usage decision) and the set of units to maintain (maintenance decision). Since the degradation rate differs for active and standby units, the units to be maintained depend on the units that have been activated, and vice versa. In order to minimize maintenance costs, usage and maintenance decisions should be jointly optimized. We formulate this problem as a Markov decision process and provide some structural properties of the optimal policy. Moreover, we assess the performance of usage policies that are commonly implemented for maritime systems. We show that the cost increase resulting from these policies is up to 27% for realistic settings. Our numerical experiments demonstrate the cases in which joint usage and maintenance optimization is more valuable. © 2017 Wiley Periodicals, Inc. Naval Research Logistics 64: 418–434, 2017     

Monotone optimal preventive maintenance policies for stochastically failing equipment

This paper examines various models for maintenance of a machine operating subject to stochastic deterioration. Three alternative models are presented for the deterioration process. For each model, in addition to the replacement decision, the option exists of performing preventive maintenance. The effect of this maintenance is to “slow” the deterioration process. With an appropriate reward structure imposed on the processes, the models are formulated as continuous time Markov decision processes. the optimality criterion being the maximization of expected discounted reward earned over an infinite time horizon. For each model conditions are presented under which the optimal maintenance policy exhibits the following monotonic structure. First, there exists a control limit rule for replacement. That is, there exists a number i* such that if the state of machine deterioration exceeds i* the optimal policy replaces the machine by a new machine. Secondly, prior to replacement the optimal level of preventive maintenance is a nonincreasing function of the state of machine deterioration. The conditions which guarantee this result have a cost/benefit interpretation.     

Minimizing the false alarm rate in systems with transient abnormality

We consider a stochastic partially observable system that can switch between a normal state and a transient abnormal state before entering a persistent abnormal state. Only the persistent abnormal state requires alarms. The transient and persistent abnormal states may be similar in appearance, which can result in excess false alarms. We propose a partially observable Markov decision process model to minimize the false alarm rate, subject to a given upper bound on the expected alarm delay time. The cost parameter is treated as the Lagrange multiplier, which can be estimated from the bound of the alarm delay. We show that the optimal policy has a control‐limit structure on the probability of persistent abnormality, and derive closed‐form bounds for the control limit and present an algorithm to specify the Lagrange multiplier. We also study a specialized model where the transient and persistent abnormal states have the same observation distribution, in which case an intuitive “watchful‐waiting” policy is optimal. © 2016 Wiley Periodicals, Inc. Naval Research Logistics 63: 320–334, 2016     

Simultaneous determination of production and maintenance schedules using in‐line equipment condition and yield information

In many manufacturing environments, equipment condition has a significant impact on product quality, or yield. This paper presents a semi‐Markov decision process model of a single‐stage production system with multiple products and multiple maintenance actions. The model simultaneously determines maintenance and production schedules, accounting for the fact that equipment condition affects the yield of each product differently. It extends earlier work by allowing the expected time between decision epochs to vary by both action and machine state, by allowing multiple maintenance actions, and by treating the outcome of maintenance as less than certain. Sufficient conditions are developed that ensure the monotonicity of both the optimal production and maintenance actions. While the maintenance conditions closely resemble previously studied conditions for this type of problem, the production conditions represent a significant departure from earlier results. The simultaneous solution method is compared to an approach commonly used in industry, where the maintenance and production problems are treated independently. Solving more than one thousand test problems confirms that the combination of both features of the model—accounting for product differences and solving the problems simultaneously—has a significant impact on performance. © 2007 Wiley Periodicals, Inc. Naval Research Logistics, 2008     

基于风险评估的检测与修理决策模型

研究了基于故障风险的检测与维修策略，运用威布尔比例风险模型建立系统故障率与工作时间及所处状态的关系。分析了两类基于间接状态信息维修决策的风险，建立了系统检测及维修策略的决策树，通过比较检测与不检测情况下的期望费用确定最优的检测间隔。算例表明，所提出的方法能够有效控制系统运行风险，降低系统运行费用。     

基于二维量度的复杂设备预防性维修决策优化

针对使用与维修具有两个测量维度的复杂设备,开展了其预防性维修决策的优化研究。基于二维量度的故障模式,给出了二维故障率的定量描述方法;分析了其预防性更换过程的基本过程,探讨了二维量度下更换周期对维修效果的影响,并从经济性角度建立了二维工龄更换费用模型;最后,采用算例的形式,对某设备维修决策同时考虑日历使用时间和行驶里程的情况,进行了二维更换间隔期的优化求解,从而验证了所建立方法与模型的实用性。     

基于IDEF3的部队装备中修业务流程优化重组

以IDEF3为工具描述了部队装备中修业务流程,分析了流程中存在的问题,通过采用基于状态的维修方式、无损检测先进技术手段、共享维修保障信息和调整器材发放审批权限等方法,对部队现行装备中修业务流程进行了优化,可有效解决装备维修过剩、维修不足、维修资源浪费和维修周期过长等问题．     

Criteria and approximate methods for path-constrained moving-target search problems

A search is conducted for a target moving in discrete time among a finite number of cells according to a known Markov process. The searcher must choose one cell in which to search in each time period. The set of cells available for search depends upon the cell chosen in the last time period. The problem is to find a search path, i.e., a sequence of search cells, that either maximizes the probability of detection or minimizes the mean number of time periods required for detection. The search problem is modelled as a partially observable Markov decision process and several approximate solutions procedures are proposed. © 1995 John Wiley & Sons, Inc.     

相控阵雷达维修方案优选方法研究

针对相控阵雷达维修方案的选择问题,确定了维修方案的决策属性,并提出一种基于改进TOPSIS法的相控阵雷达维修方案优选方法。该方法通过引入Vague集来处理维修方案选择中存在的模糊信息,并采用博弈论综合赋权法来确定决策属性的综合权重。通过计算各备选方案与正、负理想方案的灰色关联度,得到了维修方案的灰色关联相对贴近度,进而实现对维修方案的排序和选择。最后,通过算例应用与分析,对提出的方法进行了验证。     

主战飞机维修保障效能评估方法

针对空军主战飞机维修保障效能难以评估的问题,首次提出了主战飞机维修保障效能评估方法。分析了飞机维修保障系统运行过程,按照"投入-产出"的原则,建立了主战飞机维修保障效能评估指标体系,运用信息熵理论和灰色系统理论,构建出了基于熵权和灰色聚类的评估指标赋权模型。结合实例验证了方法的正确性和合理性。     

A bayesian approach to environmental stress screening

The article presents a Bayesian analysis for the environmental stress screening problem. The decision problem of deriving optimal stress screen durations is solved. Given a screen duration, the optimal stress level can also be determined. Indicators of the quality of a screen of any duration are derived. A statistical model is presented which allows a posterior density for the rate of early failures of the production process to be calculated. This enables the user to update his opinion about the quality of the process. © 1994 John Wiley & Sons, Inc.     

Optimal joint preventive maintenance and production policies

We study joint preventive maintenance (PM) and production policies for an unreliable production‐inventory system in which maintenance/repair times are non‐negligible and stochastic. A joint policy decides (a) whether or not to perform PM and (b) if PM is not performed, then how much to produce. We consider a discrete‐time system, formulating the problem as a Markov decision process (MDP) model. The focus of the work is on the structural properties of optimal joint policies, given the system state comprised of the system's age and the inventory level. Although our analysis indicates that the structure of optimal joint policies is very complex in general, we are able to characterize several properties regarding PM and production, including optimal production/maintenance actions under backlogging and high inventory levels, and conditions under which the PM portion of the joint policy has a control‐limit structure. In further special cases, such as when PM set‐up costs are negligible compared to PM times, we are able to establish some additional structural properties. © 2005 Wiley Periodicals, Inc. Naval Research Logistics, 2005.     

A non-myopic scheduling method of radar sensors for maneuvering target tracking and radiation control

In decades, the battlefield environment is becoming more and more complex with plenty of electronic equipments. Thus, in order to improve the survivability of radar sensors and satisfy the requirement of maneuvering target tracking with a low probability of intercept, a non-myopic scheduling is proposed to minimize the radiation cost with tracking accuracy constraint. At first, the scheduling problem is formulated as a partially observable Markov decision process (POMDP). Then the tracking accuracy and radiation cost over the future finite time horizon are predicted by the posterior carmér-rao lower bound (PCRLB) and the hidden Markov model filter, respectively. Finally, the proposed scheduling is implemented efficiently by utilizing the branch and bound (B&B) pruning algorithm. Simulation results show that the performance of maneuvering target tracking was improved by the improved interacting multiple model (IMM), and the scheduler time and maximum memory consumption were significant reduced by the present B&B pruning algorithm without losing the optimal solution.     

基于兰切斯特方程的网络中心战模型

为了更全面、客观地评价信息优势和决策优势对网络中心战进程的影响,在分析经典兰切斯特方程的基础上,引入信息?决策概率,提出了基于兰切斯特方程的网络中心战模型,仿真实验表明该模型能够有效地描述信息?决策概率在网络中心战进程中发挥的作用,为研究分析网络中心作战过程提供了理论参考。