On the reduction of total‐cost and average‐cost MDPs to discounted MDPs

This article provides conditions under which total‐cost and average‐cost Markov decision processes (MDPs) can be reduced to discounted ones. Results are given for transient total‐cost MDPs with transition rates whose values may be greater than one, as well as for average‐cost MDPs with transition probabilities satisfying the condition that there is a state such that the expected time to reach it is uniformly bounded for all initial states and stationary policies. In particular, these reductions imply sufficient conditions for the validity of optimality equations and the existence of stationary optimal policies for MDPs with undiscounted total cost and average‐cost criteria. When the state and action sets are finite, these reductions lead to linear programming formulations and complexity estimates for MDPs under the aforementioned criteria.© 2017 Wiley Periodicals, Inc. Naval Research Logistics 66:38–56, 2019     

对多次命中时舰艇主动力装置生命力问题的研究

针对舰艇主动力装置生命力 ,以及多次命中的生命力难以进行定量分析 ,定义了主动力装置多次命中后所处的各种可能状态 ,并运用马尔科夫过程的有关理论 ,推导出了多次命中条件下舰艇处于各种状态的概率 .     

The LP/POMDP marriage: Optimization with imperfect information

A new technique for solving large‐scale allocation problems with partially observable states and constrained action and observation resources is introduced. The technique uses a master linear program (LP) to determine allocations among a set of control policies, and uses partially observable Markov decision processes (POMDPs) to determine improving policies using dual prices from the master LP. An application is made to a military problem where aircraft attack targets in a sequence of stages, with information acquired in one stage being used to plan attacks in the next. © 2000 John Wiley & Sons, Inc., Naval Research Logistics 47: 607–619, 2000     

具有多类索赔到达保险风险模型的破产问题

应用Markov骨架过程的方法和补充变量技巧研究了索赔为多类一般到达的保险风险模型 ,分别得到了破产时间与破产时刻前后资产盈余的联合分布以及破产时间的分布。使得索赔为一般到达的保险风险问题的研究取得了较大的进展。     

基于GSPN的编队网络化防空指挥流程研究

为了提高编队指挥员在信息化海战中的作战指挥效率和应变能力,利用广义随机Petri网与马尔可夫链的等价关系,得到一种Petri网与马尔可夫链理论相结合的指挥流程时间性能分析的新方法,通过对模型进行分析与求解,得出了舰艇编队网络化防空作战指挥流程活动的准确作战周期,从而为指挥流程的优化提供有力的支撑,对评估系统效能水平也具有重要指导意义。     

Optimal production ramp-up in the smartphone manufacturing industry

Motivated by challenges in the smartphone manufacturing industry, we develop a dynamic production ramp-up model that can be applied to economically satisfy nonstationary demand for short-life-cycle products by high-tech companies. Due to shorter life cycles and more rapid evolution of smartphones, production ramp-up has been increasingly critical to the success of a new smartphone. In the production ramp-up, the key challenge is to match the increasing capacity to nonstationary demand. The high-tech smartphone manufacturers are urged to jointly consider the effect of increasing capacity and decreasing demand. We study the production planning problem using a high-dimensional Markov decision process (MDP) model to characterize the production ramp-up. To address the curse of dimensionality, we refine Monte Carlo tree search (MCTS) algorithm and theoretically analyze its convergence and computational complexity. In a real case study, we find that the MDP model achieves revenue improvement by stopping producing the existing product earlier than the benchmark policy. In synthetic instances, we validate that the proposed MCTS algorithm saves computation time without loss of solution quality compared with traditional value iteration algorithm. As part of the Lenovo production solution, our MDP model enables high-tech smartphone manufacturers to better plan the production ramp-up.     

Revenue maximization in two-station tandem queueing systems

We study optimal pricing for tandem queueing systems with finite buffers. The service provider dynamically quotes prices to incoming price sensitive customers to maximize the long-run average revenue. We present a Markov decision process model for the optimization problem. For systems with two stations, general-sized buffers, and two or more prices, we describe the structure of the optimal dynamic pricing policy and develop tailored policy iteration algorithms to find an optimal pricing policy. For systems with two stations but no intermediate buffer, we characterize conditions under which quoting either a high or a low price to all customers is optimal and provide an easy-to-implement algorithm to solve the problem. Numerical experiments are conducted to compare the developed algorithms with the regular policy iteration algorithm. The work also discusses possible extensions of the obtained results to both three-station systems and two-station systems with price and congestion sensitive customers using numerical analysis.     

Reliability of a two-dimensional demand-based networked system with multistate components

Existing research on multistate system reliability has mainly focused on one-dimensional systems such as parallel systems, linear sliding window systems, and linearly consecutively connected systems. However, two-dimensional networked systems widely exist in real-world applications such as lighting systems, monitoring systems, and computer network systems. This research considers a two-dimensional networked system consisting of multistate components. The system fails if the cumulative performance of any row or any column cannot meet a predetermined demand. A novel reliability evaluation algorithm is proposed for the considered two-dimensional networked system by extending the universal generating function technique. Furthermore, the proposed model and reliability evaluation algorithm are extended to a two-dimensional networked system with phased missions. The proposed models and algorithms are illustrated by a matrix heating system in a thermoforming machine.     

一种针对基因识别的GHMM简化算法

广义隐Markov模型是计算机基因识别的一种重要模型,它克服了传统隐Markov模型的状态段长成几何分布的缺陷,更加适合于计算机基因识别。其缺点在于计算量大,需要采用有效的简化算法。利用基因的结构特点,在不附加额外限制条件的情况下,提出了一种新的简化算法,其计算复杂度是序列长度的线性函数。对实际生物序列数据的测试结果表明了此简化算法的有效性。     

马尔可夫链最优停止的值函数

本文讨论了时间无限的马尔可夫链的最优停止问题。对于无限状态情况,给出了其最优停止变量以及值函数存在的一个充分条件;对于有限状态情况,这个充分条件以及问题的计算等价于解一个线性规划问题。