Decision support based on adaptive fuzzy situational networks and deep reinforcement learning

Alexander Shapkin¹, Vadim Borisov^{1, 2}

¹Branch of the National Research University Moscow Power Engineering Institute in Smolensk.
²Military Academy of Army Air Defence A.M. Vasilevsky.

DOI 10.24412/2410-9916-2026-4-088-102

Full text

XML JATS

Abstract

Problem statement: To support control decision-making in complex technical systems operating under uncertainty, it is necessary to determine the possible system states, available control actions, and a control strategy. Known fuzzy situational models allow one to represent a complex system as a network of situations and control actions; however, they do not provide automatic evaluation of permissible control decisions taking into account the selected control strategy. Objective: to develop a method for supporting control decision-making in adaptive fuzzy situational networks based on deep reinforcement learning, ensuring the evaluation of permissible control actions taking into account their long-term consequences. Methods used: the process of generating recommendations is presented as a reinforcement learning problem. The deep reinforcement learning method is used to evaluate control decisions. A target network and an experience replay buffer are used to stabilize learning. Inadmissible actions are excluded using a binary transition mask generated based on the current structure of the adaptive fuzzy situational network. The control strategy is defined by a parameterized reward function including components of achieving the target situation, the cost of the control transition, and the values of the parameters of the next fuzzy situation. Novelty: a method is proposed that enables the use of control decision states not only for autonomous action selection but also for intelligent operator support. For each feasible control decision, an estimate is determined that characterizes its expected utility, taking into account subsequent transitions. Based on the obtained estimates, a ranked set of alternatives is formed. Joint processing of the observation history and features of all fuzzy situations allows for the application of a single agent architecture to networks of varying dimensions, while dynamic masking ensures the compliance of recommendations with the current model structure. Result: a method for generating a ranked set of feasible control decisions consistent with a given control strategy has been developed. An example of decision support for managing an energy mini-grid is considered. Numerical estimates are generated for alternative transitions, allowing for the justification of the choice of a safe solution, taking into account not only the immediate cost of the impact but also the expected consequences of further movement through the network. Practical significance: The proposed method can be used in intelligent decision support systems for operators of complex technical and organizational systems. The system provides the operator with feasible alternatives, their ranking, and quantitative estimates of expected utility, while the final selection and implementation of the control decision remains with the decision maker.

Key words

decision support, situational management, control decision evaluation, adaptive fuzzy situational networks, deep reinforcement learning, Deep Recurrent Q-Network.

Reference for citation

Shapkin A. P., Borisov V. V. Decision support based on adaptive fuzzy situational networks and deep reinforcement learning. Systems of Control, Communication and Security, 2026, no. 4, pp. 88-102 (in Russian). DOI: 10.24412/2410-9916-2026-4-088-102

References

1. Melikhov A. N., Bershteyn L. S., Korovin S. Ya. Situatsionnye sovetuyushchie sistemy s nechetkoy logikoy [Situational Advising Systems with Fuzzy Logic]. Moscow, Nauka Publ., 1990. 272 p. (in Russian).

2. Borisov V. V., Avramenko D. Yu. Fuzzy Situational Control of Complex Systems Based on Composite Hybrid Modeling. Systems of Control, Communication and Security, 2021, no. 3, pp. 207–237, DOI: 10.24412/2410-9916-2021-3-207-237 (in Russian).

3. Borisov V. V., Zernov M. M. Realizatsiya situatsionnogo podkhoda na osnove nechetkoy ierarkhicheskoy situatsionno-sobytiynoy seti [Implementation of the Situational Approach Based on a Fuzzy Hierarchical Situational-Event Network]. Artificial Intelligence and Decision Making, 2009, no. 1, pp. 17–30 (in Russian).

4. Denisenkov M. A., Borisov V. V. Modeling the Behavior of Intelligent Agents Based on Adaptive Fuzzy Situational Networks. Proceedings of the 3rd Russian-Pacific Conference on Computer Technology and Application (RPC), Vladivostok, Russia, 18–25 August 2018, pp. 1–5, DOI: 10.1109/RPC.2018.8482217.

5. Sokolov A. M., Borisov V. V. Recurrent Fuzzy Situational-Precedent Models for Operational Control of Complex Technical Objects. Programmnye produkty i sistemy, 2024, vol. 37, no. 4, pp. 554–565, DOI: 10.15827/0236-235X.148.554-565 (in Russian).

6. Dli M. I., Chernovalova M. V., Sokolov A. M. Construction of Fuzzy Situational-Precedent Models of Regional Economic Systems Using Ontologies. Journal of Applied Informatics, 2025, vol. 20, no. 2, pp. 112–125, DOI: 10.37791/2687-0649-2025-20-2-112-125 (in Russian).

7. AlMahamid F., Grolinger K. Reinforcement Learning Algorithms: An Overview and Classification. arXiv, 2022. Available at: https://arxiv.org/abs/2209.14940 (accessed 16 March 2024).

8. Borisov V. V., Shapkin A. P. Fuzzy Situational Networks with Reinforcement Learning-Based Adaptation. MPEI Vestnik, 2025, no. 3, pp. 135–143, DOI: 10.24160/1993-6982-2025-3-135-143 (in Russian).

9. Shapkin A. P., Borisov V. V. Sposoby vnedreniya obucheniya s podkrepleniem v adaptivnye nechetkie situatsionnye seti [Methods for Integrating Reinforcement Learning into Adaptive Fuzzy Situational Networks]. XXII Mezhdunarodnaya nauchno-tekhnicheskaya konferentsiya studentov i aspirantov. Trudy konferentsii [Proceedings of the 22nd International Scientific and Technical Conference of Students and Postgraduates], Smolensk, 23–24 April 2025, Smolensk, Universum Publ., 2025, vol. 1, pp. 470–474 (in Russian).

10. Mnih V., Kavukcuoglu K., Silver D., Rusu A. A., Veness J., Bellemare M. G., Graves A., Riedmiller M., Fidjeland A. K., Ostrovski G., Petersen S., Beattie C., Sadik A., Antonoglou I., King H., Kumaran D., Wierstra D., Legg S., Hassabis D. Human-Level Control through Deep Reinforcement Learning. Nature, 2015, vol. 518, pp. 529–533, DOI: 10.1038/nature14236.

11. Hochreiter S., Schmidhuber J. Long Short-Term Memory. Neural Computation, 1997, vol. 9, no. 8, pp. 1735–1780, DOI: 10.1162/neco.1997.9.8.1735.

12. Hausknecht M., Stone P. Deep Recurrent Q-Learning for Partially Observable MDPs. arXiv, 2015. Available at: https://arxiv.org/abs/1507.06527 (accessed 16 March 2024).

13. Moreno-Vera F. Performing Deep Recurrent Double Q-Learning for Atari Games. arXiv, 2019. Available at: https://arxiv.org/abs/1908.06040 (accessed 16 March 2024).

14. Narvekar S., Peng B., Leonetti M., Sinapov J., Taylor M. E., Stone P. Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey. Journal of Machine Learning Research, 2020, vol. 21, no. 181, pp. 1–50.

15. Huang S., Ontanon S. A Closer Look at Invalid Action Masking in Policy Gradient Algorithms. arXiv, 2020. Available at: https://arxiv.org/abs/2006.14171 (accessed 16 March 2024).

16. Ba J. L., Kiros J. R., Hinton G. E. Layer Normalization. arXiv, 2016. Available at: https://arxiv.org/abs/1607.06450 (accessed 22 June 2026).

17. Dulac-Arnold G., Evans R., van Hasselt H., Sunehag P., Lillicrap T., Hunt J., Mann T., Weber T., Degris T., Coppin B. Deep Reinforcement Learning in Large Discrete Action Spaces. arXiv, 2015. Available at: https://arxiv.org/abs/1512.07679 (accessed 22 June 2026).

18. Zhu P., Li X., Poupart P., Miao G. On Improving Deep Reinforcement Learning for POMDPs. arXiv, 2018. Available at: https://arxiv.org/abs/1804.06309 (accessed 16 March 2024).

 

 This article is distributed under a license Creative Commons Attribution 4.0 License.

  The metadata of the article is distributed under a license CC0 1.0 Universal