Skip to main content
Centre for Cyber Defence and Information Security

Self-Learning Systems for Cyber Defense

A growing problem in system security stems from the fact that both attack methods and target systems constantly evolve: on the one hand attacks increase in sophistication over time, on the other hand target systems keep changing due to functional upgrades and innovation. As a consequence, a defender mustconstantly adapt and improve the target system in order to remain effective, which imposes an increasing burden on system designers and operators.

The goal of this project is to study, develop, and demonstrate strategies that will automate the defender’s task and keep a system secure in a changing environment. To this end, the project will investigate various approaches for self-learning systems that produce ever better defender strategies. The research adopts an adversarial approach, based on reinforcement learning, genetic algorithms, and game theory, whereby the strategies of simulated attackers and defenders co-evolve without human intervention.

Publications

Nyberg, J., & Johnson, P. (2026) "Vejde: A framework for inductive deep reinforcement learning based on factor graph color refinement. Transactions on Machine Learning Research.

Escriche, E. S., Nyberg, J., Kim, Y., & Dán, G. (2024) "Channel-centric spatio-temporal graph networks for network-based intrusion detection." In 2024 IEEE Conference on Communications and Network Security (CNS), 1–9.

Nyberg, J., & Johnson, P. (2024) "Structural generalization in autonomous cyber incident response with message-passing neural networks and reinforcement learning." In 2024 IEEE International Conference on Cyber Security and Resilience (CSR).

Gökstorp, S., Nyberg, J., Kim, Y., Johnson, P., & Dán, G. (2024) "Anomaly detection in security logs using sequence modeling." In NOMS 2024: IEEE Network Operations and Management Symposium, 1–9.

K. Hammar and R. Stadler, "Intrusion Tolerance for Networked Systems through Two-Level Feedback Control," 2024 54th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), Brisbane, Australia, 2024, pp. 338-352, doi: 10.1109/DSN58291.2024.00042

Nyberg, J., & Johnson, P. (2023). "Training automated defense strategies using graph-based cyber attack simulations," arXiv preprint arXiv:2304.11084.

K. Hammar and R. Stadler, ”Scalable Learning of Intrusion Response Through Recursive Decomposition”. In: Fu, J., Kroupa, T., Hayel, Y. (eds) Decision and Game Theory for Security. GameSec 2023. Lecture Notes in Computer Science, vol 14167. Springer, Cham. doi: 10.1007/978-3-031-50670-3_9  

K. Hammar and R. Stadler, "Learning Near-Optimal Intrusion Responses Against Dynamic Attackers," in IEEE Transactions on Network and Service Management, doi: .

K. Hammar and R. Stadler, "Digital Twins for Security Automation," NOMS 2023-2023 IEEE/IFIP Network Operations and Management Symposium, Miami, FL, USA, 2023, pp. 1-6, doi: .

Nyberg, J., Johnson, P., & Méhes, A. (2022). Cyber threat response using reinforcement learning in graph-based attack simulations. In NOMS 2022: IEEE/IFIP Network Operations and Management Symposium, 1–4.

K. Hammar and R. Stadler, "An Online Framework for Adapting Security Policies in Dynamic IT Environments," 2022 18th International Conference on Network and Service Management (CNSM), Thessaloniki, Greece, 2022, pp. 359-363, doi: .

K. Hammar and R. Stadler. "Learning Security Strategies through Game Play and Optimal Stopping", ICML Ml4Cyber Workshop 2022: International Conference on Machine Learning, arXiv: .

K. Hammar and R. Stadler, "A System for Interactive Examination of Learned Security Policies," NOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium, 2022, pp. 1-3, doi: 10.1109/NOMS54207.2022.9789707

K. Hammar and R. Stadler, "Intrusion Prevention through Optimal Stopping," in IEEE Transactions on Network and Service Management, .

K. Hammar and R. Stadler, "Learning Intrusion Prevention Policies through Optimal Stopping," 2021 17th International Conference on Network and Service Management (CNSM), 2021, pp. 509-517, doi: .

K. Hammar and R. Stadler, "Finding Effective Security Strategies through Reinforcement Learning and Self-Play," 2020 16th International Conference on Network and Service Management (CNSM), Izmir, Turkey, 2020, pp. 1-9,