Doctoral Thesis

Supervisors:

Main Supervisor: Prof. György Dán

Co-Supervisor: Prof. Pontus Johnson

 

Thesis title: Machine Learning for Semi-Autonomous Cybersecurity Operations: From Anomaly Detection to Threat Response

Abstract:

     Intrusion detection systems (IDS) are indispensable for defending networked systems, yet security analysts are often overwhelmed by the large volume of false alerts they generate. Despite numerous attempts to reduce false positives, even highly accurate IDSs can trigger excessive alerts due to concept drift, such as changes in benign behaviors. This flood of alerts presents two critical challenges: the burden of investigating a large number of alerts and the uncertainty of situational awareness when choosing defensive actions. Since false alerts are inevitable, analysts need to: i) continuously update IDSs to address concept drift, ii) prioritize informative alerts to support efficient investigations, and iii) respond quickly to potential threats under uncertainty. Therefore, security analysts consume a significant amount of time on the three tasks, highlighting the necessity of reducing time consumption.

 

     The papers included in this thesis are organized into two parts. The first part introduces an anomaly-based intrusion detection system using a transformer. Each field in a log line is mapped to a token using both field-based and global vocabularies. To reduce irrelevant noise, fields unrelated to attacks are excluded from the anomaly score computation. The model is trained in a self-supervised manner by masking tokens and attempting to infer the masked fields. An anomaly score is then computed from the inference error, and alerts are raised when the score exceeds a predefined threshold. Our simulations demonstrate that the bidirectional transformer with field-based vocabulary outperforms baseline LSTM approaches. Then, we explore an intrusion detection framework that integrates flow-level anomaly detection, flow-context-based anomaly detection, and concept drift detection. A generative model maps flows in the networked system into an anomaly detection-friendly latent space and identifies anomalous behavior in individual log lines, forming the basis of flow-level anomaly detection. To capture system-wide context, a graph neural network correlates each log line with others across time and hosts, producing flow-context-based anomaly scores. Through anomaly masking, comparing predictions with and without suspicious log lines, the framework highlights significant differences when malicious activities are present. The flow-level and flow-context scores are then combined into a final anomaly score. In parallel, once concept drift is detected by a second generative model, two sample selection strategies are employed to identify log lines for investigation, enabling analysts to update the model efficiently and maintain detection accuracy over time.

 

     The second part of this thesis models the networked system using a Markov framework to study operations under false alerts. We first address the challenge of alert investigation for improved situational awareness by proposing two prioritization policies that rank alerts at each time step according to their informational value. To enhance practicality, we introduce methods that reduce computational complexity, with simulations demonstrating significant improvements in both situational awareness and efficiency. Building on this, we employ likelihood ratio testing to detect intrusions and prioritize alerts that confirm attacker presence. By pruning attack graph states into multiple hypotheses and investigating alerts to increase likelihood ratios, our approach achieves a faster mean time to detection. Finally, we examine automated incident response that combines low-cost and high-cost defensive measures, including blocking and shutdown, while preserving belief in a distributed manner. We propose a novel method for distributed belief maintenance that reduces communication overhead, and extensive simulations show that our solution achieves low total cost and high practicality.

 

     Overall, the deep learning–based anomaly detection and Markov model–based operational approaches developed in this thesis provide practical tools that enhance both time efficiency and cost effectiveness.

 

Papers Included in the Thesis

1. Paper A: S. Gökstorp, J. Nyberg, Y. Kim, P. Johnson and G. Dán, "Anomaly Detection in Security Logs using Sequence Modeling," IEEE Network Operations and Management Symposium (NOMS), Seoul, Korea, Republic of, 2024, pp. 1-9.

2. Paper B: Y. Kim, A. Mårtensson and G. Dán, "CANDID: Label-Efficient Adaptive Network Intrusion Detection under Concept Drift," submitted for publication.

3. Paper C: Y. Kim and G. Dán, "Dynamic Alert Prioritization for Real-time Situational Awareness: a Hidden Markov Model Framework with Active Learning," in IEEE Transactions on Dependable and Secure Computing, 2025.

4. Paper D: Y. Kim, G. Dán and Q. Zhu, "Human-in-the-loop Cyber Intrusion Detection using Active Learning," in IEEE Transactions on Information Forensics and Security, vol. 19, pp. 8658-8672, 2024.

5. Paper E: Y. Kim, G. Dán and Q. Zhu, "ADAPTD: Adaptive Detection and Proactive Threat Defense for Autonomous APT attacks", submitted for publication.

Papers Not Included in the Thesis

1. Y. Kim and G. Dán, "An Active Learning Approach to Dynamic Alert Prioritization for Real-time Situational Awareness," IEEE Conference on Communications and Network Security (CNS), Austin, TX, USA, 2022, pp. 154-162.

2. E. S. Escriche, J. Nyberg, Y. Kim and G. Dán, "Channel-Centric Spatio-Temporal Graph Networks for Network-based Intrusion Detection," IEEE Conference on Communications and Network Security (CNS), Taipei, Taiwan, 2024, pp. 1-9.

3. Y. Kim, Z. Sun and G. Dán, "VAS²PER: A Variational Graph Autoencoder Model for Structure and Semantics Preserving Attack Graph Compression," submitted for publication.