New Paradigms for Large-Scale Network Security based on Machine Learning

Traditional signature-based intrusion detection methods often struggle to detect the latest exploits and Advanced Persistent Threats (APTs), prompting a transition towards anomaly-based approaches. We address the following three challenges in large-scale environments for anomaly-based detection: (i) the comparability of models trained on disparate Deep Packet Inspection (DPI) libraries is limited, (ii) malware detection circumvention of adversaries relies on protocol-specific blind spots that evade feature characteristics of machine learning classifiers, and (iii) newly developed models require suitable solutions to operate in production environments with high-throughput.
This dissertation aims to bridge these gaps in large-scale network environments and explores new paradigms for network security leveraging machine learning methods. Our contributions are focusing on malware traffic detection in raw network traffic to streamline the comparability issues of DPI libraries, Domain Name System (DNS) inspection for tunnelling attacks and malware communication to address DNS covert channels in protocols and operate inspection at large-scale, and an experimental outlook on Large Language Models (LLMs) for network security.

Collectively, these contributions deliver deployable, DPI-independent, protocol-agnostic detection capabilities that advance the operational security of large-scale networks.