OAC Core: Enhancing Network Security by Implementing an ML Malware Detection and Classification Scheme in P4 Programmable Data Planes and SmartNICs

University Of South Carolina At Columbia, Columbia SC

Investigators

Abstract

Malware attacks represent significant threats to organizations, which use a variety of approaches to protect against them. Examples include intrusion detection systems, intrusion prevention systems, and other security systems that run on general-purpose computers. Such schemes perform "deep packet inspection" (DPI), a process by which a security device protecting an organization thoroughly examines incoming traffic and alerts administrators about suspicious activities. While DPI may be effective in some scenarios, it requires significant processing. Furthermore, if the organization receives a high volume of traffic from the Internet, DPI may not keep up with the traffic and may only inspect a fraction of it. Additionally, the inspection may not be conducted in real-time and may only detect the malware after the attack. This project proposes to leverage the capability of P4 programmable data plane (PDP) switches and smartNICs to perform DPI. The project has four objectives. 1) Develop a malware detection and classification application running on PDPs, operating at line rate. The application will perform DPI of Domain Name System (DNS) packets, preventing malware from communicating with the corresponding C2 server. Traffic will be monitored in real-time, and functions commonly executed on general-purpose CPUs will be offloaded to PDPs. The plan includes analyzing DNS data, characterizing traffic patterns, and feeding such information to a machine learning (ML) algorithm. The ML algorithm will detect and classify malware according to their family (e.g., trojan, backdoor, ransomware). 2) Develop a malware detection and classification application running on a SmartNIC, for encrypted DNS packets. Research will be conducted to perform feature extraction for malware generating such packets. An ML algorithm will use the features to detect and classify the malware. 3) Develop a control application that shares threat intelligence and avoids malware propagation. As PDP switches and smartNIC detect malware, they share the threat intelligence with a centralized controller. 4) Expand the eX-IoT platform to fingerprint, store, and index newly detected and classified malware. The eX-IoT platform is a real-time platform for fingerprinting compromised devices on the Internet. The prototype running on PDPs and smart NICs will be built with open-source components available to the community, and the developed knowledge will be disseminated by synthesizing it as virtual lab libraries for courses and self-paced learning. The libraries will be distributed to colleges and universities. Finally, tutorials on malware detection and PDPs will be organized with organizations such as Internet2 and FABRIC. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.