TC-Small-Virtual Machine Introspection-based Live Forensics for Detection of Malicious Software

University Of New Orleans, New Orleans LA

Investigators

Abstract

Modern malware is used extensively in computer crime and cyber-warfare and poses a serious threat to the cyber-infrastructure of the United States, at the military, civil, and corporate levels. Malware can employ a number of techniques to gain access to needed resources and to prevent detection, including hooking or modifying system calls, adding new system calls, inserting new kernel modules, and directly patching kernel code. Furthermore, malware is increasingly stealthy, being both difficult to detect and to analyze, and current-generation schemes for detection, analysis, and mitigation will become increasingly ineffective as the trend toward additional stealth increases, with more esoteric infection vectors, complex packing schemes, polymorphism, and metamorphism being employed. This proposal leverages emerging live digital forensics techniques, to create powerful techniques for malware detection and mitigation. These live forensics techniques deeply analyze memory dumps and build accurate models of kernel and application structures that reflect the state of the machine at the time of an investigation. By integrating live forensics techniques into a virtual machine monitor (VMM) and developing hardware-supported introspection techniques to analyze system state, malware detection facilities can be created that prevent malware from interfering with detection and mitigation strategies. The proposal discusses a number of necessary tasks to support this research agenda, including the design of and development of a hardware-assisted VMM introspection architecture and deep, portable modeling of kernel data structures and other guest VM state, including the filesystem. These modeling techniques can then be used for real-time verification of critical kernel code, cross-verification of kernel structures, application state analysis, and protection of critical system files. A novel aspect of the proposed research is the use of commodity Graphics Processing Units (GPUs), protected by hardware directed-I/O virtualization, as malware detection accelerators. The intellectual merit of the proposed research is to increase the depth, flexibility, and capabilities of introspected live forensics analysis and to expand the scope of live forensics to the detection of sophisticated malware. The proposed techniques expand state-of-the-art in live forensics techniques, virtual machine introspection, and kernel-level malware detection and will provide a foundation on which to build even more powerful techniques. The broader impacts of the proposed work touch all sectors of society, since individual citizens, as well as the law enforcement, military, and corporate communities all benefit from the deployment of more sophisticated malware detection mechanisms. The proposed work also enhances the existing curriculum in information assurance at the University of New Orleans, since research results from this effort will be incorporated into both undergraduate and graduate courses, exposing students to an important area of study in which the supply of practitioners falls far short of the demand. For further information see the project web site at the URL http://www.cs.uno.edu/~golden/live-forensics.html.