Polymorphic and Metamorphic Malware Detection

Show full item record

Title: Polymorphic and Metamorphic Malware Detection
Author: Zhang, Qinghua
Advisors: S. Purushothaman Iyer, Committee Member
Peng Ning, Committee Member
Wenye Wang, Committee Member
Douglas S. Reeves, Committee Chair
Abstract: Software attacks are a serious problem. Conventional anti-malware software expects malicious software, malware, to contain fixed and known code. Malware writers have devised methods of concealing or constantly changing their attacks to evade anti-malware software. Two important recent techniques are polymorphism, which makes uses of code encryption, and metamorphism, which uses a variety of code obfuscation techniques. This dissertation presents three new techniques for detection of these malware. The first technique is to recognize polymorphic malware that are encrypted and that self-decrypt before launching the attacks in network traffic. We propose a new approach that combines static analysis and instruction emulation techniques to more accurately identify the starting location and instructions of the decryption routine, which is characteristic of such malware, even if self-modifying code is used. This method has been implemented and tested on current polymorphic exploits, including ones generated by state-of-the-art polymorphic engines. All exploits have been detected (i.e., a 100% detection rate), including those for which the decryption routine is dynamically coded or self-modifying. The method has also been tested on benign network traffic and Windows executables. The false positive rates are approximately .0002% and .01% for these two categories, respectively. Running time is approximately linear in the size of the network payload being analyzed and is between 1 and 2 MB/s. The second technique is a means of recognizing metamorphic malware which has a transformed program image with equivalent or updated functionalities. We propose a new approach that uses fully automated static analysis of executables to summarize and compare program semantics, based primarily on the pattern of library or system functions which are called. This method has been prototyped and evaluated using randomized benchmark programs, instances of known malware program variants, and utility software available in multiple releases. The results demonstrate three important capabilities of the proposed method: (a) it does well at identifying metamorphic variants of common malware. (b) it distinguishes easily between programs that are not related and, (c) it can identify and detect program variations, or code reuse. Such variations can be due to the insertion of malware (such as viruses) into the executable of a host program. The third technique improves the applicability of a semantic metamorphic malware detector which is the second technique of this dissertation. We propose an automated approach to generate common malware behavior patterns for detection of metamorphic malware or new malware instances. This method combines static analysis and data-mining techniques. This method has been prototyped and evaluated on real world malicious bot software and benign Windows programs. Through the experimental comparison with the metamorphic malware detector, this method results in an about 80% reduction in semantic pattern population to detect known and new malware instances. It is more robust to a junk behavior pollution attack than the malware detector is. A set of experiments was performed to test the quality of the common behavior patterns which were generated with different parameter configurations. Two optimized common behavior patterns were obtained. The corresponding detection rates and true false positive rates are 94%, 8.3%, and 78%, 0.32% respectively. According to a recent paper [1], for indirect comparison and simple reference, the values of the two detection rates which are 94% and 78% more than double the detection rate of signature-based methods on unknown malware programs, which is 33.75%.
Date: 2009-05-16
Degree: PhD
Discipline: Computer Science
URI: http://www.lib.ncsu.edu/resolver/1840.16/5484

Files in this item

Files Size Format View
etd.pdf 1.268Mb PDF View/Open

This item appears in the following Collection(s)

Show full item record