Polymorphic and Metamorphic Malware Detection
No Thumbnail Available
Files
Date
2009-05-16
Authors
Journal Title
Series/Report No.
Journal ISSN
Volume Title
Publisher
Abstract
Software attacks are a serious problem. Conventional anti-malware software expects malicious software, malware, to contain fixed and known code. Malware writers have devised methods of concealing or constantly changing their attacks to evade anti-malware software. Two important
recent techniques are polymorphism, which makes uses of
code encryption, and metamorphism, which uses a variety of
code obfuscation techniques. This dissertation presents three new
techniques for detection of these malware.
The first technique is to recognize polymorphic
malware that are encrypted and that self-decrypt before launching the
attacks in network traffic. We propose a new
approach that combines static analysis and instruction emulation
techniques to more accurately identify the starting location and
instructions of the decryption routine, which is characteristic of
such malware, even if self-modifying code is used. This method has
been implemented and tested on current polymorphic exploits,
including ones generated by state-of-the-art polymorphic engines.
All exploits have been detected (i.e., a 100% detection rate),
including those for which the decryption routine is dynamically
coded or self-modifying. The method has also been tested on benign
network traffic and Windows executables. The false positive rates are
approximately .0002% and .01% for these two categories, respectively.
Running time is approximately linear in the size of the network
payload being analyzed and is between 1 and 2 MB/s.
The second technique is a means of recognizing metamorphic
malware which has a transformed program image with equivalent or
updated functionalities. We propose a new approach that uses fully
automated static analysis of executables to summarize and compare
program semantics, based primarily on the pattern of library or
system functions which are called. This method has been prototyped
and evaluated using randomized benchmark programs, instances of
known malware program variants, and utility software available in
multiple releases. The results demonstrate three important
capabilities of the proposed method: (a) it does well at
identifying metamorphic variants of common malware. (b) it
distinguishes easily between programs that are not related and,
(c) it can identify and detect program variations, or code
reuse. Such variations can be due to the insertion of malware (such as
viruses) into the executable of a host program.
The third technique improves
the applicability of a semantic metamorphic malware detector
which is the second technique of this dissertation. We propose an
automated approach to generate common malware behavior patterns for
detection of metamorphic malware or new malware instances. This
method combines static analysis and data-mining techniques.
This method has been prototyped and evaluated on real world malicious bot software
and benign Windows programs. Through the experimental comparison with the
metamorphic malware detector, this method results in an about 80% reduction
in semantic pattern population to
detect known and new malware instances.
It is more robust to a junk behavior pollution attack than the
malware detector is. A set
of experiments was performed to test the quality of the common
behavior patterns which were generated with different parameter
configurations. Two optimized common behavior patterns were
obtained. The corresponding detection rates and true false positive
rates are 94%, 8.3%, and 78%, 0.32% respectively. According to a recent paper [1],
for indirect comparison and simple reference, the values of
the two detection rates which are 94% and 78% more than double the
detection rate of signature-based methods on unknown malware
programs, which is 33.75%.
Description
Keywords
emulation, semantic-aware, hierarchical clustering, static analysis, malware, metamorphism, polymorphism
Citation
Degree
PhD
Discipline
Computer Science
