Improving the Security of the Heap through Inter-Process Protection and Intra-Process Temporal Protection

Show full item record

Title: Improving the Security of the Heap through Inter-Process Protection and Intra-Process Temporal Protection
Author: Kharbutli, Mazen Mahmoud
Advisors: Yan Solihin, Committee Chair
Milos Prvulovic, Committee Member
Eric Rotenberg, Committee Member
Gregory Byrd, Committee Member
Edward Gehringer, Committee Member
William Boettcher, Committee Member
Abstract: In most current implementations, memory allocations and deallocations are performed by user-level library code which keeps heap meta-data (heap structure information) and the application's heap data stored in an interleaved fashion in the same address space. Such implementations are inherently unsafe: they allow attackers to use application's vulnerabilities (e.g. lack of heap-based buffer overflow checking) to corrupt its heap meta-data in order to execute malicious code or cause denial-of-service. In this dissertation, we propose an approach where heap meta-data and heap data are protected separately. Our first solution exploits existing inter-process protection mechanisms through Heap Server, a separate process that maintains heap meta-data on behalf of the application and runs in parallel with it. To perform memory allocations and deallocations, the application sends requests to the Server, which responds to the requests and updates the meta-data. Since the heap meta-data is kept in the Heap Server's address space, attacks on the application can no longer corrupt it. Heap Server is directly implementable in current systems because it does not require new hardware. To optimize Heap Server's performance, we explore non-blocking communication, bulk deallocation, and pre-allocation optimizations. Evaluated on a real system, a fully-optimized Heap Server performs almost identical to a Base heap management library with no protection mechanisms. As an alternative solution, we propose a new User-level Temporal Intra-Process Protection (UTIPP) mechanism in which a process protects itself from its own vulnerabilities by write-protecting its own heap meta-data, and only removing the protection for legitimate stores in the heap management library. Unlike existing kernel-level page protection which can only be modified in the privileged mode, UTIPP allows a process to modify the new write-protection bit with a single instruction without disrupting normal pipeline flow. Evaluated on a cycle-accurate simulator, UTIPP adds negligible overhead in most benchmarks. Another contribution of this dissertation is a new heap layout obfuscation technique which relies on randomizing the space between heap chunks and the order of chunks in the heap, making heap data attacks more difficult. This obfuscation is integrated with Heap Server and UTIPP.
Date: 2005-12-07
Degree: PhD
Discipline: Computer Engineering

Files in this item

Files Size Format View
etd.pdf 3.409Mb PDF View/Open

This item appears in the following Collection(s)

Show full item record