Welcome to the Nested Kernel.
Monolithic operating system designs undermine the security of computing systems by allowing single exploits anywhere in the kernel to enjoy full supervisor privileges. The nested kernel operating system architecture addresses this problem by “nesting” a small, isolated kernel within a traditional monolithic kernel. The “nested kernel” interposes on all updates to virtual memory translations to assert protections on physical memory, thus significantly reducing the trusted computing base for memory access control enforcement.
We incorporated the nested kernel architecture into FreeBSD on x86-64 hardware by write-protecting MMU translations and de- privileging the untrusted part of the kernel, thereby enabling the entire operating system, trusted and untrusted components alike, to operate at the highest hardware privilege level. Our implementation inherently enforces kernel code integrity while still allowing dynamically loaded kernel modules, thus defending against code injection attacks.
We also demonstrate, by introducing write-mediation and write-logging services, that the nested kernel architecture allows kernel developers to isolate memory in ways not possible in monolithic kernels. Performance of the nested kernel prototype shows modest overheads: < 1% average for Apache, <3.7% average for SSHD, and 2.7% average for kernel compile. Overall, our results and experience show that the nested kernel design can be retrofitted to existing monolithic kernels, providing important security benefits.
Links
Our full ASPLOS '15 paper can be found on here.
A link to the presentation given at ASPLOS '15 --- 20 minute version: Nested Kernel Presentation 20 Minutes
A link to the presentation given at FreeBSD Dev Summit '15 --- 50 minute version: Prezi Nested Kernel Presentation 45 Minutes
PDF: Nested Kernel Presentation 45 MinutesDetails on our prototype nested kernel, PerspicuOS, can be found here. PerspicuOS retrofits the nested kernel design into FreeBSD 9.0 for the AMD64 architecture.