A self-study program for system security research on operating systems, firmware, and devices: kernel internals, memory safety, exploit mitigations, SELinux, virtualization, Secure Boot, mobile isolation, and side-channels.
TEE topics are excluded. TrustZone, SGX, and related TEE design are assumed already known. This program covers everything else in the OS/device-protection stack.
This is kernel / device-protection work, not web-app security. After six months you should be able to:
- Reason about OS internals: virtual memory, scheduling, kernel/user isolation.
- Write and debug C / C++ / Rust at the syscall, kernel, and firmware boundary.
- Explain memory corruption (spatial and temporal) well enough to invent mitigations, not just recite CVE names.
- Know modern preventions: ASLR, NX/DEP, CFI, PAC, MTE, sandboxing, SELinux / MAC.
- Talk fluently about hardware roots of isolation: MMU, IOMMU, VT-x/AMD-V / ARMv8-A virtualization, Secure Boot.
- Prototype defenses for kernels, firmware, and mobile devices, then argue threat models like a researcher.
Modern phones and many servers are ARM64. x86_64 still matters for servers and for reading Intel/AMD virtualization docs. Both ISAs appear in this program; ARM64 is the default lab target.
| Month | Folder | Focus |
|---|---|---|
| 1 | 01-os-internals/ |
Kernel/user split, VM, scheduling, syscalls |
| 2 | 02-native-programming/ |
C, C++, Rust, debugging, tracing |
| 3 | 03-memory-safety/ |
Overflows, UAF, heap, kernel allocators |
| 4 | 04-mitigations-isolation/ |
ASLR/DEP/CFI, sandboxing, SELinux, LSM |
| 5 | 05-hardware-virtualization/ |
MMU, IOMMU, hypervisors, Secure Boot |
| 6 | 06-firmware-mobile-research/ |
Firmware, mobile OS, side-channels, research capstone |
Each monthly document contains theory, compilable labs, exercises, papers, books, and whiteboard prompts. Work them in order; later months assume earlier ones.
Assume 12–15 hours/week (evenings + one weekend block).
| Week pattern | Time | What to do |
|---|---|---|
| Theory pass | 3–4 h | Read the month’s theory + linked chapters/papers |
| Lab block | 5–6 h | Type, compile, break, and extend every code listing |
| Exercises | 3–4 h | Written + coding exercises; keep notes in notes/ |
| Paper / whiteboard | 1–2 h | One paper + 4–5 answers out loud |
Do not skip the “explain it on a whiteboard” exercises. The useful loop is architecture diagrams, not LeetCode.
Use a dedicated Linux VM (or a spare machine). Do not build unsigned kernel modules or disable mitigations on your daily driver.
Recommended baseline:
- Ubuntu 24.04 or Fedora 41, 8+ GB RAM, nested virtualization enabled
- Packages:
build-essential gdb lldb qemu-system-x86 qemu-system-arm gcc-aarch64-linux-gnu clang llvm rustc cargo binutils linux-headers-$(uname -r) linux-tools-common linux-tools-generic bpftrace strace ltrace perf pahole sparse libselinux1-dev policycoreutils setools selinux-utils - A second QEMU guest for kernel hacking (build your own kernel; never the host)
- Optional: Raspberry Pi 4/5 or an ARM64 QEMU virt machine for AArch64 labs
Kernel tree (clone once, reuse all six months):
git clone --depth=1 --branch v6.12 https://github.com/torvalds/linux.git ~/src/linuxKeep a running notes file:
mkdir -p notes
printf '# Lab log\n' > notes/lab-log.md| Topic | Months |
|---|---|
| OS internals: memory management, scheduling, kernel/user separation | 1, 5 |
| Native language: C, C++, or Rust | 2 |
| Low-level system programming and debugging | 2, 3 |
| Sandboxing, CFI, ASLR, DEP | 4 |
| Buffer overflows, use-after-free, heap exploitation concepts | 3 |
| Virtualization (Hypervisors, VT-x/AMD-V) | 5 |
| MMU, IOMMU, Secure Boot | 5 |
| TEE (TrustZone, SGX) | skipped |
| Memory corruption, privilege escalation | 3, 4, 6 |
| Side-channel attacks | 6 |
| Firmware exploitation (defensive / analysis) | 5, 6 |
| Isolation, access control, exploit resistance, threat detection | 4, 6 (SELinux is the MAC spine) |
| Mobile systems and devices | 6 |
| Research prototyping and papers | 6 + every month’s paper slot |
Read these across the six months; each monthly file points to specific chapters.
- Operating Systems: Three Easy Pieces — Remzi & Andrea Arpaci-Dusseau (free): https://pages.cs.wisc.edu/~remzi/OSTEP/
- The Linux Programming Interface — Michael Kerrisk
- Linux Kernel Development (3rd ed.) — Robert Love
- Computer Systems: A Programmer’s Perspective (3rd ed.) — Bryant & O’Hallaron
- ARM64 Architecture: A Programmer’s Overview / ARM ARM (DDI0487): https://developer.arm.com/documentation/ddi0487/latest/
- Intel 64 and IA-32 Architectures SDM: https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html
- The Rustonomicon: https://doc.rust-lang.org/nomicon/
- Rootkits and Bootkits — Matrosov, Rodionov, Bratus
- Linux Device Drivers (3rd ed., still useful for models): https://lwn.net/Kernel/LDD3/
- AArch64 Linux Kernel Hacking notes + LWN kernel index: https://lwn.net/Kernel/
- The SELinux Notebook (free): https://github.com/SELinuxProject/selinux-notebook
Rotate one of these if a month’s reading list is light:
- Abadi et al., Control-Flow Integrity (CCS 2005)
- Szekeres et al., SoK: Eternal War in Memory (IEEE S&P 2013)
- Lipp et al., Meltdown / Kocher et al., Spectre
- Konoth et al., ZebRAM or similar DRAM-isolation papers
- Project Zero write-ups on Android / kernel / hypervisor bugs: https://googleprojectzero.blogspot.com/
- Skim objectives and the week plan.
- Read theory sections with a notebook; redraw every diagram yourself.
- Compile every lab. If a listing is marked
HOST-SAFE, run it on the host. If markedVM-ONLY, run it in QEMU. - Complete exercises before moving to the next week.
- End of month: write a 1-page “what I can defend on a whiteboard” memo in
notes/month-0N-memo.md.
- TEEs, TrustZone, SGX, CCA / Realm management at the TEE-app layer
- Web, cloud IAM, and application pentesting
- CTF pwn challenge grinding as a primary method (concepts yes; scoreboard no)
- Weaponized exploit chains against real products
Memory-safety labs are intentionally vulnerable toy programs so you can talk about the bug class and the mitigation. They are not exploit development training.