We run backend code we've never seen, from thousands of sites, on Kubernetes, and every one of those pods runs inside gVisor. Most of my day goes to it. This is the map I wish I'd had before I started sending fixes upstream.
Why a second kernel
A normal container is a Linux process with walls: namespaces decide what it can see, cgroups how much it can use, seccomp which system calls it may make. Every call it's allowed still lands in the host kernel, and every container on the node shares that kernel. One kernel bug is enough to escape.
A VM fixes that with a guest kernel and a hypervisor, and pays for it in boot time and memory. gVisor sits in the middle. A sandbox starts like a container, but the app's system calls never reach the host kernel.
The pieces
- Sentry: the core. A from-scratch reimplementation of the Linux system call interface in Go: memory management, filesystems, a network stack, signals, even the TCP connections. It runs as a normal user-space process, locked down with seccomp and namespaces of its own.
- Gofer: a separate, more trusted process. At startup it opens each mount's root on the host and hands those file descriptors to the Sentry.
- runsc: the OCI runtime that starts all of it. containerd runs it where it would run runc, and Kubernetes picks it per pod with a RuntimeClass. One pod, one sandbox.
+---------------------------+
| Stub processes |
| run the app's code |
+---------------------------+
| every syscall traps
v
+---------------------------+ +---------------------+
| Sentry | <--- | Gofer |
| Linux kernel in Go | FDs | one per container |
+---------------------------+ +---------------------+
| few, filtered calls | opens mount roots
v v
+-----------------------------------------------------------+
| Host kernel |
+-----------------------------------------------------------+
For the most common escape, a bug in the system call interface, an attacker now needs two bugs: one to take over the Sentry, then one in the much smaller set of host calls the Sentry may make. gVisor describes them as two kernels "which do not share any code".
The stubs, where I got it wrong
On cloud VMs gVisor catches system calls with a platform called systrap, the default since mid-2023. The app's code doesn't run inside the Sentry. It runs in stub processes, plain host processes whose system calls the kernel turns into a SIGSYS signal. A small handler in the stub saves the thread's registers into a shared slot called a context and wakes the Sentry. The Sentry handles the call in Go and puts the context back on a queue, where any idle stub thread can pick it up.
My first upstream fix was for one thread whose context never came back, and a pod that wouldn't die for days. I proposed killing the Sentry when that happens. The maintainer's review: "If there's one stuck subproc but the others are fine, we'd end up killing everything." gVisor already had a path that kills one stub and reports that thread as dead. What merged kills that one stub after 30 seconds.
The stub's memory
Every stub reserves one block at high addresses, shared with the Sentry: per-thread stacks, the context queue, a spinning queue where idle threads wait, and one slot per context. At startup systrap shifts most of that block by a random, page-aligned gap, so one sandbox's layout tells an attacker nothing about the next. Since a 2023 change the spinning queue stayed put while everything around it moved, so a small gap could land it on top of a thread's stack. The stub then spins forever. That one was a one-line fix.
What you pay
- Every system call is a trap, a trip to the Sentry and a trip back. Code that does many small reads and writes feels it.
- Not every syscall and flag is there. Check the compatibility list before you move a workload.
- Debugging moves. A hung app is now a hung Sentry, a stub process and a queue in shared memory, and
runsc debug --stacks only answers if the Sentry still does.
For code we've never seen, we take that trade. Happy to answer questions about running it in production.
Longer version with diagrams and every PR linked: https://www.catchkill9.dev/posts/intro-gvisor-a-kernel-in-go.html
