Snapshots are images: how a warm box starts in 0.3 seconds

Containers made images cheap. kvbox does the same for whole VMs, by freezing a running box and starting new ones from it with every process still alive.

Oliver HartleyEngineer, kvbox
Oct 27, 20265 min read
warm

An agent that needs a sandbox doesn't want a blank machine. It wants the machine you already set up: toolchain installed, repo cloned, dev server listening. Rebuilding that on every run is where most of the time goes.

kvbox's answer is to treat a running box the way Docker treats an image. You get it right once, freeze it, and every new box after that starts from the frozen state instead of from scratch.

The cold-start tax

A fresh box boots its own kernel in seconds. That part is fast, and it is not the problem. The slow part is everything after: installing packages, warming caches, cloning the repo, starting services. For an agent that runs fifty short jobs a day, that setup is paid fifty times.

Containers solved this years ago with images. You build the filesystem once and every container starts from it. But an image is only a disk. The processes still have to start, the caches are still cold, and nothing is listening yet. A VM snapshot goes one step further: it is the disk and the memory and the CPU, frozen mid-sentence.

What a snapshot holds

A kvbox snapshot is three things saved together under one name: the box's memory, its CPU state, and its disk. Taking one pauses the box for about a second, then it keeps running as if nothing happened.

terminal
# freeze memory + disk; dev keeps running
$ kvbox snapshots new dev warm
✓ snapshot warm saved from box dev (412 MB on disk); start a warm clone with: kvbox new --from warm

A few hundred MB for a box with 4 GB of RAM and 8 GB of disk is not a typo. The snapshot only stores what the box actually touched: the memory pages the guest wrote and the disk blocks it changed. An idle dev box with a running server freezes small.

Snapshots of the same box are taken incrementally. Only what changed since the last one is read out of the VM, which is why the pause stays short, and the host merges them into standalone images afterwards. Deleting an older snapshot never breaks a newer one.

Mapping memory, not copying it

The fast path comes from not reading the snapshot up front. A new box maps its memory straight from the snapshot file. Pages load the first time the guest touches them, and each clone keeps its own private copy of anything it writes. The disk is a sparse copy, so a clone starts out owning only the blocks it changes.

So starting ten clones does not mean reading 412 MB ten times. They share the same file until they diverge, and most of them never touch most of it.

Set up once. Start warm forever.

A new name, the same running state

A clone is not a copy of dev. It comes up under its own name and IP, with tools installed and every process from the snapshot still running. Your dev server is already listening when the agent connects.

terminal
$ kvbox new --from warm agent-1
✓ box agent-1 is running at 172.30.0.3 from snapshot warm: 2 vCPU, 4 GB RAM, 8 GB disk
connect with: kvbox connect agent-1 (already up: everything that ran in the snapshot is running)

There is one catch, and it is where most of the engineering went. The guest itself still remembers being dev at the old address: its hostname, its network configuration, its idea of who it is. Nothing inside a frozen Linux knows it has been copied.

kvbox fixes that before the clone is reachable. A small service in the image, kvbox-agent, listens on vsock, a direct host-to-guest channel that needs no network. The host tells the woken guest its new name and address through it, then plugs the box's network in. From the outside, agent-1 was never anything else. The source box and its clones run side by side.

The warm pool

This is the pattern snapshots exist for. Boot one box. Install everything, clone the repo, start the services, run the test suite once so every cache is hot. Snapshot it.

From then on, every kvbox new --from warm is a ready environment in well under a second: one per test run, one per agent, one per preview. When the job is done the box is deleted and the snapshot is untouched. When the toolchain changes, you update one box and take one new snapshot.

The source box and the clones are ordinary boxes. kvbox ls shows them, kvbox connect reaches them, and a clone can be snapshotted again to make a template of a template.

Try it

Install kvbox, create a box, set it up the way your agents need it, and freeze it. From then on, every kvbox new --from starts warm.

terminal
$ kvbox new dev
$ kvbox connect dev
$ kvbox snapshots new dev warm
$ kvbox new --from warm agent-1

The snapshots guide in the docs has the full command reference, including kvbox snapshots ls and rm.

SnapshotsFirecrackerAgents
Oliver Hartley

Engineer at kvbox, working on microVMs, snapshots and the plumbing that makes a box start fast.

Install kvbox.
Run kvbox new.

$curl -fsSL https://kvbox.sh/install.sh | bash