ls-judge: a tour of the engine behind every Run button

The isolation story

Every run — not every submission, every test case — executes in a brand new Firecracker microVM.

Not Docker. Not a chroot. Not a Linux user namespace. A full virtualisation boundary with its own kernel. A compromised payload cannot see the host, cannot see the previous submission's files, cannot even reach the next submission's VM. Outbound network is default-deny.

Why Firecracker specifically? Three reasons:

1. Boot time: a Firecracker microVM boots in under 100 ms from a pre-warmed snapshot. Docker containers are faster to start but share a kernel; traditional VMs are safer but slow. Firecracker sits exactly where we want to be — true VM, container speed.
2. Memory: a microVM for a Python runtime occupies ~5 MiB of host RAM beyond the guest allocation. That lets a modest host run hundreds of concurrent runs.
3. Blast radius: AWS Lambda runs on Firecracker for the same reasons. If it is good enough to isolate hostile code at AWS's scale, it is good enough for us.

The language zoo

Right now ls-judge ships 120+ runtimes across 65+ languages:

• Systems: C (clang, gcc), C++ (23, 20, 17, 14), Rust (stable, beta, nightly), Zig, D, Nim, Odin.
• Managed: Java (24, 21, 17, 11, 8), Kotlin, Scala, Clojure, Groovy, C# (.NET 9, 8, 7), F#, VB.NET.
• Dynamic: Python (3.13 down to 3.10), Node.js (22, 20), Deno, Bun, Ruby, PHP, Perl, Lua, Elixir, Erlang.
• Esoteric and interesting: Haskell, OCaml, Scheme, Common Lisp, Racket, Prolog, Idris, Agda, Julia, Raku, Crystal, Dart, Gleam.

Each runtime is a pre-built root filesystem image. Adding one is a Dockerfile and a build pipeline — not a code change.

Batching that respects a budget

Problems typically have a lot of test cases — twelve or fifteen for an easy, dozens for a medium. Submitting them one at a time would be chatty and slow. ls-judge accepts batches:

POST /batch/execute
{
  "submissions": [
    { "language": "cpp", "source": "...", "stdin": "..." },
    { "language": "cpp", "source": "...", "stdin": "..." }
  ],
  "timeLimitMs": 2000,
  "memoryMb": 256
}

The server groups consecutive jobs by runtime so the compile is shared across the batch, then dispatches them in parallel up to the host's capacity. Each test case gets a streamed result — the client sees case 1's verdict before case 50 even starts, which matters when case 3 is WA and you want to stop reading.

Verdicts, not just stdout

Every call returns one of a strict set of verdicts: AC (accepted), WA (wrong answer), TLE (time limit), MLE (memory limit), RE (runtime error), CE (compile error), IE (internal error). Alongside: stdout, stderr, timeMs.

Verdict discipline is the thing most hobby judges get wrong. Mixing TLE and RE, or treating OOM as RE, or passing a submission that printed trailing whitespace — all common, all wrong. Our checkers run per-case and support three modes: EXACT, TOKEN (whitespace-insensitive), and FLOAT (per-token tolerance for numerical answers).

What runs where

• Boot time is kept below 100 ms by maintaining a pool of pre-booted guests per language — the slowest step is usually copying the submission in, not spinning the VM up.
• The API is a Go service in front of the host pool, with submission metadata and per-case verdicts in Postgres so a reader can refetch a submission long after the host has moved on.
• Idempotency keys are mandatory on /execute and /batch/execute so a retry during a network blip does not double-charge quota or re-run a flaky test.
• Concurrency per host scales with CPU count — on a modest shared box you can keep several hundred microVMs warm; a dedicated host does more. Nothing here is tied to a specific cloud or SKU.

What it is not

• Not an auto-grader for proof-based maths. The judges we ship are stdin-stdout. If you want to grade a correctness proof, you need something else.
• Not a replacement for CI. Submissions are capped (default 2 s, 256 MiB) because the model is "one problem, one test batch", not "run your test suite".
• Not free of all limits. Anonymous callers get an aggressive rate limit; registered callers get a daily quota generous enough for anything but mass automation.

Try it

The public endpoint is at runtimes.laxmi.solutions. Grab a key, POST some code, read the verdicts. If you build something interesting on top of it, send it over.