Why we write our own C and C++ inference engines
cppdashboard.dev/r/2026/08/why-we-write-our-own-c-and-cpp-inference-enginesA 66 MiB binary instead of a 9.1 GiB virtualenv, depth estimation that beats PyTorch on CPU in half the memory, and biometrics that match insightface bit for bit. The method, the measurements, and what it costs us. Ettore Di Giacinto 24…
From the article
A 66 MiB binary instead of a 9.1 GiB virtualenv, depth estimation that beats PyTorch on CPU in half the memory, and biometrics that match insightface bit for bit. The method, the measurements, and what it costs us.
Ettore Di Giacinto 24 July 2026 7 min read
Most LocalAI backends wrap somebody else’s engine, and that is the right default. llama.cpp, vLLM, whisper.cpp, stable-diffusion, MLX and the rest are maintained by people who are better at those models than we are, and wrapping them costs a Dockerfile and a gRPC shim.
Share this resource