Writing your own inference engine is becoming a real strategy
A thread on why some teams write their own C and C++ inference engines got serious engagement. The concrete case: one commenter ported Model2Vec and NER inference from ONNX Runtime to plain C compiled for WASM, cutting binary size from 30MB to 300KB with a 1.5x speedup. The AirLLM thread on running 70B inference with a single 4GB GPU pointed at the same impulse: squeezing performance by owning the stack.
The pattern is that off-the-shelf inference runtimes are optimized for the median case, and teams building for edge, mobile, or cost-sensitive deployments are hitting the ceiling of what general-purpose tooling can offer. Writing your own is expensive upfront but pays off in deployments where binary size, memory footprint, or latency are hard constraints.
This is not a signal that everyone should write their own inference engine. It is a signal that the inference layer is not a solved commodity yet, and there is real differentiation available for teams willing to go deeper.
So what?
If you are deploying models on edge devices, in the browser, or in cost-sensitive cloud environments, the gap between a custom inference path and a general-purpose runtime is measurable and worth quantifying. For most teams it will not be worth the build cost, but for anyone with tight binary size or memory constraints, the WASM path in particular has concrete numbers behind it now.