Open source local inference keeps gaining ground

The llama.cpp thread drew significant discussion around installation, GPU support across non-Nvidia hardware (Arc A770 in particular), and the friction of compiling with OpenVINO support. Separately, Nvidia released Nemotron 3.5 Lightning alongside NeMo Switchyard, an open-source routing library that intelligently directs requests to the most capable model for a given job. Mojo 1.0 also launched, with discussion about its licensing model: the language itself is proprietary but the standard library is Apache 2.

The pattern: the open-source local inference stack is maturing fast but still has real rough edges, especially for non-Nvidia hardware. The community is doing the work of smoothing those edges, but it is slow. Meanwhile, Nvidia is trying to establish itself as the orchestration layer for model routing with NeMo Switchyard, which is a smart move: if you own the routing logic, you stay relevant even as models commoditize.

Mojo's licensing is a live debate. The compiler being proprietary while the standard library is open creates an awkward dependency, and commenters questioned whether this matters less now that AI writes most of the code anyway.