On-device AI is real but the tooling is still a mess
The Apple Silicon thread has an exec from Apple talking about Mac Mini AI demand and an on-device future driven by privacy and latency. The comments tell a different story: one commenter lays out the actual pain of running models locally. BF16, FP8, BF16+FP8, NVFP4, INT8, GGUF. The format zoo is 'non-obvious' is an understatement. Another thread shows someone getting GLM 5.2 running on a slow computer with swap-file tricks, which the community celebrates as 'the hacker spirit.'
The pattern: the hardware capability is ahead of the developer tooling. Apple Silicon can run meaningful models. The question is whether a normal developer can figure out how to actually do it without spending a week on format compatibility. The GLM thread suggests motivated hackers can, but it's not yet a smooth path.
This is the same tension playing out in the Hy3 thread, where a model is 'shockingly small for how capable it is.' Small capable models are the precondition for on-device AI working at scale. The models are arriving. The runtime ergonomics are lagging.
So what?
If you're building a product that could benefit from on-device inference (privacy, latency, offline capability), the hardware story is solid but the tooling story will eat your timeline. Budget real engineering time for format compatibility and runtime selection. Alternatively, watch the MLX and llama.cpp ecosystems, which are moving fast specifically on Apple Silicon ergonomics.