Local LLMs Are Real but Slow and Rough Around the Edges
A thread on running Kimi K3 locally at 0.50 tok/s using 29 GB of RAM got comments focused on whether macOS Metal acceleration was being used, where to download the model weights, and whether the README itself was written by an LLM. That last point landed: multiple people noted that LLM-authored READMEs are now a recognizable genre, and it's not a flattering one.
The performance numbers here tell the real story. Half a token per second is too slow for most interactive use cases. The thread on AMD MI450 GPU kernel optimization ran in parallel, showing that serious inference performance still requires serious hardware and significant low-level engineering. The gap between 'runs on your laptop' and 'useful in production' is measured in orders of magnitude.
The question about Metal support on macOS is a real one. Apple Silicon can do meaningful local inference, but many open-source model runners don't fully utilize it, leaving performance on the table for a large chunk of developer hardware.
So what?
Local model deployment is not yet a shortcut to avoiding cloud inference costs for anything latency-sensitive. Founders evaluating 'run it yourself' options should benchmark on their actual hardware before making architecture decisions, and should treat any README that reads like it was auto-generated as a warning sign about the codebase.