Distributed AI Compute: Interesting Idea, Real Latency Problem
Two threads touched on distributed AI compute today. The Mesh LLM post proposed running LLM inference across distributed nodes using the iroh networking library. The Show HN for Ant, a new JavaScript runtime, raised questions about whether it could be used for distributed workloads.
The Mesh LLM thread had the most useful signal: 'the throughput over a network is incredibly slow. It's not usable for interactive use.' One commenter pointed to cocompute.ai as already doing this. The fundamental physics problem, network bandwidth between commodity nodes is orders of magnitude slower than the interconnects inside a GPU cluster, keeps showing up as the blocker for distributed inference schemes.
This is a pattern that keeps recurring in the HN discussions: the idea of democratizing AI compute by stitching together cheap hardware is appealing and keeps getting built, but the latency math doesn't work for anything a user is waiting on. Batch workloads are a different story.
So what?
Don't build a product that depends on distributed commodity inference being fast enough for interactive use. It isn't, and the physics aren't changing soon. If you want cheap inference for batch processing or async tasks, the distributed compute angle is worth watching. For anything user-facing, you need real GPU clusters.