I recently wrote a Bluesky thread about Sam Altman's takes on local compute.
Broadly, I agree with Sam (as much as this pains me). Compute benefits from gains to scale, demand pooling, and centralization.
However, I do not want to live in a world where most of the compute is owned by Anthropic, OpenAI, and Google. I would prefer a variety of collectives that serve the inference needs of their members. These collectives would aggregate demand, handle the annoyances of building an inference service, and provide quality open-weight models to their members.
I have been pondering what collective compute would look like, and wrote this:
Jarvin asked for some back of the envelope calculations on what it would cost for 100 people:
So, here's some analysis on inference collectives. I am assuming one Kimi K3.1 node, which requires 8x B300 GPU nodes. I am doing very rough/handwavy math to communicate a sense of scale.
How it would work
Broadly speaking, inference is really only cost effective to run on your own if you intend to do a significant amount of volume. Nodes have to be as close to maximal utilization as possible.
I would prefer any collective to offer unlimited tokens, with queuing and prioritization as key design components. Users should become accustomed to building asynchronous systems at scale, and allowing agents to "tick" whenever compute is available. Most agents do not currently work this way, but I think it would be straightforward to design an ATProto-native agent harness to permit large-scale agent systems that can utilize sporadic compute at scale.
If members use a lot of compute in a cycle, they will be de-prioritized if another member that has not used a lot of compute comes along. This helps maximize node utilization by supporting queuing as a default interaction method with the collective, and it encourages members to think more in terms of systems that accomplish tasks rather than agents that respond immediately to messages.
The goal is a system that can handle useful work at scale. CLI agents and 1:1 agent systems are nice, but they work best when operated by a large, monolithic company like Anthropic or OpenAI that can pool demand and revenue across many users. Collectives are smaller and may struggle with pooling.
Broadly speaking, I see two pathways to building such a system. One is a small service using spot GPUs as needed with intelligent queuing, the other is longer-term and requires significant capital to build and deploy an inference node using owned hardware.
Spot systems
The first is running a service that uses Modal or Baseten. I prefer Modal -- I know a lot of the folks there and think it is an incredible product.
An inference collective would build a system that would pool inference requests asynchronously and spin up a node when there is enough demand.
Baseten is also an excellent product but I think it's more expensive. I've been on the inference rental side with Baseten in my prior work and think they care a lot about customers + have fast nodes + good support.
Modal is great because it is designed for the kind of bursty/as needed cloud systems that this would require. Queuing can be easily built on top of Modal's existing primitives, and GPUs are generally affordable and easy to get.
Jarvin offered 100 people for the envelope math. For 100 people, it's quite expensive: $365 per person/month. This would however be a fixed cost that would partly replace your existing subscriptions to Anthropic/OpenAI/etc. However, there would be no hard token limit, and you would be encourage to enqueue as much work as possible.
100 people could get on average 33 million output tokens, which could be between $330 and $800 using the Kimi 3.1 Pro API (assuming 264m uncached input tokens). This isn't an obvious win over API pricing IMO, so we'd likely need more people.
At 500 people, we could offer something closer to $83 a month, though users would end up waiting longer and longer to get jobs run. If you were to run the nodes at max throughput and evenly distribute, that'd be about 6.5 million output tokens. If you were to go through Kimi's API directly, the same amount of output tokens would cost about $169 dollars, or $101 if you went through their batch API.
Buying compute
Buying nodes is expensive, but long-term seems like a good move. Buying a Kimi node is something like $500-700k, plus a bunch of headaches for power, maintenance, failures, etc.
Buying a node is a huge deal:
100 people: each person pays ~$5,500 upfront to fund the node, plus $157 a month. Gets 33 million output tokens assuming maximum throughput.
500 people: each person pays $1,129 upfront to pay the node, $31 a month for ongoing costs. About 6.5 million output tokens at max throughput.
I would tilt away from buying a node because it's a pain in the ass. Much easier to prove out the collective with spot compute before buying hardware.
Building a harness
It strikes me that a key component of offering spot collective inference would be designing a harness that can use it effectively. Most harnesses are not built with queuing in mind -- all inference is usually on-demand, and people expect immediacy. The economics of collective inference tilt towards slower, bursty workloads at high throughput.
A harness can help guide users to understand this economic force and help them build useful agents to encourage a particular style of AI system.
I have in mind something like Infinity, which is an experimental harness designed for extremely concurrent agent systems. A harness for collective inference might use ATProto to queue and store pending agent work at scale and asynchronously.
Existing work
Check out existing work in collective inference:
Thanks for reading.
-- Cameron