Skip to content
Breaking the Intelligence Curse

The open weight floor

Guarantee that capability within a fixed distance of the frontier is always publicly available, so no actor can hold a capability nobody else can reach.

Breaking power

63

Partial

readers 44 · 1 vote

  • Toothiness5
  • Feasibility4
  • Affordability2
  • Public acceptance3
  • Lab acceptance1
  • Independence4

What these mean

The chokepoint is the load bearing wall of the intelligence curse. Remove it and most of the rest of the structure has to be rebuilt from scratch.

The proposal is a floor rather than a ceiling. Rather than capping what the frontier may do, guarantee that capability within some fixed distance of it is always available to everybody, as open weights, permanently. If the frontier moves, the floor moves behind it.

Why a floor and not a cap

Caps fail for the same reason prohibition fails. They require the agreement of everyone capable of defecting, and defection is both profitable and hard to detect. A cap also has the perverse effect of entrenching whoever is at the cap when it is imposed.

A floor needs no such agreement. It requires one actor with enough compute to train and release. Nobody has to consent to someone else publishing weights, which is why this is one of very few interventions here that a single party can execute unilaterally.

Mechanism

Three components, in rough order of difficulty.

A publicly funded training programme with a standing mandate to release. Not a research institute that might publish, but an institution whose output is weights, on a schedule, with the release as the deliverable rather than a byproduct. The compute budget is the policy.

A distance target, stated in advance and in public. Something like: no more than twelve months behind the best measured capability on a fixed public benchmark. The target makes the commitment auditable, and auditable commitments are the only kind that change anyone's planning.

An inference subsidy, because weights that nobody can afford to run are a gesture. The floor is capability actually reachable, not capability nominally licensed.

Cost

This is the proposal's real weakness and it should not be soft pedalled. A frontier scale training run plus the compute to serve it is a serious line item, on the order of a national research facility rather than a grant programme, and it recurs annually because the frontier keeps moving.

Against that: it is small next to what states already spend on far less consequential infrastructure, and the expenditure is legible. You buy compute, you train, you release, and anyone can check whether the floor is where you said it would be.

The objection that actually lands

Releasing near frontier weights releases them to everyone, including the people you would least like to have them.

I do not think this objection can be dismissed and I do not think it is decisive either. The honest position is that this is a trade. You are accepting a wider distribution of misuse capability in exchange for breaking a concentration of power. Whether that trade is worth taking depends on which failure you think is more likely to be irreversible, and reasonable people land in different places.

What I would resist is the move where this objection is used to justify concentration while never being weighed against it. The concentration has costs too. They are just slower and less dramatic, so they are easier to leave out of the comparison.

What the public vote is for

I have this at the top of the scoreboard and I am aware that this is a minority position among people who work on AI safety. I would like to know how lonely it actually is.

Score it yourself

Only the averages are published.

Sign in to vote

Google confirms your email and nothing else. One ballot each, private, changeable.

Discussion

3 comments

Google confirms your email, nothing else.

Charlie Wyatt3h ago

hello!

Charlie Wyatt3h ago

nice

Charlie Wyatt3h ago

another