Weight-space programming is a new idea from MIT CSAIL researchers: instead of training a neural network with gradient descent, you ask a frontier language model to read the network’s raw weights, work out what they do, and edit them by hand. In their tests, OpenAI’s GPT-6 Astra identified what small networks were trained on, repaired a broken control policy and wrote game-playing networks from scratch. Its biggest result was editing 0.016% of Qwen3.5-0.8B’s weights, which lifted the model’s MATH-500 score from 7.9% to 48.1% with no training at all.
Key points
- The work comes from Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL (Khattab created DSPy). It is a research blog post with full logs, not a peer-reviewed paper.
- Astra never wrote a training loop and never used gradients. It reasoned about what each part of a network computes and changed the numbers directly.
- On Qwen3.5-0.8B, Astra traced the model’s known thinking-loop problem to one direction written by two of its 24 layers and removed it from 131,072 weights.
- That single edit raised GSM8K from 13.8% to 67.1% and IFEval, an instruction-following test it was never targeted at, from 21.0% to 62.3%.
- The limits are real. Every network was small, only Astra was tested, and the method failed to improve Gemma 3 270M at math.
What is weight-space programming?
Every neural network is a large table of numbers called weights. Normally, nobody chooses those numbers. An optimizer nudges them millions of times, using gradients, until the network gets better at a task. The result works, but it is close to unreadable for humans.
The MIT team defines weight-space programming as a general-purpose LLM reading and writing a network’s weights directly, around a plain-language goal, without going through an optimizer. Sometimes the model gets no feedback from the environment at all. In practice, you “just ask” the model questions like “which game was this network trained on?” or “make this model better at math,” and it answers or edits by reasoning over the numbers.
The researchers ran OpenAI’s gpt-6-astra through the Codex CLI with every built-in tool switched off: no shell, no web search, no file editing and no sub-agents. In the experiments that allowed code, Astra could write one Python block per turn, which the team ran in an offline sandbox and returned the printed output. The team says all 1,125 logged turns were audited as tool-free.
What did GPT-6 Astra actually do?
The post covers eight experiments, grouped into reading, fixing and writing weights.
| Task | What Astra got | Result |
|---|---|---|
| Name the game a policy network learned | 17,253 raw weights, no tools | Correct on all four games, including Breakout at 99% confidence |
| Name the shape a small diffusion model draws | Weights only, no forward passes | Correctly identified circle, square and triangle models |
| Name the language a small language model learned | Middle-layer weights in a Python sandbox | Named the right language (Python, romaji or English) every time and found every induction head |
| Fix a car policy that never crosses a strip | One shot, no tools | Edited 3.5% of weights; success went from 0% to 100% over 1,024 episodes |
| Write MinAtar game networks from scratch | No environment feedback | Breakout score 517.4 vs 15.8 for PPO trained on 100M steps |
| Write a 9×9 Go network from scratch | About 1.5M weights, up to 50 forward passes | Beat a same-size PPO network trained on 20M games |
| Spot which of two small LMs is sycophantic | Weights, then an attention-pattern tool | Failed from weights alone; succeeded once it could inspect attention |
| Make Qwen3.5-0.8B better at math | Python kernel over the weights, practice problems, 10 to 50 forward passes | MATH-500 7.9% to 48.1% with 50 passes |
The MinAtar results were not a clean sweep. Astra’s hand-written Seaquest network scored 53.1, well above PPO at 100M steps (5.6) but below PPO at 500M steps (88.9). On Asterix it scored 53.2 against 25.8 and 55.2. Breakout was the standout, beating both PPO baselines by a wide margin.
How did the Qwen math fix work?
Qwen’s own model card warns that Qwen3.5-0.8B is prone to thinking loops. In the team’s evaluation, 93% of its MATH-500 answers ran into the 32,768-token limit, so the model rarely finished a solution.
Astra got a Python session with the weights loaded, but no GPU and no training. It could pick practice problems from the GSM8K and MATH training sets and spend a fixed budget of forward passes to test ideas. It never saw the test sets. Working from those probes, it found a single direction, written by the MLPs in layers 15 and 19, that pushed the model into repeating itself. It removed that direction, changing 131,072 weights.
| Measure | Original Qwen3.5-0.8B | After Astra’s edit |
|---|---|---|
| MATH-500 accuracy | 7.9% | 48.1% |
| GSM8K accuracy | 13.8% | 67.1% |
| IFEval (instruction following) | 21.0% | 62.3% |
| MATH-500 answers hitting the token limit | 93.0% | 33.1% |
The budget mattered. With 50 forward passes Astra found the fix. With 10 or 25, the gains were much smaller and the loops mostly stayed. The sampling settings were identical before and after, so the jump comes from the weight change, not from a different decoding setup.
Does this mean models can now teach themselves new skills?
Not from this evidence. The Qwen result is best read as removing an obstacle. The small model already had enough math ability to solve many problems, and it kept tripping over its own loops. Once the loop direction was gone, that ability showed up in the scores.
The follow-up on Gemma 3 270M makes this clearer. Gemma rarely loops, with only 8.5% of its answers hitting the token limit. Its mistakes were ordinary ones, like wrong arithmetic and invalid algebra steps, with no single component to switch off. Astra could not improve it, which suggests direct editing can unlock skills a network already has but cannot add skills that are missing. The authors themselves call the results “exciting but still limited,” and say context length and tooling hold the method back today.
It is also worth being clear about what was not shown. The team tested only one model doing the editing, every target network was small, and there is no claim that direct editing beats ordinary fine-tuning on cost, speed or reliability. VentureBeat’s coverage made the same point: the Qwen run still needed computation, practice problems and testing.
Why does weight-space programming matter for builders?
Most of today’s work on understanding models, known as interpretability, relies on humans and purpose-built tools to find circuits one by one. If a frontier model can read weights and point at the part that causes a failure, that work could get much faster. The authors highlight a few uses:
- Auditing without running a model. Spotting hidden traits, such as backdoors or sycophancy, by inspecting weights rather than only testing outputs.
- Targeted repairs. Fixing a known failure, like Qwen’s loops, without a full fine-tuning run.
- Adding or removing behaviors. Editing a capability in or out without retraining from scratch.
For teams shipping small open-weight models, the practical takeaway is narrower but useful. Some “the model is just bad at this” problems may be specific, fixable defects rather than missing ability. The edited weights and every prompt, code cell and evaluation are public, so you can inspect exactly what changed before trusting it.
There is a safety angle too. A model that can edit network weights could eventually edit its own. That is the kind of self-improvement loop safety researchers want to watch closely, and it is one reason the full, audited logs matter.
What should you watch next?
- Whether other teams reproduce the Qwen fix using the released logs and weight diffs.
- Whether other frontier models, such as Claude Opus 5.5 or Gemini 4, can do the same thing, or whether this is specific to GPT-6 Astra.
- Results on larger models, where context length becomes the main bottleneck.
- A peer-reviewed version with comparisons against standard fine-tuning and existing interpretability tools.
If you are tracking OpenAI’s model lineup, our GPT-6.1 Sol breakdown explains how Astra fits against OpenAI’s cheaper coding model.
