I Dropped a Neural Net
A recent Dwarkesh Patel podcast with John Collison and Elon Musk featured an interesting puzzle from Jane Street: they trained a neural net, shuffled all 96 layers, and asked to put them back in order.
I was the first to solve it!
The puzzle
Oh no! I dropped an extremely valuable trading model and it fell apart into linear layers! I need to rebuild it before anyone notices, but I can’t remember how these pieces go together, or how it was trained. All I have left are the pieces of the model and some historical data. Can you help me figure out how to put it back together?
We’re given 97 weight matrices (the “pieces”) and 10,000 data points. The architecture is a 48-block ResNet where each block computes:
with
The problem decomposes into two sub-problems:
- Pairing. Each residual block has an input projection and an output projection. They’ve been separated. Match them back up:
possibilities. - Ordering. Arrange the 48 reassembled blocks in the correct sequence: another
possibilities.
The combined search space is
Pairing via diagonal dominance
I trained a small model with the same architecture on the same data and noticed something: the product
Why? If a residual block satisfies dynamic isometry (
Taking the trace and using the half-activation assumption (
This gives us a metric. For each candidate pair
Correct pairs:
Ordering
With all 48 blocks correctly paired, we need to put them in order.
Seed initialization
Sort blocks by their delta-norm, the mean magnitude of the residual contribution:
Residual perturbation magnitude tends to increase with depth in trained networks. Sorting by ascending
Surprisingly, a completely data-free proxy, sorting by
Bradley-Terry ranking
For all
Sorting by BT strength yields MSE = 0.00299, a 12x improvement over the seed.
Out of
Bubble repair
Finally, greedy adjacent-swap hill-climbing on MSE: sweep through all 47 adjacent pairs, swap if it reduces MSE, repeat until a full sweep produces zero swaps.
| Round | Swaps | MSE | Cumulative swaps |
|---|---|---|---|
| 0 (BT init) | - | 0.002989 | 0 |
| 1 | 21 | 0.001025 | 21 |
| 2 | 7 | 0.000553 | 28 |
| 3 | 5 | 0.000200 | 33 |
| 4 | 3 | 0.000052 | 36 |
| 5 | 1 | 0.000000 | 37 |
Five rounds, 37 swaps, exact solution.
You don’t even need Bradley-Terry
In practice, direct hill-climbing from the delta-norm seed converges to MSE = 0 on its own, no pairwise comparisons needed. It takes 13 rounds and 122 swaps instead of 5 rounds and 37, but the BT step requires 1,128 forward passes for the pairwise comparisons, so the end-to-end cost is comparable.
Even more surprising: the
Open questions
- What is this data? I spent time trying to figure out what the features represent and how they were processed. Very difficult, possibly synthetic. I’m curious to find out.
- Why is delta-norm such a good proxy for depth? Are there theoretical guarantees?
- Why does the worse initialization converge faster? Is the loss landscape more favorable from the
starting point?
Links
- Paper: arXiv:2602.19845
- Code: github.com/hynwprk/droppedaneuralnet