I spent a couple of days on a Kaggle Playground Series competition (S6E7, predicting a three-class health_condition from tabular health data), mostly as an excuse to try Google Antigravity 2.0 with Gemini 3.1 Pro, as part of my Google Developer Expert activity. I wanted to see if an agentic IDE can take a competition from a cold start to a good leaderboard submission.

Google Antigravity 2.0 is Google’s agentic development platform, launched a couple of months ago. Unlike a normal IDE, it is built around AI agents that plan tasks, write and run code, and operate the editor, terminal, and browser on their own, then present the work as reviewable artifacts. The agent runs on Gemini 3.1 Pro, Google’s model for complex reasoning and coding.

I did not start from scratch. Before opening Antigravity, I used Claude Code to build a starter kit for tabular competitions based on my previous experience: a general playbook, two prompts (one to run the pipeline end-to-end, one to drive an improvement loop), and a set of agent prompts/commands. Then I added the competition data and ran the initial prompt in Antigravity. The full code is available on GitHub.

This post is about the experience of running a Kaggle competition in an agentic environment and about the importance of setting correct constraints and validation checks.

The starting direction

A multi-agent pipeline only works if all models agree on the rules. If different model scripts create their own cross-validation splits, they can’t be compared and their out-of-fold predictions can’t be blended. That’s why I defined the rules beforehand: a fixed fold split that every model reads, a pre-defined out-of-fold and test-prediction schema that every model writes, and a “memory” (EXPERIMENTS_LOG.md) that every run appends to.

My initial prompt included a standard approach to tabular competitions: EDA, feature engineering, three models trained in parallel (LightGBM, CatBoost, and a PyTorch MLP), a hill-climbing blend, and a submission to Kaggle. The most important rule was to spend considerable time setting up cross-validation and then using it for all subsequent runs. The second prompt was a loop that read the scoreboard, picked the highest-value ideas, implemented them, judged on CV, and logged the result. The loop repeated until the goal was met or ideas were exhausted.

Running the loop in Antigravity

I pasted the first prompt, and Gemini set up the folds, ran the EDA, built about 140 features (missing indicators, frequency encodings, numeric ratios, out-of-fold target encoding without leaks), and trained the three models. Antigravity runs long jobs as background tasks, so the three models trained concurrently, and Gemini could set a wake-up timer and check the results instead of monitoring a running process. Given the second prompt, the loop mostly ran itself: read the scoreboard, pick the highest-value ideas, implement them, evaluate on CV, log the result.

I used high reasoning to ensure the best quality, but it was at a cost. On the high reasoning setting, trivial edits took minutes and anything substantial took hours. Switching Gemini to low reasoning for implementation sped up the iterations significantly, and I kept high reasoning only for the few steps that actually needed it.

The wall

The first pipeline produced a CV of 0.967 accuracy. I submitted it and saw that something went wrong - the public leaderboard score was 0.874. A Kaggler who sees such a gap usually assumes the CV is wrong. However, the agent’s instinct was to get more data. It proposed downloading a Kaggle dataset of pseudo-labels and “known noisy IDs” for this competition, and it made the case with real confidence: this, it said, was “the only way to break past the 0.874 wall.”

The problem was that the dataset was just a user-uploaded file. Its pseudo-labels are one competitor’s model output and its “noisy ID” list is that model’s guesses, so training on them would have meant fitting our models to someone else’s solution, with no guarantee the labels were right. I told the agent it was unofficial and to stop. It stopped, reverted the change, and then did something more plausible: it dug up a public dataset the synthetic competition data had been inspired by and used that to augment instead.

That was the better hunch, but it was wrong again: the original competition stated that the second dataset was an inspiration, but the feature distributions were different. As expected, this external data didn’t improve CV and was dropped. Another attempt was confident-learning label cleaning - it was worse too, scoring 0.866 on the leaderboard, lower than the previous score.

The wall was two bugs

Both real causes were simple, and both were in the evaluation pipeline.

The first was the metric. The competition scored balanced accuracy, but the config and the EDA had been left on plain accuracy. With a 14.9x class imbalance (592561 at-risk rows against 39803 fit), plain accuracy rewards a model for predicting the majority class and ignoring the rest, which is what the models learned: a strong accuracy CV and a low balanced-accuracy leaderboard score. The fix was to go back to my original rules: use the competition metric and set up the validation based on it.

The second was subtler. An earlier label-cleaning experiment had written a file listing the rows to keep, and the data loader used it: load_train() dropped the noisy rows and reset the index, so the training data became smaller, while the precomputed fold indices still held the original row references. Two files didn’t match, so fold 0’s out-of-fold predictions were zeros.

I admit it was my mistake not to notice it immediately. By the way, Claude failed to diagnose this issue either (I showed it the code repository and asked it to analyze possible causes of the CV-LB gap), so it is a common trap for agents.

The fix and score improvement

Both problems were simple to fix once found. I ensured the training data was used correctly, used the appropriate evaluation metric, added class weights to the model training, and optimized the blend based on balanced accuracy. The blend scored 0.97498 out-of-fold and 0.94886 on the public leaderboard. The models trained noticeably slower with class weight, because reweighting the rare class changes the gradient and early-stopping iterations changed too.

This score is far below public notebooks that have already been tuned, but it is a good score for several experiments. And my goal was to make a submission with a reasonable score and iterate on it.

Where Gemini and Antigravity helped, and where they didn’t

Where it worked well:

• Running sub-agents. Gemini spawned the EDA, feature, model, and blender sub-agents, and they shared the context.
• Background tasks and scheduling. It ran the three models in parallel, and I could check the results later.
• Systematic iteration. Given the loop prompt, it assessed the scoreboard and logged every run without being reminded.

Where it struggled:

• Speed on high reasoning. Minutes for small tasks, hours for large ones; low reasoning was a good trade-off between quality and speed.
• Lack of logging or intermediate outputs. It was hard to see what it was doing in the middle of a long run, and I had to wait for the final result to know if it worked.
• Confidence at wrong times. It was ready to train on an unofficial dataset and call it “the only way” forward.

The bug that never existed

I first drafted a write-up with Gemini, and it added one funny hallucination. While explaining the fold-0 zeros, it blamed a folds < f typo in the training template, which was a plausible explanation that fits the symptom exactly. I almost believed it, as it convinced me that the bug was “pre-existing”. Only when I opened the template and showed Gemini folds != f it grepped through the project history and confirmed folds < f had never been there. The real cause was the row mismatch above. The agent invented a bug to fit a symptom I described, defended it when I pushed back, and I nearly believed it.

Conclusion

Antigravity and Gemini 3.1 Pro were fun to use, and they made starting and iterating on a competition much easier. But using them reinforced one of the oldest lessons on Kaggle: setting the correct cross-validation and using the correct metric is the foundation on which everything else is built. The agents have lowered the cost of participating in a competition, but they are not a substitute for understanding the problem and the data. The human in the loop is still essential for reviewing the code and ensuring that the agents are taking the right direction.

Over the last couple of months, I worked on two Kaggle competitions in the CayleyPy series: IHES Picture Cube and Megaminx. Both are combinatorial puzzle solvers, and they can be reformulated as a search in an enormous implicit Cayley graph: each puzzle state is a vertex, each possible move is an edge, and solving means finding a short path back to the identity. This is a long term-activity: we have been researching these puzzles for a couple of years and have published several papers. This year we took another step toward solving them.

I remember working on this in 2024. At that time, all code was written by hand, and I spent a lot of time debugging it and digging into documentation. But much has changed in these two years: nowadays, agentic workflows are a significant part of my workflow, and I can delegate much of the implementation work to an LLM, focusing on decisions rather than specifics (even though I still have to review them).

I used Opus inside Claude Code as the implementation agent for both competitions. Claude wrote essentially all of the training scripts, beam-search variants, TPU ports, submission automation, and post-processing tools. I still reviewed the code, checked outputs, and made the experiment-level decisions. I focused on guiding it and preventing it from going too far down the wrong path (which it did quite often): I told it which ideas to try, which areas to research, when to stop iterating on a dead-end model, when to pivot, etc.

So this post is about two things. One is about working with Cayley graphs: how to combine learned distance heuristics, wide beam search, symmetry, and path post-processing to solve puzzle graphs far too large to search with DFS/BFS. The second one is about agentic workflow: what it actually looks like to run a research loop when an agent handles implementation and the human role shifts toward experimental design, plateau detection, and search strategy.

I share the scores reached by different methods as a proxy of progress. The leaderboard shows a single number (total moves over a fixed set of scrambles), but it is the logic behind the solutions that matters. Score improvement indicates we were able to find shorter paths, so higher places on the leaderboard reflect a better understanding of the search problem.

What’s a Cayley graph, and why would you solve one?

Let’s start with an explanation of what a Cayley graph is and how we can work with it.

We have a puzzle (like a Rubik’s cube). Possible moves (generators) are a small set of operations (like turning a face of the cube). The puzzle has a solved state, and we can scramble it by applying a sequence of moves. The goal is to find a sequence of moves that returns the puzzle to the solved state.

In a Cayley graph, we have one vertex per element (every reachable configuration of the puzzle), and an edge between two vertices whenever a single generator takes one to the other.. The solved state is the identity vertex; a scramble is some other vertex. Solving the puzzle is finding a path from the scramble back to the identity, and solving it well (the score that we care about) is finding a short one. The shortest possible path is the scramble’s true distance-to-solved; the largest such distance over all scrambles is the graph’s diameter, known to cubers as “God’s number”.

By Cayley’s theorem, every finite group is a group of permutations, so permutation puzzles are a concrete handle on finite group theory in general.

The significance of Cayley graphs

A Cayley graph is one of the standard ways of representing a group as a geometric object: vertices are elements, edges are generators, and the graph metric becomes a distance measure in the group. This is why these competitions combine discrete optimization with geometric group theory. The basic questions are genuinely hard — nobody proved the 3×3×3 cube’s God’s number is 20 until 2010, and the value for megaminx is unknown.

Why a brute search can’t solve it

These graphs are far too big to write down. The 3×3×3 cube has about 4.3 × 10^19 states; the megaminx has roughly 10^68, astronomically more. In practice, you don’t even try to store the graph — you keep one state and generate its neighbors on the fly. The graph is defined implicitly, by its generators.

Breadth-first from solved is fine for a few moves — on megaminx. the numbers of states are 1, 24, 408, 6208, 90144, 1.28M, then ~18M states at depth six — but depth seven is ~250M, and a real scramble is dozens of moves deep.

So you can’t compute the true distance-to-solved; you estimate it with a learned heuristic and search against it. The whole problem boils down to two questions: how good is the estimate and how wide a beam can you afford. And one unfortunate consequence is that achieving the best results often means cramming the largest possible beam onto a TPU.

CayleyPy, IHES, and megaminx

CayleyPy is the umbrella for this line of work on finding short paths in Cayley graphs, using learned heuristics and beam search instead of the hand-built solvers that cubers traditionally use. It spun off a series of Kaggle competitions, each a different graph, implemented in the open-source CayleyPy library.

The main puzzle is the IHES Picture Cube: a 3×3×3 with every face distinct. This puzzle is more complex than Rubik’s Cube, as the orientation of the pieces matters, even the centers. There are 1003 scrambles to solve as short as possible. It is named after the Institut des Hautes Études Scientifiques outside Paris, the institute where Mikhail Gromov, the one who turned the study of groups into the geometry of their Cayley graphs, has spent his career.

Megaminx is not a cube but a dodecahedron with twelve pentagonal faces turning in fifths of a full rotation. My solver encodes a state as a length-120 vector with 24 generators (12 faces × 2 directions), and the competition provides 1001 scrambles.

Megaminx is an interesting challenge for two reasons. First, there is no strict algorithm to fall back on (unlike the smaller puzzles where I could use a Kociemba-style two-phase solver), so learned-heuristic search is basically the only viable approach. Second, because the total score is calculated over a thousand scrambles, even small improvements can provide a visible improvement.

I had modest computational resources: a 16 GB RTX 4090 laptop as my primary device, a GCP L4 machine for long jobs, free Kaggle GPU and TPU kernels for parallel exploration, and later I got access to a larger TPU kernel. I got free GCP credits thanks to being a Google Developer Expert, and they helped us run heavy jobs on GPUs and wide-beam search on TPUs.

The core approach

Both competitions follow the same approach, and the focus is on making it work better, faster, and more efficiently:

1. Train V(s), a neural estimate of distance-to-solved, on random walks from the solved state.
2. Run beam search, using V to rank children and keep the best B at each step.
3. Solve each puzzle in decorrelated ways: different beam directions, symmetries, and search configurations.
4. Post-process the completed paths to shorten them.
5. Min-merge every solution, keeping the shortest path per puzzle. Merging solutions created by different people is often a key to significantly improving the score.

This approach is straightforward, but each stage can be complex and computationally expensive to run. Here is what the Claude role was in each stage of the search loop:

The rest of the post is about my approaches to improving the solutions for both competitions.

The scaffolding

Over the course of the two competitions, I built a scaffolding that let me run the research loop with minimal intervention. I needed a solution that could survive context resets when starting new sessions, track long-running jobs, and keep work state in a single place.

There were three main parts:

• Markdown state files. IDEAS_CATALOG.md, EXPERIMENTS.md, HANDOFF.md. Claude Code session lose context when the sessions is too long or when you end it, so anything important was written down into one of these files.
• Slash commands for repeated tasks. At first, I created commands to quickly upload submissions to Kaggle; then added commands to check job status, merge submissions, and more.
• Background tasks and Monitor for long jobs. I often had to launch many hour-long runs (sometimes more than 24h), and needed to monitor them to make sure they don’t fail or get stuck - Claude was able to manage it.

One of my favorite parts was giving a command like “run this training script on GCP GPU and report the results”; Claude then could spin up a VM, run the job, and show the results. It saved me a lot of time.

I have published the clean code and agent settings here - my working repo is private as it is a mess.

IHES — creating and polishing the core solution

In the IHES Picture Cube, a state is a permutation of 72 stickers (8 corners × 3 + 12 edges × 2 + 6 centers × 4), with 18 generators structured as 3 axis families × 3 layer indices × 2 signs. An adapted Kociemba two-phase solver (a classical computer algorithm for solving a 3x3x3 cube) gave 42718 moves. For a comparison: on the 3×3×3, God’s number is 20, so an “optimal on every puzzle” solver would score ~20k.

From baseline to a small architecture

The Claude library provides a small MLP distance heuristic and a beam search. Claude analyzed the baseline architecture (a sequential MLP) and rewrote it, adding residual blocks, an embedding encoder, a faster training recipe (bf16, torch.compile, fused AdamW, larger batches, 3× throughput), a wider beam, and BFS-d5 post-processing. This improved the score to 27106.

And at this point, I had already noticed the limitations of using agents: the MSE loss wasn’t improving, and after several attempts, the agent started training variations of the same model and ensembling them (just like a real Kaggler aiming for fast score improvement!).

I had to explicitly tell Claude to stop iterating on the same architecture and go after a better single model. I asked it to do deep research on specific topics, and we found multiple relevant works. The next best idea turned out to be not a model improvement but a beam-search improvement. At that point, I was running a beam search with a beam width of 8K, and the GPU memory was already fully used. Claude implemented multiple optimizations (priority-queue-free top-K via torch.topk, contiguous beam buffers reused across steps, and others), which allowed to increase the beam width to 65K, which improved the score by 2k point. This proved that beam search width often plays a more important role than the model architecture.

Bellman, NISS, and the search improvements

As mentioned before, models are usually trained on random walks starting from the solved state, and the labels are the walk depth. But these labels are merely an upper bound on true distance, because the random walk that produced a label might not be optimal. Bellman self-bootstrap (target(s) = 1 + min_a target_net(apply(s, a))) reduces that bias. Adding it helped solve several very hard puzzles.

NISS was another noticeable improvement. I asked Claude to mine the speedcubing forums, and it encountered NISS - Normal-Inverse Scramble Switch. Given a scramble, you solve the inverse problem, then reverse the resulting path and invert each move. The expected length is the same, but we have different hash seeds and tie-breaking, so the search is directionally decorrelated. It was simple to implement, and together with Bellman, the score reached 24068.

This is an old idea in the cubing community, but I didn’t even know I could search for it.

The next improvements were incremental: running wider beam search for the hardest puzzles, optimizing beam search (int8 state encoding) to run at 2M width, and Q-function distillation (from Vlad Kuznetsov’s writeup) improved the score to 23322.

Q-function distillation is a very interesting trick: you train a network with 18 outputs, the i-th predicting V(apply(s, action_i)); one forward pass then gives all 18 neighbor scores (no need to run 18 passes), and children are generated only for the top-B among the 18·B candidates. That was an ~8× speedup at beam 524K, and it cuts beam memory as well as time.

The final solution was a combination of all the above and took ~9 hours to run training and 4M beam search on a GCP L4. I min-merged it with my previous best submission and got 23224 moves, about 1400 moves behind Tomas Rokicki.

Min-merging my solution with the best public one gave me 21972, but I don’t count it as my own - this is a team effort.

What didn’t work

I have tried a lot of ideas, and many didn’t work; for example, training larger models produced worse results.

• A piece-decomposition input encoding trained worse than the flat 72-sticker embedding and ran 2.5× slower at inference.
• 24× rotational symmetry augmentation converged like the baseline and didn’t beat the previous record.
• A commutator-library window-replacement pass had zero hits, because beam paths aren’t commutator-structured the way hand-written FMC solutions are.
• BFS-d6 window replacement gave nothing over d5.
• A twips/twsearch integration would have been days of work for uncertain payoff.

The result: model improvements over the small-arch + Bellman + n_back=1 didn’t help. Most of the improvements came from improving the beam search.

Megaminx — reusing and improving the previous approach

In Megaminx, we have 12 faces, ~120 stickers, 24 generators (12 face rotations × two directions), and the same metric (total move count) across 1001 test puzzles. I reused most of the code and soon reached 82481. As a comparison, Tomas Rokicki’s score was 93606. This shows that while in smaller puzzles heuristics can be better than search, in larger puzzles search works better.

The first plateau

After the first success, I soon realized that I had reached the plateau. Claude tried 13 variations of Bellman training, but all of them had similar loss. The agent was happy to continue iterations, so I had to stop it and started researching.

Sym-ensemble: 360 rotations as a single model

The first breakthrough idea was symmetry-aware ensembling. The megaminx has 360 rotational symmetries (the A₅ × C₆ group acting on the dodecahedron). For a given scramble σ, every rotation g maps it to a rotated scramble g·σ that has the same optimal length; solving each rotated version and inverting back gives many valid paths per puzzle, of which we keep the shortest.

Using all 360 would be too computationally expensive. But even using just four of them was enough to push the average path length from ~89 to 88.20 — a nice improvement without retraining the model.

TPU for a wider beam

As it was mentioned before, increasing the beam width was the most productive lever in IHES. The same was true for megaminx, but I wasn’t able to go past 2-4M on GPU. TPU kernels have much more memory, so I decided to use them and asked Claude to port beam search to TPU.

It took several attempts to get right, but eventually we had a working TPU beam search on Kaggle’s free v5e-8. But it had only 1m width, which was even smaller than the 2-4M I had on GPU.

To make it better, I had Claude shard the beam across all 8 cores of one TPU — shared-beam SPMD, hash-partitioned so that per-step memory is bounded by the local shard rather than the global beam, which lets one beam grow far larger than any single core could hold. And here I hit a hard wall with Claude  - it couldn’t implement it correctly after 5 attempts using PyTorch-XLA and 5 attempts using JAX. I asked Codex to review the attempts and suggest fixes, and it produced a working version on its first attempt! This shows that sometimes an agent just doesn’t have sufficient capacity to implement a complex idea, and another agent can do it better.

The problem was in a cross-rank reduce running inside the per-step JIT’d body; moving it onto the host resolved the issue. After that, Codex worked on a few rounds of memory optimization and pushed the shared beam from 8M to 48M states on Kaggle’s free TPU v5e-8, and later to 256M on a bigger v6e-8 TPU on GCP. I will share more details in a separate blogpost.

AlphaZero without self-play: breaking the model ceiling

The previous improvements (sym-ensemble, wider TPU beam with shared-beam SPMD) improved the search, but I was stuck on the model side. I have tried many variations of the model, but the final results (the lengths of the found paths) were roughly the same - ~89 on average on a small 51-puzzle validation set. This led me to conduct in-depth research into alternative directions. Multiple approaches didn’t work (GFlowNets, dataset distillation, admissibility-aware losses), but one did help me improve the score - AlphaZero.

I have used only the architecture part (no MCTS or self-play): a shared trunk with two heads (a 24-way policy head and a scalar value head) trained on a joint CE(policy) + MSE(value) loss. The policy learned by imitating strong solution paths; the value followed the Bellman recipe.

It took several attempts to make it work: I had to retrain the whole pipeline from scratch and mix different types of labels. One more important thing: training longer usually hurt the value head, so I had to stop early. The epoch-24 checkpoint decreased the mean path length down to 87.5.

The interesting thing was that while the policy head was useful for training, it was not useful at inference. It helped produce a better value function, but for inference, I used the value head itself directly. The full five-stage training pipeline is reproducible in a public Kaggle notebook.

Simulated annealing on completed paths

Another new thing useful for megaminx was post-processing. I took the completed beam path and tried to hill-climb on it. Claude built a simulated-annealing pass with three operators:

• commuting_swap: swap adjacent moves that commute (act on disjoint stickers), which occasionally finds a shorter local segment.
• tail_resolve: re-run beam search on the last K moves with a wider beam.
• macro_insert: try inserting known short macros (commutators, fixed-period sequences) at every position.

It took ~36 hours on a single L4 to run the SA pass on the top-200 longest paths. 122 of them were improved for 471 moves in total.

The next steps were about pushing post-processing further, adding small tricks and min-merging solutions together with other competition participants.

What working with Claude looks like

As I have mentioned before, I usually set direction and let Claude implement everything.

Where Claude worked well

• Auto-research for model improvement and ablations on possible issues. Claude was able to suggest improvements to the model, run them, and do the next steps. When it implemented multiple changes at once and the results were worse, it ran ablations to isolate the issue.
• Code velocity. Creating working code took much less time than writing it by hand.
• Paper or idea implementation. It was easy to tell Claude to implement a specific paper or idea; it usually did so relatively well.
• Training on different hardware. I trained locally on a 4090, on Kaggle, and on a GCP GPU - Claude easily managed it.
• Documentation. I asked Claude to document every single experiment, decision, and idea. It helped reproduce previous solutions and avoid getting lost in the sheer volume of tried ideas.

Where Claude failed

• Endless iterations without significant improvements. The agent was happy to try dozens of small ideas that didn’t improve the results or to train many models for blending.
• Using non-optimal hardware for wrong reasons. Claude sometimes launched long processes on my local laptop that could take 1-2 days rather than using GCP or Kaggle.
• Being unable to find novel ideas. Most of the significant improvements came from my nudging, not from it itself.

Conclusion

This was my first time using agents for working on a long-term research project. It took me ~4–6 weeks of iteration to implement everything described in this post, and the main time sink was waiting for model training and beam search. Without using Claude, it would have taken me many months to implement everything, and I would have spent a lot of time on debugging and implementation details.

I was surprised how much of the implementation work could be delegated to an agent, and how much time it saved me. I was also surprised how much of the research work still required human intuition and creativity. The agents can’t handle ambiguity well and often prefer to follow safer ideas (despite being asked to be “bold”). But I’m sure the agents will become better in the future. The main benefit for me was that the cost of trying an idea has dropped enough that the bottleneck is now the ideas themselves. But that works only if you keep the agent on a tight leash and don’t let it pursue wrong ideas or avoid implementing complex ideas.

Working with Cayley graphs is a fascinating area of research, and I hope this post inspires others to explore it. If you are interested in this project, you are welcome to participate in the Kaggle competitions IHES Picture Cube and Megaminx. These competitions are useful because they turn abstract group searches into an engineering benchmark. Every improvement shows up as shorter paths and a better leaderboard score.

Amazon

Packt

Niels Cautaerts on Medium

Hossein Ghorbanfekr on GitHub

I was offered the opportunity to read GPU-Accelerated Computing with Python 3 and CUDA by Niels Cautaerts and Hossein Ghorbanfekr in exchange for an honest review. As a Google Developer Expert in AI/PyTorch, I work with PyTorch a lot, so a deep book on GPU computing in Python was in my area of interest. The book teaches Python developers and scientists to write GPU-accelerated code without dropping into C or C++, and it does this by moving up an abstraction ladder: you start by writing raw CUDA kernels with Numba-CUDA, then switch to high-level libraries like CuPy, RAPIDS, and JAX, and finally build four full applications end-to-end. I often use GPUs, but I use PyTorch/JAX without thinking about the kernels, the memory hierarchy, and the profiler. This book is about the layer underneath that, and I liked how consistently it pushes you to measure what is actually happening on the device rather than guess.

The overall structure

flowchart TB
    P1["Parts 1-2: Write your own kernels<br/>Numba-CUDA: grid/block/thread, occupancy,<br/>coalescing, shared memory, streams, multi-GPU"]
    P3["Part 3: Drop-in high-level libraries<br/>CuPy = NumPy/SciPy<br/>RAPIDS cuDF/cuML = pandas/scikit-learn<br/>JAX = jit/grad/vmap"]
    P4["Part 4: Real-world applications<br/>heat equation, computer vision,<br/>molecular dynamics, transformer LM"]
    P5["Part 5: Beyond the book<br/>Tensor Cores, nvmath, ROCm, OpenCL, Vulkan"]
    PROF["Measure, don't guess:<br/>Nsight profiling runs through every chapter"]
    P1 --> P3 --> P4 --> P5
    PROF -.-> P1
    PROF -.-> P3
    PROF -.-> P4

The book is organized into five parts:

• Part 1 (Chapters 1–4) covers the fundamentals: why GPUs are fast, how to set up an environment locally or in the cloud, how to write your first Numba-CUDA kernels, and how to profile and debug them with the Nsight tools.
• Part 2 (Chapters 5-7) is about performance: the GPU execution, occupancy, coalesced memory access, shared memory, overlapping work with CUDA streams, and scaling across multiple GPUs with Dask and JAX.
• Part 3 (Chapters 8–10) moves on to high-level libraries: CuPy as an alternative to NumPy and SciPy; RAPIDS cuDF and cuML for pandas and scikit-learn; and JAX for optimization problems.
• Part 4 (Chapters 11–14) brings it together in four applications: solving the heat equation, image processing and computer vision, a molecular dynamics simulation, and a transformer language model built from scratch.
• Part 5 (Chapter 15) is an overview of where to go next, from Tensor Cores and other CUDA Python libraries to other platforms like ROCm and OpenCL.

Every chapter follows the same structure: derive the math or the concept, write a CPU version, port it to the GPU, then profile and optimize, and see a set of questions and worked answers. The environment is set up using the Pixi package manager, and you need an NVIDIA GPU (Chapter 2 shows how to rent if you do not have one).

What I liked

There were many things I liked in this book, and I want to highlight several in particular:

• Nearly every optimization is measured, and there are cases when something doesn’t work: manual loop unrolling that the compiler had already done or a shared-memory version of the heat-equation kernel that barely contributes.
• The sections about high-level libraries include “when to use” guidance. You can accelerate an existing pandas codebase with cudf.pandas and a single %load_ext, but the book shows when the host-to-device transfer outweighs the speedup and the GPU is not worth using.
• Every chapter ends with a set of questions and worked answers, and the running examples are real scientific problems.

The CUDA streams chapter was interesting. Overlapping data transfer with computation reduces a 600 ms image pipeline to 285 ms, and we can see the actual gain in this illustration.

The computer vision case compares three ways of classifying noisy objects in an image: seven Hu moment shape descriptors, template matching in log-polar space, and a small CNN. The Hu moments reach 66.7% accuracy in milliseconds. Template matching gets 80% but takes over a minute per run. The CNN reaches 88% on clean held-out Fashion-MNIST, then drops to 60% on the noisy extracted objects, which the authors call “somewhat disappointing”.

The JAX chapter implements a physics-informed neural network, a great demonstration of why automatic differentiation matters. A plain MLP fits the training window for a damped RLC-circuit oscillation but fails to extrapolate past it; incorporating the circuit’s differential equation into the loss lets the network predict the new region.

The final project is about building a small transformer language model with JAX, Flax, and Optax: scaled dot-product attention, multi-head attention, the causal mask, the decoder stack, and a top-k sampling head. In the past, I have built transformers from scratch and by loading them from Hugging Face, and I’m convinced it is very useful to know the implementation details.

What could have been better

There are a couple of small things that could have been handled differently:

• The transformer chapter uses IMDb (a sentiment-classification dataset), which is a bit of a weird choice for demonstrating LLM capabilities. The model output is barely coherent, but that is to be expected given such a short training.
• The book is closely tied to specific library versions, and some implementation details may become outdated. But it isn’t a problem for the core concepts, and the authors have a GitHub repository with the code and a “known issues” page.

But these are small nitpicks that are completely overshadowed by the good sides of the book.

Conclusion

This book is a good fit for Python developers, scientists, and data scientists who want to understand GPU performance instead of treating the GPU as a black box behind a framework. It is especially useful if you do scientific or numerical computing and occasionally need more control than PyTorch or a high-level library gives you.

GPU tooling moves fast, and the specifics here will change. What lasts is the way of working the book teaches: profile before you optimize, know where your data lives in the memory hierarchy, and understand when the GPU actually earns its place.

Paper

Project

Demo

Modern detection-and-grounding VLMs treat a bounding box as text: each box becomes a short string of coordinate tokens, decoded one at a time, left to right. This means a model predicts box coordinates one token at a time, despite all coordinates belonging to the same geometric object. The approach inherits the limitations of language modeling rather than exploiting the structure of spatial prediction. The usual fix for the latency half is multi-token prediction (MTP): emit several tokens per step and accept some accuracy loss for throughput. LocateAnything introduces Parallel Box Decoding (PBD), which predicts an entire bounding box as a single atomic unit. This simultaneously improves localization quality and decoding speed.

The approach

LocateAnything-3B is a native-resolution VLM built from a Moon-ViT vision encoder and a Qwen2.5 language decoder joined by a two-layer MLP projector. LocateAnything uses a block-based representation for bounding boxes. Coordinates are quantized and grouped into fixed-length blocks that become the fundamental prediction unit.

Parallel Box Decoding

The central idea is Parallel Box Decoding (PBD): reformulate localization output as a sequence of fixed-length, box-aligned blocks rather than a flat coordinate-token stream. Every block has the same length (six positions, four quantized coordinates in a 0-to-1000 grid plus two structural markers). There are four block types: a Semantic block carrying the queried label, a Box block carrying the coordinates, a Negative block when the object is absent, and an End block to stop.

Because a box now lives inside one constant-length block, the decoder can output that whole block in a single step. Within a block, the four coordinate positions attend to each other bidirectionally, so the top edge is predicted with knowledge of the left edge rather than being blind to it; across blocks, attention is block-causal. That makes the scheme semi-autoregressive rather than fully parallel: the tokens inside a block are predicted together, but blocks are still emitted one after another, so a busy image costs roughly one step per box rather than a flat two.

Dual NTP and MTP training

Training only with parallel box prediction would risk losing the causal reasoning behavior inherited from autoregressive language models. LocateAnything therefore trains on two representations of the same target simultaneously: a standard next-token prediction (NTP) sequence and a block-level masked-token prediction (MTP) sequence. The NTP stream preserves the model’s original language modeling capabilities, while the MTP stream teaches it to predict an entire box block in a single step. If the block size is reduced to one token, the formulation naturally becomes standard NTP.

To make this work, the authors introduce a block-causal attention mask. The NTP stream uses standard causal attention and cannot access the MTP stream. The MTP stream can attend to previously generated blocks but not future ones, while tokens within the same block use bidirectional attention. This allows the model to jointly reason about all coordinates of a bounding box while maintaining dependencies between different box predictions.

The final objective combines the losses from both streams, enabling the model to retain autoregressive reasoning while learning the structured predictions required for Parallel Box Decoding.

The authors use a large data engine, LocateAnything-Data: 12M images, 138M natural-language queries, and 785M boxes across six task families. General object detection dominates the supervision (about two-thirds of queries and over 80% of boxes), with GUI grounding, referring comprehension, OCR, layout grounding, and point localization filling out the rest. Training runs a world-knowledge alignment pass with detection data held out, then two SFT stages: the first mixes in the full query set, the second cuts general data to a fifth and over-samples dense-scene datasets like MOT20Det and SKU110K to sharpen crowded detection.

Three inference modes and the hybrid fallback

Parallel Box Decoding is fast, but the authors identify two situations where it can fail. The first is format irregularity: in complex scenes with many object categories, the model can become confused when transitioning between categories and generate malformed box syntax. The second is spatial ambiguity: when objects are densely packed in regular patterns, parallel decoding can predict coordinates that fall between two neighboring objects rather than on either object itself.

To address this, LocateAnything introduces a hybrid inference strategy that combines the speed of MTP with the reliability of traditional autoregressive decoding. During generation, the model continuously checks for malformed outputs and low-confidence coordinate predictions. When a problematic block is detected, generation rolls back to the last valid prefix and temporarily switches to standard next-token prediction (NTP) for that block. Once the difficult region has been resolved, decoding returns to the faster MTP mode.

This leads to three inference modes. Slow Mode uses pure autoregressive decoding. Fast Mode uses Parallel Box Decoding throughout. Hybrid Mode combines both approaches, using MTP by default and falling back to NTP only when necessary.

Conceptually, Hybrid Mode resembles speculative decoding and other modern acceleration techniques. Most predictions are generated using a fast approximate procedure, while a slower but more reliable mechanism is invoked only for difficult cases. This allows LocateAnything to retain most of the throughput benefits of parallel decoding while avoiding its most common failure modes.

Experiments

On standard object detection benchmarks, LocateAnything consistently outperforms similarly-sized grounding VLMs such as Rex-Omni. More interestingly, the gains persist in dense detection scenarios such as VisDrone and Dense200, suggesting that the model learns robust spatial representations rather than simply memorizing common object layouts. The authors also report strong results on a diverse set of localization tasks, including UI grounding, document layout understanding, and referring expression comprehension, where LocateAnything reaches or exceeds the performance of larger general-purpose VLMs and several specialized systems.

Under the default Hybrid Mode, LocateAnything achieves roughly 12.7 boxes per second, compared to 5.0 for Rex-Omni and 1.1 for Qwen3-VL. The advantage becomes even larger in dense scenes, where the number of predicted boxes grows and autoregressive coordinate generation becomes increasingly expensive.

Ablations

The ablation studies reinforce the paper’s main argument: the gains come from the output representation rather than from scaling the backbone.

• First, replacing coordinate-token generation with box-aligned outputs improves localization quality even in pure autoregressive decoding. This suggests that representing a bounding box as a structured object provides a better learning signal than serializing it into a sequence of coordinate tokens.
• Second, the authors compare Parallel Box Decoding against generic multi-token prediction approaches such as SDLM and Block Diffusion. These methods treat blocks as arbitrary token groups, while LocateAnything aligns blocks with meaningful geometric units. The aligned formulation proves both faster and more accurate, supporting the paper’s broader thesis that output structure matters.
• Finally, the Hybrid Mode successfully balances speed and accuracy. Pure MTP decoding delivers the highest throughput but suffers occasional quality degradation in difficult scenes. The NTP fallback mechanism recovers most of this lost accuracy while retaining the majority of the speedup.

Conclusions

The “predict all the boxes at once” approach is not new; it is the idea behind the DETR and DINO: represent detection as a set prediction problem rather than generating coordinates token-by-token. Those models remain among the strongest closed-set detectors, but they achieve this through specialized detection architectures with fixed label spaces. They are not open-vocabulary systems, instruction-following models, or general-purpose VLMs.

LocateAnything’s bet is to bring some of that set-prediction intuition into a generative VLM. Instead of treating a bounding box as a sequence of coordinate tokens, it treats the entire box as a structured prediction unit and decodes it in parallel. The result is somewhere between DETR-style detectors and coordinate-token VLMs such as Qwen-VL and Rex-Omni.

In hindsight, coordinate-token grounding always felt like a compromise. DETR showed years ago that boxes are naturally set-valued geometric objects, yet many VLMs reverted to serializing them into text because language models only knew how to generate sequences. LocateAnything revisits that earlier intuition and asks what happens if we bring structured box prediction back into the VLM era.

What I like here is that the speed argument and the accuracy argument come from the same design choice rather than being traded against each other. Parallel decoding is normally a latency optimization that is a trade-off for quality; reframing the parallel unit as a coupled box makes it more precise, and the IoU=0.95 jump on LVIS is concrete proof. On the other hand, the advantage disappears at IoU=0.5 and the model still trails specialized detectors on COCO, and the hybrid fallback shows that pure parallelism is not yet enough in dense or multi-class scenes.

Paper

Code

Model

Long-context LLMs keep promising the same thing: feed more tokens into the prompt and let the model reason over them. The bottleneck is rarely the window itself: it is the cost of attending over it once agentic workflows, repo-scale code reasoning, and persistent memory push the context into the hundreds of thousands or millions of tokens. Softmax attention is quadratic, so million-token context is not just a modeling challenge, but an inference-cost and deployment challenge.

MiniMax Sparse Attention (MSA) is the attention design behind MiniMax M3, and it tackles this problem with blockwise sparsity built on top of Grouped Query Attention. The idea is to keep exact softmax attention but run it over a tiny, query-dependent subset of the key-value history instead of the whole thing. A lightweight Index Branch decides which blocks matter, and the expensive Main Branch perform exact softmax attention only over those selected blocks. At 1M-token context, MSA cuts per-token attention FLOPs by 28.4x against a dense GQA baseline of the same 109B-parameter configuration, while staying on par with it on quality.

This is the same framing DeepSeek used for its sparse attention. The interesting question is no longer maximum context length, but whether the model can compute over a long context cheaply enough to deploy. MSA’s particular bet is per-group block selection, and most of the design follows from that choice.

You can read about my experience with MSA in the MiniMax M3 review, which covers its multimodal capabilities.

The approach

Two branches over one GQA backbone

MSA splits each attention layer into an Index Branch and a Main Branch that share the same GQA backbone. The Index Branch adds one index-query head per GQA group and a single shared index-key head, scores every causally visible key token, then max-pools those scores up to the block level. For each GQA group it keeps the top-16 of those 128-token blocks, always force-including the local block that holds the query, for a fixed 2,048-token selection budget. The Main Branch then runs standard exact softmax attention restricted to exactly those selected blocks. Per-query cost drops from growing with sequence length to a constant set by the budget, so attention compute stays flat as context grows.

The design choice that separates MSA from its neighbors is that the Top-k selection is shared per GQA group rather than across all query heads, and it applies at block granularity rather than token granularity. Each group retrieves its own blocks, which preserves multi-group selectivity, while block-level selection keeps the KV reads contiguous so the kernel can stay efficient.

Training a non-differentiable selector

Top-k selection is not differentiable, so the Index Branch cannot learn from the language-model loss directly. MSA trains it with a KL alignment loss that matches the index branch’s block-score distribution to the Main Branch’s actual attention distribution over the selected tokens, so the selector learns to predict which blocks the exact attention would have wanted. Gradient detach confines this auxiliary loss to the index projections alone, keeping it from perturbing the rest of the model. Training starts with an indexer warmup that runs full attention before switching to sparse, which gives the local block a stable target to imitate. The same warmup recipe is what converts a pretrained dense checkpoint into a sparse one.

That conversion path is worth separating from training from scratch, because the two behave differently. MSA-PT is trained sparse from the start; MSA-CPT takes an existing dense GQA checkpoint, swaps in MSA, and continues training.

Experiments

On quality, MSA-PT matches or slightly beats the dense Full-Attention baseline across most of the benchmark table, with the largest margins on multimodal and long-context tasks. MSA-CPT, the conversion route, stays close to the dense checkpoint it started from and is strongest on text, code, and perplexity. The sparse beats dense claims warrant some caution: these are research-scale 109B models trained on only 3T tokens, and the multimodal jumps may reflect native sparse pretraining acting as a form of regularization at this scale rather than evidence that sparse attention wins at frontier scale.

The ablations show that a sliding-window baseline held to the same FLOP budget has uniformly higher perplexity than MSA.

At 1M context, MSA delivers a measured 14.2x prefill and 7.6x decode wall-clock speedup on H800 against the dense GQA baseline, both growing with context length. The paper is honest that the runtime gains are smaller than the 28.4x FLOPs reduction would suggest: index construction, Top-k selection, and a less regular memory-access pattern all decrease the theoretical win.

Conclusions

There are two major approaches within sparse-attention research. One sparsifies a model that was already pretrained densely: methods such as H2O, SnapKV, Quest, MInference, and InfLLM reduce the cost of long-context inference by pruning, compressing, or selectively retrieving from the KV cache. While effective, they inherit the quadratic cost of dense pretraining and can only approximate the attention patterns the model originally learned.

MSA belongs to the second group, where sparsity is built into training from the beginning. The model learns to operate under a fixed attention budget rather than being sparsified after the fact. Several architectures pursue this idea in different ways. NSA combines compressed, selected, and sliding-window attention branches; MoBA performs routing at block granularity using block-level summaries and learns routing implicitly through the language-model objective; InfLLM-V2 retrieves relevant blocks without introducing a learned routing network. The most similar is DeepSeek’s DSA, which performs sparse token-level retrieval on top of Multi-head Latent Attention using a lightweight ReLU-based indexer.

MSA’s distinguishing feature is its combination of per-GQA-group retrieval and block-level sparsity. Instead of forcing all heads to share the same sparse view of the context, each GQA group selects its own relevant blocks. At the same time, operating on contiguous blocks rather than individual tokens produces a sparsity pattern that maps efficiently onto GPU kernels, allowing MSA to improve both retrieval flexibility and hardware efficiency.

I like that MSA is a complete, deployed system rather than a proposal, with a released inference kernel and the cheap MSA-CPT conversion route that lets an existing dense model become sparse without retraining from scratch. At the same time, there are certain caveats: the runtime speedup trails the FLOPs reduction, retrieval-heavy long-context subtasks don’t show outstanding performance, and the scope focuses on pretraining, with no RL or post-training and no benchmark against NSA, MoBA, or DSA.

I got early access to MiniMax M3, so I plugged it into Claude Code and used it to work on a few tasks that I wanted to complete for some time: a code audit and refactor of my old web game, two UI bugs from it that I had been putting off, and a music-recommendation experiment built from my Spotify history. I used M3 for the implementation work, then asked Opus 4.8 to review it.

M3 is the first open-weights model (will soon be fully open-sourced on HuggingFace and GitHub) to combine three things in one release: frontier-level coding and agentic ability, a 1M-token context window, and native multimodality. I reviewed MiniMax M2.7 earlier, and M3 is a clear step up from M2.7 in the areas I tested.

M3 was most useful when I gave it concrete artifacts — a repo, tests, screenshots, and data exports. It did a lot of real work quickly, but an independent review still caught some regressions.

What MSA is, and why MiniMax keeps changing its attention

MiniMax has changed its attention twice (if you want to know more about attention, you can read my note). MiniMax-01 and M1 used lightning attention, a linear-attention variant, in a 7:1 hybrid — seven linear layers per softmax layer. M2 and M2.7 then reverted to full attention; the team’s candid post Why Did M2 End Up as a Full Attention Model? blamed linear attention’s precision sensitivity, immature infra, and multi-hop deficits — all costs of approximating the softmax.

M3 uses MiniMax Sparse Attention (MSA), which keeps the softmax exact and only narrows where it runs. An index branch cheaply scores blocks of context (one lightweight query per GQA group → block-max-pool → top-k), then the real query heads run ordinary full attention over just the selected blocks. MiniMax reports it running 4× faster than Flash-Sparse-Attention, at ~1/20 the per-token compute of M2, with 9× prefill and 15× decode speedups — their own numbers, unreproducible until the weights and report ship.

So MSA “matching full attention on the vast majority of capabilities” isn’t surprising: it doesn’t approximate, it selects. The only thing that can break is the selection — drop a block that mattered and the answer is gone. The real question is how good the selector is at long range.

Auditing and refactoring an old idle game

A year ago, I vibe-coded an idle game, Eternum Alchemist, with Sonnet, and I wanted to pick it up again. Before adding anything new, I asked M3 to carefully review the code for bugs, security issues, and logic problems. It spent roughly 30 minutes on the repository understanding and analysis, which isn’t surprising given it has ~100 files and ~26k lines of code.

The report was quite good. It was organized by severity (12 critical, around 20 high, 30 medium, 20 low), carried file paths and line numbers, and included a recommended order of work. Some of the most important issues were:

• shouldAttack using an integer-modulo model that made every enemy with an attack speed above 1 always attack, so the snake monster was effectively slower than intended.
• A lot of unfinished code/configs. For example, after skills reached prestige, they couldn’t level up, because their XP scaling was nested under ranks[rank] while the function read a top-level field and got NaN.

I asked M3 to fix all issues. It worked for ~2h 40m across three phases, increased the number of tests from 188 to 237, and most of the fixes were correct and well tested.

But then I asked Opus to review the changes, and it found two critical regressions that M3’s own green tests had hidden.

• M3 added schema validation to the import path, changing the data format and conflicting with the save format. Thankfully, the game is in alpha or pre-alpha stage, so this is fine, but if this were in production, the saves would be broken.
• M3 fixed non-working multipliers, but forgot that the crit hit chance was applied in two places, which resulted in it scaling as 1.05 to the power of twice the level. It was exactly the config-drift pattern the audit itself had flagged elsewhere and not fixed here.

Other than that, Opus found that six fixes were partial and six issues were untouched. As a takeaway, I can say that M3 did a large amount of correct, well-structured work quickly. But it was my mistake to let M3 both write the tests and fix the code issues. Next time, I’ll use two separate sessions for it.

Two UI bugs that needed a screenshot

The next two problems were UI-related, and that’s where M3’s multimodality came in handy.

The first was a freeze. On the Skills screen, clicking a skill froze the whole panel, and every click after the first did nothing. Describing the symptom in words got me nowhere — the model kept guessing at event handlers and React state. So I shared the screen: a screenshot plus the open DevTools. After about fifteen minutes of reading the actual rendered DOM, it found the cause, and it was not in any handler. There were two stacked global-CSS collisions, both from Create React App shipping non-scoped CSS. The first was real but only cosmetic: a .main-content grid rule squeezed the Skills window into the left half. The actual click-blocker was a .progress-text rule in ProgressBar.css — position: absolute; top: 0; left: 0; width: 100%; height: 100% — meant to center a percentage label over a progress bar. That global class leaked onto SkillsView’s own .progress-text, and because the nearest positioned ancestor was .main-content, it expanded into a full-size invisible overlay covering the entire view. That caused the issue that I encountered: with no skill selected, there is no XP-info element and no overlay, so the first click renders the overlay and every later click hits it instead. The fix was to put ProgressBar.css under a .progress-bar-container, and a regression test now fails if this breaks.

The second bug was simple. The “Current Bonuses” panel listed every future unlock — at level 10+, 25+, 30+, and so on — as if it were already active. I took a screenshot, described the problem (“this section shows all future bonuses; it should show only the current ones”), and asked it to ultrathink and fix it. It split the panel: “Current Bonuses” now shows only what is active at the current level (a single “3% increased gathering speed” for my level 3 skill), and the future unlocks moved into their own “Upcoming Bonuses” section.

Both fixes were small once found, and both were much easier to diagnose with screenshots. Being able to ask a model to reason about the image demonstrates the value of multimodal models.

Music recommendations from years of Spotify history

The last task was for fun. I have listened to music offline for years (Winamp, AIMP, VOX), but several years ago, I switched to Spotify after my friends pressured me to try new music, and I was curious. At first, I loved Spotify recommendations, but over time, they drifted into either repetition or noise. I wanted to see if M3 could do better, so I exported my extended streaming history and asked it to analyze it in depth, identify my tastes, and recommend new and exploratory artists and songs, with the output as an HTML report plus a CSV of the full list.

The analysis was mostly what I expected (I know what I listen to, and Spotify Wrapped helps too). M3 builds a listening profile from about 74k streams over five years (4.3k hours, 2.1k artists): a melancholic-romantic core with an anime and synth-pop streak and a symphonic-metal second life, broken into genre pillars by hours — Japanese city pop and 80s J-pop near 700 hours, symphonic and power metal near 500, anime and game OST near 400, then everything else.

One funny thing was the visualization of the countries. I have never been to France or Germany, but they were at the top of the chart thanks to VPN.

The analysis was cool and interesting, but some of the plots were questionable. The Platforms chart unnecessarily showed 6 platforms when two of them had like 99.9% of the streams, and the rest were insignificant.

The recommendations were organized into tiers, from safe to exploratory (“cross-niche bridges,” “adventurous picks with hidden bridges”), each with a one-line reason explaining how the artist connects to something I already listen to. I have not worked through all of them, but the results are positive: most songs are okay, a few are completely off, and several are my new favorites. One of the best finds was 中森明菜 (Akina Nakamori) with DESIRE -情熱-, the kind of song I like, the one Spotify never showed me before.

Conclusion

Across the audit, the UI bugs, and the music experiment, M3 was most useful where the task gave it something concrete to work against: a test suite, a screenshot, a data export. It is fast and cheap enough to run several supervised passes, and its multimodality is a real practical advantage for debugging. The one thing I would not skip is an independent review of anything that matters, because a model that writes both the fix and the tests can be confidently wrong on both, and here it took a second model to catch it. I will keep using it for supervised refactors and screenshot-driven debugging, with a separate reviewer in the loop for the parts I care about.

This post was written in a paid partnership with the MiniMax team. If you want to try MiniMax, you can use this code for a 12% discount.

Amazon

Author’s website

I was offered to read 50 ML Projects To Understand LLMs by Mike X Cohen in exchange for an honest review. Rather than building, fine-tuning, or prompting LLMs, the book treats GPT-2 as a scientific specimen and teaches you to investigate it with code, statistics, and controlled experiments across 50 hands-on projects. As I have spent a lot of time working on model validation and LLM evaluation, I liked that the author focuses on the statistical discipline throughout the book — permutation tests, multiple-comparison corrections, control baselines, manipulation checks. That kind of validation rigor is what separates a real result from a lucky one, both at work and in ML competitions.

The overall structure

The 50 projects are organized into six chapters that loosely follow the flow of data through a transformer model:

• Tokenization: how text becomes integers, whether tokenization is really compression, and how strongly tokenizers favor English.
• Embeddings: cosine similarity, comparing models with representational similarity analysis, semantic axes, and analogy vectors.
• Output logits: softmax, sampling strategies, the loss function, perplexity, evaluation with HellaSwag, and measuring language bias.
• Transformer outputs: the residual stream, hidden states, the logit lens, and patching hidden states to find where a capability lives.
• Attention: query-key-value weights and activations, raw versus softmax attention scores, head silencing, and patching attention heads.
• MLP: neuron characteristics, grammar tuning, lesioning neurons, supervised probing with XGBoost, and a deliberately silly recommender-system capstone.

Each project has a similar structure: a bit of background, a task box telling you what to build, a figure to reproduce, and an interpretation of what you found. Every project includes two notebooks, a “helper” with gaps to fill and a complete solution, so you can pick your difficulty: code from a blank notebook, fill in the helper, or read the solution. Cohen explicitly tells you it is not cheating to look at the solutions. The whole book uses GPT-2, which is small enough to run on a laptop and open enough to inspect completely.

What I liked

There were many things I liked in this book, and I want to highlight several in particular:

• The statistical rigor is a part of analysis: Cohen’s d for effect sizes, permutation testing with a proper discussion of exchangeability, FDR and Bonferroni corrections, split-half cross-validation, and manipulation checks.
• The controls and ablations are thorough and precise. One project ablates the least-tuned neurons as a baseline before moving the interesting ones; another compares real hidden states against shuffled ones to estimate effective dimensionality. Without that kind of baseline, it is far too easy to convince yourself that a meaningless number means something.
• The “try the obvious thing, watch it fail, learn the right approach” structure. You compare embeddings across two models with plain cosine similarity, see why it cannot work, and arrive at representational similarity analysis. You fit a linear regression to categorical token positions, watch it misbehave, and switch to logistic regression.
• GPT-2 by itself. Because the model is small, you actually run every experiment yourself instead of reading about someone else’s results.
• I liked the conversational tone of the book. “Are you disappointed in the results? I was when I first saw them!” appears more than once, and one whole project ends with the weight distributions being “not terribly interesting.” That is an unusual and welcome thing in a teaching book.
• The tokenizer-bias project, where the same sentence needs 36 GPT-2 tokens in English but 557 in Tamil.

The projects I had most fun with are the ones that go deep into model internals. In Project 32, you patch hidden states inside an indirect-object-identification task and fit a sigmoid to find the layer where the model settles on the answer. In Project 44, you go looking for grammar in MLP projections: at the population level, nouns and verbs look identical, but if we drill down to the level of individual neurons, we can notice the difference. In Project 35, you analyze why negative raw attention scores enforce sparsity after softmax. These are real interpretability experiments, and you come out able to run your own.

I also liked the short section on AI assistance in the introduction. Cohen writes that “the more time goes on, the less I use AI for coding and writing,” explains that he mainly used LLMs for code review and brainstorming, and warns readers that “if you have code you do not understand and solutions you cannot explain, then you’re probably relying too heavily on AI”.

What could have been better

There are a few small things that could have been handled differently:

• Top-p sampling is called “nuclear sampling” in a couple of places, but the common name is “nucleus sampling”. A tiny fix, but it would make it easier to find the original paper.
• The book focuses almost entirely on GPT-2, with only occasional references to BERT, RoBERTa, and Pythia. Several findings are presented as general properties of transformers, and one or two more cross-model or cross-scale comparisons would make those claims more convincing. The author flags this himself, so it is more a wish than a complaint.
• The tokenization chapter spends seven projects on tokenizers but doesn’t work through a byte-pair-encoding example by hand. A short walk-through of how BPE builds its vocabulary would round out an otherwise excellent chapter.

But these are small nitpicks that are completely overshadowed by the good sides of the book.

Conclusion

This book is a good fit for people who already use transformers and want to see inside them: ML engineers and data scientists comfortable in Python, students looking for a hands-on on-ramp into mechanistic interpretability, and anyone who learns better by running experiments than by reading papers. You do need real Python skills and some patience, but you do not need a background in LLM internals, since that is exactly what the book gives you.

GPT-2 is old by the standards of the field, and the specific numbers in any interpretability result will date quickly. The value that lasts is the way of thinking: form a hypothesis, build the right control, test it properly, and stay skeptical of your own results even when they look exciting. Those habits carry over to any model, and they are what I will keep from this book long after GPT-2 stops being a useful teaching tool.

Paper

Code

Project

Most interactive video world models still assume a single agent: one user, one action stream, one generated future. γ-World adopts a harder, more realistic setting: several independently acting agents share the same evolving world. This is essential for games, robotics, embodied AI, social simulation, and agent training environments, where the key problem is not only visual fidelity, but whether multiple agents can act, interact, and remain consistent over time.

The paper’s central contribution is a clean multi-agent design for generative world modeling. It introduces Simplex Rotary Agent Encoding to represent agent identities without fixed slots or arbitrary ordering, Sparse Hub Attention to let agents exchange information without expensive all-to-all attention, and a teacher-student distillation setup that turns a full-context diffusion model into a causal streaming model. Gamma-World itself is a DiT-based latent video diffusion model trained with flow-matching, extended along an explicit agent axis. The result is a model that can produce action-responsive multi-agent rollouts in real time, while preserving independent controllability and even generalizing from two to four players without additional training.

The approach

The model uses a transformer-based latent video diffusion model adapted for autoregressive generation, in which rotary position embeddings encode spatial and temporal locations. The authors modify this position embedding to account for agent identities and implement a multi-agent aware attention masking mechanism to reduce computational cost. The model is trained in two steps: a bidirectional teacher model and a causal student model that supports the streaming setting.

Unlike traditional world models that generate a future for a single player, γ-World generates futures for all agents simultaneously.

The model:

• Receives the first observation from each agent.
• Receives an action sequence for each agent.
• Predicts future observations for every agent jointly.

The key goal is consistency across agents and time. If agent A moves left and agent B observes agent A, both generated views should agree on what happened.

Shared Action Conditioning

Every agent has its own action sequence. A single shared encoder maps actions to latent representations, action features are injected into the transformer at every layer as additive biases. This means that each agent can be controlled independently while the parameters are shared.

Simplex Rotary Agent Encoding

Standard 3D RoPE gives video transformers rotary bands for time, height, and width. Simplex Rotary Agent Encoding (SRAE) adds a fourth band for agent identity: instead of assigning each agent a learned ID embedding, which fixes the roster and breaks symmetry the moment you reorder players, SRAE puts agents at the vertices of a regular simplex in rotary-angle space. Every pair of agents then sits at equal distance, so no agent is privileged, and the encoding does not care which slot a player occupies. The architecture never changes when an agent is added, which is what lets a model trained on two players accept four without retraining.

Sparse Hub Attention

The other half of the design is how agents share information. Dense all-to-all attention across agents is quadratic in the number of agents, so it scales poorly for large numbers of agents. Sparse Hub Attention (SHA) routes cross-agent interaction through a small set of learnable hub tokens that act as a compact representation of the environment state. Each agent attends to its own stream and to the hub tokens. The information flow is linear (agent -> hub -> agent) rather than quadratic.

This design significantly reduces computational cost while maintaining a communication pathway between agents. Together, these two ideas allow the model to generate coherent multi-agent rollouts, preserve consistency across viewpoints, and scale beyond the two-player settings that dominate most previous work.

Model training and inference

A major challenge in world modeling is balancing generation quality with real-time interactive inference. High-quality diffusion models typically rely on bidirectional attention, allowing them to look into the future during training, but this makes them unsuitable for online generation. Conversely, causal models support streaming generation but often suffer from exposure bias because they are trained on ground-truth histories while being evaluated on their own predictions.

γ-World uses a three-stage training pipeline:

• First, the authors train a powerful bidirectional teacher model that has access to the full multi-agent trajectory and can learn rich temporal dependencies and cross-agent interactions.
• Next, they train a causal student model using Diffusion Forcing, enabling autoregressive generation while preserving multi-agent communication through Sparse Hub Attention.
• Finally, the causal model is distilled into a few-step generator using Conditional Self-Forcing, where the model learns under its own rollout distribution and is encouraged to remain faithful to both the initial observations and the specified action sequences.

This training strategy allows γ-World to combine the strengths of both paradigms: the visual quality and consistency of diffusion models with the low-latency streaming capabilities required for interactive simulation. During inference, the distilled model generates future blocks autoregressively using KV-cached attention, while maintaining cross-agent coordination through shared hub states. The result is a real-time multi-agent world model capable of streaming coherent rollouts at 24 FPS.

Experiments

Training is on two-agent Minecraft trajectories, and the generation-quality numbers are all two-player. Against the concurrent Solaris (a multiplayer-Minecraft world model that uses dense joint attention and learned per-player IDs) and a frame-concatenation baseline, γ-World achieves significantly lower FID and FVD scores in tasks requiring memory, grounding, movement, building, and cross-view consistency. On Memory it cuts FVD from Solaris’s 333.8 to 184.1 and FID from 51.7 to 24.8; on Consistency, the hardest protocol, FVD drops from 443.1 to 280.0.

The ablation studies indicate that each of the paper’s major design decisions contributes to performance. Treating agents as separate streams is more effective than spatially concatenating their observations, Simplex Rotary Agent Encoding consistently outperforms learned view embeddings, and Sparse Hub Attention preserves quality while providing a scalable communication mechanism between agents. Together, these components produce the strongest overall results, supporting the authors’ central claim that agents should be modeled as distinct but exchangeable entities connected through a shared interaction state.

Four-player generation is shown only qualitatively, zero-shot, with no metrics.

Conclusions

This idea loosely echoes recent work such as DroPE, in the sense that both papers treat rotary embeddings as an architectural degree of freedom rather than a fixed implementation detail. However, the motivation is almost opposite: DroPE removes positional embeddings to improve length extrapolation in LLMs, while γ-World reallocates rotary dimensions to introduce a permutation-symmetric agent axis for multi-agent world modeling.

γ-World belongs to the emerging family of interactive video world models, alongside systems such as Oasis/Genie-style single-agent worlds, Matrix-Game-style real-time long-horizon models, MultiWorld-style multi-agent multi-view models, and ActionParty-style action-binding models. Compared with these, γ-World’s strongest distinguishing idea is not just “better video generation”, but a principled treatment of agent exchangeability: the model should not depend on Player 1/Player 2 ordering, learned identity slots, or dense pairwise attention.

Its contribution is therefore architectural and conceptual. Matrix-Game 3.0 emphasizes high-resolution real-time generation and long-horizon memory; MultiWorld emphasizes multi-view consistency; ActionParty emphasizes subject-action binding. γ-World contributes a scalable, permutation-symmetric, independently controllable multi-agent generation. The paper is important because it pushes world models closer to actual shared environments rather than controllable video demos.

I recently got access to some MiniMax M2.7 API credits, so I decided to plug this model directly into Claude Code and run it on three workflows I do regularly. The same tasks were run using Claude Opus 4.7 as the comparison baseline.

The three workflows: scaffolding an entry for an active Kaggle competition, drafting and auditing knowledge-base notes for my Obsidian vault, and updating an old PyTorch project that became outdated. I wanted to find out how well M2.7 works inside an agentic loop when the task has clear boundaries. The results were consistent across the three runs: M2.7 was useful when the constraints were explicit, and the output format was concrete. It stumbled when important context was left implicit, though some of the same gaps appeared with Opus 4.7 as well.

For the more open-ended cases, I would still keep a human review pass in the loop.

Setup

I added a claude-mm command that points Claude Code at the MiniMax API and ran M2.7 with thinking set to max in the CC interface. I ran on MiniMax’s Plus tier (High-Speed, $40/month), where the context window and per-day throughput no longer became bottlenecks for multi-step agentic work.

claude-mm() {
  ANTHROPIC_BASE_URL="https://api.minimax.io/anthropic" \
  ANTHROPIC_AUTH_TOKEN="$MINIMAX_API_KEY" \
  ANTHROPIC_MODEL="MiniMax-M2.7" \
  ANTHROPIC_DEFAULT_SONNET_MODEL="MiniMax-M2.7" \
  ANTHROPIC_DEFAULT_OPUS_MODEL="MiniMax-M2.7" \
  ANTHROPIC_DEFAULT_HAIKU_MODEL="MiniMax-M2.7" \
  ANTHROPIC_SMALL_FAST_MODEL="MiniMax-M2.7" \
  API_TIMEOUT_MS="3000000" \
  CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC="1" \
  claude "$@"
}

In agentic work, the harness can be as important as the model itself. Most of the failures I describe below had similar reasons: the prompt did not explicitly state a constraint the task depended on, and the model filled the gap with a plausible default. In practice, model quality and harness design are hard to separate. A stronger model may infer missing constraints; a better harness may make those constraints explicit. I treated this as a workflow test, not a pure model benchmark.

Refactoring an old PyTorch project

The first workflow was a refactor: my pytorch_tempest repo is a framework for training neural nets using Hydra + PyTorch Lightning. I wanted to update dependencies, modernize the tooling, and clean up the code issues that had accumulated over time. The merged result is PR: refactoring old code and updating dependencies.

The changes:

• Updated CI versions and pre-commit hooks.
• Replaced black and flake8 with ruff for both linting and formatting.
• Enabled fsdp_sharding_strategy in the Lightning trainer config.
• Refreshed the documentation.
• Added uv for environment management.
• Switched to modern Python typing (list[X] over List[X], X | None over Optional[X]).
• Removed duplicate code paths.
• Fixed a lot of small issues.

I guided M2.7 explicitly: provided step-by-step requirements (“switch black + flake8 to ruff”, “update the pre-commit config”), reviewed each change before moving to the next, and provided feedback when the diff went outside scope. I had enough tests to check whether anything broke after the changes, and rerunning model training took only several minutes. I had some challenges running CI, and the agent helped me fix them one by one.

A lot of engineers I know do not want to give an agent free rein over a codebase they care about; they want to supervise the execution and know every existing line of code. M2.7 fits this approach well. You can write short, narrow-scope prompts, conduct line-level review, and then move to the next step.

Knowledge notes for the Obsidian vault

The second workflow was writing and auditing notes for my Obsidian vault, where I keep around ML reference notes. I write most of them by hand; sometimes I have an LLM draft a parallel version to compare against and take inspiration from.

It is important to remember that different models prefer different prompt styles. A 100-line prompt tuned for Opus 4.7 does not transfer one-to-one to M2.7. To handle that, I did a small bootstrap: I asked both models to generate notes from the same starting prompt, then asked M2.7 to read both notes and propose an improved prompt for itself. The next iteration used the M2.7-tuned prompt.

I used two prompts (a writer command and a critic agent), each around 100 lines. Here is a condensed version of the first one:

Fill one broken-link stub in the DSWoK vault: research the topic, draft the note in DSWoK voice, run draft-critic-mm, save to the right folder.

1. Read context: writing style guide, frontmatter taxonomy, alias rule.
2. Pick the stub.
3. Locate references — Grep for [[<title>]] across the vault.
4. Pick the destination folder based on topical group.
5. Find a structural template from neighbouring notes.
6. Research via 3–5 sources, search-first — don't trust memory for citations, formula conventions, or post-2024 work.
6.5. Verify each cited URL before pasting it. Hard-to-verify URLs are blocking errors.
7. Determine note type and structure.
8. Draft the note with frontmatter taxonomy + style rules.
9. Cross-link inline to adjacent notes.
10. Run draft-critic-mm and address every blocking issue.

The critic agent has a similarly explicit checklist. The point of writing two detailed prompts is to make the evaluation criteria concrete: this means the model needs to make fewer judgment calls and can self-audit its output.

I shared gists with the command and the critic.

I tested both M2.7 and Opus on four notes: Negative Sampling, MAP (Mean Average Precision), Cold Start (a recommender-systems problem), and RMSE.

In the RMSE note, M2.7 got several things right:

• It flagged that RMSE “does not decompose into bias and variance” the way MSE does, because the square root is nonlinear.
• It cited Hyndman & Koehler 2006 (the canonical forecasting paper introducing MASE and scaled errors) at the right place.
• The Properties section used inline mini-headers with bold formatting, as defined in the style guide.
• The intro was tighter than Opus’s version.

What needed editing:

• Bullet-label bold (**Rating prediction.**, **Not robust to heavy-tailed noise.**) - this is against the style guide, but easily fixed.
• Missing Variants section: RMSLE, NRMSE, and weighted RMSE are absent. This wasn’t defined in the prompt, but it would be a very good addition to the text.
• The Willmott reference pointed to a 2006 JAM paper (DOI 10.1175/JAM2472.1) rather than the canonical 2005 Climate Research paper that practitioners usually cite.

The other three notes had the same pattern: solid first drafts, accuracy in the technical core, occasional citation mistakes, and occasional ignoring of style rules. Most of these issues (except the hallucinations) are easy to notice and to fix.

One additional experiment: I asked M2.7 to audit my existing notes and find possible problems. The audit was useful: the model found many formatting issues, including incorrect tags, typos, and missing cross-links. One flagged item was funny, though:

Metrics and losses/f1 score.md

Only 1 tag (evaluation); missing domain tag (recsys, nlp, or cv)

The F1 score is a general classification metric and does not need a domain tag by my taxonomy. M2.7 inferred a rule by analyzing larger notes, even though such a rule doesn’t exist. The fix was to include the tag hierarchy in the prompt next time, just as I include the writing-style guide for the drafting task.

Across the four notes and the audit run, M2.7 worked well for creating a first draft. It created useful tables and small visualizations, and the technical content was usually right, but references needed checking.

Here are the final versions of the notes after review, adding more ideas and heavy editing:

• Negative Sampling
• Cold start

Kaggle: ROGII — Wellbore Geology Prediction

The final task was the ROGII Wellbore Geology Prediction competition: predicting geological layer tops along well paths from drilling-time measurements. Quasi-spatial data, anisotropic distances, a handful of wells with target labels, and per-well prediction error as the scoring metric.

I’m a Competition Master and Notebook Grandmaster, and I was curious to see how well an agent could perform in a new competition. I intentionally started with a high-level prompt rather than a fully specified implementation plan, because that is a realistic simulation for a first interaction with Kaggle. I’ve accumulated notes, code, and write-ups from earlier Kaggle competitions over the years; I shared them as context, along with explanations of what Kaggle is, what competitions are, and how to participate. The goal was to create a first submission that could be iterated on.

M2.7 spent a considerable time on the analysis. The first working result was this notebook rogii-wellbore-final-kriging: a 5-fold validation split by well, ~40 features, and training a gradient boosting model. The validation split was not standard:

# 5-fold GroupKFold by well_id
unique_wells = pre_ps_train["well_id"].unique()
well_to_fold = {w: i % 5 for i, w in enumerate(unique_wells)}
fold_assignments = pre_ps_train["well_id"].map(well_to_fold)

The usual approach would be to use GroupKFold from sklearn, but this “cheap” version was fine for a first pass.

There were two issues, both due to Kaggle-specific mechanics rather than the model’s ML reasoning. ROGII is a kernel-only competition: at submission time, the test set you see (three rows with target values exposed) gets swapped out for the real test set (much larger, no target values). This means that models can miss these mechanics unless they are stated in the prompt. As a result, I noticed two problems:

• The model assumed the three exposed test rows were the entire test set and hardcoded them.
• It treated the exposed target column as a regular feature and used it during feature engineering.

The first submission didn’t succeed: with the target leaked into the feature set, the model trained against a column it would not have at inference, and the submission failed due to hardcoding the three available test samples.

Interestingly, Opus 4.7 also used the exposed target in feature engineering in the same setup. The kernel-only rules are not something either model picks up from “this is a Kaggle competition” — they have to be in the prompt. After I explicitly explained the mechanics, M2.7 fixed both bugs in one pass, and the submission worked.

It then produced a more advanced version: the rogii-idw-lightgbm-residual notebook, with inverse-distance-weighting features and a LightGBM residual model (without leaks) on top, scoring better than the first attempt.

In terms of participating in Kaggle competitions, M2.7 worked well for building a scaffold for future work: setting up basic validation, starting feature engineering, and training a model. After that, it can iteratively improve the solution if you provide strict constraints and specify the direction (e.g., improving a specific metric).

Cost and throughput

I ran this on MiniMax’s $40/month Plus plan and never came close to the rate limits across five days of intensive Claude Code sessions. The subscription dashboard showed that M2.7 processed roughly 91M total tokens, with most of them cache reads. At M2.7’s PAYG rates ($0.30/$1.20 per million input/output, $0.06 per million cache reads), that’s around $8 worth of usage. I didn’t log Opus 4.7’s token usage, but at its rates ($5/$25, $0.50 cache reads), it would cost around 10x.

In terms of speed, M2.7 returned tool calls and completed multi-step plans noticeably faster than Opus 4.7 on the same tasks — subjectively around 2x. I didn’t benchmark rigorously, but the difference was noticeable. Combined with the cost ratio, this means you can run several supervised iterations on M2.7 within the time and budget of one Opus iteration

Where I’d use M2.7 going forward, and where I wouldn’t

Across the ROGII submission, the four Obsidian notes, and the pytorch_tempest refactor, the results are similar. M2.7 works well when the task has clear boundaries, explicit evaluation criteria, and concrete output requirements. The cases where it fell short had a common cause: the prompt left a piece of context unstated, and the model filled the gap with a reasonable but wrong assumption. In some cases, the same prompt produced the same gap in Opus.

I would use M2.7 going forward for:

• Supervised refactors with a narrow scope and rapid iteration.
• First-draft technical content that I am going to review anyway: knowledge notes, drafts, or boilerplate for new repos.
• Audit of existing documents: when I explicitly provide the taxonomy or a list of checks.
• Iterating over existing machine learning code to improve the metrics given explicit constraints.

What I would not yet hand to M2.7 unsupervised:

• Open-ended ML competition strategy beyond the initial setup. The decisions should be made by humans or by an advanced model. When the direction is split into tasks, M2.7 can start implementing them.
• Reference-heavy technical writing without verification. This is not specific to M2.7 — citation hallucinations happen with most models I have tested. The workaround is the same: verify every URL, and treat it as another step in the workflow.

Across the three workflows, M2.7 was the right tool when I could define the constraints. When the task required the model to figure out the constraints itself (what “kernel-only” implies, what taxonomy applies to F1) both M2.7 and Opus failed, and Opus failed less. The trade is roughly 10x in cost per equivalent task. For supervised work with rapid iteration, using M2.7 is worth it.

This post was written in partnership with the MiniMax team. If you are interested in trying MiniMax, you can use this code for 12% discount.

UPD: Now you can read my next blogpost: Testing MiniMax M3 on real tasks: repo refactor, screenshot debugging, and Spotify recommendations.

Paper

Project

Long-context LLMs usually promise a simple capability: put more tokens into the prompt and let the model reason over them. This works up to a point, but it hides a structural bottleneck: a long context window is only useful if the model can actually afford to attend over it during inference, tool use, and long reasoning trajectories.

DeepSeek-V4 changes the focus from maximum context length to efficient long-horizon computation. Both available models (V4-Pro with 1.6T total / 49B active parameters and V4-Flash with 284B / 13B active) support 1M-token context windows. The whole architecture is built around making that window usable: hybrid compressed attention (Compressed Sparse, Heavily Compressed, and Sliding Window, interleaved across layers), a scaled MoE with Manifold-Constrained Hyper-Connections, Muon optimizer, reduced KV-cache cost, and a post-training recipe that replaces unified-policy RL with on-policy distillation of independently trained domain specialists

The central claim is that future reasoning and agentic systems will not be limited only by model quality, but also by whether the model can maintain useful state over very long trajectories. DeepSeek-V4 is interesting because it treats long context as an infrastructure problem inside the model itself.

Sparse MoE, mHC, and training stability

The DeepSeekMoE backbone from V3 scales up: V4-Flash has 256 routed experts + 1 shared, V4-Pro 384 + 1 shared, both activating 6 experts per token. Load balancing uses V3’s auxiliary-loss-free scheme plus a sequence-wise balance loss to prevent pathological routing on individual sequences. DeepSeek-V4 keeps multiple parts of the previous DeepSeek design: DeepSeekMoE for feed-forward layers, Multi-Token Prediction, and the broader MoE approach. The model also replaces dense FFN layers in the early transformer blocks with MoE layers using hash routing, while keeping the MTP strategy from DeepSeek-V3.

V4 integrates mHC directly into the backbone, projecting the residual mixing matrix onto the Birkhoff polytope of doubly stochastic matrices via Sinkhorn–Knopp with ~20 normalization iterations. This keeps the residual connection in the generalized identity regime that plain Hyper-Connections break. V4 is the first frontier-scale deployment of mHC, and the authors report it trains cleanly where unconstrained HC diverges.

Two stability mechanisms get added on top, both mentioned as empirical without theoretical grounding. Anticipatory Routing computes and caches routing indices Δt steps earlier, using historical router parameters, then applies them during the later main training step. SwiGLU Clamping clamps the gate’s linear component to [-10, 10] and caps the gate component above at 10, and reportedly eliminates loss spikes that emerge at trillion-parameter scale.

Hybrid attention

• Compressed Sparse Attention (CSA) first compresses the KV cache along the sequence dimension, then applies DeepSeek Sparse Attention over the compressed representation. Instead of allowing every query to attend densely to the full history, it compresses groups of tokens into fewer KV entries and then selects a limited number of compressed blocks for each query.
• Heavily Compressed Attention (HCA) uses a much larger compression ratio, but removes sparse selection. The compressed sequence becomes short enough that dense attention over compressed blocks is affordable. In other words, CSA preserves more selectivity, while HCA provides an aggressively compressed global view.

• Attention sinks add learnable sink logits to the attention denominator in CSA and HCA. This means each query head does not have to distribute all attention mass over previous tokens or compressed blocks: the total attention assigned to actual context can be less than 1, and even close to 0. This is useful in long-context attention because not every query should be forced to attend to some distant or weakly relevant context block.

Systems and precision

Pretraining uses Muon as the main optimizer (AdamW on embeddings, prediction heads, and RMSNorm weights) across 32T tokens for Flash and 33T for Pro. Sequence length is ramped 4K → 16K with dense attention; the sparse-attention path is switched on at a 64K stage. The query-key indexer path is quantized FP32 → BF16 for a 2× speedup with 99.7% recall on the top-k set.

Compared with DeepSeek-V3.2, DeepSeek-V4-Pro uses only 27% of the single-token inference FLOPs and 10% of the KV cache size at one million tokens. DeepSeek-V4-Flash reduces this further to 10% of the FLOPs and 7% of the KV cache size.

In DeepSeek-V3.2, reasoning traces were preserved across tool-result rounds but discarded when a new user message arrived. DeepSeek-V4 changes this for tool-calling scenarios. If the conversation contains tool calls, the reasoning content is preserved across the entire conversation, including across user message boundaries. A long-running coding agent needs to remember why it changed a file, which tests failed, which hypotheses were rejected, and what the next step should be.

The authors introduce Quick Instruction tokens for auxiliary tasks such as deciding whether to trigger search or recognizing intent. Instead of using a separate small model that requires redundant prefilling, these special tokens reuse the already-computed KV cache. The point is not just model quality; it is reducing orchestration overhead around the model.

Post-training: On-Policy Distillation and Generative Reward Models

The post-training recipe diverges from V3 and DeepSeek-R1. R1 ran GRPO on a single unified policy with rule-based rewards; V4 instead trains N domain specialists independently (math, competitive coding, agent use, instruction following, and others), each with its own RL loop on high-quality in-domain data. The merge happens via On-Policy Distillation (OPD): a weighted sum of full-vocabulary KL divergences from each specialist’s output distribution into a single student policy, with the student trained on its own on-policy rollouts.

The KL is computed over the full vocabulary rather than a token-level estimate, stabilizing gradients when specialists disagree. The per-specialist weighting is tunable, which means specialists explore different regions of behavior, and the final model learns how to absorb their distributions in contexts generated by itself.

This is why DeepSeek-V4 supports multiple reasoning-effort modes: the model is trained to operate under different inference budgets.

For hard-to-verify tasks, V4 also moves away from conventional scalar reward models. The authors use rubric-guided RL data and a Generative Reward Model, where the actor itself functions as the evaluator. This is less clean than rule-based verification, but it gives them a way to apply RL to tasks where correctness cannot be reduced to tests or exact answers.

Experiments

For base models, DeepSeek-V4-Pro-Base improves over DeepSeek-V3.2-Base across many knowledge, reasoning, coding, and long-context benchmarks. For long context, the MRCR results show stable retrieval up to 128K tokens, with degradation beyond that point but still meaningful performance at one million tokens. DeepSeek-V4-Pro-Max reports 0.59 average MMR on MRCR 8-needle at 1M tokens, while V4-Flash-Max reports 0.49.

Overall, the authors claim reaching open-source SOTA in agentic coding, strong world knowledge among open models, and reasoning performance that rivals top closed models.

Some evaluations are blank because APIs were too busy to return responses, and GPT-5.4 was not evaluated on some long-context tasks because its API failed to respond to many queries.

Limitations

The model is released as a preview. The technical report is detailed, but many practical questions will only be answered by external usage: how stable the one-million-token context is across real agent traces, how often compression loses critical details, and how well the tool-use thinking path generalizes outside DeepSeek’s own harness.

Second, the evaluations are strong but not fully independent. Several evaluations use internal frameworks, internal tasks, or vendor-controlled harnesses. This is normal for frontier model reports, but it means the most useful evidence will come from external SWE-bench-style, terminal, retrieval, and long-context evaluations.

Conclusions

V4 is the first DeepSeek release where the architectural part is more interesting than the RL one. Hybrid compressed attention makes 1M context servable at a fraction of V3.2’s cost, and on-policy distillation of independent domain specialists replaces the unified GRPO pipeline from DeepSeek-R1 with a compositional alternative. R1 showed that RL on a base model can elicit reasoning; V4 now claims that decomposing into specialists and merging via full-vocabulary KL is better than holding every skill in one policy. Compared to Kimi K2.5, V4 and Kimi K2.5 focus on different bottlenecks: K2.5 on native multimodality and learned agent orchestration, V4 on sparse attention and compositional post-training.

Instead of treating context length as a static model property, DeepSeek-V4 treats it as part of the runtime system for reasoning and tool use. This is the right direction. Long-horizon agents will not work just because models become smarter. They need memory that is cheap enough to keep, structured enough to retrieve from, and stable enough to support many steps of reasoning.

I like that the paper is honest about what is still open. The stability tricks are empirical without theory. Opus 4.6 retains a 13-point lead on internal R&D coding, and long-context performance degrades gradually rather than staying flat at 1M.