Quick Summary: A 535B Model Enters Live Training

The open foundation model project Marin, associated with Stanford’s Center for Research on Foundation Models (CRFM), has started training Marin 535B-A23B. The main story is not just model size, but the decision to expose training curves, data recipes, model configurations, and technical discussions while the run is still underway.

Key facts:

• Project background: Marin originated at Stanford CRFM and was publicly announced in May 2025
• Model parameters: 535B total, ~23B activated per token under an MoE architecture
• Training scale: 18.75T tokens, with about 80% for pretraining and 20% for mid-training
• Hardware: 11 NVIDIA GB200 NVL72 systems
• Duration and compute: about three months, totaling roughly 2.7×10²⁴ FLOPs
• Current status: training is still in progress, with post-training to follow

The important nuance: although the model is named 535B, it is a mixture-of-experts model, so each token activates only about 23B parameters. Before scaling to 535B-A23B, Marin ran a set of smaller scaling experiments to test the recipe, loss behavior, and stability risks at lower cost.

Open Lab: From Releasing Results to Exposing the Process

Marin is led by Stanford CS associate professor and CRFM director Percy Liang and other collaborators. The project’s announcement lists David Hall, Percy Liang, and researchers from Stanford, Open Athena, and the open community. Its central question is whether foundation models can be studied and built collaboratively like open-source software at a time when compute is concentrated and training recipes are increasingly closed.

Marin’s “open lab” workflow includes declaring each experiment’s goals and hypotheses through GitHub Issues, submitting configurations as code and Pull Requests, inviting outside review, and publishing training metrics through W&B. Crucially, successful runs, failed experiments, and mid-course changes are meant to remain part of the public record, while data, code, recipes, and final models continue to be opened.

Marin is not the first project to make large-model training more transparent. BLOOM, Pythia, LLM360, and OLMo have previously released training data, code, logs, intermediate checkpoints, or related artifacts to varying degrees. Marin’s distinguishing feature is that it tries to turn openness from a one-time model release into the default workflow of a research lab: hypotheses, code changes, training progress, and failures are exposed as early as possible.

Percy Liang previously served as chief scientist at Semantic Machines, which was acquired by Microsoft in 2018, and he is also a co-founder of Together AI. Andrew Ng reposted the news and described Marin as a “precious demonstration” of defending openness in AI, noting that the project opens not only model code but also data, training recipes, and the experimental process.

Technical Challenge: Expert Parallelism and Token Dropping

Marin 535B-A23B uses an MoE architecture. According to the public technical notes, each layer keeps two shared experts and activates eight routed experts. Both types of experts use a half-width design, and the routed experts also use 2× compression. The team describes this as roughly one hidden-layer width of neurons from shared experts and two hidden-layer widths from routed experts.

In other words, about one-third of expert compute comes from always-on shared experts. This design is not just about making the parameter count look large; it is intended to reduce the risk of token dropping during MoE training.

Expert Parallelism is one of the major bottlenecks. When tokens are routed to experts distributed across GPUs, the system must perform All-to-All communication: first sending tokens to the selected experts, then returning the computed results to the original path. Sparse computation reduces per-token compute cost, but it introduces engineering challenges around cross-GPU communication, expert load balance, and memory access.

The Marin team disclosed that in earlier experiments, token dropping rose from about 7% at a 4K context length to about 40% at 65K. One reason is that, when the total token batch size is roughly fixed, a longer context means fewer independent sequences per batch, making token distribution less balanced across experts.

For this reason, Marin 535B starts pretraining at a 4K context length instead of immediately targeting very long contexts. Compared with 8K, the same token batch can contain roughly twice as many independent sequences, helping experts receive more even input. A new pooled/wave expert-parallel scheme reduced token dropping to about 3% at 4K, but Marin also states that the implementation is still experimental and may again produce excessive dropping when extended to 65K.

JAX Training Stack and GB200 Cluster Challenges

Marin’s training stack is mainly built on JAX, XLA, and Levanter. Earlier Marin 8B and 32B models mostly ran on Google TPU. Moving the 535B model to NVIDIA GB200 NVL72 systems means the team must address expert parallelism and communication efficiency on GPU clusters.

In its public records, Marin says the team could not find an off-the-shelf expert-parallel solution with sufficient performance in a JAX/XLA GPU environment, so it implemented Expert Parallelism itself. GB200 NVL72 organizes 72 Blackwell GPUs and 36 Grace CPUs within a rack-scale NVLink domain, which is well suited to communication-heavy MoE workloads, but hardware alone does not guarantee a successful run.

Public materials also show that Marin has contingency plans for hardware failures, lower model FLOPs utilization, and schedule delays. If issues appear within the first 25% of the token budget, the team may shorten the final token count and adjust the linear learning-rate decay and data mix rather than rigidly following the original plan.

Scaling Ladder: A Failure Detector Before the Main Run

Before launching the 535B run, Marin built a four-level scaling ladder. The idea is to use smaller MoE models to test whether the training recipe scales reliably and to fit relationships among loss, compute, and model size.

According to public information, the ladder includes:

Marin says the entire scaling exercise uses only about 1% of the final compute budget, while serving several key purposes:

1. Loss trajectory prediction: estimate the expected loss at different stages of the 535B run, so major deviations can flag data, routing, or optimizer issues early
2. Stability screening: earlier scaling experiments found gradient norms rising above 4 as training continued; ablations suggested that under conditions such as high batch size, failing to address the issue could cause training to diverge
3. Anomaly detection: in some smaller models, gradient norms rose during the first 40% of training and then fell as the learning rate decayed, helping the team distinguish normal fluctuation from warning signs

The team ultimately added logit z-loss. By penalizing overly large logit normalization terms, it constrains the numerical scale of the output distribution and reduces instability risks in softmax and routing during training.

Practical Takeaways and Industry Implications

For teams that want to follow now: most research groups cannot reproduce the full 535B run, but they can study Marin’s public expert-parallel implementation, token-dropping monitoring, scaling-ladder design, and failure-handling records. Those lessons may transfer to smaller MoE training runs.

For users waiting on model quality: if the goal is final capability, benchmark comparison, or deployment, it is too early to judge. Pretraining loss mainly measures how well the model fits the data distribution; coding, math, tool use, long-context behavior, and agentic abilities will also depend heavily on data quality, mid-training, and post-training methods.

Final thought: open models have largely addressed who can use a model. Marin pushes on a deeper question: who gets to know how the model was trained? If the run succeeds, the community gains a rare large-scale MoE training case study. If it runs into trouble, the public failure path may be just as valuable for research.