Extracting Large Model Circuits Directly from Weights

IQuest Research, in collaboration with Safe AI Forum, the University of Oxford, Stanford University, and Tsinghua University, proposes Sparse Weight Decomposition (SWD) to bypass the traditional approach of “training a substitute network to understand a model,” instead extracting intervenable task circuits directly from pretrained weights.

In mechanistic interpretability research, a task circuit refers to a set of internal computations that have a causal effect on a model’s ability: the ability persists when only they are retained, and performance degrades when they are removed. Prior methods such as Transcoder and sparse feature modules typically require learning new representations to approximate a given layer or module of the original model, then performing attribution and ablation on the new units. These methods are effective but demand additional data and optimization, and may conflate explanations of the original model’s behavior with errors from the substitute module.

How SWD Constructs “Sparse Paths”

The core of SWD is to approximate a dense weight matrix W as the product of two sparse matrices A and B, i.e. W ≈ AB. The shared intermediate dimension becomes the bottleneck unit: a unit first reads a scalar from the input direction and then writes toward the output direction, effectively forming a rank-one read-write path. A rank-one update can be understood as the minimal matrix formed by one input direction and one output direction.

The method uses a small amount of calibration text to generate layer inputs and optimizes the output error before and after decomposition. During solving, the two factors are updated alternately, with hard thresholding controlling the number of nonzero connections and refitting the retained weights. After decomposition, researchers can score, rank, select, and ablate bottleneck units without needing to train a separate neural surrogate network. Unlike SVD, whose components typically have dense read-write directions, SWD emphasizes that connections within each unit must also be sparse, so retaining only a few units still significantly reduces the number of actually active connections.

Key Results: Less Than 1% Data, Scaling to 27B

The paper reports three core findings:

• In single-matrix replacement fidelity-matching experiments, SWD uses less than 1% of the data required by training-based baselines such as Transcoder.
• In task circuit experiments on GPT-2, Qwen2.5, and Qwen3.5-27B, SWD typically achieves the same sufficiency and necessity targets with fewer bottleneck units and active connections.
• The method scales to Qwen3.5-27B, extends across all 48 attention and MLP weight matrices in GPT-2 Small, and includes a zero-data variant.

The team measures whether the replaced model remains close to the original using held-out text cross-entropy difference (CE delta), supplemented by KL divergence and activation reconstruction error. In the GPT-2 Small layer 8 mlp.c_proj experiment, SWD enters the low-error regime with only thousands of calibration tokens, whereas training-based baselines require roughly 10⁶ optimizer-replay tokens. A similar trend appears across Qwen2.5-0.5B, 1.5B, 3B, and Qwen3.5-27B.

From Single Matrices to Full Models and Semantic Units

For circuit extraction, the team first ranks candidate units using first-order task attribution, then constructs nested top-k circuits and evaluates sufficiency and necessity on held-out data. On GPT-2 Small, across four tasks—GreaterThan, IOI, Docstring, and Gendered Pronoun—SWD consistently uses fewer units and connections than Transcoder and VPD-Recon-CI to reach the same targets; this advantage persists even under zero ablation and extends to the Qwen series of models.

In the full-model experiment, researchers replace the 48 attention and MLP weight matrices across all 12 Transformer blocks of GPT-2 Small with SWD decompositions, while retaining the embedding, LayerNorm, nonlinearities, and output head. Since errors accumulate across layers, they fix the sparse structure and only fine-tune the retained factor values, yielding SWD-FT: with the number of connections held constant, the model CE drops from 3.90 to 3.44, slightly outperforming a sparsity-pretrained baseline of 3.45 under a comparable nonzero parameter budget; the former uses approximately 20.6 million tokens, while the latter requires 28.84 billion.

The paper also presents zero-data SWD: without calibration text, it directly minimizes the Frobenius error in weight space. Results show that it more closely approximates the original matrix in weight space, while the activation-calibrated variant better preserves typical text behavior at high sparsity levels. Even without calibrated activations, the zero-data version can still extract meaningful task circuits.

Industry Implications and Future Directions

SWD’s value lies not only in reducing training data but in shifting the object of interpretability research from “another learned surrogate module” back to the original checkpoint. It provides sparse weight paths that are scoreable, ablative, and editable; in the GreaterThan task, it surfaces highly attributed units related to numbers, years, and quantities. A single bottleneck direction was also used to selectively boost the logit margin of “down” relative to “left” in the prompt “The opposite of up is,” with negligible KL divergence on unrelated prompts.

For a general technical audience, this suggests that mechanistic studies of large models may be moving closer to engineering practice: there is no longer a need to retrain complex surrogate representations for every layer and every checkpoint. Whether this approach becomes mainstream will depend on its stability across more models, tasks, and rigorous safety scenarios. But based on the paper’s results, directly searching for sparse causal circuits within weights is emerging as a significant new branch for understanding and manipulating the internal computation of large models.