Core Event: Jalapeño Benchmarks Revealed, Marking Strategic Divergence in Inference Chip Design

OpenAI has publicly revealed benchmark results for its proprietary inference chip Jalapeño, with metrics focused on Token throughput, generation latency, and power efficiency per watt. The chip features a rated power of 700W with sustained operation under 550W during testing.

Key facts:

• Chip name: Jalapeño (continuing OpenAI’s spicy chip naming convention)
• Tested models: GPT-OSS 120B, DeepSeek R1 670B, Kimi K2.5 1T
• Performance gains: 1.5–1.9× improvement in tokens per watt; 1.7–3.6× reduction in end-to-end latency; up to 2.1–4.1× improvement in interactive scenarios
• Status: Still in production qualification and pre-deployment phase; results achieved without speculative decoding

Crucially, these benchmarks cover both Prefill and Decode phases, underscoring a key industry shift in how inference workloads are understood and optimized.

Technical Foundation: Token Cost Replaces FLOPS as the New Benchmark

Post-deployment, AI chips now serve continuous online inference—ChatGPT responding iteratively, Reasoning models generating long chains, and Agents orchestrating tool calls. This creates two distinct computational patterns:

• Prefill phase: Input Prompt (e.g., 8K tokens) enables massive parallelization; weights are reused across tokens, yielding high arithmetic intensity, limited by compute units.
• Decode phase: Tokens generated sequentially, dependent on KV Cache; low parallelism but frequent weight and cache access, making memory bandwidth the bottleneck.

The Roofline model clarifies this tradeoff: in low-batch decoding, arithmetic intensity is roughly 2/b FLOPs/Byte (where b is bytes per parameter). Increasing model size does not alleviate Decode bottlenecks. Jalapeño’s architecture addresses this by explicitly positioning KV Cache, preserving data locality, and coordinating compute-memory-network subsystems, using a homogeneous design capable of handling both Prefill and Decode.

Diverging Paths Across Major Vendors

Jalapeño is not alone—AI inference hardware is undergoing structural segmentation:

NVIDIA’s approach: Integrates Groq 3 LPU within Vera Rubin to form a specialist division. GPUs handle Prefill (leveraging their strength in large, parallel matrix math), while LPUs manage Decode (prioritizing low latency). Each LPX rack contains 256 LPUs with only 128 GB SRAM (vs. GPU HBM), yet achieves 40 PB/s aggregated SRAM bandwidth; LPU uses deterministic execution, with software pre-scheduling to skip traditional hardware flow control. A key insight: when only aggregate throughput matters and higher latency is acceptable, Rubin GPU remains efficient; but as single-user latency demands rise, LPX gains—but adding more LPUs eventually reduces overall throughput efficiency.

Google’s approach: Directly splits TPU 8 into TPU 8t (training) and 8i (inference) at the silicon level. Hardware details confirm TPU 8i has 8 HBM groups (vs. 6 on 8t), more SRAM capacity, and supports BoardFly networking (max 7 hops vs. 3D Torus’s 16), plus a Collective Acceleration Engine to bypass chip-internal data movement for collectives.

Common thread: None pursuit a single universal chip. Instead, each adjusts resource allocation based on workload characteristics—推理 chips favor HBM bandwidth, SRAM capacity, low-latency interconnects, and KV Cache management; training chips prioritize compute and large-scale connectivity.

Buyer Recommendations: Match Hardware to Workload Profile

Evaluate Jalapeño now if your use case involves:

• Latency-sensitive services (e.g., real-time agent interactions)
• Datacenters prioritizing tokens-per-watt (Tokens/kW) efficiency
• Medium-to-long context deployments (128K+ tokens) without exotic异构 infrastructure

Wait or observe further if your use case involves:

• Online batch-heavy processing where high aggregate throughput matters more than per-token latency, where Rubin GPU remains cost-effective
• Existing Groq or TPU ecosystems where migration overhead may outweigh marginal gains
• AutoML workflows尚未 validated on KV Cache movement overhead

Final Thoughts

Jalapeño’s benchmark data confirms the industry consensus: the inference bottleneck has shifted from compute to data delivery. As FLOPS no longer accurately reflects online service efficiency, metrics like Tokens/s/user, TTFT (time-to-first-token), and TBT (token-interval throughput) are gaining prominence. The infrastructure split is still evolving, but the direction is clear—the next AI chip race will be won by those who can most precisely disaggregate and optimize the end-to-end Token lifecycle cost.