The core bet: GPUs still have headroom

French startup Kog is challenging a growing assumption in AI infrastructure: that conventional GPUs are poorly suited to the next wave of agentic AI workloads. Instead of starting from new purpose-built silicon, Kog is trying to extract more inference performance from the datacenter GPUs companies already own.

That positioning lands in a market where inference speed and cost have become central constraints. Inference is the phase in which a trained model generates an answer, and for end users it directly affects latency, throughput, and ultimately the price of an AI service. While Cerebras received a warm market reception in May with its IPO and its AI-focused chips, Kog is taking a software-first route.

The company drew attention in May with a technical preview that reached the front page of Hacker News. Its stated goal was to show that very fast single-request decoding could be achieved on standard enterprise GPUs, including AMD MI300X and Nvidia H200 hardware used in its demonstration.

What Kog has shown so far

Kog’s most visible benchmark is an inference demo reaching roughly 3,000 tokens per second per request. A token is the unit of text a language model processes and generates; higher token speed generally means faster output. The important caveat is that the demo used a purpose-built small model, Laneformer 2B, with around 2 billion parameters. Laneformer 2B has since been open sourced.

The company’s larger promise is much more ambitious: up to 30x faster LLM inference. That remains a proof point Kog still has to establish on larger models, where inference is harder because of model size, memory movement, and bandwidth demands.

Key facts from the company’s current position include:

• The demo ran on standard datacenter GPUs such as AMD MI300X and Nvidia H200.
• It reached about 3,000 per-request TPS in the preview.
• The demonstrated model was the roughly 2B-parameter Laneformer 2B.
• Kog currently has a team of 11 people.
• CEO Gaël Delalleau said the company aims to implement its first major model at 10x speed in September, then use that to demonstrate customer traction and raise a Series A.

According to Delalleau, the preview generated 200 tangible business leads. Early demand appears strongest in software engineering workflows, where users of coding agents sometimes wait a long time for complex tasks to complete. Anthropic’s Claude Fast Mode also shows that users are willing to pay a premium for speed.

Why Kog is moving toward larger models

Kog’s early market conversations pushed the company away from focusing mainly on small fine-tuned models. Fine-tuning means adapting an existing model with additional data for a more specific use case, but many prospective customers were not ready to invest in that process. Delalleau said this is why the company has shifted its attention toward accelerating larger models that better match current demand.

That shift raises the difficulty level. Large language model decoding is the step-by-step generation of the next token, and it often cannot be parallelized as cleanly as training. Skeptics argue this makes GPUs less ideal for such workloads. Delalleau’s view is the opposite: modern GPUs have increasing memory bandwidth, and the problem is unlocking it efficiently.

Kog is not alone in believing software can make existing GPUs do more. Another French startup, ZML, has released hardware-agnostic software that bypasses Nvidia CUDA to support fast inference across competing chips. CUDA is Nvidia’s widely used parallel computing platform. Delalleau describes Kog as closer to Stanford’s Hazy Research in spirit, with a deeper emphasis on low-level GPU acceleration rather than only high-level framework support.

A low-level engineering culture

Delalleau’s background helps explain Kog’s approach. He studied solid-state physics at École Polytechnique in France and later worked in offensive cybersecurity, or white-hat hacking. He has also been a four-time finalist at DEFCON’s CTF tournament. He says physics shaped a mindset of understanding the “laws” of the GPU, while hacking taught him to reverse-engineer systems down to assembly language and binary code.

That hands-on method can reveal performance opportunities hidden beneath standard software layers. But it is costly in time and engineering effort. Delalleau said that for each new GPU, Kog may spend several weeks or even months studying the hardware in detail and conducting GPU engineering research. With only 11 people, the number of chips the company can support in the near term is naturally limited.

Over time, Kog hopes to put its methodology into agent-based pipelines so it can support more chips and models. The company may also benefit from Europe’s push to build stronger domestic capabilities in AI models and compute. Kog is supported by Scaleway and backed by France’s Bpifrance and the French Tech 2030 program. Its seed round was co-led by Varsity VC, whose Kamel Zeroual previously co-founded Stribe with Delalleau.

Why it matters

Kog’s thesis matters because the AI infrastructure debate is not only about new chips. If software can materially improve inference on GPUs already installed in enterprise datacenters, the economic impact could be immediate: faster responses, better utilization, and potentially lower serving costs.

The open question is whether Kog can move from an impressive small-model demonstration to credible results on major LLMs. A 10x acceleration on a major model would make the company’s argument far more concrete for customers and investors. Until then, its 30x claim remains an ambition rather than a proven market outcome. Still, Kog reflects a broader trend worth watching: the next phase of inference competition may be fought as much in low-level GPU software as in new AI hardware.