Real-SWE Benchmarks AI on Private Enterprise Codebases

Core Announcement and Key Details

Real-SWE is a benchmark for evaluating frontier AI models on private, real-world enterprise production codebases. Its tasks are drawn from licensed codebases of actual companies, focusing on the problems engineers encounter within existing products and their surrounding context.

The codebases and their solutions are not publicly available on the internet. The benchmark uses native harnesses intended to reflect enterprise engineering practice, evaluating model-and-harness combinations rather than models in isolation.

Three Characteristics of Real Enterprise Tasks

Real-SWE highlights three features common to enterprise software work:

• Private code and context: Agents must investigate proprietary systems and understand their architecture without relying on publicly available answers.
• Direct business consequences: Tasks can involve billing, tax calculations, and customer migrations—work that can affect business operations.
• Company-specific engineering complexity: Agents must make correct changes while following existing coding conventions, business rules, and multi-service workflows.

The source argues that much of the code and context inside real enterprises is outside the data available to frontier models. The central question is therefore not only whether a model can generate working code, but whether it can understand a particular organization’s engineering patterns and implicit constraints.

How a Tax Fix Can Span Multiple Systems

One example task involves fixing invoice tax calculations. An agent must account for different business tax configurations: some businesses maintain their own rates, some price invoices through a tax authority based on the buyer’s destination, and some collect no tax. Customers with recorded exemptions should not be taxed.

The task also requires handling addresses, line items, and product categories when calling a tax service in either a sandbox or production environment. If the authority rejects an address, the issue must be reported without stopping invoice issuance. Once an invoice is settled, the sale must be filed back under that invoice number so that returns can reconcile. The environment spans a tax service, a ledger, a NestJS service, and TypeScript code, among other components.

Real-SWE task environments expose the tools and services needed for each workflow. These can include an AWS emulator, Docker, Kubernetes, code-hosting and project-management tools, and databases such as PostgreSQL, MySQL, MongoDB, and Redis. Projects may use stacks including Go, Python, and Node.js.

Short and Long Rollouts Fail at Similar Rates

The source reports that 71.4% of rollouts under 10 minutes failed, compared with 73.4% of longer rollouts. The small difference suggests that simply allowing more execution time does not necessarily address the underlying difficulty.

The harder problem is that agents must triage requirements in complex existing codebases, identify cross-system dependencies, understand business logic and company-specific coding patterns, and verify their assumptions. For enterprise adoption, this suggests that coding agents should be evaluated not only on one-off code generation, but also on their ability to gather context, follow constraints, and validate changes.

Conclusion

Real-SWE shifts attention toward real production codebases and operationally meaningful tasks. It suggests that current AI coding agents still face substantial challenges when working with internal business rules, established architectures, and cross-service enterprise workflows.