Using a third-party LLM relay to access Claude, GPT, or Gemini is cheap and convenient—but have you considered this: the relay operator can see every single word you send to the model.

That includes API keys you casually paste in, bank card numbers, login passwords, ID numbers, medical records… all sitting in someone’s server logs, in plaintext.

This isn’t paranoia—it’s architecture. A relay is fundamentally a reverse proxy: your request hits their server, gets unwrapped, forwarded to the upstream provider, and the response comes back through the same path. Your content is completely transparent to the operator. Some relays even keep request logs for billing audits.

Today, I’m going to walk through every viable open-source solution on GitHub, organized into three tiers by privacy strength. All data verified via GitHub API on 2026-08-31.

First, the Risk: What an Audit of 428 Relays Found

“Architecturally possible” only means the operator can misbehave—a security research team turned it into hard numbers. They audited 428 relay stations (28 paid + 400 free):

• 9 stations actively injected malicious code into responses sent back to users;
• 17 stations stole credentials—they exfiltrated AWS honeypot credentials the testers had deliberately planted in prompts;
• 1 station directly drained test crypto assets.

And there’s more:

• Logging your full text takes one config line. The open-source forwarding tool openai-forward documents LOG_CHAT=true dropping complete conversations to disk; the one-api family already logs token usage, timestamps, and IPs by default—switching that to full text is a few lines of code.
• Leaked credentials have an underground supply chain. Keys you paste into prompts get scraped, re-packaged, and resold without your knowledge. Sysdig’s LLMjacking report describes the mirror image: stolen cloud credentials are used to hammer LLMs, then resold through oai-reverse-proxy style relays—the cash-out channel for stolen keys is, precisely, relay stations.
• Exit scams and bait-and-switch are the norm. The “¥44.9/month unlimited” stations vanish within a month (merchant unreachable, support groups disbanded); peak-hour traffic gets dynamically routed to cheaper models—one independent test claimed 45% of stations were fake models. “¥1 = 2.85M tokens” prices below official cost only add up three ways: credit-card-fraud accounts, free-tier abuse, or selling your data.
• The base software has holes too. ~90% of relays are skins on the same open-source shell (one-api and its forks), and one-api issue #2409 documents a second-order SSRF and token leak via its WeChat login config—reported June 2026, still open.

The rest of this post walks three lines of defense: secrets never entering chat, replacing the relay with one you run, and local inference plus confidential computing. Every project mentioned below has been individually verified for stars, license, and last commit.

Tier 1: Secrets Should Never Enter the Chat

Before we talk about gateways, let’s address the root problem: your API keys and passwords should never appear in any prompt—even with a self-hosted gateway, you’re just moving plaintext from a stranger’s server to your own, where logs and context still capture it.

The Right Way to Give Agents Access to Secrets

All tools below are open-source and actively maintained:

The core pattern is one sentence: secrets go through environment variables or process injection, never through chat content. Your agent only references ${OPENAI_API_KEY} placeholders; the actual value gets injected by direnv / sops exec-env / infisical run at process startup.

If you’re using Claude Code, there are two additional layers of defense:

1. PreToolUse hook interception—write a script matching sk-, AKIA, .env paths and block before the agent reads files. This is a deterministic guardrail, far more reliable than writing “please don’t read secrets” in CLAUDE.md.
2. Permission deny rules—in settings, forbid Read access to .env*, ~/.aws/credentials.

If even temporary injection makes you uneasy, there’s the broker school—plaintext only exists in “the one process that uses it, at the moment it uses it”:

• agentsecrets (172★, MIT)—zero-knowledge credential infrastructure for AI agents; the agent operates credentials but never consumes plaintext
• toolhive (2.1k★, Apache-2.0)—enterprise MCP server hosting; MCP runs in isolated containers, secrets injected only into container env
• secretless-broker (387★, Apache-2.0)—the classic “apps never hold secrets” pattern: the app talks to the broker, which pulls real credentials from vault and injects them at the protocol layer

One red line to remember: prompt injection can make even the most obedient agent leak secrets (demonstrated in empirical research as of April 2026). Input-side hook interception, output-side redaction, and broker-style “never enter context” are layered, not redundant—any single layer can be bypassed.

What About Bank Cards and Passwords?

They should never go to any LLM. This isn’t just my recommendation—it’s the consensus from OpenAI and Anthropic’s official documentation:

• OpenAI Safety Best Practices: explicitly recommends constraining user input; all API content enters abuse-monitoring logs retained ~30 days
• Anthropic Claude Code Security: credentials should go through credential stores/permission boundaries, not chat content

The correct approach is to let agents operate indirectly: payments go through Stripe SDK’s tokenized interfaces (model sees tokens, not card numbers), passwords go through Bitwarden/1Password CLI references instead of plaintext.

Tier 2: Self-Hosted Gateways—Be Your Own Relay Operator

This is the main event. Instead of handing plaintext to a stranger’s relay, run one on your own VPS—you’re the only operator, upstream official keys live only in your own database.

The Transitional Option: Local Redaction Proxies (when you’re not ready to self-host)

A gateway takes maintenance. If you’re still using an existing relay for now, the minimum bar is making sure requests get sanitized before they leave your machine. These “redaction proxies” sit between your agent and the upstream API, replacing keys, card numbers, and ID numbers with placeholders before forwarding:

GitGuardian’s gateway writeup shows the minimal version of the same pattern: detect a database password in the request body, rewrite it as REDACTED, then forward. Claude Code users have dedicated projects too—claude-code-redaction-hooks (20★, Apache-2.0) intercepts secrets and PII before tool input enters context; cc-redact and claude-code-redact take similar approaches (the latter preserves type/structure so the model can still reason about the value). To be honest: these are double-digit-star, very-early-stage projects—right direction, audit them yourself before production use.

This class of tool catches “accidentally pasted” secrets. It cannot stop an agent from paraphrasing them out loud—so it stacks with, never replaces, Tier 1 secret injection and the self-hosted gateway below.

Recommended Option A: LiteLLM Proxy + Presidio (Most Complete)

For: those who want a one-stop solution and don’t mind configuring a few containers.

• LiteLLM (57.6k stars, MIT)—AI gateway covering 100+ providers, with virtual keys, budgets, rate limiting
• Presidio (10.7k stars, MIT)—Microsoft’s PII detection and anonymization framework, recognizes credit cards (with Luhn validation), emails, phone numbers, API keys, and 30+ entity types

LiteLLM Proxy has built-in Presidio integration that can automatically strip or mask PII before requests leave:

1
2
3

agent → LiteLLM Proxy (Presidio PII masking)
         ↓ sanitized request
      upstream LLM API (official or relay)

Presidio supports MASK (replace with placeholder) and BLOCK (reject outright) modes, with configurable confidence thresholds. The downside: Chinese PII detection requires additional configuration (English works best by default), and the component chain is the longest—LiteLLM + Presidio analyzer + Presidio anonymizer + Postgres.

Recommended Option B: new-api (Simplest)

For: those who just want to quickly replace a third-party relay without needing PII masking.

• new-api (46.8k stars, AGPL-3.0)—actively maintained fork of one-api

One Docker command, comes with a web admin UI, supports OpenAI/Claude/Gemini format conversion. Upstream official keys live only in your own SQLite/MySQL; request logging can be disabled.

It has no built-in PII masking, but you can add a simple regex middleware layer in front (e.g., matching sk- prefixes, credit card number formats).

Note: new-api is AGPL-3.0 licensed. Fine for personal use, but the copyleft clause applies if you offer it as a service to others.

Two self-hosting reminders: first, new-api disclosed CVE-2026-25802 (High 7.6) in 2026-02—self-hosters must keep it updated and never expose the admin panel to the public internet; second, one-api itself has been essentially dormant since 2026-01 (36.7k★)—new deployments should go straight to the actively maintained new-api fork.

Others Worth Watching

Tier 3: Local Inference & Confidential Compute

For: those with extreme privacy requirements and a decent GPU.

If your content is so sensitive that you don’t even trust your own VPS provider, the only thorough solution is—run models on your own hardware, data never leaves the network.

Ollama (179.8k stars, MIT)

The most popular local model runner. One command to start, with OpenAI-compatible API:

1

ollama run deepseek-r1:32b   # or qwen3:30b

Local models in 2026 are quite capable:

• 24GB VRAM smoothly runs 32B dense or 30B MoE models (deepseek-r1:32b, qwen3:30b)—perfectly adequate for everyday sensitive tasks (code analysis, document rewriting, debugging logs containing secrets)
• 671B/235B full-size models still require server-grade hardware—don’t bother if you’re a regular user

Privacy killer feature: zero egress. Model, prompts, responses all stay on your machine. No third party—including your VPS provider—sees plaintext.

The tradeoff: capability ceiling still trails frontier closed models; top-tier reasoning and long-chain planning will show the gap.

LocalAI (48.8k stars, MIT)

Similar to Ollama—also a pure local OpenAI-compatible inference engine—but supports broader modalities (vision/speech/image/video) and can run without a GPU (just slower). Good for lightweight models when you don’t have a graphics card.

Want Cloud-Grade Intelligence and Opacity: TEE Confidential Inference

If you want frontier-model capability while keeping every intermediary blind to plaintext, 2026’s answer is confidential computing (TEE): inference runs inside the GPU’s trusted execution environment; clients verify remote attestation first, then send prompts over an encrypted channel—cloud providers and relays only ever see ciphertext. This route has few surprises but plenty of pitfalls, sorted by maturity:

• dstack (538★, Apache-2.0, active 2026-08)—a confidential-AI framework that demands attestation up front; it’s the foundation of Phala Cloud and is production-usable. Its companion private-ai-gateway (30★) provides an attested private inference gateway, still experimental.
• Tinfoil (components AGPL-3.0/Apache-2.0)—confidential VM images plus client-side attestation verification; already operating in public production, with Red Hat co-building a confidential-AI reference architecture. Self-hosting means assembling the components yourself.
• Edgeless Privatemode (public measurements & artifacts)—core proxy is closed-source; trust rests on published measurements rather than auditable code. Its VS Code extension demos Claude Code doing end-to-end encrypted coding through the proxy.
• The avoid list: NEAR private-ml-sdk has been archived (dormant since 2025-10); on the fully-homomorphic-encryption front, concrete-ml (1.5k★) only handles small models and remains orders of magnitude away from real LLMs—anyone claiming “FHE runs big models” owes you a benchmark first.

The Practical Middle Ground: Sensitive Goes Local, Ordinary Goes to the Relay (Sensitivity Routing)

All-local caps your capability; all-TEE has a steep entry cost. The middle path is making sensitivity routing the default: vLLM semantic-router (5.4k★, Apache-2.0, active) ships PII/jailbreak classifiers—requests carrying sensitive entities get pinned to a local model, and only benign requests are released to external APIs. LiteLLM can build the same with an ollama backend plus pre-call guardrail redaction. Avoid the traps: RouteLLM has been dormant since 2024-08 and is privacy-oblivious; the TensorZero repo has been officially archived—skip both for new projects.

My Recommendation: Layered Combination

In practice, there’s no silver bullet. Use different solutions for different scenarios:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

┌─────────────────────────────────────────────────┐
│  High-sensitivity (keys, card numbers,           │
│  passwords, medical records, financial data)     │
│  → Local Ollama, zero egress                     │
├─────────────────────────────────────────────────┤
│  General content (coding, translation, Q&A)      │
│  → Self-hosted gateway (new-api or               │
│    LiteLLM+Presidio)                             │
│    Upstream via official API or trusted relay    │
├─────────────────────────────────────────────────┤
│  Secrets management (never do this:              │
│  paste into chat)                                │
│  → direnv/sops/Infisical env injection           │
│  → PreToolUse hook + gitleaks safety net         │
└─────────────────────────────────────────────────┘

Bottom line: your prompt is your privacy. Either keep it on your machine, or pass it through a sanitization gateway you control. At the absolute minimum—stop pasting API keys and card numbers into chat.

Do These Five Things—Starting Today

1. Tonight: no API key, bank card number, or password goes into any chat again (official models included).
2. A 15-minute guardrail: in Claude Code, add permissions.deny rules blocking reads of .env and credential files, plus a PreToolUse hook regex-matching sk-, AKIA, and card-number formats.
3. This week: if you want ZERO maintenance, run new-api (docker compose up -d)—upstream official keys live only in your own database, and your agents point at 127.0.0.1 or your VPS from then on. Keep it updated and never expose the admin panel publicly.
4. Ongoing: use api-relay-audit (819★, AGPL-3.0, active) to spot-check the relays you still use—it has dedicated checks for prompt injection, model swapping, and credential theft; worth rolling into routine audits.
5. Escalate as needed: sensitive tasks go to local ollama; cloud-intelligence purists can look at dstack/Phala-class TEE services.

None of these layers are hard to build. The hard part is realizing you’re running naked. After reading this, at least don’t hand your bank password to a ¥44.9/month station that may be gone next month.

Data Notes

All 40+ repos’ star counts, licenses, and maintenance status were verified via GitHub API on 2026-08-31; the three audit counts for the 428-relay study come from the public report of the tests. Presidio has migrated to data-privacy-stack/presidio (old URL auto-redirects). llm-guard has been archived by protectai (no commits since 2026-07, verified) and is no longer recommended. Lakera was acquired by Check Point—no open-source product available.