Model Security and API Key Management

You wake up to an email: “Your OpenAI bill this month is $47,000.” You budgeted $500. What happened?

An API key leaked. Someone found it, used it to run their own workloads, and you’re paying for it. This isn’t theoretical—it happens regularly.

LLM API keys are high-value credentials. Leak them, and attackers can rack up enormous bills, exfiltrate data, or poison your systems. Protecting them is critical.

Why API Keys Are Valuable Targets

Direct monetary cost: Using LLM APIs costs money. If an attacker has your key, they can generate unlimited requests and you pay the bill.

Data exfiltration: Your API key might have access to conversation history, user data, or fine-tuned models. Attackers can retrieve and steal this.

Service disruption: Attackers can exhaust your rate limits or quotas, causing your production system to fail.

Reputation damage: If someone uses your key for harmful purposes (generating spam, misinformation, or offensive content), it’s associated with your account.

Treat API keys with the same care as credit card numbers or passwords.

Common Ways Keys Leak

Hardcoded in source code: The developer puts the key directly in a code file. It gets committed to git. Now it’s in version history forever, even if deleted later.

Client-side code: The key is embedded in JavaScript or mobile app code. Anyone can inspect the code and extract it.

Logs: The key is logged (accidentally or intentionally). Logs are often less protected than code repositories.

Environment variables in shared environments: The key is in an environment variable, but the environment is accessible to multiple people or services.

Phishing and social engineering: An attacker tricks someone into revealing the key.

Insecure transmission: The key is sent over unencrypted channels (plain HTTP, unsecured email).

All of these are preventable with proper practices.

Secret Management Basics

Never hardcode secrets: Not in code files, not in configuration files committed to git, not anywhere visible.

Use environment variables: Store keys in environment variables, not in code. This separates configuration from code.

Use secret management systems: Services like AWS Secrets Manager, Azure Key Vault, Google Secret Manager, or HashiCorp Vault store secrets securely and provide access control.

Principle of least privilege: Only grant access to secrets to services and people who absolutely need them.

Rotate keys regularly: Change keys periodically. If a key leaks, rotation limits how long it’s valid.

Keeping Keys Out of Version Control

Never commit keys to git: Check before committing. Use git hooks to prevent accidental commits.

Use .gitignore: Add files containing secrets (like .env) to .gitignore so they’re never committed.

Scrub history: If a key was accidentally committed, scrubbing it from history is hard. You need to rewrite git history and consider the key compromised. Rotate it immediately.

Use git-secrets or similar tools: Automated tools can scan commits for potential secrets and block them.

Environment Variables Done Right

Development: Use .env files (and keep them in .gitignore). Load them with libraries like dotenv.

Production: Don’t use .env files. Use secret management systems or orchestration tools (Kubernetes secrets, AWS ECS task definitions) that inject secrets securely.

Access control: Limit who can access production environments and their secrets.

Client-Side Security

Never put API keys in client-side code: JavaScript, mobile apps, desktop apps—all can be inspected by users.

Use backend proxies: The client calls your backend, which holds the API key and makes the LLM API call. The client never sees the key.

Rate limiting on your backend: Even with a proxy, protect your backend from abuse. Limit how many requests a user can make.

API Key Scoping and Permissions

Many providers let you create keys with limited scopes.

Read-only vs. write: If a service only needs to read data, don’t give it a write-capable key.

Model-specific keys: If you only need access to GPT-3.5, don’t create a key with access to GPT-4 or fine-tuning APIs.

Project-specific keys: Create separate keys for different projects or environments (dev, staging, prod). If one is compromised, it doesn’t affect the others.

Monitoring API Key Usage

Track where and how your keys are used.

Usage dashboards: Most providers offer dashboards showing request counts, costs, and patterns. Check them regularly.

Anomaly detection: Set up alerts for unusual patterns (sudden spike in usage, requests from unexpected IPs, usage at odd times).

Cost alerts: Configure budget alerts. If spending exceeds a threshold, you get notified immediately.

Request logging: Log metadata about every API call (timestamp, endpoint, user). If you suspect misuse, you can investigate.

Key Rotation

Rotate proactively: Even without a known leak, rotate keys every few months. This limits the window of exposure if a key leaks undetected.

Zero-downtime rotation: Support multiple active keys simultaneously. Deploy the new key, verify it works, then retire the old one. This prevents service disruption.

Rotate immediately after incidents: If you suspect a key leaked (accidental commit, employee departure, security breach), rotate it immediately.

Detecting and Responding to Leaks

GitHub scanning: Tools like GitGuardian or TruffleHog scan public repositories for leaked secrets. If your key appears, you’ll be notified.

Provider alerts: Some LLM providers detect unusual usage patterns and alert you to potential leaks.

Manual checks: Periodically search GitHub, Pastebin, and other code-sharing sites for your organization name + “API key.”

When a leak is detected:

1. Rotate the key immediately
2. Audit usage logs to understand the scope of misuse
3. Report to the provider if necessary
4. Investigate how the leak happened and fix the root cause

Multi-Tenant Systems

If you’re building a SaaS product, you might manage keys for multiple customers.

Customer-specific keys: Each customer uses their own LLM provider key (BYOK: Bring Your Own Key). This isolates risk and cost.

Key isolation: If you use a shared key pool, ensure strict logical separation so Customer A can’t use Customer B’s quota or access Customer B’s data.

Billing transparency: Track usage per customer so you can bill appropriately.

Least Privilege for Internal Access

Not everyone on your team needs access to production API keys.

Role-based access: Only production-support engineers have access to production secrets. Developers use separate dev/staging keys.

Audit trails: Log who accessed secrets and when. This helps with compliance and incident response.

Temporary access: If someone needs production access for debugging, grant it temporarily and revoke it afterward.

Secrets in CI/CD Pipelines

Continuous integration and deployment pipelines often need API keys.

Use CI/CD secret management: GitHub Actions, GitLab CI, CircleCI, and other platforms have built-in secret storage.

Scope access tightly: Only specific jobs or workflows should access secrets, not every build.

Avoid echoing secrets: Don’t log secrets in CI output. Even masked logs can be reversed.

Key Compromise Scenarios

Public repository commit: Key is committed and pushed to a public repo. Assume it’s compromised immediately. Rotate.

Employee departure: An employee with access to secrets leaves the company. Rotate keys they had access to.

Third-party breach: A tool or service you use is breached. If it had access to your keys, rotate them.

Suspicious activity: Unexpected usage spikes, requests from unknown IPs, or anomalous behavior. Investigate and rotate if in doubt.

Multi-Factor Authentication for Provider Accounts

Even if API keys are secure, your provider account needs protection.

Enable MFA: Use two-factor authentication on your OpenAI, Anthropic, Google, or other provider accounts.

Strong passwords: Use unique, complex passwords managed by a password manager.

Limit account access: Only a few trusted people should have access to the provider console where keys are created.

What Good Looks Like

A secure API key management system:

• Never commits secrets to version control
• Uses secret management systems (not .env files in production)
• Scopes keys to minimal necessary permissions
• Rotates keys regularly and after incidents
• Monitors usage for anomalies
• Uses backend proxies instead of exposing keys to clients
• Audits access to secrets
• Has incident response procedures for leaks

API key management isn’t glamorous, but it’s fundamental. Get it wrong, and you face financial loss, service disruption, or data breaches. Get it right, and it’s invisible—exactly as it should be.

Build good habits from the start. Use secret managers. Never commit keys. Monitor usage. Rotate regularly. And always assume that, somewhere, someone is trying to steal your credentials.

Paranoia about secrets is healthy. Complacency is expensive.