LLM Provider API Key Management

Setting up secure key storage for customers who need a key management process before configuring the AIRS AI Gateway

AI Gateway Deployment Guide AI Gateway TRD

API Key Storage

LLM provider API keys are a hard requirement for AI Gateway

The AIRS AI Gateway must have credentials for each LLM provider it routes to. Without valid API keys, the Gateway cannot forward requests. How you store and manage those keys is your choice.

If your organization already has a secrets management process (HashiCorp Vault, AWS Secrets Manager, CyberArk, or another Key Vault), use it. This guide is for customers who do not yet have a process in place and need to set one up before configuring the AI Gateway.

This guide uses Azure Key Vault with Entra ID because it is the most common choice for customers already running in Azure, but the principles apply to any secrets manager: store one credential per provider, restrict read access to the Gateway's identity, and set up rotation alerts before keys expire.

Azure OpenAI is different from other providers

Azure OpenAI supports Entra ID authentication natively; no API key is needed at all. Instead of storing a key, you assign an Entra role to the AI Gateway's identity on the Azure OpenAI resource. This guide covers both approaches: keyless Entra auth for Azure OpenAI and Key Vault-backed keys for all other providers.

Prerequisites

Already have a secrets manager? Skip this guide.

If your organization uses HashiCorp Vault, AWS Secrets Manager, CyberArk, or any other secrets management platform, store your LLM provider keys there using your existing process. This guide only applies to customers setting up key management for the first time using Azure Key Vault.

To follow this guide you will need:

Azure subscription with Contributor access (for resource creation)
Azure CLI installed and signed in: az login
Entra ID permissions to create Service Principals or Managed Identities
API keys already obtained from each LLM provider you plan to use
A resource group created or identified for the Key Vault

Collect provider API keys before starting

Have your keys ready before creating the vault. You will need: an OpenAI API key, AWS IAM access key ID and secret, Anthropic API key, and/or a Google Cloud service account JSON. Azure OpenAI requires no key: only the endpoint URL and deployment names.

Step 1 — Create the Key Vault

Create a dedicated Key Vault for AI Gateway credentials. Using a dedicated vault isolates LLM keys from other secrets and makes it easy to scope access precisely.

▶1.1 — Set variables

Set shell variables to keep the remaining commands consistent. Adjust to match your environment.

SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
RESOURCE_GROUP="rg-ai-gateway"
LOCATION="eastus"
VAULT_NAME="kv-ai-gateway-llm"   # must be globally unique, 3–24 chars

Vault name uniqueness

Key Vault names are globally unique across all Azure tenants. If the name is taken, append your org abbreviation or a short random suffix (for example, kv-ai-gateway-llm-contoso).

▶1.2 — Create the vault with RBAC access model

Create the Key Vault with RBAC-based access control (recommended over the legacy access policy model) and enable soft-delete and purge protection.

az keyvault create \
  --name "$VAULT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --location "$LOCATION" \
  --enable-rbac-authorization true \
  --enable-soft-delete true \
  --retention-days 90 \
  --enable-purge-protection true

Verify: the command returns a JSON object with "provisioningState": "Succeeded". Note the vault URI, which you will need later:

VAULT_URI="https://${VAULT_NAME}.vault.azure.net/"
echo "Vault URI: $VAULT_URI"

▶1.3 — Grant yourself admin access

Assign yourself the Key Vault Administrator role so you can create and manage secrets during setup. This is a one-time step for the person performing the initial configuration.

MY_OBJECT_ID=$(az ad signed-in-user show --query id -o tsv)
VAULT_ID=$(az keyvault show --name "$VAULT_NAME" --resource-group "$RESOURCE_GROUP" --query id -o tsv)

az role assignment create \
  --role "Key Vault Administrator" \
  --assignee "$MY_OBJECT_ID" \
  --scope "$VAULT_ID"

Verify: role assignment appears in the Azure Portal under the Key Vault's Access Control (IAM) blade, or run:

az role assignment list --scope "$VAULT_ID" --query "[].{Role:roleDefinitionName, Principal:principalName}" -o table

Step 2 — Store Provider API Keys

Store one secret per provider credential using a consistent naming convention. Set an expiry date on every secret so Key Vault can alert you before keys expire.

Naming convention

Use the format llm-<provider>-<credential-type>. This makes secrets scannable and predictable for the AI Gateway configuration. See the reference table at the bottom of this guide for the full list.

▶2A — OpenAI

Store the OpenAI API key. Set an expiry 90 days out. OpenAI keys do not auto-expire, so the vault expiry is your rotation trigger.

EXPIRY=$(date -u -d "+90 days" +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -v+90d +"%Y-%m-%dT%H:%M:%SZ")

az keyvault secret set \
  --vault-name "$VAULT_NAME" \
  --name "llm-openai-api-key" \
  --value "your-openai-api-key" \
  --expires "$EXPIRY" \
  --description "OpenAI API key for AIRS AI Gateway"

Replace the placeholder

Replace sk-proj-... with your actual OpenAI API key. Avoid pasting keys into shell history: use read -s OPENAI_KEY to enter it interactively, then pass --value "$OPENAI_KEY".

Verify:

az keyvault secret show --vault-name "$VAULT_NAME" --name "llm-openai-api-key" --query "{name:name, expires:attributes.expires}" -o table

▶2B — Azure OpenAI (keyless Entra auth)

No API key needed for Azure OpenAI

Azure OpenAI supports Entra ID authentication natively. Instead of storing an API key, you assign the AI Gateway's identity the Cognitive Services OpenAI User role on the Azure OpenAI resource. This is more secure than key-based auth and eliminates a rotation burden.

Store the endpoint URL and deployment names as secrets so they are centrally managed alongside other provider config:

az keyvault secret set \
  --vault-name "$VAULT_NAME" \
  --name "llm-azure-openai-endpoint" \
  --value "https://my-instance.openai.azure.com/" \
  --description "Azure OpenAI endpoint URL"

az keyvault secret set \
  --vault-name "$VAULT_NAME" \
  --name "llm-azure-openai-deployment-gpt4o" \
  --value "gpt-4o" \
  --description "Azure OpenAI deployment name for GPT-4o"

The role assignment that replaces the API key is done in Step 4 (Grant Access). No expiry is needed on endpoint/deployment name secrets; they change only when you update the Azure OpenAI resource.

Verify the secrets were stored:

az keyvault secret list --vault-name "$VAULT_NAME" --query "[?starts_with(name,'llm-azure-openai')].name" -o table

▶2C — AWS Bedrock

AWS Bedrock requires an IAM access key pair and a region. Store all three as separate secrets. Create a dedicated IAM user in AWS with only bedrock:InvokeModel permissions. Do not reuse admin credentials.

az keyvault secret set \
  --vault-name "$VAULT_NAME" \
  --name "llm-aws-bedrock-access-key-id" \
  --value "YOUR_AWS_ACCESS_KEY_ID" \
  --expires "$EXPIRY" \
  --description "AWS IAM access key ID for Bedrock"

az keyvault secret set \
  --vault-name "$VAULT_NAME" \
  --name "llm-aws-bedrock-secret-access-key" \
  --value "YOUR_AWS_SECRET_ACCESS_KEY" \
  --expires "$EXPIRY" \
  --description "AWS IAM secret access key for Bedrock"

az keyvault secret set \
  --vault-name "$VAULT_NAME" \
  --name "llm-aws-bedrock-region" \
  --value "us-east-1" \
  --description "AWS region for Bedrock inference"

Least-privilege IAM policy

The AWS IAM user should have only the permissions required. Example policy:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "bedrock:InvokeModel",
      "bedrock:InvokeModelWithResponseStream"
    ],
    "Resource": "arn:aws:bedrock:us-east-1::foundation-model/*"
  }]
}

Verify:

az keyvault secret list --vault-name "$VAULT_NAME" --query "[?starts_with(name,'llm-aws-bedrock')].name" -o table

▶2D — Anthropic

Store the Anthropic API key. Anthropic keys are generated in the Anthropic Console and are not auto-expiring; use the vault expiry as your rotation signal.

az keyvault secret set \
  --vault-name "$VAULT_NAME" \
  --name "llm-anthropic-api-key" \
  --value "your-anthropic-api-key" \
  --expires "$EXPIRY" \
  --description "Anthropic API key for AIRS AI Gateway"

Verify:

az keyvault secret show --vault-name "$VAULT_NAME" --name "llm-anthropic-api-key" --query "{name:name, expires:attributes.expires}" -o table

▶2E — Google Vertex AI

Google Vertex AI authenticates via a GCP service account. Create a dedicated service account in GCP with only aiplatform.endpoints.predict permission, download its JSON key file, and store it as a single secret. Also store the GCP project ID separately.

# Store the full service account JSON as a single secret
SA_JSON=$(cat path/to/service-account-key.json)

az keyvault secret set \
  --vault-name "$VAULT_NAME" \
  --name "llm-google-vertex-sa-key" \
  --value "$SA_JSON" \
  --expires "$EXPIRY" \
  --description "GCP service account JSON for Vertex AI"

az keyvault secret set \
  --vault-name "$VAULT_NAME" \
  --name "llm-google-vertex-project-id" \
  --value "my-gcp-project-id" \
  --description "GCP project ID for Vertex AI inference"

Delete the local JSON file after storing

Once the key is stored in Key Vault, delete the local .json file. It is a high-value credential and should not persist on disk or in source control.

Verify:

az keyvault secret list --vault-name "$VAULT_NAME" --query "[?starts_with(name,'llm-google-vertex')].name" -o table

Step 3 — Create an Identity for the AI Gateway

The AI Gateway needs an Entra ID identity to authenticate to Key Vault at runtime. Use a User-Assigned Managed Identity if the Gateway runs on Azure (no credentials to manage). Use a Service Principal if the Gateway runs outside Azure.

▶3A — User-Assigned Managed Identity (Azure-hosted, recommended)

Managed Identities are the preferred option for workloads running on Azure VMs, App Service, AKS, or Container Instances. There are no credentials to rotate; Azure handles authentication automatically.

az identity create \
  --name "id-ai-gateway" \
  --resource-group "$RESOURCE_GROUP" \
  --location "$LOCATION"

# Capture the identity details for later steps
MI_CLIENT_ID=$(az identity show --name "id-ai-gateway" --resource-group "$RESOURCE_GROUP" --query clientId -o tsv)
MI_OBJECT_ID=$(az identity show --name "id-ai-gateway" --resource-group "$RESOURCE_GROUP" --query principalId -o tsv)

echo "Client ID: $MI_CLIENT_ID"
echo "Object ID: $MI_OBJECT_ID"

Assign the managed identity to the Azure resource running the AI Gateway (example for a VM):

az vm identity assign \
  --resource-group "$RESOURCE_GROUP" \
  --name "vm-ai-gateway" \
  --identities "/subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCE_GROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/id-ai-gateway"

Verify: the identity appears in the VM's Identity blade in the Azure Portal, or:

az vm identity show --resource-group "$RESOURCE_GROUP" --name "vm-ai-gateway" -o table

▶3B — Service Principal (non-Azure or on-premises)

If the AI Gateway runs outside Azure, create a Service Principal and store its credentials securely. The SP authenticates to Entra ID to obtain a token, which is then used to access Key Vault.

az ad sp create-for-rbac \
  --name "sp-ai-gateway" \
  --skip-assignment

# Output includes appId (client ID), password (client secret), and tenant
# Store these values securely — the password is shown only once

Store the Service Principal secret immediately

The client secret shown in the output is displayed only once. Store it in a secure location (a separate, non-AI-Gateway-accessible vault or a password manager) before proceeding. Set a calendar reminder to rotate it before it expires (default: 1 year).

# Capture for use in the next step
SP_OBJECT_ID=$(az ad sp show --id "sp-ai-gateway" --query id -o tsv)

Step 4 — Grant Key Vault Access

Assign the AI Gateway identity the minimum required Key Vault role. For read-only access at runtime, use Key Vault Secrets User. Reserve Key Vault Secrets Officer (write) for the ops team doing rotations.

▶4.1 — Assign Key Vault Secrets User to the AI Gateway identity

Use the Object ID from Step 3. Replace $MI_OBJECT_ID with $SP_OBJECT_ID if you created a Service Principal.

az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee "$MI_OBJECT_ID" \
  --scope "$VAULT_ID"

Verify:

az role assignment list \
  --scope "$VAULT_ID" \
  --query "[?roleDefinitionName=='Key Vault Secrets User'].{Principal:principalName, Role:roleDefinitionName}" \
  -o table

▶4.2 — Azure OpenAI: grant Cognitive Services OpenAI User role

For Azure OpenAI, the role is assigned on the Azure OpenAI resource (not Key Vault). This replaces the API key: the AI Gateway identity authenticates to Entra ID and calls Azure OpenAI directly with a bearer token.

AZURE_OAI_RESOURCE_ID=$(az cognitiveservices account show \
  --name "my-azure-openai-instance" \
  --resource-group "$RESOURCE_GROUP" \
  --query id -o tsv)

az role assignment create \
  --role "Cognitive Services OpenAI User" \
  --assignee "$MI_OBJECT_ID" \
  --scope "$AZURE_OAI_RESOURCE_ID"

Verify: role appears on the Azure OpenAI resource's IAM blade, or:

az role assignment list \
  --scope "$AZURE_OAI_RESOURCE_ID" \
  --query "[?roleDefinitionName=='Cognitive Services OpenAI User'].{Principal:principalName}" \
  -o table

▶4.3 — Grant the ops team rotation access

Assign Key Vault Secrets Officer to the security/ops team who will perform key rotations. This role can create, update, and delete secret versions without granting read access to secret values (that stays with the AI Gateway identity and Key Vault Secrets User).

OPS_GROUP_ID=$(az ad group show --group "sg-ai-ops-team" --query id -o tsv)

az role assignment create \
  --role "Key Vault Secrets Officer" \
  --assignee "$OPS_GROUP_ID" \
  --scope "$VAULT_ID"

Secrets Officer has full read and write access

Key Vault Secrets Officer includes secrets/get, so members of this role can read secret values as well as create, update, and delete versions. If your security policy requires rotation access without the ability to read values, create a custom role that excludes Microsoft.KeyVault/vaults/secrets/getSecret/action.

Step 5 — Retrieve Secrets at Runtime

The AI Gateway retrieves secrets from Key Vault using the Azure SDK at startup or on demand. The SDK uses DefaultAzureCredential, which automatically selects the Managed Identity when running on Azure or falls back to the Service Principal credentials when running outside Azure.

▶5.1 — Python: retrieve all provider keys at startup

from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient

VAULT_URI = "https://kv-ai-gateway-llm.vault.azure.net/"

credential = DefaultAzureCredential()
client = SecretClient(vault_url=VAULT_URI, credential=credential)

# Retrieve provider keys
openai_key       = client.get_secret("llm-openai-api-key").value
anthropic_key    = client.get_secret("llm-anthropic-api-key").value
aws_key_id       = client.get_secret("llm-aws-bedrock-access-key-id").value
aws_secret       = client.get_secret("llm-aws-bedrock-secret-access-key").value
aws_region       = client.get_secret("llm-aws-bedrock-region").value
vertex_sa_json   = client.get_secret("llm-google-vertex-sa-key").value
vertex_project   = client.get_secret("llm-google-vertex-project-id").value

# Azure OpenAI uses the same credential object — no key needed
azure_oai_endpoint = client.get_secret("llm-azure-openai-endpoint").value

▶5.2 — Azure OpenAI: keyless authentication

When using Azure OpenAI with Entra auth, pass the token from DefaultAzureCredential directly to the OpenAI client. No API key is used.

from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from openai import AzureOpenAI

credential = DefaultAzureCredential()

# Create a token provider scoped to Cognitive Services
token_provider = get_bearer_token_provider(
    credential,
    "https://cognitiveservices.azure.com/.default"
)

client = AzureOpenAI(
    azure_endpoint=azure_oai_endpoint,
    azure_ad_token_provider=token_provider,
    api_version="2024-02-01"
)

response = client.chat.completions.create(
    model="gpt-4o",   # deployment name stored as llm-azure-openai-deployment-gpt4o
    messages=[{"role": "user", "content": "Hello"}]
)

▶5.3 — Service Principal: set environment variables for DefaultAzureCredential

When running outside Azure with a Service Principal, DefaultAzureCredential reads credentials from environment variables. Set these on the host running the AI Gateway:

export AZURE_TENANT_ID="your-tenant-id"
export AZURE_CLIENT_ID="your-sp-client-id"
export AZURE_CLIENT_SECRET="your-sp-client-secret"

Store these in your secrets manager, not in .env files

These three environment variables are the only credentials required to access all LLM keys. Treat them with the same care as root credentials. Use your deployment platform's native secrets injection (Kubernetes secrets, systemd credentials, etc.) rather than plain .env files.

Step 6 — Key Rotation and Expiry Alerting

Set a rotation cadence and configure Key Vault to alert before secrets expire. This is the step most teams skip. Do not skip it.

▶6.1 — Configure expiry alerts via Azure Monitor

Enable Key Vault diagnostic logs and create an alert rule that fires 30 days before any secret expires.

# Enable diagnostic logs for Key Vault
LOG_ANALYTICS_ID=$(az monitor log-analytics workspace show \
  --workspace-name "law-ai-gateway" \
  --resource-group "$RESOURCE_GROUP" \
  --query id -o tsv)

az monitor diagnostic-settings create \
  --name "kv-diag-ai-gateway" \
  --resource "$VAULT_ID" \
  --workspace "$LOG_ANALYTICS_ID" \
  --logs '[{"category":"AuditEvent","enabled":true}]' \
  --metrics '[{"category":"AllMetrics","enabled":true}]'

Create an alert for secrets expiring within 30 days:

az monitor alert create \
  --name "alert-llm-secret-expiry" \
  --resource-group "$RESOURCE_GROUP" \
  --target "$VAULT_ID" \
  --condition "count 'AuditEvent | where OperationName == \"SecretNearExpiry\"' > 0" \
  --description "One or more LLM provider API keys expire within 30 days"

▶6.2 — Rotate a key (standard procedure)

Key rotation follows the same pattern for every provider: generate a new key in the provider console, store it as a new secret version, verify the Gateway picks it up, then disable the old version.

Generate a new API key in the provider's console (OpenAI, Anthropic, AWS IAM, etc.). Do not revoke the old key yet.
Store the new key as a new version of the existing secret:

EXPIRY=$(date -u -d "+90 days" +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -v+90d +"%Y-%m-%dT%H:%M:%SZ")

az keyvault secret set \
  --vault-name "$VAULT_NAME" \
  --name "llm-openai-api-key" \
  --value "your-new-openai-api-key" \
  --expires "$EXPIRY"

Verify the AI Gateway is using the new key (check Gateway logs for successful requests).
Disable or revoke the old key in the provider console.
Disable the previous secret version in Key Vault:

# List versions to find the previous one
az keyvault secret list-versions \
  --vault-name "$VAULT_NAME" \
  --name "llm-openai-api-key" \
  --query "[].{Version:id, Enabled:attributes.enabled, Created:attributes.created}" \
  -o table

# Disable the old version (replace VERSION_ID with the previous version)
az keyvault secret set-attributes \
  --vault-name "$VAULT_NAME" \
  --name "llm-openai-api-key" \
  --version "VERSION_ID" \
  --enabled false

Reference — Secret Naming Convention

Provider	Secret Name	Value	Expiry
OpenAI	`llm-openai-api-key`	API key (`sk-proj-...`)	90 days
Azure OpenAI	`llm-azure-openai-endpoint`	Resource endpoint URL	None
Azure OpenAI	`llm-azure-openai-deployment-<model>`	Deployment name	None
AWS Bedrock	`llm-aws-bedrock-access-key-id`	IAM access key ID	90 days
AWS Bedrock	`llm-aws-bedrock-secret-access-key`	IAM secret access key	90 days
AWS Bedrock	`llm-aws-bedrock-region`	Region (e.g., `us-east-1`)	None
Anthropic	`llm-anthropic-api-key`	API key (`sk-ant-...`)	90 days
Google Vertex AI	`llm-google-vertex-sa-key`	Service account JSON	90 days
Google Vertex AI	`llm-google-vertex-project-id`	GCP project ID	None

Reference — RBAC Role Matrix

Identity	Key Vault Role	Purpose
AI Gateway (Managed Identity / SP)	Key Vault Secrets User	Read secrets at runtime. Cannot write or list all secrets.
Security / Ops team (Entra group)	Key Vault Secrets Officer	Create, update, disable, and read secret versions. Full secrets management for rotation workflows.
Initial setup admin (person)	Key Vault Administrator	Full control during setup. Remove or scope down after initial configuration.
AI Gateway (Managed Identity / SP)	Cognitive Services OpenAI User	Call Azure OpenAI without an API key. Assigned on the Azure OpenAI resource, not the vault.

Remove setup admin access after initial configuration

The Key Vault Administrator role granted in Step 1.3 is for initial setup only. Once secrets are stored and access is verified, remove or narrow this assignment to Key Vault Secrets Officer for the ongoing ops team. Running day-to-day with administrator access violates least-privilege principles.