Streaming, Semantic Caching, and Agent-to-Agent Traffic

Add stream: true to the request body. Nothing else about the call changes, and the gateway key, the model slug, and any config you already use all stay the same.

curl https://aigw.portkey.ai/v1/chat/completions \
  -H "Authorization: Bearer $PORTKEY_API_KEY" \
  -H "Content-Type: application/json" \
  -N \
  -d '{
    "model": "@openai-prod/gpt-4o",
    "stream": true,
    "messages": [{"role": "user", "content": "Name three UK cities."}]
  }'

The -N flag disables curl's output buffering. Without it the response still streams, but curl holds it and prints everything at once, which makes a working stream look like a non-streaming call.

Verify: you see a run of data: lines appear progressively rather than all together, each carrying a small delta, and the sequence ends with data: [DONE]. If the whole body lands at once, either -N is missing or stream: true did not reach the provider.

On a streaming route, an input guardrail is the only guardrail that can stop anything. It runs before the stream opens, while the gateway still owns the entire response. Two settings decide whether it blocks:

• async = false — the check runs before the request goes to the provider rather than alongside it. Left at the default of true, the check only logs.
• deny = true — a failed check kills the request with 446. Left at the default of false, a failed check returns 246 and the request proceeds anyway.

Attach the guardrail through the config's input_guardrails list, by ID:

{
  "input_guardrails": ["pii-and-prompt-injection-guardrail"],
  "output_guardrails": ["response-policy-guardrail"]
}

Send a prompt that the guardrail should reject, with stream: true set.

Verify: the call returns HTTP 446 and no data: chunks at all. A 246 means the guardrail ran and failed but deny is off. A 200 with a full stream means the guardrail either is not attached to this config or is still asynchronous.

By default the gateway holds itself to strict OpenAI response compliance, and a hook_results chunk is not part of the OpenAI schema. The verdicts are therefore suppressed. Set the header to false to receive them.

curl https://aigw.portkey.ai/v1/chat/completions \
  -H "Authorization: Bearer $PORTKEY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-portkey-strict-open-ai-compliance: false" \
  -N \
  -d '{
    "model": "@openai-prod/gpt-4o",
    "stream": true,
    "messages": [{"role": "user", "content": "Summarise the onboarding policy."}]
  }'

Input guardrail verdicts arrive under before_request_hooks before the content starts. Output guardrail verdicts arrive under after_request_hooks in an extra chunk after [DONE].

In the OpenAI Python client the results sit on the chunk's extra fields, because the SDK model does not know about them:

async for chunk in response:
    extra = chunk.model_extra or {}
    hooks = extra.get("hook_results")
    if hooks:
        if hooks.get("before_request_hooks"):
            print("input guardrails:", hooks["before_request_hooks"])
        if hooks.get("after_request_hooks"):
            print("output guardrails (informational):", hooks["after_request_hooks"])
    print(chunk.choices, end="", flush=True)

Verify: at least one chunk carries a hook_results object. If every chunk is a plain content delta, the header did not reach the gateway, since the default of true silently drops these chunks rather than erroring.

Providers omit usage data from streaming responses unless asked for it. No token counts means no cost calculation, so every streamed request records as zero spend. Budgets built on that figure never trip, and chargeback reports under-count whichever teams stream the most.

Ask for usage explicitly:

{
  "model": "@openai-prod/gpt-4o",
  "stream": true,
  "stream_options": { "include_usage": true },
  "messages": [{"role": "user", "content": "Name three UK cities."}]
}

This applies to every OpenAI-compatible provider, Azure OpenAI included. Non-streaming requests return usage automatically and need no change.

Verify: open the log entry for the call and confirm the token counts and cost are non-zero. Then compare against a streamed call made without stream_options, which should show zero, to confirm you are looking at the right lever.

A request timeout on a streaming route does not mean what it means elsewhere. The timeout will not be triggered if the gateway receives at least one chunk within the configured duration. A provider that returns the first token quickly and then stalls for several minutes will not produce a 408, and the client will sit waiting.

Two things follow for an application that streams:

• Client-side idle timeout — the application needs its own timer measuring the gap between chunks, because the gateway's timeout only guards the gap before the first one.
• Timeout value — the configured duration is effectively a time-to-first-chunk budget on these routes, so set it against that figure rather than against a full-response target.

Verify: this one is best confirmed by reading the config rather than by inducing a provider stall. Check that any route carrying stream: true has a client-side idle timeout, and that the gateway timeout on that route is sized for first-chunk latency.

Semantic mode adds an embedding call and a vector search to every request that misses the exact-match check, and on a self-hosted gateway it adds two components to run. Work out whether the traffic will repay that before building it.

Verify: before enabling anything, turn on simple mode and read the hit rate on the Analytics cache tab after a day of real traffic. If simple caching is already hitting well, semantic mode has little left to add. If it hits almost never and the prompts are human-written, that is the signal to continue.

Skip this step and the next on a SaaS data plane, where the components are managed for you. On a self-hosted or hybrid gateway, set these in the gateway's environment:

SEMANTIC_CACHE_EMBEDDING_PROVIDER=openai
SEMANTIC_CACHE_EMBEDDINGS_URL=https://api.openai.com/v1/embeddings
SEMANTIC_CACHE_EMBEDDING_MODEL=text-embedding-3-small
SEMANTIC_CACHE_EMBEDDING_API_KEY=<openai-api-key>
SEMANTIC_CACHE_SIMILARITY_THRESHOLD=0.95
SEMANTIC_CACHE_EMBEDDING_DIMENSIONS=1536

The provider accepts openai, azure-openai, google for Gemini embeddings, or vertex-ai. Set the URL, model, and dimensions to match whichever you pick. For Azure OpenAI:

SEMANTIC_CACHE_EMBEDDING_PROVIDER=azure-openai
SEMANTIC_CACHE_EMBEDDINGS_URL=https://YOUR-RESOURCE.openai.azure.com/openai/v1/embeddings
SEMANTIC_CACHE_EMBEDDING_MODEL=text-embedding-3-small
SEMANTIC_CACHE_EMBEDDING_API_KEY=<azure-openai-api-key>
SEMANTIC_CACHE_SIMILARITY_THRESHOLD=0.95
SEMANTIC_CACHE_EMBEDDING_DIMENSIONS=1536

Verify: restart the gateway and confirm it starts without an embedding configuration error in the logs. The variables are not exercised until a semantic lookup runs, so a clean start is only a partial check. The real confirmation comes in Step 2.5.

The gateway supports Milvus and Pinecone. Four variables point it at whichever you run:

VECTOR_STORE=milvus
VECTOR_STORE_ADDRESS=<your-vector-store-address>
VECTOR_STORE_COLLECTION_NAME=<your-collection-name>
VECTOR_STORE_API_KEY=<your-vector-db-api-key>

Each store has its own setup conditions, and they are not interchangeable.

Verify: confirm the vector dimension configured on the store matches SEMANTIC_CACHE_EMBEDDING_DIMENSIONS. A mismatch here produces no lookups and no obvious error, just a cache that never hits.

Caching is a config property, not a header. Add a cache block:

{ "cache": { "mode": "semantic", "max_age": 3600 } }

In a config with routing targets, the block can sit at the top level so it applies to all of them, or on an individual target. A target-level block takes precedence over a top-level one, which lets an expensive model cache aggressively while a cheap one does not:

{
  "strategy": { "mode": "fallback" },
  "targets": [
    {
      "override_params": { "model": "@openai-prod/gpt-4o" },
      "cache": { "mode": "semantic", "max_age": 3600 }
    },
    {
      "override_params": { "model": "@anthropic-prod/claude-3-5-sonnet-20241022" },
      "cache": { "mode": "simple", "max_age": 200 }
    }
  ]
}

Verify: send any request through the config and check the log line. The status column should read Cache Miss rather than Cache Disabled. A first request can never hit, so a miss here is the correct result and confirms the cache is active.

Send one request, then send a reworded version of the same question. Everything other than the user text has to stay identical, including the model and every body parameter.

# First call — populates the cache
curl https://aigw.portkey.ai/v1/chat/completions \
  -H "Authorization: Bearer $PORTKEY_API_KEY" \
  -H "x-portkey-config: pc-cache-xxx" \
  -H "Content-Type: application/json" \
  -d '{"model": "@openai-prod/gpt-4o",
       "messages": [{"role": "user", "content": "Who is the US president?"}]}'

# Second call — different wording, same meaning
curl https://aigw.portkey.ai/v1/chat/completions \
  -H "Authorization: Bearer $PORTKEY_API_KEY" \
  -H "x-portkey-config: pc-cache-xxx" \
  -H "Content-Type: application/json" \
  -d '{"model": "@openai-prod/gpt-4o",
       "messages": [{"role": "user", "content": "Tell me who the president of the US is."}]}'

Verify: the second call returns noticeably faster, returns the same text as the first, and its log line reads Cache Semantic Hit. A status of Cache Hit on the second call means the two requests were byte-identical and you tested the simple cache by accident, so change more of the wording and try again.

If the second call reads Cache Miss, work down the gates in the order the diagram shows them:

• Mode — is the config's cache block set to semantic rather than simple?
• Route — semantic matching only runs on /chat/completions and /completions.
• Size — under 8,191 tokens and four messages or fewer.
• User message — at least one is required. A request carrying only system messages is dropped rather than matched.
• Exact-match fields — the model and every other body parameter, and the metadata header if you use one, still have to match exactly. Only the user text is matched by meaning.
• Threshold — two phrasings may simply not be similar enough for 0.95. Lower SEMANTIC_CACHE_SIMILARITY_THRESHOLD deliberately and in small steps, because a loose threshold returns confidently wrong answers.

Two controls decide how long an entry survives and who can be served it.

Lifetime comes from max_age in seconds on the cache block. The minimum is 60 seconds, the maximum is 90 days, and leaving it out gives 7 days. An organisation-level default set under Admin Settings, Organisation Properties, Cache Settings changes the arithmetic:

A per-request value can therefore only ever shorten the lifetime, which is what makes the organisation default a usable data-retention control rather than a suggestion.

Partitioning defaults to every request header, so two callers differing in any header never share an entry. That is safe and it is usually why a hit rate looks lower than expected. Narrow it to one string of your choosing:

curl https://aigw.portkey.ai/v1/chat/completions \
  -H "Authorization: Bearer $PORTKEY_API_KEY" \
  -H "x-portkey-config: pc-cache-xxx" \
  -H "x-portkey-cache-namespace: tenant-42" \
  -H "Content-Type: application/json" \
  -d '{"model": "@openai-prod/gpt-4o",
       "messages": [{"role": "user", "content": "Hello!"}]}'

Verify: send the same prompt under two different namespace values and confirm the second one misses. Then repeat it within one namespace and confirm it hits. That pair of results proves the partition is real rather than assumed.

A cached answer that has gone stale needs replacing before its TTL expires. Send the force-refresh header, which is a per-request header rather than a config property:

curl https://aigw.portkey.ai/v1/chat/completions \
  -H "Authorization: Bearer $PORTKEY_API_KEY" \
  -H "x-portkey-config: pc-cache-xxx" \
  -H "x-portkey-cache-force-refresh: true" \
  -H "Content-Type: application/json" \
  -d '{"model": "@openai-prod/gpt-4o",
       "messages": [{"role": "user", "content": "Who is the US president?"}]}'

The request still needs its cache config attached. Without one the header does nothing, because there is no cache to refresh.

Verify: the log line reads Cache Refreshed, and a subsequent ordinary request returns the new text rather than the old.

Agent access is off by default on every key. Two separate things have to be set, and missing either one produces an authorization failure rather than a useful message.

1. Open the API Keys page in Strata Cloud Manager and create or edit a key.
2. Turn on the Agent Gateway toggle in the key's permissions.
3. Enable the agents.invoke scope on any key that will actually call an agent, as opposed to one used only for administration.

Verify: reopen the key and confirm both the toggle and the scope are set. Record which key you used, because the registry's access decisions in Step 3.3 are made against it.

In the Agent Registry, add the upstream agent server URL. This is the address of the real agent, and it is also where the gateway will fetch the agent card and authenticate.

If you are testing before a production agent exists, two public A2A agents are published for the purpose:

Press Fetch Agent Card. The gateway reads /.well-known/agent.json from the URL you gave it and validates the transport and authentication schemes the agent declares.

Verify: the card comes back and lists the skills you expect. This is the step that proves the gateway can reach and authenticate to the agent, so do not move on from a failure here. Everything downstream assumes this call works.

Choose which workspaces the agent is available in, then narrow what those workspaces can call. Access control operates at two grains today:

• The agent — whether a caller may reach it at all.
• The skill — which of the capabilities the agent publishes a caller may invoke.

Scope to the skills a client was built for rather than to the agent's whole surface. An agent that can both read a record and delete one should not expose deletion to a client that only ever reads, and the skill grain is what makes that distinction enforceable at the gateway instead of inside the agent.

Verify: from a client holding a key scoped to one skill, attempt a skill you did not grant. The call should be refused. A refusal here is the control working, and it is worth capturing as evidence rather than treating as a failed test.

Open the Agents tab and copy the virtual server URL for the agent. It follows the agent's slug:

https://aigw.portkey.ai/agent/{agent-slug}

Replace the agent server URL in your client with that one, and send the gateway API key as the bearer token. No other client change is needed, because the virtual server speaks A2A to your client exactly as the real agent did.

Confirm the swap worked by fetching the card through the gateway:

curl --location --request POST \
  'https://aigw.portkey.ai/agent/{agent-slug}/.well-known/agent.json' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer {PORTKEY_API_KEY}'

In Python, the A2A SDK's card resolver needs nothing beyond the new base URL and the gateway key:

import asyncio, json
import httpx
from a2a.client import A2ACardResolver

BASE_URL = "https://aigw.portkey.ai/agent/{agent-slug}"
API_KEY = "{PORTKEY_API_KEY}"

async def main():
    async with httpx.AsyncClient(
        base_url=BASE_URL,
        headers={"Authorization": f"Bearer {API_KEY}"},
    ) as client:
        resolver = A2ACardResolver(httpx_client=client, base_url=BASE_URL)
        card = await resolver.get_agent_card()
        print(json.dumps(card.model_dump(mode="json", exclude_none=True), indent=2))

asyncio.run(main())

Verify: the card returned through the gateway matches the one the registry fetched in Step 3.2, and the skills listed are the ones this key is scoped to. Then run a real task against the agent and confirm it completes as it did before the swap.

Logging is the reason most organisations put a gateway in front of their agents in the first place. Every request to a virtual server is recorded, and those records are what the observability and governance features are built on.

Verify: open the Logs view and find the calls you made in Step 3.4. Confirm the agent and the invoking key are both identifiable on the log line. Then make a call that the registry should refuse, from Step 3.3, and confirm it appears too. A refused call that leaves no record is worse than no control at all, so check both.

Everything in this guide is drawn from the Prisma AIRS AI Gateway product documentation as mirrored on 21 September 2026. Nothing here is inferred from behaviour observed in a tenant, so each claim can be checked against the page it came from.