You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: add AgentgatewayPolicy alternative for multiple endpoints configuration
Add a new alternative approach in the Multiple Endpoints documentation
showing how to define routes in a separate AgentgatewayPolicy resource
attached to an HTTPRoute instead of directly on the AgentgatewayBackend.
This decouples routing configuration from the backend definition.
Copy file name to clipboardExpand all lines: assets/agw-docs/pages/agentgateway/llm/providers/multiple-endpoints.md
+55-1Lines changed: 55 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,6 +4,8 @@ Configure access to multiple OpenAI API endpoints such as for chat completions,
4
4
5
5
To set up multiple LLM endpoints, use the `ai.routes` field in the `policies` section of the {{< reuse "agw-docs/snippets/backend.md" >}} resource. This field maps the API paths to supported route types. The keys are URL suffix matches, like `/v1/models`. The values are the route types, like `Completions` or `Passthrough`.
6
6
7
+
Alternatively, you can define the routes in a separate {{< reuse "agw-docs/snippets/policy.md" >}} resource attached to an {{< reuse "agw-docs/snippets/agentgateway.md" >}} {{< reuse "agw-docs/snippets/gateway-api.md" >}}. This approach decouples routing configuration from the backend definition, which can simplify management when you have multiple routes pointing to the same backend.
8
+
7
9
-`Completions`: Parses the request, translates it to the LLM provider format, and fully processes it as an LLM request. This route type unlocks the full set of {{< reuse "agw-docs/snippets/agentgateway.md" >}} LLM features, such as tokenization and token-based rate limiting, prompt guards, prompt enrichment, model aliasing, transformations, cost tracking, and detailed observability.
8
10
-`Detect`: Forwards the request as-is, but makes a best effort to extract the model and token counts so that a subset of policies still applies, specifically token-based rate limiting and telemetry. Guardrails and other request-shaping policies do not apply. Use this route type for endpoints with a format that cannot be automatically parsed by {{< reuse "agw-docs/snippets/agentgateway.md" >}}, but where you still want metrics and rate limiting.
9
11
-`Passthrough`: Forwards the request to the LLM provider as-is, with no parsing, processing, or policies. Use passthrough for endpoints that do not need any traffic policy or manipulation, such as health checks or custom endpoints. Otherwise, use the other route types.
@@ -79,7 +81,59 @@ Configure access to multiple endpoints in your LLM provider, such as for chat co
3. Send requests to different OpenAI endpoints. With the routes configured, you can access different OpenAI endpoints by including the full path in your requests:
84
+
85
+
3.**Alternative: define routes in a {{< reuse "agw-docs/snippets/policy.md" >}}**
86
+
87
+
Instead of setting up the `routes` map on the {{< reuse "agw-docs/snippets/backend.md" >}}, you can configure them in a separate {{< reuse "agw-docs/snippets/policy.md" >}} attached to the HTTPRoute. This decouples routing configuration from the backend definition.
88
+
89
+
1. Create the {{< reuse "agw-docs/snippets/backend.md" >}} resource without the `routes` field:
3. Send requests to different OpenAI endpoints. With the routes configured, you can access different OpenAI endpoints by including the full path in your requests:
0 commit comments