Skip to content

Commit a101fbb

Browse files
committed
docs: add AgentgatewayPolicy alternative for multiple endpoints configuration
Add a new alternative approach in the Multiple Endpoints documentation showing how to define routes in a separate AgentgatewayPolicy resource attached to an HTTPRoute instead of directly on the AgentgatewayBackend. This decouples routing configuration from the backend definition.
1 parent 246599d commit a101fbb

1 file changed

Lines changed: 55 additions & 1 deletion

File tree

assets/agw-docs/pages/agentgateway/llm/providers/multiple-endpoints.md

Lines changed: 55 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ Configure access to multiple OpenAI API endpoints such as for chat completions,
44

55
To set up multiple LLM endpoints, use the `ai.routes` field in the `policies` section of the {{< reuse "agw-docs/snippets/backend.md" >}} resource. This field maps the API paths to supported route types. The keys are URL suffix matches, like `/v1/models`. The values are the route types, like `Completions` or `Passthrough`.
66

7+
Alternatively, you can define the routes in a separate {{< reuse "agw-docs/snippets/policy.md" >}} resource attached to an {{< reuse "agw-docs/snippets/agentgateway.md" >}} {{< reuse "agw-docs/snippets/gateway-api.md" >}}. This approach decouples routing configuration from the backend definition, which can simplify management when you have multiple routes pointing to the same backend.
8+
79
- `Completions`: Parses the request, translates it to the LLM provider format, and fully processes it as an LLM request. This route type unlocks the full set of {{< reuse "agw-docs/snippets/agentgateway.md" >}} LLM features, such as tokenization and token-based rate limiting, prompt guards, prompt enrichment, model aliasing, transformations, cost tracking, and detailed observability.
810
- `Detect`: Forwards the request as-is, but makes a best effort to extract the model and token counts so that a subset of policies still applies, specifically token-based rate limiting and telemetry. Guardrails and other request-shaping policies do not apply. Use this route type for endpoints with a format that cannot be automatically parsed by {{< reuse "agw-docs/snippets/agentgateway.md" >}}, but where you still want metrics and rate limiting.
911
- `Passthrough`: Forwards the request to the LLM provider as-is, with no parsing, processing, or policies. Use passthrough for endpoints that do not need any traffic policy or manipulation, such as health checks or custom endpoints. Otherwise, use the other route types.
@@ -79,7 +81,59 @@ Configure access to multiple endpoints in your LLM provider, such as for chat co
7981
kind: {{< reuse "agw-docs/snippets/backend.md" >}}
8082
EOF
8183
```
82-
3. Send requests to different OpenAI endpoints. With the routes configured, you can access different OpenAI endpoints by including the full path in your requests:
84+
85+
3. **Alternative: define routes in a {{< reuse "agw-docs/snippets/policy.md" >}}**
86+
87+
Instead of setting up the `routes` map on the {{< reuse "agw-docs/snippets/backend.md" >}}, you can configure them in a separate {{< reuse "agw-docs/snippets/policy.md" >}} attached to the HTTPRoute. This decouples routing configuration from the backend definition.
88+
89+
1. Create the {{< reuse "agw-docs/snippets/backend.md" >}} resource without the `routes` field:
90+
91+
```yaml
92+
kubectl apply -f- <<EOF
93+
apiVersion: agentgateway.dev/v1alpha1
94+
kind: {{< reuse "agw-docs/snippets/backend.md" >}}
95+
metadata:
96+
name: openai
97+
namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}
98+
spec:
99+
ai:
100+
provider:
101+
openai:
102+
model: gpt-3.5-turbo # Optional: specify default model
103+
# host: api.openai.com # Optional: custom host if needed
104+
# port: 443 # Optional: custom port
105+
policies:
106+
auth:
107+
secretRef:
108+
name: openai-secret
109+
EOF
110+
```
111+
112+
2. Create an {{< reuse "agw-docs/snippets/policy.md" >}} resource that targets the HTTPRoute and defines the `routes`:
113+
114+
```yaml
115+
kubectl apply -f- <<EOF
116+
apiVersion: agentgateway.dev/v1alpha1
117+
kind: {{< reuse "agw-docs/snippets/policy.md" >}}
118+
metadata:
119+
name: openai-routes
120+
namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}
121+
spec:
122+
targetRefs:
123+
- group: gateway.networking.k8s.io
124+
kind: HTTPRoute
125+
name: openai
126+
backend:
127+
ai:
128+
routes:
129+
"/v1/chat/completions": "Completions"
130+
"/v1/embeddings": "Passthrough"
131+
"/v1/models": "Passthrough"
132+
"*": "Passthrough"
133+
EOF
134+
```
135+
136+
3. Send requests to different OpenAI endpoints. With the routes configured, you can access different OpenAI endpoints by including the full path in your requests:
83137

84138
{{< tabs >}}
85139
{{% tab name="Cloud Provider LoadBalancer" %}}

0 commit comments

Comments
 (0)