LLM inference callback support

[SteveSandersonMS]

Summary

This PR adds SDK support for intercepting LLM inference requests and handling them in user code across all six SDK languages: Node.js, .NET, Python, Go, Rust, and Java.

Consumers register one client-global LlmRequestHandler (constructed once, no args). The runtime invokes it over JSON-RPC (llmInference.*) whenever it would otherwise issue a model-layer HTTP or WebSocket request — for both BYOK and CAPI — fully replacing the outbound call. A handler that overrides nothing is a transparent pass-through.

It includes the full feature work on this branch:

• wire up LLM inference provider registration and generated RPC types
• add the raw chunked callback protocol for outbound inference requests and responses
• cover plain HTTP, streaming responses, cancellation (runtime- and consumer-initiated), error mapping, session-id threading, and WebSocket transport
• port the feature to every SDK language with idiomatic HTTP types
• keep the public callback surface to a single LlmRequestHandler model with forwarding helpers

What changed

Shared protocol and plumbing

• add generated RPC / session event types needed for LLM inference callbacks across all SDK surfaces
• add SDK-side registration for a process-global LLM inference provider (llmInference.setProvider)
• route outbound inference requests through the callback bridge instead of requiring provider-specific hooks

Per-language ports

Each language exposes the same LlmRequestHandler model, mapped onto the most canonical HTTP representation available in that ecosystem:

Language	HTTP request/response type	WebSocket type
Node.js	Request / Response (Fetch)	per-connection handler
.NET	HttpRequestMessage / HttpResponseMessage	ClientWebSocket
Python	httpx request/response	per-connection handler
Go	*http.Request / *http.Response	per-connection handler
Rust	http::Request / http::Response	per-connection handler
Java	java.net.http HttpRequest / HttpResponse	java.net.http.WebSocket

All ports thread cancellation and session id through the request context, and provide a ForwardingWebSocketHandler (or equivalent) for the common mutate-and-forward case.

API shape

• collapse HTTP interception to a single send hook (SendRequestAsync / sendRequest / sendHttp / language equivalents)
• expose WebSocket interception through OpenWebSocketAsync / openWebSocket, returning a per-connection handler object
• allow consumers to mutate, drop, duplicate, or fully replace request/response messages while keeping the common forwarding case straightforward

Usage examples

C#

using GitHub.Copilot;
using System.Net.Http;

sealed class MyHandler : LlmRequestHandler
{
    protected override async Task<HttpResponseMessage> SendRequestAsync(
        HttpRequestMessage request,
        LlmRequestContext ctx)
    {
        request.Headers.Add("X-Debug-Session", ctx.SessionId ?? "none");
        return await base.SendRequestAsync(request, ctx);
    }

    protected override Task<CopilotWebSocketHandler> OpenWebSocketAsync(LlmRequestContext ctx)
        => Task.FromResult<CopilotWebSocketHandler>(new MyForwardingSocket(ctx));
}

sealed class MyForwardingSocket : ForwardingWebSocketHandler
{
    public MyForwardingSocket(LlmRequestContext ctx)
        : base(ctx)
    {
    }

    public override Task SendRequestMessageAsync(LlmWebSocketMessage message)
    {
        var text = message.GetText().Replace("model-A", "model-B");
        return base.SendRequestMessageAsync(LlmWebSocketMessage.Text(text));
    }
}

Node.js

import {
    ForwardingWebSocketHandler,
    LlmRequestContext,
    LlmRequestHandler,
} from "@github/copilot";

class MyHandler extends LlmRequestHandler {
    protected override async sendRequest(request: Request, ctx: LlmRequestContext): Promise<Response> {
        const headers = new Headers(request.headers);
        headers.set("x-debug-session", ctx.sessionId ?? "none");

        return super.sendRequest(new Request(request, { headers }), ctx);
    }

    protected override async openWebSocket(ctx: LlmRequestContext) {
        return new MyForwardingSocket(ctx);
    }
}

class MyForwardingSocket extends ForwardingWebSocketHandler {
    override sendRequestMessage(data: string | Uint8Array) {
        if (typeof data === "string") {
            return super.sendRequestMessage(data.replace("model-A", "model-B"));
        }
        return super.sendRequestMessage(data);
    }
}

Java

import com.github.copilot.ForwardingWebSocketHandler;
import com.github.copilot.LlmRequestContext;
import com.github.copilot.LlmRequestHandler;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.io.InputStream;

final class MyHandler extends LlmRequestHandler {
    @Override
    protected HttpResponse<InputStream> sendHttp(HttpRequest request, LlmRequestContext ctx) throws Exception {
        HttpRequest mutated = HttpRequest.newBuilder(request, (n, v) -> true)
                .header("X-Debug-Session", ctx.sessionId() == null ? "none" : ctx.sessionId())
                .build();
        return super.sendHttp(mutated, ctx);
    }

    @Override
    protected CopilotWebSocketHandler openWebSocket(LlmRequestContext ctx) {
        return new ForwardingWebSocketHandler(ctx.url(), ctx.headers()) {
            @Override
            protected byte[] onSendRequestMessage(byte[] data, boolean binary) {
                String text = new String(data, java.nio.charset.StandardCharsets.UTF_8).replace("model-A", "model-B");
                return text.getBytes(java.nio.charset.StandardCharsets.UTF_8);
            }
        };
    }
}

Register the handler when constructing the client, e.g. in Java:

CopilotClient client = new CopilotClient(
    new CopilotClientOptions().setLlmInference(new LlmInferenceConfig().setHandler(new MyHandler())));

Tests

Each language adds e2e coverage (mirroring a shared reference suite) for:

• callback provider registration
• HTTP inference interception
• streaming inference interception
• error mapping
• runtime-initiated and consumer-initiated cancellation
• session-id threading
• WebSocket callback handling
• an idiomatic handler test exercising mutate-and-forward over both HTTP and WebSocket

Plus per-language unit coverage where applicable (e.g. .NET handler/adapter behavior, Node.js unit flows).

Resolves github/copilot-sdk-internal#88

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[github-actions]

Cross-SDK Consistency Review

This PR adds LLM inference callback support to Go and .NET, bringing the feature to 4 of the 6 SDK implementations (Node.js and Python already had it). Overall the API shapes are well-aligned across languages. A few observations:

✅ Well-aligned across the four covered SDKs

Concept	Node.js	Python	Go	.NET
Entry class	LlmRequestHandler	LlmRequestHandler	LlmRequestHandler	LlmRequestHandler
HTTP hook	sendRequest(req, ctx)	send_request(req, ctx)	Transport field (http.RoundTripper)	SendRequestAsync(req, ctx)
WS hook	openWebSocket(ctx)	open_web_socket(ctx)	OpenWebSocket field (func)	OpenWebSocketAsync(ctx)
Context	LlmRequestContext	LlmRequestContext	LlmRequestContext	LlmRequestContext
WS handler	CopilotWebSocketHandler	CopilotWebSocketHandler	CopilotWebSocketHandler	CopilotWebSocketHandler
Forwarding WS	ForwardingWebSocketHandler	ForwardingWebSocketHandler	ForwardingWebSocketHandler	ForwardingWebSocketHandler
Config	LlmInferenceConfig { handler }	LlmInferenceConfig { handler }	LlmInferenceConfig { Handler }	LlmInferenceConfig { Handler }
Low-level interface	LlmInferenceProvider	LlmInferenceProvider	LlmInferenceProvider	ILlmInferenceProvider

Naming and structure follow each language's conventions (camelCase/snake_case/PascalCase, AbortSignal/asyncio.Event/context.Context/CancellationToken for cancellation, etc.).

⚠️ Remaining feature gap: Java and Rust SDKs

LLM inference callback support is not yet implemented in the Java and Rust SDKs. These two SDKs currently have no LlmInferenceProvider, LlmRequestHandler, or related types. If Java and Rust parity is intended, they'd need equivalent implementations. Is there a tracking issue or follow-up planned for those two?

i️ Language-idiomatic design differences (not issues)

These are expected given the languages involved:

• Go's LlmRequestHandler uses struct fields for customization (Transport http.RoundTripper, OpenWebSocket func(...)) instead of method overrides. This is the idiomatic Go approach with no class inheritance.
• Go's ForwardingWebSocketHandler adds OnSendRequestMessage and OnSendResponseMessage function fields for inline message transformation, avoiding the need for a full subtype. The other SDKs accomplish the same thing via subclassing.
• .NET uses LlmWebSocketMessage (a value-type wrapper with IsBinary/GetText()) where Node.js and Python use string | Uint8Array / str | bytes union types. Both representations carry the same semantic content.

Generated by SDK Consistency Review Agent for issue #1689 · sonnet46 2M · ◷

[SteveSandersonMS]

Why do we have this wrapper type and not just put LlmRequestHandler directly onto the create/resume config?

[SteveSandersonMS]

Is this IVT really needed?

[SteveSandersonMS]

Why is this public when everything else in here is internal? Maybe it could move to the file with all the public API.

[SteveSandersonMS]

Why are you doing that? What's wrong with holding the RPC reply until the consumer's I/O completes? Isn't tat necessary to achieve natural backpressure and error propagation?

Altogether I'm surprised by how much complexity there is in this file and wonder if you're trying to make it too clever. Why isn't there a significantly simpler way of adapting the RPC protocol to the public base class, for example by eliminating any separate queueing/buffering/etc?

[SteveSandersonMS]

Seems really odd to have an adapter like this when the source data is already in the right format. What stops us from simplifying all this?