Custom Code LLM Provider

A Custom Code provider is an LLM provider whose wire protocol you write yourself, in C#, inside the workspace. It compiles and runs in the server process like any other workspace code, and once saved it appears in the chat and agent model pickers next to the built-in providers.

Reach for it when the twenty built-in providers don't cover your case:

• an internal LLM gateway that requires its own auth handshake, routing headers, or request signing;
• a vendor API that is neither OpenAI- nor Anthropic-shaped;
• a model served behind a proxy that rewrites the request or response;
• a provider you want to wrap with your own retry, redaction, or per-tenant routing logic.

If the endpoint speaks the OpenAI HTTP protocol, you do not need this page — point a plain API-key provider at its base URL instead.

Prerequisites

• Custom API feature flag. If the Custom Code tile is missing from the provider list, or clicking it opens a license-check modal, the feature isn't enabled for the workspace.
• System administrator account. Custom Code providers are not offered in the Curiosity desktop app — they are a server-workspace feature.

Step 1 — Create the provider

1. Go to Manage → AI → LLM Providers (#/manage/ai/llm-providers).
2. Pick Custom Code from the provider list on the left.
3. Click Set up Custom Code.

The provider is created straight away as Custom Chat AI Provider and its code editor opens. Unlike the API-key providers, a Custom Code provider has no credentials form and no model catalog — the editor is its settings screen, so clicking the settings icon on the provider card later reopens the same editor.

The editor header holds three things:

Step 2 — Understand the contract

The body of the editor is a C# script, not a class file. It may declare as many types as it needs, and its last statement must return a factory:

return (HttpClient httpClient, ILogger logger) => new MyProvider(httpClient, logger);

The factory type is Func<HttpClient, ILogger, IChatProvider>. The workspace calls it once per completion, handing it an HttpClient from a named pool and the provider's logger; the compiled script itself is cached per provider and reused.

IChatProvider (namespace Mosaik.GraphDB.Tasks) has two members you must implement and two properties you must supply:

Four more members have working defaults you only override when you need to:

Step 3 — Write the provider

The editor opens pre-filled with a complete, working provider for the Anthropic Messages API — streaming SSE, a real cancellation path, and the DTOs it deserializes into. That template is the fastest starting point: keep its shape, replace the protocol.

The shape it establishes looks like this:

public class MyProvider : IChatProvider
{
    private string APIKey = "YOUR API KEY";

    private HttpClient _client;
    private ILogger    _logger;

    public MyProvider(HttpClient client, ILogger logger)
    {
        _logger = logger;
        _client = client;
        _client.DefaultRequestHeaders.Add("Authorization", "Bearer " + APIKey);
    }

    public async IAsyncEnumerable<IChatAICompletion> StreamCompletionEnumerableAsync(Graph graph, UID128 taskUID, UID128 chatUID, UID128 userUID, List<IChatAIMessage> prompts, string userIdentifier, int? maxCompletionTokens = null, Dictionary<UID128, _ChatAITool.CachedChatAITool[]> tools = null, LlmTornado.Chat.ChatRequestResponseFormats responseFormat = null, ChatCompletionOptions options = null, [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        // The chat's Stop button cancels through this token, so a generation that ignores it
        // keeps running (and keeps costing) after the user has stopped it.
        ChatAIProviderShared.GetChatStopCancellationToken(chatUID, out var stopTokenSource, out var stopToken);

        using var stopSource    = stopTokenSource;
        using var combinedCts   = CancellationTokenSource.CreateLinkedTokenSource(stopToken, cancellationToken);
        var       stoppingToken = combinedCts.Token;

        // ... build the request from `prompts`, send it, and yield a part per chunk:
        yield return new ChatAICompletion(ChatAuthorRole.Assistant, "…");
    }

    public async Task<List<IChatAICompletion>> GetFullCompletionAsync(Graph graph, UID128 taskUID, UID128 chatUID, List<IChatAIMessage> prompts, UID128 userUID, string userIdentifier, int? maxCompletionTokens = null, Dictionary<UID128, _ChatAITool.CachedChatAITool[]> tools = null, LlmTornado.Chat.ChatRequestResponseFormats responseFormat = null, ChatCompletionOptions options = null, CancellationToken cancellationToken = default)
        => await StreamCompletionEnumerableAsync(graph, taskUID, chatUID, userUID, prompts, userIdentifier, maxCompletionTokens, tools, responseFormat, options, cancellationToken).ToListAsync(cancellationToken: cancellationToken);

    public int TotalContextLimit => 128_000;
    public int OutputTokenLimit  => 4096;
}

return (HttpClient httpClient, ILogger logger) => new MyProvider(httpClient, logger);

Reading the conversation

prompts is the conversation the workspace wants answered, as IChatAIMessage values. Each carries an AuthorRole (User, Assistant, System, Context, Error, Unknown) and renders to text with .Render(). Map the roles onto whatever your API calls them, and run the text through ChatAI.StripControlCharacters(...) — the template does — so a stray control character doesn't make the request unparseable.

Yielding the answer

Everything you yield is an IChatAICompletion. The parts the chat understands:

A provider that yields only ChatAICompletion parts is a perfectly good text-only provider — that is what the shipped template does.

Tools

The tools dictionary carries the AI tools available for this turn. Supporting them means serializing them into your API's function-calling format, emitting ToolInvoke parts when the model asks for a call, and feeding ToolResult parts back on the following request.

Ignoring the parameter is a supported choice — the provider then answers from the prompt alone, and tools simply never fire on chats using it.

Cancellation

ChatAIProviderShared.GetChatStopCancellationToken(chatUID, out var cts, out var token) is how the chat's Stop button reaches your code. Link it with the incoming cancellationToken, pass the combined token to every SendAsync and ReadLineAsync, and dispose the source when the enumeration ends. A provider that ignores it keeps generating — and keeps billing — after the user has stopped it.

Leave CancelAsync alone unless you have a provider-side cancel call to make as well.

Step 4 — Save and check it works

The editor compiles as you type and reports diagnostics inline; Save stays disabled until the code compiles. Compilation uses the same references and imports the running provider will use, so a clean editor means a provider that will load.

These namespaces are imported for you — no using needed:

System                              System.Linq
System.Collections.Generic          System.Collections.Concurrent
System.IO                           System.Threading
System.Threading.Tasks              System.Threading.Channels
System.Text                         System.Text.RegularExpressions
System.Text.Json                    System.Text.Json.Serialization
System.Net.Http                     Microsoft.Extensions.Logging
Mosaik.Core                         Mosaik.Schema
Mosaik.AI                           Mosaik.GraphDB
Mosaik.GraphDB.Tasks                Mosaik.GraphDB.Indexes
Mosaik.GraphDB.Training             Mosaik.GraphDB.WebSockets
GraphDB.Schema                      Catalyst
UID                                 LlmTornado
LlmTornado.Images                   LlmTornado.Images.Models

Anything else needs its full name (LlmTornado.Chat.ChatRequestResponseFormats) or a package. NuGet packages installed in the workspace are available to provider code the same way they are to endpoints.

After saving, prove it end to end:

1. Open a chat and pick the provider from the model dropdown — a Custom Code provider contributes one entry, labelled with its display name.
2. Send a message and confirm text streams back token by token rather than arriving in one block.
3. Press Stop mid-answer and confirm generation actually halts.
4. Check Manage → Operate → LLM Usage if you record usage (see below).

The compiled code is cached per provider and keyed by a hash of the source, so a save takes effect on the next completion. Installing or removing a workspace package clears the cache too.

API keys and secrets

The provider's API key lives in the code, as in the shipped template. That string is stored in the graph with the rest of the provider definition, and encrypted on disk when MSK_GRAPH_MASTER_KEY is set — see Encryption at rest.

Two consequences worth planning for:

• The workspace secret store is not reachable from this scope. Endpoints and AI tools get a GetSecretAsync; a provider script does not. Read the key from the code, or from an environment variable on the host with Environment.GetEnvironmentVariable(...) if you would rather keep it out of the workspace entirely.
• The key travels with the definition. If you export provider definitions into git, the key goes with them — use an environment variable in that case, or scrub the export.

Usage and cost tracking

Nothing is recorded automatically: the built-in providers read token counts off their provider's response and write them to the usage ledger, and a custom provider has to do the same if it wants to appear on Manage → Operate → LLM Usage. LlmUsageTracker (namespace Mosaik.GraphDB.Tasks) is the type to use — construct it at the start of a completion, call RecordUsageTokens(...) once the response reports its counts, and Save() when the completion ends.

Skipping this costs you the dashboard row, not the completion. The provider works either way.

Version control and promotion

A Custom Code provider is part of the workspace configuration, so it exports and imports like every other definition:

code/ai-providers/ai-code-provider-<display-name>.cs

The file is the provider's source with two header attributes that identify it:

[provider: Curiosity.ChatAI.Name("My Gateway")]
[provider: Curiosity.ChatAI.UID("…")]

public class MyProvider : IChatProvider
{
    …
}

Importing that file on another workspace re-creates the provider under the same UID. To remove one from a target workspace, ship a file carrying only the deletion marker:

[provider: Curiosity.ChatAI.UID("…")]
[provider: Curiosity.ChatAI.Deleted]

The admin assistant's filesystem exposes the same path, so rm code/ai-providers/<file>.cs in a shell session does the same thing.

Troubleshooting

Next steps

• The rest of the provider configuration: LLM Configuration.
• A local model without writing any code: Self-hosted model with Ollama.
• Relaying a configured provider to your own clients: LLM Router.
• Giving the provider something to call: AI Tools.
• Promoting the definition between workspaces: Configuration sync.