Extract files to Markdown

Every file the workspace indexes is extracted once, and the extraction stores the file's content in two forms, page by page. The text is what the search, embedding and NLP indexes read. The Markdown keeps headings, lists and tables, for AI assistants and your own code. This page shows how to read that Markdown from C#. It also shows how to run the extractor on demand with ChatAI.GetFileAsMarkdownAsync(fileUID) when a file has none stored or you need its pictures read.

For an overview of what the extractor does and when, see OCR & File Extraction.

Read the stored Markdown

When the workspace has indexed a file, read its Markdown out of the graph. No extraction runs:

// Inside any code scope that exposes ChatAI
var markdown = ChatAI.GetMarkdownFromNode(fileUID);

The Markdown is stored only when it says more than the text. A .txt file, a source file or a recognized page of plain prose stores none, so falling back to the text is the ordinary case, not a failure. Use GetStoredMarkdown when your code has to tell "no Markdown stored" from "the Markdown is the text". Scans and images have Markdown too: OCR writes it from the page layout.

The Markdown is stored as _MarkdownDocument nodes, one per page and text box, beside the _Document nodes the indexes read and joined to the file the same way. They are not indexed, and the generic node readers do not return them: read them through the methods above.

Extract on demand

GetFileAsMarkdownAsync is available wherever the safe-graph ChatAI helper is: Custom Endpoints, Code Indexes, scheduled tasks, search scopes:

// Inside any code scope that exposes ChatAI
var markdown = await ChatAI.GetFileAsMarkdownAsync(fileUID, cancellationToken: CancellationToken);

A few things to know:

• It returns the stored Markdown when there is some. The extractor runs only for a file with no stored Markdown: one not indexed yet, one extracted before Markdown was stored, or one whose Markdown said no more than its text. The call then blocks on extraction, which can take minutes for a large scanned PDF. The cancellation token is honored at every stage.
• withOCR: true also reads the pictures inside the file (a screenshot on a slide, a scanned table in a Word file). It always runs a fresh extraction, because the stored Markdown was produced under the workspace's own OCR settings, and each picture costs tens of seconds. It needs the PaddleOCR-VL model; see OCR & File Extraction → Pictures inside documents. A scan with no text layer goes through OCR without it.
• The result is the file's content only. Thumbnails, captions and page renders are not part of it; for page images see Render file pages as images.
• An empty result means the file produced no text (for example a blank scan). It is not an error.

Recommended pattern: extract, cache on a Source node, attach to the file

Store the extract on a separate Source node linked to the file when the content should live on a node of your own schema. Your own indexes, edges and summaries can then attach to it. The early-out in the example below also keeps a file with no stored Markdown from being extracted twice.

In a Code Index attached to the File type

This is the canonical place to run extraction: every time a File node is added or re-indexed, the workspace calls into your code with the batch of UIDs as ToIndex. Cache the result and skip already-extracted files.

var failed = new List<UID128>();

foreach (var fileUid in ToIndex)
{
    if (CancellationToken.IsCancellationRequested)
        return ToIndex;

    if (!await TryExtractAndSaveAsync(fileUid))
        failed.Add(fileUid);
}

return failed;

async Task<bool> TryExtractAndSaveAsync(UID128 fileUid)
{
    LockedNode fileNode   = null;
    LockedNode sourceNode = null;

    try
    {
        var fileKey   = fileUid.ToString();
        var sourceKey = "extracted-from-" + fileKey;
        fileNode   = await Graph.GetOrAddLockedAsync(N._FileEntry.Type, fileKey);
        sourceNode = await Graph.GetOrAddLockedAsync(N.Source.Type,     sourceKey);

        var existing = sourceNode.GetString(N.Source.ExtractedText);
        if (!string.IsNullOrWhiteSpace(existing))
        {
            Logger.LogDebug("Skipping {UID}: already extracted", fileUid);
            await Graph.CommitAsync(fileNode, sourceNode);
            return true;
        }

        var markdown = await ChatAI.GetFileAsMarkdownAsync(fileUid, cancellationToken: CancellationToken);

        sourceNode.SetString(N.Source.ExtractedText, markdown);
        fileNode.AddUniqueEdge(E.HasDocument, sourceNode);

        await Graph.CommitAsync(fileNode, sourceNode);

        Logger.LogInformation("Extracted {Length} chars for {UID}", markdown.Length, fileUid);
        return true;
    }
    catch (Exception ex)
    {
        Logger.LogError(ex, "Failed to extract {UID}", fileUid);
        if (fileNode   != null) Graph.AbandonChanges(fileNode);
        if (sourceNode != null) Graph.AbandonChanges(sourceNode);
        return false;
    }
}

How this maps to the code index execution scope:

The return value matters: anything you return from the code index is treated as failed UIDs and requeued with backoff. Returning ToIndex on cancellation hands the remaining batch back so the next run picks up where this one stopped.

Idempotency

Two pieces make this re-runnable safely:

• Deterministic key for the Source node. "extracted-from-" + fileKey means the same file always maps to the same source node, so a second pass finds the existing node rather than creating a duplicate.
• The early-out check. If Source.ExtractedText is non-empty, GetFileAsMarkdownAsync is never called. Re-indexing the file won't re-OCR documents you've already processed.

To force a re-extraction for a specific file, for example after downloading the optional OCR models, clear Source.ExtractedText on its Source node and re-queue the file.

In a Custom Endpoint

Same call, returned straight to the caller. Good when the UI wants to render a file as Markdown on demand:

// POST /api/file/extract   — file UID in the request body
var fileUID = ParseBody<UID128>();
var markdown = await ChatAI.GetFileAsMarkdownAsync(fileUID, cancellationToken: CancellationToken);
return Ok(markdown);

Because OCR can be slow on large PDFs, prefer Pooling mode for an endpoint that does extraction. See Creating endpoints — Mode picker.

Linking the Code Index to the File type

When you create the Code Index in the UI (Manage → Indexes → Code Indexes → + New), set its target type to File. From then on, every time a File node is added or re-indexed, this code runs against it, and the extractor — including OCR for image-only PDFs — is invoked transparently.

Related pages

• Render file pages as images — image-only sibling for the same _FileEntry, returning blob UIDs ready to send to a multimodal LLM.
• File metadata as related facets — the same _FileEntry index target, but promoting the extractor's Metadata dictionary to nodes.
• OCR & File Extraction — what the extractor does, what's supported, when it runs.
• Code Indexes — Introduction — when to reach for a code index over a connector or a search scope.
• Code Index Scope — the full list of helpers (Graph, ToIndex, ChatAI, Logger, …) available inside the body.
• Creating Endpoints — endpoint authoring, modes, and authorization.
• Multimodal Search — supported formats and OCR limits.