Curiosity

Multimodal Search

Curiosity Studio extracts searchable text from images, scanned PDFs, audio, and video. Once extracted, the text becomes a regular searchable property — text search, vector search, NLP pipelines, and graph queries all see it like any other field.

This page is the reference for what's supported, what the limits are, and what to expect operationally. For a runnable walkthrough see Multimodal tutorial.

Optical Character Recognition (OCR)

OCR runs automatically on images and scanned PDFs, and is on by default. The recognized text is stored with the file, page by page, beside a Markdown version of the same pages. The text and embedding indexes read it the same way they read other text. For how a page is routed between the three OCR engines (NanoOCR, Florence2 and PaddleOCR-VL), see OCR & File Extraction.

Supported formats

Class Extensions
Raster images .png, .jpg, .jpeg, .gif, .bmp, .webp, .tif, .tiff
Camera and phone formats .heic, .heif, .dng, .raw
Scanned PDFs .pdf files with no text layer, or a text layer that cannot be read. PDFs with a readable text layer are extracted directly, with no OCR.

A multi-page TIFF is read page by page. An animated GIF is read by its first frame.

Languages

There are no OCR language packs. NanoOCR, which reads every page first, reads printed Latin-script languages (about 50), Cyrillic, Greek, Devanagari, Bengali, and Chinese. PaddleOCR-VL, once downloaded, reads the pages NanoOCR reads with low confidence, and its model card lists 109 languages. See OCR & File Extraction → Languages.

Limits and performance

Limit Value Notes
NanoOCR Under a second per page on a CPU Built in, no download.
PaddleOCR-VL About a minute or two per dense page on a few CPU cores At most five minutes per page.
Long scanned PDFs More than 100 pages, or 20 once PaddleOCR-VL is installed Re-queued at the lowest priority, so shorter files are extracted first.
Pictures inside documents Off by default Up to 16 pictures per document, two minutes each. Needs PaddleOCR-VL.

What to expect by kind of page

  • Printed text (clean scans, exported pages) is read by NanoOCR, with or without the optional models.
  • Tables and forms are read by PaddleOCR-VL once it is installed, and their structure is kept in the Markdown.
  • Handwriting is read by PaddleOCR-VL once it is installed. Without it, a handwritten page keeps NanoOCR's reading, which is trained on printed text.
  • Photos (text in a scene, a photographed receipt) are read by Florence2 once it is installed.
  • Degraded scans that NanoOCR reads with low confidence go to PaddleOCR-VL once it is installed.

Without the optional models, every page keeps NanoOCR's reading. Download them from Manage → AI → Models or from Manage → Data → File Indexing → AI Processing; see Installing the OCR models.

Speech-to-Text (STT)

STT transcribes audio and video files. The transcript is stored on the file node alongside word-level timestamps so the UI can deep-link to the relevant moment in a media file.

Supported formats

Class Extensions
Video .mp4, .wmv, .mpeg, .avi, .mkv, .mov, .ogv, .3gp, .flv
Audio .mp3, .wav, .mka, .wma, .flac, .aac, .aiff, .m4a, .oga, .weba, .webm

Models

The workspace uses Whisper for transcription. Pick the model size per workspace based on the accuracy/cost tradeoff:

Whisper model Approx. RTF (CPU) Approx. RTF (GPU) Quality
tiny 0.3× 0.05× Acceptable for clean speech.
base 0.5× 0.07× Good for monologue / dictation.
small 1.0× 0.10× Solid default.
medium 2.5× 0.20× Strong, handles accents.
large-v3 6.0× 0.30× Best, multilingual.

RTF = real-time factor (time to transcribe ÷ media duration). 0.1× means 1 hour of audio takes 6 minutes.

Features

  • Searchable transcripts. Find spoken phrases in any media file.
  • Timestamped navigation. Search hits link to the segment offset; the player jumps directly there.
  • Language detection. Auto-detected per file. Force a language in the admin UI for known mono-lingual corpora to skip the detection step.
  • Speaker diarization (optional). Mark up speaker turns. Disabled by default — turn on per workspace if needed.

Limits and performance

Limit Default Notes
Max file size 2 GB Configurable. Larger files split into segments.
Max duration 8 hours
Word-level timestamps enabled Disable for marginal throughput gains on long media.
Concurrent workers 1 per GPU / 4 per CPU Configure per worker pool.

Cost and capacity notes

  • OCR runs inside the workspace. Every OCR engine runs on the workspace's own machine; there is no external OCR service. Speech to text also runs locally unless the Azure speech keys (MSK_AZURE_STT_SUBSCRIPTIONKEY, MSK_AZURE_STT_ENDPOINT) are set.
  • GPU workers move the needle. For STT especially, GPU throughput is 10–30× CPU.
  • Schedule big backfills off-hours. A back-catalog of 100 000 PDF scans will saturate a CPU worker for days. Plan capacity ahead.
  • Embeddings on multimodal text. Once extracted, the text is just text — and embeddings cost the usual per-token amount. Disable embedding on short OCR fields (single labels, captions) to save on rebuild costs.

Retrieval shape

After extraction, multimodal content participates in normal retrieval. Two extras:

  • SearchHit.Highlights[] includes the transcript fragment that matched (text) or the OCR span that matched (image-with-bounding-box on supported viewers).
  • ChildHit carries the page/timestamp identifier — use it to deep-link from the hit list to "page 7" or "minute 32".

See Search DSL for the full hit shape.