REST API recipe

Source: RestApiSample/ · cursor-paginated REST API with bearer-token auth. Falls back to local JSON pages when no API URL is configured.

Owns in the academic graph: courses, terms, instructors.

What it teaches

• Cursor pagination ({ items: [...], nextCursor: "..." | null } envelope) — preferred over offset.
• Bearer-token auth via the Authorization: Bearer <token> header.
• Rate-limit handling — honors 429 Too Many Requests with Retry-After, exponential backoff on 5xx.
• A single IRestApiSource interface implemented by both HttpRestApiSource and LocalRestApiSource, so the ingestion code doesn't care where pages come from.

Driving the source

IRestApiSource source = string.IsNullOrWhiteSpace(apiBaseUrl)
    ? new LocalRestApiSource(localRoot)
    : new HttpRestApiSource(apiBaseUrl, apiBearer);

await foreach (var course in source.StreamAsync<CoursesIngest.CourseDoc>("courses"))
{
    CoursesIngest.Ingest(graph, course);
}

Under the hood HttpRestApiSource:

• Honors Retry-After: <seconds> and Retry-After: <HTTP-date>.
• Backs off exponentially (cap 60 s) on 429 and 5xx.
• Threads nextCursor into the ?cursor=… query param across pages.

Configuration

Reuse notes

• Same IRestApiSource contract works with offset paging, link-header paging, page numbers — change only the source class.
• LocalRestApiSource uses the page-1.json, page-2.json naming convention — handy for unit-style testing of the ingest logic without hitting the network.