Incremental sync
A full reload every run is expensive. Use a cursor to read only what changed.
var cursor = await state.LoadCursorAsync() ?? DateTimeOffset.MinValue;
await foreach (var row in source.ReadModifiedSinceAsync(cursor))
{
// TryAdd · Link · RestrictAccess ...
if (++count % 500 == 0) await graph.CommitPendingAsync();
}
await graph.CommitPendingAsync();
await state.SaveCursorAsync(DateTimeOffset.UtcNow); // ← only after a successful commit
Sync strategies:
| Strategy | Use when |
|---|---|
| Full refresh | Small datasets; source has no timestamps |
| Watermark incremental | Source has a reliable updated_at — most common |
| Change feed | Source has a native event stream (Kafka, CDC, webhooks) |
| Reconciliation pass | Needed to catch deletes and data drift |
Deletes: the graph doesn't auto-remove nodes when source records disappear. Handle with:
- A soft-delete flag on the source
- A periodic reconciliation scan (
graph.RemoveNode(uid)for missing records) - Delete events from a source webhook