Data pipelines for AI: sync changes with recovery

Updates must also survive failures

An AI product can combine documents, records and events from several systems. Reprocessing everything each cycle costs database work, network transfer and processing. Publishing only new records is also insufficient: corrections and removals must reach the destination.

The flow must know what changed and which version the destination confirmed. If a transfer fails, the difference remains pending. If confirmation state disappears, a full load must not start without checking the cause and cost.

Use identity, version and removal records

In a data platform I implemented, processing compares content digests and writes differences in batches. Removing a source creates an explicit removal record. The source record comes before the search projection, which can be reconciled later.

This separation avoids treating the search index as the only copy of the data. It also distinguishes a version not yet published from a removed record. The consumer must apply the version repeatedly without duplicating its effect.

Bound each cycle and confirm after writing

In another enrichment flow, reads fix an end time and follow changes ordered by update time and identifier. A small overlap covers delays near the boundary. The recovery position advances only after batches have been written.

The overlap has limited coverage. Longer delays need planned reconciliation. Also define row, memory, concurrency and cost limits for each cycle. An alert dashboard does not by itself stop a load that exceeded its budget.

Make AI data supply a controlled operation

Consulting can review collection, preparation, synchronisation and publication. Deliverables can include recovery, reconciliation, delay monitoring and executable budgets. The proposal defines source data, destinations and the delay acceptable to the business.