Serving Derived Data: How Should Batch Output Reach Production?

Serving Derived Data: How Should Batch Output Reach Production? A workflow diagram generated by Archify. 01 / Decision EX / Avoid — Direct Writes 03 / Full Rebuild Paths 04 / Incremental Runs 05 / Streaming Paths Batch Output Ready · derived dataset on disk · Decision · serve it to prod Batch Output Ready derived dataset on disk serve it to prod Write Direct to Prod DB? · per-record network calls · Decision Write Direct to Prod DB? per-record network calls Rebuild or Bootstrap? · full dataset from scratch · Decision Rebuild or Bootstrap? full dataset from scratch Bulk-Import Tool? · bulk-load API or job · Decision Bulk-Import Tool? bulk-load API or job Direct DB Writes · overloads DB, leaks output · Avoid — Direct Writes · duplicates on retry Direct DB Writes overloads DB, leaks output duplicates on retry Publish Full Snapshot · to a stream topic · Full Rebuild Paths · ingest at own pace Publish Full Snapshot to a stream topic ingest at own pace Bulk-Load Files + Swap · then atomic version swap · Full Rebuild Paths · fastest, hard to update Bulk-Load Files + Swap then atomic version swap fastest, hard to update All-or-Nothing per Run? · no partial output visible · Incremental Runs All-or-Nothing per Run? no partial output visible Stream + Completion Notice · notify when the job is done · Streaming Paths · invisible until notified Stream + Completion Notice notify when the job is done invisible until notified Stream Incremental Rows · updates as they are produced · Streaming Paths · consumers throttle reads Stream Incremental Rows updates as they are produced consumers throttle reads yes no yes no yes no no yes assess Legend Agent logic Tool action Context / trace External system