Conveyor — proposed ingestion & storage architecture

1Target architecture

flowchart TB
  C(["Client / SDK"])
  C -->|"submit (small) · 201"| API
  C -->|"presign + PUT large list"| S3S[("S3 — staged task lists")]
  API["API · Fargate · stateless
(decides sync vs async)"]
  API -->|"large/open: 202 + ingest msg"| CQ[["SQS — control queue"]]
  API -->|"small closed: sync write"| DDB
  API -->|"leases · counters · hot-read cache"| RDS[("Redis / MemoryDB
coordination")]
  CQ --> IW["Ingest worker
(stream · write · enqueue · latch)"]
  IW -->|"read chunks"| S3S
  IW -->|"write tasks
(shard only if large)"| DDB[("DynamoDB
jobs · runs · tasks")]
  IW -->|"enqueue refs"| RQ[["SQS — per-run queues"]]
  CQ -. "permanent fail" .-> DLQ[["DLQ + reconcile sweeper"]]
  DLQ -.-> DDB
  RQ --> TW["Task workers · Fargate"]
  TW -->|"scrape"| GW(["gateway"])
  TW -->|"result bodies"| S3R[("S3 — results")]
  TW -->|"stats INCR"| RDS
  TW -->|"terminal status"| DDB
  TW -->|"on run complete"| NOT["Step Functions / webhooks"]
  EXP["Results export (offline)"] -->|"full-set reads → zip"| S3R

Each arrow is a job for the engine best at it: S3 absorbs the giant body, SQS is the durable buffer, Redis owns hot coordination, DynamoDB stays the point-read/TTL system of record, the gateway/result flow is unchanged. The request thread returns in milliseconds.

2Submit decides sync vs async

flowchart TD
  S["POST /v1/jobs"] --> Q{"size & type?"}
  Q -->|"small closed
(tasks < threshold)"| SY["sync: write + enqueue
→ 201, single-partition"]
  Q -->|"large closed / open / async"| AS["stage list → 202
+ ingest_job message"]
  SY --> RS["tasks queryable now
cheap single-Query reads"]
  AS --> RA["tasks appear after ingest
sharded writes · poll job status"]

Today both ceilings are hit because submit is always synchronous and every job is stamped shard_count=16. Here, sharding and async are switched on only when the job is actually large or open — so the common small job keeps the old, fast, single-partition path (the regression we found in #117).

3Ingestion pipeline — durable, resumable

flowchart LR
  A["ingest_job (SQS)"] --> B["stream staged
source, chunk N"]
  B --> C["write tasks
deterministic ids"]
  C --> D["enqueue refs"]
  D --> E{"more
chunks?"}
  E -->|yes| B
  E -->|no| F["SetRunTotal
(absolute, idempotent)"]
  F --> G["latch last_batch"]
  G --> H["run completes via
normal lifecycle"]

At-least-once safe (this is #120's core, already tested): deterministic task ids (sha256(run_id:index)) make re-writes overwrite not duplicate; absolute SetRunTotal (not ADD) can't double-count and is set before the latch so completion never fires early; re-enqueued refs are deduped by the task worker's status guard. A crash at any step resumes on redelivery and converges; permanent failures land in a DLQ with the run left recoverable via /rerun or DELETE.

4State on the right engine (polyglot)

5Read paths — bounded by design

• Point reads (GetTask, history) — one partition, derived from task_id. Cheap, unchanged.
• Run-scan (results listing) — single Query for small jobs; bounded N-way fan-out only for the large jobs that opted into sharding. Reads are the cheap side (3× the per-partition headroom of writes, eventually-consistent, parallel).
• Full-set reads (export, "give me every result") — go through the offline export op to S3, never a synchronous hot-path scan. This is what kills the scary /rerun "load 1M rows into memory under the 60 s wall" failure mode.

6What changes vs today

7Evolution path — ship in slices

8Why not the other options