unit U5 — 5 of 5

Files too big for memory

streaming, batching, checkpoints

Everything above assumed the file fits in memory. Past a certain size it does not, and the failure is abrupt: the worker is killed by the runtime with no stack trace and no run history entry explaining why. A 50 MB CSV parsed into objects can occupy several hundred megabytes; a 2 GB export cannot be loaded at all, on any instance you would want to pay for.

Stream instead. Read the file in chunks, process each row or record as it arrives, and write results incrementally, so memory stays flat regardless of file size. Batch the writes — accumulate a few hundred rows and write them in one call rather than one call per row — which is the same lesson as bounded parallelism, applied to I/O. Keep a checkpoint of how far you got, because a two-hour job that fails at 90% and has to restart from zero will fail there again.

Where it breaks: the timeout nobody accounted for. A flow step has a maximum duration, and a large file will exceed it long before it exceeds memory. Move long file work to a queued job rather than doing it inside the request or the step that received the file — the intake acknowledges, the job does the work, and progress is visible while it runs.

worked example

Streaming a large CSV with batched writes and a checkpoint, so memory stays flat and a failure resumes.

const BATCH = 500;

export const code = async (inputs) => {
  let batch = [];
  let processed = await loadCheckpoint(inputs.jobId);   // resume, do not restart

  for await (const row of streamCsvRows(inputs.storageKey, { skip: processed })) {
    batch.push(normalise(row));

    if (batch.length >= BATCH) {
      await writeBatch(batch);              // one call per 500 rows, not per row
      processed += batch.length;
      await saveCheckpoint(inputs.jobId, processed);
      batch = [];                           // memory stays flat, whatever the size
    }
  }

  if (batch.length) {
    await writeBatch(batch);
    processed += batch.length;
    await saveCheckpoint(inputs.jobId, processed);
  }

  return { processed };
};

field checklist

Stream large files; never load one fully into memory.
Batch the writes rather than writing per row.
Checkpoint progress so a failure resumes instead of restarting.
Move long file work to a queued job, out of the intake step.
Test with a file an order of magnitude larger than today's.

common failure — A worker killed with no stack trace

A nightly import loaded each CSV into memory to parse it. The files grew, and one night a 400 MB export pushed the worker past its limit; the runtime killed the process, so there was no error, no run history entry and no alert — the import had simply stopped happening. Stream the file, batch the writes, checkpoint progress, and run it as a queued job so a long import cannot be killed by a step timeout either.

check your understanding

A nightly CSV import began failing with no error, no run-history entry and no alert. File sizes had grown over several months. What happened?

sandbox validation

The check above confirms you followed the unit. Marking the module COMPLETED takes more: build the automation in your own engine and submit the exported flow and its run evidence, signed, to your unique validation URL. See the module page for that spec.