Waafir
Cookbook

Retry safely with an idempotency key

This recipe shows how to retry the start-import call safely after a network failure or an ambiguous error. The Idempotency-Key header on POST /api/v1/monitoring/loan/imports makes the second call return the same import ID as the first, with a freshly-minted upload URL — no duplicate import is created, no second S3 staging object is allocated.

When to use this

Whenever you cannot tell whether the previous start-import call landed. The two canonical cases:

  • Your HTTP client raised a timeout before the response arrived. The request may have reached Waafir and started an import; you do not know.
  • You got a 5xx error from a non-idempotent surface and want to retry without risking duplicate work.

The platform-wide convention is that every mutating endpoint accepts Idempotency-Key — the start-import endpoint here, the export endpoint, the audit-mutation endpoints elsewhere. Use a fresh key per logical operation; reuse it across retries of the same operation. A common pattern is a UUID v4 generated once per "daily-import-batch" intent and kept in your job state until the call succeeds.

The contract

  • The header is Idempotency-Key: <your-string>. Maximum 255 characters.
  • The body also accepts an idempotencyKey field; the header takes precedence when both are present. Prefer the header — it is the canonical platform-wide placement.
  • A repeat call with the same (orgId, idempotencyKey) returns:
    • the same importId,
    • reused: true in the response,
    • a freshly-minted presigned upload URL (the original URL expires; the new one is the safe one to use).
  • A repeat call with a different body but the same key is rejected; the key is bound to the original request shape.

The recipe

# curl — start an import with an idempotency key, then retry safely
set -euo pipefail

PAT="${WAAFIR_PAT}"
BASE_URL="${WAAFIR_BASE_URL:-https://app.waafir.io}"
NDJSON_FILE="loans-2026-05-22.ndjson"
# Generate a fresh key per logical batch intent; keep it stable across retries.
IDEMPOTENCY_KEY="loan-import-2026-05-22-$(uuidgen)"

start_import() {
  curl -fsS -X POST "${BASE_URL}/api/v1/monitoring/loan/imports" \
    -H "Authorization: Bearer ${PAT}" \
    -H "Content-Type: application/json" \
    -H "Idempotency-Key: ${IDEMPOTENCY_KEY}" \
    -d "$(cat <<JSON
{
  "kind": "loan",
  "filename": "${NDJSON_FILE}",
  "sizeBytes": $(stat -c%s "${NDJSON_FILE}")
}
JSON
)"
}

# First call. If the network bombs out we have no idea whether the
# request landed — but the same key on the next attempt guarantees no
# duplicate import.
RESULT=$(start_import || true)

if [ -z "${RESULT}" ] || ! echo "${RESULT}" | jq -e .importId > /dev/null; then
  echo "first call ambiguous; retrying with the same idempotency key…"
  RESULT=$(start_import)
fi

echo "${RESULT}" | jq '{importId, reused, uploadUrl}'
# On a retry, reused == true; the importId matches the first call's; the
# uploadUrl is fresh.
// TypeScript — retry start-import safely with an Idempotency-Key
import { randomUUID } from "node:crypto";
import { statSync } from "node:fs";

const PAT = process.env.WAAFIR_PAT!;
const BASE_URL = process.env.WAAFIR_BASE_URL ?? "https://app.waafir.io";
const NDJSON_FILE = "loans-2026-05-22.ndjson";
const idempotencyKey = `loan-import-2026-05-22-${randomUUID()}`;

async function startImport() {
    return fetch(`${BASE_URL}/api/v1/monitoring/loan/imports`, {
        method: "POST",
        headers: {
            Authorization: `Bearer ${PAT}`,
            "Content-Type": "application/json",
            "Idempotency-Key": idempotencyKey,
        },
        body: JSON.stringify({
            kind: "loan",
            filename: NDJSON_FILE,
            sizeBytes: statSync(NDJSON_FILE).size,
        }),
    });
}

// Retry up to 3 times on transient failures; the same key makes this safe.
let lastError: unknown;
let result: { importId: string; reused: boolean; uploadUrl: string } | null = null;
for (let attempt = 1; attempt <= 3; attempt++) {
    try {
        const res = await startImport();
        if (!res.ok) throw new Error(`HTTP ${res.status}: ${await res.text()}`);
        result = (await res.json()) as typeof result;
        break;
    } catch (err) {
        lastError = err;
        console.warn(`attempt ${attempt} failed; retrying with same idempotency key`, err);
        await new Promise((r) => setTimeout(r, 1_000 * attempt));
    }
}

if (!result) throw lastError;
console.log({ importId: result.importId, reused: result.reused, uploadUrl: result.uploadUrl });
# python — retry start-import safely with an Idempotency-Key
import os
import uuid
import time
import requests
from pathlib import Path

PAT = os.environ["WAAFIR_PAT"]
BASE_URL = os.environ.get("WAAFIR_BASE_URL", "https://app.waafir.io")
NDJSON_FILE = Path("loans-2026-05-22.ndjson")

idempotency_key = f"loan-import-2026-05-22-{uuid.uuid4()}"

def start_import():
    return requests.post(
        f"{BASE_URL}/api/v1/monitoring/loan/imports",
        headers={
            "Authorization": f"Bearer {PAT}",
            "Content-Type": "application/json",
            "Idempotency-Key": idempotency_key,
        },
        json={
            "kind": "loan",
            "filename": NDJSON_FILE.name,
            "sizeBytes": NDJSON_FILE.stat().st_size,
        },
        timeout=30,
    )

# Retry up to 3 times on transient errors. The same key makes a second
# successful call a guaranteed replay of the first.
last_exc = None
result = None
for attempt in range(1, 4):
    try:
        response = start_import()
        response.raise_for_status()
        result = response.json()
        break
    except requests.RequestException as exc:
        last_exc = exc
        print(f"attempt {attempt} failed; retrying with same idempotency key: {exc}")
        time.sleep(attempt)

if result is None:
    raise RuntimeError("all retries exhausted") from last_exc

print({
    "importId": result["importId"],
    "reused": result["reused"],
    "uploadUrl": result["uploadUrl"],
})

What to verify

On a successful first call, reused is false. On any subsequent call with the same key, reused is true and importId equals the first call's. The uploadUrl is regenerated on every call so retrying after a stale URL expires is safe — always use the URL from the most recent response.

If you retry with a different request body and the same key, the API returns an error indicating the key is bound to a conflicting request shape; rotate the key for the new intent.

This recipe anchors the friction scenario import-idempotency-replay — the catalog entry pins the contract that a second start-import call with the same (orgId, idempotencyKey) returns the original row ID with no second S3 PUT side effect.

Validate a load with dry-run before committing data

Use the dry-run mode of the async import endpoint to validate an NDJSON file end to end — schema, enum, and row-level checks — without writing any silver rows.

Inspect partial-failure errors after an import

Read the error summary and page through per-row errors when an async import lands as partial or failed — pinpoint the offending rows by row number, error code, and raw payload.