Cookbook

Inspect partial-failure errors after an import

This recipe shows how to inspect the per-row errors of an async import after the worker finishes. The status endpoint surfaces a short sample on every read; the paged-errors mode of the same endpoint streams every error with row number, error code, error message, and the raw row payload.

When the partial / failed status arrives

A single bad row never aborts an import — Waafir's worker validates each row independently, persists every valid one, and writes a per-row error record for every bad one. The import then settles into one of two terminal statuses:

partial — at least one row succeeded and at least one row failed. The valid rows landed; the invalid ones are queryable via this recipe.
failed — zero rows succeeded. Either every row was invalid, or the import hit a fatal error (malformed NDJSON, S3 read failure, worker exception); the errorSummary carries a fatal-shape error in that case.

For both terminal states, the recipe below is the right tool to drill in.

The shape of the error surface

The status endpoint (GET /api/v1/monitoring/loan/imports/{id}) returns an errorSummary field with two shapes depending on what happened:

Row-level errors (the common case for partial and most failed runs): errorSummary is { samples, totals }. The samples array carries the first ten errors as { rowNumber, errorCode, errorMessage, rawRow }. The totals field surfaces an aggregate count.
Fatal errors (an import that died before processing any row): errorSummary is { fatal } — a single error describing the worker-level failure.

For more than the first ten errors, walk the paged-errors mode: GET /api/v1/monitoring/loan/imports/{id}?errors=true&limit=&offset=.

The recipe

# curl — inspect a partial / failed import's errors
set -euo pipefail

PAT="${WAAFIR_PAT}"
BASE_URL="${WAAFIR_BASE_URL:-https://app.waafir.io}"
IMPORT_ID="${1:?supply the import id}"

# Step 1 — fetch the terminal status, see the error summary.
STATUS=$(curl -fsS "${BASE_URL}/api/v1/monitoring/loan/imports/${IMPORT_ID}" \
  -H "Authorization: Bearer ${PAT}")

echo "status:" && echo "$STATUS" | jq '{status, rowCountProcessed, rowCountSucceeded, rowCountFailed}'
echo "first ten errors (sampled):"
echo "$STATUS" | jq '.errorSummary'

# Step 2 — page through every row error. Limit/offset are bounded
# (limit max 100; default 20).
LIMIT=100
OFFSET=0
TOTAL_FAILED=$(echo "$STATUS" | jq -r .rowCountFailed)

while [ "${OFFSET}" -lt "${TOTAL_FAILED}" ]; do
  PAGE=$(curl -fsS \
    "${BASE_URL}/api/v1/monitoring/loan/imports/${IMPORT_ID}?errors=true&limit=${LIMIT}&offset=${OFFSET}" \
    -H "Authorization: Bearer ${PAT}")
  echo "$PAGE" | jq -c '.items[] | {rowNumber, errorCode, errorMessage}'
  OFFSET=$((OFFSET + LIMIT))
done

// TypeScript — inspect a partial / failed import's errors
const PAT = process.env.WAAFIR_PAT!;
const BASE_URL = process.env.WAAFIR_BASE_URL ?? "https://app.waafir.io";
const importId = process.argv[2];
if (!importId) throw new Error("usage: ts-node inspect.ts <importId>");

const auth = { Authorization: `Bearer ${PAT}` };

// Step 1 — terminal status + error summary.
const statusRes = await fetch(`${BASE_URL}/api/v1/monitoring/loan/imports/${importId}`, {
    headers: auth,
});
type Status = {
    status: string;
    rowCountProcessed: number;
    rowCountSucceeded: number;
    rowCountFailed: number;
    errorSummary: unknown;
};
const status = (await statusRes.json()) as Status;
console.log("status:", {
    status: status.status,
    rowCountProcessed: status.rowCountProcessed,
    rowCountSucceeded: status.rowCountSucceeded,
    rowCountFailed: status.rowCountFailed,
});
console.log("error summary (first ten):", status.errorSummary);

// Step 2 — page through every row error.
type RowError = {
    rowNumber: number;
    errorCode: string;
    errorMessage: string;
    rawRow: unknown;
    createdAt: string;
};
const limit = 100;
for (let offset = 0; offset < status.rowCountFailed; offset += limit) {
    const url = `${BASE_URL}/api/v1/monitoring/loan/imports/${importId}?errors=true&limit=${limit}&offset=${offset}`;
    const pageRes = await fetch(url, { headers: auth });
    const page = (await pageRes.json()) as { items: RowError[]; limit: number; offset: number };
    for (const error of page.items) {
        console.log({
            rowNumber: error.rowNumber,
            errorCode: error.errorCode,
            errorMessage: error.errorMessage,
        });
    }
}

# python — inspect a partial / failed import's errors
import os
import sys
import requests

PAT = os.environ["WAAFIR_PAT"]
BASE_URL = os.environ.get("WAAFIR_BASE_URL", "https://app.waafir.io")
import_id = sys.argv[1] if len(sys.argv) > 1 else None
if not import_id:
    raise SystemExit("usage: python inspect.py <importId>")

auth = {"Authorization": f"Bearer {PAT}"}

# Step 1 — terminal status + error summary.
status = requests.get(
    f"{BASE_URL}/api/v1/monitoring/loan/imports/{import_id}",
    headers=auth,
    timeout=30,
).json()
print("status:", {
    "status": status["status"],
    "rowCountProcessed": status["rowCountProcessed"],
    "rowCountSucceeded": status["rowCountSucceeded"],
    "rowCountFailed": status["rowCountFailed"],
})
print("error summary (first ten):", status.get("errorSummary"))

# Step 2 — page through every row error.
LIMIT = 100
offset = 0
total_failed = status["rowCountFailed"]
while offset < total_failed:
    page = requests.get(
        f"{BASE_URL}/api/v1/monitoring/loan/imports/{import_id}",
        params={"errors": "true", "limit": LIMIT, "offset": offset},
        headers=auth,
        timeout=30,
    ).json()
    for error in page["items"]:
        print({
            "rowNumber": error["rowNumber"],
            "errorCode": error["errorCode"],
            "errorMessage": error["errorMessage"],
        })
    offset += LIMIT

What to verify

For a partial import, the sum of rowCountSucceeded and rowCountFailed equals rowCountProcessed, and rowCountSucceeded is non-zero. For a failed import, rowCountSucceeded is zero — every row that was reachable failed validation, or the errorSummary.fatal field describes a worker-level failure.

The rowNumber field is the one-indexed line in your NDJSON file. Map it back to your source rows to find the offending records, fix them, and resubmit either the corrected rows alone or the whole file (use a fresh idempotency key for the new intent — see the retry recipe).

The rawRow field carries the parsed payload as Waafir received it (so you can verify the wire shape matches what your code generated); the errorCode is a stable identifier for the validation rule that fired (so you can match groups of errors against a fix-list).

This recipe anchors the friction scenario import-partial-failure-row-errors — the catalog entry pins the contract that mixed valid-and-invalid imports land every valid row, persist one error record per bad row, mirror the first ten into error_summary.samples, and transition to partial.

Retry safely with an idempotency key

Send the same import request twice without creating duplicate work — the Idempotency-Key header makes the start-import endpoint replay the first call's result on every retry.

Cancel an in-flight import

Stop a long-running async import at the next batch boundary — the cancel endpoint flips the worker to the cancelled state without leaving partially-written rows.