Waafir
Cookbook

Bulk import a loan tape via async NDJSON

This recipe shows the foundational ingestion flow for a Waafir loan portfolio: submit a newline-delimited-JSON (NDJSON) file containing one row per loan-tape entity to the async-import endpoint, then poll the import's status until the worker finishes.

When to use this flow

The async-NDJSON-via-presigned-S3 flow is the canonical ingestion path for any non-trivial volume of rows. It scales to historical backloads of millions of rows and is the same flow used for incremental daily submissions. The cookbook uses a loan import in the examples below; the same flow accepts borrower, scheduled_repayment, actual_payment, collateral, and collateral_valuation kinds — change the kind field and the row shape and everything else is identical.

The shape of the flow

There are three steps your code must take, and one step your file does in between:

  1. POST /api/v1/monitoring/loan/imports — declare what you are about to upload (kind, filename, size). The response includes a presigned S3 upload URL and the import ID.
  2. PUT the NDJSON body to the presigned URL — your file is staged in Waafir's bronze archive (raw, immutable).
  3. POST /api/v1/monitoring/loan/imports/{id}/commit — tell Waafir the upload finished. The import moves from pending_upload into queued, and the worker begins processing batches.
  4. GET /api/v1/monitoring/loan/imports/{id} — poll until status is one of succeeded, partial, failed, or cancelled.

The recipe

# curl — bulk import flow (loan kind)
set -euo pipefail

PAT="${WAAFIR_PAT}"
BASE_URL="${WAAFIR_BASE_URL:-https://app.waafir.io}"
NDJSON_FILE="loans-2026-05-22.ndjson"

# Step 1 — start the import and capture the presigned URL + import ID.
START=$(curl -fsS -X POST "${BASE_URL}/api/v1/monitoring/loan/imports" \
  -H "Authorization: Bearer ${PAT}" \
  -H "Content-Type: application/json" \
  -d "$(cat <<JSON
{
  "kind": "loan",
  "filename": "${NDJSON_FILE}",
  "sizeBytes": $(stat -c%s "${NDJSON_FILE}")
}
JSON
)")

IMPORT_ID=$(echo "$START" | jq -r .importId)
UPLOAD_URL=$(echo "$START" | jq -r .uploadUrl)

# Step 2 — PUT the NDJSON body to the presigned URL. The `headers` field
# on the start response carries the headers the presigned URL was signed
# with — apply them all, or S3 rejects the PUT with a signature
# mismatch. At minimum `Content-Type: application/x-ndjson`; production
# tenants also receive `x-amz-server-side-encryption*` headers.
HEADER_ARGS=$(echo "$START" | jq -r '.headers | to_entries[] | "-H \"\(.key): \(.value)\""')
eval curl -fsS -X PUT --data-binary @"\"${NDJSON_FILE}\"" $HEADER_ARGS "\"${UPLOAD_URL}\""

# Step 3 — commit so the worker enqueues the import.
curl -fsS -X POST "${BASE_URL}/api/v1/monitoring/loan/imports/${IMPORT_ID}/commit" \
  -H "Authorization: Bearer ${PAT}"

# Step 4 — poll status until it reaches a terminal value.
while :; do
  STATUS=$(curl -fsS "${BASE_URL}/api/v1/monitoring/loan/imports/${IMPORT_ID}" \
    -H "Authorization: Bearer ${PAT}" | jq -r .status)
  echo "import ${IMPORT_ID}: ${STATUS}"
  case "${STATUS}" in
    succeeded|partial|failed|cancelled) break ;;
  esac
  sleep 5
done
// TypeScript — bulk import flow (loan kind)
import { readFileSync, statSync } from "node:fs";

const PAT = process.env.WAAFIR_PAT!;
const BASE_URL = process.env.WAAFIR_BASE_URL ?? "https://app.waafir.io";
const NDJSON_FILE = "loans-2026-05-22.ndjson";

const headers = { Authorization: `Bearer ${PAT}`, "Content-Type": "application/json" };

// Step 1 — start the import.
const startRes = await fetch(`${BASE_URL}/api/v1/monitoring/loan/imports`, {
    method: "POST",
    headers,
    body: JSON.stringify({
        kind: "loan",
        filename: NDJSON_FILE,
        sizeBytes: statSync(NDJSON_FILE).size,
    }),
});
if (!startRes.ok) throw new Error(`start failed: ${startRes.status} ${await startRes.text()}`);
const { importId, uploadUrl, headers: uploadHeaders } = (await startRes.json()) as {
    importId: string;
    uploadUrl: string;
    headers: Record<string, string>;
};

// Step 2 — PUT the NDJSON body to the presigned URL with the headers
// the URL was signed against. Skipping them causes S3 to reject the
// upload with a signature mismatch in production tenants where SSE-KMS
// is enabled.
const putRes = await fetch(uploadUrl, {
    method: "PUT",
    headers: uploadHeaders,
    body: readFileSync(NDJSON_FILE),
});
if (!putRes.ok) throw new Error(`upload failed: ${putRes.status}`);

// Step 3 — commit.
const commitRes = await fetch(`${BASE_URL}/api/v1/monitoring/loan/imports/${importId}/commit`, {
    method: "POST",
    headers: { Authorization: `Bearer ${PAT}` },
});
if (!commitRes.ok) throw new Error(`commit failed: ${commitRes.status} ${await commitRes.text()}`);

// Step 4 — poll.
const TERMINAL = new Set(["succeeded", "partial", "failed", "cancelled"]);
while (true) {
    const statusRes = await fetch(`${BASE_URL}/api/v1/monitoring/loan/imports/${importId}`, {
        headers: { Authorization: `Bearer ${PAT}` },
    });
    const body = (await statusRes.json()) as { status: string };
    console.log(`import ${importId}: ${body.status}`);
    if (TERMINAL.has(body.status)) break;
    await new Promise((r) => setTimeout(r, 5_000));
}
# python — bulk import flow (loan kind)
import os
import time
import requests
from pathlib import Path

PAT = os.environ["WAAFIR_PAT"]
BASE_URL = os.environ.get("WAAFIR_BASE_URL", "https://app.waafir.io")
NDJSON_FILE = Path("loans-2026-05-22.ndjson")

headers = {"Authorization": f"Bearer {PAT}", "Content-Type": "application/json"}

# Step 1 — start the import.
start = requests.post(
    f"{BASE_URL}/api/v1/monitoring/loan/imports",
    headers=headers,
    json={
        "kind": "loan",
        "filename": NDJSON_FILE.name,
        "sizeBytes": NDJSON_FILE.stat().st_size,
    },
    timeout=30,
)
start.raise_for_status()
payload = start.json()
import_id = payload["importId"]
upload_url = payload["uploadUrl"]
upload_headers = payload["headers"]  # signed-into the URL; must be sent on PUT

# Step 2 — PUT the NDJSON body to the presigned URL with the headers
# the URL was signed against. Skipping them causes S3 to reject the
# upload with a signature mismatch in production tenants where SSE-KMS
# is enabled.
with NDJSON_FILE.open("rb") as fh:
    put = requests.put(upload_url, data=fh, headers=upload_headers, timeout=300)
    put.raise_for_status()

# Step 3 — commit.
commit = requests.post(
    f"{BASE_URL}/api/v1/monitoring/loan/imports/{import_id}/commit",
    headers={"Authorization": f"Bearer {PAT}"},
    timeout=30,
)
commit.raise_for_status()

# Step 4 — poll.
terminal = {"succeeded", "partial", "failed", "cancelled"}
while True:
    status = requests.get(
        f"{BASE_URL}/api/v1/monitoring/loan/imports/{import_id}",
        headers={"Authorization": f"Bearer {PAT}"},
        timeout=30,
    ).json()
    print(f"import {import_id}: {status['status']}")
    if status["status"] in terminal:
        break
    time.sleep(5)

What to verify

When status reaches succeeded, every row in your NDJSON file landed as a live silver row. The response also surfaces rowCountProcessed, rowCountSucceeded, and rowCountFailed; on a clean run the succeeded count equals the row count of your file.

When status reaches partial, some rows landed and some did not. Walk to the partial-failure inspection recipe to page through the per-row errors and decide whether to fix the offending rows and resubmit, or accept the partial result.

A failed status means zero rows landed. The errorSummary.samples field carries the first ten error payloads; the rest are queryable via the paged-errors endpoint described in the partial-failure recipe.

This recipe is the success-path companion to the friction scenario import-fatal-error-failed-status — the failed-on-every-row case is what the catalog covers, and the happy path here is its inverse.

Cookbook

Practical recipes for integrating a fintech's loan-tape portfolio with Waafir's monitoring API — bulk ingestion, dry-run validation, idempotent retries, partial-failure inspection, cancellation, and reconciliation, with cURL · TypeScript · Python side by side.

Validate a load with dry-run before committing data

Use the dry-run mode of the async import endpoint to validate an NDJSON file end to end — schema, enum, and row-level checks — without writing any silver rows.