Waafir
Cookbook

Bulk import a loan tape via async NDJSON

This recipe shows the foundational ingestion flow for a Waafir loan portfolio: submit a newline-delimited-JSON (NDJSON) file containing one row per loan-tape entity to the async-import endpoint, then poll the import's status until the worker finishes.

When to use this flow

The async NDJSON import flow is the canonical ingestion path for any non-trivial volume of rows. It scales to historical backloads of millions of rows and is the same flow used for incremental daily submissions. The cookbook uses a loan import in the examples below; the same flow accepts borrower, scheduled_repayment, actual_payment, collateral, and collateral_valuation kinds — change the kind field and the row shape and everything else is identical.

The shape of the flow

There are three steps your code must take, and one step your file does in between:

  1. POST /api/v1/monitoring/loan/imports — declare what you are about to upload (kind, filename, size). The response includes an upload URL and the import ID.
  2. PUT the NDJSON body to the upload URL — your file is stored as submitted, unchanged.
  3. POST /api/v1/monitoring/loan/imports/{id}/commit — tell Waafir the upload finished. The import moves from pending_upload into queued, and the worker begins processing batches.
  4. GET /api/v1/monitoring/loan/imports/{id} — poll until status is one of succeeded, partial, failed, or cancelled.

The recipe

# curl — bulk import flow (loan kind)
set -euo pipefail

PAT="${WAAFIR_PAT}"
BASE_URL="${WAAFIR_BASE_URL:-https://app.waafir.io}"
NDJSON_FILE="loans-2026-05-22.ndjson"

# Step 1 — start the import and capture the upload URL + import ID.
START=$(curl -fsS -X POST "${BASE_URL}/api/v1/monitoring/loan/imports" \
  -H "Authorization: Bearer ${PAT}" \
  -H "Content-Type: application/json" \
  -d "$(cat <<JSON
{
  "kind": "loan",
  "filename": "${NDJSON_FILE}",
  "sizeBytes": $(stat -c%s "${NDJSON_FILE}")
}
JSON
)")

IMPORT_ID=$(echo "$START" | jq -r .importId)
UPLOAD_URL=$(echo "$START" | jq -r .uploadUrl)

# Step 2 — PUT the NDJSON body to the upload URL. The `headers` field
# on the start response carries the headers the upload URL was signed
# with — apply them all, or the upload is rejected with a signature
# mismatch. At minimum `Content-Type: application/x-ndjson`.
HEADER_ARGS=$(echo "$START" | jq -r '.headers | to_entries[] | "-H \"\(.key): \(.value)\""')
eval curl -fsS -X PUT --data-binary @"\"${NDJSON_FILE}\"" $HEADER_ARGS "\"${UPLOAD_URL}\""

# Step 3 — commit so the worker enqueues the import.
curl -fsS -X POST "${BASE_URL}/api/v1/monitoring/loan/imports/${IMPORT_ID}/commit" \
  -H "Authorization: Bearer ${PAT}"

# Step 4 — poll status until it reaches a terminal value.
while :; do
  STATUS=$(curl -fsS "${BASE_URL}/api/v1/monitoring/loan/imports/${IMPORT_ID}" \
    -H "Authorization: Bearer ${PAT}" | jq -r .status)
  echo "import ${IMPORT_ID}: ${STATUS}"
  case "${STATUS}" in
    succeeded|partial|failed|cancelled) break ;;
  esac
  sleep 5
done
// TypeScript — bulk import flow (loan kind)
import { readFileSync, statSync } from "node:fs";

const PAT = process.env.WAAFIR_PAT!;
const BASE_URL = process.env.WAAFIR_BASE_URL ?? "https://app.waafir.io";
const NDJSON_FILE = "loans-2026-05-22.ndjson";

const headers = { Authorization: `Bearer ${PAT}`, "Content-Type": "application/json" };

// Step 1 — start the import.
const startRes = await fetch(`${BASE_URL}/api/v1/monitoring/loan/imports`, {
    method: "POST",
    headers,
    body: JSON.stringify({
        kind: "loan",
        filename: NDJSON_FILE,
        sizeBytes: statSync(NDJSON_FILE).size,
    }),
});
if (!startRes.ok) throw new Error(`start failed: ${startRes.status} ${await startRes.text()}`);
const { importId, uploadUrl, headers: uploadHeaders } = (await startRes.json()) as {
    importId: string;
    uploadUrl: string;
    headers: Record<string, string>;
};

// Step 2 — PUT the NDJSON body to the upload URL with the headers
// the URL was signed against. Skipping them gets the upload
// rejected with a signature mismatch.
const putRes = await fetch(uploadUrl, {
    method: "PUT",
    headers: uploadHeaders,
    body: readFileSync(NDJSON_FILE),
});
if (!putRes.ok) throw new Error(`upload failed: ${putRes.status}`);

// Step 3 — commit.
const commitRes = await fetch(`${BASE_URL}/api/v1/monitoring/loan/imports/${importId}/commit`, {
    method: "POST",
    headers: { Authorization: `Bearer ${PAT}` },
});
if (!commitRes.ok) throw new Error(`commit failed: ${commitRes.status} ${await commitRes.text()}`);

// Step 4 — poll.
const TERMINAL = new Set(["succeeded", "partial", "failed", "cancelled"]);
while (true) {
    const statusRes = await fetch(`${BASE_URL}/api/v1/monitoring/loan/imports/${importId}`, {
        headers: { Authorization: `Bearer ${PAT}` },
    });
    const body = (await statusRes.json()) as { status: string };
    console.log(`import ${importId}: ${body.status}`);
    if (TERMINAL.has(body.status)) break;
    await new Promise((r) => setTimeout(r, 5_000));
}
# python — bulk import flow (loan kind)
import os
import time
import requests
from pathlib import Path

PAT = os.environ["WAAFIR_PAT"]
BASE_URL = os.environ.get("WAAFIR_BASE_URL", "https://app.waafir.io")
NDJSON_FILE = Path("loans-2026-05-22.ndjson")

headers = {"Authorization": f"Bearer {PAT}", "Content-Type": "application/json"}

# Step 1 — start the import.
start = requests.post(
    f"{BASE_URL}/api/v1/monitoring/loan/imports",
    headers=headers,
    json={
        "kind": "loan",
        "filename": NDJSON_FILE.name,
        "sizeBytes": NDJSON_FILE.stat().st_size,
    },
    timeout=30,
)
start.raise_for_status()
payload = start.json()
import_id = payload["importId"]
upload_url = payload["uploadUrl"]
upload_headers = payload["headers"]  # signed-into the URL; must be sent on PUT

# Step 2 — PUT the NDJSON body to the upload URL with the headers
# the URL was signed against. Skipping them gets the upload
# rejected with a signature mismatch.
with NDJSON_FILE.open("rb") as fh:
    put = requests.put(upload_url, data=fh, headers=upload_headers, timeout=300)
    put.raise_for_status()

# Step 3 — commit.
commit = requests.post(
    f"{BASE_URL}/api/v1/monitoring/loan/imports/{import_id}/commit",
    headers={"Authorization": f"Bearer {PAT}"},
    timeout=30,
)
commit.raise_for_status()

# Step 4 — poll.
terminal = {"succeeded", "partial", "failed", "cancelled"}
while True:
    status = requests.get(
        f"{BASE_URL}/api/v1/monitoring/loan/imports/{import_id}",
        headers={"Authorization": f"Bearer {PAT}"},
        timeout=30,
    ).json()
    print(f"import {import_id}: {status['status']}")
    if status["status"] in terminal:
        break
    time.sleep(5)

What to verify

When status reaches succeeded, every row in your NDJSON file landed as a live row. The response also surfaces rowCountProcessed, rowCountSucceeded, and rowCountFailed; on a clean run the succeeded count equals the row count of your file.

When status reaches partial, some rows landed and some did not. Walk to the partial-failure inspection recipe to page through the per-row errors and decide whether to fix the offending rows and resubmit, or accept the partial result.

A failed status means zero rows landed. The errorSummary.samples field carries the first ten error payloads; the rest are queryable via the paged-errors endpoint described in the partial-failure recipe.