Bulk import a loan tape via async NDJSON
This recipe shows the foundational ingestion flow for a Waafir loan portfolio: submit a newline-delimited-JSON (NDJSON) file containing one row per loan-tape entity to the async-import endpoint, then poll the import's status until the worker finishes.
When to use this flow
The async NDJSON import flow is the canonical ingestion path for any non-trivial volume of rows. It scales to historical backloads of millions of rows and is the same flow used for incremental daily submissions. The cookbook uses a loan import in the examples below; the same flow accepts borrower, scheduled_repayment, actual_payment, collateral, and collateral_valuation kinds — change the kind field and the row shape and everything else is identical.
The shape of the flow
There are three steps your code must take, and one step your file does in between:
POST /api/v1/monitoring/loan/imports— declare what you are about to upload (kind, filename, size). The response includes an upload URL and the import ID.PUTthe NDJSON body to the upload URL — your file is stored as submitted, unchanged.POST /api/v1/monitoring/loan/imports/{id}/commit— tell Waafir the upload finished. The import moves frompending_uploadintoqueued, and the worker begins processing batches.GET /api/v1/monitoring/loan/imports/{id}— poll untilstatusis one ofsucceeded,partial,failed, orcancelled.
The recipe
# curl — bulk import flow (loan kind)
set -euo pipefail
PAT="${WAAFIR_PAT}"
BASE_URL="${WAAFIR_BASE_URL:-https://app.waafir.io}"
NDJSON_FILE="loans-2026-05-22.ndjson"
# Step 1 — start the import and capture the upload URL + import ID.
START=$(curl -fsS -X POST "${BASE_URL}/api/v1/monitoring/loan/imports" \
-H "Authorization: Bearer ${PAT}" \
-H "Content-Type: application/json" \
-d "$(cat <<JSON
{
"kind": "loan",
"filename": "${NDJSON_FILE}",
"sizeBytes": $(stat -c%s "${NDJSON_FILE}")
}
JSON
)")
IMPORT_ID=$(echo "$START" | jq -r .importId)
UPLOAD_URL=$(echo "$START" | jq -r .uploadUrl)
# Step 2 — PUT the NDJSON body to the upload URL. The `headers` field
# on the start response carries the headers the upload URL was signed
# with — apply them all, or the upload is rejected with a signature
# mismatch. At minimum `Content-Type: application/x-ndjson`.
HEADER_ARGS=$(echo "$START" | jq -r '.headers | to_entries[] | "-H \"\(.key): \(.value)\""')
eval curl -fsS -X PUT --data-binary @"\"${NDJSON_FILE}\"" $HEADER_ARGS "\"${UPLOAD_URL}\""
# Step 3 — commit so the worker enqueues the import.
curl -fsS -X POST "${BASE_URL}/api/v1/monitoring/loan/imports/${IMPORT_ID}/commit" \
-H "Authorization: Bearer ${PAT}"
# Step 4 — poll status until it reaches a terminal value.
while :; do
STATUS=$(curl -fsS "${BASE_URL}/api/v1/monitoring/loan/imports/${IMPORT_ID}" \
-H "Authorization: Bearer ${PAT}" | jq -r .status)
echo "import ${IMPORT_ID}: ${STATUS}"
case "${STATUS}" in
succeeded|partial|failed|cancelled) break ;;
esac
sleep 5
done// TypeScript — bulk import flow (loan kind)
import { readFileSync, statSync } from "node:fs";
const PAT = process.env.WAAFIR_PAT!;
const BASE_URL = process.env.WAAFIR_BASE_URL ?? "https://app.waafir.io";
const NDJSON_FILE = "loans-2026-05-22.ndjson";
const headers = { Authorization: `Bearer ${PAT}`, "Content-Type": "application/json" };
// Step 1 — start the import.
const startRes = await fetch(`${BASE_URL}/api/v1/monitoring/loan/imports`, {
method: "POST",
headers,
body: JSON.stringify({
kind: "loan",
filename: NDJSON_FILE,
sizeBytes: statSync(NDJSON_FILE).size,
}),
});
if (!startRes.ok) throw new Error(`start failed: ${startRes.status} ${await startRes.text()}`);
const { importId, uploadUrl, headers: uploadHeaders } = (await startRes.json()) as {
importId: string;
uploadUrl: string;
headers: Record<string, string>;
};
// Step 2 — PUT the NDJSON body to the upload URL with the headers
// the URL was signed against. Skipping them gets the upload
// rejected with a signature mismatch.
const putRes = await fetch(uploadUrl, {
method: "PUT",
headers: uploadHeaders,
body: readFileSync(NDJSON_FILE),
});
if (!putRes.ok) throw new Error(`upload failed: ${putRes.status}`);
// Step 3 — commit.
const commitRes = await fetch(`${BASE_URL}/api/v1/monitoring/loan/imports/${importId}/commit`, {
method: "POST",
headers: { Authorization: `Bearer ${PAT}` },
});
if (!commitRes.ok) throw new Error(`commit failed: ${commitRes.status} ${await commitRes.text()}`);
// Step 4 — poll.
const TERMINAL = new Set(["succeeded", "partial", "failed", "cancelled"]);
while (true) {
const statusRes = await fetch(`${BASE_URL}/api/v1/monitoring/loan/imports/${importId}`, {
headers: { Authorization: `Bearer ${PAT}` },
});
const body = (await statusRes.json()) as { status: string };
console.log(`import ${importId}: ${body.status}`);
if (TERMINAL.has(body.status)) break;
await new Promise((r) => setTimeout(r, 5_000));
}# python — bulk import flow (loan kind)
import os
import time
import requests
from pathlib import Path
PAT = os.environ["WAAFIR_PAT"]
BASE_URL = os.environ.get("WAAFIR_BASE_URL", "https://app.waafir.io")
NDJSON_FILE = Path("loans-2026-05-22.ndjson")
headers = {"Authorization": f"Bearer {PAT}", "Content-Type": "application/json"}
# Step 1 — start the import.
start = requests.post(
f"{BASE_URL}/api/v1/monitoring/loan/imports",
headers=headers,
json={
"kind": "loan",
"filename": NDJSON_FILE.name,
"sizeBytes": NDJSON_FILE.stat().st_size,
},
timeout=30,
)
start.raise_for_status()
payload = start.json()
import_id = payload["importId"]
upload_url = payload["uploadUrl"]
upload_headers = payload["headers"] # signed-into the URL; must be sent on PUT
# Step 2 — PUT the NDJSON body to the upload URL with the headers
# the URL was signed against. Skipping them gets the upload
# rejected with a signature mismatch.
with NDJSON_FILE.open("rb") as fh:
put = requests.put(upload_url, data=fh, headers=upload_headers, timeout=300)
put.raise_for_status()
# Step 3 — commit.
commit = requests.post(
f"{BASE_URL}/api/v1/monitoring/loan/imports/{import_id}/commit",
headers={"Authorization": f"Bearer {PAT}"},
timeout=30,
)
commit.raise_for_status()
# Step 4 — poll.
terminal = {"succeeded", "partial", "failed", "cancelled"}
while True:
status = requests.get(
f"{BASE_URL}/api/v1/monitoring/loan/imports/{import_id}",
headers={"Authorization": f"Bearer {PAT}"},
timeout=30,
).json()
print(f"import {import_id}: {status['status']}")
if status["status"] in terminal:
break
time.sleep(5)What to verify
When status reaches succeeded, every row in your NDJSON file landed as a live row. The response also surfaces rowCountProcessed, rowCountSucceeded, and rowCountFailed; on a clean run the succeeded count equals the row count of your file.
When status reaches partial, some rows landed and some did not. Walk to the partial-failure inspection recipe to page through the per-row errors and decide whether to fix the offending rows and resubmit, or accept the partial result.
A failed status means zero rows landed. The errorSummary.samples field carries the first ten error payloads; the rest are queryable via the paged-errors endpoint described in the partial-failure recipe.
Cookbook
Practical recipes for integrating a fintech's loan-tape portfolio with Waafir's monitoring API — bulk ingestion, dry-run validation, idempotent retries, partial-failure inspection, cancellation, and reconciliation, with cURL · TypeScript · Python side by side.
Validate a load with dry-run before committing data
Use the dry-run mode of the async import endpoint to validate an NDJSON file end to end — schema, enum, and row-level checks — without writing any rows.