Bulk NDJSON

The bulk NDJSON surface accepts up to ten thousand rows in a single synchronous HTTP request, applies per-row validation against the same schemas the single-record routes use, and returns a row-level error report. Use it for end-of-day batches that comfortably fit in one HTTP request; for historical loads or larger batches use async imports.

The wire format is newline-delimited JSON: one valid JSON object per line, no trailing comma, no surrounding array, no opening or closing envelope. Blank lines between objects are accepted and skipped.

Routes

There is one route per silver entity. The schema applied per line matches the entity in the URL.

Each line in the request body must be a valid JSON object that satisfies the entity's schema as documented on the single-record page. The schemas are identical; only the wire envelope differs.

Request

POST /api/v1/monitoring/loan/bulk/loans HTTP/1.1
Authorization: Bearer <PAT>
Content-Type: application/x-ndjson
Idempotency-Key: <opaque-key>

Headers and query parameters:

The bulk surface reads portfolioId from the query string only; the per-row body field of the same name is ignored at the batch wrapper layer. Every row in the body must reference the same portfolio.

The body is the NDJSON payload itself — UTF-8 encoded, terminated by a single newline. Lines beyond ten thousand or bodies above ten megabytes are rejected — see size caps.

Per-row contract

Every row is validated independently against the entity schema. The same DX-bar rules apply per row as on a single-record write:

• Every row must carry its own id UUID and portfolioId. Caller-supplied IDs are the per-row dedup key for the database's partial unique index.
• The whole batch shares the request-level Idempotency-Key. See idempotency below.
• metadata is per-row. A row may carry its own pass-through metadata; the platform persists it verbatim.
• Currency, dates, enums, and money rules are identical to single-record. A row that fails zod validation surfaces as an entry in the per-row error report with code: "validation_failed".

A row that fails validation does not abort the batch — the failure surfaces as an entry in errors[] and the remaining rows continue to be processed.

Response

200 OK is returned for every batch where the request itself parsed, regardless of how many individual rows succeeded or failed. The response body carries the full per-row report:

{
  "totalRows": 4,
  "succeeded": 3,
  "failed": 1,
  "bronzeSubmissionId": "1a2b3c4d-5e6f-7890-abcd-ef0123456789",
  "reused": false,
  "errors": [
    {
      "rowNumber": 2,
      "code": "fk_violation",
      "message": "borrowerId references a borrower that does not exist in this portfolio"
    }
  ]
}

The per-row error shape carries only rowNumber, code, and message. There is no per-row fieldErrors array, and no per-row rawRow — the original NDJSON is preserved verbatim in the bronze archive (replayable via the operations tools) rather than echoed back on the wire.

A succeeded of 0 is not an error response — the batch parsed and the platform processed every row; the report tells you nothing landed. A genuinely failed batch (malformed NDJSON across the whole body, body too large, unknown portfolio) returns 4xx per the table below.

Status codes

The top-level error envelopes on this surface use lower-case code values, distinct from the upper-case codes used by the single-record routes. The two codespaces are intentionally separate.

Size caps

The bulk surface caps payloads at the smaller of:

• Ten thousand non-blank lines. Rows beyond that are rejected before processing begins.
• Ten megabytes of request body.

A request that exceeds either cap receives 413 Payload Too Large:

{
  "error": "batch payload exceeds 10485760 bytes; use the async import endpoint at /api/v1/monitoring/loan/imports",
  "code": "batch_too_large",
  "fieldErrors": [{ "path": "(body)", "message": "max 10485760 bytes" }]
}

The byte-cap envelope and the row-cap envelope both use code: "batch_too_large"; the error and fieldErrors[].message distinguish the trigger. The canonical recovery path is to switch the same payload to the async imports flow, which uploads via presigned S3 PUT and processes through a streaming worker with no row cap.

Idempotency

The bulk surface deduplicates against Idempotency-Key at the batch envelope level. The mechanics:

• First call with key K. The body is archived to bronze as a single envelope, every row is processed, and the per-row report is persisted alongside the bronze row.
• Second call with key K, same body. No re-archive, no re-insertion. The platform reconstructs the original report from the persisted row-errors plus the bronze envelope's metadata and returns it with reused: true.
• Second call with key K, different body. The platform still returns the original report — the original submission is the source of truth. A re-POST with a corrected body must use a different key.

This means a client retrying after a network blip is safe even if the second request's body was truncated or modified in transit — the platform returns the row-level outcomes from the first successful processing.

Bronze envelope semantics

Each bulk request archives one bronze submission containing the whole NDJSON body — not one per row. The reasoning:

• Per-row bronze writes would multiply S3 PUT operations without buying anything, because replay reads the raw NDJSON and dedup is enforced at the silver layer by the partial unique index on (org_id, portfolio_id, external_id) WHERE deleted_at IS NULL.
• The bronze row records submissionMethod: 'bulk_ndjson' and the entity-specific entityType, so the replay path can route the body back through the same handler that processed it the first time.

The bronzeSubmissionId in the response is the archive id; persist it alongside your local submission record for audit and reconciliation.

cURL example

cat <<'EOF' > batch.ndjson
{"id":"a1...","portfolioId":"ddf63d12-1234-5678-9abc-def012345678","borrowerId":"b1...","externalId":"L-001","principalAmount":"5000","currencyCode":"USD","interestRateApr":"0.18","disbursementDate":"2026-01-15","termMonths":12,"repaymentFrequency":"monthly","status":"active"}
{"id":"a2...","portfolioId":"ddf63d12-1234-5678-9abc-def012345678","borrowerId":"b2...","externalId":"L-002","principalAmount":"3000","currencyCode":"USD","interestRateApr":"0.20","disbursementDate":"2026-01-15","termMonths":24,"repaymentFrequency":"monthly","status":"active"}
{"id":"a3...","portfolioId":"ddf63d12-1234-5678-9abc-def012345678","borrowerId":"b3...","externalId":"L-003","principalAmount":"7500","currencyCode":"USD","interestRateApr":"0.22","disbursementDate":"2026-01-15","termMonths":18,"repaymentFrequency":"monthly","status":"active"}
EOF

curl -X POST "https://app.waafir.io/api/v1/monitoring/loan/bulk/loans?portfolioId=ddf63d12-1234-5678-9abc-def012345678" \
  -H "Authorization: Bearer $WAAFIR_PAT" \
  -H "Content-Type: application/x-ndjson" \
  -H "Idempotency-Key: nightly-2026-01-15-loans" \
  --data-binary @batch.ndjson

When to choose bulk NDJSON vs async imports

Choose bulk NDJSON when:

• Your batch fits in ten thousand rows and ten megabytes.
• You want an immediate per-row report in the HTTP response.
• You can tolerate a long-running HTTP request — bulk batches typically complete in a few seconds, but a large batch with many database FK lookups can take longer.

Choose async imports when:

• Your batch exceeds either size cap.
• You are loading a historical archive of payments, scheduled repayments, or borrowers — typical historical loads run to millions of rows.
• You want to retain the option to cancel an in-flight load.