Async imports

The async-imports surface is the path for historical loads and any batch above the bulk-NDJSON ten-thousand-row or ten-megabyte cap. The flow is two-phase: you describe what you intend to upload, receive a presigned S3 URL, PUT the file directly to S3, then ask the platform to commit it. From there a worker streams the file row-by-row, capturing per-row errors and surfacing progress through a status endpoint.

The implementation is shared with future non-loan domains — inventory, cashflows — so the routes you see here describe loan-domain entry points into a domain-agnostic framework.

Lifecycle

The lifecycle has seven states. Most production loads pass through pending_upload → queued → processing → succeeded; the others surface failure modes.

A row that never moves out of pending_upload represents a presigned URL that was issued but never committed; these are routinely garbage-collected after the URL's TTL expires.

Routes at a glance

POST /api/v1/monitoring/loan/imports

Start a new import. Returns the import id, a presigned PUT URL, and the upload headers your client must echo when it PUTs the file to S3.

Request body

The Idempotency-Key HTTP header takes precedence over the body field when both are supplied.

Response — 202 Accepted

{
  "importId": "1234abcd-5678-90ef-1234-567890abcdef",
  "uploadUrl": "https://s3.amazonaws.com/waafir-bronze-prod/...?X-Amz-Signature=...",
  "headers": { "x-amz-server-side-encryption": "aws:kms" },
  "s3Key": "imports/abc.../...",
  "reused": false,
  "statusUrl": "/api/v1/monitoring/loan/imports/1234abcd-5678-90ef-1234-567890abcdef",
  "commitUrl": "/api/v1/monitoring/loan/imports/1234abcd-5678-90ef-1234-567890abcdef/commit"
}

Status codes

Uploading the file to S3

Once you hold a uploadUrl, PUT the file directly to S3. Echo every header from the response's headers map; the signature on the URL covers them and S3 rejects the upload otherwise.

curl -X PUT "$UPLOAD_URL" \
  -H "x-amz-server-side-encryption: aws:kms" \
  --data-binary @your-file.ndjson

The upload is direct S3 traffic — it does not pass through Waafir. The file format is the same NDJSON wire format documented on the bulk-NDJSON page, with no row cap.

POST /api/v1/monitoring/loan/imports/{id}/commit

After PUT-ing the file to S3, call commit to tell the platform the upload is complete. The platform HEADs the S3 object to confirm it exists, flips the import row from pending_upload to queued, and enqueues the worker.

Request

No body. Idempotency-Key is implicit — a second commit on a non-pending_upload row is a no-op that returns the current state.

Responses

200 OK:

{ "importId": "1234abcd-...", "status": "queued" }

GET /api/v1/monitoring/loan/imports/{id}

Read the current status, counts, and error summary for an import. This is the polling endpoint your client calls to monitor progress.

Request

Response — default mode

{
  "importId": "1234abcd-...",
  "kind": "actual_payment",
  "domain": "loan",
  "status": "succeeded",
  "filename": "payments-2025-full-year.ndjson",
  "sizeBytes": 524288000,
  "portfolioId": "ddf63d12-...",
  "rowCountDeclared": 1500000,
  "rowCountProcessed": 1500000,
  "rowCountSucceeded": 1499987,
  "rowCountFailed": 13,
  "createdAt": "2026-05-20T10:00:00Z",
  "startedAt": "2026-05-20T10:00:15Z",
  "completedAt": "2026-05-20T10:18:42Z",
  "dryRun": false,
  "errorSummary": {
    "totals": { "validation_error": 9, "fk_violation": 4 },
    "samples": [
      { "rowNumber": 12, "errorCode": "validation_error", "errorMessage": "..." },
      { "rowNumber": 1041, "errorCode": "fk_violation", "errorMessage": "..." }
    ]
  }
}

errorSummary is a small embedded summary — totals by error code and up to ten sample rows. For the full per-row list, switch to paged-errors mode.

Response — ?errors=true&limit=&offset=

{
  "items": [
    {
      "rowNumber": 12,
      "errorCode": "validation_error",
      "errorMessage": "interestRateApr must be >= 0",
      "rawRow": "{\"id\":\"...\",\"interestRateApr\":\"-0.05\",...}",
      "createdAt": "2026-05-20T10:01:14Z"
    }
  ],
  "limit": 20,
  "offset": 0
}

limit defaults to 20, max 100. offset defaults to 0. Pagination iterates the persisted error list in rowNumber order, so a stable cursor is offset += limit.

Status codes

GET /api/v1/monitoring/loan/imports

List the caller's loan imports, most recent first.

Request

Response

{
  "items": [
    {
      "importId": "...",
      "kind": "actual_payment",
      "status": "succeeded",
      "filename": "...",
      "createdAt": "...",
      "completedAt": "...",
      "rowCountSucceeded": 1499987,
      "rowCountFailed": 13
    }
  ],
  "limit": 20,
  "offset": 0
}

DELETE /api/v1/monitoring/loan/imports/{id}

Cancel an in-flight import. The worker stops at the next chunk boundary; rows already inserted remain in silver. Cancelling an already-terminal row (succeeded, partial, failed, or cancelled) is a no-op that returns the existing state.

Responses

200 OK:

{ "importId": "1234abcd-...", "status": "cancelled" }

Kinds

The kind field on the start request determines the schema each row is validated against and which silver table receives the inserts. The six kinds map one-to-one to the silver entities:

A kind that is not in this list returns 400 UNKNOWN_KIND with the allowed values listed in the response, so a typo is recoverable without trial and error.

When to use this surface

Use async imports when:

• You are loading a historical archive — typical historical loads are millions of rows.
• Your batch exceeds either bulk-NDJSON cap (10 000 rows or 10 MB).
• You want the option to cancel mid-load — bulk NDJSON has no cancel surface because the request itself is the load.
• Your client tolerates polling — there is no push notification when an import completes.

For small operational batches that fit in a single synchronous request, bulk NDJSON is the simpler path.