From Batch ETL to Event-Driven Credit Risk Architecture

Core and servicing data flow through a nightly extract, warehouse transformation and feature mart before models update collections, limits and early warning. Nothing is inherently wrong with that pipeline. The question is whether its cadence matches the economic half-life of each downstream decision.

A mature credit platform can use four processing modes

Monthly ECL does not need second-by-second recomputation. A 15-minute micro-batch may capture most value without streaming complexity. Start with payment settled, drawdown, limit changed, DPD changed or bureau received—then map the state and decisions each business event affects.

Source transport changes; canonical business meaning should not

The database delta does not explain payment, correction, write-off or migration. Preserve raw source lineage for audit and reprocessing, but serve downstream consumers canonical semantics such as PAYMENT_SETTLED, FACILITY_LIMIT_CHANGED and ACCOUNT_STATE_CHANGED.

Ingestion validates structure, assigns canonical metadata, enforces idempotency and routes invalid events. Source contracts declare identity, semantics, latency, ordering and correction behaviour. schemaVersion and compatibility/upcasting protect consumers as events evolve.

Canonical events produce rebuildable state and targeted feature updates

Query-optimised projections such as account_state, facility_state and customer_credit_state are derived and rebuildable. A settled payment can update payment state, DPD, payment_ratio_90d and a behavioural score without recomputing unrelated income or bureau features.

Each projection carries effective_as_of, available_as_of and updated_at. The decision freshness guard verifies critical input age against versioned policy budgets.

State changes trigger decisions only when policy says they should

Repeated updates must not create repeated actions. Stable decision and action IDs protect exactly-once business effect even when delivery is at-least-once. Coalescing is micro-batching inside an event-driven path; it prevents event storms from turning into noisy rescoring and operational overload.

Streaming reduces latency; it does not eliminate temporal complexity

Preserve event, effective and processing time. Partition by a stable aggregate such as facility or account when same-aggregate ordering matters; global institutional ordering is usually unnecessary.

Late events, reversals and corrections still require restatement and replay. Duplicate delivery is normal: unique event identity, idempotent consumers and transactional state mutation protect the business effect.

Consumer lag is decision staleness, not merely infrastructure health

If a payment consumer falls behind, collections state goes stale. Monitor lag by stream, consumer and event type. Scale, prioritise material events, degrade non-critical work or coalesce safely—but never drop financial events silently.

The worst failure is silent: an API remains healthy while serving yesterday's state as current. Staleness must be visible to decision policy.

Replay rebuilds state without replaying external consequences

Consumers distinguish LIVE from REPLAY. Replay rebuilds projections and features but must not resend emails, repeat collections actions or issue external commands. Action executors use stable IDs so a repeated decision event cannot duplicate its effect.

type ProcessingContext = "LIVE" | "REPLAY";

if (context === "LIVE") {
  await actionExecutor.executeOnce(actionId, command);
}

Durable canonical history makes state rebuild, feature rebuild and incident recovery possible. Event loss is worse than delay; source-to-canonical count and financial-total reconciliation must expose missing records.

Fast operational state and slower authoritative reconciliation complement each other

Compare derived state with system-of-record state and classify timing, missing event, duplicate, correction or mapping differences. Avoid separate batch and speed codebases with divergent business logic. One path is the operational projection; the other is its reference/control—not a second truth.

Migrate the vertical slice with the largest material latency gap

A 24-hour process can first become a 15-minute micro-batch. A legacy core can use Core DB → CDC → Adapter → Canonical Event. A daily file can be parsed into canonical events today, allowing later API or CDC transport without changing downstream semantics.

Shadow, align cut-offs, canary and preserve rollback

Do not compare 14:00 real-time state with end-of-day batch state. Align effective cut-offs, store mismatch reasons and track latency improvement. Run new decisions without executing them, then canary a controlled population with rollback.

Rollback must preserve event history. Before reactivation, replay events accumulated while the old path ran. Irreversible cutovers turn a recoverable consumer defect into data loss.

Measure the complete event-to-action path and its tail

Measure p50, p95 and p99; an average hides operationally damaging tails. Feature update lag is feature availability minus source-event time, decision trigger lag is decision time minus material-event time, and action lag completes the business path.

Streaming introduces stateful operations, cost and observability burden. Formal reporting can remain periodic while risk signals update quickly. Use low latency only where its decision value justifies that burden.

A settled payment should stop a same-day collections action

The new slice reduces action latency without weakening accounting control or forcing monthly ECL into streaming.

A golden event stream tests state, triggers and side effects together

Operate the pipeline through decision-centric evidence

Monitor event counts, payment amounts, drawdown volumes, reversal rates and DPD transitions. A technically healthy pipeline can still carry wrong economics after a silent semantic change.

The Entimema architecture connects source change to controlled action

An Event-Driven Risk Infrastructure Agent can diagnose flow without changing production

A controlled agent can monitor ingestion and consumer lag, compare source and canonical counts, identify backpressure and stale decision paths, compare batch with stream projections, trace material events to actions and surface latency-budget breaches by business impact.

Continue with Point-in-Time Correct Features, Building a Credit Risk Feature Store, Point-in-Time Customer State Reconstruction, Event Time vs Processing Time vs Posting Time, Idempotency in Payment and Credit Event Processing, Why Batch Risk Is Becoming a Business Risk and The Hidden Infrastructure Debt of Modern Lending. Canonical event modelling, streaming early-warning features and backpressure recovery remain future research directions—not fabricated routes.