Idempotency in Payment and Credit Event Processing

A consumer receives payment pmt_55192, applies €500 to a loan and crashes before acknowledging the message. The source retries. One real payment now threatens to create €1,000 of state effect.

{
  "sourceEventId": "pmt_55192",
  "accountId": "acc_1042",
  "amountMinor": 50000,
  "currency": "EUR"
}

Without protection, the balance is wrong, cure and DPD may be false, behavioural features change, collections can stop incorrectly and servicing no longer reconciles. This is not merely a repeated message. It is corrupted lending state.

Exactly-once delivery is not the financial objective

The handler must produce one complete business effect; every low-level write need not be naturally idempotent. Network timeouts, consumer crashes, producer retries, queue redelivery, batch resends, API retries, disaster recovery and manual replay ensure that DuplicateDelivery > 0 eventually.

Broker offsets and checkpoints record delivery progress. They do not, by themselves, prove a payment changed the balance exactly once.

Deduplication starts with authoritative business identity

A canonical event should preserve eventId, sourceSystem and sourceEventId. An internal event ID identifies the canonical record; the source ID links repeated deliveries to one source fact.

// Unsafe: each retry becomes a new canonical identity.
const event = { eventId: randomUUID(), ...sourcePayload };

// Conceptual stable source-scoped key.
function idempotencyKey(event: SourceEvent): string {
  return event.sourceSystem + ":" + event.sourceEventId;
}

The key must be deterministic across retries, scoped to the domain where the source identifier is unique and collision-resistant for real business events. 12345 may be unique only within one provider; source scope is therefore material.

A fingerprint such as hash(source, account, amount, currency, eventTime, reference) can be evidence where no source ID exists, but two legitimate payments may share those values. False suppression loses a real payment and is as dangerous as duplicate application. Do not manufacture certainty at lower identity levels.

Commands and events need distinct semantics

A command asks to create a payment; an event states that a payment settled. An API command may accept an Idempotency-Key so a client retry refers to the same created operation. The resulting event still needs stable consumer identity. Producer controls help, but every state-changing consumer remains a final protection boundary.

Check-then-write is a concurrency bug

async function handle(event: FinancialEvent) {
  await applyToLoanState(event);
  await markProcessed(event.eventId);
}

If state application succeeds and the process dies before markProcessed, retry applies the event twice. A preliminary SELECT is also unsafe: workers A and B can both observe absence before either inserts.

CREATE TABLE processed_events (
  consumer_name   TEXT NOT NULL,
  source_system   TEXT NOT NULL,
  source_event_id TEXT NOT NULL,
  processed_at    TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (consumer_name, source_system, source_event_id)
);

INSERT INTO processed_events (
  consumer_name, source_system, source_event_id, processed_at
)
VALUES ($1, $2, $3, NOW())
ON CONFLICT DO NOTHING
RETURNING source_event_id;

If no row returns, this consumer already claimed the event. Other SQL dialects use different conflict or merge syntax; the requirement is one atomic uniqueness operation backed by a constraint.

await db.transaction(async (tx) => {
  const claimed = await tx.processedEvents.insertIfAbsent({
    consumerName: "account-state-v3",
    sourceSystem: event.sourceSystem,
    sourceEventId: event.sourceEventId
  });
  if (!claimed) return;
  await applyEventToState(tx, event);
});

The unique claim and complete financial mutation share one transaction. Either both commit or neither does.

Transaction order determines the failure window

When the event store, state database and broker are separate systems, one local ACID transaction cannot cover all three. Reliable architecture then narrows the atomic boundary and uses durable handoff patterns rather than pretending a distributed commit occurred.

Inbox and outbox close different reliability gaps

CREATE TABLE event_inbox (
  consumer_name   TEXT NOT NULL,
  source_system   TEXT NOT NULL,
  source_event_id TEXT NOT NULL,
  payload         JSONB NOT NULL,
  received_at     TIMESTAMPTZ NOT NULL,
  processed_at    TIMESTAMPTZ,
  PRIMARY KEY (consumer_name, source_system, source_event_id)
);

The consumer inbox durably stores delivery under a unique business identity. Local processing can then mutate consumer state and mark the inbox row processed inside one database transaction.

CREATE TABLE event_outbox (
  event_id      TEXT PRIMARY KEY,
  aggregate_id  TEXT NOT NULL,
  event_type    TEXT NOT NULL,
  payload       JSONB NOT NULL,
  created_at    TIMESTAMPTZ NOT NULL,
  published_at  TIMESTAMPTZ
);

The producer writes domain state and its publication intent together; a publisher sends unsent rows. This prevents “business state committed, event lost.” It does not remove consumer idempotency because publication may repeat after an uncertain acknowledgement.

Retry count must not affect financial state

Retry transient timeouts and temporary database failures with bounded backoff. Route invalid schemas and impossible references to controlled failure or quarantine; endless retry does not repair permanent data. Unknown failures require evidence and investigation.

Replay has a destination and a purpose

One canonical event may feed account, collections and feature projections. Idempotency is therefore often (consumer, event), not a global “processed somewhere” flag. Consumer A completing must never cause Consumer B to skip its legitimate projection work. Dedupe records must also outlive plausible late retry, replay and audit horizons; no universal retention period fits every source.

An idempotent database update can still send two customer actions

Email, collections messages, API calls and payment instructions may sit outside the database transaction. Give each outgoing economic action a stable actionId or decisionId, persist execution state under a unique constraint and require downstream retry recognition where possible.

actionId = "collections_notification:dec_8821"

If the same state triggers recalculation twice, distinguish model recomputation, a genuinely new decision and re-execution of the existing action. A conceptual decision identity might include account, decision type, trigger event and strategy version—but only where those fields match the business lifecycle.

CREATE TABLE executed_actions (
  action_id    TEXT PRIMARY KEY,
  action_type  TEXT NOT NULL,
  executed_at  TIMESTAMPTZ NOT NULL
);

Protect the whole allocation, not just the input message

One €500 payment can allocate €50 to fees, €100 to interest and €350 to principal. The idempotency boundary must protect the complete allocation transaction. A crash after principal but before interest cannot be recovered safely by re-running untracked additive writes.

Event infrastructure does not replace a ledger. It preserves identity and delivery evidence while servicing and accounting enforce their own balanced, controlled mutations.

A reversal has its own stable identity and a reversalOf relationship. It must also be idempotent: duplicate delivery of a reversal must not reverse twice. Do not dedupe it merely because it references the original; different corrections can legitimately refer to the same event.

Streams, files and snapshots require different layers

An identical balance snapshot can be harmless when handling means SET balance = 500, while balance += 500 is not. Likewise, “set limit to €5,000” is easier to retry safely than “increase limit by €1,000” where business semantics permit an absolute desired-state command. The analogy resembles PUT versus POST, but HTTP method labels alone do not establish financial idempotency.

Idempotency, ordering, finality and immutability solve different problems

UPDATE loan_state
SET balance_minor = $1,
    version = version + 1
WHERE account_id = $2
  AND version = $3;

If zero rows update, the aggregate changed concurrently. Expected-version control is an additional safeguard for sequence and concurrency; it is not a replacement for event identity.

A payment and reversal can each apply once but in the wrong order. A unique payment can later be legitimately reversed. Immutable history records both; idempotency prevents either effect from multiplying.

One provider payment, two deliveries, one €250 effect

{
  "sourceSystem": "payment-provider",
  "sourceEventId": "pmt_10001",
  "eventType": "PAYMENT_SETTLED",
  "accountId": "acc_4002",
  "amountMinor": 25000,
  "currency": "EUR"
}

If the worker crashes inside the transaction, claim and mutation roll back together; retry performs both. If it crashes after commit but before ACK, retry finds the committed claim and returns success without mutation. Correctness no longer depends on the exact crash instant.

Inject duplicates and crashes where production will

For any valid ordered event stream, inserting arbitrary duplicate payments, fees, limit changes and reversals must leave the final state unchanged. This property catches paths missed by example-only testing.

Crash-injection tests stop the worker before claim, after claim, after mutation, around commit and before acknowledgement. Where the claim and mutation are one transaction, intermediate failures roll back; after commit, redelivery suppresses. Add deliberate failure around outbox publication and external action execution.

Observe suppression, collision risk and reconciliation

Some redelivery is normal in at-least-once systems. Investigate sudden change, concentration by source/event type/provider/time window, financial materiality or failed suppression—not a universal threshold.

False suppression is harder to see. Audit collisions, suspicious same-key/different-payload cases and unresolved identities. Quarantine material ambiguity rather than guessing. Operational reconciliation should explain SourceEventCount → CanonicalUniqueEventCount → AppliedBusinessEventCount; financial reconciliation should connect unique settled payments to servicing application and accounting posting subject to timing and allocation differences.

Design every retry from the economic event outward

Banks face batch resend, message redelivery and internal integration retries. API-first lenders face webhook retries whenever a response is uncertain. Legacy and cloud-native estates therefore share the same requirement: could this event arrive again after failure, and can we prove state and action remain correct?

An Event Duplication & Processing Integrity Agent can turn retries into evidence

A future controlled agent can monitor duplicate delivery, idempotency-key conflicts and retry patterns; attribute suppression by source; compare source, canonical and applied counts; identify likely false collisions or unsafe consumer sequences; reconstruct incident impact; and prepare evidence for human engineers.

Continue with Event Time vs Processing Time vs Posting Time in Credit Systems, The Payment Is Not the Balance, The Hidden Infrastructure Debt of Modern Lending, Why Batch Risk Is Becoming a Business Risk and Credit Risk Model Validation Pipeline. Future Engineering work can develop account-state reconstruction, backdated corrections, reversals, DPD, event-driven triggers and historical replay; these are research directions, not fabricated routes.