Polling responses overlap by design. The newest page can contain swaps already returned in the previous poll, and retries can replay an entire response. Deduplication is therefore a correctness requirement, not an optimization.
Use the swap ID as the identity
Persist the Fomo swap id under a unique database constraint. Do not create identity from timestamp and amount; separate swaps can legitimately share both.
create table processed_swaps (
swap_id text primary key,
trader_id uuid not null,
created_at timestamptz not null,
payload jsonb not null,
processed_at timestamptz not null default now()
);
Insert before producing effects
Attempt the unique insert inside a transaction. Only the worker that successfully inserts may create a normalized event or outbox message. If the constraint conflicts, the swap was already claimed.
insert into processed_swaps (swap_id, trader_id, created_at, payload)
values ($1, $2, $3, $4)
on conflict (swap_id) do nothing
returning swap_id;
Process in chronological order
Sort unseen swaps by createdAt ascending before applying position transitions. API order should not be treated as a permanent contract unless documented.
Make downstream work idempotent
Deduplicating ingestion is not enough if a worker crashes after sending an alert but before committing. Use a transactional outbox: insert the swap and an effect request in the same transaction, then let a separate dispatcher deliver it under its own idempotency key.
signal:{strategyId}:{swapId}
alert:{channelId}:{swapId}
order:{accountId}:{swapId}
Keep checkpoints conservative
A timestamp checkpoint helps query planning but should not replace IDs. Allow an overlap window so delayed or equal-timestamp events are still fetched. The unique constraint safely removes repeats.
Test duplicate pages, reversed event order, equal timestamps, process crashes and two workers claiming the same swap. Exactly-once delivery is difficult; idempotent effects make at-least-once processing safe.