A product event — a mention, a price drop, a delivery update — has to reach users by push, email, and SMS. One event can fan out to hundreds of thousands of recipients.
Two constraints shape everything. First, delivery is slow: an external provider takes 100–300ms per send and will rate-limit you if you exceed your quota. Second, the user action that triggers a notification must not wait for any of that.
So this is an asynchronism problem, not a throughput one. The API's job is to accept the event and get out of the way; the work happens behind a queue, at whatever rate your workers and your providers allow.
You start with an API writing straight to the database, doing the sends inline. It works at low volume and collapses the first time a campaign goes out.
Requirements- Events accepted
- ~2K/s sustained
- Fan-out per event
- up to 500K recipients
- Provider latency
- 100–300 ms per send
- Delivery guarantee
- at-least-once, deduplicated
- p99 API latency
- < 150 ms
- Error rate
- < 0.5%