Your Webhooks Shouldn't Trust the URL They Were Given

Product

A webhook is the one feature where a customer types an address into a form and your servers go and call it. Every other request your backend makes goes somewhere you chose. This one goes wherever the customer says, as often as your product produces events, for as long as the endpoint exists.

This week Kookee shipped webhooks, and on top of them integrations with Slack, Discord, Telegram, GitHub, Jira, Asana and Freshdesk. Publish a post and your static site rebuilds; a visitor leaves feedback and it lands in your team's Slack channel or becomes a Jira task.

The Integrations page with GitHub Issues, Slack and Jira connected, and Telegram, GitHub Actions and Discord ready to configure

The form with the URL field took an afternoon. Everything below took the rest of the week, and a few of these we only caught when reviewing the plan, before a line of the code existed.

Eight things a webhook sender has to get right

1. Check the address when you send, not when you save

The obvious check is to refuse http://localhost and 10.0.0.5 when someone saves an endpoint. It is also the useless one. The URL is checked once, but the name in it is looked up again on every delivery, and nothing stops hooks.example.com from resolving to a public address on Monday and to 169.254.169.254, the cloud metadata service, on Tuesday.

So the real check happens at send time. Kookee resolves the host, refuses the delivery if any address it gets back is private, loopback, link-local or reserved (IPv4 written inside IPv6 included), and then connects to exactly the address it checked, so a second lookup cannot swap it. Redirects are not followed: a 302 to an internal address is just an answer.

One range is easy to forget: your own servers' public addresses. Behind a CDN, your origin's IP is the thing an attacker most wants to reach, and a webhook would happily call it from inside your firewall. Ours are on the block list.

2. Sign it the way everybody else does

We could have invented a signature header. Instead, Kookee signs requests the Standard Webhooks way: a webhook-id, a webhook-timestamp and an HMAC in webhook-signature. The person writing the receiver gets a library in their language, a five-minute replay window for free, and one less format to learn.

The signing secret is encrypted at rest, and can be revealed and regenerated from the endpoint's menu. Regenerating it stops the old one at once, which is what you want after a leak.

3. A slow receiver should only slow itself

Every delivery waits on someone else's server. If a customer's endpoint takes nine seconds to answer and your workers handle a few deliveries at a time, one customer quietly becomes everyone's queue.

We give every request a hard 10-second deadline, DNS lookup included, and run deliveries 25 at a time per process. That alone is not enough, because a slow endpoint retried all day still eats those slots. So each endpoint gets 2 and a half minutes of waiting an hour, and each account 7 and a half minutes across its endpoints. A delivery over that budget is skipped, and the endpoint says so in plain words. A skip never counts as a failure: the receiver is slow, not broken, and turning it off would punish the wrong thing.

Content events and visitor events are counted apart. A flood of spam feedback should never be the reason your site did not rebuild after you published.

4. Retry for hours, then stop, and tell someone

Servers go down for deploys and come back. A delivery that fails is retried ten times over about eight and a half hours, waiting 1, 2, 4 and up to 256 minutes, with the same webhook-id every time so the receiver can drop duplicates.

What we did not want is an endpoint that has been failing silently since March. An answer of 410 Gone turns an endpoint off at once. Five events in a row that fail every retry turn it off too. Either way the project owner gets a notification and an email saying which endpoint and why, and turning it back on is one click.

5. Send the event when the page changes, not when the row does

"Entry updated" sounds simple until you count what fires it. Autosave, a typo fix on a draft, a translation being linked: a naive hook rebuilds a static site forty times an afternoon.

Kookee's content events follow what a visitor can see. entry.published fires when an entry becomes visible, and that includes a scheduled post going live at 9:00 with nobody at the keyboard: each future date has a job waiting for it. entry.updated fires only for an entry that was visible and still is. A draft saved ten times sends nothing.

6. Every destination has its own way to ping people

Integrations post text a stranger wrote into your team's tools, and each tool has its own markup for getting attention. A feedback post titled <!channel> urgent would ping a whole Slack channel. In Discord the same trick is @everyone, plus a masked link that hides where it goes. In GitHub, #123 or owner/repo#1 in an issue body leaves a cross-reference on someone else's issue, from your account.

So every service gets its own escaping. Slack's special characters are escaped and unfurling is off. Discord gets every Markdown character escaped, mentions disabled and embeds suppressed. Telegram text is HTML-escaped. In a GitHub issue the visitor's text sits in a code block with a fence longer than any run of backticks inside it. Freshdesk gets HTML-escaped text, because its ticket body is HTML. And Jira refuses a description with an empty paragraph, so blank lines are dropped before they reach it.

None of this is visible when it works. It is the difference between a feedback tool and a way for anyone to message your whole company.

7. Credentials never ride in the queue

A Slack webhook URL is a password: anyone who has it can post to your channel. Same for a Telegram bot token, which sits right in the API's URL. The tempting design puts the decrypted config in the delivery job so the worker has everything it needs. Then every token sits in Redis in plain text, in every retry, for hours.

Kookee's jobs carry an integration id, an event id and the event. The worker loads the row, decrypts the config at the last moment, and never writes a request URL to a log. Tokens are also kept out of the audit log, which records who changed what, but not the secret they typed.

8. Read the other side's rules before you write your own

Our first plan treated 403 Forbidden as "this integration will never work", and turned the integration off. Node's HTTP client sends no User-Agent header unless you set one, and GitHub answers a request without one with 403. Every GitHub integration would have switched itself off on its first delivery.

The plan review caught it, and every request now carries content-type and a User-Agent. The lesson went further than one header: GitHub also answers rate limits with 403, so that case retries instead, and Telegram reports a deleted chat as a 400 with a message, so that case turns off. Each service gets a short list of exceptions to a simple rule, written down next to the code that sends to it.

If you build webhooks

  • Resolve the host at send time, refuse private and reserved addresses, connect to the address you checked, and do not follow redirects.

  • Put your own servers' public addresses on the block list.

  • Sign with Standard Webhooks instead of a format of your own.

  • Give every delivery a hard deadline, and give each endpoint and account a time budget, so one slow receiver cannot hold everyone up.

  • Retry for hours with a stable message id, then turn the endpoint off and tell its owner.

  • Fire content events on what visitors can see, not on every save.

  • Escape text for each destination's own markup before you post it.

  • Keep credentials out of the queue, the logs and the audit trail.

Webhooks are on every Kookee plan, and integrations on Pro and Ultra. Send events to your site with webhooks covers the events, the payload and verifying the signature.

Continue reading