Healthy Webhook Consumption with Rails

Introduction

What is a Webhook?

In a nutshell, webhooks are a semi-standardized way of managing and delivering push notifications over HTTP.

Webhooks usually form part of a larger, more traditional pull-based API to provide an alternative to polling for updates or as a way of broadcasting events in real-time. Instead of having the client make a request to the API provider for data, a webhook is a request made by the provider to inform the client of a change to the system.

Why use Webhooks?

Webhooks offer many advantages over other methods of keeping your app up-to-date with changes in remote data.

Reduced Resource Use

Webhook consumers use fewer system resources than their polling counterparts. Because they passively listen for updates they only consume appreciable amounts of CPU time, memory, etc. when changes occur. This contrasts a polling system that makes web requests on a set schedule whether or not anything has changed on the other end.

Data Freshness

Polling systems suffer from an inherent delay in their reflection of remote state. If I poll every 30 minutes I must accept that data I have may be 29 minutes out of date in the worst case. This can be mitigated by increasing the polling frequency, but there are associated resource costs for both you and the API provider. Many APIs have a request limit that places a hard cap on the rate at which you can poll.

On the other hand, webhooks can be sent immediately when a change occurs. A client can therefore guarantee that the data it has is only seconds old, even if the last update was hours ago.

Who uses Webhooks?

Anatomy of a Webhook

Topics and Subscriptions

Webhook providers expose one or more topics that correspond to events in their system. For example, a blogging platform might expose the following topics:

Webhook consumers subscribe to one or more topics that they're interested in. This is done in a RESTful fashion. POSTing creates subscriptions, GETing views current subscriptions, and DELETEing removes them.

Here's a sample webhook subscription creation call:

POST: /app/webhooks.json

  {
    "webhook": {
      "topic": "blog_post/create",
      "callback": "http://example.com/webhooks/blog_post/create"
    }
  }

As you can see, there are two fields that need to be passed: The topic the consumer wishes to subscribe to, and the callback URL at which the consumer wishes to receive webhook calls.

Webhook Callbacks

Here's what an outgoing webhook call made by the API provider might look like:

POST: http://example.com/webhooks/blog_post/create

  {
    "blog_post": {
      "id": 1234,
      "title": "10 ways to get rich quick!",
      "created_at": "2013-11-21 20:15:01 -0500"
    }
  }

In this case, it is simply a JSON-encoded copy of the blog post object that has been created.

Extra data you might see include signature headers, topic headers, and associated/nested objects (e.g. comments on a blog post).

Some services also employ the concept of a lightweight webhook that has no content beyond that which is necessary to make a new call to the API to retrieve the full object. Typically this is the ID or URL of the object in question.