left arrow Back to posts

Announcing SQL writes: Write data back to APIs using SQL in your database, synchronously

Anthony Accomazzo
3 min read
main image of the article

We’re Sequin. We stream data from services like Salesforce, Stripe, and AWS to messaging systems like Kafka and databases like Postgres. It’s the fastest way to build high-performance integrations you don’t need to worry about.

💡
Update: SQL writes was a neat feature, and we learned a lot. Currently, we're prioritizing HTTP for writing back to APIs. We might refine and implement SQL writes again in the near future.

No one is happy with HTTP APIs today.

For developers that need to build on top of third-party APIs (i.e. consumers), each integration is bespoke. There are few standards. Even OAuth 2.0 can be implemented in dozens of ways, let alone the shape of JSON responses, the format of errors, supported query params, or pagination behavior.

On top of that, poor throughput and rate limits mean consumers often need to cache or sync data. Eventually, even relatively small integrations turn into complex distributed system problems.

The platforms that provide HTTP APIs for consumers (hereafter providers) are equally unhappy. There are few standards that the industry adheres to, resulting in a process that is full of bike-shedding. And when it's finished, it's always disappointing, as even the best HTTP APIs are full of compromises and struggle to service the long-tail of use cases. This leads to frustration on both sides: the provider can't be proud of their offering, and the consumer feels inhibited by the design.

At Sequin, we realized it doesn't have to be this way. Our platform consists of two novel features that make both API consumers and providers much happier.

The first is the feature we originally launched with: we sync data from the API provider in real-time and put it in a Postgres database. By extracting data out of the API and putting it in Postgres, consumers enjoy a number of benefits:

  1. They can read their data without throughput limits, such as those imposed by rate limits.
  2. They can query their data with all the expressiveness of SQL. SQL is an order of magnitude more expressive than the typical HTTP API.
  3. They can use a standard interface they're familiar with (SQL) vs a bespoke one. In fact, they can just use their ORM to integrate, vs needing to use an under-supported API adapter library.

We're launching our second innovation today: Developers can now write back to the API via Sequin. We intercept the mutations that you make to synced records in your database and apply them to the API as well.

In fact, our Postgres Proxy commits the write to the API first, before your database. ‌This means that all the necessary policy that the API implements – like validation rules – are run against every database write. The API remains the source of truth, and the database never drifts.

If the write fails API validation, the Proxy rolls the write back and returns a native Postgres error:

insert into salesforce.contact (first_name, last_name, email)
values ('Andrea','Palladio','andrea__invalid')
returning id;

ERROR: [Salesforce]: Property values were not valid: `email` 'andrea_invalid' is invalid.

With these two components of our platform, APIs interfaces are no longer bespoke, they're familiar. We abstract away the common cruft of integrations, like syncing, cache invalidation, normalization, rate limit management, and token management. We let developers instead just focus on crafting the reads and writes that their application needs.

You get to have the tables from the APIs across your stack right there in your database, fully readable and writeable.

Now, there are some kinds of API requests or even providers that don't make sense in this paradigm. Our read/write interface is best suited for CRUD or REST operations. RPC-type requests can feel awkward, like posting a new payment to Stripe. For those situations, developers will typically still make API calls.

But what we've found is that about 80-90% of an integration's needs fall squarely into CRUD/REST territory.

The response from our early customers has been overwhelming positive. With this "crucial" feature, Sequin is the "silver bullet" for their integration challenges.

What's next

With our core read/write functionality in place, we have a number of exciting releases that build on this foundation coming soon. Here's a couple things that are just around the corner:

The first, of course, is more more sources. At the moment, we support Airtable, HubSpot, and Salesforce – but this is just the start.

The second is events. Our events system will let you build workflows based on data in your database changing. Think webhooks, but supercharged, and available for any change possible in the API. If you're interested in being an early tester of this feature, reach out to us.

Give Sequin a spin, and let us know what you think!

– Anthony & the Sequin team