No data left behind: introducing Singer, open source ETL

  • 28 March 2022
  • 0 replies

Userlevel 5

This content, written by Jake Stein, was initially posted in Looker Blog on Mar 1, 2017. The content is subject to limited support.

Looker is a powerful tool for exploring and visualizing all kinds of data, and our customers use us to gain insight into many different aspects of their business. In order to get that insight, data needs to be consolidated from different tools and sources into a data warehouse. Fast-growing organizations like Casper, InVision, Charity: Water and Laurel & Wolf all rely on Looker and to build data-driven cultures.

Stitch integrates with more than 50 different data sources, but we know that there are many more places where your data might live. When our customers need a data source that isn't supported by commercial ETL tools, their option has been to write a custom script to pull the data themselves or hire a 3rd party to write that script for them. While writing a script for a one-time data pull isn't too hard, over the long term it creates a maintenance burden. APIs change, a different data warehouse is chosen, servers go down, cron jobs stop running, API rate limits get hit, and so on. This is why we recommend commercial ETL tools in the first place, but the problem isn't totally solved if only five of your six data sources is supported.

Stitch is introducing to solve that problem. Singer is a better way to write and collaborate on ETL scripts, and it includes three parts:

  • Taps, which pull data from sources
  • Targets, which send data to destinations
  • A JSON-based format for communication between taps and targets

All taps and targets can be mixed and matched, so changing the destination you're loading data into is easy. Since it's all open source, the community can leverage each other's improvements. Some of the Taps built so far on Singer include Marketo, Braintree and Freshdesk.

Singer integrations can be run on hardware that you manage, but Stitch can also run any integration written in the Singer format. By running a Singer integration on Stitch’s platform you'll get auto-scaling, a secure infrastructure, credential management, monitoring, and alerting.

If you have data you need to bring into your data warehouse and Looker’s partners don’t already have a connector built, check out . And if you are looking get a deeper view into your data, get in touch to and trial.

0 replies

Be the first to reply!