Serverless ETL is becoming the future for teams looking to stay focused on their core responsibilities rather than running a large infrastructure to power data pipelines. Nextdoor’s mission is to build community, and I joined the Systems Infrastructure team to help support product engineers in achieving that goal. The team’s responsibilities include managing the data ingestion pipeline which services 2.5 billion daily syslog and tracking events. We found that, as our data volumes grew, keeping the data ingestion pipeline stable became a full time endeavor that distracted us from our other responsibilities.
Upon joining Nextdoor, my first project was to replace the aging Apache Flume-based pipeline, which served the business well for years. As we investigated alternative solutions, we had two use cases in mind:
- Streaming logs into Elasticsearch with an SLA of 1 minute.
- Writing partitioned batches of data into S3 with an SLA of 5 minutes. Specifically, the current hour’s data must be available no later than 5 minutes after the end of the hour. This is to support downstream ETL processes.
Obviously, a streaming solution lends itself well to these requirements and there are a lot of options in this space. The popular traditional solutions include Flume, Kafka+Storm, Kafka Streams, Flink, Spark, and many others. As well as up and coming solutions like Kinesis+Lambda from Amazon, Cloud Dataflow from Google, Azure Functions from Microsoft, and IronWorker from Iron.io.
The Divide
All of these solutions have scale, performance, operational, and monetary tradeoffs, but there are two overarching schools of thoughts which stem from the advent of cloud based infrastructure as a service. You can either invest in operating your own infrastructure or offload that responsibility to an infrastructure as a service provider. Amazon Web Services (AWS) pioneered this field and this now allows many companies, like Nextdoor, to focus more on developing product rather than running infrastructure. Cloud providers like AWS have gone beyond simply offering to offload the hardware aspect of infrastructure by packaging and offering SaaS for commonly used services like webservers, databases, and now Big Data platforms.
For instance, Hadoop’s core services haven’t gone away but rather have been packaged into services like EMR, Qubole, and Altiscale to decouple running a cluster from running your queries and ETL jobs. Nextdoor’s Data/BI team uses Qubole in this manner.
What is Serverless?
Each of the Serverless options from Amazon, Microsoft, Google, and Iron.io have subtleties but the premise is you are able to create and upload code that is automatically run for you. You no longer need to worry about managing servers, services, and infrastructure. All of that is handled for you. This, of course, does not come without monetary cost — but in most cases, you will find cost savings in no longer running servers or not having a dedicated engineer operating your pipeline.
How do you fit ETL into this concept? Most of these providers also have various data store or data stream technologies which can be linked with your code. For instance, AWS Lambda is able to trigger your code each time a file is uploaded to S3 or events streamed to Kinesis or written to DynamoDB. All you have to do is supply is code to process that data.
Get Slava Markeyev’s stories in your inbox
Join Medium for free to get updates from this writer.
Here’s an example of a Amazon Lambda function written in Python:
You can see how Serverless is a powerful shift in how ETL is performed. However, it is still a young approach and is missing some pieces such as standardization of code. Engineers are able to write lightweight functions to process data but this often leads to fragmentation of approaches within teams, organizations, and the greater open source community. Boilerplate logic for configuration, exception handling, deserialization, transformation, transport, retrying, and monitoring is constantly reimplemented, mis-implemented, and in some cases punted on. I aimed to solve this problem by developing Bender.
Introducing Bender
Press enter or click to view image in full size
Bender provides an extendable Java framework for creating serverless ETL functions on AWS Lambda. Bender is split up into two components, Bender Core handles the complex plumbing and provides the interfaces necessary to build modules for all aspects of the ETL process, while Bender Modules implement the most common use cases.
When designing Bender, we wanted a function that we could easily reconfigure to handle different use cases while leveraging common boilerplate logic like error handling, statistical reporting, retrying, configuration, etc. Bender Core performs the batching and routing of data and provides interfaces for aspects of the ETL process.
Out of the box, Bender includes modules to read, filter, and manipulate JSON data as well as semi-structured data parseable by regex from S3 files or Kinesis streams. Events can then be written to S3, Firehose, Elasticsearch, or even back to Kinesis.
It is easy to implement additional support for different data log formats found in AWS and on machines such as syslog, apache logs, VPC flow logs, etc, as well as different data sources such as DynamoDB and destinations like a SQL database.
Partitioning Example
A common use case is to re-partition data files written by Firehose into a different structure. With Bender, this is easy to do with the partition operation. For instance, if your json has 3 fields (HOST, FACILITY, and EPOCH) that you wish to partition by, your configuration may look like this:
Additional examples with full configuration and documentation are provided in Bender’s sample configurations. In a future part of this series, we will focus on the ins-and-outs of how we use Bender in our infrastructure and the implementation and design decisions we made when working with Lambda, Kinesis, and Elasticsearch.
Final Thoughts
Working with cloud based services doesn’t eliminate engineering effort — rather, it shifts it. My team, Systems Infrastructure, helps bridge the gaps between servers and services so that our engineers can focus on the main goal of Nextdoor: building a private social network for neighborhoods. This means that sometimes we run services and other times leverage cloud offerings while writing projects like Bender to make life easier.
Find this sort of stuff cool? The Nextdoor engineering team is always looking for motivated and talented engineers.