Press enter or click to view image in full size
On March 24, 2017, we started writing a new kind of workflow tool.
Today, on its two-year anniversary, we’re excited to announce that Prefect has launched as an open-source project with an Apache 2.0 license.
Click here to get started, or dive right into our repo on GitHub!
Press enter or click to view image in full size
Our team has spent years building tools for data engineers and data scientists, including as a PMC of Apache Airflow. Prefect is the culmination of those lessons. Our design is informed by hundreds of data engineering use cases that have allowed us to identify core frictions in the modern data stack. We’ve worked with over a hundred Lighthouse Partners to improve our software over the last six months. And we’ve designed every piece — from the execution engine to the logging to the data handling — to be modular and easily customizable.
As a result, at launch, Prefect is already the easiest, most powerful, and best-tested open-source workflow engine.
Here are a few things Prefect is doing right now:
- Processing terabytes of climate data
- …and running our team’s daily standup via Slackbot
- Powering a quantitative investment strategy
- …and turning on a coffee machine every morning
- Orchestrating the backend of our Cloud execution platform
- …and writing Prefect’s GitHub star count into Airtable
- Parameterizing machine learning models
- …and lots and lots (and lots) of ETL
If you can code it, Prefect can automate it.
What will you build?
Welcome to Prefect!
Prefect is a new kind of workflow management system.
We started with a simple premise:
Your code probably works. But sometimes it doesn’t.
When your code works, you might not even need a workflow system. We call that positive engineering. It’s only when things go wrong that a system like Prefect starts to be valuable. That’s negative engineering: all the little details that guarantee your code either achieves its goal, or fails successfully. In this way, workflow systems are actually risk management tools, like insurance: there when you need them, invisible when you don’t.
And yet, we don’t see a single tool designed that way. Other workflow systems seem to believe that they’re actually positive engineering tools, somehow enabling users to do things they couldn’t do otherwise. As a result, they feel no shame in asking users to generate yet another config file, or contort code into a convoluted DAG structure. Prefect already knows you can write incredible code; it just wants to make sure it works.
Prefect takes your code and transforms it into a robust, distributed pipeline. You can continue to use your existing tools, languages, infrastructure, and scripts. Prefect builds a rich DAG structure, but in a way that respects positive engineering and doesn’t inhibit it. You can use Prefect’s functional API to transform scripts with minimal hooks; or you can access the deferred computational graph directly; or any combination thereof. It’s up to you.
The most common thing we hear about negative engineering is: “This should be easy!”
Prefect is the first step toward making that true.