GitHub - OpenDCAI/DataFlow: Easy Data Preparation with latest LLMs-based Operators and Pipelines.

📰 1. News

• [2026-02-02] 🖥️ DataFlow WebUI is now available! Launch the visual pipeline builder with a single command: dataflow webui. Build and run DataFlow pipelines through an intuitive web interface. 👉 WebUI Docs

  

• [2026-01-20] 🌟 Awesome Works Using DataFlow is now live! A new section showcasing open-source projects and research built on DataFlow. Contributions are welcome! 👉 Awesome Works

• [2025-12-19] 🎉 Our DataFlow technical report is now available! Read and cite our work on arXiv: https://arxiv.org/abs/2512.16676

• [2025-11-20] 🤖 Introducing New Data Agents for DataFlow! Try them out and follow the tutorial on Bilibili: https://space.bilibili.com/3546929239689711/lists/6761342?type=season

• [2025-06-28] 🎉 DataFlow is officially released! Our data-centric AI system is now public. Stay tuned for future updates.

🔍 2. Overview

DataFlow is a data preparation and training system designed to parse, generate, process, and evaluate high-quality data from noisy sources (PDF, plain-text, low-quality QA), thereby improving the performance of large language models (LLMs) in specific domains through targeted training (Pre-training, Supervised Fine-tuning, RL training) or RAG using knowledge base cleaning. DataFlow has been empirically validated to improve domain-oriented LLMs' performance in fields such as healthcare, finance, and law.

Specifically, we are constructing diverse operators leveraging rule-based methods, deep learning models, LLMs, and LLM APIs. These operators are systematically integrated into distinct pipelines, collectively forming the comprehensive DataFlow system. Additionally, we develop an intelligent DataFlow-agent capable of dynamically assembling new pipelines by recombining existing operators on demand.

🛠️ 3. Operators Functionality

🔧 3.1 How Operators Work

DataFlow adopts a modular operator design philosophy, building flexible data processing pipelines by combining different types of operators. As the basic unit of data processing, an operator can receive structured data input (such as in json/jsonl/csv format) and, after intelligent processing, output high-quality data results. For a detailed guide on using operators, please refer to the Operator Documentation.

The design of DataFlow operators follows a PyTorch-like style, making them easy to understand and use. The code block below shows a minimal invocation example of PromptedGenerator:

Example input data (json/jsonl-style):

// input.json
[
  {"problem": "What is 17 + 25?"},
  {"problem": "If x = 3, compute 2x^2 + 1."}
]

Operator invocation code:

from dataflow.operators.core_text import PromptedGenerator
from dataflow.utils.storage import FileStorage
from dataflow.serving import APILLMServing_request

# set input file to global storage class
storage = FileStorage(first_entry_file_name="./input.json",)

# configure LLM serving (e.g., OpenAI API)
# api key needs to be set via `export DF_API_KEY=sk-xxx`
llm_serving = APILLMServing_request(
    api_url="https://api.openai.com/v1/chat/completions",
)

prompted_generator = PromptedGenerator(
    llm_serving=llm_serving,  # pre-configured LLM backend
    system_prompt="Please solve this math problem."
)

prompted_generator.run(
    storage=self.storage.step(),  # data management (details omitted)
    input_key="problem",          # read from this column
    output_key="solution"         # write to this column
)

After running, the operator will append the generated results into output_key. For example, the output data (json/jsonl-style) becomes:

// dataflow_step1.json
[
    {"problem":"What is 17 + 25?","solution":"42"},
    {"problem":"If x = 3, compute 2x^2 + 1.","solution":"19"}
]

📊 3.2 Operator Classification System

In the DataFlow framework, operators are divided into three core categories based on their functional characteristics:

🛠️ 4. Pipelines Functionality

🔧 4.1 Ready-to-Use PipeLines

Current Pipelines in Dataflow are as follows:

⚙️ 4.2 Flexible Operator PipeLines

In this framework, operators are categorized into Fundamental Operators, Generic Operators, Domain-Specific Operators, and Evaluation Operators, etc., supporting data processing and evaluation functionalities. Please refer to the documentation for details.

🤖 4.3 Agent Guided Pipelines

• DataFlow Agent: An intelligent assistant that performs data analysis, writes custom operators, and automatically orchestrates them into pipelines based on specific task objectives.

  • 
  • [HuggingFace🤗 demo input & output for DataFlow Agent]

⚡ 5. Quick Start

🛠️ 5.1 Environment Setup and Installation

DataFlow supports Python>=3.10 environments, tested passed on Windows, Linux, and MacOS with Python 3.10, 3.11, and 3.12.

Please use the following commands for environment setup and installation👇

We recommend use uv to install DataFlow for speed up.

pip install uv
uv pip install open-dataflow

If you want to use your own GPU for local inference, please use:

pip install uv
uv pip install open-dataflow[vllm]

After installation, you can use the following command to check if dataflow has been installed correctly:

If installed correctly, you should see:

open-dataflow codebase version: 1.0.0
        Checking for updates...
        Local version:  1.0.0
        PyPI newest version:  1.0.0
You are using the latest version: 1.0.0.

🐳 5.1.1 Docker Installation (Alternative)

We also provide a Dockerfile for easy deployment and a pre-built Docker image for immediate use.

Option 1: Use Pre-built Docker Image

You can directly pull and use our pre-built Docker image:

# Pull the pre-built image
docker pull molyheci/dataflow:cu124

# Run the container with GPU support
docker run --gpus all -it molyheci/dataflow:cu124

# Inside the container, verify installation
dataflow -v

Option 2: Build from Dockerfile

Alternatively, you can build the Docker image from the provided Dockerfile:

# Clone the repository (HTTPS)
git clone https://github.com/OpenDCAI/DataFlow.git
# Or use SSH
# git clone git@github.com:OpenDCAI/DataFlow.git

cd DataFlow

# Build the Docker image
docker build -t dataflow:custom .

# Run the container
docker run --gpus all -it dataflow:custom

# Inside the container, verify installation
dataflow -v

Note: The Docker image includes CUDA 12.4.1 support and comes with vLLM pre-installed for GPU acceleration. Make sure you have NVIDIA Container Toolkit installed to use GPU features.

🚀 5.2 Quick Start with Google Colab

You can start your first DataFlow translation project directly on Google Colab. By following the provided guidelines, you can seamlessly scale from a simple translation example to more complex DataFlow pipelines.

👉 Start DataFlow with Google Colab

📖 5.3 Reference Project Documentation

For detailed usage instructions and getting started guide, please visit our Documentation.

🖥️ 5.4 WebUI

DataFlow provides a Web-based UI (WebUI) for visual pipeline construction and execution.

After installing the DataFlow main repository, simply run:

This will automatically download and launch the latest DataFlow-WebUI and open it in your browser (http://localhost:<port>/ if it does not open automatically).

📚 Documentation

• Chinese: https://wcny4qa9krto.feishu.cn/wiki/F4PDw76uDiOG42k76gGc6FaBnod
• English: https://wcny4qa9krto.feishu.cn/wiki/SYELwZhh9ixcNwkNRnhcLGmWnEg

🛠️ Development Repository

• https://github.com/OpenDCAI/DataFlow-webui

🧪 6. Experimental Results

For Detailed Experiments setting, please visit our DataFlow Technical Report.

6.1 Text Pipeline

6.1.1 Pre-training data filter pipeline

From the SlimPajama-627B corpus, we extract a 100B-token subset and apply multiple DataFlow text-pretraining filters. We train a Qwen2.5-0.5B model from scratch for 30B tokens using the Megatron-DeepSpeed framework, the results are as follows:

6.1.2 SFT data filter and synthesis pipeline

To study small-scale SFT data quality, we fine-tune the Qwen2.5-7B base model using LLaMA-Factory on WizardLM and Alpaca datasets. For each dataset, we compared a randomly sampled set of 5K instances against a set of 5K instances filtered by DataFlow's SFT pipeline. Additionally, we synthesize a 15k-size dataset, DataFlow-SFT-15K, using DataFlow’s Condor Generator and Condor Refiner pipeline, followed by DataFlow’s SFT filtering pipeline (excluding the Instagram filter). Benchmarks include comprehensive Math, Code, and Knowledge evaluation suites.

Math Benchmarks

Code Benchmarks

Knowledge Benchmarks

6.1.3 Conversation Synthesis Pipeline

We synthesize DataFlow-Chat-15K using DataFlow's conversation-generation pipeline and fine-tune Qwen2.5-7B-Base on it. Baselines include ShareGPT-15K, UltraChat-15K, and their full (non-truncated) versions. We evaluate on domain-specific tasks (TopDial, Light) and general benchmarks (MMLU, AlpacaEval, Arena-Hard).

Conversation Benchmarks

General Benchmarks

6.2 Reasoning Pipeline

We adopt the NuminaMath dataset as a high-quality seed dataset. We compare three training sources: (1) a random 10K subset from Open-R1, (2) a random 10K subset from Synthetic-1, and (3) our 10K synthesized DataFlow-Reasoning-10K dataset constructed using DataFlow.

6.3 Code PipeLine

We randomly sample 20k instances from the Ling-Coder-SFT corpus and process them through the DataFlow Code Pipeline. This yields three curated code instruction datasets of different scales, DataFlow-Code-1K, DataFlow-Code-5K, and DataFlow-Code-10K, each designed to provide high-quality, pipeline-refined supervision signals for code generation tasks.

We compare our synthesized datasets against Code-Alpaca-1k and Self-OSS-Instruct-SC2-Exec-Filter-1k.

Trained on Qwen2.5-7B-Instruct

Trained on Qwen2.5-14B-Instruct

📄 7. Publications

Our team has published the following papers that form core components of the DataFlow system:

Contributing Institutions:

🏆 8. Awards & Achievements

We are honored to have received first-place awards in two major international AI competitions, recognizing the excellence and robustness of DataFlow and its reasoning capabilities:

💐 9. Acknowledgements

We sincerely thank MinerU for their outstanding work, whose powerful PDF/document text extraction capabilities provided essential support for our data loading process. We also thank LLaMA-Factory for offering an efficient and user-friendly framework for large model fine-tuning, which greatly facilitated rapid iteration in our training and experimentation workflows. Our gratitude extends to all contributors in the open-source community—their efforts collectively drive the development of DataFlow. We thank Zhongguancun Academy for their API and GPU support.

🌟 10. Awesome Work Using DataFlow & DataFlow Ecosystem

This section highlights projects, research works, and applications built on top of DataFlow or deeply integrated with the DataFlow ecosystem.

📌 Curated list of featured projects: [Awesome Work Using DataFlow]

We warmly welcome the community to contribute new entries via Pull Requests. 🙌 Detailed Guidance can help you creating a Dataflow extension repository from DataFlow-CLI.

🤝 11. Community & Support

Join the DataFlow open-source community to ask questions, share ideas, and collaborate with other developers!

• 📮 GitHub Issues: Report bugs or suggest features

• 🔧 GitHub Pull Requests: Contribute code improvements

• 💬 Join our community groups to connect with us and other contributors!

📜 12. Citation

If you use DataFlow in your research, feel free to give us a cite.

@article{liang2025dataflow,
  title={DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI},
  author={Liang, Hao and Ma, Xiaochen and Liu, Zhou and Wong, Zhen Hao and Zhao, Zhengyang and Meng, Zimo and He, Runming and Shen, Chengyu and Cai, Qifeng and Han, Zhaoyang and others},
  journal={arXiv preprint arXiv:2512.16676},
  year={2025}
}