Quickstart · sqlflow docs

Turbolytics

2 min read Original article ↗

Create a stream processor that reads data from Kafka in less than 5 minutes.

Getting Started

Run a stream processor that reads a Kafka topic, executes SQL, and writes the output to the console.

What you’ll need

  • Docker
  • A copy of turbolytics/sql-flow cloned on your local machine
git clone https://github.com/turbolytics/sql-flow.git && cd sql-flow
  • The turbolytics/sql-flow docker image. Use v1.0.5 or later; this page pulls v1.0.6.
docker pull turbolytics/sql-flow:v1.0.6
  • Kafka running on your local machine
docker compose -f dev/kafka-single.yml up -d
  • The repo’s Python dependencies, for the sample-data script that publishes events into Kafka. sqlflow itself is a Go binary and needs no Python. Dependencies are locked with uv:
uv sync

Test the sqlflow configuration file

sqlflow’s CLI tests a stream configuration against a fixture file, so you can validate a config before running it against a live stream.

Run the invoke command to test the configuration file against a set of test data:

docker run -v $(pwd)/dev:/tmp/conf -v /tmp/sqlflow:/tmp/sqlflow turbolytics/sql-flow:v1.0.6 dev invoke /tmp/conf/config/examples/basic.agg.mem.yml /tmp/conf/fixtures/simple.json

The following output should show:

{"city":"New York","city_count":28672}
{"city":"Baltimore","city_count":28672}

Run sqlflow against a Kafka stream

This section runs sqlflow as a stream processor that reads from a Kafka topic and writes to the console. sqlflow runs as a daemon: it continuously reads from Kafka, executes the SQL, and writes the output.

  • Publish test messages to the Kafka topic
uv run python3 cmd/publish-test-data.py --num-messages=10000 --topic="input-simple-agg-mem"
  • Start the Kafka Console Consumer, to view the sqlflow output
docker exec -it kafka1 kafka-console-consumer --bootstrap-server=kafka1:9092 --topic=output-simple-agg-mem
  • Start sqlflow
docker run -v $(pwd)/dev:/tmp/conf -v /tmp/sqlflow:/tmp/sqlflow -e SQLFLOW_KAFKA_BROKERS=host.docker.internal:29092 turbolytics/sql-flow:v1.0.6 run /tmp/conf/config/examples/basic.agg.mem.yml --max-msgs=10000

The following output should begin to show in the kafka console consumer:

{"city":"San Francisco","city_count":181}
{"city":"New York","city_count":207}
{"city":"Miami","city_count":210}
{"city":"Baltimore","city_count":209}
{"city":"Asheville","city_count":193}

That is one batch, not the whole run. The config aggregates every batch_size messages, which this file sets to 1,000, and the sample data picks one of five cities at random per event, so each batch emits five rows whose counts sum to 1,000. Ten batches cover the 10,000 messages: 50 rows out, summing to exactly 10,000, roughly 2,000 per city. Your counts will differ run to run.