Buildkite Test Engine | Remove flaky tests, optimize performance

3 min read Original article ↗

Buildkite Test Engine

Deliver confidently with fast and reliable tests

Buildkite Test Engine speeds up builds with real-time flaky test management and intelligent test splitting. Drive accountability, customize actions, and get more out of your existing CI compute with performance insights and analytics.

Test suite reliability
89.73%

Flaky assigned to

Marketing team

Cart abandonment email triggers after 1 hour of inactivity

./spec/

cart_abandonment_email_triggers_after_1_hour_of_inactivity_spec.rb

💻

:ruby:

Protagonist

Rails monolith

📱

:swift:

Flaky tests? React in real time

Identify, categorize, and take action on flaky tests with workflows to save time and resources throughout your testing process and keep the critical path clear.

Detect with precision

Configure workflows with multiple detection heuristics that identify intermittent failures and flaky test patterns to eliminate wasteful retries, and keep merge queues moving by automatically categorizing and quarantining problematic tests.

Assign team ownership

Take control of test management by mapping tests to teams and automatically assigning ownership when troublesome test cases are detected.

An example CODEOWNERS file window with a foreground visual showing assigned flaky tests

Customize your flaky test response

Configure workflows for different test types or teams, with fine-grained control. Create separate monitoring strategies for feature tests, unit tests, or specific environments. Generate custom notifications that range from webhooks to Slack notifications and creating Linear tickets.

A visualization of workflows A visualization of workflows

Find your slowest tests, fast

See which tests are slowing your builds sooner, so you can solve your biggest problems faster.

Understand test performance

Identify and optimize slow or poor performing tests using timing data analysis from all test runs.

Use timing data to split work

Group tests into parallel jobs based on timing data—using the built-in intelligent sorting or your own rules for distributing tests.

Optimize tests on any platform

Get test insights from any CI/CD tool: Buildkite Pipelines, GitHub Actions, Jenkins, CircleCI, and more.

Buildkite, CircleCI, Jenkins, and GitHub Actions logos

Real-time insights and analytics

Gain critical test insights with a unified view.

Get a comprehensive view of all tests

Understand what’s running and what needs attention across your test suites, all from one place. Unit tests, integration tests, end-to-end tests—see analysis of your regression testing health in a single view.

💻

:ruby:

Protagonist

Rails monolith

Runs

0/week

📱

:swift:

Runs

0/week

:react:

:jest:

Runs

0/week

Report on progress

Automate reports on the status of unreliable or troublesome tests across teams on a configurable cadence.

A digest email report on top 5 flaky tests A digest email report on top 5 flaky tests

Deep tracing and instrumentation

Automatic tracing helps you see why tests are slow or behaving differently between executions.

Slowest SQL queries

Slowest HTTP requests

Span timeline

HTTP

GET http://selenium:4444/wd/hub/session

349ms

SQL

SELECT session_id FROM sessions WHERE

8ms

HTTP

GET http://selenium:4444/wd/hub/session

25ms

HTTP

POST http://selenium:4444/wd/hub/session

8ms

SQL

SELECT session_id FROM sessions WHERE

8ms

SQL

INSERT INTO otp_attempts (user_id, created_at

8ms

Frequently asked questions

Got a question that’s not on our list? Want a demo? Just want to chat? Get in touch.

Resources