Passmark: The open-source Playwright library for AI regression testing

4 min read Original article ↗

Open-source AI regression testing

Covers your browser regression testing end-to-end and helps you catch regressions early. Fast. AI executes natural language steps via Playwright, with intelligent caching, auto-healing, and multi-model assertion verification.

Your tests stay stable without needing to update AI prompts or retrain models.

The problem

Your test suite can't keep up with your team.

Your engineers ship 50 PRs a week. AI coding tools made them faster. But testing? Still manual. Still brittle. Still someone else's problem. Most teams live in one of three places:

Passmark takes a different approach. ↓

No tests.

You skipped e2e testing because nobody had time to write and maintain a Playwright suite. Regressions hit production. Users find the bugs.

Broken tests.

You wrote tests six months ago. Half are skipped as the user flows have changed. The rest flake on every CI run. Your team ignores the failures. The suite is dead weight.

AI tools too slow.

AI testing tools promise to fix this. But they run every action through an LLM, every time. That works for checking a single PR. It doesn't work for 500 regression tests running on every commit. Too slow. Too expensive. Too unpredictable for CI.

How it works

AI once. Playwright speed forever.

Write test steps in plain English. No selectors. No page objects. No boilerplate.

On first run, AI agents navigate your app using accessibility snapshots and screenshots. Every action is cached to Redis when possible.

Every run uses cached Playwright actions when available. Native speed. No LLM calls. No API costs. When a cached action fails because the UI changed, AI re-engages only for that step and re-caches.

Code

This is a Passmark test.

8 lines of test logic. Plain English. Runs on Playwright. Self-heals when your UI changes. Cached after first run.

Compare that to the 40-line Playwright test with data-testid selectors, waitForSelector calls, and hardcoded timeouts you'd write by hand. Then imagine maintaining 200 of those.

Comparison

Where Passmark sits.

Hand-written Playwright

  • Plain English test authoring
  • Runs at native Playwright speed
  • Self-heals on UI changes
  • Built for CI (500+ test suites)
  • Cross-test state sharingManual wiring
  • Email and OTP testingExternal tools
  • AssertionsBrittle
  • $0 AI cost on repeat runs
  • Open-source
  • Plain English test authoring
  • Runs at native Playwright speedYes (cached)
  • Self-heals on UI changes
  • Built for CI (500+ test suites)
  • Cross-test state sharingBuilt-in (Redis)
  • Email and OTP testingBuilt-in
  • AssertionsVision models with consensus
  • $0 AI cost on repeat runs
  • Open-source

Real-time AI tools

  • Plain English test authoring
  • Runs at native Playwright speed
  • Self-heals on UI changesPartial
  • Built for CI (500+ test suites)
  • Cross-test state sharing
  • Email and OTP testing
  • AssertionsVision models
  • $0 AI cost on repeat runs
  • Open-sourceSome

Built for regression

Everything you need at test 200 that you don't think about at test 1.

Cross-test state

Sign up in test 1. Log in with the same credentials in test 2. Global placeholders persist across tests via Redis.

Built-in email testing

Comes bundled with disposable inboxes via emailsink integration. Extract OTPs, magic links, or any content from incoming emails with natural language prompts.

Consensus assertions

Every assertion is evaluated by Claude and Gemini independently. A third model breaks ties. Passes only on consensus. Returns a confidence score (0-100).

Dynamic test data

Every run gets unique IDs, emails, names, phone numbers. Parallel tests never collide. Three scopes: {{run.*}} (per test), {{global.*}} (per execution), {{data.*}} (per project).

CI integration

npx playwright test. That's it.

Your existing Playwright config, fixtures, and CI setup work as-is. Passmark is a library you import inside Playwright tests - not a separate runner.

  • GitHub Actions, GitLab CI, Jenkins, any CI runner
  • Parallel execution with zero shared-state collisions
  • Tracing for every step, cache hit, and assertion
  • Redis for action caching and cross-test state
About

What is Passmark?

Passmark is an open-source AI regression testing library built on Playwright. Developers write end-to-end tests in plain English. AI agents execute tests on first run and cache every browser action to Redis. Subsequent runs replay cached actions at native Playwright speed with zero LLM calls. When UI changes cause a cached action to fail, the AI automatically re-discovers the correct interaction and updates the cache. Passmark includes built-in email testing, cross-test state management, dynamic test data generation, and multi-model consensus assertions. It requires Node.js 18+, Playwright 1.59+, Redis, and API keys for Anthropic and Google AI.

Team

Built by the team behind Bug0 and Hashnode.

Passmark is the open-source core of Bug0, an AI-native QA platform trusted by 200+ engineering teams. Built by Fazle Rahman and Sandeep Panda, who previously created Hashnode, a developer blogging platform with millions of monthly readers.

Backed by Accel, Guillermo Rauch (Vercel), and Naval Ravikant.

Star on GitHub

Go managed

Love the engine? Let someone else run it.

Passmark is the open-source core. Bug0 Managed is the service built on top of it. A dedicated QA engineer writes your tests, triages every failure, and gates your releases. Same engine. Zero maintenance on your end.

  • Test enginePassmark

  • Who writes testsYou

  • Who triages failuresYou

  • InfrastructureYours

  • PriceFree

Get Started

  • Test enginePassmark

  • Who writes testsDedicated QA engineer

  • Who triages failuresHuman-verified reports

  • InfrastructureBug0 AI agents for testing

  • PriceFrom $2,500/mo

Learn more at bug0.com