Trying Claude for E2E Testing on an Airbnb Clone

TLDR: I built Bunkly, a test-friendly Airbnb clone, and ran Claude Code through every stage of E2E testing: test planning, spec generation, parallel runs, GitHub Actions, and updating tests after a UI redesign. Claude worked best as a contributor to my process, not a replacement for the decisions I still had to make. You can use its knowledge of Playwright APIs and common testing patterns, and it will scaffold setup and auth projects and fill out happy-path flows faster than writing them by hand. But it needs you to guide it: app routes, seed data, environment variables, and workflow boundaries are things Claude cannot infer on its own. It will produce specs that look complete but silently pass for the wrong reasons, especially when product changes shift behavior under the hood. Treat Claude as a way to accelerate a first draft, then review its output for correctness before you trust it in production.

You hear all the time about teams struggling to create and maintain E2E tests that keep pace with the speed at which they’re shipping. They might initially invest time in creating an initial suite only to watch the suite turn flaky, slow, or get ignored when failures pile up [1].

I’ve heard people talk about using Claude Code for QA and wanted to see if Claude could help with the planning side of QA work: test planning, generating specs, running tests locally, setting up GitHub Actions, and updating tests when the UI changed.

As part of my test, I built a production-ready Airbnb-style hotel app called Bunkly, complete with search, checkout, loyalty, and messaging. I decided to start with a clean slate, an app with no Playwright tests yet.

I have attached the full Claude Code transcript [2] for you to read alongside the blog.

The Bunkly codebase was designed to be deliberately test-friendly. Interactive elements carry `data-testid` attributes, seed data is deterministic (fixed slugs and known passwords in `CLAUDE.md`), and `ARCHITECTURE.md` explains how Server Components and server actions split responsibilities.

You’d think that would make things straightforward. Instead, I hit selectors that weren’t in the code, odd feature priorities (wishlists ranked above booking), and CI jobs that skipped hidden directories.

This post covers what happened at each step of trying Claude Code for QA testing, what worked, what broke, and what I still had to verify in the code, seed data, and CI logs.

Set up Bunkly locally

The first step was to get Bunkly running locally. I cloned the repository and seeded the database with the following commands:

git clone https://github.com/rishi-raj-jain/bunkly
cd bunkly
npm install
cp .env.example .env   # set DATABASE_URL
npm run db:migrate     # or db:push in dev
npm run db:seed
npm run dev            # http://localhost:3000

The seed step creates about ten to fifteen properties across five cities, rolls inventory forward for a year, and adds users with plaintext passwords for test automation. Without that seed data, flows like logging in as a user named Sarah, viewing upcoming bookings, or testing loyalty enrollment would have nothing to assert against.

To enable browser-aware changes in Claude Code, I also set up the Playwright MCP so threads could refer to real navigation and DOM behavior along with file changes.

Claude for QA Test Planning

Before automating the tests via code, I asked Claude for a test plan keyed to Bunkly’s main workflows, e.g., discover properties, complete a booking end-to-end, manage profile and payment methods, notifications, message threads, loyalty (where seed allows), and secondary flows like wishlists and price alerts. Here is the prompt I used:

Produce a prioritized E2E test matrix in docs/cc-test-plan.md with P0/P1/P2 cases.

Claude’s first attempt at a test plan was thorough on the surface. It covered the major features and some lesser-used ones, such as price alerts, wishlists, reviews after check-out, and flows for editing bookings. The plan was laid out as a large table with P0, P1, and P2 priorities and tried to balance common paths with edge cases.

Taking stock of the app’s features was genuinely helpful. It forced me to consider corners of the product I might have skipped otherwise.

Here are a few prompts I sent after the initial test plan was generated (source):

can you move the tests from P2 to PQ that makes sense, we are a booking company so we need the ability to make sure that search is the top priority

but how is wishlist a P1?

why adding payment method is P1?

why did you move payment method to P2? It was in P1 and it should be a P0 instead since we need to be able to process money!

why in the world is forgot password a P2?

Several important gaps around using Claude for QA became clear when I reviewed the test plan:

• The priorities in the plan (e.g., high/medium) didn’t always match practical risk-cosmetic toggles, which were marked higher than critical for end-to-end booking flows.
• Some scenarios described in the plan didn’t align with what’s possible in the current codebase. For example:
  - The plan included saving wishlists, but the UI and seed data didn’t allow wishlist actions for any seeded user, so these tests were not feasible.
  - Review tests referenced features without establishing the necessary preconditions. Review flows require users with prior completed bookings. Without seeded bookings for the user, these tests could never run as intended.
  - Messaging tests assumed the existence of seeded conversations linked to a user. Without a specific database setup for these conversations, messaging tests could not execute successfully.

There was a disconnect between feature coverage on paper and what could be executed, as the plan didn’t specify the test prerequisites or acknowledge app limitations. Overall, these mismatches highlighted the need to bridge the gap between the features listed in the plan and the test scenarios that can actually be run manually.

For this kind of app, making every test scenario actually useful means spelling out things like the specific user account (with credentials pulled from CLAUDE.md), how to navigate to the right page, which selectors to use (and making sure they match real data-testids from the codebase), what should change in the database if the test performs a mutation, and how to reset everything between runs to keep things clean and reproducible.

Claude was great at firing off a wide-ranging list so I didn’t forget anything major, but it couldn’t translate that list into something directly actionable, something I could truly trust day-to-day, or something I could hand to someone to use for proper QA.

Claude for E2E Test Generation

Once the test plan was refined (and I had it create the final one in `cc-test-plan-after.md`), I asked Claude to generate Playwright tests accordingly, with the following prompt:

now read the cc-test-plan-after.md file and then write the tests using playwright (feel free to use the playwright mcp if required)

Once the initial tests were scaffolded, I followed up with operational instructions (wonder why Claude hadn’t done that):

add the instructions to run the tests in README.md

And to ensure the automated runner would actually hit the production build, I needed to specify (since Claude didn’t pick that up automatically):

to run the web server, use npm run start and not npm run dev

The output included a standard Playwright structure: a playwright.config.ts file (with the webServer command), global setup scripts, a set of initial spec files in the tests/e2e/ directory, and some helpers for navigation and repeated interactions. While I hadn’t used Playwright’s multiple project configuration model before, the generated split between a general setup project and an authenticated-user project matched what I would have done.

Next, I ran npx playwright test and encountered a set of initial failures, including:

• Incorrect selectors and URLs in the tests
• Assertions that did not match the actual DOM structure
• Incorrect step order and issues related to async UI updates
• Test navigation/routes and business logic that were not aligned with the codebase
• Use of test.skip() in place of a proper seed-backed test setup
• Tests missing authentication contexts

To better understand the causes of these failures, I examined each issue in detail. Below, I break down what went wrong in each area.

Issue 1: Tests didn’t use the right selectors and URLs

Although CLAUDE.md carefully documents the expected data-testid attributes for interactive UI elements, the generated Playwright specs referenced the test IDs and selector names that simply do not exist in the codebase.

• For example, the specifications for profile updates assumed the presence of a profile-form wrapper, however, upon inspecting the relevant React components, it became clear that no such element is defined with that identifier. To fix this, I referenced the source code directly.

Another source of test failure was the mismatch in the query parameters used:

• For example, the application expected a destination query parameter on the results page. However, both the generated specs and a reusable helper function used the parameter name location instead. This single issue affected multiple specs because it was present in shared navigation code, resulting in every search-related test failing.

These adjustments definitely made me aware that anyone using Claude Code for QA testing needed to be someone who understood how the application source and test code behaved, so they could correct them rather than rely on Claude’s assumptions, especially as both code and tests change over time.

Issue 2: Tests didn’t make assertions based on the actual UI

For example, the payment method specs asserted that the first four digits of a card should be displayed after adding it. However, the wallet UI is designed to display only the last 4 digits to comply with PCI best practices. As a result, the test assertion failed every time.

I noticed similar issues when tests made assertions about UI text or record counts that didn’t match the seeded data (for example, expecting a certain number of notifications even though no notification rows were populated during setup).

These mismatches suggested the tests were partly based on assumptions about the intended UI rather than on the actual state and structure of the codebase Claude was working on. The need to check output for this adds extra time and risk if using Claude Code for E2E testing.

Issue 3: Tests didn’t handle async UI timing or state

One flow expected a service-section to be visible before the user opened the service request entry point. The app renders that section after the click.

Another spec toggled a notification category and immediately called page.reload(), overlapping React useTransition on that settings page. The runner reported timeouts and stale state that looked like flakes until I reordered steps and waited on the right locators.

I told Claude to always use expect(locator).toBeVisible() (not assumptions), reference real data-testids from components, check server actions for business rules, and use networkidle only when truly needed. After tightening waits and removing a reload, failures dropped from 8 to 3, showing which were timing vs. logic issues.

However, investigating and fixing these issues took time and reduced the time savings I was hoping to realize from using Claude Code for QA.

Issue 4: Tests didn’t check real routes or logic

For example, some test specs made assumptions about app routes or business logic without checking the implementation:

• Price alert tests tried to visit /alerts, assuming a standalone alerts page existed, but in reality, price alert functionality lives within each property (properties/[slug]/) and in the account’s price alert settings (not a top-level route).

• One of the messages spec navigated to /messages/ (with a trailing slash) and tried to confirm the presence of that URL even though the app is navigated at /messages (no trailing slash).

• Review-related specs attempted to review upcoming bookings, when in fact (as enforced by both reviews.ts and seeded data) only completed stays are eligible for reviews.

Issue 5: Tests didn’t set up required state

When flows needed two payment methods on file, or a non-default card to delete or to switch default, the agent sometimes emitted test.skip() with a comment that setup was hard. Skipped tests do not exercise the feature and still count toward green in CI.

I asked for explicit navigation to payment settings, seed-backed users, or a beforeEach that creates the second method through the same UI path a user would take.

Issue 6: Tests didn’t handle authentication properly

Initially, several tests attempted to access protected routes without logging in, resulting in redirect and timeout errors. Playwright’s dependencies config (like dependencies: [‘setup’]) wasn’t used by default, so not all tests reused the established session until I pointed it out.

Claude for QA: Parallel test runs and the impact of shared database state locally

After debugging specs in isolation, I enabled fuller parallel execution (fullyParallel and multiple workers). Loyalty point assertions and review submission collided when workers logged in as the same seeded user and mutated the same rows.

I pasted worker logs and stack traces into Claude. It suggested grouping the unstable suites with test.describe.configure({ mode: ‘serial’ }) so those no longer run concurrently. I had not used that API on this project before and it was the right tool for the symptoms.

Cases that failed only under parallel load included loyalty balance checks after redemption-style flows and review submission that changes booking or review rows. Serial mode stabilized them locally and I kept a mental list of which domains are read-only versus write-heavy for future sharding when I deploy the tests to CI.

Claude for QA: Managing environment variables, jobs, artifacts, and dotfiles in GitHub Actions

The next step after confirming that everything worked fine locally was for me to move the suite to GitHub Actions (CI).

My Claude Code QA agent’s produced workflow was recognizable i.e. checkout, install dependencies, run seed, shard tests, upload reports. I added DATABASE_URL to the repository secret so that it was accessible. It didn’t go smoothly:

• The first failure was environmental i.e. the seed script read DATABASE_URL from .env locally and required me to prompt the agent to wire it down via the repository secrets in the workflow code.

• Authenticated specs depended on files under tests/.auth/ produced during a setup job. The setup job logged in and wrote storageState to disk. Consumer jobs restored node_modules from cache but not the auth directory. Auth specs then failed or passed depending on whether they accidentally performed login again in the spec body.

• I still had a gap after auth-related tests improved. The setup job had run in the logs, but tests/.auth/ did not appear in the uploaded artifact bundle. The agent’s first explanation was that seed had not run but I knew that seed had run. So, I pointed at an artifact upload configuration and dot directories. GitHub’s default upload patterns often omit paths like .auth unless you include them explicitly in actions/upload-artifact.

• After adding the dotfile paths (or an explicit path to tests/.auth/**), later jobs downloaded the auth bundle and the workflow passed end to end.

Claude for QA: Updating tests when the product changes

After a stable CI run, I asked Claude to update with two UX changes in the application:

• To merge the stay and guest details into one step in the checkouts, and leave payment as the clearer second step.
• Drop the tabbed “upcoming / past” navigation in the bookings view for a single list with filters.

And as a result:

• Checkout edits by Claude mostly matched the new step order. The agent rewrote navigation through the combined step and adjusted payment-step assertions. It also tightened waits on a spec that had been green under loose timing (a race that showed up when step boundaries changed!).
• Bookings tests became harder to pass after the Claude edits as one spec silently passed. I knew it shouldn’t have happened since I removed the tab clicks, but I found that the tests were written so they searched for a specific property name that appeared regardless of the filter state. The assertion did not prove that the “past” filter changed the list (a false negative, CI green without guarding the behavior we changed).

I asked Claude to see what’s going on:

can you read the code and then tell me if clicking past-tab changes the booking-card- list? or does it just takes me to the same page but to the past bookings below the upcoming views instead of it being tabbed

and it finally came to senses with the following proposal:

The fix is to scope the locator to the past section:
const cards = page.locator('[data-testid="bookings-section-past"]
[data-testid”="booking-card-"]"');
That way the loop only iterates over cards inside #bookings-section-past,
regardless of scroll position. Want me to apply that fix?

After that, I simply asked it to read through the failed test results that were automatically reported in a standalone directory by Playwright:

can you read the test-results folder and then fix the tests?

From the logs, I saw that it fixed the add-on scenarios (like breakfast on pending versus confirmed bookings) with the new list-and-filter UI model and also clarified which seed user owned which booking states. For handling booking checks, it added serial grouping to ensure parallel workers were not writing to the same booking (or add-on records).

Advice for developers thinking about using Claude for QA testing

Overall, after having Claude update the code and write tests for the new feature, I saw that simple selector renames get handled quickly if I point the agent directly at the changes, but bigger workflow changes need new assertions that will actually fail if the old UX is still present. For example, things like asserting on filter state in the DOM, row counts, or booking ids that only appear under the right filter, rather than just checking if some property name appears anywhere on the page need new assertions.

After a redesign, I had to clearly define for the agent what “correct” means, rather than assuming it could infer it from the component diffs alone.

While Claude can be helpful for setting up E2E testing, it works best as a contributor to your process rather than an all knowing test author.

You can leverage its knowledge of Playwright APIs and common testing patterns, and it can help with initial scaffolding and fill out happy-path test flows, however, that’s all it does. If you have more complicated use flows, it will likely be very time consuming for you to use Claude for QA testing and it could miss testing critical things.

There’s also considerable manual work required in using Claude for QA. You need to guide it on your specific app’s routes, seed data nuances, environment setup, and workflow boundaries to be fully effective. While it can sort out issues like parallelization and CI artifact handling, you still need to review its output for context-specific correctness, as it may produce false positives or fragile test logic if left unchecked.

Blindly trusting Claude E2E test results or assuming your Claude Code QA agent fully understands new workflows might cause issues to slip through, especially when product changes affect how things work under the hood. Ultimately, Claude might be able to help you get started creating an end-to-end coverage plan but it’s not purpose-built for generating QA tests and so often misses things or creates flaky tests.

[1]: https://medium.com/codetodeploy/our-e2e-tests-were-flaky-we-deleted-half-of-them-8393bb419322

[2]: https://github.com/rishi-raj-jain/bunkly/blob/main/2026-05-26-185638-this-session-is-being-continued-from-a-previous-c.txt

(Co-authored with

)