# Stop Writing Brittle Playwright Tests

> Why YAML-Based Testing is the Future

**Published by:** [AI Dev Journal](https://paragraph.com/@terryso/)
**Published on:** 2025-06-16
**Categories:** claudecode, playwright, uitesting, testing, mcp
**URL:** https://paragraph.com/@terryso/stop-writing-brittle-playwright-tests

## Content

How a simple YAML configuration built for Claude Code and Playwright MCP transformed our testing workflow and made automation accessible to everyone on the teamIf you've ever maintained a large Playwright test suite, you know the pain. Hundreds of lines of JavaScript scattered across dozens of files. Hardcoded values that break when environments change. Test logic so complex that only the original author dares to modify it. What if I told you there's a better way? A way to write tests that are readable by anyone, maintainable by design, and powerful enough to handle complex workflows? Enter YAML-based Playwright testing for Claude Code — a paradigm shift that's changing how teams approach test automation by leveraging the power of Claude Code's AI capabilities and Playwright MCP's browser automation.The Problem with Traditional Playwright TestsLet's be honest about traditional Playwright tests:// Traditional Playwright test - 50+ lines of code test('complete order flow', async ({ page }) => { await page.goto('https://example.com'); await page.fill('[data-testid="username"]', 'user123'); await page.fill('[data-testid="password"]', 'pass456'); await page.click('[data-testid="login-btn"]'); await expect(page.locator('h1')).toContainText('Dashboard'); // ... 40+ more lines of clicking, filling, asserting // ... hardcoded values everywhere // ... no reusability between tests }); Problems:❌ Verbose and complex — Simple actions buried in boilerplate❌ Hardcoded values — Environment changes break everything❌ Poor reusability — Copy-paste leads to maintenance nightmares❌ Technical barrier — Only developers can write/modify tests❌ Scattered logic — Related tests live in different filesThe YAML Revolution: Tests That Make SenseNow imagine the same test written in YAML:# test-cases/order.yml tags: - smoke - order - checkout steps: - include: "login" - "Click Add to Cart button for first product" - "Click Add to Cart button for second product" - "Click shopping cart icon in top right" - "Enter First Name" - "Enter Last Name" - "Enter Postal Code" - "Click Continue button" - "Click Finish button" - "Verify page displays Thank you for your order!" - include: "cleanup" Immediate benefits:✅ Crystal clear intent — Anyone can understand what this test does✅ Natural language — Steps read like user stories✅ Reusable components — Login and cleanup are shared across tests✅ Environment agnostic — No hardcoded values in sightThe Magic Behind the Simplicity1. Reusable Step LibrariesCommon workflows become building blocks:# steps/login.yml steps: - "Open {{BASE_URL}} page" - "Fill username field with {{TEST_USERNAME}}" - "Fill password field with {{TEST_PASSWORD}}" - "Click login button" - "Verify page displays Swag Labs" Write once, use everywhere. No more copy-paste madness.2. Environment Variable MagicDifferent environments? No problem:# .env.dev BASE_URL=https://dev.example.com TEST_USERNAME=dev_user # .env.prod BASE_URL=https://example.com TEST_USERNAME=prod_user Same tests, different environments. Automatically.3. Intelligent Tag FilteringRun exactly what you need:# Run only smoke tests /run-yaml-test tags:smoke # Run order AND checkout tests /run-yaml-test tags:order,checkout # Run smoke OR critical tests /run-yaml-test tags:smoke|critical No more running the entire suite when you only changed the login flow.4. Smart ReportingAutomatically generated HTML reports with:✅ Step-by-step execution details✅ Environment configuration✅ Screenshots and artifacts✅ Success/failure statisticsReal-World Impact: A Case StudyBefore YAML testing:📊 2,000+ lines of Playwright JavaScript⏱ 3 days to onboard new QA team members🐛 15+ test failures per environment change👥 Only 3 developers could modify testsAfter YAML testing:📊 200 lines of readable YAML⏱ 30 minutes to onboard new team members🐛 0 test failures during environment changes👥 Entire team can write and modify testsWhy This Matters for Your TeamFor Developers:Less time writing boilerplate, more time building featuresTests that actually document your application's behaviorNo more "let me just quickly fix this test" rabbit holesFor QA Engineers:Focus on test strategy, not JavaScript syntaxRapid test creation and modificationClear visibility into test coverageFor Product Managers:Tests that read like acceptance criteriaEasy to verify that tests match requirementsConfidence that important flows are coveredFor DevOps:Predictable test execution across environmentsClear failure reporting and debuggingEasy integration with CI/CD pipelinesTechnical Architecture: How It WorksThis YAML Playwright testing framework is specifically designed for Claude Code and Playwright MCP. The framework consists of several key components:Claude Code IntegrationAI-Powered Execution: Claude Code's AI interprets natural language test steps and converts them to Playwright actionsSmart Step Recognition: Advanced understanding of testing intent from plain English descriptionsContext Awareness: Maintains context across test steps for more intelligent automationPlaywright MCP FoundationBrowser Automation: Leverages Playwright MCP for reliable cross-browser testingElement Detection: Intelligent element finding and interactionScreenshot & Reporting: Built-in capture and documentation capabilitiesMulti-Environment Configuration├── .env.dev # Development environment ├── .env.test # Test environment ├── .env.prod # Production environment Reusable Step Libraries├── steps/ │ ├── login.yml # Authentication flows │ ├── cleanup.yml # Cleanup procedures │ └── navigation.yml # Common navigation Test Cases with Natural Language├── test-cases/ │ ├── order.yml # E-commerce order flow │ ├── user.yml # User management │ └── search.yml # Search functionality Intelligent Execution EngineThe framework automatically:Loads environment-specific configurationExpands include references from step librariesSubstitutes environment variables ({{BASE_URL}})Executes tests using Playwright MCPGenerates comprehensive reportsGetting Started: Your First YAML TestThe beauty of YAML-based testing is its simplicity. Here's how to get started:1. Prerequisites# Install Claude Code (if not already installed) # Follow instructions at: https://claude.ai/code # Install Playwright MCP for Claude Code claude mcp add playwright -- npx -y @playwright/mcp@latest # Clone the YAML testing framework git clone https://github.com/terryso/claude-code-playwright-mcp-test.git cd claude-code-playwright-mcp-test 2. Project Structureyour-project/ ├── .env.dev # Environment config ├── steps/ # Reusable step libraries ├── test-cases/ # Your test cases ├── screenshots/ # Test artifacts └── reports/ # Generated reports 3. Write Your First Test# test-cases/login.yml tags: - smoke - auth steps: - "Open {{BASE_URL}} page" - "Fill username with {{TEST_USERNAME}}" - "Fill password with {{TEST_PASSWORD}}" - "Click login button" - "Verify successful login" 4. Execute and Iterate# In Claude Code, use the built-in commands /run-yaml-test file:test-cases/login.yml env:dev # Or run with tag filtering /run-yaml-test tags:smoke env:dev Within hours, you'll have tests that are more maintainable than anything you've written before. The magic happens through Claude Code's AI understanding your natural language steps and Playwright MCP executing them as browser actions.Advanced FeaturesComplex Tag Filtering# Multiple conditions /run-yaml-test tags:smoke,login|critical # Environment-specific execution /run-yaml-test tags:order env:prod Dynamic Step Parameterssteps: - "Add product {{PRODUCT_NAME}} to cart" - "Set quantity to {{QUANTITY}}" - "Apply discount code {{DISCOUNT_CODE}}" Comprehensive ReportingHTML Reports: Beautiful, interactive test reportsJSON/XML Output: For CI/CD integrationScreenshot Capture: Automatic failure documentationPerformance Metrics: Execution timing and statisticsThe Future is ReadableWe're moving toward a world where:Tests are documentation that executesAnyone can contribute to test automationMaintenance is a joy, not a choreEnvironments are just configurationYAML-based Playwright testing isn't just a tool — it's a philosophy. It's the belief that tests should be clear, maintainable, and accessible to everyone on the team.Common Questions AnsweredQ: How does this compare to existing solutions like Cucumber? A: While Cucumber requires learning Gherkin syntax and step definitions, this YAML testing framework uses natural language directly with Claude Code's AI interpreting the intent. No step definition mapping needed - Claude Code understands what you want to do. Q: What about test debugging? A: Claude Code provides detailed execution logs, Playwright MCP captures screenshots on failure, and you get clear error messages that map back to your YAML steps. The AI context helps identify issues quickly. Q: Can I integrate this with CI/CD? A: Absolutely. The framework generates standard exit codes and multiple report formats (HTML, JSON, XML) for seamless CI/CD integration. Q: How do you handle complex assertions? A: Claude Code's AI makes natural language assertions surprisingly powerful: "Verify page contains 'Thank you'", "Verify cart total equals $43.18", "Verify 2 items in cart". The AI understands context and intent.Take Action TodayThe question isn't whether this approach is better. The question is: How much time are you willing to waste on brittle, complex tests? Start your YAML testing journey:Get Claude Code: Install Claude Code and Playwright MCPTry the demo: Clone the project from https://github.com/terryso/claude-code-playwright-mcp-test and run your first YAML testConvert one test: Take your most complex Playwright test and rewrite it in YAMLShare with your team: Show them how readable tests can beScale gradually: Convert more tests as you see the benefitsReady to transform your testing workflow with Claude Code and Playwright MCP? The future of test automation is readable, maintainable, and accessible to everyone. 🔗 Get Started: https://github.com/terryso/claude-code-playwright-mcp-test What's your biggest pain point with current Playwright tests? How would YAML-based testing with Claude Code solve it for your team?

## Publication Information

- [AI Dev Journal](https://paragraph.com/@terryso/): Publication homepage
- [All Posts](https://paragraph.com/@terryso/): More posts from this publication
- [RSS Feed](https://api.paragraph.com/blogs/rss/@terryso): Subscribe to updates

## Optional

- [Collect as NFT](https://paragraph.com/@terryso/stop-writing-brittle-playwright-tests): Support the author by collecting this post
- [View Collectors](https://paragraph.com/@terryso/stop-writing-brittle-playwright-tests/collectors): See who has collected this post