# Stop Writing Brittle Playwright Tests *Why YAML-Based Testing is the Future* By [AI Dev Journal](https://paragraph.com/@terryso) · 2025-06-16 claudecode, playwright, uitesting, testing, mcp --- _How a simple YAML configuration built for Claude Code and Playwright MCP transformed our testing workflow and made automation accessible to everyone on the team_ * * * If you've ever maintained a large Playwright test suite, you know the pain. Hundreds of lines of JavaScript scattered across dozens of files. Hardcoded values that break when environments change. Test logic so complex that only the original author dares to modify it. What if I told you there's a better way? A way to write tests that are **readable by anyone**, **maintainable by design**, and **powerful enough** to handle complex workflows? Enter **YAML-based Playwright testing for Claude Code** — a paradigm shift that's changing how teams approach test automation by leveraging the power of Claude Code's AI capabilities and Playwright MCP's browser automation. The Problem with Traditional Playwright Tests --------------------------------------------- Let's be honest about traditional Playwright tests: // Traditional Playwright test - 50+ lines of code test('complete order flow', async ({ page }) => { await page.goto('https://example.com'); await page.fill('[data-testid="username"]', 'user123'); await page.fill('[data-testid="password"]', 'pass456'); await page.click('[data-testid="login-btn"]'); await expect(page.locator('h1')).toContainText('Dashboard'); // ... 40+ more lines of clicking, filling, asserting // ... hardcoded values everywhere // ... no reusability between tests }); **Problems:** * ❌ **Verbose and complex** — Simple actions buried in boilerplate * ❌ **Hardcoded values** — Environment changes break everything * ❌ **Poor reusability** — Copy-paste leads to maintenance nightmares * ❌ **Technical barrier** — Only developers can write/modify tests * ❌ **Scattered logic** — Related tests live in different files The YAML Revolution: Tests That Make Sense ------------------------------------------ Now imagine the same test written in YAML: # test-cases/order.yml tags: - smoke - order - checkout steps: - include: "login" - "Click Add to Cart button for first product" - "Click Add to Cart button for second product" - "Click shopping cart icon in top right" - "Enter First Name" - "Enter Last Name" - "Enter Postal Code" - "Click Continue button" - "Click Finish button" - "Verify page displays Thank you for your order!" - include: "cleanup" **Immediate benefits:** * ✅ **Crystal clear intent** — Anyone can understand what this test does * ✅ **Natural language** — Steps read like user stories * ✅ **Reusable components** — Login and cleanup are shared across tests * ✅ **Environment agnostic** — No hardcoded values in sight The Magic Behind the Simplicity ------------------------------- ### 1\. **Reusable Step Libraries** Common workflows become building blocks: # steps/login.yml steps: - "Open {{BASE_URL}} page" - "Fill username field with {{TEST_USERNAME}}" - "Fill password field with {{TEST_PASSWORD}}" - "Click login button" - "Verify page displays Swag Labs" Write once, use everywhere. No more copy-paste madness. ### 2\. **Environment Variable Magic** Different environments? No problem: # .env.dev BASE_URL=https://dev.example.com TEST_USERNAME=dev_user # .env.prod BASE_URL=https://example.com TEST_USERNAME=prod_user Same tests, different environments. Automatically. ### 3\. **Intelligent Tag Filtering** Run exactly what you need: # Run only smoke tests /run-yaml-test tags:smoke # Run order AND checkout tests /run-yaml-test tags:order,checkout # Run smoke OR critical tests /run-yaml-test tags:smoke|critical No more running the entire suite when you only changed the login flow. ### 4\. **Smart Reporting** Automatically generated HTML reports with: * ✅ Step-by-step execution details * ✅ Environment configuration * ✅ Screenshots and artifacts * ✅ Success/failure statistics Real-World Impact: A Case Study ------------------------------- **Before YAML testing:** * 📊 **2,000+ lines** of Playwright JavaScript * ⏱ **3 days** to onboard new QA team members * 🐛 **15+ test failures** per environment change * 👥 **Only 3 developers** could modify tests **After YAML testing:** * 📊 **200 lines** of readable YAML * ⏱ **30 minutes** to onboard new team members * 🐛 **0 test failures** during environment changes * 👥 **Entire team** can write and modify tests Why This Matters for Your Team ------------------------------ ### **For Developers:** * Less time writing boilerplate, more time building features * Tests that actually document your application's behavior * No more "let me just quickly fix this test" rabbit holes ### **For QA Engineers:** * Focus on test strategy, not JavaScript syntax * Rapid test creation and modification * Clear visibility into test coverage ### **For Product Managers:** * Tests that read like acceptance criteria * Easy to verify that tests match requirements * Confidence that important flows are covered ### **For DevOps:** * Predictable test execution across environments * Clear failure reporting and debugging * Easy integration with CI/CD pipelines Technical Architecture: How It Works ------------------------------------ This YAML Playwright testing framework is specifically designed for **Claude Code** and **Playwright MCP**. The framework consists of several key components: ### **Claude Code Integration** * **AI-Powered Execution**: Claude Code's AI interprets natural language test steps and converts them to Playwright actions * **Smart Step Recognition**: Advanced understanding of testing intent from plain English descriptions * **Context Awareness**: Maintains context across test steps for more intelligent automation ### **Playwright MCP Foundation** * **Browser Automation**: Leverages Playwright MCP for reliable cross-browser testing * **Element Detection**: Intelligent element finding and interaction * **Screenshot & Reporting**: Built-in capture and documentation capabilities ### **Multi-Environment Configuration** ├── .env.dev # Development environment ├── .env.test # Test environment ├── .env.prod # Production environment ### **Reusable Step Libraries** ├── steps/ │ ├── login.yml # Authentication flows │ ├── cleanup.yml # Cleanup procedures │ └── navigation.yml # Common navigation ### **Test Cases with Natural Language** ├── test-cases/ │ ├── order.yml # E-commerce order flow │ ├── user.yml # User management │ └── search.yml # Search functionality ### **Intelligent Execution Engine** The framework automatically: 1. Loads environment-specific configuration 2. Expands `include` references from step libraries 3. Substitutes environment variables (`{{BASE_URL}}`) 4. Executes tests using Playwright MCP 5. Generates comprehensive reports Getting Started: Your First YAML Test ------------------------------------- The beauty of YAML-based testing is its simplicity. Here's how to get started: ### **1\. Prerequisites** # Install Claude Code (if not already installed) # Follow instructions at: https://claude.ai/code # Install Playwright MCP for Claude Code claude mcp add playwright -- npx -y @playwright/mcp@latest # Clone the YAML testing framework git clone https://github.com/terryso/claude-code-playwright-mcp-test.git cd claude-code-playwright-mcp-test ### **2\. Project Structure** your-project/ ├── .env.dev # Environment config ├── steps/ # Reusable step libraries ├── test-cases/ # Your test cases ├── screenshots/ # Test artifacts └── reports/ # Generated reports ### **3\. Write Your First Test** # test-cases/login.yml tags: - smoke - auth steps: - "Open {{BASE_URL}} page" - "Fill username with {{TEST_USERNAME}}" - "Fill password with {{TEST_PASSWORD}}" - "Click login button" - "Verify successful login" ### **4\. Execute and Iterate** # In Claude Code, use the built-in commands /run-yaml-test file:test-cases/login.yml env:dev # Or run with tag filtering /run-yaml-test tags:smoke env:dev Within hours, you'll have tests that are more maintainable than anything you've written before. The magic happens through Claude Code's AI understanding your natural language steps and Playwright MCP executing them as browser actions. Advanced Features ----------------- ### **Complex Tag Filtering** # Multiple conditions /run-yaml-test tags:smoke,login|critical # Environment-specific execution /run-yaml-test tags:order env:prod ### **Dynamic Step Parameters** steps: - "Add product {{PRODUCT_NAME}} to cart" - "Set quantity to {{QUANTITY}}" - "Apply discount code {{DISCOUNT_CODE}}" ### **Comprehensive Reporting** * **HTML Reports**: Beautiful, interactive test reports * **JSON/XML Output**: For CI/CD integration * **Screenshot Capture**: Automatic failure documentation * **Performance Metrics**: Execution timing and statistics The Future is Readable ---------------------- We're moving toward a world where: * Tests are **documentation** that executes * **Anyone** can contribute to test automation * **Maintenance** is a joy, not a chore * **Environments** are just configuration YAML-based Playwright testing isn't just a tool — it's a philosophy. It's the belief that tests should be **clear**, **maintainable**, and **accessible** to everyone on the team. Common Questions Answered ------------------------- **Q: How does this compare to existing solutions like Cucumber?** A: While Cucumber requires learning Gherkin syntax and step definitions, this YAML testing framework uses natural language directly with Claude Code's AI interpreting the intent. No step definition mapping needed - Claude Code understands what you want to do. **Q: What about test debugging?** A: Claude Code provides detailed execution logs, Playwright MCP captures screenshots on failure, and you get clear error messages that map back to your YAML steps. The AI context helps identify issues quickly. **Q: Can I integrate this with CI/CD?** A: Absolutely. The framework generates standard exit codes and multiple report formats (HTML, JSON, XML) for seamless CI/CD integration. **Q: How do you handle complex assertions?** A: Claude Code's AI makes natural language assertions surprisingly powerful: "Verify page contains 'Thank you'", "Verify cart total equals $43.18", "Verify 2 items in cart". The AI understands context and intent. Take Action Today ----------------- The question isn't whether this approach is better. The question is: **How much time are you willing to waste on brittle, complex tests?** Start your YAML testing journey: 1. **Get Claude Code**: Install Claude Code and Playwright MCP 2. **Try the demo**: Clone the project from https://github.com/terryso/claude-code-playwright-mcp-test and run your first YAML test 3. **Convert one test**: Take your most complex Playwright test and rewrite it in YAML 4. **Share with your team**: Show them how readable tests can be 5. **Scale gradually**: Convert more tests as you see the benefits * * * _Ready to transform your testing workflow with Claude Code and Playwright MCP? The future of test automation is readable, maintainable, and accessible to everyone._ 🔗 **Get Started:** https://github.com/terryso/claude-code-playwright-mcp-test **What's your biggest pain point with current Playwright tests? How would YAML-based testing with Claude Code solve it for your team?** * * * --- *Originally published on [AI Dev Journal](https://paragraph.com/@terryso/stop-writing-brittle-playwright-tests)*