# Test Runner Migration: Karma → Web Test Runner

## Table of Contents
1. [Overview](#overview)
2. [Understanding JavaScript Testing Environments](#understanding-javascript-testing-environments)
3. [Current State Analysis](#current-state-analysis)
4. [Migration Strategy](#migration-strategy)
5. [Implementation Plan](#implementation-plan)
6. [Technical Details](#technical-details)
7. [Success Metrics](#success-metrics)
8. [Rollback Plan](#rollback-plan)

## Overview

This document outlines the migration from the deprecated Karma test runner to the modern Web Test Runner (@web/test-runner) for the Rollbar.js SDK. This migration is part of the broader modernization effort to enable TypeScript support and improve developer experience.

### Goals
- Replace deprecated Karma with actively maintained Web Test Runner
- **Eliminate Grunt dependency** - run tests directly via npm scripts
- Improve test execution speed by 50-70%
- Enable native TypeScript support for future migration
- Simplify test configuration and maintenance
- Maintain 100% test compatibility

### Approach
- **Strategy**: Incremental migration with parallel validation
- **Risk Level**: Low (keeping existing setup as fallback)

## Understanding JavaScript Testing Environments

### The Two Worlds of JavaScript

JavaScript runs in two fundamentally different environments, and understanding this distinction is crucial for our testing strategy:

#### 1. Browser Environment
- **What it has**: DOM, window object, document, localStorage, fetch API
- **What it lacks**: Filesystem access, process control, native modules
- **Security**: Sandboxed for safety (can't read your hard drive!)
- **Examples**: Chrome, Firefox, Safari, Edge

#### 2. Node.js Environment
- **What it has**: Filesystem (`fs`), process object, Buffer, native modules
- **What it lacks**: DOM, window, browser-specific APIs
- **Security**: Full system access (can read/write files)
- **Examples**: Your terminal, web servers, build tools

### Why This Matters for Rollbar.js

The Rollbar SDK must work in BOTH environments:

```javascript
// Browser usage
<script src="rollbar.min.js"></script>
<script>
  Rollbar.init({ accessToken: 'abc123' });
</script>

// Node.js usage
import Rollbar from 'rollbar';
const rollbar = new Rollbar({ accessToken: 'abc123' });
```

This dual-environment requirement means we need different test strategies:
- **Browser tests**: Need a real browser to test DOM manipulation, window.onerror, etc.
- **Server tests**: Need Node.js to test filesystem operations, process handling, etc.

### Test Framework vs Test Runner

**Test Framework (Mocha)**
- Provides the structure for writing tests
- Defines `describe()`, `it()`, `beforeEach()`, etc.
- Framework-agnostic (works in any JavaScript environment)

```javascript
describe('Calculator', () => {
  it('should add numbers', () => {
    expect(1 + 1).to.equal(2);
  });
});
```

**Test Runner**
- Executes the tests in the appropriate environment
- For browsers: Launches browser, serves files, collects results
- For Node.js: Just runs the JavaScript file directly

### Our Testing Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                      Rollbar.js Tests                       │
├─────────────────────────────┬───────────────────────────────┤
│      Browser Tests (~478)   │      Server Tests (~126)      │
├─────────────────────────────┼───────────────────────────────┤
│   Environment: Browser      │   Environment: Node.js        │
│   Runner: Web Test Runner   │   Runner: Node.js (direct)    │
│   Browser: Playwright       │   Browser: N/A                │
│   Framework: Mocha          │   Framework: Mocha            │
│   Assertions: Chai/Expect   │   Assertions: Chai            │
└─────────────────────────────┴───────────────────────────────┘
```

## Current State Analysis

### Test Infrastructure Metrics
- **Total Tests**: ~604 (478 browser + 126 server)
- **Test Runner**: Karma 6.4.4 (deprecated, last update 2023)
- **Execution Time**: ~27 seconds (23s browser + 4s server)
- **Configuration Complexity**: 91 lines Karma + 273 lines Grunt

### Current Grunt Dependency
The tests currently run through Grunt tasks:
- `grunt test` → runs `grunt test-browser`
- `grunt test-browser` → orchestrates multiple Karma runs
- `grunt test-replay` → runs replay-specific tests
- `grunt karma:api`, `grunt karma:browser.core`, etc.

**We will eliminate this entire layer**, running Web Test Runner directly via npm scripts.

### Current Pain Points

1. **Maintenance Burden**
   - Karma is deprecated with security vulnerabilities
   - Complex configuration spread across multiple files
   - Difficult debugging experience

2. **Developer Experience**
   - Poor error messages from Karma
   - No stack traces on test failures (obfuscated/minified code)
   - No way to manually test directly in browsers
   - No native ES module support
   - Complex watch mode setup

3. **Performance Issues**
   - Sequential test suite execution
   - Webpack rebundling for each suite (500ms overhead × 13 suites)
   - Bundle sizes up to 4.82MB for single test suites

### Test Categories and Distribution

| Category       | Test Count | Current Runner | Bundle Size |
| -------------- | ---------- | -------------- | ----------- |
| API            | 5          | Karma          | 691 KB      |
| API Utility    | 14         | Karma          | 425 KB      |
| Browser Core   | 16         | Karma          | 2.31 MB     |
| DOM Utility    | 15         | Karma          | 73.4 KB     |
| Predicates     | 4          | Karma          | 369 KB      |
| Replay Tests   | 50         | Karma          | 2.43 MB     |
| Browser Tests  | 67         | Karma          | 4.82 MB     |
| Transforms     | 42         | Karma          | 1.74 MB     |
| Utility        | 37         | Karma          | 395 KB      |
| Tracing        | 28         | Karma          | 444 KB      |
| Examples       | 163        | Karma          | Various     |
| Server Tests   | 126        | Mocha          | N/A         |

## Migration Strategy

### Architectural Decision: Two Test Runners

After careful consideration, we'll maintain two separate test runners:

1. **Web Test Runner** for browser tests (using **Playwright** for browser automation)
2. **Direct Mocha execution** for server tests

**Rationale:**
- Clear separation of concerns
- Optimal performance for each environment
- Simpler debugging and maintenance
- Aligns with JavaScript ecosystem best practices

**Browser Automation: Playwright**
We're using Playwright as our browser launcher because:
- Cross-browser support (Chromium, Firefox, WebKit/Safari)
- Actively maintained by Microsoft
- Better performance than legacy WebDriver solutions
- Native support in Web Test Runner
- Future-proof for advanced testing scenarios

### Phased Approach

We'll migrate in 5 phases to minimize risk and ensure thorough validation:

#### Phase 1: Foundation & Proof of Concept
- Install and configure Web Test Runner
- Migrate simplest test suite (API - 5 tests)
- Validate core functionality
- Document learnings

#### Phase 2: Core Test Migration
- Migrate utility and transform tests
- Set up HTML fixture loading
- Configure proxy routes
- Establish patterns for common scenarios

#### Phase 3: Complex Test Migration
- Migrate browser-specific tests
- Handle replay and tracing tests
- Address edge cases
- Performance optimization

#### Phase 4: CI/CD Integration
- Update GitHub Actions workflows
- Parallel test execution
- Coverage reporting
- Remove Karma dependencies

#### Phase 5: Documentation & Cleanup
- Update developer documentation
- Remove old configurations
- Team training
- Final validation

## Implementation Plan

### Phase 1: Foundation Setup

#### 1.1 Install Dependencies
```bash
npm install --save-dev @web/test-runner @web/test-runner-mocha @web/test-runner-playwright
```

Note: `@web/test-runner-playwright` provides the Playwright integration for browser automation.

#### 1.2 Create Basic Configuration
Create `web-test-runner.config.mjs`:
```javascript
import { playwrightLauncher } from '@web/test-runner-playwright';

export default {
  files: 'test/**/*.test.js',
  nodeResolve: true,
  
  browsers: [
    playwrightLauncher({ product: 'chromium' }),
  ],

  testFramework: {
    config: {
      ui: 'bdd',
      timeout: 5000,
    },
  },
};
```

#### 1.3 Migrate First Test Suite
Start with the API tests (smallest suite):
1. Copy existing test file
2. Update imports for ES modules
3. Run with Web Test Runner
4. Compare results with Karma output

### Phase 2: Progressive Migration

#### 2.1 Migration Order
Based on complexity and dependencies:

1. **Simple Tests**
   - api (5 tests)
   - apiUtility (14 tests)
   - predicates (4 tests)

2. **Utility Tests**
   - utility (37 tests)
   - transforms (42 tests)
   - domUtility (15 tests)

3. **Core Tests**
   - browser.core (16 tests)
   - browser.init (23 tests)
   - browser.rollbar (67 tests)

4. **Feature Tests**
   - replay.* (50 tests)
   - tracing.* (28 tests)

5. **Example Tests**
   - All example tests (163 tests)

#### 2.2 Configuration Enhancements
Add support for:
- HTML fixtures
- Proxy routes for `/dist/` and `/examples/`
- Custom CSP headers
- jQuery and other global dependencies

### Phase 3: Advanced Features

#### 3.1 Parallel Execution
Configure concurrent test execution:
```javascript
export default {
  concurrency: 4,
  concurrentBrowsers: 2,
  // ... rest of config
};
```

#### 3.2 Coverage Reporting
Add code coverage:
```javascript
import { esbuildPlugin } from '@web/dev-server-esbuild';

export default {
  coverage: true,
  coverageConfig: {
    include: ['src/**/*.js'],
    threshold: {
      statements: 90,
      branches: 85,
      functions: 85,
      lines: 90,
    },
  },
  plugins: [esbuildPlugin({ js: true })],
};
```

### Phase 4: CI/CD Updates

#### 4.1 GitHub Actions Workflow
Update `.github/workflows/test.yml`:
```yaml
- name: Run browser tests
  run: npm run test:browser
  
- name: Run server tests
  run: npm run test:server
```

#### 4.2 NPM Scripts (Direct Execution - No Grunt!)
Update package.json:
```json
{
  "scripts": {
    "test": "npm run test:browser && npm run test:server",
    "test:browser": "web-test-runner",
    "test:server": "mocha 'test/server.*.test.mjs'",
    "test:watch": "web-test-runner --watch",
    "test:replay": "web-test-runner 'test/**/replay/**/*.test.js'",
    "test:unit": "web-test-runner 'test/**/*.test.js' --exclude '**/integration/**'"
  }
}
```

**Command Comparison:**
| Current (via Grunt)               | New (Direct)                  | Benefit                |
| --------------------------------- | ----------------------------- | ---------------------- |
| `npm test` → `grunt test` → Karma | `npm test` → Web Test Runner  | Remove 1 layer         |
| `grunt test-replay`               | `npm run test:replay`         | Direct execution       |
| `grunt karma:api`                 | Covered by `npm test:browser` | Unified approach       |
| 273 lines of Grunt config         | 0 lines                       | Massive simplification |

## Technical Details

### Handling Special Requirements

#### HTML Fixtures
```javascript
// Current (Karma)
__html__['test/fixtures/frame.html']

// New (Web Test Runner)
import { fixture, html } from '@open-wc/testing-helpers';
const el = await fixture(html`<div>content</div>`);
```

#### Proxy Routes
```javascript
export default {
  middleware: [
    // Serve dist files
    function serveDistFiles(context, next) {
      if (context.path.startsWith('/dist/')) {
        context.path = context.path.replace('/dist/', '/');
      }
      return next();
    },
  ],
};
```

#### Global Dependencies
```javascript
// For tests requiring jQuery
export default {
  testRunnerHtml: testFramework => `
    <html>
      <body>
        <script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
        <script type="module" src="${testFramework}"></script>
      </body>
    </html>
  `,
};
```

### Performance Optimizations

1. **Parallel Execution**: Run up to 4 test files concurrently
2. **Shared Dependencies**: Load common libraries once
3. **No Bundling**: Use native ES modules
4. **Smart Watch Mode**: Only re-run affected tests

### Expected Performance Gains

| Metric      | Current (Karma) | Expected (WTR) | Improvement |
| ----------- | --------------- | -------------- | ----------- |
| Total Time  | 23 seconds      | 8-10 seconds   | 57-65%      |
| Startup     | 3 seconds       | 0.5 seconds    | 83%         |
| Per Test    | 48ms            | 17ms           | 65%         |
| Bundle Size | 4.82MB max      | 0 (native)     | 100%        |

## Success Metrics

### Quantitative Metrics
- ✅ All 478 browser tests passing
- ✅ Test execution time < 10 seconds
- ✅ Code coverage maintained at > 90%
- ✅ Zero test flakiness
- ✅ CI pipeline success rate 100%

### Qualitative Metrics
- ✅ Improved developer experience
- ✅ Easier debugging workflow
- ✅ Simplified configuration
- ✅ Better error messages
- ✅ TypeScript-ready infrastructure

## Rollback Plan

If issues arise during migration:

1. **Immediate**: Tests continue running on Karma (parallel setup)
2. **Short-term**: Revert Web Test Runner changes
3. **Investigation**: Debug and fix issues
4. **Retry**: Attempt migration with fixes

### Risk Mitigation
- Keep Karma configuration until fully validated
- Run both test runners in CI during migration
- Incremental migration allows partial rollback
- Each phase independently validated

## Next Steps

1. **Review**: Team reviews this plan
2. **Approval**: Get stakeholder sign-off
3. **Setup**: Create feature branch `feature/web-test-runner`
4. **Execute**: Begin Phase 1 implementation
5. **Validate**: Run parallel validation until confident
6. **Complete**: Remove Karma after full validation

## Appendix: Learning Resources

### For JavaScript Testing Beginners
- [MDN: Introduction to automated testing](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Automated_testing)
- [Modern Web: Web Test Runner docs](https://modern-web.dev/docs/test-runner/overview/)
- [Mocha documentation](https://mochajs.org/)

### Understanding the Migration
- Why Karma is deprecated: Maintenance burden, architectural limitations
- Why Web Test Runner: Modern architecture, native ES modules, active development
- Why keep Mocha: Minimal change, widely understood, works everywhere

### Key Concepts
- **Test Framework**: Provides test structure (describe/it)
- **Test Runner**: Executes tests in target environment
- **Assertion Library**: Provides expectation syntax (expect/assert)
- **Test Environment**: Where tests execute (browser/Node.js)