Skip to content

Comprehensive Testing Framework & Test Coverage

Overview

Establish a comprehensive testing framework that ensures all MCP tools work correctly and reliably for Claude agents. This includes unit tests, integration tests, end-to-end tests, and performance tests that validate the entire system.

Claude Agent Success Criteria

A Claude agent should be able to:

  • ✅ Run all tests and get clear pass/fail results
  • ✅ Understand test failures with actionable error messages
  • ✅ Add new tests when implementing features
  • ✅ Validate system reliability through comprehensive test coverage

Detailed Acceptance Criteria

1. Test Framework Setup

Configure comprehensive testing infrastructure:

{
  "scripts": {
    "test": "vitest",
    "test:run": "vitest run",
    "test:coverage": "vitest run --coverage",
    "test:coverage:ci": "vitest run --coverage --reporter=junit --reporter=json --reporter=verbose",
    "test:unit": "vitest run tests/unit",
    "test:integration": "vitest run tests/integration", 
    "test:e2e": "vitest run tests/e2e",
    "test:performance": "vitest run tests/performance",
    "coverage:check": "vitest run --coverage && node scripts/check-coverage.js",
    "coverage:report": "vitest run --coverage && open coverage/index.html"
  }
}

Test Structure:

  • tests/unit/ - Unit tests for individual functions and components

  • tests/integration/ - Integration tests with real databases and CLI

  • tests/e2e/ - End-to-end workflow tests

  • tests/performance/ - Performance and load tests

  • tests/fixtures/ - Test data, configurations, and mock databases

  • Configure Vitest with TypeScript support

  • Set up test coverage reporting (≥80% required)

  • Create test database setup and teardown utilities

  • Implement test fixtures and mock data

  • Add performance testing capabilities

2. Unit Test Suite

Comprehensive unit tests for all core functionality:

Core Utilities Tests:

// tests/unit/graphile-migrate/executor.test.ts
describe('GraphileMigrateExecutor', () => {
  test('executes commands correctly', async () => {
    const executor = new GraphileMigrateExecutor();
    const result = await executor.execute('status', [], { workingDirectory: testDir });
    expect(result.success).toBe(true);
  });

  test('handles command failures gracefully', async () => {
    const executor = new GraphileMigrateExecutor();
    const result = await executor.execute('invalid-command', [], { workingDirectory: testDir });
    expect(result.success).toBe(false);
    expect(result.stderr).toContain('Unknown command');
  });
});

MCP Tools Tests:

// tests/unit/tools/migration-status.test.ts
describe('get_migration_status tool', () => {
  test('returns comprehensive status for valid project', async () => {
    const result = await getMigrationStatus({ projectPath: testProjectPath });
    expect(result.success).toBe(true);
    expect(result.data.project.configValid).toBe(true);
    expect(result.data.committed).toBeArrayOf(expect.objectContaining({
      filename: expect.any(String),
      path: expect.any(String)
    }));
  });
});
  • Test all MCP tools with valid inputs
  • Test error handling and edge cases
  • Test validation logic and pattern detection
  • Test file operations and I/O
  • Test utility functions and helpers

3. Integration Test Suite

Integration tests with real databases and systems:

Database Integration Tests:

// tests/integration/database.test.ts
describe('Database Integration', () => {
  let testDb: TestDatabase;

  beforeEach(async () => {
    testDb = await createTestDatabase();
  });

  afterEach(async () => {
    await testDb.cleanup();
  });

  test('schema analysis returns accurate results', async () => {
    await testDb.executeSql('CREATE TABLE users (id SERIAL PRIMARY KEY)');
    const analysis = await analyzeSchema({ projectPath: testDb.projectPath });
    expect(analysis.data.tables).toHaveLength(1);
    expect(analysis.data.tables[0].name).toBe('users');
  });
});

CLI Integration Tests:

// tests/integration/cli.test.ts
describe('Graphile Migrate CLI Integration', () => {
  test('executes real migration commands', async () => {
    const project = await createTestProject();
    const result = await executeCommand('migrate', [], { workingDirectory: project.path });
    expect(result.exitCode).toBe(0);
  });
});
  • Test with real PostgreSQL databases
  • Test actual Graphile Migrate CLI integration
  • Test file system operations
  • Test network operations and timeouts
  • Test multi-database scenarios

4. End-to-End Test Suite

Complete workflow tests simulating real usage:

Developer Workflow Tests:

// tests/e2e/developer-workflow.test.ts
describe('Complete Developer Workflow', () => {
  test('full migration development cycle', async () => {
    const project = await setupTestProject();
    
    // 1. Create new migration
    const created = await createMigration({
      projectPath: project.path,
      migrationName: 'add_users_table',
      template: 'table'
    });
    expect(created.success).toBe(true);
    
    // 2. Validate migration
    const validation = await analyzeMigrationSafety({
      projectPath: project.path
    });
    expect(validation.data.idempotency.score).toBeGreaterThan(80);
    
    // 3. Test idempotency
    const idempotency = await testMigrationIdempotency({
      projectPath: project.path,
      iterations: 3
    });
    expect(idempotency.data.test.passed).toBe(true);
    
    // 4. Commit migration
    const commit = await commitMigration({
      projectPath: project.path,
      message: 'Add users table'
    });
    expect(commit.success).toBe(true);
  });
});

Shadow Database Workflow:

// tests/e2e/shadow-database.test.ts
describe('Shadow Database Workflow', () => {
  test('complete shadow testing cycle', async () => {
    const project = await setupTestProject();
    
    // Setup shadow database
    const setup = await setupShadowDatabase({
      projectPath: project.path,
      seedData: true
    });
    expect(setup.success).toBe(true);
    
    // Test migration
    const test = await testAgainstShadow({
      projectPath: project.path
    });
    expect(test.data.test.execution.successful).toBe(true);
    
    // Compare with main
    const comparison = await compareWithShadow({
      projectPath: project.path
    });
    expect(comparison.data.comparison.identical).toBe(false);
    
    // Cleanup
    const cleanup = await cleanupDevelopmentDatabases({
      projectPath: project.path
    });
    expect(cleanup.success).toBe(true);
  });
});
  • Test complete migration development workflows
  • Test error recovery and rollback scenarios
  • Test multi-user development scenarios
  • Test edge cases and boundary conditions
  • Test integration with different PostgreSQL versions

5. Performance Test Suite

Performance and load testing for scalability:

Performance Benchmarks:

// tests/performance/benchmarks.test.ts
describe('Performance Benchmarks', () => {
  test('schema analysis performance', async () => {
    const largeDb = await createLargeTestDatabase(1000); // 1000 tables
    
    const startTime = Date.now();
    const analysis = await analyzeSchema({ projectPath: largeDb.path });
    const duration = Date.now() - startTime;
    
    expect(analysis.success).toBe(true);
    expect(duration).toBeLessThan(5000); // Should complete within 5 seconds
    expect(analysis.data.tables).toHaveLength(1000);
  });

  test('idempotency testing performance', async () => {
    const project = await createTestProject();
    await project.addMigration('large-migration.sql', createLargeMigration());
    
    const startTime = Date.now();
    const test = await testMigrationIdempotency({
      projectPath: project.path,
      iterations: 5
    });
    const duration = Date.now() - startTime;
    
    expect(test.success).toBe(true);
    expect(duration).toBeLessThan(30000); // Should complete within 30 seconds
  });
});

Load Testing:

// tests/performance/load.test.ts
describe('Load Testing', () => {
  test('concurrent MCP tool usage', async () => {
    const projects = await Promise.all(
      Array(10).fill(0).map(() => createTestProject())
    );
    
    const startTime = Date.now();
    const results = await Promise.all(
      projects.map(project => getMigrationStatus({ projectPath: project.path }))
    );
    const duration = Date.now() - startTime;
    
    expect(results.every(r => r.success)).toBe(true);
    expect(duration).toBeLessThan(10000); // Should handle 10 concurrent requests within 10 seconds
  });
});
  • Test performance with large databases
  • Test concurrent tool usage
  • Test memory usage over time
  • Test resource cleanup efficiency
  • Test scalability limits

6. Test Infrastructure Utilities

Create comprehensive test utilities and helpers:

Test Database Manager:

// tests/utils/test-database.ts
export class TestDatabaseManager {
  async createDatabase(name: string): Promise<TestDatabase>;
  async createProject(options: ProjectOptions): Promise<TestProject>;
  async seedDatabase(db: TestDatabase, data: SeedData): Promise<void>;
  async cleanup(): Promise<void>;
}

export interface TestProject {
  path: string;
  name: string;
  database: TestDatabase;
  addMigration(filename: string, content: string): Promise<void>;
  getStatus(): Promise<ProjectStatus>;
  cleanup(): Promise<void>;
}

Mock Data Generators:

// tests/utils/mock-data.ts
export function generateMockMigration(type: 'table' | 'function' | 'index'): string;
export function generateLargeDatabase(tableCount: number): DatabaseSchema;
export function generateTestProject(options: ProjectOptions): ProjectConfiguration;
  • Create database setup/teardown utilities
  • Implement project scaffolding helpers
  • Add mock data generation functions
  • Create test assertion helpers
  • Implement performance measurement utilities

Coverage Requirements

Maintain ≥80% code coverage across all metrics:

  • Lines: 80% minimum coverage
  • Functions: 80% minimum coverage
  • Branches: 80% minimum coverage
  • Statements: 80% minimum coverage

Coverage verification script:

// scripts/check-coverage.js
const coverage = require('./coverage/coverage-summary.json');
const required = 80;

Object.entries(coverage.total).forEach(([metric, data]) => {
  if (data.pct < required) {
    console.error(`${metric} coverage ${data.pct}% below required ${required}%`);
    process.exit(1);
  }
});
console.log('✅ All coverage requirements met');

Claude Agent Guidance

Running Tests

# Run all tests
npm test

# Run specific test suites
npm run test:unit
npm run test:integration
npm run test:e2e

# Run with coverage
npm run test:coverage

# Check coverage requirements
npm run coverage:check

# Generate coverage report
npm run coverage:report

Test Development Workflow

// 1. Write failing test first (TDD)
test('should create migration with template', async () => {
  const result = await createMigration({
    projectPath: '/test/project',
    migrationName: 'add_users',
    template: 'table'
  });
  expect(result.success).toBe(true);
  expect(result.data.file).toContain('CREATE TABLE IF NOT EXISTS');
});

// 2. Implement feature to make test pass
// 3. Refactor while keeping tests green

Expected Test Behavior

  • All tests should pass consistently
  • Test failures should provide clear error messages
  • Performance tests should complete within time limits
  • Coverage should meet minimum requirements

Common Test Issues

  • Database connection failures: Ensure test databases are properly configured
  • File permission issues: Use proper test directories with write permissions
  • Timeout errors: Increase timeout for integration tests with real databases
  • Flaky tests: Use proper setup/teardown and avoid race conditions

Definition of Done

  • Complete test framework configured and working
  • All test suites implemented with comprehensive coverage
  • Performance tests validate scalability requirements
  • Test coverage meets ≥80% requirement across all metrics
  • CI/CD integration tests pass consistently
  • Test documentation complete and clear
  • Error messages provide actionable guidance
  • Test data cleanup works reliably

Estimated Effort

3-4 days

Dependencies

  • All previous issues (this validates the entire system)

Technical Notes for Claude Agents

Key Files to Create

  • tests/unit/**/*.test.ts - Unit test files
  • tests/integration/**/*.test.ts - Integration test files
  • tests/e2e/**/*.test.ts - End-to-end test files
  • tests/performance/**/*.test.ts - Performance test files
  • tests/utils/ - Test utilities and helpers
  • tests/fixtures/ - Test data and configurations
  • scripts/check-coverage.js - Coverage validation script

Test Database Strategy

  • Use Docker containers for consistent test environments
  • Create isolated test databases for each test
  • Implement proper cleanup to avoid test interference
  • Use database transactions for fast test isolation
  • Support multiple PostgreSQL versions

Performance Testing Guidelines

  • Set realistic performance targets based on typical usage
  • Test with various database sizes and complexities
  • Monitor memory usage and resource consumption
  • Use consistent test environments for reliable metrics
  • Implement automated performance regression detection