Skip to content

Add comprehensive integration tests for complete template generation and validation

Problem Statement

Currently, our integration tests only cover individual components (git hooks, basic template structure) but lack comprehensive end-to-end testing of the complete template generation process. This creates risk that:

  1. Generated projects may not build or run correctly
  2. Template files may be missing or incorrectly processed
  3. Placeholder substitution may fail in some files
  4. New template features may break existing functionality

Required Integration Tests

1. Complete Template Generation Test (test:complete-generation)

Test Requirements:

  • Use actual createServer() function with mock inquirer prompts
  • Generate a complete project in temporary directory
  • Verify ALL template files are copied (not just a subset)
  • Validate placeholder substitution works across ALL files
  • Test with various project configurations (different names, emails, licenses)

Validation Criteria:

// Template files that MUST be present and correctly processed
const requiredFiles = [
  'package.json',          // Must have substituted package name, author, etc.
  'tsconfig.json',         // Must be valid JSON
  'src/index.ts',         // Must have substituted SERVICE_NAME placeholders
  'src/server.ts',        // Must have substituted SERVER_NAME placeholders  
  'src/core.ts',          // Must have substituted placeholders
  'src/schemas.ts',       // Must have substituted placeholders
  'src/example-module.ts', // Must have substituted SERVICE_NAME placeholders
  'tests/unit/schemas.test.ts',     // Must have substituted placeholders
  'tests/integration/server.test.ts', // Must have substituted placeholders
  'tests/e2e/workflows.test.ts',      // Must have substituted placeholders
  'vitest.config.ts',     // Must be valid TypeScript
  '.gitlab-ci.yml',       // Must have substituted PACKAGE_NAME
  '.gitignore',           // Must be comprehensive (see below)
  '.githooks/pre-push',   // Must be executable and contain quality gates
  'README.md',            // Must have ALL placeholders substituted
  'CLAUDE.md',            // Must have ALL placeholders substituted  
  'CONTRIBUTING.md',      // Must have substituted repository URLs
  'LICENSE',              // Must have substituted AUTHOR_NAME and YEAR
  '.env.example'          // Must exist for API token guidance
];

Placeholder Validation:

// These placeholders MUST be substituted in generated files
const requiredSubstitutions = {
  '{{SERVER_NAME}}': 'test-server',
  '{{PACKAGE_NAME}}': '@testauthor/test-server', 
  '{{AUTHOR_NAME}}': 'Test Author',
  '{{AUTHOR_EMAIL}}': 'test@example.com',
  '{{SERVICE_NAME}}': 'Test Service',
  '{{REPOSITORY_URL}}': 'https://github.com/test/test-server',
  '{{YEAR}}': new Date().getFullYear().toString(),
  '{{PACKAGE_NAME//\//-}}': '@testauthor-test-server' // Escaped version for CI
};

// NO file should contain unreplaced placeholders after generation
const forbiddenPatterns = [
  /\{\{[A-Z_]+\}\}/g,  // Any unreplaced placeholder
  /\{\{.*?\}\}/g       // Any template syntax
];

2. GitIgnore Validation Test (test:gitignore-completeness)

Test Requirements:

  • Verify .gitignore is copied to generated project
  • Validate .gitignore contains comprehensive patterns
  • Test that .gitignore actually works (create test files that should be ignored)

Validation Criteria:

// .gitignore MUST contain these patterns
const requiredGitIgnorePatterns = [
  // Dependencies
  'node_modules/',
  '*.tgz',
  
  // Build outputs  
  'dist/',
  'build/', 
  '*.tsbuildinfo',
  
  // Environment files
  '.env',
  '.env.local',
  '.env.production', 
  '.env.test',
  
  // Coverage reports
  'coverage/',
  '*.lcov',
  '.nyc_output',
  
  // OS files
  '.DS_Store',
  'Thumbs.db',
  
  // IDE files
  '.vscode/',
  '.idea/',
  '*.swp',
  '*.swo',
  
  // Logs
  '*.log',
  'logs',
  
  // Temporary files
  'tmp/',
  'temp/'
];

// Test actual git ignore functionality
const testIgnoredFiles = [
  'node_modules/fake-package/index.js',
  'dist/compiled.js', 
  '.env',
  'coverage/lcov-report/index.html',
  '.DS_Store',
  'debug.log'
];

3. Generated Project Functionality Test (test:generated-project-works)

Test Requirements:

  • Generate complete project
  • Run npm install successfully
  • Execute all development commands successfully
  • Verify generated project passes its own test suite

Validation Criteria:

# These commands MUST succeed in generated project
npm install                    # Dependencies install without errors
npm run build                 # TypeScript compiles successfully  
npm run test:run              # All tests pass
npm run test:coverage         # Coverage meets thresholds
npm run format:check          # Code formatting is valid
npm run health-check          # MCP server starts and responds

# Generated dist/ MUST contain expected files
dist/index.js                 # Main entry point
dist/server.js               # MCP server implementation
dist/core.js                 # Utilities
dist/schemas.js              # Type definitions

# Package.json scripts MUST be functional
"scripts": {
  "build": "tsc && shx chmod +x dist/*.js",
  "watch": "tsc --watch", 
  "prepare": "npm run build",
  "test": "vitest",
  "test:run": "vitest run",
  // ... all other scripts must work
}

4. Git Hooks Integration Test (test:git-hooks-complete)

Test Requirements:

  • Verify git repository initialization works
  • Test git hooks configuration is applied correctly
  • Validate pre-push hook actually executes quality gates
  • Test hook failure scenarios

Validation Criteria:

// Git configuration MUST be set correctly
const requiredGitConfig = {
  'core.hooksPath': '.githooks'
};

// Pre-push hook MUST be executable and contain quality gates
const prePushRequirements = [
  'npm run format:check',   // Code formatting validation
  'npm run build',         // Build validation  
  'npm run test:run'       // Test validation
];

// Hook MUST prevent push on failures
const hookFailureScenarios = [
  'formatting errors',     // Bad formatting should block push
  'build failures',       // TypeScript errors should block push
  'test failures'         // Failing tests should block push
];

5. Template Consistency Test (test:template-consistency)

Test Requirements:

  • Compare generated files with current template files
  • Ensure no template files are accidentally excluded
  • Verify file permissions are set correctly
  • Test different project configurations produce consistent results

Validation Criteria:

// ALL files in templates/ directory MUST be included in generation
// Exception list for files that shouldn't be copied:
const excludedTemplateFiles = [
  // Add any files that should NOT be copied to generated projects
];

// File permissions MUST be preserved/set correctly
const executableFiles = [
  '.githooks/pre-push'     // Must be chmod 755
];

// Generated projects with same inputs MUST be identical
const deterministicGeneration = true;

Test Implementation Requirements

Test Structure

tests/
├── integration/
│   ├── complete-generation.test.ts     # End-to-end generation test
│   ├── gitignore-validation.test.ts    # .gitignore functionality test  
│   ├── generated-project.test.ts       # Generated project works test
│   ├── git-hooks-complete.test.ts      # Complete git hooks test
│   └── template-consistency.test.ts    # Template consistency test
└── fixtures/
    ├── test-configs.ts                  # Test project configurations
    └── expected-outputs.ts              # Expected file contents

Test Utilities Required

// Helper functions needed for comprehensive testing
interface TestUtilities {
  generateTestProject(config: ProjectConfig): Promise<string>; // Returns temp dir path
  validateAllPlaceholders(projectPath: string, substitutions: Record<string, string>): void;
  runNpmCommands(projectPath: string, commands: string[]): Promise<void>;
  validateGitIgnore(projectPath: string, patterns: string[]): void;
  compareDirectoryStructures(expected: string, actual: string): void;
  cleanupTestProject(projectPath: string): void;
}

Mock Data Required

// Test project configurations to validate different scenarios
const testConfigurations = [
  {
    name: 'basic-server',
    author: 'Test Author', 
    email: 'test@example.com',
    service: 'Test API',
    repository: 'https://github.com/test/basic-server',
    license: 'MIT'
  },
  {
    name: 'complex-mcp-server',
    author: 'Complex Author',
    email: 'complex@company.com', 
    service: 'Enterprise API Service',
    repository: 'https://gitlab.com/company/complex-mcp-server',
    license: 'Apache-2.0'
  }
  // Test edge cases: special characters, long names, different licenses
];

Success Criteria

✅ All new tests pass consistently
✅ Generated projects build and run successfully
✅ 100% of template files are validated
✅ All placeholders are correctly substituted
✅ Git hooks work correctly in generated projects
✅ Generated projects pass their own CI/CD pipelines
✅ Test coverage increases significantly
✅ Integration tests run in reasonable time (< 2 minutes)

Acceptance Testing

Before merging, the following manual validation should be performed:

  1. Generate test project: npm run build && node dist/index.js test-integration-project
  2. Validate project works: cd test-integration-project && npm install && npm test && npm run build
  3. Test git hooks: Make intentional formatting error and try to push (should fail)
  4. Verify MCP server: Test generated server with MCP client
  5. Check all files: Manually spot-check that all expected files exist and look correct

Timeline Expectations

  • Analysis and Planning: 1-2 hours
  • Test Implementation: 4-6 hours
  • Manual Validation: 1 hour
  • Documentation Updates: 30 minutes

Total Estimated Effort: 6-9 hours

Dependencies

  • Existing template files in templates/ directory
  • Current createServer() function in src/generator.ts
  • Vitest testing framework
  • Temporary directory management utilities

Risk Mitigation

  • Slow tests: Use efficient temporary directory cleanup and parallel test execution where possible
  • Flaky tests: Ensure deterministic test data and proper async handling
  • Resource usage: Clean up temporary projects after each test
  • Platform differences: Account for Windows vs Unix file permission differences