Skip to content

Add NPX Execution Testing to CI/CD Pipeline

🔄 Enhancement: Add NPX Execution Testing to CI/CD Pipeline

📋 Summary

Following the successful fix of NPX entry point compatibility in MR #6 (closed), we need to add comprehensive NPX execution testing to our CI/CD pipeline to prevent regression of this critical functionality.

🎯 Objectives

  1. Prevent NPX Regressions - Ensure NPX execution continues working in future releases
  2. Comprehensive Coverage - Test all supported installation methods in CI
  3. Early Detection - Catch execution context issues before they reach production
  4. Quality Assurance - Validate documented installation methods work correctly

🛠️ Implementation Tasks

Phase 1: NPX Testing Infrastructure (2-3 hours)

  • Add NPX execution test job to .gitlab-ci.yml
  • Create test script that validates NPX startup sequence
  • Test NPX execution with timeout/process management
  • Validate server initialization and MCP protocol handshake

Phase 2: Multi-Context Testing (2-3 hours)

  • Direct Node Execution: node dist/index.js
  • NPX Local: npx . (local package)
  • NPX Published: npx @haleyio/mcp-servers-gitlab
  • Global Install: npm install -g + binary execution
  • CI Environment: Test in fresh container environment

Phase 3: Integration with Existing Pipeline (1-2 hours)

  • Integrate with existing test stages
  • Add appropriate job dependencies and artifact handling
  • Configure proper environment variables for testing
  • Add cache optimization for NPX operations

Phase 4: Documentation and Maintenance (1 hour)

  • Document NPX testing approach in CLAUDE.md
  • Add troubleshooting guide for NPX test failures
  • Create maintenance checklist for NPX testing updates

🧪 Technical Requirements

Test Script Structure

test-npx-execution:
  stage: test
  script:
    # Build package first
    - npm run build
    
    # Test direct execution
    - timeout 10s node dist/index.js || test $? -eq 124
    
    # Test NPX local execution  
    - timeout 10s npx . || test $? -eq 124
    
    # Test server startup sequence
    - echo '{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"protocolVersion": "2024-11-05", "capabilities": {}, "clientInfo": {"name": "test", "version": "1.0.0"}}}' | timeout 5s node dist/index.js

Validation Criteria

  • Server process starts without errors
  • MCP protocol initialization succeeds
  • Server responds to basic MCP commands
  • Process terminates cleanly
  • No memory leaks or hanging processes

🎯 Success Metrics

  • Coverage: All 4 execution methods tested in CI
  • Reliability: NPX tests pass consistently (>99% success rate)
  • Speed: NPX tests complete within 2-3 minutes
  • Maintenance: Clear failure diagnostics and troubleshooting

🔗 Dependencies

  • Prerequisite: MR #6 (closed) (NPX compatibility fix) must be merged
  • Related: Issue #10 (closed) (Comprehensive Testing Plan)
  • Follows: Current CI/CD pipeline structure

📊 Priority

Medium-High - Important for preventing regressions of critical functionality

🏷️ Labels

enhancement testing ci/cd npx quality-assurance

📝 Acceptance Criteria

  • CI/CD pipeline includes NPX execution testing
  • All supported installation methods are tested
  • Tests prevent regression of NPX functionality
  • Clear failure diagnostics and troubleshooting documentation
  • Tests complete within reasonable time limits
  • Documentation updated with testing approach

🎉 Success Definition

Done when: CI/CD pipeline automatically validates NPX execution on every build, preventing future regressions of the critical NPX compatibility issue resolved in MR #6 (closed).

Context: This issue directly follows the critical NPX compatibility fix and ensures we don't regress this essential functionality in future releases.