How to properly run contract testing?

Contract testing can be initiated in various ways depending on the architecture and chosen tools.

Main approaches:

• During Build Time: Tests are run as part of the CI/CD pipeline after the service is built. This helps detect integration errors before deployment.

• After Deployment: Tests are run after deploying the service to a test or staging environment. This is more realistic but may lead to discovering issues later.

• Scheduled: Tests are run regularly (e.g., hourly or daily) to check the stability of contracts between services.

• On Demand: Tests are run manually when needed, such as before a release or after significant changes.

Tools and approaches:

• Pact: A popular framework for Consumer-Driven Contract Testing.

  • The consumer writes tests that define the expected data format.
  • Contracts (Pact files) are generated.
  • The provider (API service) verifies these contracts, ensuring it meets the consumer's expectations.
  • Run: typically as part of both consumer's and provider's build pipeline.

• Swagger/OpenAPI: API specifications can be used to generate and run tests.

  • Tools like Swagger Codegen or Stoplight can generate test code based on the specification.
  • Tests verify compliance with the specification.
  • Run: can be integrated into CI/CD.

• Custom frameworks: Creating bespoke solutions for contract testing using standard client libraries and testing frameworks (e.g., JUnit, TestNG, Pytest).

  • Tests explicitly call APIs and check the structure and content of responses.
  • Run: flexibly configured in CI/CD.

Recommendations for proper execution:

1. Integration into CI/CD: The most effective way is to run contract tests within the CI/CD pipeline. This ensures automatic detection of issues.
2. Separation by test type: Divide tests into "Contract Tests" in your test suite.
3. Test environment: Ideally, have a dedicated test environment for contract testing that ensures dependency stability.
4. Reports: Generate detailed reports on the results of contract tests.
5. Notifications: Set up alerts for test failures.

Example of CI integration (Pact with Maven):

<!-- pom.xml -->
<build>
    <plugins>
        <plugin>
            <groupId>au.com.dius</groupId>
            <artifactId>pact-maven-plugin</artifactId>
            <version>4.6.0</version> // Use the latest version
            <configuration>
                <pactBrokerUrl>${pact.broker.url}</pactBrokerUrl>
                <projectVersion>${project.version}</projectVersion>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>publish</goal> // Publish pact files
                        <goal>verify</goal>   // Verify contracts
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Example in GitLab CI (.gitlab-ci.yml):

# .gitlab-ci.yml
image: maven:latest # Or another image

stages:
  - build
  - test
  - deploy

build:
  stage: build
  script:
    - mvn clean package

contract_test:
  stage: test
  script:
    - mvn verify
  artifacts:
    paths:
      - target/pact/reports/

deploy:
  stage: deploy
  script:
    - echo "Deploying..."
  when: manual # Example of manual deployment after tests