The Evolution of Testing in the Age of Serverless

Introduction

As organisations adopt microservices for rebuilds and new applications, many will realise that the established ‘Test Pyramid’ approach to testing is considerably less effective than it was with traditional applications.

The Test Pyramid emphasises unit tests that focus on the application’s individual functions and code. These comprise the bulk of this testing strategy, reflected in their position in the larger bottom pyramid section. Unit tests are followed by integration tests that validate the interactions between two or more connected components. While contract testing is not officially included in this strategy, we have added it in the same section as Integration testing for awareness purposes. We will address contract testing later in this article. At the top, and the smallest part of the pyramid, are the end-to-end tests, also called E2E, which test the output of application workflows. As we move up the pyramid, the cost and effort required to develop and run each type of test increases. This is one of the reasons unit tests are preferred, as lower effort and cost typically translates to better developer productivity.

However, when we switch to a microservices architecture, we find that unit tests do not provide sufficient application coverage. Compared to traditional applications, microservice applications tend to have less code and more workflows of many connected consumers (services that initiate a request for external services) and providers. Consumers are services that initiate a request for external services. For example, a checkout service that calls an external payment service to process payments. Providers are services that receive and process a request. In our previous example, the payment service will receive the request, process the payment, update the database and then return the payment status back to the consumer.

As such, we need to prioritise integration tests over unit tests. End-to-end or E2E tests are of similar importance to unit tests in this architecture. These preferences resulted in the development of the ‘Test Honeycomb’. In this design, the integration tests in the middle are now the largest section, and the smaller unit and E2E sections are equal in size. As with the pyramid testing strategy, contract testing is not officially mentioned, but we have again added it in the integration testing section for awareness purposes.

Challenges with Integration and E2E Tests

While Integration and E2E tests can provide better microservice testing coverage, they come with their own challenges, such as dependence on external resources and state.

When a component changes the structure of its request or response payload – the messages it exchanges with other components – it is difficult to determine the change’s impact on any components that depend on it. One mitigation is to run integration tests on all connections to identify any failures resulting from the change. Depending on the architecture, this can potentially require a significant number of tests to be executed.

For example, if we have ten components interacting with one another, we could potentially end up with 90 integration points to test – if connectivity is two-way. Realistically, it will be closer to 15 integrations in most well-architected microservice designs. Still, maintaining a list of integrations and ensuring compatibility across dependencies can be challenging as the application grows.

To add, all components, including any services they depend on, and their connections must be updated and available for the tests to pass. A common approach to microservice development is for multiple developers or small teams to work on multiple microservices in parallel. This approach further complicates the challenge of needing all components to be available for testing. Besides synchronising updates across teams, significant coordination is needed to plan deployments so the required tests can be performed before the changes go to production.

To address these challenges, we need a solution that can eliminate the dependency on external services for tests to be conducted, and at the same time, still enable developers to confidently test their changes before deploying them to production.

The Pact

Pact is a popular open-sourced tool for running Consumer-Driven Contract Tests. Pact provides a suite of libraries for consumers and providers to build their test cases and a broker for maintaining the collection of contracts.

With Pact contract testing, the focus is on what – the request or response payload, that is sent between consumers and providers, and not the how – such as specific messaging services or formats. Pact uses REST APIs to receive and send contracts between the consumers and providers.

The value of Pact lies in it providing a platform that providers and consumers trust and, facilitating the agreement and verification of contracts between them. It aims to solve a people challenge – collaborating effectively during integration tests, rather than any particular technical challenge.

Consumers publish a contract that defines the expected response required from the provider. As a provider, it may have multiple consumers using it for various purposes. Let’s take an example of a customer making an order on an e-commerce website. When the order is submitted, several consumer microservices, such as payment, customer loyalty and warehouse inventory, would be interested to know about it. Each consumer will have its own set of values they would like to receive from the provider, which is defined in a contract. The role of the Pact broker is to collect all these contracts that were created by the consumers to send to the provider. The provider then uses these contracts to verify that the response it returns matches what is expected by the consumers.

With this system in mind, consumers and providers can work at their own cadence while confident that new updates will not break existing integrations within the application. The consumer developer can create and test new features without waiting for the provider, and the provider developer can prioritise the list of features, updates and fixes for the service. Again, keep in mind that this flexibility only applies within a project milestone – all service updates will still need to be deployed into production for the application to work.

Challenges

Pact grew out of a container microservices environment where a microservice-to-microservice communication pattern is more commonly used. However, such a pattern is considered an anti-pattern for Serverless microservices. In Serverless, messages from a provider are passed to a managed message router such as Amazon SQS, SNS or EventBridge, which forwards them to one or more consumers. This raises questions about where the Pact broker should sit in this architecture and which request/response payloads we should test.

Besides message routers, Serverless generally prefers using purpose-built managed services such as API Gateway and DynamoDB over developing custom code. There is limited value in writing contract tests for managed services since they have stable APIs, which we are unlikely able to influence even if the test fails. The information about the availability and implementation of these services is usually provided in their respective documentation.

Serverless Contract Testing

Despite the challenges mentioned, we still can find value with Contract Testing by focusing on the content of these messages instead of how and to whom the messages are being sent.

Let’s walk through a simple example of how we can implement Contract Testing using Pact to get a better idea of how to implement Contract Testing for Serverless. Do note that we are not doing a deep dive into Pact for this article, instead we are exploring the concepts of using Contract Testing for Serverless projects.

Step 1: Consumer defines the contract

As mentioned earlier, in a Consumer-Driven Contract Test, the consumer is responsible for determining the contract between itself and the provider.

In this example, the consumer defines a test case that specifies the following when the receiveTransactionUpdate() method is executed on the consumer:

1. Test case (“successful transaction event”) that should be triggered on the provider’s end
2. Content that the provider should verify (transaction_id, customer_id and status)

describe("transaction complete", () => { 
  it("accepts a successful transaction", () => { 
    const pact = messagePact 
      .expectsToReceive("successful transaction event") 
      .withContent({ 
        transaction_id: like("1111-2222-3333-4444"), 
        customer_id: like("9999-8888-7777-6666"), 
        status: "SUCCESS", 
      }) 
      .verify(asynchronousBodyHandler(receiveTransactionUpdate)); 
 
    return pact 
  }); 
});

Step 2: Consumer publishes the contract

Once the developer has completed the test cases, the next step is to publish the contract to the broker. First, the Pact library will compile a list of all the test cases to create a contract, like the JSON file we see below.

Once the contract is created, the developer can add other attributes such as tags to define the feature branch/environment and the version information before uploading it to the broker.

"messages": [ 
  { 
    "description": "successful transaction event", 
    "contents": { 
      "transaction_id": "1111-2222-3333-4444", 
      "customer_id": "9999-8888-7777-6666", 
      "status": "SUCCESS" 
    }, 
    ... 
  } 
]

Step 3: Provider retrieves and verifies the contract

Whenever a consumer publishes or updates a contract, a Webhook can be configured to run automated builds or notify the provider’s developer.

To ensure that new consumer contracts do not break any existing builds, the provider can filter the tests to run based on the defined tags in the contracts. For example, we can configure the Pact library to only run tests against all consumer contracts tagged with “team_a_tests” to limit the scope of the tests.

The provider initialises the Verifier class in the Pact Library to run the test. It will automatically retrieve all the contracts associated with the provider class and run all the tests.

Here, the provider retrieves the “successful transaction event” test case published by the consumer in the contract. The provider then writes the implementation of the code and test it against the test case.

Here are two possible reasons why a contract could fail:

1. There was a code change in the provider class that breaks existing consumers
2. A consumer changed its expected payload, resulting in a newly-generated contract

To resolve the failed test cases, the provider will either have to modify their code or reach out to the respective consumer to re-align the requirements, such as requesting justifications for a change in contract.

const opts = { 
  ...baseOpts, 
  ...(process.env.PACT_URL ? pactChangedOpts : fetchPactsDynamicallyOpts), 
  messageProviders: { 
    "successful transaction event": () => { 
      response = createEvent(new Transaction("1111-2222-3333-4444",  
                                             "9999-8888-7777-6666",  

       "SUCCESS")) 
      return response 
    } 
  }, 
}; 
 
 
const p = new MessageProviderPact(opts); 
 
describe("successful transaction", () => { 
  it("successful transaction event", () => { 
    results = p.verify(); 
 

    return results 
  }); 
});

Step 4: Provider publishes the contract

Once the provider completes the contract, the results are published to the broker. The results of the tests can be accessed via the broker’s web interface.

Alternatively, the provider or consumer can check if their changes are ready to be deployed by running the can-i-deploy command found in the Pact library to determine if all dependencies and dependents have been successfully validated.

> pact-broker can-i-deploy \
              --pacticipant MyConsumerService \
              --broker-base-url http://localhost:8000 \
              --broker-username pact_workshop \
              --broker-password pact_workshop \
              --version 000000001
Computer says yes \o/

Conclusion

We want to emphasise that the conventional approach to testing with a strong emphasis on Unit Tests is not the best option for Serverless. Using a Microservice architecture greatly increases the number of integrations while reducing the amount of code developers need to test to ensure sufficient coverage.

Integration and E2E tests are preferred for this architecture, but they come with challenges such as all components needing to be available for testing, which can be logistically challenging when multiple teams are working on a single application. Contract testing can help address these challenges. Providers don’t need to maintain a list of consumers, and they can react to breaking changes before the code is deployed. This helps shorten development time, improve the quality of tests and increase developer productivity.

Additional Resources

Thomas Smart

Thomas is a Lead Consultant at Sourced with over 18 years of experience in delivering digital products. He gained a passion for evangelising Serverless architecture through his enthusiasm for cloud and entry into the AWS Ambassador programme.

KangZheng Li

KangZheng is an up-and-coming AWS Cloud Engineer with two years of professional experience in architecting and implementing cloud solutions. He enjoys researching the latest technology trends and applying them to client projects.