What is the API Testing Coverage?

When discussing code quality, test coverage typically refers to the percentage of the codebase exercised by automated tests. For APIs, the concept is similar but focuses on the API’s contract and functionality rather than the underlying code. Calculating API test coverage can be done at various levels of complexity.

Basic Endpoint Coverage

The simplest method is to measure endpoint coverage. This involves identifying which API endpoints or resources are being tested. This is a binary metric: either an endpoint is included in a test suite or it is not. While this method is easy to implement and provides a quick way to identify untested paths, it has significant limitations because it doesn’t account for different HTTP methods (GET, POST, PUT, etc.) or varying request and response schemas.

Comprehensive API Coverage

A more robust approach involves a deeper analysis of the API’s contract, often defined by a specification like OpenAPI. This method assesses coverage across multiple dimensions, including:

• Operation Coverage: Verifying that every HTTP method for each endpoint is tested.

• Request Parameter Coverage: Testing all required and optional parameters, including different combinations of values.

• Response Status Code Coverage: Ensuring that tests validate both successful and error responses (e.g., 200 OK, 400 Bad Request, 500 Internal Server Error).

• Schema Coverage: Testing that the data types, formats, and constraints for both requests and responses, as defined in the schema, are properly handled. This is where concepts like oneOf, allOf, and anyOf become crucial, as they require dedicated test cases to ensure all possible valid schemas are tested.

Calculating this comprehensive coverage is more complex, but it provides a more accurate measure of the test suite’s effectiveness and reduces the risk of undetected bugs in the API’s public interface. Specialized tools can automate this process by analyzing the OpenAPI specification and comparing it against executed test cases.