Measure your collection's quality with the Postman API and Spectral rules

Ask the Experts and Postman Tips
, ,
1

Collections are an excellent tool for collaborating with others in the API design, development, testing, and documentation processes. You can share your collection with others, who can then help in the development process by providing their feedback in comments.

To help your API consumers, you should always define the following items in a collection:

  • Documentation. Adding descriptions to the collection and its requests helps consumers understand the collection’s purpose and how your API works. At the collection’s top level, provide details for how they can get started. Describing requests in your collection helps consumers to understand how it works and what to expect from it.
  • Examples. Providing examples of different responses (such as success, unauthenticated errors, and bad requests) for each request helps consumers understand the expected responses and adapt their code.
  • Variables. Defining generic requests with global, environment, or collection variables lets consumers easily test your API with different environments or conditions by simply changing the variables’ values.
  • Authentication. Defining authentication at the global or request level makes it much easier for consumers to test your API.
  • Tests. Providing tests in your collection can help consumers get started testing your API’s behavior.

When a collection lacks these items, it becomes difficult for consumers to get any value from it or use it. If you intend to share your collection, whether internally or publicly, it’s good practice to properly define these items to improve the quality of your collections.

In this post, we’ll show you how to create a tool for your automation that evaluates the quality of a collection using the Postman API, Spectral CLI, and jq. You can find the code for this tutorial in our postman-collection-spectral-linter GitHub repository.

Best practices for defining the quality rules for a Postman Collection

The postman-collection-spectral-linter tool uses Spectral rules to detect any quality issues in a Postman Collection. Spectral rules are a method for defining the conditions that a JSON or YAML file must meet. The Spectral CLI tool processes a given JSON or YAML file and checks if that file meets all the rules in this file.

For example, we’ll define a collection-description-is-mandatory rule that checks the collection.info property in the collection’s JSON for a description property:

rules:
  collection-description-is-mandatory:
    description: Collection must have a description.
    given: $.collection.info
    severity: error
    then:
      field: description
      function: truthy

Measuring quality

Based on our experience as Postman users and our experience working with Postman users around the world, we’ve noticed several key elements that a collection must or should include to be considered high-quality. We’ve defined these features in the tool’s rules.yaml file:

Errors

  • All requests must have a description.
  • Requests must have a URL.
  • All requests have at least a 2XX or 3XX example.
  • All responses must have a status code.
  • DELETE responses must have a 200 or 204 status code.
  • Requests with path variables require a 404 response.
  • All request query parameters must have a description.
  • All request path parameters must have a description.
  • All PATCH requests require a request body.
  • All responses must have a status code.
  • All authenticated requests require a 401 response.

Warnings

  • HTTP should not be used.
  • Response bodies should be in JSON.
  • All folders should have a description.
  • Request names should not end in “Copy”.

This ruleset isn’t exhaustive, and it may not necessarily meet your use case. You can add your own customized rules to fit your needs, extend any of the default rules, or remove rules you don’t need in the rules.yaml file.

How it works

When you run the tool’s shell script (Bash) command, it:

  • Retrieves the collection in JSON format with the Postman API’s GET /collections/{collectionId} endpoint.
  • Uses Spectral CLI to validate a set of defined rules against the collection.
  • Checks whether the Spectral results contain errors and returns them the script’s exit status. If any of the rules fail, then the command fails.

The tool uses the following script:

# Perform a curl request against the Postman API. Save the request body in a file called collection.json
curl -s -H "x-api-key: $API_KEY" https://api.postman.com/collections/$COLLECTION_UID | jq . > _collection.json

# Lint the collection using spectral. Save the result in a JSON file
spectral lint _collection.json --ruleset $RULES_PATH -f json --quiet > _result.json

# search for errors (severity=0) in the result file. If there are errors, exit with a non-zero status code
jq '.[] | select(.severity==0)' -e _result.json >/dev/null

if [ $? -eq 0 ]
  then
    exit 1
fi

Note that:

  • In line 2, the tool uses the Postman API to retrieve the collection in JSON format. It then uses jq to beautify the output and saves it as a temporary _collection.json file.
  • In line 5, Spectral CLI checks the rules in the rules.yaml file, then dumps the results to a temporary _results.json file.
  • In line 8, jq performs a search for errors with a 0 (error) severity.
  • In line 10, the tool checks if the previous command succeeded with an execution status of 0. In that case, we return an error execution status of 1. You can use this status code in scripts to evaluate if the collection contains any errors. Additionally, you can use the _result.json file contents with the error details to perform any kind of notification.

Using the tool

After you’ve cloned the repo, run the shell script with the following command:

./linter.sh -c <collection_uid> [-k <postman_api_key>] [-r path_to_ruleset_file]

In this command, the postman_api_key parameter defaults to a POSTMAN_API_KEY environment variable and path_to_ruleset_file defaults to the rules.yaml ruleset. Feel free to extend it by adding additional rules or turn off any rules you don’t need.

Summary

The Postman API provides full CRUD capabilities that let you manage your collections. You can easily retrieve the collection JSON with the Postman API and a valid API key. To continuously validate whether your collection follows your quality standards, you can define your Spectral rules and check those rules automatically with the Spectral CLI. It’s important to integrate quality checks in your CI/CD workflows to improve the overall quality of your collections.

Next steps

  • Define your own rules, adapted to the quality level you want in your collections. You can extend the best practices rules we provided in this blog post.
  • Use the tool to automatically notify users when they don’t meet the quality standards to help your team maintain great collections.
  • Use the detailed results of the Spectral CLI to send notifications or add comments to a collection with the Postman API.

Do you have any use cases related to collection quality? We would love to hear about it—please let us know in a comment!

4 Likes