Open API Schema 3.0 Contract Testing

🙋 Help
1

Hi,
I am using the contract testing functionality in the API section. I have create a collection to test and I have imported the OpenAPI definition but when I click on validate I always receive Validation OK, even if i use a mock to force an invalid json response that does not match with the schema definition.

image
image1608Ă—521 22.8 KB

Do you have any idea?

thanks a lot

2

Hi @Moon, I’m also having this issue and wondering if you solved it. Seems like the contract testing tests some parts but not the validity of the response body. I can copy and past the schema into the javascript tests and validate it with tv4 just fine, but that sort of diminishes the value here.

The documentation seems to indicate support for this feature: https://learning.postman.com/docs/postman/design-and-develop-apis/validating-elements-against-schema/

Hard to believe such an obvious feature would be missing.

3

There is a whole lot of validation going on so it can be confusing to follow each step. This validation is making sure the collection you generated from your OpenAPI still in alignment. This validation isn’t making sure the response of each API is valid, it is for making sure OpenAPI contract is met, not the individual API schema contract. Does that make sense? We have a lot of work to do in defining the right words to use for this process. Try changing a path name in your collection, and you should see it not validate.

4

That make sense.
Thanks a lot for your help!

1 Like
5

Hi @Moon

I have created an extension on the OpenApi-to-Postman library, that would allow you to generate your Postman collection including several Postman tests .

The goal of the Portman package is to facilitate contract testing via Postman, based on your based on your OpenAPI specification.

Currently the following Postman tests are generated automatically:

  • Success status check
  • Response time check
  • Content-type check
  • JSON schema validation
  • JSON body check

Perhaps this will solve your request with regards to the contract testing.
Feel free to give feedback.

2 Likes
6

Just out of curiosity, why doesn’t it validate the response schema is valid as well?

I’ve been using AJV for response schema validation, but it seems like it’s essentially just re-writing (with some differences) what’s in the OpenAPI definition.

1 Like
7

That with some differences part might be more difficult than you think, @robmcec :slight_smile:

I’ve pointed you at this before, but I’ll reference it again because you seem to be doing a lot of manual work, but I created a Postman collection that will automatically generate contract tests off of your open api spec and validate response codes and response schema. You are more then welcome to use it as-is or piece together what i did for the automatic response validation.

1 Like
8

Hi Allen, thanks for the link.

If memory serves, the api spec needs to have certain requirements for the generator to work.

I’m not the spec designer and the current one is still in the process of developing the basic endpoints.

If I have some time, I’ll create a new collection, copy our current spec and try the test generator.

Those schema’s are not something I’ll want to be manually editing if we implement contract testing. That’s for sure.

Thanks!

1 Like
9

@robmcec The test suite generation PR New Feature - Generate tests from OpenAPI / Swagger - automated contract testing #56 by thim81 · Pull Request #225 · postmanlabs/openapi-to-postman · GitHub will re-use the OpenAPI schema to generate a “pm.response.to.have.jsonSchema” test.

// Test whether the response matches the schema
pm.test("[PUT] /lists/v2/audiences/{audienceId}/contacts/{contactId} - Schema is valid", function() {
   pm.response.to.have.jsonSchema(schema,{unknownFormats: ["int32", "int64"]});
});
1 Like