Hi,
I am using the contract testing functionality in the API section. I have create a collection to test and I have imported the OpenAPI definition but when I click on validate I always receive Validation OK, even if i use a mock to force an invalid json response that does not match with the schema definition.
Hi @Moon, I’m also having this issue and wondering if you solved it. Seems like the contract testing tests some parts but not the validity of the response body. I can copy and past the schema into the javascript tests and validate it with tv4 just fine, but that sort of diminishes the value here.
There is a whole lot of validation going on so it can be confusing to follow each step. This validation is making sure the collection you generated from your OpenAPI still in alignment. This validation isn’t making sure the response of each API is valid, it is for making sure OpenAPI contract is met, not the individual API schema contract. Does that make sense? We have a lot of work to do in defining the right words to use for this process. Try changing a path name in your collection, and you should see it not validate.
I have created an extension on the OpenApi-to-Postman library, that would allow you to generate your Postman collection including several Postman tests .
The goal of the Portman package is to facilitate contract testing via Postman, based on your based on your OpenAPI specification.
Currently the following Postman tests are generated automatically:
Success status check
Response time check
Content-type check
JSON schema validation
JSON body check
Perhaps this will solve your request with regards to the contract testing.
Feel free to give feedback.
Just out of curiosity, why doesn’t it validate the response schema is valid as well?
I’ve been using AJV for response schema validation, but it seems like it’s essentially just re-writing (with some differences) what’s in the OpenAPI definition.
That with some differences part might be more difficult than you think, @robmcec
I’ve pointed you at this before, but I’ll reference it again because you seem to be doing a lot of manual work, but I created a Postman collection that will automatically generate contract tests off of your open api spec and validate response codes and response schema. You are more then welcome to use it as-is or piece together what i did for the automatic response validation.
// Test whether the response matches the schema
pm.test("[PUT] /lists/v2/audiences/{audienceId}/contacts/{contactId} - Schema is valid", function() {
pm.response.to.have.jsonSchema(schema,{unknownFormats: ["int32", "int64"]});
});