Schema validation help

🙋 Help
1

I’m trying to add schema validation to PM.
This is a response:

{

    "data": {

        "statusItem": [

            {

                "ID": 53,

                "cost": 0,

                "discountType": 0,

                "discountValue": 0,

                "flags": 96,

                "flags3": 256,

                "flags4": 0,

                "flags5": 0,

                "futureBreakID": 0,

                "group": 4,

                "highlightColor": 0,

                "inActive": 0,

                "miscItemID": 0,

                "name": "Acct Correction",

                "notificationID": 0,

                "overrideRejectMins": false,

                "POReturnReasonID": 0,

                "priority": 0,

                "rejectStatusMins": 0,

                "seqNum1": 0,

                "seqNum2": 8192,

                "userOrGroupID": 0

            }

Is there a tool that I can use to convert the above response to a schema in which I can use for something like this:

const schema = {
    //valid schema to test against
};

pm.test("Validate status_items"), () => {
    pm.response.to.have.jsonSchema(schema);
}

I’m reading\watching tutorials, but it’s just not clicking. Is there an easy way to do this?

P.S.
This passes in postman. Does anyone know how?

const scshema = {
    "type": "objedfasgct"
};

pm.test("Validate status_items"), () => {
    pm.response.to.have.jsonSchema(schsdfddfema);
}
2

I tend to use jsonschema to help create the schema.

It has a number of different options that will create a number of different schema structures - depending on your needs. :grin:

It’s also tricky creating the schemas, it’s a whole new syntax and way of structuring the object that easy to get wrong and give false positives.

I think that both @allenheltondev and @vdespa have knowledge in this area and might be able to provide more assistance. :grin:

3

Yes. The false positives are making me think of not even attempting this.

4

I use jsonschema as well and validate with ajv - which coincidentally is included in the postman code sandbox!

I’m not sure what that assertion is doing, but loading the schema in ajv and validating against a json object is pretty powerful.

If you’re defining your API in an OpenAPI Spec, then you’re already halfway there to getting what you’re looking for.

5

We are defining our API in OpenAPI Spec in PM.

What’s the other half of what I need to do?

6

Are you defining it in yaml or json?

In theory, you can just take the schema definition defined in the schema section and add that to ajv.

I have a dynamic test generator collection I made to test scenarios exactly like this. If you’re familiar with javascript, you should be able to follow along what it’s doing:

1 Like
7

json

Ok. I’ll have to check that out.

Thanks!

8

@robmcec

pm.test takes two parameters:

  1. A string representing the name of the test
  2. A callback function containing the assertions.

You are only providing the first parameter and closing the function. Does that make sense?

9

I used jsonschema to convert a response body into a schema to test against.

let Ajv = require('ajv'),
ajv = new Ajv({logger: console}),
schema = {
    "$schema": "http://json-schema.org/draft-07/schema",
    "$id": "http://example.com/example.json",
    "type": "object",
    "title": "The root schema",
    "description": "The root schema comprises the entire JSON document.",
    "default": {},
    "examples": [
        {
            "data": {
                "statusItem": [
                    {
                        "ID": 12,
                        "cost": 0,
                        "discountType": 0,
                        "discountValue": 0,
                        "flags": 0,
                        "flags3": 0,
                        "flags4": 0,
                        "flags5": 0,
                        "futureBreakID": 0,
                        "group": 1,
                        "highlightColor": 65535,
                        "inActive": 0,
                        "miscItemID": 0,
                        "name": "Surface",
                        "notificationID": 0,
                        "overrideRejectMins": false,
                        "POReturnReasonID": 0,
                        "priority": 0,
                        "rejectStatusMins": 5,
                        "seqNum1": 0,
                        "seqNum2": 0,
                        "userOrGroupID": 0
                    }
                ]
            }
        }
    ],
    "required": [
        "data"
    ],
    "properties": {
        "data": {
            "$id": "#/properties/data",
            "type": "object",
            "title": "The data schema",
            "description": "An explanation about the purpose of this instance.",
            "default": {},
            "examples": [
                {
                    "statusItem": [
                        {
                            "ID": 12,
                            "cost": 0,
                            "discountType": 0,
                            "discountValue": 0,
                            "flags": 0,
                            "flags3": 0,
                            "flags4": 0,
                            "flags5": 0,
                            "futureBreakID": 0,
                            "group": 1,
                            "highlightColor": 65535,
                            "inActive": 0,
                            "miscItemID": 0,
                            "name": "Surface",
                            "notificationID": 0,
                            "overrideRejectMins": false,
                            "POReturnReasonID": 0,
                            "priority": 0,
                            "rejectStatusMins": 5,
                            "seqNum1": 0,
                            "seqNum2": 0,
                            "userOrGroupID": 0
                        }
                    ]
                }
            ],
            "required": [
                "statusItem"
            ],
            "properties": {
                "statusItem": {
                    "$id": "#/properties/data/properties/statusItem",
                    "type": "array",
                    "title": "The statusItem schema",
                    "description": "An explanation about the purpose of this instance.",
                    "default": [],
                    "examples": [
                        [
                            {
                                "ID": 12,
                                "cost": 0,
                                "discountType": 0,
                                "discountValue": 0,
                                "flags": 0,
                                "flags3": 0,
                                "flags4": 0,
                                "flags5": 0,
                                "futureBreakID": 0,
                                "group": 1,
                                "highlightColor": 65535,
                                "inActive": 0,
                                "miscItemID": 0,
                                "name": "Surface",
                                "notificationID": 0,
                                "overrideRejectMins": false,
                                "POReturnReasonID": 0,
                                "priority": 0,
                                "rejectStatusMins": 5,
                                "seqNum1": 0,
                                "seqNum2": 0,
                                "userOrGroupID": 0
                            }
                        ]
                    ],
                    "additionalItems": true,
                    "items": {
                        "$id": "#/properties/data/properties/statusItem/items",
                        "anyOf": [
                            {
                                "$id": "#/properties/data/properties/statusItem/items/anyOf/0",
                                "type": "object",
                                "title": "The first anyOf schema",
                                "description": "An explanation about the purpose of this instance.",
                                "default": {},
                                "examples": [
                                    {
                                        "ID": 12,
                                        "cost": 0,
                                        "discountType": 0,
                                        "discountValue": 0,
                                        "flags": 0,
                                        "flags3": 0,
                                        "flags4": 0,
                                        "flags5": 0,
                                        "futureBreakID": 0,
                                        "group": 1,
                                        "highlightColor": 65535,
                                        "inActive": 0,
                                        "miscItemID": 0,
                                        "name": "Surface",
                                        "notificationID": 0,
                                        "overrideRejectMins": false,
                                        "POReturnReasonID": 0,
                                        "priority": 0,
                                        "rejectStatusMins": 5,
                                        "seqNum1": 0,
                                        "seqNum2": 0,
                                        "userOrGroupID": 0
                                    }
                                ],
                                "required": [
                                    "ID",
                                    "cost",
                                    "discountType",
                                    "discountValue",
                                    "flags",
                                    "flags3",
                                    "flags4",
                                    "flags5",
                                    "futureBreakID",
                                    "group",
                                    "highlightColor",
                                    "inActive",
                                    "miscItemID",
                                    "name",
                                    "notificationID",
                                    "overrideRejectMins",
                                    "POReturnReasonID",
                                    "priority",
                                    "rejectStatusMins",
                                    "seqNum1",
                                    "seqNum2",
                                    "userOrGroupID"
                                ],
                                "properties": {
                                    "ID": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/ID",
                                        "type": "integer",
                                        "title": "The ID schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            12
                                        ]
                                    },
                                    "cost": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/cost",
                                        "type": "integer",
                                        "title": "The cost schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "discountType": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/discountType",
                                        "type": "integer",
                                        "title": "The discountType schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "discountValue": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/discountValue",
                                        "type": "integer",
                                        "title": "The discountValue schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "flags": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/flags",
                                        "type": "integer",
                                        "title": "The flags schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "flags3": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/flags3",
                                        "type": "integer",
                                        "title": "The flags3 schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "flags4": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/flags4",
                                        "type": "integer",
                                        "title": "The flags4 schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "flags5": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/flags5",
                                        "type": "integer",
                                        "title": "The flags5 schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "futureBreakID": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/futureBreakID",
                                        "type": "integer",
                                        "title": "The futureBreakID schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "group": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/group",
                                        "type": "integer",
                                        "title": "The group schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            1
                                        ]
                                    },
                                    "highlightColor": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/highlightColor",
                                        "type": "integer",
                                        "title": "The highlightColor schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            65535
                                        ]
                                    },
                                    "inActive": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/inActive",
                                        "type": "integer",
                                        "title": "The inActive schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "miscItemID": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/miscItemID",
                                        "type": "integer",
                                        "title": "The miscItemID schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "name": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/name",
                                        "type": "string",
                                        "title": "The name schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": "",
                                        "examples": [
                                            "Surface"
                                        ]
                                    },
                                    "notificationID": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/notificationID",
                                        "type": "integer",
                                        "title": "The notificationID schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "overrideRejectMins": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/overrideRejectMins",
                                        "type": "boolean",
                                        "title": "The overrideRejectMins schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": false,
                                        "examples": [
                                            false
                                        ]
                                    },
                                    "POReturnReasonID": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/POReturnReasonID",
                                        "type": "integer",
                                        "title": "The POReturnReasonID schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "priority": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/priority",
                                        "type": "integer",
                                        "title": "The priority schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "rejectStatusMins": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/rejectStatusMins",
                                        "type": "integer",
                                        "title": "The rejectStatusMins schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            5
                                        ]
                                    },
                                    "seqNum1": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/seqNum1",
                                        "type": "integer",
                                        "title": "The seqNum1 schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "seqNum2": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/seqNum2",
                                        "type": "integer",
                                        "title": "The seqNum2 schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "userOrGroupID": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/userOrGroupID",
                                        "type": "integer",
                                        "title": "The userOrGroupID schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    }
                                },
                                "additionalProperties": true
                            }
                        ]
                    }
                }
            },
            "additionalProperties": true
        }
    },
    "additionalProperties": true
}

Next, I used the following test:

pm.test('Schema is valid', function() {
    var error = pm.response.json()['errors'];
    pm.expect(ajv.validate(schema, {errors: error})).to.be.true;
});

It failed with this very sad error:
Schema is valid | AssertionError: expected false to be true

This error doesn’t provide nearly enough information to troubleshoot.
Should I be using different parameters for the function?

BTW, I got this method from

10

You should be able to just have this:

let schema = {
    "type": "object",
    "required": [
        "data"
    ],
    "properties": {
        "data": {
            "type": "object",
            "required": [
                "statusItem"
            ],
            "properties": {
                "statusItem": {
                    "type": "array",
                    "items": {
                        "anyOf": [
                            {
                                "type": "object",
                                "required": [
                                    "ID",
                                    "cost",
                                    "discountType",
                                    "discountValue",
                                    "flags",
                                    "flags3",
                                    "flags4",
                                    "flags5",
                                    "futureBreakID",
                                    "group",
                                    "highlightColor",
                                    "inActive",
                                    "miscItemID",
                                    "name",
                                    "notificationID",
                                    "overrideRejectMins",
                                    "POReturnReasonID",
                                    "priority",
                                    "rejectStatusMins",
                                    "seqNum1",
                                    "seqNum2",
                                    "userOrGroupID"
                                ],
                                "properties": {
                                    "ID": {
                                        "type": "integer"
                                    },
                                    "cost": {
                                        "type": "integer"
                                    },
                                    "discountType": {
                                        "type": "integer"
                                    },
                                    "discountValue": {
                                        "type": "integer"
                                    },
                                    "flags": {
                                        "type": "integer"
                                    },
                                    "flags3": {
                                        "type": "integer"
                                    },
                                    "flags4": {
                                        "type": "integer"
                                    },
                                    "flags5": {
                                        "type": "integer"
                                    },
                                    "futureBreakID": {
                                        "type": "integer"
                                    },
                                    "group": {
                                        "type": "integer"
                                    },
                                    "highlightColor": {
                                        "type": "integer"
                                    },
                                    "inActive": {
                                        "type": "integer"
                                    },
                                    "miscItemID": {
                                        "type": "integer"
                                    },
                                    "name": {
                                        "type": "string"
                                    },
                                    "notificationID": {
                                        "type": "integer"
                                    },
                                    "overrideRejectMins": {
                                        "type": "boolean"
                                    },
                                    "POReturnReasonID": {
                                        "type": "integer"
                                    },
                                    "priority": {
                                        "type": "integer"
                                    },
                                    "rejectStatusMins": {
                                        "type": "integer"
                                    },
                                    "seqNum1": {
                                        "type": "integer"
                                    },
                                    "seqNum2": {
                                        "type": "integer"
                                    },
                                    "userOrGroupID": {
                                        "type": "integer"
                                    }
                                }
                            }
                        ]
                    }
                }
            }
        }
    }
}

pm.test('Schema is valid', function() {
    pm.response.to.have.jsonSchema(schema)
});

I’m not really sure what you’re trying to do in that test :smiley:

11

Holy cr*p that works as expected! Sorry about my profanities.

I can even get it to fail how I’d expect.

How did you turn my incorrect thing into the correct thing?

It looks like you took mine and just chopped out everything except “type”.

Did you use some other schema generator, or did you just manually change it?

Also, we’re using a defined Open API Spec in the API section. After looking at this is the schema check (as a test script) even necessary?

12

It really depends on the implementation of your API.

The open api spec in Postman is just a definition. In my opinion it’s always worth the effort it takes to run a quick ajv.validate on a response to make sure the developers did what they were supposed to do.

13

So do I need to add the schema definition in the test itself,
Like this:

let schema = {
    "type": "object",
    "required": [
        "data"
    ],
    "properties": {
        "data": {
//and so on

or can I just tell adj to validate against the defined schema somehow?

14

If the schema has been structured correctly, this would validate that against the response body.

pm.test('Schema is valid', function() {
    pm.response.to.have.jsonSchema(schema)
});
15

Sorry, by schema, you mean the schema that lives with the test case, not the schema in the API -> Define section?

I guess the question I have is "Is there a way to use the schema in the API -> Define tab in test cases instead of having to create that schema monstrosity and test against it?

It seems like we’d be doing the same thing twice if we did that.

16

This schema:

let schema = {
    "type": "object",
    "required": [
        "data"
    ],
    "properties": {
        "data": {
//and so on

You could change the name to avoid confusion if you like :smiley:

This might be what you’re talking about for validation between an API Spec and a Collection?

https://learning.postman.com/docs/designing-and-developing-your-api/validating-elements-against-schema/

17

I use the following syntax so that all validation errors are reported back, where ‘schema’ is what you would have defined for the desired validation. I feel using ‘allErrors: true’ is key:

var Ajv = require('ajv'),
    ajv = new Ajv({ logger: console, allErrors: true }); 

tests["Schema is valid"] = ajv.validate(schema, pm.response.json());
if (ajv.errors !== null) {
    tests["Schema validation error(s): " + JSON.stringify(ajv.errors) + ";"] = false;
}
1 Like
18

Ok, thank you. I’ll take a look at that.

I think what confuses me about schema testing is that we have an open api definition for the collection.

Won’t the response results get validated against the open api schema? If our response is bad, we should see a notification that says we have “issues” above the endpoint URL.

This seems like doing the same thing twice. Once when PM validates the response and once when avj validates against the schema provided to it in the test script.

var schema = {schema stuff}

Is this correct, or am I missing something?

If we have an open api definition, do we still need to do the schema validation in the test script?

Thanks!

19

If you have created your API (Validation is available for OpenAPI 3.0 schemas.) in Postman and generated Collections from that or linked the existing collections to that - It would validate those requests in the same way and give you a summary of the issues found.

https://learning.postman.com/docs/designing-and-developing-your-api/validating-elements-against-schema/#validating-elements

Is this what you have set up for your API now? Are you able to validate the Collections and Request against the API spec?

20

Yes, we have the schema defined and linked to the collection that I’m testing.

It will be versioned and updated using the schema validator in PM.

Since we do that, we don’t really need to use Avj correct?