Schema validation help

robmcec · November 10, 2020, 9:08pm

I’m trying to add schema validation to PM.
This is a response:

{

    "data": {

        "statusItem": [

            {

                "ID": 53,

                "cost": 0,

                "discountType": 0,

                "discountValue": 0,

                "flags": 96,

                "flags3": 256,

                "flags4": 0,

                "flags5": 0,

                "futureBreakID": 0,

                "group": 4,

                "highlightColor": 0,

                "inActive": 0,

                "miscItemID": 0,

                "name": "Acct Correction",

                "notificationID": 0,

                "overrideRejectMins": false,

                "POReturnReasonID": 0,

                "priority": 0,

                "rejectStatusMins": 0,

                "seqNum1": 0,

                "seqNum2": 8192,

                "userOrGroupID": 0

            }

Is there a tool that I can use to convert the above response to a schema in which I can use for something like this:

const schema = {
    //valid schema to test against
};

pm.test("Validate status_items"), () => {
    pm.response.to.have.jsonSchema(schema);
}

I’m reading\watching tutorials, but it’s just not clicking. Is there an easy way to do this?

P.S.
This passes in postman. Does anyone know how?

const scshema = {
    "type": "objedfasgct"
};

pm.test("Validate status_items"), () => {
    pm.response.to.have.jsonSchema(schsdfddfema);
}

danny-dainton · November 10, 2020, 10:07pm

I tend to use jsonschema to help create the schema.

It has a number of different options that will create a number of different schema structures - depending on your needs.

It’s also tricky creating the schemas, it’s a whole new syntax and way of structuring the object that easy to get wrong and give false positives.

I think that both @allenheltondev and @vdespa have knowledge in this area and might be able to provide more assistance.

robmcec · November 10, 2020, 10:14pm

Yes. The false positives are making me think of not even attempting this.

allenheltondev · November 10, 2020, 10:21pm

I use jsonschema as well and validate with ajv - which coincidentally is included in the postman code sandbox!

I’m not sure what that assertion is doing, but loading the schema in ajv and validating against a json object is pretty powerful.

If you’re defining your API in an OpenAPI Spec, then you’re already halfway there to getting what you’re looking for.

robmcec · November 10, 2020, 10:29pm

We are defining our API in OpenAPI Spec in PM.

What’s the other half of what I need to do?

allenheltondev · November 10, 2020, 11:10pm

Are you defining it in yaml or json?

In theory, you can just take the schema definition defined in the schema section and add that to ajv.

I have a dynamic test generator collection I made to test scenarios exactly like this. If you’re familiar with javascript, you should be able to follow along what it’s doing:

robmcec · November 10, 2020, 11:25pm

json

Ok. I’ll have to check that out.

Thanks!

vdespa · November 11, 2020, 7:33am

@robmcec

pm.test takes two parameters:

A string representing the name of the test
A callback function containing the assertions.

You are only providing the first parameter and closing the function. Does that make sense?

robmcec · November 12, 2020, 5:30pm

I used jsonschema to convert a response body into a schema to test against.

let Ajv = require('ajv'),
ajv = new Ajv({logger: console}),
schema = {
    "$schema": "http://json-schema.org/draft-07/schema",
    "$id": "http://example.com/example.json",
    "type": "object",
    "title": "The root schema",
    "description": "The root schema comprises the entire JSON document.",
    "default": {},
    "examples": [
        {
            "data": {
                "statusItem": [
                    {
                        "ID": 12,
                        "cost": 0,
                        "discountType": 0,
                        "discountValue": 0,
                        "flags": 0,
                        "flags3": 0,
                        "flags4": 0,
                        "flags5": 0,
                        "futureBreakID": 0,
                        "group": 1,
                        "highlightColor": 65535,
                        "inActive": 0,
                        "miscItemID": 0,
                        "name": "Surface",
                        "notificationID": 0,
                        "overrideRejectMins": false,
                        "POReturnReasonID": 0,
                        "priority": 0,
                        "rejectStatusMins": 5,
                        "seqNum1": 0,
                        "seqNum2": 0,
                        "userOrGroupID": 0
                    }
                ]
            }
        }
    ],
    "required": [
        "data"
    ],
    "properties": {
        "data": {
            "$id": "#/properties/data",
            "type": "object",
            "title": "The data schema",
            "description": "An explanation about the purpose of this instance.",
            "default": {},
            "examples": [
                {
                    "statusItem": [
                        {
                            "ID": 12,
                            "cost": 0,
                            "discountType": 0,
                            "discountValue": 0,
                            "flags": 0,
                            "flags3": 0,
                            "flags4": 0,
                            "flags5": 0,
                            "futureBreakID": 0,
                            "group": 1,
                            "highlightColor": 65535,
                            "inActive": 0,
                            "miscItemID": 0,
                            "name": "Surface",
                            "notificationID": 0,
                            "overrideRejectMins": false,
                            "POReturnReasonID": 0,
                            "priority": 0,
                            "rejectStatusMins": 5,
                            "seqNum1": 0,
                            "seqNum2": 0,
                            "userOrGroupID": 0
                        }
                    ]
                }
            ],
            "required": [
                "statusItem"
            ],
            "properties": {
                "statusItem": {
                    "$id": "#/properties/data/properties/statusItem",
                    "type": "array",
                    "title": "The statusItem schema",
                    "description": "An explanation about the purpose of this instance.",
                    "default": [],
                    "examples": [
                        [
                            {
                                "ID": 12,
                                "cost": 0,
                                "discountType": 0,
                                "discountValue": 0,
                                "flags": 0,
                                "flags3": 0,
                                "flags4": 0,
                                "flags5": 0,
                                "futureBreakID": 0,
                                "group": 1,
                                "highlightColor": 65535,
                                "inActive": 0,
                                "miscItemID": 0,
                                "name": "Surface",
                                "notificationID": 0,
                                "overrideRejectMins": false,
                                "POReturnReasonID": 0,
                                "priority": 0,
                                "rejectStatusMins": 5,
                                "seqNum1": 0,
                                "seqNum2": 0,
                                "userOrGroupID": 0
                            }
                        ]
                    ],
                    "additionalItems": true,
                    "items": {
                        "$id": "#/properties/data/properties/statusItem/items",
                        "anyOf": [
                            {
                                "$id": "#/properties/data/properties/statusItem/items/anyOf/0",
                                "type": "object",
                                "title": "The first anyOf schema",
                                "description": "An explanation about the purpose of this instance.",
                                "default": {},
                                "examples": [
                                    {
                                        "ID": 12,
                                        "cost": 0,
                                        "discountType": 0,
                                        "discountValue": 0,
                                        "flags": 0,
                                        "flags3": 0,
                                        "flags4": 0,
                                        "flags5": 0,
                                        "futureBreakID": 0,
                                        "group": 1,
                                        "highlightColor": 65535,
                                        "inActive": 0,
                                        "miscItemID": 0,
                                        "name": "Surface",
                                        "notificationID": 0,
                                        "overrideRejectMins": false,
                                        "POReturnReasonID": 0,
                                        "priority": 0,
                                        "rejectStatusMins": 5,
                                        "seqNum1": 0,
                                        "seqNum2": 0,
                                        "userOrGroupID": 0
                                    }
                                ],
                                "required": [
                                    "ID",
                                    "cost",
                                    "discountType",
                                    "discountValue",
                                    "flags",
                                    "flags3",
                                    "flags4",
                                    "flags5",
                                    "futureBreakID",
                                    "group",
                                    "highlightColor",
                                    "inActive",
                                    "miscItemID",
                                    "name",
                                    "notificationID",
                                    "overrideRejectMins",
                                    "POReturnReasonID",
                                    "priority",
                                    "rejectStatusMins",
                                    "seqNum1",
                                    "seqNum2",
                                    "userOrGroupID"
                                ],
                                "properties": {
                                    "ID": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/ID",
                                        "type": "integer",
                                        "title": "The ID schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            12
                                        ]
                                    },
                                    "cost": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/cost",
                                        "type": "integer",
                                        "title": "The cost schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "discountType": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/discountType",
                                        "type": "integer",
                                        "title": "The discountType schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "discountValue": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/discountValue",
                                        "type": "integer",
                                        "title": "The discountValue schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "flags": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/flags",
                                        "type": "integer",
                                        "title": "The flags schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "flags3": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/flags3",
                                        "type": "integer",
                                        "title": "The flags3 schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "flags4": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/flags4",
                                        "type": "integer",
                                        "title": "The flags4 schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "flags5": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/flags5",
                                        "type": "integer",
                                        "title": "The flags5 schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "futureBreakID": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/futureBreakID",
                                        "type": "integer",
                                        "title": "The futureBreakID schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "group": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/group",
                                        "type": "integer",
                                        "title": "The group schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            1
                                        ]
                                    },
                                    "highlightColor": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/highlightColor",
                                        "type": "integer",
                                        "title": "The highlightColor schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            65535
                                        ]
                                    },
                                    "inActive": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/inActive",
                                        "type": "integer",
                                        "title": "The inActive schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "miscItemID": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/miscItemID",
                                        "type": "integer",
                                        "title": "The miscItemID schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "name": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/name",
                                        "type": "string",
                                        "title": "The name schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": "",
                                        "examples": [
                                            "Surface"
                                        ]
                                    },
                                    "notificationID": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/notificationID",
                                        "type": "integer",
                                        "title": "The notificationID schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "overrideRejectMins": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/overrideRejectMins",
                                        "type": "boolean",
                                        "title": "The overrideRejectMins schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": false,
                                        "examples": [
                                            false
                                        ]
                                    },
                                    "POReturnReasonID": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/POReturnReasonID",
                                        "type": "integer",
                                        "title": "The POReturnReasonID schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "priority": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/priority",
                                        "type": "integer",
                                        "title": "The priority schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "rejectStatusMins": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/rejectStatusMins",
                                        "type": "integer",
                                        "title": "The rejectStatusMins schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            5
                                        ]
                                    },
                                    "seqNum1": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/seqNum1",
                                        "type": "integer",
                                        "title": "The seqNum1 schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "seqNum2": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/seqNum2",
                                        "type": "integer",
                                        "title": "The seqNum2 schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    },
                                    "userOrGroupID": {
                                        "$id": "#/properties/data/properties/statusItem/items/anyOf/0/properties/userOrGroupID",
                                        "type": "integer",
                                        "title": "The userOrGroupID schema",
                                        "description": "An explanation about the purpose of this instance.",
                                        "default": 0,
                                        "examples": [
                                            0
                                        ]
                                    }
                                },
                                "additionalProperties": true
                            }
                        ]
                    }
                }
            },
            "additionalProperties": true
        }
    },
    "additionalProperties": true
}

Next, I used the following test:

pm.test('Schema is valid', function() {
    var error = pm.response.json()['errors'];
    pm.expect(ajv.validate(schema, {errors: error})).to.be.true;
});

It failed with this very sad error:
Schema is valid | AssertionError: expected false to be true

This error doesn’t provide nearly enough information to troubleshoot.
Should I be using different parameters for the function?

BTW, I got this method from

danny-dainton · November 12, 2020, 5:56pm

You should be able to just have this:

let schema = {
    "type": "object",
    "required": [
        "data"
    ],
    "properties": {
        "data": {
            "type": "object",
            "required": [
                "statusItem"
            ],
            "properties": {
                "statusItem": {
                    "type": "array",
                    "items": {
                        "anyOf": [
                            {
                                "type": "object",
                                "required": [
                                    "ID",
                                    "cost",
                                    "discountType",
                                    "discountValue",
                                    "flags",
                                    "flags3",
                                    "flags4",
                                    "flags5",
                                    "futureBreakID",
                                    "group",
                                    "highlightColor",
                                    "inActive",
                                    "miscItemID",
                                    "name",
                                    "notificationID",
                                    "overrideRejectMins",
                                    "POReturnReasonID",
                                    "priority",
                                    "rejectStatusMins",
                                    "seqNum1",
                                    "seqNum2",
                                    "userOrGroupID"
                                ],
                                "properties": {
                                    "ID": {
                                        "type": "integer"
                                    },
                                    "cost": {
                                        "type": "integer"
                                    },
                                    "discountType": {
                                        "type": "integer"
                                    },
                                    "discountValue": {
                                        "type": "integer"
                                    },
                                    "flags": {
                                        "type": "integer"
                                    },
                                    "flags3": {
                                        "type": "integer"
                                    },
                                    "flags4": {
                                        "type": "integer"
                                    },
                                    "flags5": {
                                        "type": "integer"
                                    },
                                    "futureBreakID": {
                                        "type": "integer"
                                    },
                                    "group": {
                                        "type": "integer"
                                    },
                                    "highlightColor": {
                                        "type": "integer"
                                    },
                                    "inActive": {
                                        "type": "integer"
                                    },
                                    "miscItemID": {
                                        "type": "integer"
                                    },
                                    "name": {
                                        "type": "string"
                                    },
                                    "notificationID": {
                                        "type": "integer"
                                    },
                                    "overrideRejectMins": {
                                        "type": "boolean"
                                    },
                                    "POReturnReasonID": {
                                        "type": "integer"
                                    },
                                    "priority": {
                                        "type": "integer"
                                    },
                                    "rejectStatusMins": {
                                        "type": "integer"
                                    },
                                    "seqNum1": {
                                        "type": "integer"
                                    },
                                    "seqNum2": {
                                        "type": "integer"
                                    },
                                    "userOrGroupID": {
                                        "type": "integer"
                                    }
                                }
                            }
                        ]
                    }
                }
            }
        }
    }
}

pm.test('Schema is valid', function() {
    pm.response.to.have.jsonSchema(schema)
});

I’m not really sure what you’re trying to do in that test

robmcec · November 12, 2020, 6:07pm

Holy cr*p that works as expected! Sorry about my profanities.

I can even get it to fail how I’d expect.

How did you turn my incorrect thing into the correct thing?

It looks like you took mine and just chopped out everything except “type”.

Did you use some other schema generator, or did you just manually change it?

Also, we’re using a defined Open API Spec in the API section. After looking at this is the schema check (as a test script) even necessary?

allenheltondev · November 13, 2020, 12:23pm

It really depends on the implementation of your API.

The open api spec in Postman is just a definition. In my opinion it’s always worth the effort it takes to run a quick ajv.validate on a response to make sure the developers did what they were supposed to do.

robmcec · November 13, 2020, 2:24pm

So do I need to add the schema definition in the test itself,
Like this:

let schema = {
    "type": "object",
    "required": [
        "data"
    ],
    "properties": {
        "data": {
//and so on

or can I just tell adj to validate against the defined schema somehow?

danny-dainton · November 13, 2020, 3:00pm

If the schema has been structured correctly, this would validate that against the response body.

pm.test('Schema is valid', function() {
    pm.response.to.have.jsonSchema(schema)
});

robmcec · November 13, 2020, 3:17pm

Sorry, by schema, you mean the schema that lives with the test case, not the schema in the API -> Define section?

I guess the question I have is "Is there a way to use the schema in the API -> Define tab in test cases instead of having to create that schema monstrosity and test against it?

It seems like we’d be doing the same thing twice if we did that.

danny-dainton · November 13, 2020, 3:24pm

This schema:

let schema = {
    "type": "object",
    "required": [
        "data"
    ],
    "properties": {
        "data": {
//and so on

You could change the name to avoid confusion if you like

This might be what you’re talking about for validation between an API Spec and a Collection?

https://learning.postman.com/docs/designing-and-developing-your-api/validating-elements-against-schema/

SLaBoss · November 17, 2020, 4:11pm

I use the following syntax so that all validation errors are reported back, where ‘schema’ is what you would have defined for the desired validation. I feel using ‘allErrors: true’ is key:

var Ajv = require('ajv'),
    ajv = new Ajv({ logger: console, allErrors: true }); 

tests["Schema is valid"] = ajv.validate(schema, pm.response.json());
if (ajv.errors !== null) {
    tests["Schema validation error(s): " + JSON.stringify(ajv.errors) + ";"] = false;
}

robmcec · November 17, 2020, 4:47pm

Ok, thank you. I’ll take a look at that.

I think what confuses me about schema testing is that we have an open api definition for the collection.

Won’t the response results get validated against the open api schema? If our response is bad, we should see a notification that says we have “issues” above the endpoint URL.

This seems like doing the same thing twice. Once when PM validates the response and once when avj validates against the schema provided to it in the test script.

var schema = {schema stuff}

Is this correct, or am I missing something?

If we have an open api definition, do we still need to do the schema validation in the test script?

Thanks!

danny-dainton · November 17, 2020, 5:28pm

If you have created your API (Validation is available for OpenAPI 3.0 schemas.) in Postman and generated Collections from that or linked the existing collections to that - It would validate those requests in the same way and give you a summary of the issues found.

https://learning.postman.com/docs/designing-and-developing-your-api/validating-elements-against-schema/#validating-elements

Is this what you have set up for your API now? Are you able to validate the Collections and Request against the API spec?

robmcec · November 17, 2020, 5:42pm

Yes, we have the schema defined and linked to the collection that I’m testing.

It will be versioned and updated using the schema validator in PM.

Since we do that, we don’t really need to use Avj correct?

Topic		Replies	Views
Schema validation Ask the Experts and Postman Tips tests , pm , schema , jsonschema , schemavalidation	0	477	June 9, 2021
Postman test script for json schema validation failing for some custom format types in json schema Ask the Experts and Postman Tips tests , jsonschema , schemavalidation	0	476	August 25, 2021
Validating a schema has all the right properties and nothing more Ask the Experts and Postman Tips tests , schema	2	527	July 1, 2022
Time-stamp json schema Ask the Experts and Postman Tips jsonschema	5	1656	February 8, 2024
Schema Validation of SOAP in Postman Ask the Experts and Postman Tips tests	5	4554	December 15, 2021

Schema validation help

Related topics