Collection Runner doesn't mark the tests as failure when they fail

[uguryiilmz (Ugur)]

Hello,

Thank you for the great work and helping community to grow . Postman has been pretty useful so far

I’m trying to test my endpoint by using collection runner and json data set. I am implementing new APIs and also have old APIs that supposed to return the same values, so i mainly check if the response from old-new endpoint are equal to each other. It works well except that it doesn’t mark my iterations as failure when they fail. For example; i am checking the length from two different endpoints as below

pm.expect(new_endpoint.data).to.have.lengthOf(old_endpoint.length);

and in the console i’m getting an error saying AssertionError: expected [ Array(57) ] to have a length of 69 but got 57. However the collection runner still marks this test as passed. You can see the screenshot (Iteration 5 errors out because the lengh doesn’t match but it is still showed as success. I don’t know if this is expected or a postman bug, just wanted to get an idea in case i’m doing something wrong.

[ianb120 (ian120)]

Hi,

Just to check you are wrapping that assert in a pm.test function first of all and then using the assertion as a callback function in the test?

Otherwise it would lead to something like this, where the assert error shows but postman isn’t logging it. You’d need something like this.

pm.test("test description here...", function() {
    pm.expect(new_endpoint.data).to.have.lengthOf(old_endpoint.length);
}

[uguryiilmz (Ugur)]

Hi Ian,

yeah it’s wrapped in the function. Here’s what i have. I wonder if it’s happening because i have another async request with sendRequest

pm.test(`response from two endpoint are equal to each other with ${media_types} media_type`, function () {
    let new_endpoint = pm.response.json();
    new_endpoint.data.sort((a, b) => {
        return moment(a.date).unix() - moment(b.date).unix()
    })    

    const headerRequest = {
        url:`url`,
        method: 'GET',
        header:{
                'token': token,
        }
    }

    pm.sendRequest(headerRequest, (error, response) => {
        const old_endpoint = response.json().data
        console.log("old endpoint",old_endpoint)
        pm.expect(new_endpoint.data).to.have.lengthOf(old_endpoint.length);