Postman fail a test step but move on and report as one test failure?

🙋 Help
1

Hi all

So we were testing the APIs using pm.expect etc but the issue was inside a test once the first pm.expect fails, it would move on to the next test and so wouldn’t report on any other failures.

So we changed the test now to do a try catch on each test step:

try {
pm.expect(actualValue).to.eql(expectedValue);
}
catch (e)
{
pm.test(“FAILURE on the expected value vs the Actual”, () =>
{
throw new Error(e.message)
});
}
}

Problem is though, if we have say two of these try catches inside a test to test 2 different json objects, postman will report two test failures

Is there anyway to get postman to just report this as one test case failure, with multiple failed test steps? Otherwise we are going to end with 1000s of failed test cases instead of say 60

Thanks

2

I’m not sure this is possible. Postman uses the Chai library for assertions.

Chai does not allow soft asserts, it is against their assertion philosophy.

The assertion should either prove or disprove a thing, and that should fail the entire test (because whatever the description of “pm.test” is, no longer holds true).

Therefore its a hard fail as soon as you hit an error within a pm.test block.

The principle here is that the tests should be singular as possible in design.

Maybe a feature request is needed.

Therefore unless Postman decides to also include a soft assertion library in the sandbox, I don’t think there is an easy way for you to achieve what you want.

I suspect you can create a wrapper like you are doing with try\catch. Perhaps have a look on a JavaScript forum for an answer on how to implement that. As this is more of a JavaScript query than Postman at this point. Although a quick search seems to advise you to use a different assertion library. (Which isn’t possible with Postman).

On a side note, Postman doesn’t really fall neatly into the testing principles you may have with other products, so it does need some thought about how you want to layer your requests\tests.

In my mind, the Collection is the Test Suite, the Folders are the Test Cases, the requests are the test steps. Within the test steps, you can have multiple assertions, and I generally try and make these as singular as possible so I can get the feedback on all of the assertions I include. Which means multiple pm.tests. Otherwise you have the issue you are finding in that you don’t get the feedback on the other assertions after the first one fails. (This may or may not be important to the system you are testing). If we are developing the API ourselves, then it generally is important. If we are consuming a 3rd party API, then it might not be so important.

Your situation seems to be a different use case so the above may not work for you.

3

Thankyou! Yyeah I kind of thought this would be the case. I am used to doing UI Testing where one test has many steps, The test fails overall but inside the report it will list each fail in the test

No worries anyway I am looking at potentially parsing the json output to group together these test steps that come from one test. If I make the name start with the same test name I can group them together as one. Cheers