Test Summary export data not showing Assertion messages

Ask the Experts and Postman Tips
1

Exporting the results of a Test do not show the Assertion message. Message expected to be found on the report:
AssertionError: Password Invalida: expected ‘ErrorUserBlockedByPassword’ to equal ‘Success’

{
	"id": "0d974bc-083a-4e05-b5bf-f4fd54aa31a4",
	"name": "Public Services",
	"timestamp": "2024-05-15T18:48:12.538Z",
	"collection_id": "25665066-3f5add1-d7f1-4131-882b-306ac9ec695e",
	"folder_id": 0,
	"environment_id": "25665066-dfca6a3-7fa3-4176-8c1d-215363362e11",
	"totalPass": 0,
	"delay": 0,
	"persist": true,
	"status": "aborted",
	"startedAt": "2024-05-15T18:47:59.777Z",
	"totalFail": 2,
	"results": [
		{
			"id": "734447e0-ab51-457c-93ed-51b298ac80d1",
			"name": "Login-Batch",
			"url": "https://bleh/api/General/SignIn",
			"time": 5662,
			"responseCode": {
				"code": 200,
				"name": "OK"
			},
			"tests": {
				"¿Login es valido?": false
			},
			"testPassFailCounts": {
				"¿Login es valido?": {
					"pass": 0,
					"fail": 2
				}
			},
			"times": [
				5911,
				5662
			],
			"allTests": [
				{
					"¿Login es valido?": false
				},
				{
					"¿Login es valido?": false
				}
			]
		}
	],
	"count": 3,
	"totalTime": 11573,
	"collection": {
		"requests": [
			{
				"id": "734447e0-ab51-457c-9ed-51b298ac80d1",
				"method": "POST"
			}
		]
	}
}
2

Hey @joseluisyr :wave:

Welcome to the Postman Community! :postman:

Those messages have never been part of that output from the runner, or at least as far back as I can remember. That report is more of a metric based output of passed and failed tests.

How are you intending to use the output? It’s quite a manual step each time, what’s the workflow you had in mind?

3

Hi @danny-dainton
What a bummer I was hoping that something was missing… I thought I could export this data and use it in a more complex report for non-technical people who could understand why each iteration failed.

For example the Assertion is comparing the pm.response.json().OperationResult and for this example is giving me ‘ErrorUserBlockedByPassword’ so I want this value to be shown so the user can actually understand why the test failed.

4

I’m not trying to change your flow but if you use Newman to run the Collections, you could use a custom reporter to show a report to those folks.

5

Thanks I’m going to give it a look. But quite shocked to find that there is no way to actually know why the test failed in the report.

6

You see that in the UI for the Collection Run or in any individual requests.

image
image795×468 75.3 KB

image
image1844×1564 156 KB

You can even share a specific run with your teammates.

Are you all not part of the same team?

7

This also shows the need to consider your test policy\strategy including your naming conventions and assertion rules.

I personally recommend that the test case name needs to be relevant and ideally should be able to tell you exactly what has failed by the test case name alone.

Be careful with the number of assertions.

I would recommend 1-2 assertions max for each test block.

Anything more than that, and you might want to consider breaking it into separate tests, so you can a more accurate feedback.

You might also want to consider defining variables for each element you want to check.

So you can do something like…

pm.expect(variable1).to.eql(variable2)

Instead of

pm.expect(largeJavaScriptObject).to.eql(someVariable)

As that is likely to be very hard to read in the GUI or anywhere else, when it fails.

8

Yes but what if my test is running thousand of transactions so the user would need to scroll on this UI report one by one and on top of that they have no way to group the errors or do any sort/search function.

9

In the Collection Runner, you can filter the run by failed tests which would show that information.

They wouldn’t need to scroll through all of the requests over multiple iterations, only the failed tests.

If you’re consistently seeing failed tests, enough that they would need to be scrolled through each time, that feels like a different problem. :grimacing:

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.