Newman htmlextra reporter - More user input needed 😀

Hey folks :wave:

It’s been a while now since I first created a new HTML reporter for Newman. I have modified and changed different parts of the report over time but I would really love some more input from the Postman community, to help me improve the reports even more.

Any suggestions, improvements, issues or whatever, are very welcome! :slight_smile:

If you’ve used the reporter already, that’s awesome! Thank you! :trophy:

For people wanting to give it a go, you can install the NPM package locally and give it a go.

If you wanted to try before you buy - Here is an example report for you to take a look at and click through the different parts.

Thank you!! :star:

10 Likes

All the prasies for your excellent work aside, full marks for the “try before you buy” link! :trophy: :100:

1 Like

Hi,
First at all thank you for you job and thank you for share it with us.
we use daily your report and we think that is great .
The only think is the size of the HTML Page .
Just one point
We run 500 iteration of the same collection with different data and the size of the report it around 200 MB . Maybe need to find a way to load dynamically the request and repose data and not insert it inside the HTML .
Best Regards ant thank you again

2 Likes

Great example report. Thank you so much spending effort on these and make our life easier. Starwars quotes: “May the force always be with you”. :upside_down_face:

Just an suggestion that:
1.) Appreciate if you can enhance it further like Gitlab CI dashboard. At one glance, all the stakeholders are able to look at the whole scenario. Dashboard will show with colors that collection names, descriptions, iterations, assertions, response times, data file names that we are executed with newman or even better with which lines in json file that it hits error. Just an example idea that may make your day with following link gitlab-ci-dashboard - npm

2.) Moreover, I notice that there is no pie chart for it. Probably you can consider develop it.

3.) Unfortunately the runner itself couldn’t select or pick any collection stored in team workspace. Otherwise, it should made the report more perfect by identify which collection are actually runned by using team member workspace, it can show which members are actually executed it.

Thanks much! :pray:

1 Like

Thanks for the feedback :heart:

The default static report is generated from the Summary Object from Newman and the more information that is added into the collection (More Folders, More Requests, More Tests, More Console Log Statements etc.) times by the number of Iterations, will make the static report continue to grow in size.

I love that people want to continue to use the default template but sometimes, this just contains too much information - The way I think about the output, Is asking myself who is it for? What information would be valuable to them? What do they actually care about?

If the receiver of the report wants to know all the request/response details from X request inside X folder on the 356th iteration, then report as it is, provides that value. If they just want a very high level view of Collection run, that’s where the beauty of creating a new custom template comes in and you can tailor the information captured and ultimately reduce the size of the file report.

Thank you for your feedback and the suggestions :trophy:

I think the context of the report differs from the information that the gitlab-ci-dashboard would provide. If it’s a case of displaying certain relevant and important information on a one-page dashboard, that can be achieved with a new custom template.

It’s just a case of deciding what’s useful to have on that page, the Collection and run details are available to you.

Things like adding different charts (bar, pie, doughnut, etc.) are simple enough but it’s just a case of styling and getting the layout right, as you can see, just dumping something at the bottom of a page looks a bit nasty :frowning:

I’m not too sure what you mean on your 3rd point, would you be able to explain this more or maybe provide an example?

2 Likes

Hey Danny,

Thank you and major Kudo’s for the detailed documentation on your Report. I’ve used postman for a couple of years and haven’t had much luck in getting “Reporters” to work. I got yours working on the first try!!!

I like your report format and plan on using it.

2 Likes

Hi Danny… congrats for your work! I’m using it a lot, it’s exactely what I needed. I even tried to make some adjustments on the template, although it’s my very first time with HBS templates.

I have a question, I cannot get my SOAP response body to be formatted with the pretty print css class. I get the whole response in one huge line. Have you heard of other people having a similar issue? Could it be the HTTP headers?

Take a look at this image

Let me know if you have any suspect :slight_smile:

Also, for newer versions, could you add a Copy to Clipboard button to the request body as well?

I’m using [email protected]

Hey @obisignano-c,

Thank you for using the reporter!! :trophy:

Can you update to the latest version, currently 1.8.8 and give it another go, please? :slight_smile:

It should print those responses like this:

It’s a short term fix at the moment but hopefully, it makes life easier for you.

I’ve also added a Copy to Clipboard button for the request body :slight_smile:

1 Like

You rock man! It works perfectly, thanks for your quick response… I’ll let you know if I find something else that could be interesting to enhance, but this
definitely does my life better these days J

2 Likes

Just wanted to use this thread as a quick place to post an update - The latest version of the htmlextra reporter (1.9.2) includes a couple of new flags/arguments, that allows you to exclude certain pieces of sensitive data from the final report.

To exclude a specific Request/Response Header, in this case, the Authorization header, the following CLI argument can be used - --reporter-htmlextra-skipHeaders.

An example of using this within the full newman run command would be:

From the CLI:

newman run https://www.getpostman.com/collections/631643-f695cab7-6878-eb55-7943-ad88e1ccfd65-JsLv --folder 'Authentication Methods' -r htmlextra --reporter-htmlextra-skipHeaders Authorization

From a Script:

const newman = require('newman');

newman.run({
    collection: 'https://www.getpostman.com/collections/631643-f695cab7-6878-eb55-7943-ad88e1ccfd65-JsLv',
    folder: 'Authentication Methods',
    reporters: 'htmlextra',
    reporter: {
        htmlextra: {
            skipHeaders: [ 'Authorization' ]
        }
    }
}, function (err) {
    if (err) { throw err; }
    console.log('collection run complete!');
});

To exclude Headers and Bodies in all the requests and responses, leaving only the high-level request metadata and Test results, the following CLI argument can be used - --reporter-htmlextra-skipSensitiveData.

An example of using this within the full newman run command would be:

From the CLI:

newman run https://www.getpostman.com/collections/631643-f695cab7-6878-eb55-7943-ad88e1ccfd65-JsLv --folder 'Authentication Methods' -r htmlextra --reporter-htmlextra-skipSensitiveData

From a Script:

const newman = require('newman');

newman.run({
    collection: 'https://www.getpostman.com/collections/631643-f695cab7-6878-eb55-7943-ad88e1ccfd65-JsLv',
    reporters: 'htmlextra',
    reporter: {
        htmlextra: {
            skipSensitiveData: true
        }
    }
}, function (err) {
    if (err) { throw err; }
    console.log('collection run complete!');
});

Hopefully, these options will give you extra flexibility when sharing the final reports with other people outside of your team.

1 Like

Hey @danny-dainton! So I recently nerded out a good bit with the htmlextra reporter and wanted to say thanks for your work on it and the great documentation!

I think there was a suggestion earlier in the thread about bringing visualizations into the reporter. I am not sure if that was before the Postman Visualize feature? I’m relatively new to Postman.

Okay feature idea-> it would be very cool to be able to marshal postman visualizations created in a request\collection over to a reporter in a standard fashion. Might be worth the squeeze? I know that we are limited by what the newman summary object contains, but I do believe there is a script section, in the Json reporter output at least that I saw. I guess there would be a decent challenge on getting a handle on what part of the script potentially contained the visualization code. Would have to maybe have standard expectation for where to find it.

Anyway -truly thank you for creating the reporter and your time!

Hey @enufacas,

That’s a really interesting post, looking forward to reading the second part :trophy:

Bringing high-level visualizations to the report wouldn’t really be that difficult, it’s just a HTML page that uses the data from Newman so that can be displayed in whatever format we can now do on the Visualizer.

I spent a few mins adding something horrendous looking to a report to prove it could be done. That’s a doughnut chart but it could have been a bar, pie or any other chart. :slight_smile:

Getting something in for each request would be interesting but you kinda need to think about what value that’s providing. If the person/s consuming the report just wanted a visual output from the run, I would just be more inclined to create a new template to handle that and show the big number value that they want rather than all the detailed information about the request.

This other consideration would be the size on the reports - These are dynamically created so the more you add to the report, the larger it’s going to get and opening them up on a browser is not going to be that performant. I’ve seen this with runs that have a huge number of iterations and the final report is pretty big. I haven’t done anything around filesize compression of the final report so, at the moment, that’s not going to be great user experience. :frowning:

Thanks again for taking a look at the reporter and for your kind words. Makes it all worthwhile :heart:

@danny-dainton I follow! Yeah actually I end up adding a bar chart. Part 2 is here, complete with an Azure devops pipeline. https://www.esmith.dev/posts/share-azuredevops-stats-in-slack-using-a-postman-collection-part2/
And yeah, the chart load time is a bit slow but I did not do anything to try and optimize it yet. Anyway thanks very much again for the reporter and your reply.

1 Like

Hi @danny-dainton I have only one test in postman pm.test(‘Validate status code is 200’, function() {
pm.response.to.have.status(200);
console.log(“first time execution”);

});

but in html report it shows me 2 test please let me know how do i read this report

Without more contextual information about the collection or a copy of the final report, it’s tough for me to know what’s happening here from just that image. :thinking:

  • Do you also have a test with, the same name, at the Collection or Folder level?

  • Are you using pm.sendRequest() in the script sandbox?

1> I have a collection with one request
image

2> in collection level I have a Pre-req -script to get token but no test

3> In one request > I have a pre req script with one test

image

so passed test should be 1 right ?

CLi report

HTML report

let me know if I need to send entire collection

Yes @danny-dainton i am sending
pm.sendRequest(getTokenRequest, (err, response) => {
const jsonResponse = response.json();
const newAccessToken = jsonResponse.access_token;
//console.log(newAccessToken);

pm.collectionVariables.set(“token”,newAccessToken);
});

in collection pre script is this why it is taking as 2, in that case if i don’t give any thing in test tab of request in request I should get at least one but I am not able to see anything test shows blank

Thanks @danny-dainton so its an open issue with no solution till now.