Is it possible to have 1 html report on all collections run in newman?

t=mytest1

newman run --bail -e $MY_ENV -g $GLOBAL_ENV $DIR/API/Collection/$t.collection.json --reporters=cli,htmlextra --reporter-htmlextra-title=“Platform-API-TestReport” --reporter-htmlextra-export $API_REPORT_DIR/$API_HTML_FILE 2>&1 | tee -a $API_REPORT_DIR/$API_LOG_FILE

t=mytest2
newman run --bail -e $MY_ENV -g $GLOBAL_ENV $DIR/API/Collection/$t.collection.json --reporters=cli,htmlextra --reporter-htmlextra-title=“Platform-API-TestReport” --reporter-htmlextra-export $API_REPORT_DIR/$API_HTML_FILE 2>&1 | tee -a $API_REPORT_DIR/$API_LOG_FILE

this will produce 2 html report
how can I make it into 1 report for all my collections?

@tohlaymui35

If you had just one collection you could have multiple folders under it that represent the different sets of tests. If you needed different environment variables between the two tests you could have it switch those in the pre-request of each folder. Then you could only have one Newman run and one report. The reporter you are using will show the different folders that were run. It would take a bit to figure out the correct way to get the variables right for each, but I think it would work.

Mick

1 Like

I’m facing the same issue as @tohlaymui35. What you generally want to have is:

  • One collection per service you own
  • One CI/CD job that does a regular health check or regression test over all services you own
  • A nice report that aggregates results from test runs in all your collections

I don’t see a way to do this without custom coding right now. Breaking this one general job into one job per service/collection or merging all collections into one doesn’t seam like a desirable option to me. I’d love to use newman-reporter-htmlextra but don’t find report aggregation of multiple collections as a feature.

Does anyone have experience with these kind of aggregated reports?

Cheers,
Matthias

Hey @maschmidt,

Unfortunately, it’s currently a single Collection run at a time. That’s the same for the native app and Newman. All the available reporters would work in the same way too (A single report per run). :frowning:

There are a few NPM modules out there that would combine collections into one so that it could be run through Newman but that would come with its own set of limitations/problems.

Thanks for mentioning that you would like to use my reporter, if there was an easy way from me add the functionality you wanted in there, I would give it a go but I honestly wouldn’t know where to start.

There are a few other versions of the Newman reporters that would output the data in a format that might help you to aggregate that all together after a certain CI/CD job but I would suspect that you would need so way to store that data after each run and then have some custom logic to give you the final result you need.

Hi @dannydainton,

thanks for getting in touch.

I agree, merging collections into one doesn’t seam like a good option. I expect issues with collection-wide settings such as variables or authentication. Also, I tried postman-combine-collections and it doesn’t look very promising.

It should be some sort of post-processing step. By default all reporters output their a file into a /newman subfolder which is the place to store the data after each run. There could be a post-processing reporter that picks up those files and aggregates general data such as total assertions, total failed tests, total skipped test, etc. This data could be presented in an overview which links to each collection run html file separately. No need to show details such as request headers or response body from all collection runs on one page.

Here is a draft of such an overview I derived from your htmlextra report. Each row of the collection run table could link to a separate htmlextra report.

Wdyt? Just brainstorming here … :slight_smile:

Cheers,
Matthias

I am looking exactly at the same thing, as I plan to run a set of tests in as script. Request the user for some manual actions and update environment variables accordingly to run an additional test case set. What I wanted is to join the generated reports into one, as it is actually a single test set.

Thanks for any help!!!
Gonzalo

I am also looking for a similar thing. I need separate collections for different modules and it does not make sense to combine it.
However, Once in a week we execute all the collections via CI integration and currently it generates different reports (one per collection)…
If there is a way we can show a single aggregated report of all the collection runs, it would be awesome…I liked the example of htmlextra report shown above by @maschmidt :slight_smile:

@dannydainton by any chance this is in your backlog or something?
However, thanks for the current HTML Report which htmlextra provides. Its comprehensive as well as very user friendly.

Thanks,
Nishant Shah

Nope :smiley:

Newman runs a single collection at a time and with that, comes a single summary object, where the reporter gets it’s data.

It’s got nothing to do with the reporter, Newman would need to be able to combine and run multiple collections at once before I could even look at the data to see what I could do in the reporter.