More advanced data driven testing - how should one structure and run tests?

I have a lot of experience writing automated tests suites for APIs in Postman. I have always used data-driven testing and as such have always been a little frustrated with Postmans limitations in this area. When I say data-driven I am trying to achieve two goals. First I want to decouple the test from the test data but more importantly, I want to be able to parameterise my tests so that I iterate over each with different data which represent different scenarios/cases. I have some API endpoints which I hit tens of times but with different data. I am looking for advice on a better way to structure and perform my testing. I will outline the issues below.

In Postman you can run a collection against a selected environment and/or a data file. The problem here is you can pass just one data file for a complete collection. While you can technically have all your required data in the single file it might become bloated that for brevity it is actually better to hard code your data in the test within the Postman UI.

What you can do is organise your collection tests into sub-directories and then select individual sub-directories to run, each using their own data. Likewise, you could split the tests up into individual collections. The big problem here is to run the whole suite of tests is no longer a click of a button but instead, you would have to go through each sub-collection of tests and run them. Linking sub-collections to test data might also become a frustration. Just as importantly, you no longer get a single concise report which is a big problem.

My workaround to date has been to use Newman to hit specific tests or groups of tests with a specific data file. This allows me to separate out my test data to match collections and also means I can kick off all my tests in one go (by firing all the commands in one go). The problem is I am still just running sub-collections when I do this and would like to end up with a single report as if all the tests were run from one collection at the same time.

My second workaround is just to just write the tests in code but this is not the ideal solution as it makes the test suite a lot more technical. Our less technical testers would not be able to contribute to the suite and someone technical would need to always be available to maintain the API tests.

I have seen pm.iterationData and can imagine creating a pre-request script before each test to load iteration data for the test but it feels a bit hacky. I don’t know if this is even possible.

Thanks

1 Like

Did you ever get an answer to this? It’s a perfectly articulated question. I have the exact same issue and would love to get an answer.

Hey @jrush,

Welcome to the community! :trophy:

The original post didn’t really feel like a question, more like a certain usecase that doesn’t quite fit with the current functionality of the runner.

Would you be able to expand on your specific usecase more and allow others to address your issues please?

I find that although things might sound the same, if you’re not working it that exact context and running those exact requests, it’s more that likely going to be different situation for yourself/others. :grin:

We have some awesome folks here that can help if you can expand on your particular context.

1 Like

I think this is fundamentally what I’m wondering about. I have a bunch of endpoints (sorry if I don’t know the right lingo) in my collected (endpoint1?variables1, endpoint2/different_variables, etc.)

I’d like to have one file with many different combinations of variables for each endpoint. There are maybe 80 endpoints and each has 10-20 different parameter combinations (which many different types of data).

I’d like a ‘one click’ test everything, which is not clear how to do in PostMan or if it’s even possible.

Thanks.