We have a Collection which contains several sub-folders. Each sub-folder contains tests to group like concerns. When using the Collection Runner, and specifying the entire collection as the run context we see test failures that we do not see when running the tests alone or via specifying the Sub-folder.
I can’t see a way to resolve this myself, can anyone point me in the right direction or any known issues related to the Collection Runner and sub-folders?
@jadams Are you seeing the tests being run in the correct order?
You dont happen to have any sort of Workflow in place in your tests do you?
The tests are being run in the correct order. There is some workflow in place, but behavior is the same even with the line removed. Below is the test failing. The ‘verify phone’ and ‘verify name’ are the assertions that are failing. Seems very straight forward for an intermittent failure?
tests["Status code is 200"] = responseCode.code === 200;
var jsonData = JSON.parse(responseBody);
var data = jsonData.data;
tests['verify source id'] = data.sourceId == postman.getEnvironmentVariable('source_id');
tests['verify channel id'] = data.channelId === 1;
tests['verify phone'] = data.phone === "new phone";
tests['verify name'] = data.name === "new name";
tests['verify source code'] = data.sourceCode === JSON.parse(pm.environment.get("postedSourceObject")).sourceCode;
tests['verify address'] = data.address === "new address";
tests['verify city'] = data.city === "new city";
tests['verify state'] = data.state === "new state";
tests["verify zip"] = data.zip === "new zip";
tests["verify email"] = data.email === "new email";
tests["verify fax"] = data.fax === "new fax";
tests["verify data source name"] = data.dataSourceName === "new dataSourceName";
tests["verify data source id"] = data.dataSourceId === "new dataSourceId";
postman.setNextRequest ("PUT Update Source");@jadams By any chance is the workflow step postman.setNextRequest ("PUT Update Source"); outside of the current folder being executed?
There are some rules for Workflows that can impact your runs.
Remember these two facts as you use this workflow:
- postman.setNextRequest() is always executed at the end of the current script. If you put this function before other code blocks, these blocks will still execute.
- postman.setNextRequest() has a scope, which is the source of your collection run. If you run a collection, you can jump to any request in the collection (even requests inside folders, using the same syntax). However, if you run a folder, the scope of postman.setNextRequest() is limited to that folder. So you can jump to any request in this folder, but not ones that are outside of the folder. It includes requests inside other folders, and also root-level requests in the collection.
I am thinking that your issue could be the scope of your Workflow.
No, ‘PUT Update Source’ is in the same folder and the very next test. Also, I get the same behavior if the postman.setNextRequest line is removed.
Ok, so we know its not the workflow then. If its the next call in the flow you can remove that from the Test as it serves no purpose.
Without knowing a bit more on the how the collection is structured its hard to determine what is going on. Feel free to PM me if there is anything you don’t feel comfortable sharing publicly.