Issue: Postman Collections run through CLI from Azure Dev Ops inconsistently writes values to global, environmental, and collection variables and causes tests to have false negatives.
Steps to Reproduce:
Setup CLI Pipeline with Postman Collections utilizing Global, Environmental, Collection Variables shared as appropriate and are carried through all calls
Do not use a --delay-request XXX interval
Trigger the collection run
Verify that variables are read, changed, and passed in a timely manner.
Current Behavior:
Variables are intermittently not getting updated in a timely manner so that followup calls may fail because they are reading old variable data.
Expected Behavior:
Variables should be updated in a timely manner so that calls made in sequence will be able to use previously set variable data correctly.
Additional Notes: We are using Azure Dev Ops East
Postman CLI Lessons Learned:
Global variables need to be called via endpoint
(new) as of last week - all Postman variables need to be all shared now to be read in a CI/CD pipeline
they’re not implicitly shared by default
grandfathered in variables are not shared by default
Are you able to create an example version of this using the Postman Echo service so that we can see where this might not be working?
Are there any variables in different scopes with the same names? When exporting or referencing these variable scopes in your command, are you ensuring that the shared values are correct before the runs. The Postman CLI would only be able to read the values that have been explicitly shared and not ones that have only been entered locally in the Value column.
Are the requests that are made time specific, do you have any that might take longer to respond that others?
If you can share all the finer details about the requests, as all requests are different, it will help us to build of a picture of what might be happening and we can get you closer to a solution.
To be honest, it looks like it resolved itself after I added a 1000ms delay between calls and reduced flaky tests. For example, I make a graphQL call that executes a lookup and writes specific values to explicitly shared collection variables.
The followup requests used those collection variables.
the issue was that when run on cli, the follow up calls would report that variables were blank, however, since the variables were shared, they would update my local installed postman and I could see that they were updated there.
