I am trying to run a collection with 22 requests using the Performance tests functionality of the Postman Collection Runner. The requests are following a workflow of creating a user, login into an account, creating a virtual machine, creating a drive, starting and stopping the machine, cloning the drives, etc. I am running the tests using the following example configurations:
20 Virtual Users for 10 minutes (The Fixed option)
20 Virtual Users for 10 minutes with ramp up duration of 5 minutes (The ramp up option)
My question is how many times each user will run each request because I noticed the following:
Tests with the ‘Ramp up’ option - the first request will be run 70 times, the second one 68 times … the last (22nd one) 10 times
How long does it normally take to run the 22 requests in order?
20 users \ 5 minutes is one user every fifteen seconds.
I’m wondering if it just kills all current requests (no matter what stage they are) when it hits the 10 minute limit.
If it takes longer than five minutes to complete all 22 actions, then you are going to have requests in different states.
Do the requests that run near the end generally take longer to process? Restarting the VM for example.
How about if you run the “fixed” options. Do you get consistent numbers then?
I haven’t done any performance testing for a while, but we always used to baseline this with 1 request through your flow, then 2 until we started hitting drops in timings rather than jumping directly into a lot of users (although I’m not sure what you expect to be a decent number of concurrent requests).
The requests in the end should take less time to run because they are just listing the different resources under the account - drives, VMs, subscriptions, transactions, etc.
Even with the ‘fixed’ option, the numbers are not consistent:
