Postman Local Variables in Performance Testing

Ask the Experts and Postman Tips
,
1

I have a series of requests where values in later ones depend on responses from earlier ones, i.e. make a GET request, get an ID from it, send that ID as part of a POST request. To do this I’m using pm.variables, setting values from one request and getting them in another. This works fine as expected for functional tests run one at a time.

When doing performance testing with multiple virtual users, though, I found the same data being sent by different virtual users, even though each was bound to one row in a .csv file, and said repeat data was bunched close together in time. I assumed that local variables, set by pm.variables, since they’re supposed to last only for a request or a collection run, were bound to their virtual user in performance tests, but this only makes sense if they’re, in fact, shared across virtual users; one VU is getting one piece of data, setting a variable according to it, then another is using that value instead of its own. Is this the case, and if so, is there a “thread-safe” way to use variables or otherwise pass data between requests?

2

Hi @wmcknight. Welcome to the Postman Community :postman_logo:.

How were you able to determine that the variables are shared across virtual users? This isn’t an expected behaviour and I’ll like your help in understanding your processs to see if we can replicate the same on our end or help clarify where necessary.

3

When I noticed the duplicate rows in the table, I reduced the performance test to make one GET request to get a unique key (itself based on a unique value from a .csv row), and then make a POST request that creates a single row containing said key in a database table. The GET requests calls pm.variables.set to set a variable with this unique key, and the POST request has a field in its request body set to its value. Running this serially for each .csv row in a functional works precisely as expected, there’s exactly one database row for each request, all of them distinct and containing this key.

When I run this as a performance test, with a delay introduced in the post-request script so I can end it after the first round of requests, I get the same number of rows, but I get duplicates among them. The only way this makes sense, when it does not happen in one-at-a-time functional testing, is if the variable the POST request is using is somehow being set by a GET request from another virtual user, i.e. one user’s GET sets the variable to 123456, and another user’s POST sends that value even though its GET fetches/sets the value as 456789.

There is the (very stupid) possibility that the vendor-supplied server software we’re testing is somehow squashing the values received by simultaneous requests, but performance testing gives very limited ability to see the requests that it sent so I can’t really check this directly. I tried making the POST request to a deliberately invalid server so I could see the errors/the bodies of the requests that caused them, but that seems to arbitrarily pick one .csv row to use in its example error even know I know for a fact the same error happened with all of them.

EDIT: I went testing some more, to catch patterns of repeated rows, and found that I was actually getting very consistent repeats for given numbers of virtual users. When I have 15, for instance, I get this set of order no each time for the same .csv file

image
image490×330 8.86 KB

Similarly for 20 VU’s
image
image487×429 11.2 KB

I also found that if I use pm.iterationData instead of pm.variables, I consistently get all the same shop order (sometimes the last in the .csv, sometimes another; this also appears to be consistent across multiple runs with the same number of VU’s), instead of multiple-with-duplicates; this also does not happen with one-by-one functional testing, where it is in order from top to bottom. These together suggest a bug, albeit not a concurrency one, in mapping virtual users to the .csv file (yes I am using ordered VU data mapping), which I’ll include for reference/to indicate that it does not include duplicates.

ShopOrder,Machine
240721,SB101
240722,SB102
240723,SB103
240724,SB104
240725,SB105
240726,SB106
240727,SB107
240728,SB109
240729,SB113
240730,SB201
240731,SB202
240732,SB203
240733,SB204
240734,SB205
240735,SB206
240736,SB207
240737,SB208
240738,SB209
240739,SB211
240740,SB212
240741,SB301
240742,SB303
240743,SB304
240744,SB306
240745,SB307
240746,SB308
240747,SB314
240748,SB401
240749,SB403
240750,SB404
240751,SB405
240752,SB406
240753,SB407
240754,SB408
240755,SB409
240756,SB410
240757,SB411
240758,SB412

EDIT 2: While the initial GET request to get the key for each .csv row is necessary in the general case, to narrow this down further I made a .csv containing the fetched keys for all of the above to take the pm.variable.set completely out of the equation, and got the exact same results. This doesn’t totally rule out variable scope leaking across virtual users like I initially suspected (the JSON body still references them with the double-curly brackets), but this further reinforces something being wrong with the VU-.csv mapping.

4

Hey @wmcknight. Will you be open to jumping on a call with an engineer from the team to take a look into with you to understand the specific issue you’re having and how you can move forward with it?

5

I would; could you also include my supervisor? I’ll provide his info however you’d prefer.

6

Yes, sure @wmcknight. Your chat is locked. If you could open it up or send me a DM here on the community forum with your email. We could continue this conversation there and find a time that works for you both.

7

I can’t find the setting to open my chats or a button to send you a DM

Literally the only thing letting me see any message-related functionality is the notification from @discobot, nothing on any profile page, nothing in my preferences or profile, nothing.

8

Hi @wmcknight. Sent you an email!

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.