Retrieve data for 1600 boards in subsets of 50 boards

[ejuenemann (Erwin)]

Hello
I’m new to Postman. I think I have a fairly simple use case, but I cannot find a similar example which would allow me to figure this out by myself.
Situation: We use a software that offers to create lots of boards with different security configuration settings. The software does not include a report feature which would allow us to retrieve a list of the boards and their configuration. However, the software offers an API which can be used to retrieve this info. Calls to the API are limited to 50 boards per run.

The info provided is that I should use the offset parameter to adjust the set of boards returned.

So, in theory, I could run the following API requests:

https://api.company.com/v2/boards?limit=50&offset=0
https://api.company.com/v2/boards?limit=50&offset=50
https://api.company.com/v2/boards?limit=50&offset=100
https://api.company.com/v2/boards?limit=50&offset=150
…

For 1600 boards, this would mean a lot of manual work. So I was hoping to be able to write a script that generates the API calls up to the point when the offset reaches 1600.

My overall idea is:

• retrieve Total (=total number of boards) and Offset from JSON response
• increment Offset by 50 each time
• use postman.setNextRequest(“GetBoards”) to repeat requests, until Offset value reaches Total value

I am facing two hurdles:

1. Passing on the updated offset value to the 2nd request
2. Prompting repeated request runs, and controlling the number of API requests carried out.

More details below. Any tips how to proceed would be much appreciated.

=== 1. Passing on the updated offset value to the 2nd request ===
TESTS tab
The following code, I hope, will increment offset after each run by 50, and place the updated value into the pm variable offsetNew

let response = pm.response.json();
let offset = response.offset
let total = response.total
console.log("Offset is: " + offset)
console.log("Total is: " + total)
offset = offset + 50
console.log("Offset + 50 is: " + offset)
pm.variables.set(‘offsetNew’,offset)
console.log("New Offset is: " + pm.variables.get(‘offsetNew’))

Now, I’d need to ‘push’ the value of offsetNew to be used in the next request run. I don’t know how to proceed. I tried a few things:
a. Request, Pre-request script: For the first run, pm.variables.get(‘offsetNew’) does not work – since it’s undefined.
b. Request, Body tab, ‘raw’ option
I tried using tab ‘Body’ of the request to feed ‘offsetNew’ into the request offset parameter: {“offset” : {{“offsetNew”}}}. This doesn’t work
c. Collection, Variables tab: I tried setting variable ‘offsetNew’ under tab Variables in the Collection that holds my request. I can’t see the impact of this.

=== 2. Prompting repeated request runs, and controlling the number of API requests carried out ===
My plan:

• Use ‘Run collection’ button for the collection in which my request (‘GetBoards’) sits
• use postman.setNextRequest(“GetBoards”)
• Logic: compare offsetNew against Total, and stop the requests once offsetNew reaches Total.

Since I haven’t figured out how to manage increment offset, I can’t test this. My initial idea is to add something like the following to tab TESTS

if (offset < total) {
postman.setNextRequest(‘GetBoards’);
}
else {
postman.setNextRequest(null);
}

[michaelderekjones (Mike Jones)]

This is similar to the Starship challenge in “15 Days of Postman for Testers”.

In theory, you just need to create a variable for the offset that you need to track and ensure its set appropriately in the params tab. (Which should end up looking like this).

https://api.company.com/v2/boards?limit=50&offset={{offset}}

You will need to define the {{offset}} variable and give it an initial value of 0. This isn’t confidential information, so a collection variable should be fine.

The rest of the magic can be done in the tests tab. (After the request has been submitted). No need for anything in the pre-request scripts.

Does the response specifically tell you if there are more pages or not?

The API in the Starship challenge returns the count, and a link for the next page.

When it hits the end, its shows null for next so you know you are at the end of the results. (You don’t really need to keep track of the total, so in your circumstance you wouldn’t necessarily need to know there are 1600 boards).

Which is then easy to control with a simple IF statement similar to the following.

if (response.next != null) {
    pm.collectionVariables.set("pageNumber", currentPageNumber);
    postman.setNextRequest("get starships");
} 

Does your API contain anything like that which you can use to determine whether you need to keep triggering setNextRequest()?

Otherwise, you will need to hardcode the 1600 and then create logic based on that.

If you want to increment a number by 50, then this is done using x += 50 which is the same as x = x + y.

Just to note, the setNextRequest() function only works with the collection runner.

[ejuenemann (Erwin)]

Hi Mike
Many thanks for replying. Based on your response, I made the edits described below.
When I run the collection, the console gives me the output I expect – it increments the Offset value for each run by 50. So far so good.
The only remaining problem: I cannot find the response. When I run the collection, the request response section (tabs Body – Cookies – Headers – Test Results) is empty.
Could you point me to where I find the response after running a collection?
Thanks
Erwin

Request, tab Params
I entered
Key: offset
Value: {{offset}}

Note:

• the URL shows the offset parameter as expected, with the curly brackets
• for testing, I set limit to 15. For the ‘real run’, this would be 50

Collection, tab Variables
I entered
Variable: offset
Initial value: 0

Then I ran the collection (Collection > Run > Run Company API requests)
Initially this threw an error. The error went away after I save request – maybe the error was caused by running the previous request version (prior to saving)

Request, tab ‘Tests’
This is interim code – repeating request until offset of 200 is reached. Later, I will replace ‘200’ with ‘total’.
My assumption is that “pm.variables.set(‘offset’,offset)” sets the postman variable ‘offset’ to the value of the JavaScript variable offset (which is calculated based on the response). The postman variable ‘offset’ is then used as a parameter for the next request.

let response = pm.response.json();
let offset = response.offset
let total = response.total
console.log("Offset is: " + offset)
console.log("Total is: " + total)
offset = offset + 50
console.log("Offset + 50 is: " + offset)
pm.variables.set(‘offset’,offset)
console.log("New value of pm variable offset is: " + pm.variables.get(‘offset’))
if (offset < 200) {
postman.setNextRequest(‘GetBoards’);
}
else {
postman.setNextRequest(null);
}

[michaelderekjones (Mike Jones)]

@ejuenemann

Could you point me to where I find the response after running a collection?

When you run the collection, you need to select “Persist responses for a session”.

Once you’ve run your collection, you can click the request and a panel should open to the right which shows your request\response details.

Or alternatively, run the request manually a few times and as we just need to see an example response.

If your offset variable is a collection variable, then use pm.collectionVariables.set to update it and pm.collectionVariables.get to retrieve it.

Otherwise you are going to get in a mess with scopes. pm.variables will set what ever the lowest scope is.

If you post code or example responses, please use the pre-formatted text option in the editor (to stop everything aligning to the left).

[ejuenemann (Erwin)]

Hi Mike
Thanks, below you see the changes I made based on your post. This is followed by the outcomes. The console output, in my understanding, suggests that 4 requests were run.
Because as new user I can only put 5 links, I remove the most part of the links

GET https://api.company.com/v2/boards?limit=4&offset=0
?limit=4&offset=50
?limit=4&offset=100
?limit=4&offset=150

So far, so good. However, after the collection was run, the Request Response section is still empty.
At the very bottom, I include an example of a (strongly redacted) response received when running the request via Send. This response looks like what I’d expect.

I tried ‘pre-formatted’ style this time.

Thanks
Erwin

=== Changes made ===

1. Request level, tab Tests, edited code

let response = pm.response.json();
let offset = response.offset
let total = response.total
console.log("Offset is: " + offset)
console.log("Total is: " + total)
offset = offset + 50
console.log("Offset + 50 is: " + offset)
pm.collectionVariables.set(‘offset’,offset)
console.log("New value of pm variable offset is: " + pm.collectionVariables.get(‘offset’))
if (offset < 200) {
postman.setNextRequest(‘GetBoards’);
}
else {
postman.setNextRequest(null);
}

2. Request level, tab Params, set limit to 4
   I simply did this to confirm the request picks up this change.
   I then saved request

3. Collection level, setting when running collection:

• ticked ‘Persist responses for a session’
• I then ran the collection

=== Outcome: Collection level, console ===

GET https://api.company.com/v2/boards?limit=4&offset=0
200
1026 ms

Offset is: 0
Total is: 1627
Offset + 50 is: 50
New value of pm variable offset is: 50

GET LINK REMOVED ?limit=4&offset=50
200
988 ms

Offset is: 50
Total is: 1627
Offset + 50 is: 100
New value of pm variable offset is: 100

GET LINK REMOVED ?limit=4&offset=100
200
684 ms

Offset is: 100
Total is: 1627
Offset + 50 is: 150
New value of pm variable offset is: 150

GET LINK REMOVED ?limit=4&offset=150
200
1026 ms

Offset is: 150
Total is: 1626
Offset + 50 is: 200
New value of pm variable offset is: 200

=== Outcome: Collection level, Variables ===
Variable offset, Initial value 0, Current value 200

=== Outcome: Request level, Response section ===
Response section has no content

=== === Outcome when running Request on Request level via Send === ===

{
“size”: 4,
“offset”: 200,
“limit”: 4,
“total”: 1627,
“data”: [
{
“id”: “uXjVNae-JQQ=”,
“type”: “board”,
“name”: “REDACTED”,
A LOT OF DATA HERE WHICH I REMOVED
],
“links”: {
“self”: “https://api.company.com/v2/boards?limit=4&offset=200”,
“first”: “LINK REMOVED ?limit=4&offset=0”,
“prev”: “LINK REMOVED ?limit=4&offset=196”,
“next”: “LINK REMOVED ?limit=4&offset=204”,
“last”: “LINK REMOVED ?limit=4&offset=1624”
},
“type”: “list”
}

[michaelderekjones (Mike Jones)]

First of all, that is not the </> Preformatted text option.

Looks like you have used the ‘’ blockquote option instead.

Your example response has a links object that has an key for the “next” page, so ideally you would use that instead of the offset. You just need to save the URL as an variable and use that instead of the offset variable.

It also includes a link for the last page, so ideally you should send a manual request to that last page and then check what is shows for next.

Hopefully its something like null which you can use to control your IF statement. In this scenario, you wouldn’t need to track the offset at all. You just need to check the link for next and keep looping until it returns null.

I don’t really know what you mean by the response having no content. As long as you have ticked the “persist” option, you just need to click on the relevant request in the Test - Run Results. This should open a tab which shows the response like in the following example.

You should have four GET requests listed in the test results. Just click on one, and the response should then show.

[ejuenemann (Erwin)]

Hi Mike
Thanks. Let’s hope others find this easier than I do. I now reached my limit for manual runs. Therefore trying out the Schedule feature, made a mistake, and the next one I can do is for Monday. I can’t see a feature ‘Persist responses for a session’ within the schedule so will see how that goes.
If the Offset approach doesn’t work I will try the Page-related approach you suggested.
Cheers, E.