How to loop API call and collect all paginated results into 1 JSON file

New to APIs/Postman
1

Searched and found no answer. Question very similar to someone else’s (who has not yet been answered) but I’ll give it a go.

I have to retrieve 50,000 records, but the page limit is 1,000. This means I have to run the call 50 times, each time clicking on the “offset=1,000” link. The problem is this takes a lot of time to run the calls, and THEN I have to stitch together the 50 individual JSON files as a result.

I’ve done a lot of looking on the internet, looking on this forum, and found nothing. It must be (must be…?!) a fairly common requirement. I hope someone can help?

I know it involves writing a script, but don’t know what language I should use, and it involves incrementing the offsets and saving the JSON files as one file…but have no idea how to actually implement using the POSTMAN tool. :frowning:

Any help would be 100% appreciated!!?! Can I post you a lattee?

Ben

2

Hey Ben,

Can you provide some visual examples of what the current manual process looks like?

Where are you clicking this offset link? Is the response body? In a header? Is there a count somewhere that tells you have X pages/records remaining? Is there something specific that tells you that you have nothing left. Is the page limit defined in a query param?

How are you stitching these files together? Do you have a script for have or is it copy and paste?

1 Like
3

After the first 1000 results are loaded, I save those 1000 results in a JSON file. There is a “next_page” link.
image

When I click it, it runs the same query but for the next 1000 results (e.g. offset=1000).
image

And so on until there are no more results and therefore no “next_page” returned:
image

I then take all of these JSON files and append in powerbi, but it is a manual process to load them all into PowerBI into a table format.

Hope this helps. Ideally this would just be a solution that:

  1. Runs first query
  2. Saves JSON file
  3. Runs the “Next_page” query
  4. Append this JSON file to previous
  5. Loop until no further “next_page” query returned

Tanks!!!

1 Like
4

Hey, what’s the answer to this? Feels impossible that there’s no way to do this~

5

I guess the best solution for that would creating an endpoint to get all records without pagination. But assuming that is not an option, a suitable solution should be based on 2 approaches:

Unfortunately I don’t have a straightforward solution for that case, but there is a pathway you can deep dive and came up with elements you’ll need to solve that.

2 Likes
6

I think the most obvious reason why there is no solution like this is related to the fact that Postman is not the best tool for solving this problem.

Postman can’t write files.

You can easily write some JavaScript that gets the required data and saves it to a file.

2 Likes
7

Hi, has anyone solved this problem, would be very interested in a solution

8

You can write responses to files with Postman you using pm.sendRequest() to send a response to a local server, and then loop your requests using postman.setNextRequest() to cycle through pagination until a certain stop point. You’d have to think of how to avoid race conditions with file appending and new incoming responses

However, like @vdespa mentioned there are the right tools for the right jobs - what you are trying to accomplish could be easily coded as a script, for example.

Here is a node script that cycles through paginated Eventbrite attendees, groups them together and writes the full output to a file: eb-attendee-sweep/index.js at main · clairefro/eb-attendee-sweep · GitHub

9

Really helpful, i was searching every where how to get link or downland

10

I know its not Postman or JavaScript, but if you know PowerShell, then this is personally what I would use for this task.

Most people have access to a Windows PC. Not everyone can just setup a node JS app or API for collecting data.

You can use Invoke-Webrequest and export the data in whatever format you want. JSON, CSV.

If you do want to go down the API route. Then I would also consider serverless (and codeless) options like Microsoft Flow or Logic Apps. You can surface both triggers with REST API’s. You could also integrate the flow with MS Application Insights if you really want to show off.

As mentioned, the right tool for the job. When I see these types of requests, it sounds like you are trying to use Postman as middleware, which isn’t really what it was designed for.