How to loop API call and collect all paginated results into 1 JSON file

Searched and found no answer. Question very similar to someone else’s (who has not yet been answered) but I’ll give it a go.

I have to retrieve 50,000 records, but the page limit is 1,000. This means I have to run the call 50 times, each time clicking on the “offset=1,000” link. The problem is this takes a lot of time to run the calls, and THEN I have to stitch together the 50 individual JSON files as a result.

I’ve done a lot of looking on the internet, looking on this forum, and found nothing. It must be (must be…?!) a fairly common requirement. I hope someone can help?

I know it involves writing a script, but don’t know what language I should use, and it involves incrementing the offsets and saving the JSON files as one file…but have no idea how to actually implement using the POSTMAN tool. :frowning:

Any help would be 100% appreciated!!?! Can I post you a lattee?

Ben

Hey Ben,

Can you provide some visual examples of what the current manual process looks like?

Where are you clicking this offset link? Is the response body? In a header? Is there a count somewhere that tells you have X pages/records remaining? Is there something specific that tells you that you have nothing left. Is the page limit defined in a query param?

How are you stitching these files together? Do you have a script for have or is it copy and paste?

1 Like

After the first 1000 results are loaded, I save those 1000 results in a JSON file. There is a “next_page” link.
image

When I click it, it runs the same query but for the next 1000 results (e.g. offset=1000).
image

And so on until there are no more results and therefore no “next_page” returned:
image

I then take all of these JSON files and append in powerbi, but it is a manual process to load them all into PowerBI into a table format.

Hope this helps. Ideally this would just be a solution that:

  1. Runs first query
  2. Saves JSON file
  3. Runs the “Next_page” query
  4. Append this JSON file to previous
  5. Loop until no further “next_page” query returned

Tanks!!!