How to get responses of multiple pages from an API that has offset instead of page numbers

Ask the Experts and Postman Tips
1

I am trying to get data from a Discord API. The page shows multiple pages, however, there is no page number attribute. Instead, in the header, there is an offset value of 25.

{ “key”: “offset”,
“value”: “25”,
“description”: “” }

So the URL changes from:

https://discord.com/api/v9/guilds/663869392152559648/messages/search?content=upset

to

https://discord.com/api/v9/guilds/663869392152559648/messages/search?content=upset&offset=25

and multiples of 25 for every subsequent page…

I have got the response to the first URL with the following code (Many thanks to Valentine Despa!!! Very grateful for his tutorials ) :

const newman = require('newman'); // require newman in your project
const fs = require('fs');
newman.run({
    collection: require('./Discorddata.json'),
    reporters: 'cli'
}).on('request', (error, data) => {
if (error) {
    console.log(error);
    return;
}
const requestName = data.item.name;
const fileName = `response-${requestName}.txt`;
const content = data.response.stream.toString();
fs.writeFile(fileName, content, function (error) {
    if (error) { 
         console.error(error); 
    }
 });
 });

I do not know how to use the offset and value to get all the pages. I know I should probably use setNextRequest() but I cannot find anything to help me understand how I can use Newman and setNextRequest.
And even in setNextRequest() how should I specify the request?

Extremely sorry if this is downright dumb. I am very new to the whole web dev and backend stuff and POSTMAN above all!! I need to collect Discord data for my ML project so I am trying to learn how all of this works.

Thank you very much!

2

Hi, the post here https://community.postman.com/t/edit-the-page-number-in-the-headers-and-loop-it-till-the-last-page-postman/6871 specifically mentions offset. I hope this helps

3

Hi @dburns74 ! Thank you very much for your reply. I had gone through that post as well. The problem was that there was no information on the page number or in my case “offset” in the request header. According to this post POSTMAN is not designed to do this. I eventually retrieved the data using nodejs. Thank you all the same for your help!

4

I was (am) having a similar issue, with just a page # being the issue and having it dynamically set.

I managed to get it working in a collection by doing the following:

I set an environment variable for page

I used the following as a Pre-Request Script:

   const page = parseInt(pm.variables.get("page"));
if(!page){
    pm.variables.set("page", 1);
}

I then used the following in my Tests to set the page variable:

tests["Got users"] = responseCode.code == 200;
let page = parseInt(pm.variables.get("page"));
let jsonData = JSON.parse(responseBody);
if (jsonData.data.has_more_data) { 
    console.log("More items available");
    postman.setEnvironmentVariable("page", page+1);
    postman.setNextRequest('Get User List');
} else {
    console.log("No more items available");
}

This allowed me to retrieve all pages for my request. I know have to figure out how to save all the responses to one (or several) files, which I understand that Postman is not good at, but I am trying anyways.

I hope this helps.

1 Like