Pagination script for looping through all response pages

[swl-swaldie (swaldie)]

New to postman, and chatgpt appears to have me on the right track, but I’m not quite there.

I’m unable to fetch all available items due to the resource’s per-page limit.

The script suggested for the pre-request is as follows:

// Initialize the environment variables only if they are not set
if (!pm.environment.get("url")) {
    // Set the initial API endpoint
    pm.environment.set("url", pm.request.url.toString());
    // Set an empty list to store all page views
    pm.environment.set("all_page_views", JSON.stringify([]));
}

…with the post-response as follows:

// Parse the response
let responseData = pm.response.json();

// Retrieve all page views from the environment variable or initialize if not present
let allPageViews = pm.environment.get("all_page_views") 
    ? JSON.parse(pm.environment.get("all_page_views")) 
    : [];

// Append the current page's data to the list
allPageViews = allPageViews.concat(responseData);

// Store the updated page views in the environment variable
pm.environment.set("all_page_views", JSON.stringify(allPageViews));

// Check if there's a next page link in the headers
let linkHeader = pm.response.headers.get("Link");
if (linkHeader && linkHeader.includes('rel="next"')) {
    // Extract the URL for the next page
    let nextUrl = linkHeader.match(/<([^>]+)>;\s*rel="next"/);

console.log(nextUrl[1]);
    if (nextUrl) {
        pm.environment.set("url", nextUrl[1]);
        // Trigger the next request using the updated method
        pm.execution.setNextRequest(pm.info.requestName);
    }
} else {
    // If there's no next page, print the total number of page views retrieved
    console.log("All page views retrieved:", JSON.parse(pm.environment.get("all_page_views")));
    
    // Clear the environment variable for the URL (pagination ends)
    pm.environment.unset("url");
}

Adding that console.log line after setting the nextUrl variable displays what I’m guessing would be the correct next link:

However, hitting send only gets me that first page; i.e., there is no looping through subsequent iterations.

Any suggestions would be appreciated.

[danny-dainton]

Hey @swl-swaldie

Welcome to the Postman Community

As that script is using the setNextRequest function to loop through the variables, you’re going to need to use the Collection Runner.

The function doesn’t work when only sending a single request.

[swl-swaldie (swaldie)]

Thank you for the quick response, @danny-dainton .

Since I’m not familiar with using the collection runner, are there any adjustments that can be made to my setup that would enable me to accomplish this with a “single” request?

[danny-dainton]

I’m not in a position to try that code out and I don’t know what the API is you’re using or what it’s returning.

Using the Collection Runner is quite simple and shouldn’t require any adjustments to your script.

Test your API using the Collection Runner | Postman Learning Center

To use it in a single request, you’d need to adjust your script to use pm.sendRequest instead but it would mean a bigger refactor, not huge but as I mentioned, I can’t really help you out with that right now.

[swl-swaldie (swaldie)]

Thanks again, @danny-dainton !
I’ll look into that.

[swl-swaldie (swaldie)]

I read through this documentation, and it sounds like the purpose is to

Send some or all of the API requests in a Postman Collection in the order…

However, I only have this one request, but it needs to be resent for each of the next URLs that are returned in the link header until all of the pages have been exhausted.

Can either the pre-request or post-response scripts be set up to resend this request?

[danny-dainton]

Did you try to run your Collection with the Collection Runner?

Did it not do what you expected it too?

[swl-swaldie (swaldie)]

I did, but it just keeps running the same request; i.e., the “next” url is not being used, and it never stops.
I’m going to add a few more log statements to see where i’m going wrong.

[gbadebo-bello]

Hi @swl-swaldie. Can you share a screenshot of what your script currently looks like?

[swl-swaldie (swaldie)]

Hi Gbadebo,
Here’s the pre:

And here’s the post:

I’m currently trying to figure out a way to get the request to GET the value of
nextUrl[1] after the first iteration, and keep looping until there are no more pages to fetch.

[swl-swaldie (swaldie)]

After exhausting my efforts with chatgpt, I switched to copilot and I finally arrived at a solution using the following scripts:

Pre-request

if (!pm.collectionVariables.get("allResponses")) {
    pm.collectionVariables.set("allResponses", JSON.stringify([]));
}

Post-response

let iterationCount = 0;

function getNextPageUrl(linkHeader) {
    const links = linkHeader.split(',');
    let nextPageUrl = null;

    links.forEach(link => {
        const [url, rel] = link.split(';');
        if (rel.includes('rel="next"')) {
            nextPageUrl = url.trim().slice(1, -1); // Remove angle brackets
        }
    });

    return nextPageUrl;
}

function processResponse(res, url) {
    if (!res || !res.headers) {
        console.error("Invalid response object:", res);
        return;
    }

    iterationCount++;
    let allResponses = JSON.parse(pm.collectionVariables.get("allResponses"));
    try {
        const responseBody = res.json();
        
        // Assuming responseBody is already an array of objects
        allResponses = allResponses.concat(responseBody);
        pm.collectionVariables.set("allResponses", JSON.stringify(allResponses));
    } catch (e) {
        console.error("Failed to parse JSON or extract data:", e);
    }

    const linkHeader = res.headers.get('link');

    if (linkHeader) {
        const nextPageUrl = getNextPageUrl(linkHeader);

        if (nextPageUrl && iterationCount < maxIterations) {
            const authHeader = pm.request.headers.find(h => h.key === 'Authorization');

            pm.sendRequest({
                url: nextPageUrl,
                method: pm.request.method,
                header: {
                    'Authorization': authHeader ? authHeader.value : '',
                    ...pm.request.headers
                },
                body: pm.request.body
            }, function (err, newRes) {
                if (err) {
                    console.error("Error on next request:", err);
                } else {
                    processResponse(newRes, nextPageUrl);
                }
            });
        } else {
            console.log("All pages fetched. You can now export the data from the Postman collection variable.");
        }
    } else {
        console.log("All pages fetched. You can now export the data from the Postman collection variable.");
    }
}

// Initial call to process the first response
processResponse(pm.response, pm.request.url.toString());

The full response body is then captured in the current value of the allResponses collection variable:

And this did not require use of the runner functionality.