How can I get more than 100 tweets?

Thanks for tagging me, @meenakshi.dhanani! Certainly some good questions in here! I had some trouble navigating the Twitter API initially myself, so I’m happy to try and jump in here. There are a few questions happening, so I’ll try to hit all of them.

For those asking how to get more than 100 tweets (@cryosat-cosmonaut-15 and @science-candidate-29), as others have mentioned, Twitter limits the number of tweets you can return in one request. It caps it at 100 if you’re on the Standard Product Track. But using the pagination feature and using the next_token, you should be able to get more than the 100, but you’ll need to make multiple requests.

To do this, you’ll have to utilize the Postman method called setNextRequest to build your request workflow. If you’re not familiar with it, here’s some more information.

I’m assuming you’re using the https://api.twitter.com/2/tweets/search/recent? endpoint, called “Recent Search”, to search for recent tweets. If so, to utilize the next_token, here is some sample code you can put in the “Tests” tab of your request:

let jsonResponse = pm.response.json();
let next_token = jsonResponse.meta.next_token;

pm.collectionVariables.set("next_token", next_token);

if(next_token){
    postman.setNextRequest("Recent search")
} else {
    postman.setNextRequest(null);
}

This code above will capture the token needed for the next request and saves it as a collection level variable if one is available. It also sets the next request to repeat the same request if next_token exists, and sets the request to null if there is no value for next_token. That will stop the collection runner from getting stuck in a never ending loop once there are no more older tweets to get.

Your params should look something like this, which include the next_token parameter.

Lastly, if it is the first time running this request, there is no need to have the next_token parameter. To cover this condition, we can add some code in the “Pre-request Script” tab that will remove that query parameter if there is no collection level variable for next_token. Put the following code in the “Pre-request Script” tab to cover that first request in the workflow:

let nextToken = pm.collectionVariables.get("next_token");

if(!nextToken){
    pm.request.removeQueryParams("next_token");
    console.log("removing next_token parameter from query since there is no highest_tweet_id to base the search off of.")
}

Lastly - for @cryosat-cosmonaut-15’s question about viewing and saving all the results at once. You won’t be able to do it in one query. But if you set up a workflow using the guidance above, you should be able to achieve this by doing something like:
Have a collection variable called tweets that is an array. For each time “Recent Search” runs, add the new tweets to the end of the array but don’t delete or overwrite the tweets that already in there. On the last run of the sequence, the array should have all of the tweets in it from every single time “Recent Search” ran.

Hope that helps and happy tweet searching! :bird: :slight_smile:

5 Likes