How can I get more than 100 tweets?

Hello to all of you

I’m new to both postman and the Twitter api and I don’t know much about programming (I’m a psychology researcher).
I would like to search for all tweets containing a word between two dates. But the question is that I only manage to download 100. How can I get ALL the tweets.

Here is an example
I want to collect all the tweets written in French from 17 January to 17 January (for example) that contain the word metooinceste.
How should I do it (I have api and researcher credentials and I have managed to do the search and display it but it only downloads 100 tweets and that’s the problem).

Hi @cryosat-cosmonaut-15, welcome to the community!

For twitter apis, there is cap of 100, which means a maximum of 100 results can be fetched at a time. You may need to check in the documentation of API being consumed if there is a possibility of pagination. If yes, then you may need to identify parameters to achieve so. For example, below is the documentation of premium search API.

There is a way to achieve next 100 results by passing additional parameter “next”.

3 Likes

Thank you! I´ve got it! And how can I save all the results at once? Or how can I display them all at once (I know how to display results but in each query only the new ones are displayed)

Hello @cryosat-cosmonaut-15, because of API design the developer has some restrictions and that’s why he have used the pagination. You can’t view all requests at once, you need to go through next :slight_smile:

I still dont get how to fetch more than 100…can you help?

Same here… I still don’t know how to get more than 100 results in recent searches when pulling tweets.

@sean.keegan tagging you here, since you might find it interesting.
Since we’re on Twitter APIs, just wanted to share Sean has a really interested Twitter recipe - https://blog.postman.com/how-to-use-twitter-api-create-hashtag-search-bot/

Thanks for tagging me, @meenakshi.dhanani! Certainly some good questions in here! I had some trouble navigating the Twitter API initially myself, so I’m happy to try and jump in here. There are a few questions happening, so I’ll try to hit all of them.

For those asking how to get more than 100 tweets (@cryosat-cosmonaut-15 and @science-candidate-29), as others have mentioned, Twitter limits the number of tweets you can return in one request. It caps it at 100 if you’re on the Standard Product Track. But using the pagination feature and using the next_token, you should be able to get more than the 100, but you’ll need to make multiple requests.

To do this, you’ll have to utilize the Postman method called setNextRequest to build your request workflow. If you’re not familiar with it, here’s some more information.

I’m assuming you’re using the https://api.twitter.com/2/tweets/search/recent? endpoint, called “Recent Search”, to search for recent tweets. If so, to utilize the next_token, here is some sample code you can put in the “Tests” tab of your request:

let jsonResponse = pm.response.json();
let next_token = jsonResponse.meta.next_token;

pm.collectionVariables.set("next_token", next_token);

if(next_token){
    postman.setNextRequest("Recent search")
} else {
    postman.setNextRequest(null);
}

This code above will capture the token needed for the next request and saves it as a collection level variable if one is available. It also sets the next request to repeat the same request if next_token exists, and sets the request to null if there is no value for next_token. That will stop the collection runner from getting stuck in a never ending loop once there are no more older tweets to get.

Your params should look something like this, which include the next_token parameter.

Lastly, if it is the first time running this request, there is no need to have the next_token parameter. To cover this condition, we can add some code in the “Pre-request Script” tab that will remove that query parameter if there is no collection level variable for next_token. Put the following code in the “Pre-request Script” tab to cover that first request in the workflow:

let nextToken = pm.collectionVariables.get("next_token");

if(!nextToken){
    pm.request.removeQueryParams("next_token");
    console.log("removing next_token parameter from query since there is no highest_tweet_id to base the search off of.")
}

Lastly - for @cryosat-cosmonaut-15’s question about viewing and saving all the results at once. You won’t be able to do it in one query. But if you set up a workflow using the guidance above, you should be able to achieve this by doing something like:
Have a collection variable called tweets that is an array. For each time “Recent Search” runs, add the new tweets to the end of the array but don’t delete or overwrite the tweets that already in there. On the last run of the sequence, the array should have all of the tweets in it from every single time “Recent Search” ran.

Hope that helps and happy tweet searching! :bird: :slight_smile:

3 Likes

I love it @sean.keegan. Super useful :nerd_face:

1 Like