How can I get more than 100 tweets?

Hello to all of you

I’m new to both postman and the Twitter api and I don’t know much about programming (I’m a psychology researcher).
I would like to search for all tweets containing a word between two dates. But the question is that I only manage to download 100. How can I get ALL the tweets.

Here is an example
I want to collect all the tweets written in French from 17 January to 17 January (for example) that contain the word metooinceste.
How should I do it (I have api and researcher credentials and I have managed to do the search and display it but it only downloads 100 tweets and that’s the problem).

Hi @cryosat-cosmonaut-15, welcome to the community!

For twitter apis, there is cap of 100, which means a maximum of 100 results can be fetched at a time. You may need to check in the documentation of API being consumed if there is a possibility of pagination. If yes, then you may need to identify parameters to achieve so. For example, below is the documentation of premium search API.

There is a way to achieve next 100 results by passing additional parameter “next”.

4 Likes

Thank you! I´ve got it! And how can I save all the results at once? Or how can I display them all at once (I know how to display results but in each query only the new ones are displayed)

Hello @cryosat-cosmonaut-15, because of API design the developer has some restrictions and that’s why he have used the pagination. You can’t view all requests at once, you need to go through next :slight_smile:

I still dont get how to fetch more than 100…can you help?

Same here… I still don’t know how to get more than 100 results in recent searches when pulling tweets.

@sean.keegan tagging you here, since you might find it interesting.
Since we’re on Twitter APIs, just wanted to share Sean has a really interested Twitter recipe - https://blog.postman.com/how-to-use-twitter-api-create-hashtag-search-bot/

Thanks for tagging me, @meenakshi.dhanani! Certainly some good questions in here! I had some trouble navigating the Twitter API initially myself, so I’m happy to try and jump in here. There are a few questions happening, so I’ll try to hit all of them.

For those asking how to get more than 100 tweets (@cryosat-cosmonaut-15 and @science-candidate-29), as others have mentioned, Twitter limits the number of tweets you can return in one request. It caps it at 100 if you’re on the Standard Product Track. But using the pagination feature and using the next_token, you should be able to get more than the 100, but you’ll need to make multiple requests.

To do this, you’ll have to utilize the Postman method called setNextRequest to build your request workflow. If you’re not familiar with it, here’s some more information.

I’m assuming you’re using the https://api.twitter.com/2/tweets/search/recent? endpoint, called “Recent Search”, to search for recent tweets. If so, to utilize the next_token, here is some sample code you can put in the “Tests” tab of your request:

let jsonResponse = pm.response.json();
let next_token = jsonResponse.meta.next_token;

pm.collectionVariables.set("next_token", next_token);

if(next_token){
    postman.setNextRequest("Recent search")
} else {
    postman.setNextRequest(null);
}

This code above will capture the token needed for the next request and saves it as a collection level variable if one is available. It also sets the next request to repeat the same request if next_token exists, and sets the request to null if there is no value for next_token. That will stop the collection runner from getting stuck in a never ending loop once there are no more older tweets to get.

Your params should look something like this, which include the next_token parameter.

Lastly, if it is the first time running this request, there is no need to have the next_token parameter. To cover this condition, we can add some code in the “Pre-request Script” tab that will remove that query parameter if there is no collection level variable for next_token. Put the following code in the “Pre-request Script” tab to cover that first request in the workflow:

let nextToken = pm.collectionVariables.get("next_token");

if(!nextToken){
    pm.request.removeQueryParams("next_token");
    console.log("removing next_token parameter from query since there is no highest_tweet_id to base the search off of.")
}

Lastly - for @cryosat-cosmonaut-15’s question about viewing and saving all the results at once. You won’t be able to do it in one query. But if you set up a workflow using the guidance above, you should be able to achieve this by doing something like:
Have a collection variable called tweets that is an array. For each time “Recent Search” runs, add the new tweets to the end of the array but don’t delete or overwrite the tweets that already in there. On the last run of the sequence, the array should have all of the tweets in it from every single time “Recent Search” ran.

Hope that helps and happy tweet searching! :bird: :slight_smile:

5 Likes

I love it @sean.keegan. Super useful :nerd_face:

1 Like

Hi Sean, please help me out I’m really desperate!

I copied your codes into the respective tabs in postman.
I only modified this part

where I exchanged “Recent search” With “Full-archive search” because I am doing the full-archive search.

However, nothing happens…
When I click on the send button, the system is just handling one request and stops even though I receive a next_token.
I am using the academic API v2 by the way.

It would mean everything to me if you help me out!

1 Like

Hi @altimetry-specialis2! Hopefully I can help.

It sounds like you’re close if you’re getting the next_token! However, if you’re only pressing the blue send button, Postman will run that specific request and no others. If you’ve copied the code from above, the logic is there to determine the next request, but only that single request is made if you click the blue Send button.

Since you’re trying to run a series of requests in a row based on some conditional logic, you’ll want to use the Postman Collection Runner. Once you select your collection and put it in the runner, you can run sets of requests, pass data between requests, and alter the request workflow. Try out the Collection Runner and see if that gets you what you’re looking for.

Hope that helps!

Hello!
I am having trouble with the collection runner. I am using “Write Responses to File” collection, and I used the code for the test section of my request but as the collection runs, it says “This request does not have any tests”. Thus, I have to put in an iteration number instead of it running automatically until there is no next_token left. The test I am using is:

let jsonResponse = pm.response.json();
let next_token = jsonResponse.meta.next_token;

pm.test("loop", function () {pm.collectionVariables.set("next_token", next_token)}) ;

if(next_token){
    postman.setNextRequest("Recent search")
} else {
    postman.setNextRequest(null);
}

I don’t know whats wrong. I would really appreciate it if anyone can help.

1 Like

Hi Sean. Thanks for your help. I was able to follow your steps and used a collection runner to pull 41,000 tweets. Now, when I try to extract the tweets, I am only getting some of them as a json file. You mentioned setting up a collection variable called tweets as an array to view and save the results at once. What are the actual steps to do this and save all the results as one json file?

I really appreciate your help.

I am facing the same problem. collection runs report me “This request does not have any tests” for all the 9 iterations.

Hi @cindyita and @dosbolsotsialkhan. This question has come up a few times, so I created a collection that should be easier to fork and use. It has all of the information on getting started in the documentation.

Please take a look at this collection: Get more than 100 tweets.

Hope that helps!

2 Likes

Thank you very much, @sean.keegan ! I forked this collection. However, I am still facing other issues:
1)How can I export the responses from JSON to csv file? Does it need me to install GitHub? or it can be done with Postman?
2) Is it possible to make full-archive request paginations from one certain twitter user name/user id? (eg. from a certain company’s official twitter account).
3) If I have a list of 1500 company’s twitter user names/user id, how can I automate the step (2) for 1500 times?

In short, my purpose is to collect finance related information from 1500 company’s tweets since they start their twitter accounts. Eg. query “from:apple revenue profit -is:retweet -is:reply -is:quote” and I also set other parameters like time start&end, but I wanna automate this process for 1500 times (each company has several 500 full-archive search paginations depends on the number of their tweets), and report the responses as 1500 seperate data frames in csv format.

Thank you very much!

Hi @dosbolsotsialkhan,

There are a few things you can do to try and achieve what you’re looking for. To answer your questions in order:

  1. You’re not currently able to export the response data, in that format, from the features within the app. But there’s more info on some options in this post here and this post here.

  2. If you want to get all the tweets from a specific Twitter user, you’ll need to know the user’s ID and then you’ll want to hit the Twitter API endpoint that allows you to search for tweets from a user ID. According to the Twitter API documentation here, you can get the most recent 3,200 tweets. Similar to the code for getting all the Tweets for a certain hashtag or search query that I linked above, you would just have to tweak it a bit to get all the Tweets from a specific user. You can fork the entire Twitter v2 collection for your convenience here.

  3. This would be a great place to make use of the Collection Runner and conditional logic in your Pre Request or Tests tabs like we used in the previous collection.

Hope that helps and good luck!

HI @sean.keegan,

Thanks for your collection. I’m just playing around with it, but I’m getting this message when I try to run it:
“TypeError: pm.response.json().meta is undefined”

I seem to be able to send the actual request “get all tweets” but when I try to run the collection allowing pagination and next token, it gives the above error. I wondered if you might be able to help please?

Thanks
Mark

Hey Sean, you’re an absolute legend! Your life-saving work on this fork is worth a standing ovation.I have a small issue though. I successfully ran my query—only to flash that dazzling 200 OK status code. But guess what? The data it generated seems to be playing hide-and-seek with me. I’m having trouble finding the resulting data. It would be incredibly helpful if I could obtain the JSON output directly in Postman. Unfortunately, my attempts to translate the code or find a solution have been unsuccessful so far. Because, let’s be honest, anything that’s not Python makes me feel like I’m deciphering ancient hieroglyphics. Help me out, wise man! May the humor be with you as you save the day. Thanks a million!

1 Like

Hi @sean.keegan! Thank you SO much for your helpful “Twitter more than 100 tweets” collection in Postman. It seems to have at least mostly successfully ran for me, but as the comments above mentioned, I can’t see any results (though my Twitter Developer portal shows that >500 tweets have been collected since I started trying it). How do I go about exporting the results? I’d be grateful for any guidance that you can give. Thanks!