Twitter rate limits

Hi I have searched the search and not found out how to approach this one.

The question is not how to do it but how best to approach downloading all followers and following of a twitter account?

I want to be able to download all of the followers details of a group of accounts on Twitter, collectively there are about 200,000-250,000 to analyse for connections as part of something I am doing for uni.

Its a social psychology project that will then do a lexicographical analysis (afterwards) on those people who have the most within group connections.

It will also ask questions before and after with some users aiming to understand drivers of attitudinal change over time dependent upon exposure. Of course it isn’t possible to see whether people actually read their tweets, I don’t read a lot of mine but there is also the effect of leadership, that is part of the downstream analysis. The key is understanding the most connected people within group.

The problem is the Twitter rate limitations. I haven’t had a problem setting up the OAuth2.0 on my extended Twitter Development AC in the Postman app to download the followers of any one follower ID but then it comes back with only a few of them (understandably) based on the Twitter rate limits system.

Within their documentation they suggest that it is possible to download 15 per 15 minutes so that is one per minute, 1440 a day, which is SO LOW ----- BUT then half a million a year, THAT IS MASSIVE, that isn’t so bad at all with a good plan.

The problem is the plan! LOL. I can set something in Python but can’t leave my MacBook running all the time. I could do it for a week but it would need to be online sitting around for about 3 months. That time isn’t a problem but its not practical to manage on a single machine.

I get how Postman works but is there a common strategy for this you can help me with please :slight_smile:

Thanks for your help!!

R

Hi @s4rob

Off the top of my head, I’d suggest looking at monitors.

Build your API call and then set a monitor to trigger it every 15 minutes, and build a loop into the call for 15 iterations. And some code to save the response / write it to a file (there are videos of that on YouTube).

As far as I understand, by doing this you shouldn’t need to leave your computer running and it would keep you within the Twitter rate limits.