Postman request triggering bot defense

🌱 Just Getting Started
1

I’am trying to scrape data from shorturl.at/lBO15. Generally when you send a get request to most of the sites with postman you get the HTML as a response. However in this case I receive the following error

Blockquote

<title></title>

<meta charset="utf-8" />
Вашата заявка беше 
отхвърлена.<br>За повече информация се свържете с Центъра за обслужване на клиенти на Интернешънъл Асет Банк АД на 0800 12 422.<br>Моля съобщете следният номер на заявка:

< 5889068425447287412><br><br><br>

    <title>Request Rejected</title>The requested URL was rejected.For more details, please call +359 800 12

    422.<br>Your support ID is:<%BOTDEFENSE.support_id%><br><br><br><a href='javascript:history.back();'>[Go

        Back]</a></body>

Blockquote
Accessing the site with a browser is working ok. Any idea how can I avoid hitting this bot defense?

2

When I tried a GET to shorturl.at/lBO15 the response was the following…

<title>Parntawan Kindergarten START - YouTube</title>

If you are hitting BOT defenses, I’d say…

  • Make sure your headers look normal.
    • User-Agent (set to a common agent, or depending on the bot defense use $randomUserAgent.
    • Referer should be set to somewhere that would normally link to the site.
    • Your IP address might be getting blocked. If so, then perhaps a VPN service to mask you IP will help.
    • If this is happening during a Postman monitor, then it may mean they block the runners IP address. In the past I’ve blocked “bad blocks of addresses” and accidently taken out some of the good traffic in the process. I’m not sure if there is a way to provide a VPN service to have calls be made through. Another solution may be to proxy it across a translation service, like Google Translate (assuming the have a public API).

Be sure that whatever you are scraping isn’t against the sites Terms and Conditions; perhaps they don’t want you to scrape the page & are stopping you for a reason. Many sites have terms that disallow scraping their pages. Also, perhaps they already expose a public API that would give you the information you seek in a nice format like JSON instead of HTML (which might break when they do a site design)?

3

Tried with different user agent with no success. I don’t think the IP is the problem since I can access the site. Probably there is API but not public.