Block bad bots
Recently my website has been crawling to a stop with 508 errors.
And then i realised there’s a stupid bot keep crawling my website non-stop
And that stupid bot is called ‘Semrush’ and below is the access log entry
46.229.168.68 - - [05/Dec/2016:16:07:55 +0800] "GET /xyz.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)"
And to block this bot all you have to do is to add the following entries to your .htaccess file
BrowserMatchNoCase Semrush bad_bot Order Deny,Allow Deny from env=bad_bot deny from 46.229.164. deny from 46.229.168.
The above codes will block both browser agent and as well as IP address
Similar Posts:
- Migrating Symfony1 to Symfony2: Weekend 1
- Premature end of script headers in perl script
- Spry with HTML codes/CDATA in XML
- Symfony 1 to Symfony 2
- Wake-On-Lan with Dlink DIR 655 over the internet
