Recently my website has been crawling to a stop with 508 errors.
And then i realised there’s a stupid bot keep crawling my website non-stop
And that stupid bot is called ‘Semrush’ and below is the access log entry
46.229.168.68 - - [05/Dec/2016:16:07:55 +0800] "GET /xyz.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)"
And to block this bot all you have to do is to add the following entries to your .htaccess file
BrowserMatchNoCase Semrush bad_bot
Order Deny,Allow
Deny from env=bad_bot
deny from 46.229.164.
deny from 46.229.168.
The above codes will block both browser agent and as well as IP address