Recently my website has been crawling to a stop with 508 errors.
And then i realised there’s a stupid bot keep crawling my website non-stop
And that stupid bot is called ‘Semrush’ and below is the access log entry
18.104.22.168 - - [05/Dec/2016:16:07:55 +0800] "GET /xyz.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)"
And to block this bot all you have to do is to add the following entries to your .htaccess file
BrowserMatchNoCase Semrush bad_bot Order Deny,Allow Deny from env=bad_bot deny from 46.229.164. deny from 46.229.168.
The above codes will block both browser agent and as well as IP address