Home Codingapache Block bad bots

Block bad bots

by Ben

Recently my website has been crawling to a stop with 508 errors.

And then i realised there’s a stupid bot keep crawling my website non-stop

And that stupid bot is called ‘Semrush’ and below is the access log entry - - [05/Dec/2016:16:07:55 +0800] "GET /xyz.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)"

And to block this bot all you have to do is to add the following entries to your .htaccess file

BrowserMatchNoCase Semrush bad_bot

Order Deny,Allow
Deny from env=bad_bot

deny from 46.229.164.
deny from 46.229.168.

The above codes will block both browser agent and as well as IP address

You may also like

Leave a Comment