Bot crawlers are essential algorithms that index your website. However, that applies to only a few of them as some bots cause more harm than good, like server load or network instability. You can block these bots using Apache with WHM/cPanel server.
To Block Bots from Apache
1. Log in to WHM using your root account.
2. Select the Service Configuration option from the navigation menu.
3. Click on Apache Configuration.
The list of configuration options for the Apache server will appear.
4. Click on Include Editor.
Here, you can apply configurations to your Apache server on a global scale.
5. Go to the Pre Main Include section and, under I wish to edit the Pre Main configuration include file for: select All Versions from the dropdown list.
A text box will appear in the Pre Main Include section.
6. Copy/enter this code in there and click the Update button.
<Directory "/example">
SetEnvIfNoCase User-Agent "MJ12bot" bad_bots
SetEnvIfNoCase User-Agent "AhrefsBot" bad_bots
SetEnvIfNoCase User-Agent "SemrushBot" bad_bots
SetEnvIfNoCase User-Agent "Baiduspider" bad_bots
<RequireAll>
Require all granted
Require not env bad_bots
</RequireAll>
</Directory>
Note: In the first line in the Directory section where we set it as /example, you have to specify the directory path to the location of your website. For cPanel servers, it is set as /home by default. Also, you can block more bad bots by adding them to the code with the same syntax.
Here’s a list of commonly termed bad bots:
| Bad_Bots List |
|---|
| MJ12Bot |
| Baiduspider |
| AhrefsBot |
| SEMrushBot |
| DotBot |
| MauiBot |
| YandexBot |
| PetalBot or AspiegelBot |
