Too many bots on your website: how to identify them and keep them in check

Not every "visitor" to your website is a person. A large share of requests comes from software: search engine crawlers (Googlebot, bingbot), SEO tools (AhrefsBot, SemrushBot, MJ12bot), crawlers collecting text for artificial intelligence (GPTBot, ClaudeBot, CCBot, Bytespider) and scripts that copy content. In small numbers you never notice them. In large numbers they eat the account's CPU and memory, and the website gets sluggish for your real customers.

Signs that bots are to blame

• The website slows down at times when few people are browsing, and cPanel under Metrics, then Resource Usage, shows CPU spikes or warnings that limits were reached.
• The access logs (Metrics, then Raw Access, or the Awstats statistics) are full of names such as AhrefsBot, SemrushBot, PetalBot, Bytespider, GPTBot, python-requests or curl.
• For online shops: hundreds or thousands of hits on addresses with ?filter_, ?orderby=, ?min_price=, or on far-off listing pages like /page/37/. No person clicks through that many filter combinations; a bot tries every one.

That is exactly why shops suffer most: every filter and sorting combination is, to a bot, a fresh page that PHP and the database have to build.

What the server already stops

Imunify360 and the server firewall automatically cut off part of the abusive traffic, for instance addresses firing an abnormal number of requests in a very short time. Crawlers from search engines and SEO services get through at server level on purpose, since many clients depend on them. Those are yours to control, using the measures below, from the gentlest to the strictest.

1. robots.txt: a request that well-behaved bots honour

The robots.txt file sits in the website's main folder (usually public_html) and tells bots where not to go. Google, Bing and most SEO tools follow it. Here is a sample for a WooCommerce shop that you can adapt:

# Pages with no search value: cart, account, filters, sorting
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?s=
Disallow: /*?add-to-cart=
Disallow: /*?orderby=
Disallow: /*?filter_
Disallow: /*?min_price=
Disallow: /*?max_price=

# SEO tools and AI crawlers: no access at all
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: MJ12bot
User-agent: DotBot
User-agent: PetalBot
Disallow: /

The first group lets search engines see products and categories while skipping thousands of filter combinations. The second group shuts the listed SEO tools and AI crawlers out completely. Take out anything you use: if you work with Ahrefs or Semrush, keep their crawler allowed.

With Yoast SEO, edit the file under Yoast SEO, then Tools, File editor; with Rank Math, under General Settings, then Edit robots.txt. Never block /wp-content/, CSS, JavaScript or images, because Google needs them to understand the page.

2. Caching: a bot visit that costs next to nothing

A page served from cache does not start PHP or query the database, so it hardly matters whether a person or a bot asked for it. On WordPress Hosting and Business Hosting plans, which run on LiteSpeed, use the LiteSpeed Cache plugin and set it up following Turning on LiteSpeed Cache for WordPress. For shops, two settings make the difference:

• do not exclude category pages from the cache, since that is where most bots end up;
• in the plugin's Cache section, under Drop Query String, list the tracking parameters (utm_*, fbclid, gclid), which stops every campaign link from counting as a separate page.

On Web Hosting SSD plans, which use Nginx and Apache, pick a caching plugin that works with Apache; the idea is the same.

3. .htaccess: turning away bots that ignore the rules

Content scrapers and aggressive bots do not read robots.txt. The server can refuse them in .htaccess before WordPress even starts, which costs no CPU.

1. In cPanel, open File Manager, go to public_html and turn on hidden files (Settings, then Show Hidden Files).
2. Download a copy of .htaccess first.
3. Paste the block below at the top of the file, above the # BEGIN WordPress line. Placed there, WordPress will not overwrite it when permalinks are saved.

# Refused bots
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|AhrefsBot|SemrushBot|MJ12bot|DotBot|PetalBot|BLEXBot|DataForSeoBot|MegaIndex|Barkrowler|Scrapy|python-requests|Go-http-client) [NC]
RewriteRule .* - [F,L]
</IfModule>

Bots on the list now get "403 Forbidden". Tailor the list: remove what you use and add names you spotted in the logs. Never put Googlebot, bingbot or facebookexternalhit (which builds Facebook link previews) in it.

To test the rule, run curl -I -A "AhrefsBot" https://your-domain.md/ from your computer: the answer should be 403. If the website shows error 500 after saving, upload your copy of the file again.

The catch: a bot that pretends to be an ordinary browser cannot be recognised by name. For that kind, Cloudflare and IP blocking on our side are what remain.

4. Cloudflare: rules for traffic you cannot filter any other way

If the website runs through Cloudflare (connection steps in the Cloudflare guide), even the free plan offers some handy tools:

• Bot Fight Mode, in the bot security settings, which slows down crawlers known to be abusive;
• Rate limiting rules, for example on filter URLs, when one IP address goes over a set number of requests per minute;
• Custom rules that shut out unwanted crawlers based on their User-Agent.

5. From inside WordPress, when .htaccess is off limits

• Wordfence (free version): in the firewall settings, under Rate Limiting, cap crawler page views (for example 60 per minute) and crawler requests for missing pages (for example 30 per minute), set to throttle or block. Also switch on the option that immediately blocks bots falsely claiming to be Google.
• LiteSpeed Cache: leave the Crawler feature off unless you need it; what matters is that category pages are cached.
• Do not run two security plugins side by side. They get in each other's way and slow the website down instead of protecting it.

When to contact us

If you have taken the steps above and the website is still slow, open a ticket that names the domain and, where possible, the bots you saw in the logs. We check the server logs, can block the abusive addresses and will tell you whether your real traffic calls for a plan with more resources. Do not let it drag on: under the Terms and Conditions, an account that overuses resources and threatens the server's stability can be suspended. For more on what drives load, read why website scripts load the server.

Pitfalls to avoid

• Do not block Googlebot or bingbot if you want to stay in search results. Google Search Console no longer offers a manual crawl-rate setting: Google slows down by itself when the server often answers with 500, 503 or 429 errors. If Googlebot is genuinely overloading the website, Google provides a form for reporting it.
• Do not edit .htaccess without a copy. A single syntax error takes the whole website down with error 500. For correct versions of the file, see .htaccess for popular CMSs.
• Do not rely on robots.txt alone. It is a request, not a wall, and the bots causing the most trouble are precisely the ones that never read it.

Related articles

• Resource Limit Is Reached (error 508): why it happens and how to clear it
• Turning on LiteSpeed Cache for WordPress
• Using Cloudflare with TopHost hosting: step-by-step connection and the right settings
• A CAPTCHA page instead of your website: why it shows up and what to do