Block specific browsers/bots in nginx

Blocking Unwanted Bots and User-Agents in Nginx

Nginx allows administrators to block specific User-Agent strings to prevent unwanted bots, crawlers, scrapers, or malicious clients from accessing a site. This is useful for reducing server load, conserving bandwidth, and protecting content from aggressive crawling—such as by AI training bots that may not respect robots.txt.

Common targets include malicious scanners, spam bots, and increasingly AI-related crawlers (e.g., those collecting data for large language models).

Identifying Unwanted Traffic

Use log analysis tools like GoAccess to inspect access logs for high-volume traffic by User-Agent or ASN. Many AI crawlers announce themselves clearly (e.g., "meta-externalagent/1.1", "GPTBot", "ClaudeBot").

Basic Blocking with if Directive

The simplest method is a conditional check inside the server {} block:

if ($http_user_agent ~* "badbot|undesired-agent") {
    return 403;  # or 404, 444, etc.
}

This performs a case-insensitive match (~*) and returns the specified status code.

Example: Blocking Meta's AI Crawler

Many administrators report heavy traffic from Meta's crawler (User-Agent: "meta-externalagent/1.1"), used for AI data collection.

To block it:

if ($http_user_agent ~* "meta-externalagent") {
    return 402;  # "Payment Required" - a humorous nod to content usage in AI training
}

The 402 Payment Required code is rarely used but valid; it implies the crawler should "pay" for data, popular among publishers concerned about AI scraping.

After changes, test configuration and reload:

nginx -t
nginx -s reload

Advanced: Using map for Multiple User-Agents

For cleaner configuration and better performance with many agents, use a map (outside server {}, in http {}):

map $http_user_agent $bad_bot {
    default 0;
    ~*meta-externalagent 1;
    ~*GPTBot              1;
    ~*ClaudeBot           1;
    ~*Bytespider          1;
    # Add more as needed
}

Then in server {}:

if ($bad_bot = 1) {
    return 402;
}

This avoids multiple if statements and is recommended for lists of bad bots.

Alternatives: Rate Limiting or Connection Closing

Instead of full blocking:

• Rate limit specific agents to reduce impact.
• Use return 444; (Nginx-specific) to close the connection without response, saving resources.

Example rate limiting:

map $http_user_agent $bot_limit {
    ~*meta-externalagent 1;
    default              0;
}

limit_req_zone $bot_limit zone=bot_zone:10m rate=5r/m;

server {
    ...
    limit_req zone=bot_zone burst=10 nodelay;
}

Complementary: robots.txt

Many crawlers respect robots.txt. For Meta's agent:

User-agent: meta-externalagent
Disallow: /

However, server-level blocking is more reliable for non-compliant bots.

External links

• Nginx if directive documentation
• GoAccess - Log Analyzer for Identifying Bots
• Dark Visitors - List of AI Bots and User-Agents
• Nginx Ultimate Bad Bot Blocker