> ## Documentation Index > Fetch the complete documentation index at: https://product-discovery.developer.voyado.com/llms.txt > Use this file to discover all available pages before exploring further. # Handling bots and crawlers > Identify, filter, and rate-limit unwanted bot traffic in your Elevate integration while allowing trusted crawlers like Googlebot with an allow list. Unwanted crawlers and bots can cause problems in your Elevate integration. While some crawlers, such as Googlebot, are legitimate and beneficial, others can artificially increase traffic and session counts, leading to: * Increased resource usage * Inflated usage metrics * Skewed analytics and statistics By following these recommendations, you can mitigate the impact of bots and crawlers by identifying, filtering, and restricting unwanted sources. Elevate can exclude certain traffic in client-side integrations, but for server-side integrations we have no visibility into the traffic source and therefore cannot assist with traffic exclusion. ## Allowing trusted bots Some crawlers are necessary for SEO or platform integrations, such as: * Googlebot * Bingbot * LinkedInBot * FacebookExternalHit Maintain an allow list of known user agents that should be permitted to access your site. These bots help with indexing and link previews and generally behave predictably. ## Managing crawler traffic Not all automated traffic is harmful, but different types of crawlers must be handled differently to avoid inflated usage metrics. ### robots.txt Use `robots.txt` to tell compliant crawlers which URLs on your own domain should not be crawled. This helps reduce unnecessary load from pages that would otherwise trigger Elevate API calls, such as internal search or utility URLs. ```bash icon="terminal" theme={null} User-agent: * Disallow: /search Disallow: /internal-endpoint/ ``` The `robots.txt` file only affects cooperative crawlers such as Googlebot. It applies only to pages on your own site, not external domains like `*.elevate-api.cloud`. It does not block bots from executing JavaScript or making API calls if they choose to ignore the file. ### Unwanted or aggressive bots Some crawlers ignore `robots.txt` entirely and may generate large numbers of Elevate sessions. These should be filtered or blocked server-side before any Elevate API calls are made. #### Inspect the User-Agent header Block requests that appear suspicious by checking the reported client identity. Requests with missing, malformed, or clearly fake identifiers should be denied immediately to prevent unauthorized or automated access. #### Rate-limit suspicious traffic Throttle excessive traffic from individual IPs or networks to reduce the risk of abuse, denial-of-service attempts, and automated attacks. Resources: * [Rate limiting AI scrapers with NGINX](https://www.eigenmagic.com/2025/09/30/rate-limiting-ai-scrapers-with-nginx/) * [Rate limiting with NGINX](https://blog.nginx.org/blog/rate-limiting-nginx) #### Monitor traffic sources Regularly review logs for spikes from: * Unknown IP ranges * Unusual geographies * Known hosting providers or bot networks This helps identify bots that bypass user-agent filtering. Resources: * [OWASP Logging Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html) ## Leveraging your CDN or hosting platform Many CDNs and hosting platforms include automatic bot detection that flags "likely bot traffic". Even so, this traffic may still reach your site and Elevate, since it is not always blocked outright, just identified. If your usage numbers indicate this, there are further steps you can take with your provider: * **Request custom blocking rules**, for example by country or network. Traffic from a region that has no relevance to your business is a common candidate for blocking. * **Ask for a challenge instead of a full block.** A challenge (commonly a human-verification popup) is triggered when traffic to a specific endpoint from a specific source crosses a defined threshold, for example, more than 50 requests per minute to a search endpoint. This significantly reduces automated traffic without blocking legitimate users outright. * **Add endpoint-specific rate limits in your own application**, particularly for "expensive" endpoints such as search. This lets you control the rate per IP directly, rather than relying solely on your provider. Keep in mind that this only reduces load on your own servers. The traffic still reaches your site, it does not necessarily prevent it from reaching Elevate. ## Generalized approach to handling automated traffic Identify and explicitly allow well-known, legitimate automated clients that are required for core business functions, such as search engine indexing or integrations. Requests that do not match trusted automated clients or typical human-driven browsers should be treated as potentially automated and flagged for special handling. Instead of blocking these clients outright, reduce the level of functionality and resource usage associated with their requests. This helps limit impact while avoiding unnecessary disruption. Ensure that the client classification is available throughout the request lifecycle so frontend behavior and third-party integrations can adapt accordingly. Turn off analytics, tracking, personalization, or other secondary services for suspected automated traffic to prevent amplification effects and traffic bursts. ## Sequence diagrams ### NGINX example NGINX example for blocking suspicious bots and crawlers using user-agent filtering and rate limiting.

### Application server example Application server example for server-side rendering (SSR) with a session limit for suspected bots and crawlers.