Server-side AI crawler tracking with Cloudflare

Why server-side

GPTBot, ClaudeBot, PerplexityBot and most other crawlers fetch HTML and leave. They show up in your server logs but not in JavaScript analytics. The Worker sees them at the edge and CortIQ classifies each one as a training crawler, agentic fetch or citation crawler, using the same registry as the JavaScript tag.

Detection is based on the user agent. AI agents that run inside a normal browser and send a standard Chrome user agent cannot be identified this way; the JavaScript tag uses browser signals for those.

Setup

1. Make sure your domain is added as a site in CortIQ. The Worker matches requests to sites by hostname; requests for unknown domains are ignored.
2. In Cloudflare, go to Workers & Pages → Create Worker, paste the code from cloudflare-worker.js and deploy it.
3. Under the Worker's Settings → Variables, add CORTIQ_INGEST_URL and CORTIQ_INGEST_KEY with the values we send you.
4. Under Triggers, add a route for your domain, for example example.com/*.

CORTIQ_INGEST_URL = https://cxmkdtgfocgbfizawlwa.supabase.co/functions/v1/cloudflare-ingest

How the Worker behaves

• It passes every request on to your site unchanged and reports to CortIQ in the background, so visitors do not wait for CortIQ.
• It skips static files (CSS, JavaScript, images, fonts, media) and Cloudflare's /cdn-cgi/ paths.
• Each request still runs the Worker, so it counts toward your Cloudflare Workers request quota.

What is stored

The Worker cuts the IP address to its network (/24 for IPv4, /48 for IPv6) before anything leaves Cloudflare. CortIQ never receives full IP addresses from it.

• Bots and automated traffic: path (without query string), user agent, referrer, the truncated network address and Cloudflare's Ray ID.
• Human visitors: counted only. No path, user agent, referrer or address is kept.

This runs without visitor consent because no personal data about human visitors is kept. Raw rows are deleted after 90 days. See the privacy policy.