Allow Vizzybl to index your site

Vizzybl crawls your website to build the content index behind brand defense and content optimization. When our crawler can't read your pages, indexing returns 0 pages and a "needs attention" warning appears in Brand Protection → Content Index. This is Vizzybl's own crawler — not the Google or Vertex AI bot — so allow it explicitly using the details below.

How to tell if Vizzybl is blocked

Open Brand Protection → Content Index. A domain that couldn't be indexed shows one of two states:

• Blocked — we reached your site but requests were refused. The site is likely blocking our crawler at the firewall or WAF.
• No readable content — we reached your pages but found nothing to index. This is usually thin content or a JavaScript-only page.
• Asks crawlers not to index — your robots.txt tells automated crawlers to stay out, so we didn't crawl. Nothing is broken; change the robots rule if you want us to index the site.

Note: A 0-page result is not always a block. An over-broad rule in your robots.txt can hide almost every URL from every crawler — check that section below before assuming a firewall problem.

When it's a competitor's site

You can't edit a competitor's robots.txt or firewall, so the guidance on this page won't help there. When a competitor's site refuses our crawler but its robots.txt does not ask crawlers to stay away, the warning offers Try another way — an alternative fetch route that reaches some sites our own crawler cannot.

It is not guaranteed, and it usually returns fewer pages, so a result from it is labelled Limited coverage. It is available on Pro and above, and it costs the same credits as a normal index (see below).

If the competitor's robots.txt does ask crawlers not to index the site, we don't offer it. That is a stated preference and we respect it.

What indexing costs

Indexing is metered on the pages we actually fetch, not the scan size you pick:

Pages fetched	Credits
1	1
20	10
100	50
1,000	275

The rate is 0.5 credits per page, dropping to 0.25 per page beyond 100. Three things are always free:

• a run that couldn't fetch anything — a blocked or robots-denied index costs nothing;
• pages we skip because your sitemap says they haven't changed since we last indexed them;
• the failed first attempt when you use Try another way — you're charged once, for the pages that were actually retrieved.

Re-indexing is therefore cheap: a weekly re-index of a 100-page site that changed three pages fetches three pages and costs 2 credits, not 50.

Allow Vizzybl by IP (recommended)

Vizzybl crawls from a single, stable egress IP. Allowlisting that IP in your firewall or WAF is the most reliable option, because our crawler falls back to a standard browser User-Agent for compatibility — so User-Agent matching alone may not catch every request.

Add this IP to your allowlist:

35.202.121.22

Important: Allowlist by IP as your primary rule, and add the User-Agent rule below as a second signal.

Using Cloudflare? Follow Allow Vizzybl in Cloudflare, step by step below. It takes about two minutes.

Allow Vizzybl in Cloudflare, step by step

Cloudflare can refuse or silently drop our crawler, even when your site loads normally in a browser. When that happens, indexing times out and returns 0 pages. One custom rule that skips Cloudflare's security checks for our IP fixes it.

1. In the Cloudflare dashboard, open your domain and go to Security → WAF → Custom rules, then click Create rule.

2. Give it a name you'll recognise, for example Allow Vizzybl crawler.

3. Under When incoming requests match…, set:

   • Field: IP Source Address
   • Operator: equals
   • Value: 35.202.121.22

   The expression preview should read (ip.src eq 35.202.121.22). The Total requests matched panel shows whether we've already tried to reach your site.

4. Under Then take action…, choose Skip.

5. Leave Log matching requests on, so you can confirm the rule works.

6. Under WAF components to skip, tick all four:

   • All remaining custom rules
   • All rate limiting rules
   • All managed rules
   • All Super Bot Fight Mode Rules

7. Open More components to skip and tick all seven:

   • Zone Lockdown
   • User Agent Blocking
   • Browser Integrity Check
   • Hotlink Protection
   • Security Level
   • Rate limiting rules (Previous version)
   • Managed rules (Previous version)

8. Click Deploy, then drag the rule to the top of your custom rules list. Skip only bypasses the rules that come after it.

This rule only matches traffic from Vizzybl's crawler IP, so it doesn't weaken protection for anyone else.

Important: Two Cloudflare features can't be bypassed by a Skip rule:

• Bot Fight Mode (the free version, under Security → Bots). If it's on, turn it off, or upgrade to Super Bot Fight Mode, which the rule above does skip.
• IP Access Rules (under Security → WAF → Tools). If you block or challenge Google Cloud or "hosting" networks there, add an IP Access Rule that Allows 35.202.121.22.

Check it worked: open Security → Events, filter by the IP address 35.202.121.22, and re-run your index. Events should show your new rule with the action Skip, and the index should report pages instead of 0.

Allow the VizzyblBot User-Agent

Our crawler identifies itself on its first request to each page as:

Mozilla/5.0 (compatible; VizzyblBot/1.0; +https://vizzybl.ai/bot)

Allow it in two places:

• robots.txt — add an explicit group for our crawler:

User-agent: VizzyblBot
Allow: /

• WAF / firewall — add an allow (or skip-bot-check) rule that matches the User-Agent VizzyblBot. Managed bot-fighting rules often challenge unfamiliar crawlers, so name us explicitly.

Note: User-Agent allowlisting is best-effort — if a site blocks VizzyblBot, our crawler retries as a browser so indexing still works where it can. The IP rule above is the deterministic one.

Check your robots.txt

Vizzybl honors robots.txt on every request, whatever User-Agent we send. An over-broad rule silently removes your content from indexing.

Warning: A rule like Disallow: /*? blocks every URL that contains a query string. If your pages live on query-parameter URLs — common on booking, search, and catalog sites — this hides them from all crawlers. Add an Allow: for the paths Vizzybl should read, or scope the Disallow more narrowly.

When both a wildcard group (User-agent: *) and a User-agent: VizzyblBot group are present, the VizzyblBot group takes precedence for us — so a specific Allow: / for VizzyblBot overrides a broad wildcard Disallow.

Verify it worked

1. Go to Brand Protection → Content Index and click Index Site to re-run.
2. A healthy run reports the number of pages indexed — not 0.
3. In your server or WAF logs, confirm requests from VizzyblBot (and our egress IP) return 200.

Related

• Cloudflare Setup — track which AI crawlers fetch your content
• Audit scores — how AI bot access affects your Technical score