Crawlability & technical signals
Files, crawler access, rendering and speed: what AI systems actually need to reach your pages.
“You need an llms.txt file to appear in AI answers.” — Debunked
Reality: llms.txt is a community proposal. No major AI search provider has documented it as a requirement or ranking input. Pages without it are cited every day.
What to do: Add one if it is cheap — it is a tidy summary for agents and developers — but prioritise crawlable, substantive pages first.
Source: llms.txt proposal
“Blocking GPTBot removes you from ChatGPT search results.” — Nuanced
Reality: OpenAI documents separate agents: GPTBot (training), OAI-SearchBot (search) and ChatGPT-User (user-triggered fetches). Blocking one does not block the others.
What to do: Decide per crawler. Many businesses allow search and user agents while choosing a training policy separately.
Source: OpenAI — Overview of OpenAI crawlers
“Blocking Google-Extended removes you from Google AI Overviews.” — Debunked
Reality: Google states Google-Extended controls use of content for Gemini model training and grounding in certain products, not inclusion in Search or AI Overviews, which follow Googlebot.
What to do: Keep Googlebot allowed if you want Search visibility; treat Google-Extended as a separate data-use decision.
Source: Google — Overview of Google crawlers (incl. Google-Extended)
“AI crawlers render JavaScript like Googlebot does.” — Debunked
Reality: Googlebot renders JavaScript, but many AI fetchers retrieve raw HTML only. A client-only app can look empty to them.
What to do: Serve meaningful HTML: server rendering, prerendering or static summaries of key pages.
Source: Google Search Central — JavaScript SEO basics
“Single-page apps cannot rank at all.” — Debunked
Reality: Google can index JavaScript sites; the risk is delay, rendering errors and invisibility to non-rendering bots, not an absolute ban.
What to do: Test pages with URL Inspection and a no-JS fetch; prerender the pages that matter commercially.
Source: Google Search Central — JavaScript SEO basics
“There is a special robots.txt directive that boosts AI citations.” — Debunked
Reality: robots.txt only controls crawling permissions. It cannot request preference, priority or citation.
What to do: Use robots.txt to allow the crawlers you want and block private areas — nothing more.
Source: Google Search Central — Introduction to robots.txt
“Robots.txt noindex hides pages from search.” — Debunked
Reality: Google does not support noindex in robots.txt. A blocked URL can still be indexed from links, without its content.
What to do: Use a noindex meta tag or header on a crawlable page to keep it out of the index.
Source: Google Search Central — Introduction to robots.txt
“Submitting a sitemap guarantees indexing.” — Debunked
Reality: Sitemaps aid discovery. Engines still choose what to crawl and index based on quality and demand.
What to do: Keep the sitemap clean, then strengthen internal links and content on unindexed pages.
Source: Bing Webmaster Guidelines
“IndexNow makes Google index pages faster.” — Debunked
Reality: IndexNow is supported by Bing, Yandex and others. Google does not participate.
What to do: Use IndexNow for Bing (which also powers Copilot grounding) and Search Console for Google.
Source: IndexNow documentation
“Page speed is a major factor in being cited by AI.” — Nuanced
Reality: Speed is a modest classic-search signal, and very slow or timing-out pages may fail to be fetched. There is no evidence of a strong citation reward for speed alone.
What to do: Fix timeouts and severe slowness; do not chase perfect scores at the cost of content work.
Source: Google Search Central — Creating helpful, people-first content
Structured data & schema
What markup does and does not do for AI answers and classic search features.
“Schema markup makes ChatGPT cite you.” — Debunked
Reality: No major LLM provider documents schema as a citation trigger. Generation works from retrieved text; markup may help indexing systems understand entities.
What to do: Use accurate schema for search features and entity clarity, not as a citation lever.
Source: Google Search Central — Introduction to structured data
“FAQ schema still produces rich results for most sites.” — Debunked
Reality: Google limited FAQ rich results to well-known authoritative government and health sites in 2023.
What to do: Write visible FAQs for users; do not expect FAQ snippets in Google for a typical business.
Source: Google Search Central — Introduction to structured data
“There is special schema required for AI Overviews.” — Debunked
Reality: Google states no special markup or optimisation is required for AI features beyond normal Search eligibility.
What to do: Focus on indexable, helpful pages that meet Search Essentials.
Source: Google Search Central — AI features and your website
“Structured data is useless for AI search.” — Nuanced
Reality: It is not a direct citation lever, but it helps search systems understand organisations, products and authors, which feeds knowledge systems that AI products may rely on.
What to do: Mark up Organization, Product, Article and LocalBusiness accurately where they genuinely apply.
Source: Google Search Central — Introduction to structured data
“More schema types on a page always helps.” — Debunked
Reality: Markup must describe visible content. Irrelevant or misleading markup can trigger manual actions.
What to do: Mark up only what the page really shows; validate with the Rich Results Test.
Source: Google Search Central — Spam policies
“JSON-LD is preferred over microdata by Google.” — Verified
Reality: Google recommends JSON-LD as the easiest to implement and maintain, while supporting other formats.
What to do: Use JSON-LD in new builds.
Source: Google Search Central — Introduction to structured data
“Schema can replace missing on-page content.” — Debunked
Reality: Structured data describes page content; it does not substitute for it. Hidden or unsupported claims violate guidelines.
What to do: Put facts in visible text first, then mark them up.
Source: Google Search Central — Introduction to structured data
“sameAs links help connect your brand entity.” — Verified
Reality: sameAs pointing to official profiles is a standard way to disambiguate an organisation in structured data.
What to do: Link only profiles you control and keep names consistent across them.
Source: Google Search Central — Introduction to structured data
“Review schema on your own homepage creates star ratings.” — Debunked
Reality: Google does not show self-serving review stars for LocalBusiness and Organization pages that host reviews about themselves.
What to do: Collect reviews on independent platforms; use Product review markup only where eligible.
Source: Google Search Central — Introduction to structured data
Content format, freshness & volume
Which content qualities make a page usable as an answer source, and which volume habits waste money.
“Publishing more articles is the fastest path to AI visibility.” — Debunked
Reality: Mass-produced, low-value pages risk scaled-content spam policies and add little that retrieval systems prefer.
What to do: Publish fewer, better pages that answer real buyer questions with specifics.
Source: Google Search Central — Spam policies
“AI-written content is penalised by Google.” — Debunked
Reality: Google evaluates quality and intent, not authorship method. Scaled low-value content is penalised however it is produced.
What to do: Use AI as a drafting aid with expert editing, original detail and fact-checking.
Source: Google Search Central — Creating helpful, people-first content
“Changing the date on a page makes it “fresh”.” — Debunked
Reality: Updating dates without substantive changes is called out by Google as not helpful and can mislead users.
What to do: Update dates only when content materially changes.
Source: Google Search Central — Creating helpful, people-first content
“Adding statistics and citations can increase visibility in generative answers.” — Nuanced
Reality: The GEO study found quotations, statistics and source citations improved visibility in its test setup. Results varied by domain and engine and are not universal guarantees.
What to do: Add genuine, sourced data where it helps readers — never invented figures.
Source: Aggarwal et al. — GEO: Generative Engine Optimization (arXiv 2311.09735)
“Keyword stuffing works on LLMs because they read words.” — Debunked
Reality: The GEO study found keyword stuffing performed poorly, and it remains a spam policy violation in search.
What to do: Write naturally; cover the concepts and questions buyers ask.
Source: Aggarwal et al. — GEO: Generative Engine Optimization (arXiv 2311.09735)
“Long-form content always beats short content.” — Debunked
Reality: Length is not a ranking or citation factor in itself. Completeness for the query matters.
What to do: Make pages as long as the answer requires and no longer.
Source: Google Search Central — Creating helpful, people-first content
“Clear headings and direct answers make content easier to reuse.” — Verified
Reality: Retrieval systems split pages into passages. Self-contained, well-labelled sections are easier to match and quote accurately.
What to do: Lead sections with a direct answer, then supporting detail.
Source: Lewis et al. — Retrieval-Augmented Generation (arXiv 2005.11401)
“You must write in Q&A format for AEO.” — Nuanced
Reality: Questions as headings can help match queries, but forcing every page into Q&A harms readability.
What to do: Use questions where buyers genuinely ask them; use normal structure elsewhere.
Source: Google Search Central — Creating helpful, people-first content
“Hidden text for bots improves AI understanding.” — Debunked
Reality: Showing bots different content from users is cloaking and violates spam policies.
What to do: Make machine-readable content visible or available to users too (for example a plain-text edition).
Source: Google Search Central — Spam policies
“Original first-hand experience matters for quality.” — Verified
Reality: Google’s helpful-content guidance explicitly asks whether content shows first-hand experience and expertise.
What to do: Include real process detail, photos, examples and named authors.
Source: Google Search Central — Creating helpful, people-first content
“AI answers only quote pages ranking in the top 3.” — Debunked
Reality: AI features often draw from a wider set of sources than the top organic results, including via query fan-out.
What to do: Cover sub-questions thoroughly; a page can be cited without ranking first.
Source: Google Search Central — AI features and your website
“Translating pages automatically creates new visibility markets.” — Nuanced
Reality: Localised content can reach new audiences, but unreviewed machine translation at scale risks low quality.
What to do: Translate priority pages with human review and proper hreflang.
Source: Google Search Central — Creating helpful, people-first content
Links, mentions & off-site signals
How backlinks, brand mentions, reviews and third-party corroboration relate to AI visibility.
“Backlinks no longer matter in the AI era.” — Debunked
Reality: Links still help discovery and authority in classic search, which many AI systems use for retrieval.
What to do: Earn links from relevant, reputable sites through useful assets and partnerships.
Source: Google Search Central — Spam policies
“Unlinked brand mentions can support AI visibility.” — Nuanced
Reality: LLMs learn from and retrieve text where your brand is discussed, link or not. The effect is plausible but hard to measure directly.
What to do: Pursue genuine mentions in industry media, podcasts and communities.
Source: Aggarwal et al. — GEO: Generative Engine Optimization (arXiv 2311.09735)
“Buying links is a safe shortcut.” — Debunked
Reality: Link schemes violate spam policies and can lead to manual actions or ignored links.
What to do: Invest in digital PR and assets worth citing instead.
Source: Google Search Central — Spam policies
“Reddit and forum discussions influence AI answers.” — Nuanced
Reality: Some AI products frequently surface community content, but presence and impact vary by product and topic.
What to do: Participate honestly where your buyers already discuss problems; never astroturf.
Source: Google Search Central — Spam policies
“Press releases alone build authority.” — Debunked
Reality: Syndicated releases are usually nofollow and duplicated. Value comes from journalists who pick up the story.
What to do: Use releases for genuine news and pitch editors directly.
Source: Google Search Central — Spam policies
“Third-party reviews help buyers and machines trust a business.” — Verified
Reality: Independent reviews provide corroborating evidence that people and systems can weigh.
What to do: Ask satisfied clients for reviews on reputable platforms.
Source: Google Search Central — Creating helpful, people-first content
“Guest posting on any site helps.” — Debunked
Reality: Large-scale guest posting with keyword links is treated as a link scheme.
What to do: Contribute occasionally to relevant publications for audience, not anchors.
Source: Google Search Central — Spam policies
“Being listed in industry directories is worthless.” — Nuanced
Reality: Spammy directories add nothing; reputable, curated industry lists provide corroboration.
What to do: Choose a handful of respected directories relevant to your sector.
Source: Bing Webmaster Guidelines
“Internal links help engines find and understand pages.” — Verified
Reality: Internal links are a primary discovery path and signal relative importance.
What to do: Link from strong pages to new or unindexed ones with descriptive text.
Source: Bing Webmaster Guidelines
Brand authority & entities
Entity clarity, consistency and the evidence that makes a business recognisable to machines.
“You need a Wikipedia page to be cited by AI.” — Debunked
Reality: Wikipedia helps notable entities, but countless businesses are cited without one, and promotional pages get deleted.
What to do: Build verifiable third-party coverage first; notability follows.
Source: Google Search Central — Creating helpful, people-first content
“Consistent business information across the web helps entity understanding.” — Verified
Reality: Conflicting names, addresses or descriptions make it harder for systems to resolve which entity is meant.
What to do: Standardise name, description, contact details and profiles.
Source: Bing Webmaster Guidelines
“A clear “about” page and named authors support trust.” — Verified
Reality: Google’s guidance asks whether it is clear who created content and why they are qualified.
What to do: Publish real author bios, credentials and company background.
Source: Google Search Central — Creating helpful, people-first content
“E-E-A-T is a direct ranking score.” — Debunked
Reality: E-E-A-T is a quality-rater framework, not a single algorithmic score.
What to do: Demonstrate experience and trust in content; do not chase a mythical score.
Source: Google Search Central — Creating helpful, people-first content
“Small businesses cannot compete with big brands in AI answers.” — Nuanced
Reality: Big brands have advantages on broad queries, but niche, local and specific questions are often answered from specialist sources.
What to do: Own a narrow topic deeply.
Source: Google Search Central — AI features and your website
“Founder visibility can strengthen a brand entity.” — Nuanced
Reality: Named experts with consistent profiles and published work create more corroborating signals, though impact is hard to isolate.
What to do: Link founder profiles, talks and writing to the company site.
Source: Google Search Central — Creating helpful, people-first content
“Claiming “#1 agency” in copy helps AI recommend you.” — Debunked
Reality: Unsupported superlatives are not evidence; systems weigh corroboration, and false claims create legal risk.
What to do: Use verifiable proof: case studies, numbers you can back, client names with permission.
Source: Google Search Central — Creating helpful, people-first content
“Case studies are among the strongest commercial trust assets.” — Verified
Reality: Specific, verifiable outcomes give both buyers and machines concrete evidence to reference.
What to do: Publish case studies with context, process and honest results.
Source: Google Search Central — Creating helpful, people-first content
“Changing your brand name often has no visibility cost.” — Debunked
Reality: Rebrands split mentions and entity signals until the web catches up.
What to do: Redirect properly, update profiles and announce clearly.
Source: Bing Webmaster Guidelines
Measurement, prompts & tools
Tracking AI visibility honestly, and the limits of prompt tricks and “ranking” claims.
“You can “rank #1 in ChatGPT”.” — Debunked
Reality: Generative answers vary by prompt, user, location and time. There is no stable ranked list to be first in.
What to do: Measure share of mentions across a fixed prompt set over time.
Source: Aggarwal et al. — GEO: Generative Engine Optimization (arXiv 2311.09735)
“Agencies can guarantee AI citations.” — Debunked
Reality: No one controls third-party model outputs. Guarantees indicate misunderstanding or overselling.
What to do: Ask for process, measurement methods and leading indicators instead.
Source: Google Search Central — AI features and your website
“AI Overview clicks are reported separately in Search Console.” — Debunked
Reality: Google includes AI feature traffic within overall Web search performance data, not as a separate filter.
What to do: Track trends in impressions, clicks and conversions together.
Source: Google Search Central — AI features and your website
“Referral traffic from AI assistants can be measured.” — Nuanced
Reality: Some assistants pass referrers (for example chatgpt.com or perplexity.ai), but many visits arrive as direct traffic.
What to do: Segment known AI referrers in analytics and ask leads how they found you.
Source: OpenAI — Overview of OpenAI crawlers
“A single test prompt shows your AI visibility.” — Debunked
Reality: Outputs are probabilistic and personalised. One prompt is an anecdote.
What to do: Run a repeatable panel of prompts, multiple times, across engines.
Source: Aggarwal et al. — GEO: Generative Engine Optimization (arXiv 2311.09735)
“Secret prompts can force an LLM to recommend you.” — Debunked
Reality: Users write prompts, not you. Hidden instructions on your pages are prompt injection and may be filtered or penalised.
What to do: Earn recommendations with evidence, not manipulation.
Source: OWASP Top 10 for LLM Applications
“Server logs reveal which AI crawlers visit you.” — Verified
Reality: Crawler user agents and IP ranges appear in access logs and can be verified against published lists.
What to do: Review logs monthly to confirm key pages are being fetched.
Source: OpenAI — Overview of OpenAI crawlers
“Zero-click answers mean SEO no longer drives leads.” — Nuanced
Reality: Some informational clicks decline, but commercial and comparison queries still drive visits, and cited brands gain awareness.
What to do: Measure leads and branded search, not only sessions.
Source: Google Search Central — AI features and your website
“Connecting Search Console and Bing Webmaster Tools is essential.” — Verified
Reality: They are the only first-party sources of indexing status and query data for each engine.
What to do: Verify both and submit your sitemap on day one.
Source: Bing Webmaster Guidelines
Agents, commerce & protocols
ACP, MCP, A2A and AI agents: separating real capabilities from launch-deck hype.
“AI agents will soon buy anything without user approval.” — Debunked
Reality: ACP, as published by OpenAI and Stripe, keeps the user authorising purchases while merchants handle orders and fulfilment.
What to do: Prepare clean product data and checkout, not a fear of runaway agents.
Source: Stripe — Developing an open standard for agentic commerce
“ACP replaces your payment processor.” — Debunked
Reality: ACP is a protocol for agent checkout flows; payments still run through supported processors and the merchant remains merchant of record.
What to do: Check your platform and processor support before planning work.
Source: Stripe — Developing an open standard for agentic commerce
“MCP gives AI unrestricted access to your systems.” — Debunked
Reality: MCP standardises how applications connect to tools and data. Access depends on the servers and permissions you configure.
What to do: Expose only scoped tools, with authentication and logging.
Source: Model Context Protocol — Introduction
“MCP is just another name for an API.” — Nuanced
Reality: MCP servers usually wrap APIs, but MCP defines a common interface so many AI apps can discover and call tools in the same way.
What to do: Use MCP when several AI clients need the same tools; a direct API is fine for one integration.
Source: Model Context Protocol — Introduction
“AI agents are ready to run whole businesses unsupervised.” — Debunked
Reality: Agents still make errors, are vulnerable to prompt injection and need oversight for consequential actions.
What to do: Start with supervised workflows and human approval steps.
Source: OWASP Top 10 for LLM Applications
“Prompt injection is a real security risk for agents.” — Verified
Reality: OWASP lists prompt injection as the top risk for LLM applications.
What to do: Separate untrusted content from instructions and limit tool permissions.
Source: OWASP Top 10 for LLM Applications
“Product feeds will matter for agentic shopping.” — Verified
Reality: Agentic checkout relies on structured, accurate product data that agents can read and act on.
What to do: Keep titles, prices, availability and policies accurate and machine-readable.
Source: Stripe — Developing an open standard for agentic commerce
“Small businesses should wait until agents are mainstream.” — Nuanced
Reality: Full agentic commerce is early, but the groundwork — clean data, clear policies, automated intake — pays off now.
What to do: Automate repetitive admin today; adopt protocols when your platform supports them.
Source: Model Context Protocol — Introduction
“A chatbot on your site makes you “AI-ready”.” — Debunked
Reality: A widget does not make your content crawlable, your data structured or your workflows automated.
What to do: Fix discoverability and processes before adding chat.
Source: Google Search Central — AI features and your website
RAG, embeddings & LLM mechanics
How retrieval and generation work, and the myths that come from misunderstanding them.
“LLMs search the live web for every answer.” — Debunked
Reality: Some products retrieve live results for some queries; others answer from training data. Behaviour varies by product, mode and query.
What to do: Be present both in long-lived web sources and in crawlable current pages.
Source: Lewis et al. — Retrieval-Augmented Generation (arXiv 2005.11401)
“RAG eliminates hallucinations.” — Debunked
Reality: Retrieval grounds answers in sources and reduces some errors, but models can still misread or misattribute retrieved text.
What to do: Keep human review for important outputs and cite sources.
Source: Lewis et al. — Retrieval-Augmented Generation (arXiv 2005.11401)
“Content is retrieved in passages, not whole pages.” — Verified
Reality: Retrieval systems typically split documents into chunks and match the most relevant ones to a query.
What to do: Make each section understandable on its own.
Source: Lewis et al. — Retrieval-Augmented Generation (arXiv 2005.11401)
“Vector search makes keywords irrelevant.” — Debunked
Reality: Many production systems combine semantic and keyword (hybrid) retrieval because exact terms still matter.
What to do: Use the real names of products, places and problems.
Source: Lewis et al. — Retrieval-Augmented Generation (arXiv 2005.11401)
“Fine-tuning a model is the best way to add company knowledge.” — Debunked
Reality: For changing facts, retrieval is usually cheaper and easier to update; fine-tuning better suits style and task behaviour.
What to do: Start with a knowledge base plus retrieval.
Source: Lewis et al. — Retrieval-Augmented Generation (arXiv 2005.11401)
“Training cut-offs mean new businesses are invisible to AI.” — Nuanced
Reality: Base models may not know you, but retrieval-enabled products can find current pages.
What to do: Make sure current pages are crawlable and corroborated.
Source: OpenAI — Overview of OpenAI crawlers
“Bigger context windows remove the need for retrieval.” — Debunked
Reality: Long contexts cost more and models can miss details in long inputs; retrieval still selects what matters.
What to do: Feed focused, relevant material.
Source: Lewis et al. — Retrieval-Augmented Generation (arXiv 2005.11401)
“LLMs cite the original source of every fact.” — Debunked
Reality: Citations point to retrieved pages, which may not be the original source, and some answers carry no citation.
What to do: Publish original data clearly attributed so others cite you.
Source: Lewis et al. — Retrieval-Augmented Generation (arXiv 2005.11401)
“Opting out of training stops all AI use of your content.” — Nuanced
Reality: Training opt-outs do not necessarily block search or user-triggered retrieval, which use different agents.
What to do: Review each crawler’s documented purpose.
Source: OpenAI — Overview of OpenAI crawlers