X (Twitter) SEO: What the Platform Actually Lets Search Engines See

X (Twitter) SEO covers two jobs that share a name and almost nothing else: ranking inside X's own search, and having X pages reachable by Google and Bing. The platform is X, formerly Twitter, and this piece uses X throughout. If the open question at your B2B SaaS or AI company is whether X deserves any organic effort at all, the fastest way to settle it is to stop reading tactics and read what the platform publishes about itself. The main source here is X's robots.txt, fetched live on September 18, 2026, which names exactly which crawlers get access and shuts the rest out.

The short version

• Read on September 18, 2026, X's robots.txt blocks every crawler it does not name. The file's last group is User-agent: * followed by Disallow: /, under X's own heading "Every bot that might possibly read and respect this file".
• In that same file, Googlebot and Bingbot share one group with named exceptions. Likes, reposts, media tabs, followers, following and realtime search are disallowed to them. Individual post URLs are not.
• Google-Extended is disallowed outright. Per Google's documentation that opts a site out of Gemini model training and Gemini grounding, and it explicitly does not affect inclusion in Google Search.
• No crawler operated by OpenAI, Anthropic or Perplexity is named anywhere in the file as we read it on September 18, 2026, so each one falls into the catch-all group.
• Robots.txt governs crawling, not indexing. What X allows is the ceiling on what a search engine can do, never a promise that it will.

What does "X SEO" actually mean?

Two different things. Treat them as one and every decision downstream gets muddled.

The first is in-platform search: getting your posts surfaced when someone types a query into X itself. That is X's algorithm, X's index, and X's published explanation of it, which is thin.

The second is external search: whether a page on x.com can be fetched, indexed and ranked by Google or Bing. How you write a post does not touch it. X decides it, in one plain text file served from its domain root, and the ruling applies to your account exactly as it applies to everyone else's.

If you want the wider picture of which social platforms are worth optimizing for, that comparison runs across four platforms and settles a different question. This piece is about what X's own files permit.

What does X's robots.txt say, and when was it read?

Read on September 18, 2026, X's robots.txt file returned HTTP 200 with content-type: text/plain;charset=utf-8 and 113 lines. It contains eleven User-agent lines forming ten groups: ten named crawler tokens plus one catch-all.

It came from the twitter.com host, because one of our requests to the same path on x.com returned a Cloudflare challenge page instead of the file. A later request to x.com the same day returned the same 113 lines, so treat the challenge as intermittent rather than as a property of the host.

Group in the file	What it is	What the file grants it
Googlebot and Bingbot, sharing one group	The crawlers behind Google Search and Bing Search	Named Allow rules, eighteen Disallow rules, and no blanket disallow. Anything the group does not name stays crawlable.
facebookexternalhit	Facebook's link preview fetcher	A narrower allow list. Disallow: /*? closes most query strings, and Disallow: /search?q= closes keyword search.
Google-Extended	Google's product token for Gemini training and grounding	Disallow: *
FacebookBot, Discordbot	Meta's and Discord's own crawlers	Disallow: * for each
meta-webindexer, meta-externalagent, meta-externalads, meta-externalfetcher	Meta AI and Meta product crawlers	Disallow: * for each
*	Every crawler not named above	Disallow: /

Three lines sit after the catch-all group opens: Disallow: /i/u, for notification-email links; Crawl-delay: 1; and Sitemap: https://x.com/sitemap.xml. X's comment beside that first line says the named crawlers stay unblocked on purpose, because they have to crawl /i/u to see the X-Robots-Tag noindex it serves. We did not verify what the sitemap contains, and this piece makes no claim about it.

The Crawl-delay line is worth a second look, because it does nothing for the crawler most readers care about. Google's documentation on how it interprets robots.txt lists the fields it supports as user-agent, allow, disallow and sitemap, and states that other fields such as crawl-delay are not supported. The same page confirms the sitemap field "isn't tied to any specific user agent".

Which parts of X are closed even to Googlebot?

Eighteen Disallow rules sit in that shared group, and they cover most of the surfaces a marketer would want to audit.

What is closed	The rules	Why it matters
Social graph	/*/followers, /*/following, /*/verified_followers	Who follows an account is not readable by Google from X itself.
Engagement history	/*/likes, /*/retweets	The trail of what an account endorsed stays inside the platform.
Media surfaces	/*/media, /*/photo$, /*/photo?	Account media tabs and standalone photo pages are off limits.
Account analytics	/*/analytics	Performance views are not crawlable.
Search result pages	/search/realtime, /search/users, /search/*/grid	X's own search output is closed to both named search crawlers.
Deactivated accounts	/account/deactivated, /settings/deactivated	Dead-end states are closed to the crawler.

What the group does not block matters more. No rule in it matches a post URL of the form /<handle>/status/<id>, it carries no blanket Disallow, and it closes with Allow: /*?, so query-string variants of a permitted path stay open too.

That is the useful shape of the file: the archive and the social graph are shut, the individual post is open.

Can Google index an X post?

The file does not answer that. Nor does X's search result FAQs page, read the same day, which does not mention Google or any external search engine anywhere on it. What the robots.txt answers is the prior question.

Crawling is permission to fetch. Indexing is a separate decision Google makes afterwards, and it is not obliged to make it.

Google's robots.txt documentation, linked above, puts the relationship plainly: "Google can't index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet." Permission and outcome move independently in both directions. Our guide to how robots.txt works, and why a blocked page can still show up in Google covers the mechanism in full, and this piece will not restate it.

So the honest formulation is this. X's robots.txt establishes that Googlebot is permitted to fetch post pages. It establishes nothing about whether Google has fetched any particular one, kept it, or will rank it. We have not measured how much of X is indexed, for the platform or for any account, and this article does not estimate it.

The check you can run yourself: search Google for site:x.com/<handle>, then repeat it against site:twitter.com/<handle>. It is a rough signal rather than a coverage report, because you do not own the domain and cannot run URL Inspection against it. Read a thin result as "not much is here", never as a number.

Can AI assistants read what you post on X?

Three separate answers, and they have to stay separate.

Crawlers X does not name: as we read the file on September 18, 2026, it names no crawler operated by OpenAI, Anthropic or Perplexity, and none operated by Common Crawl, Apple, Amazon or ByteDance. Each therefore falls into the catch-all group, where the rule is Disallow: /. A crawler that respects robots.txt and is not named is being told to take nothing.

Google-Extended: the file disallows it outright. Google's documentation on Google-Extended describes it as "a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models...", and notes that it has no separate user agent string of its own.

Google also documents the boundary of that control: blocking Google-Extended changes nothing about whether a site is included in Google Search, and it is not used as a ranking signal there. That is the half people skip: blocking Google-Extended is a training and grounding control, not a search-visibility one.

What robots.txt cannot do at all: it is a request, not a wall. It governs whether a compliant crawler fetches a URL, not what any system already knows. An assistant can describe your post perfectly well from a newsletter, a forum thread or an article that quoted it, none of which are on X's domain or covered by X's file.

Which leads to a practical conclusion, with one exception: "post on X so AI assistants pick it up" is not supportable for any assistant whose crawler X does not name, and not for Gemini training or grounding. The exception is AI Overviews, which sit inside Google Search and run on the access X grants Googlebot. Our guide to controlling AI crawlers sets out the wider policy picture, including where robots.txt stops being the right instrument.

How does search inside X rank posts?

By popularity, keywords, and an unspecified remainder. That is close to the whole of what X publishes about it on its search result FAQs page.

X's search result FAQs, read on September 18, 2026, states: "We determine relevance based on the popularity of a post (e.g., when a lot of people are interacting with or sharing via reposts and replies), the keywords it contains, and many other factors." The same page adds that "Top posts are selected through an algorithm, we do not manually curate search results."

Three things are worth naming about that. The phrase "many other factors" is carrying most of the weight and is never expanded. Results are filtered by safe search and by accounts a viewer has muted or blocked, so what you see is partly yours rather than universal. And across the full page, X does not mention Google or any external search engine at all, which is why the crawling question has to be answered from the robots.txt instead.

The one actionable read: keywords are named, so a post that states its subject in plain words is legible to the in-platform index in a way that a post carrying its topic only in an image is not. Small lever, and it moves in-platform search alone.

What does X's robots.txt not tell you?

Quite a lot, and being explicit about the boundary is what separates this from a confident guess.

Question	Does the file answer it?	What you would need instead
Has Google crawled a given post?	No. It grants permission, not evidence.	Server logs you do not have access to.
Is a given post indexed?	No. Crawling and indexing are separate.	A site: query, as a rough signal only.
What is in X's sitemap?	It references the URL and nothing more.	A verified fetch of that file, which we did not obtain.
Will Googlebot wait one second between requests?	It asks. Google documents crawl-delay as unsupported.	Google's own crawl-rate behavior.
Will an unnamed crawler actually stop?	No. Compliance is voluntary.	Server-side blocking, which only X can apply.

One more limit, and it is why almost every section above carries a date. Treat the reading as accurate on September 18, 2026 and checkable by anyone in about ten seconds, not as a standing fact.

Is X worth organic effort for a B2B company?

Rarely as a compounding asset. Sometimes as corroboration.

Our analysis suggests the deciding factor is ownership rather than reach. A post you write on X is a page on X's domain, discoverable on terms X sets and can change without notice. The work leaves an artifact, but you do not hold it. Put X through the test our guide to channels you own versus channels you rent applies to every channel and it lands on the rented side, which is where our four-platform comparison already places it and is not changed by anything in the robots.txt.

Two cases still justify the effort:

• Corroboration: a consistent, accurate profile is one more source that agrees with your website about who you are and what you do. That has value for entity understanding regardless of whether any individual post ranks.
• Your buyer genuinely lives there: in some categories, notably developer tools and AI, the people evaluating you may well be active on X. Reach is a legitimate goal on its own. It is simply not SEO, and calling it SEO is how the budget conversation goes wrong.

What to stop measuring: impressions and engagement on posts, as a proxy for search performance. They measure the feed. Nothing in X's published files connects them to anything a search engine does, and no line in the robots.txt gives a high-performing post more crawl access than a quiet one.

When this advice does not apply: if X is already your main source of qualified conversations, none of the above argues for switching it off. It argues for counting it honestly, in the reach column, and not expecting the compounding curve that owned pages produce.

How do you run this check on any platform yourself?

Six steps, and it takes about ten minutes per platform.

Step 1. Fetch the file, not a summary: request /robots.txt at the platform's root and confirm you got content-type: text/plain. A 200 that returns HTML is usually a challenge or an error page dressed as success, so check the header and not the status code alone.

Step 2. Find the group that applies to the crawler you care about: Google's documentation is explicit that "only one group is valid for a particular crawler" and that it picks the group with the most specific matching user agent. So a named group overrides the catch-all entirely. Read your crawler's group and stop reading the others.

Step 3. Read the catch-all last, and read it as the default policy: User-agent: * with Disallow: / tells you the platform's posture is closed. Every named group above it is a deliberate exception, which tells you who the platform decided to let in.

Step 4. Separate crawl control from index control: record what the file permits in one column and what you observe in search results in another. They will not always agree.

Step 5. List the AI tokens that are present, and the ones that are absent: absence is the finding, not an oversight. An unnamed AI crawler inherits the catch-all, which on a closed file means it is disallowed everywhere.

Step 6. Date the read and set a recheck: a robots.txt changes without announcement, and a stale reading is worse than none because it feels sourced.

That procedure produced every finding above, and it works identically on LinkedIn, Reddit or YouTube. If you want help turning findings like these into content and AI-search work that compounds on pages you own, that is the shape of the engagement.

One practice is worth more here than any tactic: read a platform's own files before you believe a guide about it. The file is the version a compliant crawler is given, which makes it a better starting point than any summary of it.

Our recommendation is to do it this week for the three platforms your team publishes to most. Fetch each one's robots.txt, find the group that covers Googlebot, and write down one sentence per platform about what it permits. Then compare those three sentences against what your plan currently assumes about each channel.