Only 10 of 50 Travel Sites Set Explicit AI Crawler Rules

Only 10 of 50 major travel websites explicitly address AI crawlers in their robots.txt files, according to Skift’s audit—one in five sites with a stated AI policy, and four in five relying on catch-all rules or defaults as AI-driven travel shopping expands.

Travel brands say they want AI to find them. Their websites often tell automated programs something else, or nothing specific. That mismatch forms the starting point for a market where the rules of machine access remain unsettled.

On Oct. 30, Anthropic, Google, Microsoft and OpenAI backed a proposal that would allow a website to opt out of AI training by all four providers with one robots.txt entry. The site must add it explicitly; catch-all rules will not carry the new opt-out.

Robots.txt does not generate an AI answer or replace a booking engine. It functions as a gate for automated crawlers, telling them which pages they may retrieve. The distinction now carries commercial weight because travelers increasingly use AI as a shopping layer, and sites that remain unreadable can vanish from results guiding those decisions.

In July, Adobe reported that AI visits to travel sites surged 119% year over year, more than doubling traffic from the prior period. Adobe analyst Vivek Pandya described the machine-readability gap as a “major bottleneck.” He wrote: “Significant portions of U.S. travel and retail sites remain unreadable to machines, restricting their presence across AI search results.”

The unreadability reaches the commercial details. “When an LLM cannot easily read brand content (such as amenities, pricing, or availability),” Pandya added, “potential revenue is being left on the table.” For a hotel or travel brand, those fields carry the information a prospective guest needs before a booking.

Four policy choices emerge

Among the 10 sites with explicit policies, the split favored visibility over training:

• Expedia and Vrbo give AI search crawlers broad access.
• Tripadvisor blocks training crawlers while allowing some search access.
• Best Western and ITC Hotels explicitly welcome OpenAI’s training crawler.
• Airbnb made the sample’s only major policy change, moving from allowing GPTBot and ClaudeBot in August to disallowing access across the board by October.

That spread is strategically uneven. Expedia and Vrbo prioritize discovery, Tripadvisor draws a narrower line, and Airbnb reversed course between August and October. The audit shows no shared industry standard for balancing machine visibility with model-training access.

This choice is not costless. Sites that permit search crawlers can expose prices, availability and amenities to AI-assisted shoppers, while sites that welcome training crawlers grant providers a different kind of use.

The audit does not attach a revenue figure to either permission, but Pandya’s warning makes the operator exposure plain: unreadable commercial content can limit discovery and leave potential revenue on the table.

Enforcement breaks the policy line

Posted rules did not always control what happened. Skift’s automated requests met 403 Forbidden errors at eight sites, mainly hotels. CraftedStays separately found that Marriott, Hilton, IHG, Hyatt and Four Seasons named no AI crawler on their banned lists, even as their bot protection refused fetches outright.

A permissive file therefore does not guarantee machine access. Conversely, a listed ban may not identify every automated request that operators actually stop. Skift therefore tested stated policy and technical enforcement, and found that the two could send different signals.

Cloudflare data showed Meta’s crawler, not OpenAI’s or Anthropic’s, accounted for 25.5% of AI-bot requests to travel and tourism sites—roughly one in four.

What does this change for travel operators?

For travel operators, the practical issue is not whether a file contains fashionable AI language. It is whether crawlers can retrieve the content that supports a purchase decision—price, availability and amenities—and whether the operator has deliberately chosen to permit search visibility, model training or both.

The immediate task is to audit robots.txt against actual bot behavior. A site needs an explicit policy if it wants the Oct. 30 four-provider opt-out to work. The examples also show why a file alone will not settle access: protection layers can return 403 errors, and declared rules can differ from enforcement.

The proposal may standardize opt-outs, but Skift’s findings show standardization will not solve enforcement. Operators that want AI-assisted discovery will need clear permissions, machine-readable commercial details and technical access consistent with the policy on paper. With AI traffic already up 119%, the next contest is over who can read the sale.