How to Configure llms.txt, robots.txt, and Sitemap for AI Crawlers

What does llms.txt do and how to configure it for your site?

llms.txt is a text file that helps briefly present your site's structure and key content to systems that work with content. We create it not as a promise of presence in answers, but as a managed technical document: it highlights pages that your organization considers useful for review.

Before preparing the file, we cross-check it against your actual site: which sections are public, where current information is located, and which pages should not be included in the brief navigation. For an organization with products or services, it's important that the description leads to primary sources rather than replacing them with promotional wording.

In the work, it's useful to gather in advance:

• main pages about the company, product, and services;
• links to documentation, help center, or public materials;
• information about outdated sections and pages that should not be included;
• a contact for approving editorial wording.

This way, we make the file a reflection of your site's priorities, not a random list of URLs. For general context on AI search visibility, see the AI Search Visibility page, and for technical optimization, see Technical AEO.

How to check robots.txt for AI crawlers

robots.txt sets the access rules published by the site owner for crawlers that respect this file. As part of the service, we review existing directives and agree on changes for AI user agents, so the rules match your chosen access policy and do not conflict with existing restrictions.

First, determine what should be accessible: public product pages, documentation, blog, or specific reference sections. Then, mark areas that require restrictions, such as internal pages or technical paths. We compare this policy with the current file rather than adding new directives without checking context.

When agreeing, pay attention to three things:

• which site sections contain public information;
• which rules already apply to the relevant paths;
• who from your project approves changes to the access policy.

We also document exactly which lines are changed and why, so your site team can verify the result after publication. Note that robots.txt rules and llms.txt content serve different purposes: the former describes access, the latter helps present selected materials. If your project requires work on answers in a specific system, that should be planned separately — for example, for Perplexity or Google AI Overviews.

How does a sitemap help with site page navigation?

A sitemap is a URL map published by the site owner; its role in this project is to provide a clean list of selected pages and align it with the current structure. We check the map against the site and note issues that may hinder keeping it up to date.

Optimization does not mean adding every available page to the sitemap. First, determine which URLs are important and should remain part of the public structure: for example, active product pages, documentation, and key materials. Then compare this set with what is actually published. If the site has outdated addresses or multiple versions of similar pages, it's useful to agree with the content owner on which version is primary before making changes.

For client-side verification, ask your team to show:

• which sitemap file is used;
• what types of pages are included;
• how site updates are reflected in the current map;
• whether the published list matches the agreed pages.

Such an analysis helps your team make a concrete decision about the sitemap composition and related files. It does not replace content development: the usefulness and accuracy of the pages themselves remain a separate task. To improve materials that may be used in AI answers, you can separately consider Content for AI Answers and Structured Data.

What does llms.txt setup include and how does the work proceed?

As part of the project, we review current files, prepare agreed changes, and deliver a report on the completed setup. The typical timeline is one to two business days; it is measured in the project's working context and depends on how quickly you provide access and approve the policy for site sections.

The work is carried out by a specialist who gathers technical context and connects it with your project goals. Before starting, we need the domain, available versions of llms.txt, robots.txt, and sitemap, a list of key pages, and a contact person who can approve restrictions and wording. If your site is updated through a developer, determine in advance who publishes changes.

The result includes:

• a prepared and configured llms.txt file;
• agreed edits to robots.txt for AI user agents;
• verification of the sitemap and suggestions for its configuration;
• a final report with a list of changes and their intended purpose.

After publication, we verify that the changed files are accessible and document the completed work in the report. If you need a broader analysis of technical and content factors, compare the scope with the GEO Audit service or explore the Technical AEO page to avoid mixing file setup with site content tasks.

What llms.txt does not solve for Perplexity and other systems

llms.txt, robots.txt, and sitemap help formalize the site owner's intentions and make the published structure clearer, but they do not replace editorial and technical work on the pages. The file itself does not turn content into a citable source and does not provide grounds to promise appearance in AI answers.

The limitation here is specific: individual AI platforms independently determine how to interpret these files and apply their own rules for access, indexing, and content selection. We can perform agreed configurations and report on them, but we do not control platform decisions about which sources to show or mention.

Plan the service as a technical foundation, not as a standalone way to "get into Perplexity answers." After configuration, evaluate your site more broadly: check that key pages contain accurate information, the content is current, and important claims can be verified against primary sources. For further work on presence in different AI systems, explore Visibility Monitoring and the Perplexity Optimization direction.

To get started, send us your domain, links to llms.txt, robots.txt, and sitemap if they are already published, and a list of important site sections. The MediaHype team will review the initial state, clarify your access policy, and propose the scope of configuration.

Prices

Starting prices in USD. Custom bundles and volume discounts on request. Payment in USDT, USDC, BTC, ETH, SOL, TON or your project token.

How it works

1. Collect initial dataYou send the domain, links to current files, and a list of important pages. We clarify who approves access rules.
2. Review current configurationWe cross-check llms.txt, robots.txt, and sitemap against the site structure and note areas that need decisions.
3. Agree on changesWe finalize the page list for llms.txt and rules for AI user agents, then submit changes for approval.
4. Publish and verifyAfter implementation, we verify file accessibility and prepare a final report with a list of completed tasks.