top of page
Merediths-Husband-logo.jpg

LLMS

  • Writer: Meredith's Husband
    Meredith's Husband
  • 7 days ago
  • 3 min read

How to Create and Upload an llms.txt File for AI Search Bots

Control how AI bots like ChatGPT and Gemini interact with your website

The llms.txt file is an emerging best practice to control how AI language models (LLMs) — like ChatGPT, Claude, Perplexity, or Gemini — interact with your website. Similar to robots.txt, this plain text file lets you allow or disallow access to parts of your site, specifically for AI bots.

In this guide, you'll learn how to create the file step-by-step and upload it to your website’s root directory using two popular methods.

How to Create an llms.txt File for AI Search Bots (Step-by-Step)

🔧 Step-by-Step Guide

Step 1: Open a Plain Text Editor

Use any text editor you’re comfortable with:

  • Windows: Notepad or Notepad++

  • Mac: TextEdit (set to plain text)

  • Advanced: Visual Studio Code, Sublime Text

Step 2: Copy and Paste the Sample Directives Below

This file:

  • ✅ Allows major AI bots to access your public content

  • ❌ Disallows access to common private, sensitive, or user-specific areas

  • 💬 Uses comments (lines that start with #) to explain each section

Paste the following into your text editor:

yaml
CopyEdit
# ================================================
# llms.txt – Directives for AI Search Bots
# Purpose: Control what content Large Language Models (LLMs)
#          like ChatGPT, Claude, Gemini, and others can access
# ================================================

# ----- General Rules -----
User-Agent: *
Disallow: /admin/
Disallow: /login/
Disallow: /dashboard/
Disallow: /account/
Disallow: /checkout/
Disallow: /cart/
Disallow: /settings/
Disallow: /thank-you/
Disallow: /orders/
Disallow: /user/
Disallow: /members/
Disallow: /private/
Disallow: /wp-admin/
Disallow: /cgi-bin/
Disallow: /hidden/

# ----- OpenAI GPTBot -----
User-Agent: gptbot
Allow: /
Disallow: /admin/
Disallow: /login/
Disallow: /dashboard/
Disallow: /checkout/
Disallow: /members/

# ----- Anthropic Claude Bot -----
User-Agent: anthropic-ai
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /checkout/
Disallow: /private/

# ----- Google Gemini (google-extended) -----
User-Agent: google-extended
Allow: /
Disallow: /admin/
Disallow: /login/
Disallow: /orders/
Disallow: /user/

# ----- Perplexity AI Bot -----
User-Agent: perplexitybot
Allow: /
Disallow: /login/
Disallow: /settings/
Disallow: /private/

# ----- Mistral AI Bot -----
User-Agent: mistralbot
Allow: /
Disallow: /checkout/
Disallow: /dashboard/

# ----- Default catch-all for any unknown LLM bots -----
User-Agent: *
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /members-only/
Disallow: /orders/
Disallow: /user/
Disallow: /checkout/

Step 3: Save the File as llms.txt

  • Go to File > Save As

  • Save it as llms.txt (not llms.txt.txt)

  • Set Encoding to UTF-8 (if prompted)

  • Save to a known location on your device

Step 4: Upload the File

Now that you've created your llms.txt file, the next step is just as important: uploading it to the root directory of your website.

Why the root? Because that’s where AI crawlers will automatically look for it — just like they do with robots.txt and sitemap.xml. If the file isn’t placed at the root (e.g. https://yourdomain.com/llms.txt), most bots won’t be able to find or follow its directives.

This is where the file must live:

Let’s look at two easy ways to upload your file to the right location.

🔌 Option 1: Uploading via WordPress File Manager Plugin

If your website uses WordPress, here’s how you can upload the file using a trusted plugin.

✅ Recommended Plugin

WP File Manager

  • ★ 4.8 stars | 1M+ installs

  • Gives you drag-and-drop access to site files right inside your WordPress dashboard.

📂 Steps:

  1. Go to Plugins > Add New and search for "WP File Manager."

  2. Click Install Now, then Activate.

  3. From your dashboard, click WP File Manager.

  4. Navigate to the root directory (/public_html or top-level folder).

  5. Click Upload, and select your llms.txt file.

  6. Visit https://yourdomain.com/llms.txt in your browser to confirm.

🔐 Important:

Deactivate and delete the plugin after uploading to reduce security risks. These tools provide powerful access and should not remain active longer than needed.

🖥️ Option 2: Uploading via Your Web Hosting File Manager (cPanel, GoDaddy, SiteGround, etc.)

Most web hosts provide access to a file manager. Here's how to use it:

📂 Steps:

  1. Log into your hosting dashboard (e.g., GoDaddy, Bluehost, SiteGround).

  2. Open File Manager (usually under cPanel or Site Tools).

  3. Navigate to the public_html folder (your site’s root).

  4. Click the Upload button.

  5. Select your llms.txt file from your computer.

  6. Confirm it appears in the file list.

  7. Open a browser and go to:

    https://yourdomain.com/llms.txt

If you see the file's contents, it worked!

Bonus Tips

  • 🧩 Don’t confuse llms.txt with robots.txt — they serve different purposes (AI bots vs. search engines).

  • 🛡️ Update your privacy policy to reflect your AI data usage preferences.

  • 🧠 Respect varies — while OpenAI, Google, and Anthropic say they honor llms.txt, not all crawlers comply.

🏁 What You’ve Achieved

  • ✅ Created and configured an llms.txt file

  • ✅ Uploaded it to your root directory using a method that fits your setup

  • ✅ Taken proactive steps to control how AI bots interact with your content

 
 
bottom of page