Zuper LLMs Txt Generator - Documentation

Overview

This guide explains how to configure AI crawler access and search engine bot rules for your Shopify store using the Zuper LLMs Text Generator app.

What You Will Learn

• How to generate and manage your LLMs.txt file.
• How to create and implement robots.txt rules.
• How to control which content AI crawlers and search engines can access.
• How to verify your configuration is working correctly.

Before You Begin

Generate LLMs.txt

Let's walk through creating your AI ready llms.txt file together. By the end of this section, you'll have full control over which parts of your store AI crawlers can access.

1. Accessing the LLMs.txt Generator

First, let's get you to the right place.

1. Open the Zuper LLMs Text Generator app from your Shopify admin.
2. Now, you have two easy ways to reach the LLMs.txt Generator:
   • From the dashboard: Look for the Manage Crawlers or Edit Rules buttons. Click either one, and you'll land directly on the configuration page.
   • From the left navigation: Simply click LLMs.txt Generator in the app menu.

2. Configuration Panel Overview

Great — you've made it to the LLMs.txt Generator. Here's what you'll see on this page:

• Shopify Content Access Permissions
• AI Crawler Allow/Block Rules
• Product Inclusion Rules
• Collection Inclusion Rules
• Custom Path Rules
• Live Preview: /llms.txt

Take a moment to scan these sections. Don't worry if everything isn't familiar yet — we'll walk through each one together.

3. Configure Content Access Permissions

Let's start by deciding which parts of your store AI crawlers should be able to see. You'll find five toggle switches. Turn on the ones you want to include:

• Enable Product Access
• Enable Collection Access
• Enable Blog Access
• Enable Page Access
• Enable Sitemap Access

4. Set AI Crawler Rules

Now, let's talk about who gets access. You'll see a list of AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.). For each one, you decide: Allow or Block.

As you make changes, you'll see them reflected in the Live Preview panel on the right. Remember to click Save.

5. Configure Product Inclusion Rules

Let's walk through this section together. Here, you'll decide exactly which products from your store should appear in your llms.txt file.

You'll see five options available. Keep in mind: you can only select one option at a time. Don't worry, we'll go through each one with clear examples so you can pick what works best for your store.

Your Five Options

1. All products
2. Products with selected tags
3. Exclude products with tags
4. Selected products only
5. Exclude selected products

Let's break down each one so you know exactly what to expect.

1: All Products

This one is straightforward. When you select All products, every product in your Shopify store will be included in your llms.txt file.

No filtering. No exceptions. Just all your products, visible to the AI crawlers you've allowed. If you want maximum visibility for your catalog, this is the option for you.

2: Products with Selected Tags/ Excluded Selected Tags

Want more control? This option lets you include only products that have specific tags. Same goes for the excluded selected tags also.

Here's how it works:

When you select Products with selected tags, you'll need to tell the app which tags to look for.

1. Click the Browse button. A popup window will appear, asking you to enter tags.
2. In the popup, type the tag names you want to include. For example: "featured", "summercollection", or "aiapproved".
3. After typing each tag, click the Add button to add it to your list.
4. Once you've added all your tags, close the popup.
5. Click Save to apply your settings.

After saving, you'll see the products with those tags appear in your llms.txt preview. Simple, right?

3: Selected Products Only (or) Exclude Selected Products

Prefer to pick products manually? This option gives you complete control. Same goes for the Exclude selected products also.

Here's how it works:

1. Select Selected products only.
2. Click the Browse button. A popup will appear showing your product list.
3. Search or scroll to find the products you want to include.
4. Select the products you want by checking the boxes next to them.
5. Click the Add button to confirm your selection.
6. Close the popup and click Save.

After saving, only the products you picked will appear in your llms.txt file. Great for curating a specific subset of your catalog.

Product Price Display

One more setting to consider: do you want product prices to show up in your llms.txt file?

You'll see a toggle labeled Display Product Price.

• Turn it on if you want prices included.
• Turn it off if you prefer to keep pricing information private.

This setting applies to all products that appear in your llms.txt output.

6. Configure Collection Inclusion Rules

When you selected the collection you will be press add button add it then you will save

7. Add Custom Path Rules

Sometimes, you need fine grained control over specific URLs. That's where custom path rules come in.

You can allow or block any path on your store. For example:

• Disallow: /cart
• Disallow: /checkout
• Allow: /products

Rule Status Settings

Each rule can be set to:

• Active: The rule is applied in your llms.txt file.
• Disabled: The rule is saved but ignored (useful for testing).

How to Add a Custom Rule

Let's add a custom rule together:

1. Click Add Custom Rule.
2. Choose Allow or Disallow.
3. Enter the URL path you want to control (for example: /account or /pages/about).
4. Set the status to Active or Disabled.
5. Click Add to save your rule.

8. Preview LLMs.txt

Take a look at the Live Preview panel. This shows you exactly what your llms.txt file will look like — in real time.

As you adjust settings, the preview updates automatically. It's a great way to double check your work before you publish.

Two Helpful Buttons

• Reset: Want to start over? Click this to restore all settings to their default values.
• Sync: Added new products or collections since you started? Click this to refresh the preview with your store's latest content.

Verification Checklist

Before you move on, verify these items:

• Allowed crawlers match your intentions
• Blocked crawlers are listed correctly
• Content sections reflect your toggles
• Custom rules appear as expected

9. Generate and Access the File

You've configured everything. You've saved your changes. Now, let's make sure you can find your file.

Your llms.txt file is automatically generated and live at:

https://your-store.com/llms.txt

How to Access the File

You can access it two ways:

1. From the app dashboard: Scroll to the Public File URLs section and click View LLMs.txt.
2. Directly in your browser: Visit https://your-store.com/llms.txt.

Robots.txt Editor

Let's take control of how search engines and bots interact with your store. In this section, you'll learn how to create and customize your robots.txt file using the Zuper LLMs Text Generator app.

Accessing the Robots.txt Editor

Ready to get started? Here's how to find the Robots.txt Editor:

1. From your Shopify admin, open the Zuper LLMs Text Generator app.
2. Navigate to Robots.txt Editor from the app dashboard.

That's it — you'll now see the robots configuration panel where you can start building your rules.

Use Rule Templates

Not sure where to start? No problem. We've included built in templates to help you configure common rules quickly — no technical knowledge required.

Available Templates:

• Block Cart Pages – Prevent search engines from indexing your cart pages
• Block Checkout – Keep your checkout process private
• Allow Product Pages – Ensure your products are discoverable
• Block Admin Pages – Protect your admin area from indexing
• Allow Collections – Make your collections visible to search engines

Simply click on a template to apply it instantly. You can always customize the rules later if needed.

Add Custom Rules

Want more control? You can create your own rules for specific bots and paths. Let's walk through how to configure a custom rule.

You'll need to set three things:

1. User-agent – This identifies which bot the rule applies to. Examples:
   • * (applies to all bots)
   • GPTBot (OpenAI's crawler)
   • Googlebot (Google's search crawler)
2. Path – The URL path you want to control. Examples: /cart, /collections*, /admin
3. Rule Type – Choose one:
   • Allow – Permits the bot to access this path
   • Disallow – Blocks the bot from accessing this path

Manage Rule Status

Here's a helpful feature: each rule can be set to one of two states:

• Active – The rule is applied in your robots.txt file.
• Disabled – The rule is ignored (but saved for later use).

Why is this useful? It lets you test rules safely without deleting them. If you're not sure about a rule, disable it first. See how your store performs. Then activate it when you're ready.

Preview Robots.txt

Want to see what your robots.txt file will look like before you publish it? Scroll down to the Live Preview section. Here, you'll see the generated file in real time. As you add or modify rules, the preview updates automatically.

Before moving forward, verify these items:

• All rules are correct and match your intentions
• No important pages are accidentally blocked
• Sensitive paths (like /admin, /checkout) are properly protected
• You can copy the text or liquid also.

Apply Robots.txt Changes

Step 1: Copy the Generated robots.txt Content

1. Open your Shopify admin panel and navigate to Apps > Zuper LLMs Text Generator.
2. From the left hand menu, click on Robots.txt Editor.
3. Locate the Live Preview: /robots.txt section on the right side.
4. Click the Copy button.

Step 2: Open Theme Code Editor

1. Go to Online Store > Themes.
2. Find your Current theme.
3. Click Edit Code (found in the "..." actions menu).

Step 3: Create or Locate robots.txt.liquid

1. In the left sidebar, locate the Templates folder.
2. Look for a file named robots.txt.liquid.
3. If it exists: Click to open it.
4. If it does NOT exist: Click Add a new template, name it robots.txt.liquid, and click Create template.

(or)

Step 4: Paste the Generated Content

1. Paste the content you copied in Step 1 into the robots.txt.liquid file.
2. Note: You can either replace the existing content entirely or append it after Shopify's default rules.

Step 5: Save Changes

Click Save. Until you click Save Or Press CTRL+S, nothing will update on your live store.

Step 6: Verify the Output

Visit https://yourstore.com/robots.txt in a new tab. Confirm that the rules appear correctly.

Verify Robots.txt

After saving your changes, do a final verification at your store URL: https://yourstore.com/robots.txt

Confirm the following:

• Rules are visible and properly formatted
• Changes are applied correctly
• No syntax errors in the file
• All your custom rules appear as expected