Today, we’re going to teach you guys how to set up and configure robots.txt in your Magento 2 store.
What is Magento 2 Robots.txt File?
A Magento 2 robots.txt file is a configuration file that tells search engine crawlers (Google, Bing etc.) how to interact with your website. This file defines what parts of your site to crawl or index and what to avoid. It helps control search engine behavior, protecting sensitive pages or areas that shouldn’t be indexed (like admin pages, customer data or internal search results).
The robots.txt is basically a file that tells all search engine crawlers which URLs they can access on your website or eCommerce store.
You can also keep single or multiple web pages out of Google by block indexing using noindex.
Below, we’ll show you exactly how to set up robots.txt in Magento 2. After that, we will also show you how to configure robots.txt in your Magento 2 store.
How to Set Up Robots.txt in Magento 2?
Please follow the below steps to learn how to set up robots.txt in Magento 2.
Step 1. First of all, log in to your admin panel and navigate to Stores > Settings > Configuration.
Step 2. After that, select XML Sitemap in the Catalog tab.
Step 3. Next, open the Search Engine Submission Settings section and select YES in the Enable Submission to Robots.txt field.
Step 4. Finally, click on the Save Config button to finish the process.
How to Configure and Add Magento 2 Robots.txt?
Please follow the below steps to learn how to configure and add robots.txt in your Magento 2 store.
Step 1. First of all, log in to your admin panel and navigate to Content > Design> Configuration.
Step 2. After that, find the Global line, click on the Edit button, and open the Search Engine Robots section.
Step 3. Next, select the Default Robots as per your requirements from the following options.
- INDEX, FOLLOW: Search engine crawler will regularly index your pages and update the changes.
-
INDEX, NOFOLLOW: Search engine crawler will index your store once but will not come back to update the changes.
-
NOINDEX, FOLLOW: Search engine crawlers won’t index your website. It’ll only monitor changes.
- NOINDEX, NOFOLLOW: Search engine crawler will hide your website or eCommerce store from search engine.
Step 4. Now, write any custom instructions if you want to in the Edit custom instruction of robots.txt file.
Step 5. If you want to remove custom instructions and simply apply the default settings, click on the Reset to Default button.
Step 6. Finally, click on the Save Config button to finish the process.
Conclusion
And there you go!
This is the easiest way to set up as well as configure robots.txt in your Magento 2 store.
And if you need our professional assistance, feel free to contact us at any time.