A robots.txt file is a small text file that tells search engines which parts of your website they may crawl. If you have ever wondered how to add a robots.txt file to a website, whether the name matters, or how to remove one, this guide covers all three. It takes about five minutes.
What a robots.txt file does
When a search engine visits your site, it first looks for a file at yourcompany.com/robots.txt. The file lists rules, such as which folders to skip. Well-behaved crawlers follow them. It is a request, not a lock, so never use it to hide private information.
What a robots.txt file looks like
A basic robots.txt file looks like this:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://yourcompany.com/sitemap_index.xml
User-agent says which crawler the rules apply to, and an asterisk means all of them. Disallow lists paths to skip. Allow makes an exception. Sitemap tells crawlers where your list of pages lives. Learn more in Google’s robots.txt introduction.
Does it matter what you name the file?
Yes. The file must be called robots.txt, all lowercase, with no extra letters, and it must live in the root folder of your site. A file named Robots.txt, robot.txt or robots-file.txt will be ignored.
How to add a robots.txt file to your website
- Open a plain text editor and write your rules.
- Save the file as robots.txt.
- Log in to your hosting dashboard and open the file manager, or connect with FTP.
- Upload the file to the root folder of your site, the same folder that holds your main index file.
- Visit yourcompany.com/robots.txt in your browser to confirm it loads.
On WordPress, most SEO plugins let you edit robots.txt from their settings, so you do not need to upload a file yourself. If you need help, our team can add one for you.
How to remove a robots.txt file
Delete the file from your root folder, or clear the rules in your SEO plugin. With no file, search engines may crawl everything. If you only want to unblock one folder, delete that Disallow line instead of the whole file.
Common robots.txt mistakes
- Disallow: / blocks your entire site. It is sometimes left over from a development version, and it is the most common reason a new site never shows up in Google.
- Using robots.txt to hide pages. Blocked pages can still appear in results. Use a noindex tag or a password instead.
- Blocking CSS and JavaScript files. Search engines need them to understand your pages.
Once your file is right, follow our guide to adding your website to Google search to submit your sitemap.
Robots.txt examples for common situations
Here are a few starting points. Change the paths to match your own site.
Allow everything:
User-agent: * Disallow:
Block one folder:
User-agent: * Disallow: /private/
Block a staging site completely:
User-agent: * Disallow: /
Use the last one only on a copy of your site that should never be public. If it ends up on your live site, search engines will stop crawling it.
How to test your robots.txt file
After you add or change the file, load yourcompany.com/robots.txt in a browser and read it. Then open Google Search Console, inspect a few important pages and confirm they are not reported as blocked. Test again after every change, because a small typo can block more than you intended.
Robots.txt vs noindex: which should you use?
A robots.txt file controls crawling. A noindex tag controls whether a page appears in search results. They solve different problems:
- Use robots.txt to keep crawlers out of areas such as admin folders or internal search pages.
- Use noindex to keep a page out of results while still letting Google read it.
- Do not combine them on the same page. If robots.txt blocks a page, Google cannot see the noindex tag.
Robots.txt and your sitemap
Adding a Sitemap line to your robots.txt file helps search engines find your list of pages. Most WordPress SEO plugins add it for you. You can also submit the sitemap directly in Search Console, as covered in our guide to adding your website to Google.
A robots.txt file does not protect private content
Anyone can read a robots.txt file, and listing a private folder in it can point people straight to it. If something must stay private, put it behind a login or remove it from your site. Treat the robots.txt file as a polite instruction to crawlers, not as security.
Keep your robots.txt file simple
A robots.txt file is a small tool, and the best robots.txt file is a short, clear one. Add only the rules you need, test the robots.txt file after every change and never rely on it to hide private pages. A tidy robots.txt file helps search engines spend their time on the pages that matter to your business.
Frequently asked questions
Does it matter what I name my robots.txt file?
Yes. The file must be named robots.txt in all lowercase, and it must sit in the root folder of your site so it loads at yourcompany.com/robots.txt. Search engines will not look for it anywhere else or under any other name.
How do I attach a robots.txt file to a website?
Create a plain text file named robots.txt, then upload it to your site’s root folder using your hosting file manager or FTP. On WordPress you can also edit it from your SEO plugin’s settings.
How do I remove a robots.txt file from a website?
Delete the file from your root folder, or on WordPress clear the rules in your SEO plugin. Without a robots.txt file, search engines assume they may crawl everything.
Does robots.txt hide a page from Google?
Not reliably. It asks search engines not to crawl a page, but the page can still appear in results if other sites link to it. To keep a page out of results, use a noindex tag or password protection.