The Robots.txt file is one of the most important thing to keep up with for any WordPress website. It has all the details that any web crawler will look into before starting to crawl your website content. It is critical to keep your WordPress website’s Robots.txt file up to date. In case you need to see and edit a robots.txt file, the process is really simple. Here we will see the easiest way to edit robots.txt file in WordPress.
What is robots.txt file in WordPress?
Usually when ever you create a website we have a default robots.txt file builtin and to get the info in the robots.txt we just need to enter the websitename/robots.txt in the address bar of any web browser
Format of robots.txt
There is a default format of content that you can find inside Robots.txt file which looks something like as follows
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://website/sitemap.xml
You can add any URL here which “Allow” or “Disallow” in front of it to stop all search engines from crawling that page. In case you give anything wrong here then it would create problem for your website. You may need to find ways to edit it and make it right so that crawlers from Google, Bing, Yahoo and soon can index pages in your site.
Also Read: How to Install Plugin for WordPress Website
Here the User-agent: * shows that all crawlers are allowed. Now that we have checked how robots.txt looks and what should be the content, lets see how to edit robots.txt file in WordPress.
Way to edit robots.txt in WordPress website
There are certain plugins that you can use to edit the robots.txt, but there is a common plugin that is usually used with all WordPress websites. The Yoast SEO plugin is considered to be generic and makes it easy to edit a robots.txt file.
Steps to edit robots.txt file in WordPress
- Sign in to your wordpress website.
- Now when the dashboard is shown in the sidebar scroll down to fine Yoast SEO plugin.
3. Click on the Yoast SEO button and you will be shown with additional settings menu
4. Click on Tools and you will be redirected to a page showing the list of tools in Yoast SEO plugin.
5. Click on the “File editor” button and you will be redirected to another page where you will see the editor to edit the robots.txt file in wordpress.
6. Once you are in this page you can do all the needed edits in the robots.txt file. After you have done all change done forget to click on the “Save changes to robots.txt” button.
Also Read: How to Setup AAWP plugin in WordPress
7. You should be good to go now. To confirm the changes, enter the URL: “https://website/robots.txt” in the address bar and check.
Hope this post helped in finding a solution for editing robots.txt WordPress. If you like this post please share with friends and provide your comment below. Thanks for reading.
