![]() The key word here is “discourage”: Search engines have no obligation to honor this request, especially search engines that don’t use the standard robots.txt syntax that Google does. It can also add a meta tag to your site’s header that tells Google and other search engines not to index any content on your entire site. When you tick “Discourage search engines from indexing this site,” WordPress modifies your robots.txt file (a file that gives instructions to spiders on how to crawl your site). Google uses them to decide how to rank and place your website in the search results, grab blurbs from your articles for the search results page, and pull your images into Google Images. Spiders “crawl” the web, visiting websites and logging all your content. Have you ever wondered how search engines index your site and gauge your SEO? They do it with an automated program called a spider, also known as a robot or crawler. What Does “Discourage Search Engines from Indexing This Site” Mean? ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |