It's advisable to use this option if we want to know the total number of pages indexed in search engines like Google. Respect noindex: if we mark this option, the spider will not crawl those pages with the “noindex” meta tag.Request authentication: if any of the web pages that are part of the site is password-protected, by checking this option, the program will ask us to enter a username and password to access and analyze the protected page.Limit depth search: with this option, we can set the spider only to crawl a few clicks away from the home page.Limit the total number of pages to crawl: with this option, we will limit the number of pages crawled by the “ spider.”. If we want our spider to ignore that file and inspect all areas of the website, we need to check this option. Ignore robots.txt file: if we have blocked certain areas of our site by using the file robots.txt.Crawl subdomains: if our site has multiple subdomains and wants to “ spider” it, we need to check this option. This option is very useful if we want to know the total number of “ dofollow” pages our site contains.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |