To enhance their databases search engines are constantly crawling the World Wide Web using robots or bots. These sites are then analyzed using algorithms to index and rank websites. Crawling is a regular process and so far there are 3.5 billion properties in Google’s Database. This number is growing day by day as new websites enter cyberspace.
Already known websites are crawled periodically to make out if new additions or changes have taken place. Some websites set crawling time using a well-set cache. There are many ways search engines can be invited to visit the website.
How crawling works?
Search engines have their own mechanism to crawl and index web pages. To determine indexing and crawling search engines use several metrics that could be as many as two hundred. These metrics are algorithmically set such that they accurately index and rank websites about their inherent qualities.
The search engine bots can be identified by a string they leave behind. The bots are also known as user agents.
Examples
Googlebot User Agent
Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)
Bingbot User Agent
Mozilla/5.0 (compatible; bingbot/2.0; +https://www.bing.com/bingbot.htm)
Baidu User Agent
Mozilla/5.0 (compatible; Baiduspider/2.0; +https://www.baidu.com/search/spider.html)
Yandex User Agent
Mozilla/5.0 (compatible; YandexBot/3.0; +https://yandex.com/bots)
A Reverse DNS Agent can trace the IP Address that made the lookup. Search engines crawl and index every URL they encounter but not a nontext file such as video, image, and audio. However, they can read file names, content data, and metadata.
How New Pages Are Discovered & Indexed
Search engines extract new pages using those connected to other websites. This is why external and internal links are an important part of the world wide web. Remember anchor text in links carry a lot of information to the search engines and hence is important. The web is like a spider and is interconnected all over except the private networks. Articles and directories are thus helpful in new link discovery.
Both HTML and XML Site Maps are helpful as robots crawl them and the new pages they carry. New pages can be submitted directly to search engines. Direct page submission too is helpful in new page discovery other than those discovered through the crawl.
Crawling Architecture
The search begins with crawling. The web crawling architecture can be slow, or one that downloads hundreds and millions of pages in a week. Search engines like Google crawl millions of pages in a week and store them in hundreds of data centers all over the World.
After a site has been successfully crawled the algorithms work upon it. Fast crawlers provide several challenges in system design, I/O and network efficiency, and robustness and manageability. While the crawlers can be traced the algorithms are kept secret so that spammers do not take advantage. No wonder SEO or search engine organization is a big mystery for webmasters. Crawling plays a major role in indexing and ranking hence SEO.