Search Engine Optimization Tips and Tricks – Learn how to increase visibility in Search Engines – Part 22 – Understanding the search engines crawler

In quite a few posts on this site, I have mentioned terms such as search engine crawler, bot, etc. Many readers who are somewhat new in this field will not know what these terms means, and why they are important to you. So, a simple description of how search engines operate, and what do these crawlers do. When you enter a term in a search engine, it does not go searching off into the internet to find what you want, instead the search looks at a huge database owned by the search engine company and finds the relevant search terms. To make it even more clear, your site needs to be in this index, else you can forget getting visitors from search engines. So, how do the search engines create this index, and how does your site end up in this index ? Well, this is achieved through what are called search engines bots or crawlers, or robots. No matter what the term, these are automatic scripts whose primary function is to crawl through the internet, using the links present on web pages, and keep on making a copy of all the content that they find. These are very simple scripts, which really do not have the intelligence to be able to process Javascript, or images, or any other complicated stuff used on sites. Consider the experience as similar to using a text browser on your site, and then reviewing how your site looks on this text browser.
You can actually try this, look for a browser called Lynx (Wikipedia) and view your site on it. You will find that all your complicated JavaScript / DHTML / Flash based navigation is unrecognized by this browser, and it can get stuck. Similarly, if your site does not have easy ways of finding all the posts you have done (such as a sitemap or a simple navigation method), then many of those will not get indexed, and can prevent search engines from finding such posts and making them available to search engines.
So, this is one simple lesson. You need to make sure that whatever nice jazzy stuff you show to your viewers, you should have a way to let the search engine crawlers easy access to all your site, including setting up robots.txt file properly. Further, since the search engine crawler makes it way across the internet using links, you should be linked from other sites, and you should also be linking to many relevant sites.
Once the content is available in the search engines, they rank these based on extremely secret algorithms that constantly evolve. This search engine ranking is what determines how many people visit your site from search engine results, and also means that you need to be constantly at work to ensure that the search engines rate your site high, and any changes in how they use their ranking logic (including frowning on many practices) should be incorporated in your techniques.

1 comment to Search Engine Optimization Tips and Tricks – Learn how to increase visibility in Search Engines – Part 22 – Understanding the search engines crawler

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>