So you’ve got your site hosted and uploaded, and now it’s time to go about making sure the world knows you’re here. You may have heard the terms “crawling” and “indexing” as it relates to your website. They are two different processes, that each have their own significance. Let’s take a look at each, and how you can best achieve positive results with them.
Crawling
Crawling is the process by which Google, (or other search engines) finds new (or existing but updated) pages to look at and determine whether or not to include in their search listings. Google’s robot, affectionately known as Googlebot, scours the web seeking new content to crawl.
Some of the best practices you can employ to make sure GoogleBot is coming around, and hopefully frequently, are as follows.
• A Google Sitemap is a large aid in helping GoogleBot seek and devour your content.
• Make sure your content is readable by the bots, and not hidden in Javascript or other language the bots find difficult to digest.
• RSS feeds of your content go a long way toward announcing that there is something new to look at, and often can help get your pages crawled and indexed in hours.
• Frequently updated and unique content. Nothing the bots like better!
• Incoming links, particularly from relevant authority sites, are a big help. Sites that are getting spidered daily will send the spiders right over!
Indexing
This is the process whereby Google takes the data that GoogleBot has harvested, reads the content, and makes a value judgement based on an algorithm containing more than 200 factors as to where your page should place (if at all) in the search results.
Obviously, getting your pages indexed and able to be found high in the search results is what everyone is after. Here are some ways to help that happen.
• Offer up unique, high-value content.
• Strive to acquire backlinks, particularly from other sites in your niche, and hopefully some that pass link juice!
• Institute and follow a sound site architecture that optimizes your internal linking structure.
• Make sure there are no canonical domain issues. Decide on either www or non www. Don’t split your link values!
Getting indexed can mean one of three things. You could be placed in the main index, (what you want); you might be relegated to the supplemental index (not what you want) or you could be crawled but dropped from the index altogether. (Nuff said) The last one usually happens with duplicate pages on a site, low-value pages, or pages that have unreadable content.
Google Webmaster Tools can be a great aid in helping you determine what you need to do in order to get your pages crawled and indexed. Get started in this sooner rather than later!