Search Engine Spiders

In the past I have talked about setting up your Meta Tags and submitting your site to the major search engines. Once this has been done successfully you can expect to start receiving visits from some of the search engine spiders.

What is a spider? Search engine spiders are small programs that are sent out by the search engines to scurry around the web looking at and cataloging web sits. They go by such cryptic names as
Mozilla, Slurp, Gulliver, ArchitextSpider and so on. You can expect these spiders to visit monthly,
weekly and daily and if you are using a good Stats program you will see them listed.

The most common spiders are:

Sidewinder   -Infoseek, (
Slurp           -
Inktomi, (Hotbot, Yahoo, Snap, Microsoft Search, Goto.com etc., AOL)
Scooter      -
Alta Vista
T-Rex         -
Architext     -Excite, WebCrawler
Backrub      -Google

Something to note is that not all search Engines use their own Spider. Slurp which is owned by Inktomi, is used by several different search engines.

It has been reported that some search engines will give a web page that has been "
spidered" or "crawled." a higher ranking than one that has been submitted. Unfortunately not all spiders are created equal. Setting up your web site to work well with one spider may not be appropriate for
another. In some instances different search engines will interpret the same information gathered by a shared spider differently, making the process of optimizing your web site for this purpose even more difficult.

Even so, there are some things you can do to optimize your site. Here are a few tips to ensure that you get the most from these arachnid visitors.
  1. Submit your two most important pages to the major search engines on a timely basis, usually every two weeks. Many spiders will not visit until your site has been submitted.
  2. Ensure that all your pages are within three clicks of your home page since many spiders will not index any further than that.
  3. Update your pages often. It gives the spiders a reason to continue indexing your site.
  4. Be certain all your pages have Meta Tags and that they are properly set up. See our previous articles at: for additional information on setting up your Meta Tags.
  5. Avoid spam-dexing. This includes excessive repetition of keywords, hidden text, tiny text etc.

Once your site has been catalogued, it can still take up to several weeks before the search engines list it.

To find out more about the search engine spiders, take a
look at: searchengineworld

"IMS Web Tips" ISSN 1488-7088
© Copyright 2001 Virtual Mechanics

"IMS Web Tips" is a weekly news letter for all web site managers regardless of experience who are looking for detailed information on creating, maintaining and promoting their web sites.

To subscribe send an email to join.imswebtips@list.imswebtips.com or visit www.IMSWebTips.com for subscription information and a list of past articles.


If you like the contents of this newsletter, please recommend it to a friend. Not only will you help us to continue to provide you with useful and informative articles, you could also win $10,000. Click here for details.
Click here for a free WYSIWYG pixel precision HTML editor.
Home Page | Archive by Date | Archive by Topic | Virtual Mechanics | DHTML Magic | SVG Magic