Spiders Control Organic SEO
An Analysis of Search Engine Spiders and the role they play in Organic SEO.
Let's Examine the robot spiders (Artificial Intelligence Agents/Robot Crawlers) that determine the organic search engine results.
The major search engines are: Google, Yahoo, MSN and AOL.
This set of search engines are known as the Big Four. The big four search engines of Google, Yahoo, MSN and AOL together represent 98% of keyword search activity.
Three spider families control the organic search results on the Big Four search engines of: Google, Yahoo, MSN and AOL.
Organic search results on Google are determined by the googlebot family of spiders with assistance from the Open Directory Project (www.dmoz.org) often identified in the Google results as the Google Directory.
Organic search results on AOL are also powered bt the googlebot spiders with slight assistance and support from the dated Netscape databases. This is often why AOL organic search results are not a 100% carbon copy of Google organic search results link for link.
Organic search results on Yahoo are determined by the Yahoo/Inktomi spider family.
The Yahoo spiders identify themselves on website server logs as:
Yahoo-Slurp
Inktomi-Slurp
Inktomi Archiver
IA-Archiver
YahooSeeker
also scan website server logs for additional Yahoo spider ips like this:
Inktomi
Fast Crawler v X
Fast Crawler v
UA Mozilla/4.0
UA Mozilla/4.05
UA Mozilla/5.0 Slurp/cat; slurp@inktomi.com
UA Mozilla/5.0
Yahoo! Slurp
YahooFeedSeeker/1.0
YahooSeeker-Testing/v3.9
YahooSeeker/1.1
YahooSeeker/1.2
For more information on Yahoo spiders visit: http://help.yahoo.com/help/us/ysearch/slurp/index.html
Organic search results on MSN are determined by the MSNBot spider family.
Many additional spider families also contribute to the organic keyword search results and is always serves websites best to be crawled often by as many spiders as possible. The Alexa/Amazon (www.A9.com) spiders are very helpful for sites especially ecommerce or shopping portals.
Some other leading spiders that also provide ancillary support, influence, and play a small but key role in determining organic search results are:
Lycos, InfoSeek, Alta Vista, Excite, Northern Light, Fast, and more.
Spider Track has great information and lists of spiders that are important in terms of Organic SEO for more information visit: http://www.spidertrack.org/index.php
---
Organic SEO + Spiders + Algorithms
The algorithms that are programmed to power the spiders or robot crawlers of the Big Four search engines (Google, Yahoo, MSN and AOL) have many similarities (thay all value HTML elements) however, contrary to popular belief the Google, Yahoo, and MSN spider families all are programmed with unique algorithms that have many differences.
A common misconception is that for the most part, all of the organic search spiders work the same way. It is true that organic search engine spiders are seeking text content and value URLs for content relevancy but it is not accurate to assume that spiders assign keyword ranking positions in the organic search results to webpages based exclusively on the number of backlinks.
If Google, Yahoo, or MSN were to weight backlinks so heavily what would prevent non-relevant sites with thousands of FFA Links (FFA: Free For All Link Farms) from securing and maintaining top keyword rankings on every possible keyword query.
Links Themes and content relevant links that tie pages from two sites together aid in serving users by helping to wrap and support a content theme are far more valuable and actually what the organic spider algortihtms are programmed to favor.
Google still assigns PageRank, however Google founders Larry Page and Sergey Brin have both recently stated that:
"PageRank is Dead"
In summary of their comments: "PageRank is broken" and Google is moving on.
As a public company Google has many more valuable (revenue-generating) programs on their plate to step backwards and make repairs to the non-revenue generating and outdated PageRank system.
---
The spiders that power organic search results receive algorithm updates often and require engineering management and human oversight. I will profile the algortihms, spiders, and the engineering teams that manage spider tasks in coming posts.
---
If Organic SEO is a goal contact the top ranking Organic SEO consulting firms and organic SEO consultants not SEM marketing companies that are sponsoring links and forced to pay for search advertising of their organic SEO services, but the seo consultants and organic seo companies that are ranking in top ORGANIC SEO keyword positions.
Remember that search engine spiders control Organic SEO and knowledge of all of these spiders and their algorithms is absolutly essential in organic search engine optimization.
Look for technology empowered companies that demonstrate their organic search engine optimization capabilities with organic SEO keyword ranking positions versus search marketing companies that are forced to pay for keyword exposure on keyword search phrases that best describe Organic SEO.
Also be wary of marketing-intensive search engine optimization companies that are not listed in top 20 organic keyword positions.
Many of these SEM advertising firms prefer to steer conversations away from algorithms and specific technology factors and instead continually encourage PPC activity; repeatedly inquiring about your PPC budget levels, exclusively promoting Pay Per Click search marketing as search engine optimization solutuons, despite the fact that keyword searchers prefer organic search results 7 to 1 vs. the sponsored links.
The most effective search engine optimization programs involve a mix of both Organic Search Engine Optimization (Organic SEO) and Pay Per Click Search Marketing (PPC SEM).
These most successful search engine optimization campaigns (organic SEO / PPC SEM) result in a majority of keyword targets driving new business leads, RFQ inquiries, and conversions through organic keyword positions that correspond directly with the content theme of the website.
If your website does not contain content that correlates with a target keyword set then use PPC search marketing for exposure on these keyword targets as new relevant content pieces are being developed by your marketing or public relations departments.
Dedicated content pieces are the key to long-term organic search engine optimization strength. Proven organic SEO consultants help companies with SEO copywriting and content development and also help companies technically optimize website foundations of all URLs cover to cover and these proven organic SEO consulting firms can be found in the top 20 organic keyword positions on: Google, Yahoo and MSN.
Focus on HTML code characterisitcs and spider indexing capabilities per url, establish a sound HTML foundation for your website and then apply content relevance. Websites with content-depth, technically sound URL code foundations, solid URL naming conventions, and strategically optimized dynamic databases are the sites that enjoy the best organic search engine placement.
More on Spiders and their role in Organic SEO in coming posts.
--
SEO Blog. Organic SEO Blog. Search Marketing News. SEO Done Right examines search engine optimization, the most effective form of internet marketing. Breaking SEO news and emerging developments at Google, Yahoo, and Bing. Leading Organic SEO Consultants Peak Positions debunk the many myths, hype, and spin related to SEO and search marketing.
