BIG CHANGES AHEAD FOR SEARCH ENGINES?

Part of The Alchemist's Lair Web Site
Maintained by Harry E. Pence, Professor of Chemistry, SUNY Oneonta, for the use of his students. Any opinions are totally coincidental and have no official endorsement, including the people who sign my pay checks. Comments and suggestions are welcome (pencehe@oneonta.edu).

Last Revised Oct. 9, 2003

YOU ARE HERE> Alchemist's Lair > Web Tutorial > Shakeup for Search Engines

(This article appeared in the Fall 2003 issue of the Computers in Chemical Education Newsletter.

Big Changes ahead for Search Engines, Harry E. Pence, SUNY Oneonta, Oneonta, NY, pencehe@oneonta.edu

Note to the reader: I have discussed the article with the Newsletter editor, and we have decided that some readers may find it to be helpful, even if it is (in the words of the author), "too nerdy for words." If you have little interest in the politics of search engines, you should probably skip it. Caveat Emptor!

INTRODUCTION

Regular readers of this on-line newsletter will recognize that one of the ongoing features has been a biennial report on the best search engines for chemistry (click here for the most recent evaluation), based on information gathered by the senior class at SUNY Oneonta. This discussion has focused mainly on five web search engines: Altavista, Hotbot (partially powered by Inktomi), FAST, Google, and MSNetwork. A number of major acquisitions in the past few months suggest that this roster may be in the process of changing. If so, the results would be important to anyone who uses one of these engines. Thus, an interim report seemed appropriate to keep users informed of what may be happening to their favorite engine. Of course, remembering the old adage that if you love sausage you should never go to see how it is made, some may feel that the less they know about the behind the scenes maneuvering the better. This article is not for them.

The Dominance of Google

For the past few years (eons in web time), Google has maintained a dominant position among web search engines. Danny Sullivan reports that as of August, 2003, Google is directly responsible for about 32% of the searches on the web, narrowly edging out Yahoo (28%), AOL(19%), and MSN (15%) for the honor of being the most widely used search engine on the WWW. Sullivan then goes on to point out that Google is much more dominant than these numbers indicate, because it is the search engine of choice on several other sites. As a result, Google may actually account for as much as three-fourths of the web search activity.

There are recurrent pronouncements that some new engine, like Wisenut, will replace Google as the industry standard, but so far this has failed to happen. When it became available in late 2001, Wisenut was celebrated as the next Google, but it's performance never seemed to match the hype. Recently, Wisenut has been purchased by LookSmart. Although Wisenut displays many pluses and will be kept in mind for future engine evaluations, its future continues to be murky. Danny Sullivan points out that it is unclear in the long run how well a crawler-based engine like Wisenut will fit with LookSmart, which uses human editors to compile directories. As will be noted below, recent developments with LookSmart make the future of Wisenut even more murky.

Essential Business Components of a WWW Search Engine

There are three essential components for the financial success of a web search engine. First is, of course, the web index and the search algorithm. The other two components are ownership of the portal site where the engine is located and the paid advertising links that are displayed with the search results. MSN and Yahoo own their own sites, but they had both contracted out the paid placements and they both used an independent engine to provide all or part of the search results. Google is the only engine that is in complete control of its services and so is more independent that any other search site.

The paid advertisements (often called contextual ads) that appear in search returns are probably the last thing that a user wants to hear about. To the user, they are a pain in the neck (or some other region of the anatomy), but for the company that runs the engine, they are a major source of revenue. Contextual ads appear when certain key words are used in a search. There are companies, like Overture or LookSmart, that exist primarily to provide contextual advertising for insertion into the results at sites like Yahoo or MSN. For example, if your company makes widgets, you can pay to make sure that every time a search mentions the word widget, an advertisement for your company will appear with the search results. These ads may either be provided separate from the search results (as Google does) or incorporated into the list of results (as Yahoo and MSN do). In the latter case, the ranking of the contextual ad is determined by how much the advertiser is willing to pay each time a user clicks on it; the more one pays the more prominent the position on the results page. This means that the first sites returned to the user may not be particularly relevant and are there only because someone has stacked the deck by paying for placement. This is another reason why most serious searchers prefer Google.

Engine Churning or There is a Buck to be Made Here

In 2002, Google challenged Overture by making deals with companies like AOL, Earthlink, and Ask Jeeves, that had previously been partners with Overture. This started a domino process. To simplify, Overture bought both Altavista and FAST, two of the major search engines. Yahoo, which in the past had contracted with Google to provide search results, purchased Inktomi, another of the major search engines, and finally purchased Overture. Thus, Yahoo now has control of three of the five major search engines as well as the major provider of contextual advertising. What does Yahoo plan to do with not one but three of the most powerful search engines on the web? More importantly, will Yahoo be willing to commit resources to three engines, or will one flourish while the others are ignored?

The wild card in this picture is MicrosoftÕs MSNetwork, which used contributions from Inktomi, Overture, and LookSmart. Now that both of its major competitors control a portal site, a search engine, and a source of contextual advertisements, MSN seemed to be in the weakest position of the three major engine sites. It is also the only one of the three that fails to turn a profit on searches (although Microsoft has such a huge cash reserve from other businesses that this probably is not much of a concern). This month, Microsoft announced that it would no longer use LookSmart to provide listings from its MSN search service. This represents a major financial blow to LookSmart and may mean that funding for Wisenut is questionable. (You do remember that LookSmart now owns Wisenut, don't you? ;-) Microsoft says that eliminating many of the contextual ads will make the results it returns more relevant, which is a plus for those who use MSN. Microsoft has also announced that it is developing its own search engine, presumably so that it will no longer depend on Inktomi, which is owned now by the competition, Yahoo. It seems late in the game to create a new search engine but remember how Microsoft overcame a late start to replace Netscape and Lotus 1-2-3.

Looking into the Crystal Ball (Clouded though it is)

So what does all this activity mean to the average chemist who wishes to search for chemical terms on the web? Those who are accustomed to using Google will probably be able to continue past habits with little change, but non-Google users should beware that their favorite engine may be changing and not necessarily for the better. The most significant changes may well be at Yahoo, since it now has the tools to compete more directly with Google. When the current acquisitions are complete, Yahoo will have control of three major search engines, all of which could be competitive with Google if they had access to more funds. Yahoo continues to show increasing profits, so it has the money and the competitive need to turn one of its acquisitions into a true Google killer. Anyone who is tempted by that advertising that is built on the new search capability should, however, remember that Yahoo still embeds contextual ads with legitimate search returns. Most serious searchers will find this to be a major liability. MSN should provide somewhat more relevant results than in the past now that it has separated from LookSmart, and the potential for a new Microsoft search engine represents another challenge to Google. On the other hand, the loss of revenue that this decision causes to LookSmart puts the future of Wisenut is in question. Anyone who uses Altavista, Inktomi, or AlltheWeb as their major search engine should watch carefully to see whether Yahoo continues to support all three of these.

What is coming next for search engines? Google has purchased both Kaltrix, a search engine that claims to make it easier to personalize searches for each individual, and Pyra Labs, which developed key software for web logs (or Blogs, as they are called). Both acquisitions may suggest that Google views its future as depending on more personalized search software, customized to each individual. This is an exciting direction, which could radically change the way that search engines are used, but that is a story for another time.

Acknowledgments

As noted in the earlier reports in this series, the Search Engine Watch web site, maintained by Danny Sullivan, and the Search Engine Showdownweb site, maintained by Greg Notess are excellent resources for anyone who wishes to keep up with current developments in WWW search engines. These sites are a major source for the information in this report.

Return to Web Tutorial Home Page at the Alchemist's Lair.

Return to the Fall 2003 issue of the Computers in Chemical Education Newsletter.)

You are the visitor to the Alchemist's Lair site since Jan. 10,1997.