Real Estate

Search Engine History: Web Search Before Google

Did Google always dominate the web search market? In the second of three posts on the history of search engines, I take a look at the pioneers of the early search market, including the first web crawler, WWW Wanderer. Did you know that Disney used to be one of the biggest players in the business? Or that Altavista was technically more advanced, in many ways, in 1998 than Google is now? Keep reading!

The pioneering web search engines

Really, the point where modern search engines start to appear is after the development and popularization of the MOSAIC browser in 1993. In 1994, Internet Magazine was released, along with a review of the top 100 websites listed as the ” more extensive”. ever ready to appear in a magazine. A 28.8 Kbps modem was priced at $399 and brought the Internet within reach of the masses (albeit slowly)!

At this point and for the next 4-5 years, it was almost possible to produce web-based and print directories of the best sites and for this to be useful information for consumers. However, the rapid growth in the number of www sites (from 130 in 1993 to more than 600,000 in 1996) began to make this effort seem as futile as producing printed yellow pages from every company, media, and library in the world.

While WAIS was not a lasting success, it highlighted the value of being able to search and click the full text of documents across multiple Internet hosts. The nascent Internet magazines and web directories further highlighted the challenge of being able to keep up with an Internet that was growing faster than any human being’s ability to catalog it.

In June 1993, Matthew Gray of MIT developed WWW Wanderer, a PERL-based web crawler. Initially, this was conceived simply as a tool to measure the growth of the world wide web using “gathering sites”. Later, however, Gray (now working for Google) used the crawled results to create an index called “Wandex” and added a search interface. In this way, Gray developed the world’s first web search engine and the first autonomous web crawler (an essential feature of all modern search engines).

While Wanderer was the first to send out a bot to crawl websites, it did not index the full text of the documents (as WAIS had done). The first search engine to combine these two essential ingredients was WebCrawler, developed in 1994 by Brian Pinkerton at the University of Washington. WebCrawler was the search engine where many of us early pioneers first traversed the web, and it will be fondly remembered for its (at the time) beautiful graphical interface and the incredible speed with which it returned results. 1994 also saw the launch of Infoseek and Lycos.

However, the scale of the web’s growth was beginning to put indexing beyond the reach of the average college IT department. The next big step required a capital investment. Enter, stage right, the (then huge) Digital Equipment Corporation (DEC) and its blazing-fast Alpha 8400 TurboLaser processor. DEC was an early adopter of web technologies and the first Fortune 500 company to establish a website. Its search engine, AltaVista, was launched in 1995.

Founded in 1957, DEC led the minicomputer market during the 1970s and 1980s. In fact, most of the machines that the first ARPANET hosts ran on were DEC-PDP-10 and PDP-11. However, by the early 1990s, DEC was a troubled business. In 1977, its then CEO, Ken Olsen, said that “there is no reason for a person to have a computer in his house.” Though out of context at the time, this quote was in part symptomatic of DEC’s slow response to the rise of personal computing and the client-server revolution of the 1980s.

At the time Altavista was being developed, the company was under siege from all sides by HP, Compaq, Dell, SUN, and IBM and was losing money like it was going out of style. Louis Monier and his research team at DEC were “discovered” internally as the latest public relations coup; the entire web captured – and searchable – on a single computer. What better way to showcase the company as an innovator and show off its new baby’s blazing-fast speed and 64-bit storage?

During 1995, Monier released a thousand web crawlers on the young web (an unprecedented achievement at the time). By December (site launch), Altavista had indexed more than 16 million documents comprising several billion words. In essence, Altavista was the first commercial powerhouse web-based search engine system. AltaVista enjoyed almost 300,000 visits on its first day alone, and within nine months, it was serving 19 million requests per day.

Altavista was, in fact, technically far ahead of its time. The search engine pioneered many technologies that took years for Google and others to catch up. The site included natural search queries, Boolean operators, machine translation services (babelfish), and image, video, and audio search. It was also very fast (at least at first) and (unlike other engines) well suited to indexing legacy Internet resources (and in particular the still popular UseNet newsgroups).

After Altavista, Magellan, and Excite (all launched in 1995), a host of other search engine companies debuted, including Inktomi & Ask Jeeves (1996) and Northern Light & Snap (1997). Google itself launched in 1998.

Of these early engines, each enjoyed its own enthusiastic following and a share of the then-nascent search market. Each also had their own relative strengths and weaknesses. Northern Light, for example, organized its search results into specific folders labeled by topic (something that arguably still needs improvement today) and gained a small but enthusiastic following as a result. Snap pioneered ranked search results, in part, by what people clicked on (something Yahoo! and Google are just toying with now!)

In January 1999 (at the start of the dot-com boom), the largest sites (in terms of market share) were Yahoo!, Excite, Altavista, and Disney, with 88% of all search engine referrals. Market share was not closely related to the number of indexed pages (with Northern Light, Altavista, and a then relatively unknown Google leading the pack):

Search engine Percentage of search references (December 1999)

yahoo! – 55.81%

Excite Properties (Excite, Magellan, and WebCrawler) – 11.81%

Altavista – 11.18%

Disney Search Properties (Infoseek & Go Network) – 8.91%

Lycos – 5.05%

Go To (Now Overture) – 2.76%

Complement / NBCi – 1.58%

MSN – 1.25%

north light

Leave a Reply

Your email address will not be published. Required fields are marked *