Unlabelled History of Search Engines: From 1945 to Google Today Link - 2

Thursday, 31 October 2019

History of Search Engines: From 1945 to Google Today Link - 2

Paid Inclusion

As mentioned earlier, many general web directories charge a one time flat fee or annually recurring rate for listing commercial sites. Many shopping search engines charge a flat cost per click rate to be included in their databases.
As far as major search engines go, Inktomi popularized the paid inclusion model. They were bought out by Yahoo in December of 2003. After Yahoo dropped Google and rolled out their own search technology they continued to offer a paid inclusion program to list sites in their regular search results, but Yahoo Search Submit was ended at the end of 2009.

Pay Per Click

Pay per click ads allow search engines to sell targeted traffic to advertisers on a cost per click basis. Typically pay per click ads are keyword targeted, but in some cases, some engines may also add in local targeting, behavioral targeting, or allow merchants to bid on traffic streams based on demographics as well.
Pay per click ads are typically sold in an auction where the highest bidder ranks #1 for that keyword. Some engines, like Google and Microsoft, also factor ad clickthrough rate into the click cost. Doing so ensures their ads get clicked on more frequently, and that their advertisements are more relevant. A merchant who writes compelling ad copy and gets a high CTR will be allowed to pay less per click to receive traffic.
In 1996 an 18-year-old college dropout named Scott Banister came up with the idea of charging search advertisers by the click with ads tied to the search keyword. He promoted it to the likes of Yahoo!, but their (lack of) vision was corrupted by easy money, so they couldn't see the potential of search. The person who finally ran with Mr. Banister's idea was IdeaLab's Bill Gross.

Overture (Formerly GoTo)

Overture.
Overture, the pioneer in paid search, was originally launched by Bill Gross under the name GoTo in 1998. His idea was to arbitrage traffic streams and sell them with a level of accountability. John Battelle's The Search has an entertaining section about Bill Gross and the formation of overture. John also published that section on his blog.
“The more I [thought about it], the more I realized that the true value of the Internet was in its accountability,” Gross tells me. “Performance guarantees had to be the model for paying for media.”
Gross knew offering virtually risk-free clicks in an overheated and ravenous market ensured GoTo would takeoff. And while it would be easy to claim that GoTo worked because of the Internet bubble’s ouroboros-like hunger for traffic, the company managed to outlast the bust for one simple reason: it worked.
While Overture was wildly successful, it had two major downfalls which prevented them from taking Google's market position:
  • Destination Branding: Google allowed itself to grow into a search destination. Bill Gross decided not to grow Overture into one because he feared that would cost him distribution partnerships. When AOL selected Google as an ad partner, in spite of Google also growing out their own brand, that pretty much was the nails in the coffin for Overture being the premiere search ad platform.
  • Ad Network Efficiency: Google AdWords factors ad clickthrough rate into their ad costs, which ensures higher relevancy and more ad network efficiency. As of September 2006 the Overture platform (then known as Yahoo! Search Marketing) still did not fix that problem.
Those two faults meant that Overture was heavily reliant on it's two largest distribution partners - Yahoo! and Microsoft. Overture bought out AltaVista and AllTheWeb to try to win some leverage, but ultimately they sold out to Yahoo! on July 14, 2003 for $1.63 billion.

Google AdWords

Google AdWords launched in 2000. The initial version was a failure because it priced ads on a flat CPM model. Some keywords were overpriced and unaffordable, while others were sold inefficiently at too cheap of a price. In February of 2002, Google relaunched AdWords selling the ads in an auction similar to Overture's, but also adding ad clickthrough rate in as a factor in the ad rankings.
Affiliates and other web entrepreneurs quickly took to AdWords because the precise targeting and great reach made it easy to make great profits from the comfort of your own home, while sitting in your underwear :)
Over time, as AdWords became more popular and more mainstream marketers adopted it, Google began closing some holes in their AdWords product. For example, to fight off noise and keep their ads as relevant as possible, they disallowed double serving of ads to one website. Later they started looking at landing page quality and establishing quality based minimum pricing, which squeezed the margins of many small arbitrage and affiliate players.
Google intends to take the trackable ad targeting allowed by AdWords and extend it into other mediums. Google has already tested print and newspaper ads. Google allows advertisers to buy graphic or video ads on content websites. On January 17, 2006, Google announced they bought dMarc Broadcasting, which is a company they will use to help Google sell targeted radio ads.
On September 15, 2006, Google partnered with Intuit to allow small businesses using QuickBooks to buy AdWords from within QuickBooks. The goal is to help make local ads more relevant by getting more small businesses to use AdWords.
On March 20, 2007, Google announced they were beta testing creating a distributed pay per action affiliate ad network. On April 13, 2007 Google announced the purchase of DoubleClick for $3.1 billion.

Google AdSense

On March 4, 2003 Google announced their content targeted ad network. In April 2003, Google bought Applied Semantics, which had CIRCA technology that allowed them to drastically improve the targeting of those ads. Google adopted the name AdSense for the new ad program.
AdSense allows web publishers large and small to automate the placement of relevant ads on their content. Google initially started off by allowing textual ads in numerous formats, but eventually added image ads and video ads. Advertisers could chose which keywords they wanted to target and which ad formats they wanted to market.
To help grow the network and make the market more efficient Google added a link which allows advertisers to sign up for AdWords account from content websites, and Google allowed advertisers to buy ads targeted to specific websites, pages, or demographic categories. Ads targeted on websites are sold on a cost per thousand impression (CPM) basis in an ad auction against other keyword targeted and site targeted ads.
Google also allows some publishers to place AdSense ads in their feeds, and some select publishers can place ads in emails.
To prevent the erosion of value of search ads Google allows advertisers to opt out of placing their ads on content sites, and Google also introduced what they called smart pricing. Smart pricing automatically adjusts the click cost of an ad based on what Google perceives a click from that page to be worth. An ad on a digital camera review page would typically be worth more than a click from a page with pictures on it.
Google was secretive about its revenue share since the inception of AdSense, but due to a lawsuit in Italy Google feared they would be stuck disclosing their revenue share, so they decided to do so publicly for good public relations on May 24, 2010. Google keeps 32% while giving publishers 68% of contextual ad revenues. On search ads Google keeps 49% and gives publishers 51%. Some premium publishers are able to negotiate higher rates & custom integration options as well.

Yahoo! Search Marketing / Gemini

Yahoo! Search Marketing is the rebranded name for Overture after Yahoo! bought them out. As of September 2006 their platform is generally the exact same as the old Overture platform, with the same flaws - ad CTR not factored into click cost, it's hard to run local ads, and it is just generally clunky.
Yahoo! launched another ad platform named Gemini in 2014, however they only gave it a small share of their overall search inventory & have relied heavily on Bing Ads for text ads and Google for Product Listing Ads in their search results.

Microsoft AdCenter / Bing Ads

In 2000 Microsoft launched a keyword driven ad program called keywords, but shut it down after 2 months because they feared it would cannibalize their banner ad revenues.
Microsoft AdCenter was launched on May 3. 2006. While Microsoft has limited marketshare, they intend to increase their marketshare by baking search into Internet Explorer 7. On the features front, Microsoft added demographic targeting and dayparting features to the pay per click mix. Microsoft's ad algorithm includes both cost per click and ad clickthrough rate.
Microsoft also created the XBox game console, and on May 4, 2006 announced they bought a video game ad targeting firm named Massive Inc. Eventually video game ads will be sold from within Microsoft AdCenter.
Windows 10 pushes Bing as a default search tool aggressively, which in turn boosts distribution for Bing search & Bing Ads.

Search Engine Optimization

What is SEO?

Search engine optimization is the art and science of publishing information in a format which will make search engines believe that your content satisfies the needs of their users for relevant search queries. SEO, like search, is a field much older than I am. In fact, it was not originally even named search engine optimization, and to this day most people are still uncertain where that phrase came from.

Early SEO

Early search engine optimization consisted mostly of using descriptive file names, page titles, and meta descriptions. As search advanced on the page factors grew more important and then people started trying to aim for specific keyword densities.

Link Analysis

One of the big things that gave Google an advantage over their competitors was the introduction of PageRank, which graded the value of a page based on the number and quality of links pointing at it. Up until the end of 2003 search was exceptionally easy to manipulate. If you wanted to rank for something all you had to do was buy a few powerful links and place the words you wanted to rank for in the link anchor text.

Search Gets More Sophisticated

On November 15, 2003 Google began to heavily introduce many more semantic elements into its search product. Researchers and SEO's alike have noticed wild changes in search relevancy during that update and many times since then, but many searchers remain clueless to the changes.
Search engines would prefer to bias search results toward informational resources to make the commercial ads on the search results appear more appealing. You can see an example of how search can be biased toward commercial or informational resources by playing with Yahoo! Mindset.

Curbing Link Spam

On January 18, 2005, Google, MSN, and Yahoo! announced the release of a NoFollow tag which allows blog owners to block comment spam from passing link popularity. People continued to spam blogs and other resources, largely because search engines may still count some nofollow links, and largely because many of the pages they spammed still rank.
Since 2003 Google has came out with many advanced filters and crawling patterns to help make quality editorial links count more and depreciate the value of many overtly obvious paid links or other forms of link manipulation.

Historical, Editorial, & Usage Data

Older websites may be given more trust in relevancy algorithms than newer websites (just existing for a period of time is a signal of quality). All major search engines use human editors to help review content quality and help improve their relevancy algorithms. Search engines may factor in user acceptance and other usage data to help determine if a site needs reviewed for editorial quality and to help determine if linkage data is legitimate. Navigational or branded search queries may be used as a key signal to authenticate if a site's link profile was manufactured or authentic. Sites which people repeatedly seek out by name are sites which generally provide a good user experience.
Google has also heavily pushed giving away useful software, tools, and services like free maps, email, web browser & mobile phone OS which allow them to personalize search results based on the searcher's historical preferences.

Hypocrisy in Search

Google engineer Matt Cutts frequently comments that any paid link should have the nofollow attribute applied to it, although Google hypocritically does not place the nofollow attribute on links they buy. They also have placed their ads on the leading Warez site and continued to serve ads on sites that they banned for spamming. Yahoo! Shopping has also been known to be a big link buyer.
Much of the current search research is based upon the view that any form of marketing / promotion / SEO is spam. If that was true, it wouldn't make sense that Google is teaching SEO courses, which they do.

Self Reinforcing Market Positions

In many verticals search is self reinforcing, as in a winner take most battle. Jakob Nielsen's The Power of Defaults notes that the top search result is clicked on as often as 42% of the time. Not only is the distribution and traffic stream highly disproportionate, but many people tend to link to the results that were easy to find, which makes the system even more self reinforcing, as noted in Mike Grehan's Filthy Linking Rich.
A key thing to remember if you are trying to catch up with another website is that you have to do better than what was already done, and significantly enough better that it is comment worthy or citation worthy. You have to make people want to switch their world view to seeing you as an authority on your topic. Search engines will follow what people think.

Google

Google from 1998.

Early Years

Google's corporate history page has a pretty strong background on Google, starting from when Larry met Sergey at Stanford right up to present day. In 1995 Larry Page met Sergey Brin at Stanford.
By January of 1996, Larry and Sergey had begun collaboration on a search engine called BackRub, named for its unique ability to analyze the "back links" pointing to a given website. Larry, who had always enjoyed tinkering with machinery and had gained some notoriety for building a working printer out of Lego™ bricks, took on the task of creating a new kind of server environment that used low-end PCs instead of big expensive machines. Afflicted by the perennial shortage of cash common to graduate students everywhere, the pair took to haunting the department's loading docks in hopes of tracking down newly arrived computers that they could borrow for their network.
A year later, their unique approach to link analysis was earning BackRub a growing reputation among those who had seen it. Buzz about the new search technology began to build as word spread around campus.
BackRub ranked pages using citation notation, a concept which is popular in academic circles. If someone cites a source they usually think it is important. On the web, links act as citations. In the PageRank algorithm links count as votes, but some votes count more than others. Your ability to rank and the strength of your ability to vote for others depends upon your authority: how many people link to you and how trustworthy those links are.
In 1998, Google was launched. Sergey tried to shop their PageRank technology, but nobody was interested in buying or licensing their search technology at that time.

Winning the Search War

Later that year Andy Bechtolsheim gave them $100,000 seed funding, and Google received $25 million Sequoia Capital and Kleiner Perkins Caufield & Byers the following year. In 1999 AOL selected Google as a search partner, and Yahoo! followed suit a year later. In 2000 Google also launched their popular Google Toolbar. Google gained search market share year over year ever since.
In 2000 Google relaunched their AdWords program to sell ads on a CPM basis. In 2002 they retooled the service, selling ads in an auction which would factor in bid price and ad clickthrough rate. On May 1, 2002, AOL announced they would use Google to deliver their search related ads, which was a strong turning point in Google's battle against Overture.
In 2003 Google also launched their AdSense program, which allowed them to expand their ad network by selling targeted ads on other websites.

Going Public

Google used a two class stock structure, decided not to give earnings guidance, and offered shares of their stock in a Dutch auction. They received virtually limitless negative press for the perceived hubris they expressed in their "AN OWNER'S MANUAL" FOR GOOGLE'S SHAREHOLDERS. After some controversy surrounding an interview in Playboy, Google dropped their IPO offer range from $85 to $95 per share from $108 to $135. Google went public at $85 a share on August 19, 2004 and its first trade was at 11:56 am ET at $100.01.

Verticals Galore!

In addition to running the world's most popular search service, Google also runs a large number of vertical search services, including:

Just Search, We Promise!

Google's mission is to organize the world's information and make it universally accessible and useful.
However that statement includes many things outside of the traditional mindset of search, and Google maintains that ads are a type of information. This other information includes:

Paying for Distribution

In addition to having strong technology and a strong brand Google also pays for a significant portion of their search market share.
On December 20, 2005 Google invested $1 billion in AOL to continue their partnership and buy a 5% stake in AOL. In February 2006 Google agreed to pay Dell up to $1 billion for 3 years of toolbar distribution. On August 7, 2006, Google signed a 3 year deal to provide search on MySpace for $900 million. On October 9, 2006 Google bought YouTube, a leading video site, for $1.65 billion in stock.
Google also pays Mozilla and Opera hundreds of millions of dollars to be the default search provider in their browsers, bundles their Google Toolbar with software from Adobe and Sun Microsystems, and pays AdSense ad publishers $1 for Firefox + Google Toolbar installs, or up to $2 for Google Pack installs.
Google also builds brand exposure by placing Ads by Google on their AdSense ads and providing Google Checkout to commercial websites.
Google Pack is a package of useful software including a Google Toolbar and software from many other companies. At the same time Google helps ensure its toolbar is considered good and its competitors don't use sleazy distribution techniques by sponsoring StopBadware.org.
Google's distribution, vertical search products, and other portal elements give it a key advantage in best understanding our needs and wants by giving them the largest Database of Intentions.

Editorial Partnerships

They have moved away from a pure algorithmic approach to a hybrid editorial approach. In April of 2007, Google started mixing recent news results in their organic search results. After Google bought YouTube they started mixing videos directly in Google search results.

Webmaster Communication

Since the Florida update in 2003 Google has looked much deeper into linguistics and link filtering. Google's search results are generally the hardest search results for the average webmaster to manipulate.
Matt Cutts, Google's former lead engineer in charge of search quality, regularly blogged about SEO and search. Google also has an official blog. Matt Cutts went on leave with Google in 2014 & officially resigned from the company at the end of 2016.
Google also helps webmasters understand how Google is indexing their site via Google Webmaster Central. Google continues to add features and data to their webmaster console for registered webmasters while obfuscating publicly available data.
For an informal look at what working at Google looked like from the inside from 1999 to 2005 you might want to try Xooglers, a blog by former Google brand manager Doug Edwards.

Information Retrieval as a Game of Mind Control

In October of 2007 Google attempted to manipulate the public perception of people buying and selling links by announcing that they were going to penalize known link sellers, and then manually editing the toolbar PageRank scores of some well known blogs and other large sites. These PageRank edits did not change search engine rankings or traffic flows, as the PageRank update was entirely aesthetic.

Increasing The Rate of Algorithmic Change

  • In 2009 Google did an update named the "Vince" update, which placed weight on search query chains, and had the net effect of promoting larger branded websites.
  • In 2011 Google rolled out their Panda algorithm in an attempt to make it harder to pour low quality content into well linked sites, by allowing the weaker pages to pull down the sites as a whole.
  • In 2012 Google announced the Penguin update aimed at making search spamming harder. In 2012 they also rolled out tigher anchor text filters, an algorithm to penalize ad heavy sites, and launched a ranking factor based on how many valid DMCA requests there are against a site.
The net effect of these new algorithms & other forms of obfuscation Google has introduced has been to make it much harder to rank independent websites owned by small companies, while making SEO easier for large companies that have significant usage signals associated with their websites. This has caused many SEO professionals to chase after servicing large corporate clients, as talent tends to follow the money.

Yahoo!

1995 Yahoo! Directory.

Getting Into Search

Yahoo! was founded in 1994 by David Filo and Jerry Yang as a directory of websites. For many years they outsourced their search service to other providers, considering it secondary to their directory and other content features, but by the end of 2002 they realized the importance and value of search and started aggressively acquiring search companies.
Overture purchased AllTheWeb and AltaVista in 2003. Yahoo! purchased Inktomi in December, 2002, and then consumed Overture in July, 2003, and combined the technologies from the various search companies they bought to make a new search engine. Yahoo! dumped Google in favor of their own in house technology on February 17, 2004.

Getting Social

In addition to building out their core algorithmic search product, Yahoo! has largely favored the concept of social search.
On March 20, 2005 Yahoo! purchased Flickr, a popular photo sharing site. On December 9, 2005, Yahoo! purchased Del.icio.us, a social bookmarking site. Yahoo! has also made a strong push to promote Yahoo! Answers, a popular free community driven question answering service.
On July 2, 2007, Yahoo! launched their behaviorally targeted SmartAds product.
On July 29, 2009, Yahoo! decided to give up on search and signed a 10 year deal to syndicate Bing ads and algorithmic results on their website.
Yahoo! shut down their directory service in December of 2014.
In 2014 Yahoo! signed a deal to be the default search provider in Mozilla Firefox inside the United States. They also did a distribution deal with Oracle, however those revenue gains were short lived & Yahoo kept losing share in online advertising & web search.
Over the years Yahoo! not only exited the search business, but they also exited most of their other vertical businesses. The role of the general purpose web portal was relegated to irrelevancy through the combination of:
  • general purpose search engines like Google adding interactive features & rich answers directly to their search results
  • people spending a greater share of their idle web time on social networks like Facebook which offer a highly personalized news feed
  • the rise of ad blockers & declining CPM rates for display ads
  • thin outsourced vertical offerings being inferior to the third party data sources & brands which powered them
Verizon announced they were acquiring the Yahoo! operating business in July of 2016 for $4.83 billion.

Microsoft

In 1998 MSN Search was launched, but Microsoft did not get serious about search until after Google proved the business model. Until Microsoft saw the light they primarily relied on partners like Overture, Looksmart, and Inktomi to power their search service.
They launched their technology preview of their search engine around July 1st of 2004. They formally switched from Yahoo! organic search results to their own in house technology on January 31st, 2005. MSN announced they dumped Yahoo!'s search ad program on May 4th, 2006.
Live Search.
On September 11, 2006, Microsoft announced they were launching their Live Search product.
Bing.
On June 1, 2009, Microsoft launched Bing, a new search service which changed the search landscape by placing inline search suggestions for related searches directly in the result set. For instance, when you search for credit cards they will suggest related phrases like
  • credit card types
  • apply for credit cards
  • credit cards for bad credit
  • advice on credit cards
Microsoft released a Bing SEO guide for Webmasters [PDF] which claimed that the additional keyword suggestions helped pull down search demand to lower listed results when compared against the old results 6 through 10 when using a single linear search result set. Conversely, the Google format tends to concentrate attention on the top few search listings. After extensive eye tracking Gord Hotchkiss named this pattern Google's Golden Triangle.
Eye Tracking Studies.
While Yahoo! has lost much of their relevance, Bing has built a formidable Google search competitor. They have narrowed the revenue gap against Google & have built a profitable search business.
Bing is strongest in the US market, while having a lower share outside of the US, in part due to Google driving aggressive installs of Google Chrome from Flash security updates & promoting Chrome across Google properties & the AdSense ad network.
Google is more dominant in moble search than they are in desktop due to
  • requiring Android phone manufacturers to set Google as the default search provider
  • signing a deal with Apple to be the default web search service in Safari on iPhones, iPads & Mac computers.

Other Engines

One would be foolish to think that there is not a better way to index the web, and a new creative idea is probably just under our noses. The fact that Microsoft is making a large investment into developing a new search technology should be some cause for concern for other major search engines.
Through this course of history many smaller search engines have came and went, as the search industry has struggled to find a balance between profitability and relevancy. Some of the newer search engine concepts are web site clustering, semantics, and having industry specific smaller search engines / portals, but search may get attacked from entirely different angles.
On October 5, 2004 Bill Gross ( the founder of Overture and pioneer of paid search) relaunched Snap as a search engine with a completely transparent business model (showing search volumes, revenues, and advertisers). Snap has many advanced sorting features but it may be a bit more than what most searchers were looking for. People tend to like search for the perceived simplicity, even if the behind the scenes process is quite complex.
Outside of technology there are four other frontiers search is being attacked / commoditized from
  • Browser & Software Distribution: Search companies are paying computer manufacturers or software companies an aggregated value of hundreds of millions or billions of dollars each year to bundle their search toolbar with their products.
  • Social Search: Large social networks have significant reach and a ton of page views. Many years ago Yahoo! wass rumored to express interest in buying social network Facebook nearly a billion dollars. Yahoo! has already bought social picture site Flickr and social bookmarking site Del.icio.us. In August of 2006 Google signed a 3 year $900 million contract to provide search and advertising on MySpace.

    In addition some companies, like Eurekster, are trying to create products which allow groups of webmasters to make topic or community specific search services.
  • Content Providers: Some content providers are trying to publish content on their own domains and build off their brand. Some are refusing to be included in search indexes. Some are requiring a kickback to be indexed. Some are unsure of what they want and are choosing to sue search engines, either for further brand exposure, or to gain further negotiation leverage.
  • Content Aggregators: Search is just one way of finding information. Via RSS feeds and various other technologies many sites are offering what some people consider persistent search, or a way to access any information about a specific topic as it becomes available. Google also bought YouTube for $1.65 in stock. YouTube consists largely of pirated content which Google can organize and publish ads against based on usage data and other forms of ad targeting.
Some early search pioneers have tried to reboot search, but most these efforts have failed to gain a sustainable marketshare.
Cuil was heavily hyped but quickly bust. Blekko launched with less hype & lasted longer, but ultimately sold to IBM. Gigablast was founded in 2000 by Matt Wells. They are an open source search engine which has quietly existed for nearly 2 decades. Gabriel Weinberg founded DuckDuckGo in 2008. It leverages the core Bing index but differentiates through the search interface & result features. They have done a great job of consistently growing off a small base & is popular with many web developers in part for their search privacy features & lack of result personalization.
Some foreign markets have dominant local search services. Yandex is big in Russia. Baidu leads China. Naver is popular in South Korea.

Search & Legal Issues

Privacy

In 2005 the DoJ obtained search data from AOL, MSN, and Yahoo!. Google denied the request, and was sued for search data in January of 2006. Google beat the lawsuit and was only required to hand over a small sample of data.
In August of 2006 AOL Research released over 3 months worth of personal search data by 650,000 AOL users. A NYT article identified one of the searchers by name. In 2007 the European Union aggressively probed search companies aiming to limit data retention and maintain searcher privacy rights.

Publishing & Copyright Lawsuits

As more people create content attention is becoming more scarce. Due to The Tragedy of the Commons many publishing businesses and business models will die. Many traditional publishing companies enjoyed the profits enabled by running what was essentially regionally based monopolies. Search, and other forms of online media, allow for better targeting and less wasteful / more efficient business models. Due to growing irrelevancy, a fear of change, and a fear of disintermediation, many traditional publishing companies have fought search.
In an interview by Danny Sullivan, Eric Schmidt stated he thought many of the lawsuits Google face are business deals done in a court room.

Newspapers

In September of 2006 some Belgian newspaper companies won a copyright lawsuit against Google News which makes Belgium judges look like they do not understand how search or the internet work. Some publisher groups are trying to create an arbitrary information access protocol, Agence France Presse (AFP) sued Google to get them to drop their news coverage, and Google paid the AP a licensing fee.
In early 2017 the Wall Street Journal opted out of Google's first-click free program. In turn they saw a 44% decline in organic search traffic & the WSJ saw a fourfold increase in the rate of visitors converting to paying subscribers.

Books

In September of 2005 the Authors Guild sued Google. In October of 2005 major book publishing houses also sued Google over Google Print.

Photos

Perfect 10, a pornography company, sued Google for including cached copies of stolen content in their image index, and for allowing publishers to collect income on stolen copyright content via Google AdSense.

Access to Hate Information

In May of 2000 a French judge required Yahoo! to stop providing access to auctions selling Nazi memorabilia.
Many requests for information removal are covered on Chilling Effects and by the EFF. Eric Goldman tracks these cases as well.

Pay Per Click & Ad Targeting Lawsuits

In 1999 Playboy sued Excite and Netscape for selling banner impressions sold for searches for Playboy.
Overture sued Google for patent infringement. Just prior to Google's IPO they settled with Yahoo! (who by then bought out Overture) by giving them 2.7 million shares of class A Google stock.
Geico took Google to court in the US for trademark violation because Google allowed Geico to be a keyword trigger to target competing ads. Geico lost this case on December 15, 2004. Around the same time Google lost a similar French trademark case filed by Louis Vuitton.
Lane's Gifts sued Google for click fraud, but did not have a strong well put together case. Google's lawyers pushed them into a class wide out of court settlement of up to $90 million in AdWords credits. The March 2006 settlement aimed to absolve Google of any clickfraud related liabilities back through 2002, when Google launched their pay per click model.
Ads for Shady Products / ServicesIn 2004 search engines in the United States stopped running ads for online casinos.
In 2009 a US federal government sting operation busted Google for running ads promoting the illegal sale of steroids.
The Chinese search engine Baidu faced domestic regulatory scrutiny after public outrage in response to a kid with cancer dying & blaming his death on a bogus medical procedure marketed via Baidu ads. The impact on Baidu ad sales was major, driving a year over year decline in spite of rapid growth in web usage inside China. Baidu was also investigated for running stealth gambling ads at night.

Search User Information

The US government requested that major search companies turned over a significant amount of search related data. Yahoo!, MSN, and AOL gave up search data. The Google blog announced that Google fought the subpoena
In August, Google was served with a subpoena from the U. S. Department of Justice demanding disclosure of two full months’ worth of search queries that Google received from its users, as well as all the URLs in Google’s index.
AOL not only shared information with the government, but AOL research also accidentally made search records public record.
After the AOL data leak fiasco & news of the NSA spying program most major search engines began encrypting user searches with secured search sessions. DuckDuckGo took things one step further by promising not to track or follow a user, or personalize search reuslts.
Google aggressively tracks Android users & was also fined $22.5 million by the FTC for overriding privacy features in Apple's Safari web browser.

Monopoly

On June 27, 2017 the European Commission fined Google €2.42 billion for breaching EU antitrust rules. The specific fine was for Google's preferential placement of their shopping search results & the European Commission is still investigating multiple other issues with Google including bundling defualt Android search placement.

Search as a Commoditizer

Each search company has its own business objectives and technologies to define relevancy. The three biggest issues search engines are fighting are
  • Publishing Rights: All search engines are fighting trying to gain the rights to index quality content. Some of the highest quality content is so expensive to create and market that there is not a business model for openly sharing it on the web. Worse yet, as more and more people get into web publishing the businesses that delay to get their content indexed will have lost authority and distribution the whole time they delayed. This, and the fear of disintermediation, are part of the reason there are so many lawsuits.
  • Distribution: The more distribution you have the more profit you can use to leverage the ability to buy more content or make better content partnerships. Also more distribution means that you can potentially send more visitors (and thus profit) to a person who lets you index their content. More usage data may also help engines improve their relevancy algorithms.
  • Ad Network Size & Efficiency: Efficient ad networks can afford to pay for more distribution, and thus help the search company gain more content and distribution.
In order to try to lock users in search engines offer things like free email, news search, blogging platform, content hosting, office software, calendars, and feature rich toolbars. In some cases the software or service is not only free, but it is expensive to provide. For example, Google does not profit from Google news, but they had to pay the AP content licensing fees, and hosting Google Video can't be cheap.
In an attempt to collect more data, better target ads, and improve conversion rates Google offers
  • a free analytics product
  • free cross platform tracking
  • free Wifi internet access in San Francisco and Mountainview
  • a free wallet product which makes it quick and easy to buy products
The end goal of search is to commoditize the value of as many brands and markets as possible to keep adding value to the search box. They want to commoditize the value of creating content and increase the value of spreading ideas, the value of attention, and the importance of conversion.
As they make the network more and more efficient they can eat more and more of the profits, which was a large part of the reasoning behind Jakob Nielson's Search Engines as Leeches on the Web.

Selling Search as an Ecosystem

Because search aims to gain distribution by virtually any means possible the search engines that can do the best job of branding and get people to believe most in their goals / ideals / ecosystem win. Search engines are fighting many ways on this front, but not all of them are even on the web. For example, search engines are trying to attract the smartest minds by sharing research. Google goes so far as offering free pizza!
Google hires people to track webmaster feedback across the web. Matt Cutts frequently blogs about search and SEO because to him it is important for others to see search, SEO, and Google from his perspective. He offers free tips on Google Video in no small part because it was important for Google Video to beat out YouTube for Google to become the default video platform on the web. Once it was clear that Google lost the video battle to YouTube Google decided to buy them.
Beyond just selling their company beliefs and ideology to get people excited about their field, acquire new workers, and get others to act in a way that benefits their business model search engines also provide APIs to make portions of their system open enough that they can leverage the free work of other smart, creative, and passionate people.
Selling search as an ecosystem goes so far that Google puts out endless betas, allowing users to become unpaid testers and advocates of their products. Even if the other search engines matched Google on relevancy they still are losing the search war due to Google's willingness to take big risks, Google's brand strength, and how much better Google sells search as an ecosystem.

Extending Search

Google wants to make content ad supported and freely accessible. On October 9, 2006, Google announced they were acquiring YouTube for $1.65 billion in stock. In March, 2007,Viacom sued Google / YouTube for $1 billion for copyright infringement. In 2007 Microsoft pushed against Google's market position calling Google a copyright infringer (for scanning books) and doing research stating that many of Google's blogspot hosted blogs are spam.

Social Search

In 2006 and 2007 numerous social bookmarking and decentralized news sites became popular. Del.icio.us, a popular social bookmarking site, was bought out by Yahoo. Digg.com features fresh news and other items of interest on their home page based on user votes.

Text REtrieval Conference (TREC):

In 1992 TREC was launched to support research within the information retrieval community by providing the infrastructure necessary for large-scale evaluation of text retrieval methodologies. In addition to helping support the evolution of search they also create special tracks for vertical search and popular publishing models. For example, in 2006 they created a blog track. Past TREC publications are posted here.

Other Search Conferences:

There are a number of other popular conferences covering information retrieval.
Search Science lists a number of conferences on the right side of the Search Science blog.
There are also a number of conferences which talk about search primarily from a marketer's perspective. The three most well known conferences for are

Sources and Further Reading:

Many of the following have not been updated in years, or only cover a partial timeline of the search space, but as a collection they helped me out a lot. SearchEngineWatch is amazingly comprehensive if you piece together all of the articles Danny Sullivan has published.

Books About Search

  • The Search - John Battelle's book about the history of search and how search intersects with media and culture.
  • The Google Story - David Vise's book about Google.
  • The Google Legacy - Stephen E. Arnold's book about why he believes Google is in a better market position than its competitors
  • In The Plex - Steven Levy's book about Google.

Infographics


Share this