- February 2004
past few months we've seen some of the biggest
changes in the search engine scene in years. This
is a quick update on the changes at Yahoo, Google
and MSN search.
off to a search engine-marketing conference
at the end of the week. I'll probably send out
another newsletter within a few weeks as we
learn more about the still developing changes.
below I've listed the four major search sites
on the web. They account for over 90% of the
searches done in North America. (The percentage
numbers show the percentage of searches done
by US web surfers at that site or network of
web sites in November 2003.
MSN Search: 15%
All others: About 6%
the past year and half most of the results in
a search at Yahoo actually came from Google.
all been predicting that Yahoo was going to
drop Google and switch to the Inktomi search
engine, which Yahoo purchased last year. In
our testing over the past month or so we could
see results that look like they were from Inktomi
being interspersed in the results of queries
so this tended to confirm the change was imminent.
we were all surprised to learn that Yahoo has
begun rolling out a new search engine. It's
the beginning of a rollout that will take place
over the next few weeks.
says it developed a new search engine using
technology and engineering expertise it has
acquired. The new Yahoo search engine robot
is called Yahoo! Slurp (For those of you who
like to look at web stats you may see visits
by this robot in the logs).
our testing this week since the cutover started,
the new Yahoo engine looks to be very similar
to Inktomi. It indexes many of the same page
elements and displays very similar results as
Inktomi, which leads me to believe that the
new Yahoo search is largely based on Inktomi.
In fact, in our testing we see that the Yahoo
Search index is already quite large. We're seeing
Yahoo return about 50 to 75% of the number of
results that Google returns for a specific search.
Based on this we believe that the Yahoo search
database likely started with the Inktomi database.
It's just too big to have been built in just
a couple of months.
why would Yahoo bring out a "new"
search engine instead of just building on Inktomi,
which it owns? I believe it was largely a competitive
move. For almost two years now Yahoo has been
in the awkward position of displaying results
that come from another search engine, Google.
Many asked the question, “Why use Yahoo
when the results are mostly from Google anyway”?
I believe Yahoo didn't want to replace Google
with results from Inktomi since these same results
are being used by many others including major
competitor MSN Search (see MSN Search below).
now we have a new search engine on the scene
that we'll be learning about over the coming
of the first questions of course is “how
do you get into the Yahoo search engine”.
In our testing most of our clients already have
some number of pages indexed so you'll probably
start seeing click throughs from the new Yahoo
Yahoo also announced a new Paid Inclusion option.
We'll be learning the details of it and any
changes with the Yahoo Directory (which has
had a separate cost for inclusion for a couple
of years) over the next couple of weeks. Yahoo
now has a confusing array of inclusion programs
and rumors are circulating that there will be
some consolidation soon.
MSN and Inktomi
recently the “natural” results in
a search at MSN Search came from the LookSmart
directory with secondary results from the Inktomi
search engine. In mid January MSN dropped LookSmart
(i.e. for the most part as they have still been
showing some results from LookSmart).
We've all been predicting that Inktomi, which
already was an important search engine, would
become much more important when MSN dropped
LookSmart in Janaury followed by Yahoo replacing
Google with Inktomi. But now that Yahoo has
switched to the new Yahoo search engine instead
of Inktomi the picture has changed.
Inktomi has indeed become much more important
now that it provides most of the results at
MSN Search (MSN Search accounts for about 15%
of all web searches). But the question is how
long will MSN continue to use Inktomi?
MSN has been developing its own search engine
for some time now. About six weeks ago MSN stated
that they would likely not go public with the
new search engine until 2005.
However right after Yahoo announced that it
had switched to the new Yahoo search engine,
rather than Inktomi, MSN released a Beta test
of its new search engine.
So don't be too surprised if MSN switches over
to its own search engine much sooner than it
had previously stated.
Update on the Google
that Google results are no longer displayed
on Yahoo, Google has lost a major part of its
business. Before the switch Google results were
displayed on Google, AOL, and Yahoo accounting
for about 70% of all searches. Now Google provides
about 50% of the search results. They're still
the leader with Yahoo in second place at about
still watching the major changes at Google that
began in mid November as they continue to roll
become farily clear that Google is incorporating
the technology it acquired from its purchase
of Applied Semantics, which uses artificial
intelligence to examine language patterns in
an attempt to determine what people are searching
for. It appears that word patterns, including
“Stemming” (more below) are being
examined on web pages in the Google index.
last week a Google representative stated the
goal of a good search engine should be both
to understand what a document is really about,
and to understand (from a very short query)
what a user really wants. And then match those
things as well as possible.) Better semantic
understanding helps with both those prerequisites
and makes the matching easier.
a good example is stemming. Stemming is basically
SEO-neutral, because spammers can create doorway
pages with word variants almost as easily as
they can to optimize for a single phrase …
But webmasters who never think about search
engines don't bother to include word variants--they
just write whatever natural text they would
normally write. Stemming allows us to pull in
more good documents that are near-matches. The
example I like is [cert advisory]. We can give
more weight to www.cert.org/advisories/ because
the page has both "advisory" and "advisories"
on the page, and "advisories" in the
Standard stemming isn't necessarily a win for
quality, so we took a while and found a way
to do it better.
So yes, I think semantics and document/query
understanding will be more important in the
Google continues to roll out these changes.
Just this week we've seen significant changes
in the rankings of some client pages.
A Google representative will be providing information
and answering questions at the SEM conference
I'll be at later this week. More to come