MSN: MSN battles for search crown

Kent is a busy lady. But, while she concerns herself with what must be a complex handover, she is clear about the importance of search to her firm and the scale of Microsoft's ambitions. "We have been in the search game for a while, since 1998, but the market has consolidated into three major players: Google, ourselves and Yahoo!," she explains. "Google has claimed the lion's share of query volumes and has done a great job, but there is a lot we can do to get results quicker that are more relevant. A search on 'travel' returns 48 million results on Google - it's like finding a needle in a haystack. People want more relevant results that are more personal. It is an important play for MSN, a strategic play - we want to win the lion's share of search."

Aside from the volumes of traffic delivered by search, revenues through sponsored listings are an enormous coup for media owners and advertisers. The Interactive Advertising Bureau (IAB) reported in July that paid-for search listings totalled £82.9m in the second half of 2003, 41 per cent of the overall £202.1m spent on online advertising.

Seb Bishop, co-founder of pay-per-click search network Espotting, says: "More money is being pushed into search, not just in development but money from advertisers. Search has become a core part of most marketing budgets and a core part of everyone's lives."

MSN's commitment is therefore worrying for rivals, especially given that the firm's history reveals a reluctance to play second fiddle. In its core market - software - it is number one by a huge margin despite this year's anti-trust settlements. The $100m pledge makes clear its seriousness about search. It has poached key people from rivals, such as Paul Ryan, former chief technology officer of paid-for listings specialist Overture, who is credited with playing a key role in the company's rise to fame, culminating in a $1.6 billion (£1bn) takeover of the firm by Yahoo! last year.

In its bid to get ahead of Google and Yahoo!, people are expecting Microsoft to wrap search tools in with users' PC desktops - the space it 'owns' - to help it dominate the market. This will enable it to reach users at an earlier stage, without the need to open an internet browser. The firm has made plenty of noise about its push to personalise search, enabling its system to effectively 'learn' and adapt results based on users' past queries. This area is a Holy Grail for search engines and all major players have made steps in this direction in the past year.

Kent is confident. Following the release of MSN's new home page and results display in July (see box, p33), she claims it has the fastest-loading search home page on the web with a 36 per cent rise in searches through the web site. "The new search will come in the next 12 months, but we have already made improvements," she says. "We have a faster, cleaner home page that provides greater clarity - it is easier to see which results are advertiser links. Our search home page loads quickly - there are great links to our other services, such as Hotmail, and improved relevancy."

The company has also released an initial version of its new search technology. Its crawler (the automated tool search that engines use to find and update web sites to build their index) has been notably enthusiastic, while the index itself (on which the fight with Google, Yahoo! and Ask Jeeves will be based) has had a cool welcome from experts. Dixon Jones, head of internet marketing consultancy Receptional Marketing and moderator of the MSN message board on Webmaster World, says: "People have been complaining about how aggressive the crawler has been. It has been taking up so much bandwidth that site owners have had to pay more money to ISPs, but that is a sign of how serious Microsoft is and it has reacted by putting in delays. But, the general opinion is that it needs to do more in filtering results. There seems to be an awful lot of URL spamming and results are heavily keyword-based. The interface is looking very clean - they've clearly been looking at Google for what's good."

Jones' view is confirmed by a report from US investment bank Piper Jaffray (see box, p35), which viewed the new search technology at a recent analyst briefing in the US. The firm confirmed the speed of the new system and said the relevancy of results was encouraging. However, it added that, though on track, matching the relevancy of existing players would be an enormous task. It said: "To be sure, MSN does have a basic relevancy engine running, but it needs to be improved significantly."

Robin Kellett, MSN UK's search manager, agrees there is a long way to go, but he's confident: "This is a toddler that has just started to walk. We'll release the crawler fully when it meets its objectives; when it is better than what is currently out there. Our goal is to provide the best search experience on the net."

Producing a search index that matches MSN's rivals is just the first part of the challenge. Differentiating it from those same rivals is key and moving search to a new level a must. Piper Jaffray's report suggests: "The marketing task is possibly the biggest challenge; how to convince consumers who are perfectly happy with Google or Yahoo! to switch. To do this, MSN cannot simply offer a comparable or even somewhat better search experience - it needs a major point of differentiation, coupled with aggressive marketing." And Bishop agrees: "Building an index is yesterday's news. The question is: what will it bring out on top of that - how is it going to differ?"

MSN has made it clear that wrapping search in with users' PCs and providing more personalised results is the key. In explaining its plans, MSN bosses point to Newsbot - the news service it is beta testing (http://uk.newsbot.msn.com/). The service crawls the world's news pages seeking out stories to generate headlines, much like Google News. However, crucially, it learns which headlines a user clicks on and delivers stories on similar topics in a separate section, 'The Daily Me'. A similar system is a certainty for MSN's new search services.

Kellett explains: "It's an aggregator of more than 4,800 (news) services worldwide - there are different numbers in different countries, but it gives a clue as to the way search is going. It looks at what users have looked at in the past and delivers an automatic personal element - it's a first step towards a more personalised experience."

The chase for personalisation has already led most of the big players to trial localised search in the US. Systems try to mirror classified directories more closely using surfers' IP addresses to deliver results relative to location. Yet, Receptional Marketing's Jones feels Microsoft's position as the provider of 95 per cent of the world's PC operating systems could give it an advantage. "Every time you register your operating system you offer your address. It is difficult working out where you are in the world, but Microsoft could have a different opportunity to everyone else - points of difference that Yahoo! and Google can't copy."

Additionally, Yusuf Mehdi, the global chief of MSN, unveiled in July how far his company had come in developing its search tools. He revealed a system that would search a user's own files, computer, and emails, as well as the web, to produce comprehensive results from a user's entire 'computer world'. Enabling customers to do this will help the company to take the battle for users to the desktop, where it rules the roost, rather than the web, where Google and Yahoo! offer stiff competition. They, naturally, are seeking to do the opposite.

Jones explains: "Google's blue-sky objective is to try and take everything away from the desktop - even applications like word processing - and put it on the web, so it doesn't matter what operating system you're using or where you are in the world. It might be good for users but it also takes all the advantages MSN has away. The battleground will be 'are you searching from the web or from the desktop?' Both are trying to gather that information from their user base."

As the fight heats up, there is also one more issue for Microsoft to tackle. Its paid-for listings revenue is split with Yahoo!, MSN's fiercest rival, since its listings are provided by Overture - bought by Yahoo! last June. Overture's deal with MSN lasts into the summer of 2005 and both companies publicly state that they are very happy with the relationship. However, once the current agreement expires, an extension appears unlikely.

Jones comments: "MSN has been very, very quiet about PPC (pay-per-click search listings). It has a clear relationship with Overture and the message is that that is fine. But MSN is giving money to its major competitor - you have to assume it will either build its own technology or go out and buy it."

Kent confirms that PPC is "an area we are looking at" and, as Microsoft builds towards the launch of its new search engine, it is clear there is much still to be resolved. She has other 'projects' in mind just now, but winning the search battle is a fascinating one to leave in your in-tray.

MSN AIMS TO TRUMP RIVALS WITH REVAMPED SEARCH OFFERING

In July, MSN unveiled an overhauled search service with its own new-look home page and fresh design for the results listings.

The home page had a cleaner look and offered users the chance to search not only the web but also its news service, shopping directory, community services and Encarta encyclopaedia from one location.

The search results pages received the most radical redesign, with sponsored search listings moving from the main results to the right-hand side of pages. The number of sponsored links appearing at the top of the page were reduced to three.

MSN UK search manager Robin Kellett says the aim is to boost user experience and deliver more useful results: "User tests and research revealed that people want a clean, fast interface."

MSN expects the redesign to help it deliver better advertising return rates from sponsored listings.

Matt Whittingham, head of information services at MSN UK, says that, although paid inclusion links have been removed from the main search results, the clearer definition between paid-for links and algorithmic search results should improve click-through. "It is quite within the realms of possibility that the new interface will increase, rather than reduce clickthrough because we are providing a better experience for users," he suggests.

Gillian Kent, director of MSN UK, says the web site now has the fastest-loading search home page around and traffic on MSN's search has increased 36 per cent.

The new algorithmic search index being developed by MSN also aims to deliver more relevant results than rivals, with personalised results that are based on users' previous behaviour.

PIPER JAFFRAY REPORTS ON THE ROAD AHEAD

We attended the recent MSFT analyst day in the US to take a look at the latest from MSN. Demos included unique features combining desktop (PC) and web search. Their performance was exceptionally fast. MSN also showed pure search results from its own search engine, which is almost functional but not deployed.

MSN, in our opinion, is completing the first phase of building a good algorithmic search engine: build it to scale and create a large database (index). While this effort is quite complicated, it pales in comparison to the second stage: build a robust relevancy engine on top of the index.

To be sure, MSN does have a basic relevancy engine running but it needs to be improved. The difficulty is that the effort required to reach each incremental gain in relevancy is related to the relevancy improvements gained. Witness Yahoo! - it took a year for its Inktomi engine to close the gap on the last five per cent or so of difference with Google.

The next step for MSN is to close the relevancy gap and market to users. The marketing task is possibly the biggest challenge: how to convince consumers perfectly happy with Google or Yahoo! to switch. MSN cannot simply offer comparable or even somewhat better search experience - it needs a major point of differentiation and aggressive marketing.

Differentiation in search is not easy since all the major search engines are working on basically the same new products and features. It doesn't even matter who is first since consumers' loyalty is not by features or technology but by brand, and it takes a large effort to undermine a strong brand.

The third step involves integrating commerce search and the ad network with its algorithmic engine. We think MSN, upon launching its search product (Q4 2004 or Q1 2005) will deeply integrate Overture results while it works on the algorithm part. Eventually (2006 and beyond) MSN will develop its own ad network, replacing Overture/Yahoo! But it will take MSN a significant time to change the dynamic of the market.