eBay has started rolling out Best Match in earnest on the core eBay.com site, and boy is it getting noticed.
First, here is the original post on eBay that announced the test of Best Match as the default sort in five major categories, dated January 16th. Just a few days ago, really.
I caught this blog post over the weekend from Randy Smythe, and realized that I had a few things to say about the launch of this test.
The first of which is congratulations to the eBay Finding team. The launch of this test represents an inevitable step towards the future of a search engine on the eBay.com site optimized for the best possible buyer experience. For all the back-seat driving and Monday morning quarterbacking that they receive, very few people understand the complexity of the problems that the eBay Finding team has to tackle.
The second thing I have to say here is get ready to drink from the firehose. This move is bigger than anything I can think of in the history of the eBay buyer experience, and it’s going to test eBay and the eBay community in new ways. There is no playbook for this type of change, there is no simple pattern match. There is going to be a lot of churn, a lot to learn, and lot of quick action & analysis needed to make this successful.
It might not seem obvious to outsiders how big a change this really is. But believe me, it’s huge. There is a $60 Billion economy that is all predicated on the way that hundreds of millions of buyers search through and find billions of items for sale on eBay. That’s roughly the Gross Domestic Product of the country of Vietnam.
To explain why this change is so dramatic, let me explain a bit of the background behind this change. Let’s start with how eBay search works today.
eBay search has a history of being extremely literal and transparent. Until changes were made in the last few years, eBay search would literally do only the following:
- Look at the keywords entered by the buyer
- Look at the title keywords of every listing on the site
- Return only the listings that had 100% of the keywords entered by the buyer
- Sort the listings by “time remaining”
When I worked on the eBay Finding team, it was always surprising to me how many active eBay users I would talk to, both buyers and sellers, who assumed there was “something more” to the way eBay returned items. In fact, I would sometimes ask potential product managers, interviewing at eBay, to describe how they thought the eBay search engine worked. I would get the correct answer less than 10% of the time.
This system had some clear and obvious benefits. It’s simplicity meant that it was transparent to sellers and buyers, at least, in theory. Sellers would, in theory, experiment over time to find the right keywords to use in their listings. Buyers would also experiment. Over time, assuming that eBay was a fairly efficient market, sellers would provide listings with keywords to match the keywords that buyers would use. Supply would meet demand.
Sorting by time remaining had some natural benefits too. For an auction that ending soon, the differences between zero bids, one bid, and more than one bid are stark. One bid guarantees a sale, two bids puts you on a fast path to an efficient price. There was inherent benefit for sellers and for eBay to see auctions that were ending soon get exposure to a disproportionate number of buyers.
So, if it’s not broken, don’t fix it, right? Well, the good news is, the search system was good enough to grow eBay to the giant it is today. The bad news is that it had some fairly obvious shortcomings that became unsustainable over time.
There were a few obvious ones that almost anyone who used eBay ran into. Inexperienced sellers, just casually listing, had no idea what keywords to put in their titles. Pitty the poor seller, trying to sell their $1500 PowerMac G5, if they instead called it a “PowerMac G-5”. Inexperienced buyers also had no idea that searching for “Apple Macintosh” would bring back radically different results than “Apple Mac”. eBay didn’t know how to match keywords to categories. A search for “DVDs” wouldn’t just take the buyer to the DVD category – it would literally return all listings that had “DVDs” in the title. eBay didn’t even understand plurals! “DVD” would bring back very different results than “DVDs”.
eBay started addressing these issues in earnest about five years ago. They slowly rolled out improvements like transliteration (plurals), as well as some experiments with “generic keywords” like DVDs.
Why slowly? Well, the problem is, tinkering with a multi-billion dollar economy is, to lack a better word, scary. It’s scary because you have millions of sellers who have already adapted to the old search engine. You have billions of dollars of purchasing at stake, which means a 1% blip in finding efficiency can mean the difference of tens of millions of dollars in revenue for the company. And last, but not least, it’s scary because it’s hard to objectively find a measure of success that everyone can agree on.
How do you measure the success of search? When a buyer does more searches, is that a good thing or a bad thing? If a buyer views more items, does that mean you’ve done a good job showing them relevant items, or a bad job because they have to click through a lot of items to find one they want?
This gets even more complicated when you take into account the financial relationship between eBay and its sellers. eBay gets paid basically two ways: fees paid up-front when listing the item, and fees paid when an item sells. As a result, sellers pay eBay an up-front amount assuming a certain amount of visibility for their item. eBay does not guarantee impressions, clicks, or sales, but over time, sellers get used to the rough economics of their activities on eBay. They learn which keywords, which categories, which items get them enough clicks and sales to make their business works. That’s how they decide when and where to pay eBay it’s fees.
In any case, those changes merely affected the results that were returned by eBay’s search engine when the buyer performed a search (Step 3). It didn’t affect the sort order, which determines which items are on the first page of a buyer’s search results.
Unfortunately, changing the sort order was just a matter of time. “Time Remaining” is a very good sort for auction items, but it is almost meaningless for fixed price items. Over time, as eBay grew, more and more items on eBay were fixed price. In fact, if you include eBay Stores, eBay has had vastly more fixed price items than auction items for some time. What’s more, all the Step 3 changes mentioned above added more items to the search results, making it even less likely that you’d get good results on your first page.
If you are familiar with internet search, then you know sorting your items to provide the best possible results on the first page is incredibly important. And a meaningless sort for a majority of your listings is just not going to be sustainable without sacrificing a significant amount of your buyer experience and sales.
So, this rollout of Best Match is a big deal. Best Match does not change the results that are returned by eBay for a given keyword, but it does change what appears on that first page. It is a new way to sort items. And that, by itself, is huge.
Not surprisingly, sellers have noticed. Randy’s blog post quotes a seller who has purportedly seen a 40% drop in sales. It’s certainly possible. Best Match will alter the amount of time that listings will have at the top of results. Some sellers might see no change in their activity. Most will see small changes. But there will be a few who see huge swings from their existing metrics.
Most interestingly, it is practically impossible to predict what the outcome will be for any particular seller. To be sure, eBay will guide Best Match to increase overall sales for the site. That means, more items will receive bids and be bought. The economic pie will be bigger for the eBay selling community. But there is no known way to effectively simulate what the outcome will be for any particular seller with their existing listings.
This is a fundamental challenge for eBay. eBay has stated they will focus on improving the buyer experience. eBay will also continue to manage the marketplace to a greater number of sales. However, that won’t change the fact that some sellers will do better under this new system, and others will do worse.
Don’t be surprised to see sellers start to dissect public patent applications for clues on how eBay Best Match works. This is their lifeblood, as much as Google PageRank is the lifeblood for content websites. There is huge economic value in “cracking the code”, and one thing is for sure, the eBay community is full of entrepreneurs who will try to harvest some of that value. Like Google PageRank, Best Match is designed to be opaque. As a result, eBay will make no guarantees about how it functions, and they will actively change it over time to improve it and to prevent abuse.
Also, don’t be surprised when sellers are, in the aggregate, upset about this change. This adds uncertainty to their business, and even though every other site out there is based on relevance sort, they hold eBay to a different standard, and for good reason.
The version of Best Match that eBay is rolling out now has gone through more testing than any new piece of functionality that eBay has ever released. They have gone through numerous versions of the technology, numerous experiments with different factors and systems, and elaborate economic experiments to ensure that it results in higher sales for the marketplace and happier eBay buyers.
And now the real test begins. The eBay Finding team will need to listen, learn & react more in 2008 than they ever have before. It will not be easy, for anyone. But then again, the most important changes never are.
Update (1/24/2008): It looks like this post was picked up in the internal Weekly Gazette inside of eBay. I am, of course, flattered to be highlighted. Of course, I am not an unbiased source, since it was on eBay Express that we first discovered the need to move away from “ending soonest” and “lowest price” sorts, and launched the very first, crude version of Best Match.
Psychohistory 