It’s a problem that has existed as long as there have been search engines: How do you handle complicated searches?
Although search technology has come a long way in 20 years, the answer for all large-scale search engines – even Google – has remained “not well.” Once a user searches for something that involves a complicated range of possible answers, there’s just no good way to choose which results a user will personally find most relevant. At least, not with today’s computing power.
Imagine, for example, if someone searches for panic room. Are they interested in home security, the 2002 movie, or the obscure Welsh rock band? Search engines can’t really be sure, so presently they use a sort of popularity contest. If not many people are clicking on the band Panic Room, then no one’s going to see them first in the search results – even the people who really want to know about that band. To try to correct this, major search engines profile users, so that someone who spends a lot of time on the band’s website might actually see their page in top results.
But that’s an imperfect solution, because the same user who checked out the band six months ago might be interested in an actual panic room the day after their house is broken into. (Most people intuitively understand this search-engine failing, giving rise to search strings like panic room band or panic room movie – but this creates an unnecessary barrier between users and good search results.)
Thanks to researchers at North Carolina State University, a solution is on the way. For more than a year Dr. Kemafor Anyanwu and his team have successfully used “ambient query context” to personalize search results. That’s their way of saying that they look at recent searches by a user. If you’ve just been searching for Welsh folk music, the band comes up first; if you were searching for jodie foster then the movie will be pushed toward the top – even if months ago you were shopping for safe rooms.
If that seems like an obvious solution, it is. But it takes a lot of computing power to track up-to-the-minute search history for a user, reference all related terms, and weight the results chronologically. It’s a game of predicting user intent, and that takes more memory than any search engine has been able to devote to it.
Dr. Anyanwu’s team may have fixed that, by inventing a new method for representing and indexing that contextual data – and a new architecture for organizing it. They won’t be unveiling the specifics of their technique until late October, but the results speak loudly:
“Our new indexing and search computing architecture allows us to support personalized search for about 2,900 concurrent users using an 8GB machine, whereas an earlier approach supported only 17 concurrent users.”
Somebody get this over to Google, for the sake of Welsh folk rock fans everywhere.