The PURPOSE Of Social Search

Over the last 30 years, many tasks that humans once performed have been automated, encoded and passed over to computers. It’s the drive for efficiency that Bill Gates still talks about today. The internet however, never started off as a human-powered system so there was nothing to automate. But recently we have been witnessing the emergence of human-powered applications such as Social Search that are in some way competing with fully automated systems.

Machines and the size problem

The Internet is big and it’s growing. I went to the WayBackMachine to try to find out by how much. The WayBackMachine is fabulous. It saves a copy of all the webpages it has found during its crawl and lets you go and have a look at how things were. It’s a digital archive of the early days of the web and looking back at some of the sites I was involved with 5 or 6 years ago is a cringe-inducing exercise. I’m just glad they aren’t showing up in Google these days!

I used the WayBackMachine to visit the Google homepage over a few dates and look at the number of pages in their index. Here’s an example of the Google homepage from 2005.

Google stopped publishing the size of their index in Sept 05, so I’m projecting after that, but there is hearsay evidence online for something like 20bn pages right now.

A word of warning

I am sceptical about these numbers. I wrote in a previous post about Google’s bogus results count and I’m pretty sure that these numbers are also inflated, but I think we can conclude that the internet is pretty big.

So what?

The so what is that with this vast vat of data, one would expect that only the powerful processing of a (or many) computers would be able to make sense of it all. But that turns out not to be true. The reasons are that:

1. There is an army of professionals out there trying to screw with the google rankings – some using nasty ‘Black Hat’ tactics
2. As volume grows, even machines have a harder time finding the ‘right’ content

As a result, Google results are getting worse. Google has segmented its dataset into news, blogs, books etc but that has added complexity to the interface and has not solved the problem.

The human touch

Enter Social Search – Wikipedia for example. If you know what you want to find out about, just go to the Wikipedia page and you will find generally high quality information on the subject. No spam, No machine-generated rankings, just people-edited goodness. Mahalo is a spin on this. Professional editors this time and a news feed to pull through more up-to-date info. But under the skin, Mahalo is really another wiki.

And there are more bizarre social search examples – take twitter. It’s not immediately obvious that Twitter is social search, but it can be. My friend Simon Grice told a story at Being Digital in which Loic Lemeur from Seesmic was looking to hire a raccoon (Seesmic’s logo) in San Francisco. Not having had much luck through more traditional channels, Loic posted a request on Twitter and received a number of replies. [ref: Andrew Whitehouse’s weblog]

Google itself uses another type of social interaction for it’s paid search or PPC ranking. According to Google, the number one factor in Ad Rank is the historical clickthrough rate (CTR) of the ad and of the matched keyword on Google. So people’s behaviour this time rather than their words, but it’s still social.

Then there are the discovery engines such as stumble upon and digg that are also human-powered. These are gaining popularity as it’s very rare to get something from them that is irrelevant or spammy. Due to the nature of the service, the network is self-managing.

One final example from the Arts. thepaintingfool.com will create art for you or rather is ‘a computer program that aspires to be an artist’ and Koan is software that will write music. But students at universities like Sussex are using this software in a more directional model. That’s to say they aren’t using the machines to generate music or art, they are directing them. They are tools. It’s like having an intelligent amplifier rather than having an amplifier that creates music all by itself.

The big picture take away for me is that a combination of people and technology is very powerful.

The best way I can figure to describe why this is that whilst machines create efficiency people add purpose.