A few things to note: Firstly, while the act of promoting or removing a search result seems very akin to Digg, the result is not the same. The changes you make only affect your own search results, and Google is very clear on this. However, it would be madness to believe that Google would not use the data gathered from this social experiment to further improve their algorithms. You also have the option of adding your own results to further personalize your searches and there is an option for seeing what others have recommended/promoted or removed, providing for an interesting social experiment.

Secondly, as this WSJ blog notes, this ability may annoy people who have used SEO tactics to improve their site’s placement in Google’s search rankings. However, I find this complaint misses the point: Search is supposed to simplify people’s lives, and if they’ve promoted or removed a link it was because they found something to be more useful or irrelevant.

This isn’t yet a “wisdom of the crowds” approach to search results, but its undoubtedly a step forward towards a hybrid approach that takes in more human input to determine the quality of results and their placement. One can only hope it will improve with time!

“It could be as simple as detecting whether or not a photo contains a person, or, one day, as complex as recognizing people, places, and objects…”

The article goes on to speculate that Google may use this technology to enable searching of people on the web by any photos of them that may be online. Certainly, this wouldn’t be out of their league, and they’ve been interested in this technology for some time. Indeed, Sergey Brin, one of the co-founders of Google, has said that image recognition is something they’d like to do. Despite the obvious usefulness of such a tool, it does raise privacy questions.

Add it to the list

Besides facial recognition, Google is also active in voice recognition as well, and have hinted at using it serve up web ads based on what’s playing on your TV. (Seems far-fetched at present time) Their OCR technology is also pretty decent, as Google Book Search is able to search the full text of many books online, that have had their pages scanned in – the service will also hilite the relevant passages, a neat feature. All of these services are part of Google’s future plans to catalogue and make available for search, all forms of information, not just those that are presented as text on the web.

Do no evil

Of course, while Google has been getting attention from privacy advocates as of late, because of the huge amounts of information they collect from users, they’re one of the better companies when it comes to keeping your information private. When the US DOJ tried to get search engine companies to turn over anonymized search queries for their users, all of them complied except for Google, which took the matter to court and won. Thus, Google is probably one of the companies you should worry about less, as there are many others who have less or no respect for an individual’s privacy. However, recent blunders by AOL have done little to quell the fears of privacy advocates.

Facial recognition is almost certainly used by governments, as it’s an important tool for them. So, what’s the big deal with Google getting it, then, if it’s already in use? Well, the problem is that this signals the technology is becoming more available and pervasive, and indeed, as that happens, we will become more used to it. The fact that anything you put on the Internet can, and will, be viewable by all is only driven home more by the use of this technology – some people still don’t realize this, though.

Reasonable expectation of privacy

While it’s probably true that when you go out in public, you have little to no expectation of privacy, I doubt that this is a good thing overall. Facial recognition technology, when fully developed, and combined with a network of security cameras could enable automated tracking of a person’s whereabouts – no need for them to be carrying a cellphone or other tracking device. Is this a good thing? Some would argue it would better enable the tracking of criminals and terrorists, but I believe the potential for abuse is very real and something that needs to be addressed.

While I don’t think Google would abuse this technology, others certainly would, which is why I think people need to be aware of what’s going on and for companies like Google to be open about what they’re planning on doing with it.

No end to the data

Traditional news began covering the story today, with even TIME magazine penning a little piece about the incident and its relation with privacy rights, or the lack thereof. Other people have been more enterprising, with someone setting up an online, searchable database of the results, thus saving people the time of importing all the data into their own DBMS.

After it was discovered that users were apparently searching for “how to kill your wife”, ValleyWag set up a contest to find the scariest search record. Apparently, the online community’s appetite for the grotesque is insatiable.

Even more interesting were some of the results from the contest: Someone apparently submitting a search query that looked very much like a “message in a bottle” of a castaway stranded on an island. ValleyWag suspects a viral marketing campaign for the TV series Lost. Or, maybe someone is actually stranded on an island in the South Pacific with a laptop and WiFi access – though that wouldn’t be such a bad life!

Changes upcoming

With all the bad publicity AOL has been getting as of late (problems with cancelling customer accounts, workforce reductions, etc.), this leak could not have come at a worse time. Further complicating the situation is the fact that bloggers spread the information like wildfire, and helped the data get into wide release. In fact, the previously mentioned AOL Search Database is in violation of the user agreement that came with the search data – it states that the information provided is only to be used for non-commercial (research) purposes. The search database site is clearly displaying ads and the owner hoping to profit from it.

Companies are going to tighten up their policy on data release, and I wouldn’t expect any of them to be releasing data to the research community any time soon. At the very least, the way they deal with search records will be kept even more discreet.

The download was taken offline sometime yesterday (August 6th, 2006), but enough people had already downloaded it to ensure that mirrors would quickly be set up, ensuring its continued spread. While the intent of this release of data was obviously not malicious, it was a poorly thought-out move that AOL is sure to receive more bad PR for – especially in light of their recent troubles.

Good intent, bad execution

The dataset was first released sometime on or before August 4th, 2006, on AOL Research. Like Google, Yahoo! and Microsoft, AOL maintains a Research site to inform and help members of the academic and developer community of things they’re up to, and to offer assistance to researchers. As Google’s cache of the former download page indicates, the intent of offering this download was to facilitate research into making search engine technologies better, by offering a look into real users’ search queries and behaviours.

Having a little bit of experience in the research field (I’m currently working as a summer undergraduate research student), I can say that it’s not unheard of to see industry researchers helping out the academic community, either through joint research projects or, as in this case, the release of datasets for testing some particular sort of algorithm.

For example, one of the projects of the researchers in my lab is speech quality assessment, which is essentially developing an algorithm or system that can process a speech file and assess its quality as a human would, which has enormous benefits for Telco’s since they are always seeing what speech codecs are most efficient in terms of size/quality. In order to do this, one needs a vast quantity of voice files (and their associated Mean Opinion Scores – rated by humans) to test around. These voice files and their data can only be obtained through costly (both in time and money) trials that cost lots of money – we’re talking six-figures and above here. Much of the data can thus only be obtained by Telcos who have the money to conduct these trials. Fortunately, many of them have been gracious enough to release these datasets to our lab (and others) – but the data is only released after a strict NDA (Non-Disclosure Agreement) is signed, along with perhaps other agreements.

Thus AOL’s intent certainly was not bad, as they merely wanted to help out in the academic circles, and maybe get cited in a paper or two. However, their execution of this release was poor. They should not have released this data publically, allowing just anyone to download it, but rather have followed a pattern of releasing only under an NDA and only to parties that they thought were reasonably going to use it for research purposes. In many ways, it’s worse than releasing data they paid for, because much of the data constitutes the private data of regular AOL users, data that perhaps they agreed to protect under AOL’s privacy policy.

Privacy concerns?

Some of you might be wondering why this matters at all. After all, it’s just a few million search queries that AOL users entered – containing no personally identifiable information. Well, that’s true – however, since the usernames were merely replaced by unique identifiers (eg. a username is changed to a random, unique number, so “John Doe” is always mapped to “123456″, for example), profiles of users’ searches can be built – opening so many cans of worms that I can’t even count that high.

Here’s a complete breakdown of what information is provided for each query:

AnonID

an anonymous user ID number.

Query

the query issued by the user, case shifted with most punctuation removed.

QueryTime

the time at which the query was submitted for search.

ItemRank

if the user clicked on a search result, the rank of the item on which they clicked is listed.

ClickURL

if the user clicked on a search result, the domain portion of the URL in the clicked result is listed.

This is an absolute goldmine of information to anyone concerned with search engines, SEO or anything related to that. Not only do you get a complete profile of a user’s searches, but you get the exact time they conducted the search, what, if any, results they clicked on (and the rank of that result), along with the domain name of site the clicked-through to. Running datamining techniques on this information is somewhat trivial, as it can easily be analyzed to determine patterns of most any sort.

Internet marketers would give an arm and a leg for this sort of data, but they now will have their hands on it – for free. In fact, there’s already been analysis on the dataset, producing valuable results, and claims that Google will get spammed with made-for-adsense sites (a clear violation of their TOS) targetting keywords for popular and profitable markets. This will further pollute search results, and generate more noise, creating more headaches for search engines like Google, who will have to further adapt to prevent the signal-to-noise ratio from further deterioration.

However, this dataset probably (or hopefully?) won’t have a lasting effect for spammers. As Thought Market points out, since everyone has access to this data, so does Google as well. And Google has proved themselves not to be slackers on developing data analysis techniques – after all, how did they become arguably the most successful search engine? You can bet they will be analyzing this dataset to see what results they can get, and hypothesizing about how these results might be used. If they seen anything fishy going on, you can bet they’ll remove those sites from their index.

Justice is blind

But perhaps the most important or unnerving aspect of the release of this dataset is the implications it has on legal matters. The data released is exactly the sort of data the US DOJ requested from the major search engines (Google, Microsoft, Yahoo! and AOL) earlier this year to “defend its argument that the Child Online Protection Act is constitutionally sound”. Microsoft, Yahoo! and AOL complied, while Google resisted the subpoena – and when the matter went to court, the judge ruled in Google’s favour.

Taken in this context, things do not look good for AOL – while Google’s fighting to protect your privacy, AOL’s actively working in the opposite direction, it seems. I realize it was not meant to be that way, but with something this serious, the effect is sometimes far more important than the intent. Looking at the raw search queries of some users reveals things ranging from embarrasing or weird searches, to queries that may indicate they’re up to no good.

Let’s be clear here – simply searching for “how to kill your wife” is not tantamount to murder – nor is it even enough proof of conspiracy to commit murder. At least, that’s how I see things from my “not a lawyer” perspective. However, consider a situation where someone did kill their wife. If their search results were made available, and this query turned up, how do you think law makers would respond? Or, consider sample size of 1000 murderers, and their search queries. If it were statistically established that a significant percentage of murderers search on the Internet for some aspect of murder before the actual crime, how would law makers respond?

These are complex questions, and I’m not sure what I’d do with the data. Statistics is a complex topic, but the important fact is they are often abused to support invalid and over-reaching actions. This obviously leads to questions and issues like “thought crime” and comparisons with Minority Report. I generally don’t like using slippery-slope arguments, but this is an area where it’s easy to see how abuse could happen, considering that the US DOJ has already expressed an interest in this sort of data, and with all the NSA wiretaps going on.

There were also many other weird, and often disgusting search queries going on, often by the same set of users. I won’t repeat them here, but you can easily see for yourself what some AOL users were searching for. While these searches represent a very small percentage of all the searches conducted, it doesn’t paint a good picture of the Internet for lawmakers. (Keep in mind that these sort of searches are to be expected – after all, even if the incident rate for violent crime is something low like 27 per 100 000, that’s still 0.027%, and keep in mind this search query dataset represents over 650,000 users.)

Note that I haven’t even touched on the topic of false positives. What if someone was writing a paper on the effects of torture, or of any of the other examples of “man’s inhumanity to man”? They’d obviously have to search for some pretty disturbing stuff (I’m glad I’ve never had such a writing assignment), and then these searches would be tied together by the unique identifier. There are plenty of other examples, but the idea is that using someone’s search queries for legal purposes is an easy way to 1984 – while there obviously may be statistical connections between actual crimes, the potential for “though crime” is just too large.

Waves and reverberations

As far as I can tell, this story (and thus the wide release of the data) only started sometime this weekend, probably on Friday or Saturday. But, it’s already making its way around the non-traditional news outlets (blogs, social communities) like fire through a tissue factory. There’s been talk of how personally identifiable information may be available, and of the legal ramifications. It’s already in the top five among Technorati searches, so you can believe that anyone who wants this data will get it. If this doesn’t hit the mainstream news soon, I’ll be very surprised and disappointed – so far there’s only one result for it in Google News.

Update

AOL has apparently responded to this privacy leak, in a comment available here.

Nothing really new or interesting – other than the data represented only about 1.5% of all AOL users, and those included were only US users who used AOL’s client software. As expected, Mr. Weinstein of AOL said that it was an “innocent enough attempt to reach out to the academic community with new research tools”, but as expected, it didn’t go through official channels for approval. You can be sure that AOL’s going to change their policy on this sort of stuff – don’t expect any more stuff from AOL research for a while. (Someone or some people probably also got fired.)

During this latest round of spamming, which reached its peak this weekend, it appears that well over 5 billion spam pages were indexed by Google; while this by itself is a huge number, taken in context with the total number of pages that Google had indexed at the time, around 25 billion by the source in the first link, it is simply astonishing. What’s even more impressive, or scary, is the fact that the site was started only less than a month ago, making this intrusion into the Google search indexes not only massive, but frighteningly fast as well.

From reading the posts at Digg, and from the resultant link, it appears the spammer used a script in order to serve up articles based on keywords, and furthermore, utilized many topical subdomains in order to generate the content that would appear “high” on the keywords list, and thus be indexed by Google. Comment-spam (on forums, blogs, and the like) may or may not have played a role in getting the pages ranked higher, but one thing is for certain – these useless pages made it very high onto the search results page, in many cases filling multiple spots in the top 10 results. These searchs were for common terms, such as “war on terror pros cons” and “pizza sauce recipe”.

But what’s the reason for this? Well, the same as for any spam marketing campaign – advertising. Because of the currently huge market for Internet advertising, the potential for making lots of money of ads on popular sites is an opportunity many cannot turn down. You’re likely to see these somewhat unobtrusive text ads on most any popular site nowadays – in fact it was Google who first popularized them as a replacement for the annoying animated graphic banner ads and popups, which put off many viewers. This form of advertising is, undoubtedly, the backbone of many web 2.0 companies. Companies like Digg, Flickr and even Google rely on ads for nearly 100% of their revenue.

But this potential has turned many to the dark side of advertising – creating spam sites whose sole purpose is to attract viewers for increased ad viewing. While successful web 2.0 companies may display ads on their site, they all offer some useful service that people return for. These spam sites do not offer any useful service or information, but instead manipulate search engine results in order to trick users to visiting their site. Once there, the user will find only semi-meaningful information laced intricately laced with ads, or perhaps, no information and only ads. While this clearly violates many ad providers terms-of-service (such as Google Adwords), most sites have no problem doing this or finding marketers who don’t care about such trivial things.

This is perhaps the other side of the double-edged sword that is the Internet. On the one hand, forums, blogs, and other community-based sites offer the immensive capacity for spreading useful information. On the other hand, they also offer the ability to spread useless information as well, and in some cases, search engines cannot yet discriminate between the two as most humans would. This can be seen in the huge amounts of comment spam, and spam blogs that pervade the Internet.

All of this creates problems on many levels, and in many ways, is more damaging that e-mail spam. While e-mail spam is annoying for the junk it creates in our inboxes, and the extra bandwidth it consumes, for the most part anti-spam tools have helped curb this influx. However, search engine spam targets the most basic use of the Internet, and that is the ability to find useful information. With all the spam sites out there, and the manipulation of search engine results that comes from this, the ability to conduct a search that returns useful information may be compromised in the future if proper countermeasures are not employed.

Furthermore, it creates a nightmare for the people who engineer the search engines, as they must find a way to tweak the algorithms to prevent this from happening again. In the process, false positives may be generated, causing legitimate sites to be unintentionally delisted, causing futher headaches. Google has been having delisting problems as of late, and one wonders if this is related to the recent spamming problem.

One also has to wonder how many of these problems may have come as a result of the upgrade Google did to its datacenters, dubbed “Big Daddy”. Google did not seem to have these sorts of problems before this, so perhaps there is a correlation, but maybe not a cause. It’s interesting to note, however, that the aim of the “Big Daddy” updates was to prevent this sort of thing – and to keeping meaningful sites in the index, which is exactly the opposite of what happened, since the spamming from these sites evidently bumped out important sites from the indexes.

This recent round of spamming was not unique to Google; it affected the Yahoo! and MSN search results as well, though not to the extent that it did to Google’s. It was probably not the intention of the spammer to get so many pages indexed on Google, and probably got “out of hand” quickly, however, one has to wonder if this spam site directly targeted Google in its quest for search engine manipulation, or whether this was just a coincidence. But the problem for search engines remains, and that is, how to effectively discriminate between meaningful and useless information, without making too many mistakes, one way or the other.

Thankfully, it appears the Google is working on fixing this problem, and not only by just removing the most recent spams. It will take some work, but I think they should hopefully arrive at a solution, but as always, spammers will always be working to gain the upper hand as well. Let’s hope that the combined effort of companies like Google, Yahoo! and Microsoft can thwart them, or else the Internet may become awash in the useless garbage of spam.