It’s been a while since Facebook introduced the concept of “applications” to their API, providing developers a very flexible set of tools to create and add features to Facebook itself. Applications on Facebook are best described as “widgets”, since they offer functionality that’s often provided by widgets on other sites, so why Facebook chose a different name is somewhat peculiar.

I’m not going to debate the merits of these applications, or whether they’re leading Facebook down the road to a MySpace-hell dominated by personalized profiles that are an eyesore (there’s already enough discourse on that subject, both on Facebook and outside of it), but rather I’ll take a look at this from a developer’s point of view. What is the potential for developing an application for Facebook, and at the same time, what are the pitfalls?

Pro: Relatively easy to develop

Facebook’s API is easy to work with, especially if you have experience with PHP. One of the official client libraries is written in PHP, and since it’s intended to be used for web applications, this makes perfect sense. Using it to develop an application should therefore be a relatively pain-free process.

While the documentation isn’t always complete, there’s a nice Wiki that covers additional details, and together, these two should allow you to overcome most problems you may run into during development.

Overall, the abilities offered by the API coupled with its ease of use make development relatively straightforward.

Pro: Access to a large user base

Facebook’s current user base of 30 million is rapidly growing. It’s already doubled since the start of this year, and continues to grow at close to 3% per week. If that rate continues, they’ll double again by the end of this year. But perhaps the most amazing fact here is the dedication of Facebook’s user base. It’s one thing to have millions and millions of users, with only a small percentage that are actually active. This isn’t the case with Facebook, which sees over half of its users log in daily. That’s an astonishing statistic by any measure, and even more so when you consider Facebook large and fast-growing user base.

Thus, developing an application for Facebook gives it access to a large, and more importantly, active user base. This gives it a huge potential to grow virally, that is, from person-to-person. When someone adds your application to their profile or account, it will show up as a News Feed item. This means that anyone who has this user as a friend will see a message in their News Feed about this user adding your application. The idea is that they might get interested, click on your application, and then add it to their profile as well, continuing the viral spread cycle.

They can, of course, also find your application by doing a search - if you’re lucky enough to get your application listed in Facebook’s official application directory - more on that later.

Con: Cost and scalability

I think Marc Andreessen (of Netscape fame) put it best when referring to the popularity of an application on Facebook, namely that success kills. (The rest of his article, while lengthy, is an excellent read and I suggest you take the time to read it) Marc gives the example of the iLike application, one of the first, which experienced an insane growth in the early stages. During that time, they grew by 300,000 per day, eating up all of their capacity to the point where they had to buy new servers on a daily basis.

While a company such as iLike, with sources of revenue, may be able to shoulder this kind of burden, it’s quite unlikely that a lone developer would. This creates a sort of Catch-22 - you want your application to become popular, so that you could hopefully monetize it. However, in order for that to happen, you need plenty of resources - which require money.

If all you’re looking to create is a neat little application for you and your friends, or if your application doesn’t really create a huge load on your end, you’ll probably be alright. But be aware that the same dedication and large user base of Facebook that can be so excellent for spreading an application, can be the same environment that quickly overruns your capacity to support your creation.

Con: Monetization

While we’re talking about monetization, it’s worthwhile to point out that if this is your main concern, it’s probably best not to pursue Facebook application development. Since support costs can be high for Facebook applications, attempting to derive a successful business model on this will most likely be met with limited success, if any. Some have called for Facebook to start a revenue sharing program, but this is unlikely, considering Facebook’s uncertain financial future - even if the IPO rumours turn out to be true. (Which they likely will be)

A revenue sharing program, however, would be fair, since by creating a successful application you are adding value to Facebook’s site, and directly increasing their revenues by nature of increasing the time a user spends on the site. I would not, however, hold out on this to develop.

This doesn’t mean that application development is limited to those who just want to create something “for fun” or to add something to their portfolio. On the contrary, I believe the best opportunities the Facebook platform affords is to other businesses based on web services. By utilizing Facebook, they can create applications that promote their site, or tie into their own API, thus using Facebook’s large user base as a source for new customers, users, and thus revenue. Indeed, the first applications developed were made by companies with this very intent. Other sites could follow this model.

Other Cons

Some believe that the viral nature of Facebook apps has been overstated, and that they now don’t spread as easily.

Another, more serious aspect, relates to something I alluded to when I commented on Facebook’s introduction of their Marketplace, a classified-listing type service. It seems that in the TOS of the Facebook Platform, they reserve the right to make a similar application to yours, without obligation to you. I don’t know if this is standard legalese in a TOS for API usage, but certainly it can be construed to be something more sinister, and would deny you any right you had to something you’d created. Of course, if Facebook did do something like this, I’m sure there would be an uproar from the community, with Digg fanning the flames, of course.

Privacy, as always

Privacy is also a concern, as it now not only falls on the user to determine what settings to implement, but also whether developers will respect the TOS of the Facebook Platform. The documentation clearly outlines what user (and user’s friend) info can be stored, but there is nothing to stop a developer from storing more of that information, either for the purposes of adding another feature to their application, or for malicious intent. There have already been concerns about this with popular applications.

The issue of privacy here is that when you sign into a Facebook application, it basically has access to all of the friend info that you would - after all, it probably needs this to perform whatever function it does. Though you may have consented to allow this information to be used, your friends may not have explicitly done this, and may not want their information to be used by 3rd party developers.

Though users can opt-out of participation in the Facebook Platform altogether (and thus prevent 3rd party developers having access), I’m guessing that not everyone or even a majority know about this feature. The privacy settings of Facebook have grown almost as much as the site itself, with settings divided amongst sections and sub-sections, creating somewhat of a convoluted mess. The best precaution to all of this, of course, is not posting anything on the Internet that you wouldn’t want to be known in public. This applies not only to Facebook but other sites as well; I only emphasize this for Facebook because of the notoriety its obtained from such incidents.

Google Street View has been out for some time now, and its proven to be a pretty nifty feature. Basically, Google has sent out teams of people in vans with cameras mounted on the tops to major cities across the US. They’ve been able to capture 360-degree panoramic views of most of the streets in these cities, so you can view the surrounding area on almost any point in the street. This is obviously a great feature for people who find that maps don’t provide the best experience of “getting to know a place”. With Street View, you get the best experience of seeing what a new place will be like, short of actually visiting it. It’ll be great for visitors. Google hopes to extend to feature to more cities as time goes on.

However, the new feature has obviously raised some privacy concerns, but perhaps not in the strictest legal sense. Since Google has taken these pictures from public areas, they’re probably in the clear. However, they is somewhat of a difference when these public pictures are being integrated into the largest search engine for everyone to see.

Precedent

This isn’t first service that’s offered street views. Microsoft has also offered their own limited service, and as one blogger pointed out over a year ago, it would be cool if Google did the same. However, as pointed out in this article, Google Street Views are far more detailed and interactive. In addition to being able to rotate the view a full 360 degrees, you can zoom in. The level of quality in most pictures is quite good, so zooming actually produces good results, rather than just a larger blurry section of an image. You can easily see way down the street by zooming.

Privacy

However, this has also raised what some have called “privacy concerns” - not in the “NSA wiretapping” sense, but rather in the “it’s freaky” sense. A list was quickly formed for the most intriguing Street View sightings, and people also found one of a guy digging for gold. In another more innocent list, we can see Steve Jobs’ Mercedes. In many of these images, you can easily see the faces of those involved. Now, while it’s true that you could have easily seen all of this yourself going for a stroll down the street, it’s somewhat different when a picture is taken and put onto one of the most popular search engine maps for all to see.

Google has provided some recourse, though. If there’s an offending image, you can submit a complaint to have it removed. Indeed, some of the more racier images have been removed; however, while they’re no longer on Google Maps, they’ve forever been immortalized on the various blogs and forums that have covered this topic.

A big deal?

Now, don’t get me wrong. I don’t think this is a major breach of privacy, but if a government had done this, do you think people would respond differently? I think it’s great that Google is trying to improve their search and map functions through this “information fusion”. Once it expands to more cities, it’ll definitely help people like me who get lost easily. Being able to see and identify with a place before going there will be a huge aid. Who knows, once they roll out holographic monitors, I may not even have to save up for that trip around the world…

Since the changes were put into effect, scarcely a day ago, a bevy of groups voicing their opposition to the new features have sprung up on Facebook. Many of them are united by the fact that the news feeds crowd their starting page now, and cause information overload - something I criticized as well. However, most of them decry these new features as being huge breaches of privacy and point to how they make stalking much easier.

Outrage and calls for a boycott

Many anti-changes-to-Facebook groups have been formed, but the largest one (with an online petition), has almost 500,000 users and is still growing at a fast pace. Many of the comments talk about how this is turning Facebook into a ripe target for stalkers, and of the potential for privacy violation. Certainly such a large group of people can’t be wrong?

The Internet is not private

While I admit these changes do help to advertise events in such a way that more people find out about them then some would like to, I don’t believe it fundamentally affects the privacy of Facebook. As I mentioned in my original post, this will hopefully serve as a reminder that privacy on the Internet, even it is in a seemingly locked-away area such as Facebook, is something that one cannot expect. If you post information in a publically-accessible area on the Internet, someone will eventually find out about it, either by mistake or through the Googlebot.

However, much of these concerns are ill-placed. As mentioned on the Facebook blog, no extra information that wasn’t publically available before, has been made available by this update. All it does is aggregate all of that information about your friends in a way that’s supposed to make it easier for you to read. I agree that some options should be provided - such as what information you want to see about your friends (eg. I don’t care that someone wrote on someone else’s wall), but if you don’t want your information showing up on other people’s feed, simple disable those options in your privacy setting in Facebook.

Of more concern

What’s more important to know is that Facebook is not a secretive place and has never been. It’s already been used to identify and charge people for offences because of photos someone posted of them committing incriminating acts. It’s also reportedly used by employers concerning potential employees. Don’t think that someone can access your profile if they’re not in your network? Patriot Act - Read it. (Shameless Arrested Development quotation) Are your privacy rights being violated by all this information-gathering online? I don’t know, maybe; but the lesson here is clear. If you don’t want to take the chance having something private revealed about yourself, don’t post it online.

A rude wakeup

Any big change to a user interface is bound to cause some uproar, but this has provoked extra outcry because it has outlined the true nature of “privacy” on Facebook. The default privacy options are apparently too loose for most people, but most people haven’t bothered to change them - until now. Additionally, many people simply add whomever requests it to their friends list, without thinking about whether they would want that person to be privy to certain bits of information they post on Facebook. While I do think that some changes need to be made now with the new features, I don’t think that these new features intentionally infringe on privacy. The features were meant to allow one to quickly share updates with friends - not with random people you just add as friends. Perhaps people will exercise more discretion in the future, returning Facebook into a real “social” network.

Update (2006-09-08)

Facebook has responded by making changes to allow users to choose what updates and information they want to be displayed in the News Feed and their Mini-Feed. Sounds like a good way to ease in the changes, as people can now decide what they want advertised - they should have done this from the beginning, but it’s nonetheless good to see them listening to their users.

Things like friends’ status updates, new friends of your friends, new photos of your friends, and groups that you friends have joined/left are listed, along with when these actions occurred. Basically, most of things that people would browse around for about their friends have been nicely aggregated into one page for easier viewing, making the service easier to use and more relevant. However, it’s almost information overload - I don’t really need to see every status update on what groups my friends have joined or left, do I?

Additionally, it makes it easier to snoop in on what your friends are doing on Facebook and when they’re online - though it should be noted that all the information posted in the News Feed was always publically available; this just makes it dead simple to find all of it.

The second feature that was added was what they called a “Mini News Feed”, which is displayed on every personal user’s profile. It’s basically the same as your News Feed, except it displays only the updates for that user/friend - again making it easier to keep track of your friends and what they’ve been doing, which only serves to further improve Facebook’s social networking ability.

I agree with TechCrunch’s assessment that Facebook “gets it right” when it comes to social networking - it’s more about making the end user experience better than just about increasing pageviews, and often the two are conflict. Other sites just can’t compare, in my opinion. My only suggestion for improvement would be to have some sort of settings page for the News Feed so what’s displayed can be customized.

Nothing’s changed with privacy?

However, I disagree with Facebook’s assertion that these changes “do not give out any information that wasn’t already visible“, at least in principle. While it’s true that before you could have constantly scanned each and every one of your friend’s profiles in order to figure out when they had joined certain groups, posted wall messages or otherwise updated their profile, this would have been very tedious and time-consuming, and only the most dedicated stalkers would be able to keep up. The new features make this pseudo-voyeurism all too easy. (You could have also written a scraper before, but this would probably have been against the TOS)

If there’s anything good about these changes, it’s that it’ll make people think twice about posting incriminating or otherwise personal information that they would normally want to remain private. It may also force users to reconsider who they really want to add as a “friend”, and how they want their privacy to be set.

I’ll still be using it

Of course, I’m not going to stop using Facebook. But, maybe the way I use it will change - and most likely this will be true with many other users, if the newest Facebook group, entitled “The New Facebook is Freaky as Sh*t“, is in any indication of this. How did I learn of this group? By noting that a friend of mine had just joined it, as of 11:17 PM EST tonight.

“It could be as simple as detecting whether or not a photo contains a person, or, one day, as complex as recognizing people, places, and objects…”

The article goes on to speculate that Google may use this technology to enable searching of people on the web by any photos of them that may be online. Certainly, this wouldn’t be out of their league, and they’ve been interested in this technology for some time. Indeed, Sergey Brin, one of the co-founders of Google, has said that image recognition is something they’d like to do. Despite the obvious usefulness of such a tool, it does raise privacy questions.

Add it to the list

Besides facial recognition, Google is also active in voice recognition as well, and have hinted at using it serve up web ads based on what’s playing on your TV. (Seems far-fetched at present time) Their OCR technology is also pretty decent, as Google Book Search is able to search the full text of many books online, that have had their pages scanned in - the service will also hilite the relevant passages, a neat feature. All of these services are part of Google’s future plans to catalogue and make available for search, all forms of information, not just those that are presented as text on the web.

Do no evil

Of course, while Google has been getting attention from privacy advocates as of late, because of the huge amounts of information they collect from users, they’re one of the better companies when it comes to keeping your information private. When the US DOJ tried to get search engine companies to turn over anonymized search queries for their users, all of them complied except for Google, which took the matter to court and won. Thus, Google is probably one of the companies you should worry about less, as there are many others who have less or no respect for an individual’s privacy. However, recent blunders by AOL have done little to quell the fears of privacy advocates.

Facial recognition is almost certainly used by governments, as it’s an important tool for them. So, what’s the big deal with Google getting it, then, if it’s already in use? Well, the problem is that this signals the technology is becoming more available and pervasive, and indeed, as that happens, we will become more used to it. The fact that anything you put on the Internet can, and will, be viewable by all is only driven home more by the use of this technology - some people still don’t realize this, though.

Reasonable expectation of privacy

While it’s probably true that when you go out in public, you have little to no expectation of privacy, I doubt that this is a good thing overall. Facial recognition technology, when fully developed, and combined with a network of security cameras could enable automated tracking of a person’s whereabouts - no need for them to be carrying a cellphone or other tracking device. Is this a good thing? Some would argue it would better enable the tracking of criminals and terrorists, but I believe the potential for abuse is very real and something that needs to be addressed.

While I don’t think Google would abuse this technology, others certainly would, which is why I think people need to be aware of what’s going on and for companies like Google to be open about what they’re planning on doing with it.

No end to the data

Traditional news began covering the story today, with even TIME magazine penning a little piece about the incident and its relation with privacy rights, or the lack thereof. Other people have been more enterprising, with someone setting up an online, searchable database of the results, thus saving people the time of importing all the data into their own DBMS.

After it was discovered that users were apparently searching for “how to kill your wife”, ValleyWag set up a contest to find the scariest search record. Apparently, the online community’s appetite for the grotesque is insatiable.

Even more interesting were some of the results from the contest: Someone apparently submitting a search query that looked very much like a “message in a bottle” of a castaway stranded on an island. ValleyWag suspects a viral marketing campaign for the TV series Lost. Or, maybe someone is actually stranded on an island in the South Pacific with a laptop and WiFi access - though that wouldn’t be such a bad life!

Changes upcoming

With all the bad publicity AOL has been getting as of late (problems with cancelling customer accounts, workforce reductions, etc.), this leak could not have come at a worse time. Further complicating the situation is the fact that bloggers spread the information like wildfire, and helped the data get into wide release. In fact, the previously mentioned AOL Search Database is in violation of the user agreement that came with the search data - it states that the information provided is only to be used for non-commercial (research) purposes. The search database site is clearly displaying ads and the owner hoping to profit from it.

Companies are going to tighten up their policy on data release, and I wouldn’t expect any of them to be releasing data to the research community any time soon. At the very least, the way they deal with search records will be kept even more discreet.

The download was taken offline sometime yesterday (August 6th, 2006), but enough people had already downloaded it to ensure that mirrors would quickly be set up, ensuring its continued spread. While the intent of this release of data was obviously not malicious, it was a poorly thought-out move that AOL is sure to receive more bad PR for - especially in light of their recent troubles.

Good intent, bad execution

The dataset was first released sometime on or before August 4th, 2006, on AOL Research. Like Google, Yahoo! and Microsoft, AOL maintains a Research site to inform and help members of the academic and developer community of things they’re up to, and to offer assistance to researchers. As Google’s cache of the former download page indicates, the intent of offering this download was to facilitate research into making search engine technologies better, by offering a look into real users’ search queries and behaviours.

Having a little bit of experience in the research field (I’m currently working as a summer undergraduate research student), I can say that it’s not unheard of to see industry researchers helping out the academic community, either through joint research projects or, as in this case, the release of datasets for testing some particular sort of algorithm.

For example, one of the projects of the researchers in my lab is speech quality assessment, which is essentially developing an algorithm or system that can process a speech file and assess its quality as a human would, which has enormous benefits for Telco’s since they are always seeing what speech codecs are most efficient in terms of size/quality. In order to do this, one needs a vast quantity of voice files (and their associated Mean Opinion Scores - rated by humans) to test around. These voice files and their data can only be obtained through costly (both in time and money) trials that cost lots of money - we’re talking six-figures and above here. Much of the data can thus only be obtained by Telcos who have the money to conduct these trials. Fortunately, many of them have been gracious enough to release these datasets to our lab (and others) - but the data is only released after a strict NDA (Non-Disclosure Agreement) is signed, along with perhaps other agreements.

Thus AOL’s intent certainly was not bad, as they merely wanted to help out in the academic circles, and maybe get cited in a paper or two. However, their execution of this release was poor. They should not have released this data publically, allowing just anyone to download it, but rather have followed a pattern of releasing only under an NDA and only to parties that they thought were reasonably going to use it for research purposes. In many ways, it’s worse than releasing data they paid for, because much of the data constitutes the private data of regular AOL users, data that perhaps they agreed to protect under AOL’s privacy policy.

Privacy concerns?

Some of you might be wondering why this matters at all. After all, it’s just a few million search queries that AOL users entered - containing no personally identifiable information. Well, that’s true - however, since the usernames were merely replaced by unique identifiers (eg. a username is changed to a random, unique number, so “John Doe” is always mapped to “123456″, for example), profiles of users’ searches can be built - opening so many cans of worms that I can’t even count that high.

Here’s a complete breakdown of what information is provided for each query:

AnonID

an anonymous user ID number.

Query

the query issued by the user, case shifted with most punctuation removed.

QueryTime

the time at which the query was submitted for search.

ItemRank

if the user clicked on a search result, the rank of the item on which they clicked is listed.

ClickURL

if the user clicked on a search result, the domain portion of the URL in the clicked result is listed.

This is an absolute goldmine of information to anyone concerned with search engines, SEO or anything related to that. Not only do you get a complete profile of a user’s searches, but you get the exact time they conducted the search, what, if any, results they clicked on (and the rank of that result), along with the domain name of site the clicked-through to. Running datamining techniques on this information is somewhat trivial, as it can easily be analyzed to determine patterns of most any sort.

Internet marketers would give an arm and a leg for this sort of data, but they now will have their hands on it - for free. In fact, there’s already been analysis on the dataset, producing valuable results, and claims that Google will get spammed with made-for-adsense sites (a clear violation of their TOS) targetting keywords for popular and profitable markets. This will further pollute search results, and generate more noise, creating more headaches for search engines like Google, who will have to further adapt to prevent the signal-to-noise ratio from further deterioration.

However, this dataset probably (or hopefully?) won’t have a lasting effect for spammers. As Thought Market points out, since everyone has access to this data, so does Google as well. And Google has proved themselves not to be slackers on developing data analysis techniques - after all, how did they become arguably the most successful search engine? You can bet they will be analyzing this dataset to see what results they can get, and hypothesizing about how these results might be used. If they seen anything fishy going on, you can bet they’ll remove those sites from their index.

Justice is blind

But perhaps the most important or unnerving aspect of the release of this dataset is the implications it has on legal matters. The data released is exactly the sort of data the US DOJ requested from the major search engines (Google, Microsoft, Yahoo! and AOL) earlier this year to “defend its argument that the Child Online Protection Act is constitutionally sound”. Microsoft, Yahoo! and AOL complied, while Google resisted the subpoena - and when the matter went to court, the judge ruled in Google’s favour.

Taken in this context, things do not look good for AOL - while Google’s fighting to protect your privacy, AOL’s actively working in the opposite direction, it seems. I realize it was not meant to be that way, but with something this serious, the effect is sometimes far more important than the intent. Looking at the raw search queries of some users reveals things ranging from embarrasing or weird searches, to queries that may indicate they’re up to no good.

Let’s be clear here - simply searching for “how to kill your wife” is not tantamount to murder - nor is it even enough proof of conspiracy to commit murder. At least, that’s how I see things from my “not a lawyer” perspective. However, consider a situation where someone did kill their wife. If their search results were made available, and this query turned up, how do you think law makers would respond? Or, consider sample size of 1000 murderers, and their search queries. If it were statistically established that a significant percentage of murderers search on the Internet for some aspect of murder before the actual crime, how would law makers respond?

These are complex questions, and I’m not sure what I’d do with the data. Statistics is a complex topic, but the important fact is they are often abused to support invalid and over-reaching actions. This obviously leads to questions and issues like “thought crime” and comparisons with Minority Report. I generally don’t like using slippery-slope arguments, but this is an area where it’s easy to see how abuse could happen, considering that the US DOJ has already expressed an interest in this sort of data, and with all the NSA wiretaps going on.

There were also many other weird, and often disgusting search queries going on, often by the same set of users. I won’t repeat them here, but you can easily see for yourself what some AOL users were searching for. While these searches represent a very small percentage of all the searches conducted, it doesn’t paint a good picture of the Internet for lawmakers. (Keep in mind that these sort of searches are to be expected - after all, even if the incident rate for violent crime is something low like 27 per 100 000, that’s still 0.027%, and keep in mind this search query dataset represents over 650,000 users.)

Note that I haven’t even touched on the topic of false positives. What if someone was writing a paper on the effects of torture, or of any of the other examples of “man’s inhumanity to man”? They’d obviously have to search for some pretty disturbing stuff (I’m glad I’ve never had such a writing assignment), and then these searches would be tied together by the unique identifier. There are plenty of other examples, but the idea is that using someone’s search queries for legal purposes is an easy way to 1984 - while there obviously may be statistical connections between actual crimes, the potential for “though crime” is just too large.

Waves and reverberations

As far as I can tell, this story (and thus the wide release of the data) only started sometime this weekend, probably on Friday or Saturday. But, it’s already making its way around the non-traditional news outlets (blogs, social communities) like fire through a tissue factory. There’s been talk of how personally identifiable information may be available, and of the legal ramifications. It’s already in the top five among Technorati searches, so you can believe that anyone who wants this data will get it. If this doesn’t hit the mainstream news soon, I’ll be very surprised and disappointed - so far there’s only one result for it in Google News.

Update

AOL has apparently responded to this privacy leak, in a comment available here.

Nothing really new or interesting - other than the data represented only about 1.5% of all AOL users, and those included were only US users who used AOL’s client software. As expected, Mr. Weinstein of AOL said that it was an “innocent enough attempt to reach out to the academic community with new research tools”, but as expected, it didn’t go through official channels for approval. You can be sure that AOL’s going to change their policy on this sort of stuff - don’t expect any more stuff from AOL research for a while. (Someone or some people probably also got fired.)