Props definitely go out to Automattic for creating such a reliable and accurate service. When I first wrote about it over a year ago, I was very impressed with its precise filtering of spam and non-spam (aka ham) comments along with its unobtrusiveness. Akismet truly makes spam filtering transparent to the end user, unlike other methods such as CAPTCHAs.

Of course, I can’t forget thanking the developers of WordPress as well. Without them, I would have no site from which I’d have to protect from spam.

Check back later when this site surpasses the 1,000,000 mark for spam.

I wonder if this is related to the recent update to the system (as it’s a centralized service), or something else. The update seemed to be only related to improving the statistics tracking of the service, and not something related to the spam-detection algorithm. I’ve tried disabling/enabling the plugin, so we’ll see if that helps - has anyone else been having these problems?

In the old days…

Back a few years ago, the big thing was e-mail spam. It’s been around so long that almost anyone who’s used e-mail knows about it. All those e-mails telling you how to re-finance your mortgage, get low-cost prescription drugs or how to grow a certain appendage longer tended to fill up one’s inbox, making it a pain to delete all of them and find the real e-mail that you needed to read.

For those of you thinking that e-mail spam was big annoyance, recent polls have shown that people apparently do make purchases from spam e-mails, making it a viable direct marketing tactic. However, server-side e-mail spam filters have greatly increased in their ability to weed out junk e-mails in the past few years, so for many people, spam is not as big of a problem as it once was. Unless, you’re using Hotmail.

Adapt or die

Faced with the prospect of decreased income thanks to e-mail spam filters (or just wanting to make more money), spammers began a new front in the war of direct marketing. They began to spam forums, guestbooks and most recently, commenting systems on various systems in the hopes of drawing attention to the products they were advertising. The purpose was the same - to encourage people to buy products, mostly the same ones they had been sending out in mass e-mails. So, while e-mail spam has not gone away in recent times, alternative forms of spam have increased many times.

In fact, spamming online communites with message may even be more effective, since a spammer needs only to get their message on one site in order for it to be viewed by many. However, since spam tended to be easy to distinguish from comments posted by humans, it was relatively easy for developers to combat this by building in anti-spam features to weed out spam. Spammers responded by making their “bots” - the automated programs that sent out the spam messages - better and trying to make the “quality” of their messages seem more “human”.

A personal experience

My site, unitstep.net, is far from a popular site. However, spam bots have still managed to find this site and I’ve logged 104 spam comments in just over two months’ worth of operation. That’s quite impressive, and shows that spammers are actively searching for new blogs to spam. As I mentioned before, some comment spam is aimed at promoting other websites in order to increase their ranking in SERPS (Search Engine Results Pages), thus drawing unsuspecting visitors to their sites, where they are served up advertising that looks like regular content. (WordPress and most blogs add an attribute of rel="nofollow" to comment links to defeat this, but they still try.) Most of the spam I’ve got, has been of the direct marketing variety, though.

If you haven’t seen the comment spam though, it’s because of Akismet, a truly kick-ass plugin for WordPress, written by the same team. It basically uses a central authority to check on every comment that’s submitted, and analyzes its content to determine if its likely to be spam, or likely to be real. Comments that are marked as spam aren’t shown, and are instead put in a moderation queue, for me to look at and delete or, if it’s a false positive, allow it through. Akismet appears to learn as well, so it’s success rate increases the longer it’s been in use. I apparently joined relatively late in the game, as I have yet to find Akismet report a false positive or let a spam comment through.

Until today.

However, I don’t blame Akismet for this one, as I was almost caught off guard. Here was the content of the comment:

Plato learning…

I am Karin, very interesting article that contained the information I was searching for in Google, thanks….

Upon closer inspection, the comment does look like a spammer’s, since it didn’t specifically relate to the post’s topic, instead using a generalized statement. The first sentence also made no sense. But, what threw me off was the lack of links in the post - the spam bot had just instead used the regular “homepage” field to fill in the URL of their spam blog - or splog - in the hopes that someone would visit it.

In fact, due to my curiousity, I had to visit the site to see what was on it. (This was, of course, how I determined it to be a splog) The splog consisted of a load of nonsensical posts, and of course, lots of ads, by Google no less. Since it’s against the terms-of-service of Google Adsense to make a site for the direct purpose of displaying Google ads, this spammer is obviously in clear violate of the program. The spam blog wasn’t littered with links though (besides the ads), and the posts would look human after only a quick check - a closer inspection reveals sentences merged together into nonsensical, run-on paragraphs.

The Turing test

All of the efforts by current spammers reminded me of the Turing Test. Basically, it’s a concept that if a person communicating with a computer cannot reliably tell if they are communicating with a computer or a human, then that computer system is said to have passed the test. Turing thought it was a better way of evaluating a computer over the question of “Can a computer think?”

In a way, spam and anti-spam techiques are engaged in some sort of Turing test. Except that it’s a computer system (the anti-spam system) that is trying to determine if an entity is a computer (spambot) or a real human. The anti-spam techniques described above have typically been widely used, along with CAPTCHAs or other “tests” that the typical human can easily do, but are relatively hard to design into a computer program or system.

However, as programming techniques and algorithms advance, the spam/anti-spam war is sure to heat up. As seen by some of the comment spam I’ve started to get, spammers are getting better with the bots they produce. They’re moving away from posting messages that are heavily-laden with links to gambling/porn sites that are obviously spam, to posting messages that could have conceivably come from a human, with only the regular home page link that any curious visitor might click. As these techiques advance, it’s entirely possible that a spam bot could read a blog post, analyze the content, and post a reasonably-specific message that contained a few links here and there to spam sites. The same goes for CAPTCHAs as well - advances in image processing are making optical character recognition more and more accurate.

The end result is, as I mentioned at the beginning, a lower SNR and lower quality of information on the web. Not only will more useless spam information be disseminated, making it more likely to find crap rather than helpful information when doing a search - but it’ll also be harder to post actual information that isn’t incorrectly labelled as spam. For example, if spam bots were able to post vaguly post-specific comments to sites, then quick remarks by people, such as “Thanks for the useful information!”, might also get labelled as spam. Same goes for CAPTCHAs - if we make the images more distorted and the characters harder to read, people might also have trouble - I know I do on some of them - and this creates another problem of accessibility for the disabled.

This isn’t to say I’ve lost hope. Akismet, so far, has only missed one comment - and I would have missed it had I not visited the linked site. So, it hasn’t really let me down. Others have had similar success, so for now, I think anti-spam techniques have the upper hand. Spam is merely an annoyance and a statistic currently, and I hope it stays that way. What I’ve talked about in this entry could be viewed as a worse case scenario of sorts.

Now, excuse me while I press “Delete All” on the spam comment moderation queue.

During this latest round of spamming, which reached its peak this weekend, it appears that well over 5 billion spam pages were indexed by Google; while this by itself is a huge number, taken in context with the total number of pages that Google had indexed at the time, around 25 billion by the source in the first link, it is simply astonishing. What’s even more impressive, or scary, is the fact that the site was started only less than a month ago, making this intrusion into the Google search indexes not only massive, but frighteningly fast as well.

From reading the posts at Digg, and from the resultant link, it appears the spammer used a script in order to serve up articles based on keywords, and furthermore, utilized many topical subdomains in order to generate the content that would appear “high” on the keywords list, and thus be indexed by Google. Comment-spam (on forums, blogs, and the like) may or may not have played a role in getting the pages ranked higher, but one thing is for certain - these useless pages made it very high onto the search results page, in many cases filling multiple spots in the top 10 results. These searchs were for common terms, such as “war on terror pros cons” and “pizza sauce recipe”.

But what’s the reason for this? Well, the same as for any spam marketing campaign - advertising. Because of the currently huge market for Internet advertising, the potential for making lots of money of ads on popular sites is an opportunity many cannot turn down. You’re likely to see these somewhat unobtrusive text ads on most any popular site nowadays - in fact it was Google who first popularized them as a replacement for the annoying animated graphic banner ads and popups, which put off many viewers. This form of advertising is, undoubtedly, the backbone of many web 2.0 companies. Companies like Digg, Flickr and even Google rely on ads for nearly 100% of their revenue.

But this potential has turned many to the dark side of advertising - creating spam sites whose sole purpose is to attract viewers for increased ad viewing. While successful web 2.0 companies may display ads on their site, they all offer some useful service that people return for. These spam sites do not offer any useful service or information, but instead manipulate search engine results in order to trick users to visiting their site. Once there, the user will find only semi-meaningful information laced intricately laced with ads, or perhaps, no information and only ads. While this clearly violates many ad providers terms-of-service (such as Google Adwords), most sites have no problem doing this or finding marketers who don’t care about such trivial things.

This is perhaps the other side of the double-edged sword that is the Internet. On the one hand, forums, blogs, and other community-based sites offer the immensive capacity for spreading useful information. On the other hand, they also offer the ability to spread useless information as well, and in some cases, search engines cannot yet discriminate between the two as most humans would. This can be seen in the huge amounts of comment spam, and spam blogs that pervade the Internet.

All of this creates problems on many levels, and in many ways, is more damaging that e-mail spam. While e-mail spam is annoying for the junk it creates in our inboxes, and the extra bandwidth it consumes, for the most part anti-spam tools have helped curb this influx. However, search engine spam targets the most basic use of the Internet, and that is the ability to find useful information. With all the spam sites out there, and the manipulation of search engine results that comes from this, the ability to conduct a search that returns useful information may be compromised in the future if proper countermeasures are not employed.

Furthermore, it creates a nightmare for the people who engineer the search engines, as they must find a way to tweak the algorithms to prevent this from happening again. In the process, false positives may be generated, causing legitimate sites to be unintentionally delisted, causing futher headaches. Google has been having delisting problems as of late, and one wonders if this is related to the recent spamming problem.

One also has to wonder how many of these problems may have come as a result of the upgrade Google did to its datacenters, dubbed “Big Daddy”. Google did not seem to have these sorts of problems before this, so perhaps there is a correlation, but maybe not a cause. It’s interesting to note, however, that the aim of the “Big Daddy” updates was to prevent this sort of thing - and to keeping meaningful sites in the index, which is exactly the opposite of what happened, since the spamming from these sites evidently bumped out important sites from the indexes.

This recent round of spamming was not unique to Google; it affected the Yahoo! and MSN search results as well, though not to the extent that it did to Google’s. It was probably not the intention of the spammer to get so many pages indexed on Google, and probably got “out of hand” quickly, however, one has to wonder if this spam site directly targeted Google in its quest for search engine manipulation, or whether this was just a coincidence. But the problem for search engines remains, and that is, how to effectively discriminate between meaningful and useless information, without making too many mistakes, one way or the other.

Thankfully, it appears the Google is working on fixing this problem, and not only by just removing the most recent spams. It will take some work, but I think they should hopefully arrive at a solution, but as always, spammers will always be working to gain the upper hand as well. Let’s hope that the combined effort of companies like Google, Yahoo! and Microsoft can thwart them, or else the Internet may become awash in the useless garbage of spam.