The Hacker Factor Blog

There were a couple of recent technical topics that I wanted to write about this month. However, I just cannot get enough information to write in a knowledgeable fashion. So I am just going to summarize what I know about two of the topics here...

Google versus China

Earlier this month there was some potentially very big news. Google accused China of a cyberattack. They claimed that China attacked their systems and looked up information related to dissidents.

Google responded by threatening to end their censorship service in China and to pull out of China.

The reason I say "potentially very big news" is that detailed information was lacking. What kind of cyberattack? There were rumors about a remote exploit or assistance from an insider. And while some of the claims clearly say that China was behind it, Google's CEO was less than certain. Newsweek quoted Google CEO Eric Schmidt as saying:

We came across a lot of evidence of the monitoring of Chinese dissidents through the Web. We do not have clear evidence as to who was doing the monitoring, but you can draw your own conclusions.

Without more details, this is nothing more than technical heresay. No wonder the Chinese government has denied the hacking claim and calls the accusation "groundless".

We need more details: What evidence was observed? How was it traced to China? Was it an exploit or an insider? There are even claims that other companies have been attacked by China. But without details of the kinds of attacks, how can we know if it is happening, how to detect it, and how to mitigate the threats?

IE, Germany and France

Every month Microsoft releases patches for their software. Dubbed "Patch Tuesday", companies prepare for the updates and schedule patch cycles. The updates include bug fixes and critical security patches. This is not new -- it has been going on since at least 2005. On occasion, Microsoft also does out-of-cycle releases. These are primarily for critical vulnerabilities.

As system administrators, we know that patches are par for the course. I was much more concerned with Microsoft products back when they rarely released patches. Regular and predictable updates are a good thing, as are immediate fixes for critical issues.

However, following a recent patch with "yet another critical fix for Internet Explorer", the German Federal Office for Information Security (BSI) made an announcement: "Deshalb empfiehlt das BSI, bis zum Vorliegen eine Patches von Microsoft auf einen alternativen Browser umzusteigen." ("Therefore, the BSI recommends to switch to the existence of a patch from Microsoft to an alternative browser.")

In other words, until details of this vulnerability (that was allegedly used by China to attack Google) are known and a patch is available, you should use a different browser.

Not to be outdone, France made a similar recommendation: "Dans l'attente d'un correctif de l'éditeur, le CERTA recommande l'utilisation d'un navigateur alternatif." ("Pending a patch from the publisher, CERT recommends using an alternative browser.")

Instant Replay

There are two things going on here. First, we have the reported cyberattack against Google. Second, we have a vulnerability announcement from Microsoft. I have yet to see any details linking the two. As a result, we have nations taking on big companies: Google vs. China, and Germany and France tag teaming against Microsoft.

So let's recap: Google says that China attacked them, but does not say how. Google's CEO gives no specific details and ends by saying "you can draw your own conclusions." And reports on the attack range from a very specific, directed attack to assistance from an insider. Based on this unknown and speculative information, Germany and France have decided to recommend against using IE until a patch for this unknown threat becomes available.

Just Enough Information To Cause Damage

Now keep in mind: I would not be surprised if China did a cyberattack against Google and other companies. But if we're being open and honest, then I want details. Where's the full disclosure?

While a vulnerability is known in IE (and a patch has since been made available), the recommendation to drop everything and change browsers seems a little harsh. I mean, seriously, Linux and Mac OS X both have frequent updates for vulnerabilities. Microsoft is no different in this regard. Why should I completely give up my platform if there is some vulnerability without a patch? Using this logic, no operating system is safe since there are always pending (not yet released) security patches.

And while I have spent a decade recommending against using Internet Explorer, my reasons are based on historical precedence and not a knee-jerk reaction to an unknown threat. If Germany and France had said, "IE has a long and established history of serious security vulnerabilities and the trend does not seem to be ebbing. Other browsers, like Firefox, generally have fewer vulnerabilities and issues fixes faster." then I would not criticize these countries. However, that isn't what they are saying. They are saying that there is a potentially new and unknown threat that may or may not have been used against Google and possibly other companies, so stop using IE because it is at the center of the rumor!

Keep in mind, if the attack was specific and directed toward Google, then you -- the home user -- are not at risk. China is not going to attack Ma and Pa's home computer in central Kansas. Moreover, even if IE was used as part of the attack, there is no reason to fear it if an insider is required. (Does Ma and Pa have a Chinese exchange student in their house? No? Then there is no insider threat from China.) For the average user, threats from phishing and identity theft are much more troublesome than directed attacks from China (assuming that China is doing the attacks in the first place).

Before we can make a decision, before we can logically react to a threat, we must have information. And right now? We only have speculation and vague and varying details.

About a three years ago I wrote an article for Security Focus about online impersonations. In the article, I discussed a big problem with many online services: there is no validation required. When opening an account with Google, Facebook, Wikipedia, Twitter, and all other free Internet services, you can claim to be anyone. Impersonation is easy.

While some online impersonations are nothing more than harassment, others can have very serious consequences. For example, last week a Wyoming man was arrested for impersonating his ex-girlfriend on Craigslist and soliciting a rape fantasy. (That's where a stranger shows up and acts out a rape. Yeah, really sick stuff.) Of course, the ex-girlfriend was not actually involved in the solicitation -- she was being impersonated online -- but she was still raped. (Gee... wonder why he is her "ex"?)

There are many other examples of hostile online impersonations. For example, a man from Spokane, WA impersonated his wife after killing her in order to make it appear as though she was still alive. Poker celebrity Patrik Antonius was impersonated on Twitter by a stalker. And of course, phishing is nothing more than a scam based on corporate impersonation.

Fortunately, most online impersonations are not life threatening. However, you never know what the impersonator intends. They might claim to want to be your friend in order to steal your money or manipulate you. It is for this reason that I take all forms of impersonation very seriously.

You work for who?

Of all of the social networking sites, I find LinkedIn to be the most interesting. While other sites intend for you to meet new people, LinkedIn is designed to help you contact associates of professional associates. The concept is based on common business relationships: if I know 25 people in the computer security industry and you know me, then you probably could benefit from knowing the kinds of people I know. Or to put it another way, if you and all of your Facebook friends and their friends got together, it would be a big drunken party. If you and all of your LinkedIn contacts and their contacts got together, then it would be a productive (and likely profitable) business luncheon.

I recently received a LinkedIn "request to connect" from someone I did not know. While I reject requests from strangers, this one got my attention. It was an impersonation. I know -- because he claimed to work for Hacker Factor. My company, Hacker Factor, is not very large. There are only a handful of people who can actually claim to be associated with the company -- and "Jerry siganteng" is not one of them. Moreover, in his request to connect he claimed that he "trusts me". (Clearly he does not know me that well.)

This is where things get interesting. LinkedIn makes it very easy to report self-impersonations. However, this person was not impersonating me. He was pretending to be someone who works for my company. LinkedIn claims to take personal histories very seriously -- and this person was lying about his business affiliation. On 10-Dec-2009, I reported him to LinkedIn.

Shortly after submitting my complaint (he does not work for me!), I received an automated email reply. However, the human reply did not come until Dec 17 -- a full week later:

Dear Neal,

Thank you for informing us of this situation.

It is against the terms of LinkedIn's User Agreement to misrepresent positions and qualifications in the LinkedIn profile. We will notify this member of the complaint against their account and take the appropriate action if needed.

We appreciate your assistance in making LinkedIn a professional and trust worthy site.

Regards,

LinkedIn Privacy Team

Unfortunately, LinkedIn dropped the ball. On 29-Dec-2009, this imposter was still listed on LinkedIn. I wrote to LinkedIn again. They responded a week later:

Dear Neal,

Thank you for your reply.

We will go ahead and take the appropriate action on this account since we have had no response from the account owner.

If you have further questions, please feel free to reply to this message.

Regards,

LinkedIn Privacy Team

Poof -- the impersonator was gone.

So the good news is: LinkedIn does take impersonations seriously and does disable accounts associated with imposters. The bad news is: it took nearly a month and I had to remind LinkedIn about the impersonator.

But why?

I still don't know what the impersonator really wanted. He might have been a spammer -- direct LinkedIn contacts can see your email address (unless you choose not to share it). He might have wanted to see my contacts in order to harass or con them (seeing contacts is another option that can be enabled or disabled). Or perhaps he wanted to appear legitimately associated with my company.

While impersonating a small company like mine does not seem that significant (to anyone except me and my associates), imagine if this were a bigger company. An outsider who claims to work for IBM could connect with other IBM employees and use the contacts and associated reputation to social-engineer an attack against IBM. ("Hi Bill @ IBM, I'm (fake) Matt at IBM and I know (real) Joe and (real) Ed at IBM. Could you help me reset my remote user password?")

It is unlikely that the guy impersonating a Hacker Factor employee was specifically targeting Hacker Factor. It is much more likely that he has accounts impersonating hundreds or thousands of companies registered at LinkedIn. And while a small company can easily identify a fake employee, I wonder how many big companies would notice...

Two weeks ago, USA Today featured an article titled, "Cybercrooks stalk small businesses that bank online". This article discusses some keylogging malware (banking trojan) that watches for when people login to their banks. However, it includes the following text in the first paragraph:

The American Bankers Association and the FBI are advising small and midsize businesses that conduct financial transactions over the Internet to dedicate a separate PC used exclusively for online banking.

This is amazingly bad advice.

• If a small or midsize business (SMB) doesn't have a dedicated administrator to manage one system, then how can we believe that these same SMB can easily manage two computers?
  
  
• People are going to copy data between systems. If they are connected over a local network, then the malware will spread. If a USB drive is passed from one computer to the other, then the malware will spread.
  
  
• If the antivirus software on one computer does not notice the malicious code, then you can be certain that the same ineffective antivirus software is running on the other computer!

Two computers is not a solution.

The Paper Trail

While I have always been critical of the FBI, this level of bad advice is very surprising. I tried to identify the source of this statement (since USA Today does not cite the actual source). Here's what I found:

• The USA Today story came out on 30-Dec-2009. However, neither the FBI nor the ABA released any kind of press statement around that date. In other words, this was a filler-story.
  
  
• On 3-Nov-2009 (nearly two months earlier), the FBI released a press release concerning malware that targets SMB companies and online banking accounts. (The IC3 released a similar statement on the same day.) The description from the press release matches the general malware description found in the USA Today article.
  
  
• The FBI press release references a US-CERT paper from 2006 (that was revised in 2008). On Page 4, the paper says:
  

  Avoid situations where personal information can be intercepted, retrieved, or viewed by unauthorized individuals.
  
  You should conduct online bank transactions in locations that are not subject to public monitoring. When you are entering login information, you should avoid using unsecured or public network connections (for example, at a coffee shop or library). As a general rule, you should avoid using any computer that other people can freely access; the end result could be unauthorized access of your financial information. Remember, it is possible for your account information to be stored in the web browser’s temporary memory.

Now keep in mind, the US-CERT paper, FBI press releases, and ABA press statements never says to use two different computers. In fact, I could find no reference that gives the bad advice found at USA Today. I would not be surprised if USA Today just made it up. (In industry, we call it "MUS"; Making Up Sh*t.)

The advice from the FBI and US-CERT is good, but not great. Their advice is to not use public computers or anyone else's computer. (It's the same concept as washing your hands to stop the flu.) In other parts of the paper they advise people to be vigilant and take action when you see something incorrect with your online banking account.

While these tips are good, they overlook one significant item: they put the onus of keeping your bank account safe on the end-user. However, banks should share the responsibility.

Interest-Free Banking

The banking industry has never been known to take proactive security measures. They didn't start using vaults until after people stole the safes. Alarms were introduced after burglars began robbing the vault after hours. And banks did not even begin using HTTPS until after there were compromises by packet sniffing the HTTP connections. (I remember being told by a major credit card provider that they were not interested in anti-phishing solutions because phishing was not a big enough problem.)

Requiring non-technical users to monitor and prevent online banking theft is idiotic. We don't ask investors to stand guard at the bank's front doors, so why should we ask users to stand guard online? Banks should take a proactive approach. With a well-designed security solution, users should be able to bank safely even if their computers are infected with a virus. Here are just a few ideas that can lead to more secure online banking:

• HTTPS. SSL (Secure Sockets Layer) is a security framework. HTTPS is simply HTTP with SSL. The way we normally use it, the client authenticates the server but the server does not authenticate the client. Banks need to issue client-side certificates. By tying the banking session to the certificate, they reduce the exposure from compromised login credentials. An infected system may still be remotely used to access the bank, but credentials offloaded to a system in Russia will be useless. (There are many caveats here, such as the horrible browser interfaces for installing client certs, how to transfer the cert to the user, and cert theft, but these issues can be worked around.)
  
  
• Hardware tokens. A better solution than client-side certificates for SSL is the use of hardware token generators, like SecurID cards. These handy devices generate random numbers. Even if you know my login and password, you won't be able to generate the correct random number -- so my account will be safe. The best part is: if banks could standardize this, then you won't need one SecurID card per account; you will just need one. The same SecurID card could be registered with every bank account you own.
  
  
• Dongles. If SecurID cards are too complicated, then how about a hardware dongle? Even something simple that takes a numerical challenge and computes a hash is better than nothing. (Ideally, every client gets a different hash computation. The easiest method is to seed a random number generator with a combination of a serial number and account number.) Without the dongle, an attacker will be unable to access the bank account.
  
  
• User-managed credit cards. The last few digits on your credit card is a checksum. It is algorithmic and computed. Not only that, but the LUHN Algorithm is well known. The checksum was needed back in the days when vendors took carbon images of your credit card. Since the image could be hard to read, the checksum was used to identify errors. (That's not the only reason, but it gets the point across.) Today, cards are scanned (and in some cases, a human types in the checksum value). The checksum is not essential.
  
  Rather than using the same non-essential checksum, credit card providers can leave those digits blank. Allow users to select the numbers. When you swipe your card, you will type in the last digits rather than the cashier. (The same use model as a PIN number, without the pain of changing your PIN number. Also, PINs are only used with debit cards; all cards should require them.)
  
  When users are concerned about a potentially compromised card, they contact their banks (via phone or online or text message) and change the last four digits of the card! While the decision is still up to the user to decide when to make the change, at least users still have the option. (I would buy more things online if I had the ability to change my credit card number after making purchases with smaller vendors.)
  
  Moreover, card providers could link the digits to a SecurID or hash-based token. Since transactions already record the date and time, this is not an additional burden on anyone except the bank. Now, if someone steals your credit card number, they get nothing. This is much better than building RFID chips into credit cards since the RFID chips can be easily compromised.

The idea here is two-part authentication: something you have (a cert, token, or dongle) and something you know (password). With two-part authentication, it does not matter if the user's computer is infected with malware. An attacker cannot hijack your account since -- at best -- they will only have one of the two parts. They may capture your login credentials, but they won't be able to login if they cannot access the cert, token, or dongle.

If the banks were really interested in protecting accounts, then they would take proactive measures and not put the responsibility on the consumer. In this kind of consumer utopia, we would not have to worry about infected computer systems, or mass media outlets promoting bad advice.

There is an old saying that systems are as secure as the weakest element. However, the weakest element is not always the human. In this case, the infrastructure around account management and online banking is weaker than the human element. With a minor amount of effort, credit card and online banking can be made significantly more secure without blaming the customer for account compromises.

I like the FreeImage library for rendering and saving image. From a programmer's viewpoint, it is clean code, easy to use, and supports 99% of the images I come across. Moreover, I like their license. The FreeImage Public License (FIPL) does not put on restrictive stipulations (unlike GPL, which dictates how the code must be used).

Unfortunately, FreeImage only supports 99% of the images I come across. That remaining 1% keeps biting me. The first time I got seriously bit, it was with a very odd GIF image. Firefox, Internet Explorer, and ImageMagick worked fine, but FreeImage claimed it was corrupt. (Technically it was corrupted.) I ended up writing my own GIF decoder.

Recently I needed to process some ICO files, as in "favicon.ico". On any give web site, the http://site/favicon.ico file (when present) provides the tiny graphic that appears in the web browser's address bar. (My site looks like a tiny computer with a keyboard.) Now, I know what you're thinking: what kind of analysis can you do with a 16x16 graphic? You'd be surprised...

Parsing ICOs

The biggest problem I had was finding information on the ICO image format. For example, the information at Wikipedia has a few mistakes and big oversights. You cannot implement the format from their information. Better information comes from Daubnet, but even it is not complete enough for implementation.

After some reverse engineering based on hints from Daubnet, I finally got the format details. ICO is really just a wrapper around really tiny bitmaps (the largest are 256x256 with 8-bit color). However, this is a Microsoft format -- that means it is also really screwed up.

The Easy Part

Each ICO contains three sections: header, index, and data. The header is the simplest part. It contains six bytes. The first two are reserved, the second two are the type (in little endian format), and the last two are the count. The type is either 1 or 2, indicating whether the ICO contains an icon or a cursor, while the count identifies how many images are stored within the ICO file.

typedef struct

  {

  byte Reserved[2];

  byte Type[2];

  byte Count[2];

  } ICOheader;

Immediately following the header is the index array. There is one index element per image (the header's count). Each index stores information about each image: width, height, color depth, and so on. However, each values is overloaded with alternate meanings. For example, the width and height each take one byte. If the width is 7 then it means the image is 7 pixels wide. However, a width of 0 means it is 256 pixels wide. So the smallest size is 1x1 and the largest is 256x256 (represented as 0x0). Similarly, the number of colors is one byte. A value of 3 means there are 3 colors, but a value of 0 means the image uses the maximum palette size, or true color if the image uses three color planes. (Lots of 'ifs' and conditional meanings.)

typedef struct

  {

  byte w; // image width: 1-255; 0=256

  byte h; // image height: 1-255; 0=256

  byte colorcount; // number of colors in the colorcount; 0=max possible

  byte Reserved;

  byte V1[2]; // Icon: color planes; Cursor: x for hotspot

  byte V2[2]; // Icon: bits per pixel; Cursor: y for hotspot

  byte size[4]; // size of the bitmap data

  byte offset[4]; // offset to data

  } ICOindex;

The critical values for the ICOindex are the width, height, colorcount, offset from the beginning of the file to the data, and the size of the data (number of bytes). Other values are redundant or unused. For example, the number of planes (V2, but only when the ICOheader says the Type is a cursor) is usually 0 or 1, but the value does not seem to be used for anything.

Into the Abyss

The data section itself is a variation of the bitmap (BMP) format. It contains a header that describes the image, and then the raw data itself.

typedef struct

  {

  byte HeaderSize[4];

  byte Width[4];

  byte Height[4];

  byte Planes[2];

  byte BitCount[2]; // bits per pixel: 1, 4, 8, or 32

  byte Compression[4]; // 0 for uncompressed

  byte ImageSize[4]; // 0 for uncompressed

  byte XpixelsPerM[4]; // unused

  byte YpixelsPerM[4]; // unused

  byte ColorsUnused[4]; // unused

  byte ColorsImportant[4]; // unused

  } ICObmp;

As you can see, there are lots of unused fields, like the relative ratio of pixels per meter. (Dude! It's 256x256 pixels at best! Who cares if it is 0.1 or 0.12 meters!) Similarly, at 256x256 we are not talking about huge megabyte files; compression is not used (value is always zero).

Other information is redundant. For example, the ICO index gives the image width and height. The maximum size is 256x256. So why does the ICO BMP restate the image size using two bytes per dimension? This is just wasted space.

Following the BMP header is the color map -- another conditional field: IF the bits per pixel (bpp) is less than 32 THEN use the color map, otherwise there is no color map. The number of colors either matches the index colorcount, or colorcount is zero and it matches 2^bpp. For example, if the colorcount is zero and bpp is 4 then there are 2⁴=16 colors in the color map.

The color map (when present) contains 4 bytes per color, and since this is a BMP, the colors are BGR instead of RGB. (The fourth byte is unused and should be zero.)

Finally, we have the data. One would think the data would be straightforward: One set of bpp per pixel. If the bpp is 32, then it is the color (BGR+unused), otherwise it is the index into the color map. But this is a BMP, so it needs to be a little more complicated.

Each row in the BMP is padded so the row fits into a 32-bit boundary. Thus, if there are ten 8-bit indexes in a row then there are 10 data bytes + 2 byte padding per row. With smaller bpps, like 2-bit data, then 10 pixels in a row = 20 bits + 12 unused bits for padding = 32 bits.

Sounds simple enough... in fact, it is a little too simple. This is a BMP, and there is a rule somewhere that Microsoft BMPs cannot be simple. While other image formats (like JPEG, PNG, GIF, and PPM) render from top to bottom, BMPs render from bottom to top. Oh, and when reading bits, the first bit is the MSB and not the LSB. That's right: the BMP is rendered in BGR, from bottom to top, with 32-bit aligned rows, and bits are processed from MSB to LSB! (Everything is reversed!)

Now for the stupidity: I can understand why it might be nice to align values on a 32-bit boundary. 32-bit alignment can help speed up caching and CPU pipelining. Except that this is a tiny file to begin with, so caching and pipelining are not big issues. And even more: the initial header is 6 bytes, not 8! So nothing after the header is aligned on the 32-bit boundary. That's right: ALL data, including the color map and BMP data is a multiple of 32-bits wide and offset 16-bits! Nothing is aligned! Any possible benefit from alignment is broken!

Multiple Images

ICO allows a single image file to contain the same image at multiple resolutions and color depths. This way, your browser can choose the best image to display in the address bar. While this is good in theory, it does not seem to work that way.

Here's a fun experiment. The web site Zoominfo has a favicon.ico that contains four icons:

1: 16x16, 16 colors

2: 32x32, 16 colors

3: 32x32, true color

4: 16x16, true color

Every web browser I have seen so far uses the last image in the ICO file for the address bar logo. It does not matter if the last image is the lowest quality or less optimal size. Browsers use the last icon.

However, if you load the ICO file in the browser, then you may see a different icon: http://www.zoominfo.com/favicon.ico. With Firefox, you will see the last icon. Safari will show you the best icon (in this case, the third one that has the black border). But IE6 shows the first icon.

While I consider JPEG to an inconsistent format, and TIFF is the kangaroo of image formats (because it jumps all over the place), the ICO format wins for being the stupidest format. It is as if the designers tried to wedge one file format into an unsupported purpose. In the process, they broke any potential performance benefits, replicated information, and overloaded unrelated functionality. Worst of all: nobody seems to support the file in the way it was designed.

Are we done yet?

But there's one more fiasco to the ICO format. It seems that the rest of the world did not want to be limited to a Microsoft BMP file format. In 2005, the World Wide Web Consortium (W3C) added support for favicon.ico files to be ICO, GIF, or PNG. That's right, your file that ends with ".ico" may not be an ICO at all.

Many years ago I worked with a software architect who never considered Denial-of-Service (DoS) attacks. He knew computer security, but strongly believed that all DoS attacks are the same and you could not defend against them. After I wrote my first book on network security, I became keenly aware that there are differences in DoS attacks. Moreover, if you understand the attack then you can defend against it. For example, an ICMP Smurf attack (where you become flooded by ping response packets) can be mitigated by dropping ICMP packets. A UDP Smurf attack like Fraggle, can be mitigated by blocking the destination port. And SYN Cookies are an awesome options for mitigating TCP SYN Flood attacks.

With most DoS and Distributed Denial-of-Service (DDoS) attacks, the packets have some kind of similar pattern. If the attack is TCP, then you won't see UDP. If the attack targets port 22 then you won't see it going after port 21. And if the packets are generated by a program that uses a TTL of 58, then every packet will begin with a TTL of 58. If you can fingerprint the attack, then you stand a chance of defending against it.

DoS and DDoS attacks have one other common feature: they don't last forever. Granted, they have the potential to cost you money or put you out of business, most are not that severe. The attacks may be painful, but they are only temporary; either the attacker will stop, the victim will mitigate the attack, or the resources used by the attacker will no longer be available.

I recently experienced a type of DDoS attack that I had never considered. It was the equivalent of a flash-mob. And unlike your typical DDoS, this one keeps going and going and going...

A Modest Beginning

On 2-Nov-2009 I posted a blog entry that analyzed a digital image. I didn't think much about it at the time -- I had been doing similar postings for months without any fanfare. (I joke that my blog has 12 loyal followers.) At the time, nobody even noticed the blog entry -- no excessive volume, no links from other sites, and only a few comments.

All of that changed on 27-Dec-2009. My automated monitoring system alerted me to an abnormal increase in network traffic. And abnormal was an understatement: volume increased by more than 1000x. My web logs easily identified the source: a single twitter post by Tim O'Reilly.

Body by Victoria. Fabulous CSI analysis of photoshop mods in recent Victoria's Secret photo: http://bit.ly/5QnMuB Lovely forensic analysis
2:09 PM Dec 27th, 2009 from Seesmic
timoreilly

Normally I wouldn't think much of a Twitter post (how much traffic could one tweet generate?)... except that Tim has 1.4 MILLION FOLLOWERS! And a significant number of them re-tweeted (RT) the post. For the first eight hours, my site was pummeled by Twitter users!

After about 4 hours, the attack spread. It went from Twitter to Facebook and then to Reddit where it was posted twice (one two). Eventually it even went to Digg.

While the server was able to keep up for the first 6 hours, the flood eventually became too much. Users began to get "503 Service Unavailable" errors. Users who hit re-load were able to view the page.

Now keep in mind, this server has been able to handle the Slashdot Effect without a problem. (Sites featured on Slashdot receive so much traffic that they may go down.) I've had my content slashdotted many times (including one period where my work was slashdotted 4 times in 5 weeks, with 2 of them mentioning me by name). The server could handle this kind of load. However, Twitter + Reddit + Digg was simply too much. This was a flood. This was a flash-mob. This was a distributed denial-of-service that was different: every packet was from a different user and looked different. While the attack was unintentional, nonetheless it was large enough to cause server problems.

The Impact

My mentor once told me how to determine if you are under attack: once is a fluke, twice is coincidence, three time is an attack. If a user accidentally deletes a critical file three times, then it is still an attack even if it was unintentional. In this case, I received an extended high volume of traffic. Regardless of the intention, it is an attack. Moreover, the server was not configured for this new kind of attack.

Interestingly, the impact really didn't hurt the server. Although users received "503 Service Unavailable" errors, the server itself was doing great. Even with all of my scripts and back-end monitoring systems, the server load was relatively low. The back-end database was operating fine -- no errors, not even mildly stressing the system. And the network load was negligible; the thick pipe was nowhere near clogged.

What was failing was the sheer number of concurrent connections. The web server could only handle a finite number of concurrent connections. While the server can accept more, it will return a 503 error if the total number is greater than the permitted volume.

On my blog, I can actually distinguish between a human and a bot. At the peek of the attack, I was receiving over 8000 humans per hour (up from the typical 8 per hour). Of course, bots grossly outnumber humans; at the peak I was receiving over 100,000 hits per hour.

Now, 100,000 hits per hour doesn't sound like much. (That's only 27 per second.) However, web browsers and bots don't immediately close the connection. Instead, they keep it open for a few seconds (or even a few minutes!). The idea is that a user is unlikely to perform just one hit (web request). Instead, they will perform multiple requests -- downloading images, style sheets, and more. By keeping the connection open, the browsers improve performance. (You don't need the TCP three-packet connection overhead.) Now the number of concurrent connects begins increasing. 27 open the first second, but 54 are open in the next second. At this rate, the flash-mob has the ability to consume all available network connections.

As an aside: Even if your server can handle 65,000 concurrent connections, the web server is usually restricted to fewer connections. The idea is that a busy web server should not block access to other network services. (Even though I could not connect to my blog, I could still connect using SSH and transfer files.)

The gift that keeps on giving

Although the social networks came first, they were just the start. The link was passed to other blogs in the form of comments. For example, someone mentioned it at Create Digital Motion. Each popular blog created a small burst in volume as people followed links to my site.

The social network flood began to ebb after 24 hours. However, that's when popular blogs began to reference it. The first big one was Boing Boing. They increased the volume by about 400 human references per hour for nearly 48 hours. While I thought Boing Boing was big, it was nothing compared to Jezebel (a blog that covers celebrities, sex, and fashion for women). While Jezebel didn't have the same spike as Boing Boing, they generated much more sustained volume -- 72 hours later, Boing Boing dropped off, but Jezebel was still going strong.

Today, the volume is still about 10x higher than normal, and referrers are coming from more topic-focused forums. Interestingly, repost blogs (where people just repeat what others write) are generating very little volume. In contrast, people who create their own content create spikes in referrers. For example:

• Really Useful Knowledge is a blog run by a UK professor. He blogs about topics that interest his students. His post created a noticeable spike.
  
• Il Disinformatico is a popular Italian blog that focuses on disclosing misinformation. (Their initial spike dwarfed Boing Boing.)
  
• Nerve is an online forum. A few hundred referrers in a few hours.
  
• Aid Watchers took it in a very different direction. They related digital image analysis to forensic economics. (A very interesting read.) While the topic is very focused, the volume of referrers was significant and sustained.
  
• While the German SpreeBlick site covers many different topics, they generated a significant number of referrers.
  
• Meet The Gimp was the first site that focuses on graphics.
  
• (ISC)² has a blog that focuses on computer security topics.
  
• Models -- a site that focuses on... models. Heh.

As far as a running total goes: My web logs indicate a massive increase in the number of unique IP addresses that visited my site. The first 72 hours (Dec 27 - Dec 30) had nearly the same number of unique IP address as the rest of the entire year (Jan 1 - Dec 26). I received a year's worth of traffic in three days.

As I enter the second week of the attack, I'm seeing an uptake in automated attack systems. (If the jerk in Russia doesn't stop his botnet from probing my site, I'm going to post all of his IP addresses and set up an automated block. Here's a hint to the idiot: if one automated bot probe fails, then doing the same probe from a different bot is also going to fail.)

Preventative Measures

Even though every packet came from a different kind of system, there are still methods to mitigate the impact of a flash-mob DDoS without blocking visitors. For example, in the next month or two I will be moving my site to a grid network system. This will distribute the number of connections across multiple web servers. This move alone should be enough to mitigate the impact from any future flash-mob attack. Visitors will still be able to access the site and should never notice how it is hosted, even if there is high volume.