Hacked WordPress: “Vuln!! Path it now!!” and How I Fixed It (Forensic Investigation)

This entry is regarding a very, very persistent and pesky WordPress hack I experienced in February 2019, which is identified by “Vuln!! Path it now!!” inserted into an article while other articles were overwritten with links to illegal streams for recent movies like Hell Boy, The Green Book, and Marvel Girl.

Summary of the Solution

I talk about the entire discovery and agonizing troubleshooting process in this article. For your convenience, here is the summary if you don’t care about my story and only want to know how to fix your site:

• the hack overwrites your content with links to streaming content (or whatever they inject)
• the hack probably got in through a WordPress core vulnerability or plugin vulnerability (not sure which, but I had WordPress v4.4 at the time)
• the one thing that worked for me was changing my database (mySQL) password (and also change it in your wp-config.php)
• keep checking your website every 4 hours or so for the next 2 days — the hack runs about every 2-20 hours
• after you know your system is safe, patch your plugins/core, delete unnecessary plugins/themes (points of vulnerability), make sure your folder permissions are correct, and consider installing an automated security tool like Wordfence
• for extra security, place a file called vuln.php into your wp-content folder and make it unreadable (I explain why later)

The Situation

This blog,  ronswebsite.com, is one I have barely touched since 2014. I sometimes moderate it because of all the discussion that happens at my article about Hawaii’s General Excise Tax (300+ comments). Because I don’t update it much, my WordPress system became out of date and I did not put much effort into hardening the security.

Then one day, I visit my blog, and I see that comments are on a page called “Download and Watch Full Hellboy (2019)”… ? So I visit the page… and I see that the comments are still there for the tax article, but the entire thing was rewritten!!! Editing the page revealed this:

The URL, title, and content were all rewritten. The comments remained. This affected maybe a third of my articles… but not all of them. I had no idea how it happened.

In addition, there was another page and it was titled “Vuln!! Path it Now!!” and in the body it said “Hacked my Mister Spy.” I think they were trying to say “vulnerable… patch it now!!” Either way, I understand that hackers feel a bit of joy seeing the power they have, and these pages are like trophies to them. So, imagine how proud they must feel to search for their hack words and find this:

The Solution: First Attempt

The first step was to get things up.

THANKFULLY, because of WordPress’ design and keeping old versions through its REVISIONS system, my prior articles were still there. So, the first thing I did was I restored the articles and renamed the URL to the original one, so that links and google searches going to my articles wouldn’t be broken.

The second step was to understand how the hack occurred, to know better where the vulnerability was. As a precaution I

• changed my web system user password
• asked my web hosting company to restore my backup to a day before the first occurence happened

Backups are useful for that reason! With a little effort of backing up, you can jump back in time, which is infinitely better than having to revert your things manually.

As I breathed a sigh of relief however, my website was REFILLED with the hack content!

It became apparent that either

1. the hacker was actively controlling my website (unlikely… most hacks of non-special targets are done through automated methods… there isn’t enough monetary incentive for a targeted attack)
2. there was some sort of backdoor script written in (or maybe the hack was lying dormant, and was in my website, even before the first occurence)

I had to go with #2… they did not know my user credentials, because they did not log in. I know this because in the record, it showed no author in the hacked article, and because they still accessed my website.

Understanding WordPress or Web Applications

Web applications like WordPress are essentially databases + a website front end. They are usually built in a way that you need to login, and so only a person with a valid user account in the actual database can go into the admin screen (also part of the website front end) and make changes. The content on a website like this? All it is text inside a database. However, if there is a “backdoor”, they create alternative entrances, much like a thief can go into the window rather than through a door.

This is how I knew there was a backdoor of some sort.

The Solution: Second Attempt

I remember being a bit frantic that entire day because I had no idea how it happened. Also, it was essentially a time bomb, where it would run the script every 2 to 11 hours. I did not want my site de-indexed or to be put on some quarantine (infected sites will be flagged by search engines like Google), so it was semi-important to fix this. Also, it’s a nice challenge, with real stakes :).

Since I know the WordPress web system and the Linux LAMP server, I can understand where to look to solve my issue.

Here’s what I needed to check in my blanket attack on the hack:

• Check Existing Server Files for Modification
• Check Server for New Files Created
• Examine Server Access Logs
• Patch Software/Plugins
• Update WordPress

Idea: Check existing server files for modification

When there’s a backdoor, it usually modifies a file on the server. One common method I’ve seen before is that through a vulnerability, a script can modify a file and then encode the added hacked code, with something like MD5. This means that if you were to open it, you would see nonsense and nothing that would alert you. It also means servers or protections meant to scan for specific text will not find it.

These are easy to spot because the hack will change the LAST MODIFIED date stamp of the file. Files on your server only change when you update them, so if you see a date out-of-place, then you look there.

Result: Nothing. I searched through and through, but there were no files that were modified in the time span that I was hacked. 

Idea: Check server for new files created

Another red flag is if a new file is created. It’s the same method as the previous one: you create a script to run and do bad things.

These are easy to spot because they will show a recent CREATED or LAST MODIFIED data stamp. Also, if you know what the typical files are, anything new will stick out to you.

Result: Nothing. There were no new files. 

Idea: Check server access logs

Server access logs don’t lie. You can find info there for clues, but you need to search through it. Every time someone accesses the site, that gets logged. For me, I have 1000s of people visiting this site per month and 100s of bots trying to gain access. A server access log would be hard to read… these files are text, but are 10s of MB. Also, I would need a timestamp to know what time to look up in the log. For APACHE, you can find these in your root folder, in /logs/

Result: Too long… aint nobody got time for that. 

Idea: Patch Software/Plugins

One method of hackers is to target peripheral targets. The main target is usually heavily secured, but the main target employs outside vendors and connects to other networks or systems that might not be as secure. So, hackers can attack the weaker peripheral targets. This is how TARGET was hacked… through their HVAC company.

Attacking peripheral targets in web applications means attacking plugins and extensions. WordPress itself is pretty solid and have regular security, but plugins are a weak point and have been huge vulnerabilities. Even worse — once a single vulnerability is found by a hacker, they will do it en masse, since they know WordPress is the most popular CMS and a popular plugin might be used by tens or hundreds of thousands of websites. Also, I haven’t patched since 2014 :D.

Result: Patched… hope it works?

Idea: Update WordPress Core

Similar to the above, WordPress itself needs to be patched. Most people avoid doing it because of the potential for breaking other things, due to peripheral software/plugins being dependent on that particular version.

Result: Patched… hope it works?

Guess what… it didn’t work

Despite my good efforts, the hack happened again. Shit.

The Solution: Final Attempt

Troubleshooting a problem where you only know the symptoms, but not the cause is a troublesome situation. In some cases, you might be going through a tedious trial-by-elimination process of trying everything. However, it is a common situation for many jobs and it is an important skill.

After recovering from my heart attack from my second failed attempt (figure of speech), I was not demoralized. You know why? Because most of my adult life since I was 18, I have had to deal with broken vehicles breaking down on me. My moped, motorcycles, and sometimes my car… all giving me issues. But I didn’t have money and that’s what happens when you buy used. What I learned is that if you take a strategic approach, by trying things out, you can help eliminate potential issues and narrow the problem.

Here’s what I knew:

• it’s an automated attack
• directed at WordPress systems
• there’s a backdoor, not using WordPress user credentials
• no visible, residual files by hack
• no visible, modified files by hack
• the hack inserts text into database
• restoring a backup did not work

What worked: changing the database password

I realized what I needed to try was on the database level. You see, WordPress connects to the database. Inside the wp-config.php, are actually the database user and database password. I’m not 100% sure of the technical situation, but if a hacker has access to the server, the hacker could load a script which could access the database, because the database connection is already there. I wasn’t sure about this, because I searched top to bottom for evidence of added files, but I found nothing (more on this in a bit…). Or if they learn the credentials and the location of the database server, they could access it remotely, possibly (again, I don’t know the full details of this process).

In the end, changing the database password worked, when patching did not.

Followup: I found the script… It was Chinese hackers

I track visitor data using Google Analytics. Look at what I found during the month of Feburary: it’s visits to wp-content/vuln.php from a computer in China. That is server-side file, which runs, but should not be visited:

What I understand is that the vulnerability allows a file to be created, which when placed inside the server, will inject the content and overwrite my pages. Afterward, the file is deleted, so that there is no clue for forensic investigations, like the one I just wrote about (clever girl…).

I find it strange that they stayed for an average of 2.5 minutes before I had my solution. Does the file have some sort of user interface that the hacker manually plays with? I’m not sure.

What you can also do for further protection is create a file there called vuln.php and provide minimal permissions so a hacker can’t touch it. If it’s an auto-script, that might block it from running, since it can’t create the file nor delete it.

Bonus: Is there a person behind these sites????

I wrote down all the host of the illegal streams being linked to by my hacked website:

firimu.com
playsmovies.com
xinesmas.com
bytemovies.com

It turns out, these sites were registered by a guy named Burt ____ in Atlanta, Georgia. I then looked him up on Linkedin… and the guy is possibly a middle-aged accountant (there are several Burts in Atlanta)?? I looked up the address in the registry, and it seems to be a nice neighborhood. So is Burt the Accountant doing illegal stream hacking in his spare time???

Unlikely.

If a person is doing something illegal, there is no way they should put their actual information on the registry to link them to the activity. FYI: there is no authentication of the info you put on a registry. Also, it’s possible to pay and mask your registry info with proxy info. Likely, it was registered by some hackers out there and then Burt became a scapegoat. Sorry Burt, hope you don’t end up getting arrested by Atlanta Police Department.

I hope this helped you

Reader, if this helped you, please do me a favor and add your comment below. I honestly did not find anything on this, so I had to write it up myself. Let me know my forensic work is appreciated!