Around 4am this morning, I got a call that my sites were all down and that they had been hacked. It did not take long to get them back up and running because of my server setup. However, that was the easy part. The hard part was identifying what happened so that I could stop this m*****f****r lovely individual from being able to manipulate my sites again.
I discovered that the root of the problem was the Total CMS online demo. Since it's a demo, it's open to the public. This lovely individual had figured out that they could upload any file that they wanted into the Total CMS Depot. They uploaded a PHP file that then allowed them run commands and create files on my server. They never got access to the server, they simply uploaded files and ran them by visiting the file URL in the browser. Since all of their interactions happened over the browser connection, I was able to reconstruct every move the marvelous person made. Because of Total CMS's auto-backup feature, I have copies over every file that they uploaded, even though they deleted them from the depot.
I have shipped updates to both Total CMS and Easy CMS so that php files, as well as other potentially dangerous files, can no longer be uploaded.
It is important to note that this only affected me because my Total CMS demo is open to the public. I do this so that users can edit content in order to play with the CMS. In normal everyday use, you will not have your admin pages open to the public. They should be locked down and password protected.
A side affect of this incident was that some individuals had a window of time where licenses could not be verified, registered or purchased. This brought up a great question on another post about Total CMS having a "grace period" when checking and verifying a license. I have good news, this was already in place but today's update has made it better.
There being a grace period already in Total CMS is why no one has ever had this issue for 8 years now. In that time my servers have gone down. I have migrated them multiple times to different data centers. All without a hitch. Today caused an edge case that I had not anticipated. The good news is that I have already shipped a fix so this won't happen again. There was a less than 4 hour outage. In that time, not all license checks would have failed either. There were only a couple of circumstances that some users obviously hit. Sorry for the inconvenience to those that were affected.
That is the full story. I am super busy this week and didn't really need this to happen. When it rains, it pours as they say. I am leaving for Brazil on Sunday. I will be spending the summer down there with family. I will be working while I am down there. However, I think that I will need to put a pause on the live streams. Speaking of live streams, I was planning on skipping this week. However, me getting hacked to be an interesting conversation, let me know if you would find that interesting.
I also had to miss the big Apple keynote today... I think that I may go get a beer and watch it now.