Lots of people loved the OC, and I was certainly one them. More about that in the post mortem post to follow.
Details here are to the best of my knowledge, but some things are guesses without confirmation from people that are actually there, however if you read the associated news stories, I think you may agree that this all makes sense.
The OC aka Old Chat aka Original Chat went offline a couple / few weeks ago for a couple days or so. Then it came back and everything was fine. It ran for some time, then it went offline again. It’s been about two weeks or so – and the chat server has been unresponsive. Not just for those trying to use the chat room, but also from admin access. None of my access points are working to that server. I can’t get in, the chat can’t get out.
Why did it go offline?
The OC chat server was housed in an “extra protected” data center. Our server is one computer there; housed in racks of other computers running other web sites. Apparently someone hacked into that data center, allegedly to get access to a kkk database and publicly dump details from within (more recent news stories appear to show that was not the main reason for the hack but a by-product).
When the breach was discovered, our data center unplugged those racks of computers from the internet to clear any infected files, change passwords all that.
I got notice of a breach of the systems there, and an apology, and told all would be fixed, hang tight. The system did come back up for bit, and then I started to see news stories about the breach. It appears that various law enforcement agencies were brought in to investigate the breach.
Day or so later, the rack of computers that our chat server was housed in was unplugged from the internet again. I assumed they were replacing hardware routers and switches and cables to make sure the data center was clean of any possible lingering infections, but now it seems there is much more than that going on, as it doesn’t take weeks to replace routers and hubs.
So little info
I gave them time, understanding how severe of a situation this is for them. Then I started emailing and calling. Days went by and I got a message like “We are working to restore the hosting computers, there is not ETA. It may be a while.” – more days go by. Nothing new from our hosting company.
Piecing Things Together
Pouring through news stories and web searches, this is what I gather.
Whoever got into the ‘kkk database’ did so in a way that breached the deep levels of our hosting company. These people had gained access to a ton of information, not just the database that was publicly shame exposed. It looks like they downloaded / copied a bunch of PIA (personally identifying information) from a bunch of people who were not related to the kkks site. This of course kicks in some other rules and laws and stuff beyond the hack and exposing of the one site. Which makes sense why federal agencies would be brought in to investigate everything.
I am not a lawyer nor an expert in data security laws or anything like that. However given that I have gotten notices from other companies that have been hacked in the past it seems that this is a very costly situation for the hosting company, beyond what they are losing from not getting hosting fees from customers like us who have their servers offline.
So there’s a chance the OC could be turned back on
For the past couple weeks I have been confident that our hosting facility would get all their hardware patched up and our original chat system would be reconnected to the internet doing what it does best.
However there are a couple of big problems even if our server is turned back on.
It dawned on me the other day that if the server is plugged back into the internet and we can use the system again, we must consider that the system could of been compromised by an unknown number people. I can’t trust the server it’s on, or the code that makes everything run, and neither should you or anyone else. I count at least 5 teams of people that have unfettered access to our server at this point, and turning it back on would actually be worse than permanently pulling that system down.
Next logical step would be to reinstall our chat system on another server at a different hosting provider and get the OC running again.
Unfortunately the last remaining code for that system was housed on the compromised server. The original chat script that I had on my own computer was destroyed when my computer got a virus last year. The company that made the original chat script went out of business. The guy that made the code base for our original chat system is no where to be found, I’ve tried several times, no replies to any emails in months.
This is a sad and embarrassing day / week / month – every programmer in the world would denounce this lack of backups of code, and they would be right to do so. I purposely made no backups of our chat server system, but I should of had two or three backups of the main code and all the mods, and I did… but my old main system was destroyed last year, and I did not think to pull a backup from the server. So the code is gone.
I STRONGLY encourage you to read the added privacy information on that page.
Good thing we developed backup chat systems.
Many of our older regular users have known about a backup system that at one time was called the “new chat” – it’s not so new now, and we have newer systems running actually, however many people will migrate over to the SexChatSpace.com site’s “new chat” page when the OC is broken.
Some people will find their way to our newest chat system, the “mv-chat” (creative name for “mobile / video chat” huh?
We also have a “flashy chat” system that is running as another backup option.
I have started to develop an even newer chat rooms system with an eye on the future as well.
I made a profiles section here, and you can find a similar option on the sex chat space site – another way that people may be able to find each other / message each other then the chat are down. If things get real bad I try to post on a WordPress.com blog and our twitter thing. I’m doing everything I can to keep channels open for everybody to chat with each other.
The Mv-Chat system will be getting most of the attention for code updates. The mv-chat has many bonus features that the OC system did not have. There are a few things the mv-chat does not do well that was easy with the OC system, things like actions and others are mentioned and discussed on the mv-chat-info-page.
I am looking into adding encryption options for things around here, and we are also beginning to develop a new type of chat rooms system that is going to open up many more options for people. More on that in a couple months.
I’m also very open to ideas and suggestions. Every day I read comments and reply when they are left in the blog section here. Some people only engage this way on the kicked / banned page, others have found our site suggestions page or other pages where comments are open.
I spent a lot of time listening to people in the chat rooms the past week, and it was suggested that I came into the chats once a month for an open QnA / AMA (ask me anything) kind of thing to engage users in the chat rooms to hear things, so I will do that soon – I’ll post in the blog or peeps section a day and time if anyone wants to make suggestions they can know in advance I guess.
I have been listening to the feedback from people over the years, and try to modify things according to what I see and hear from people. Some things I can do quickly, other things that I want to make happen will take time.
If you’d like to reminisce a bit about the old chat, there is a post here: The Old Chat Post Mortem (comments about the OC)
If you’d like to read some of the details as reported by various news outlets, a private search at startpage.com will give you many different places that have published varying bits of information about this hack. -> https://startpage.com/do/search?q=staminus+hack+kkk&lui=english
This was a major hack event, and they stole a lot of people’s info. You can see why they brought in the fbi and other forensic investigators to look through the equipment there.
Let this be a lesson to us all. Make a backup of your precious pictures and documents, encrypt that stuff with 7zip or something. And don’t leave that cd / dvd /us drive plugged into your computer.
If you have a router that you connect through to get to the internet (or anyone you know does) make sure the factory default password has been changed on that router. Even if you can’t find the manual, do some searches for your router model, turn off UDP, and change the default password. While your in the router settings, see if there is a firmware update.
If there is not a firmware update, and your router is more than a year old, I would trash it.
I think it’s high time that everyone realizes our data is not safe on computers at all. If you search for news stories about healthcare hacks, the Verizon enterprise super secure client breach, the opm breach, the Jp Morgan hack, Facebook flaws – you will see that the top security companies can not protect the data that we have “at rest” – social network messages and pictures, your yahoo mail, any of this data that is sitting on connected devices – your phone – unless it is encrypted, it’s not safe. Even if it is encrypted, determined hackers may find a way to get the pass codes and get the info.