Open Carnage closes doors

[Accessing]

To many here its an unfamiliar name, but to few of you I mentioned another community using similar forum software that was dedicated to Halo and general expansion of knowledge. As forums grow old a few have withered away but Open Carnage stood to be one of the last Halo CE modding forums around. Chimera and Invader being two of them on the site.  

Takka, who was the site owner not here and probably done with forums explained the below in what happened to there forums. 

Quote

 

DDoS

The story begins out of the blue with repeated DDoS attacks in April of '22. Despite CloudFlare's mitigation, the attacks were always successful, and I had no way of keeping the site online. Due to the spike in traffic, my host also believed that the site was infected with malware, and suspended the site (took it offline). I was quite out of my depth and had no way of verifying my host's claim.

I asked a friend of mine (we'll call "Rick") for advice, whom as far as those online go was a pretty close friend, and had a ton of technical knowledge. He's the CTO of a tech firm in Canada, and simply offered to host the site in his company's datacenter for free. He was confident that we could then withstand the DDoS attacks due to their more robust security and resources. He said he'd take care of moving it over and that the site would be back up and running over night.

Unfortunately however, he made this estimate with the assumption that our forum was running with a modern version of PHP. Our software version dates back to 2016, because any time you update, it requires about an intense week of work at minimum. Our unique plugins (like the raffle) break, and the styling of the forum (theme) breaks. Considering OC was on a downturn as forums have been going out of style, I didn't feel that revamping the entire site at such an expenditure was worth it. Everything worked fine. Hosts however, including Rick's company, are not very interested in running old versions of PHP. So, his plan was just to force it to run in a more modern environment, and fix PHP issues that arise. He expected that this would take up to a week, and had a couple of his company's engineers tackle it off the books when they had time.

 

Upgrades

The site then came back up on a more modern version of PHP, however several non-critical functionalities of the forum were broken. You may have noticed that changing avatars and otherwise uploading files to the site only worked sometimes. So now the long-term plan was to just upgrade the forum's software, so that it could comfortably run a modern version of PHP. Rick said that he would bankroll the development for determining the upgrade path, because everyday content can get corrupted with any IPS software update, and this was a very big jump in software versions. Plus he said he would pay for remaking our plugins and the development of a new theme. I said that I could handle making the new theme, but he insisted that his team would be much better at it than anything I could do, so that became the plan.

Sounds too good to be true, yeah? Well this guy once PayPal'd me $1.6k on a whim so that I could buy him a new phone that wasn't yet released in Canada, and at no cost mailed over some spare parts to upgrade my wife's computer (he was tired of her loading time when we all gamed together). I knew him for a while and had the impression he was good for it. He did ask though if I could give an hour-long Zoom presentation on the subject of wireless technology to about a dozen of his engineers, and I figured this was the least I could do, so I did. For those that don't know, that subject is my profession.

Now while the team was supposedly working on the upgrade plans for OC, I wasn't given any sort of back end access to anything. Not the database or even SFTP. So I was very helpless any time someone asked me why something wasn't working, and even worse, I couldn't give Kavawuvi any sort of access to her Chimera or Invader subdomains, which were necessary for her to easily provide software updates. A short-term inconvenience was the thought... Rick had said at this stage that the team would only need about a month to complete their work.

 

Stagnation

A month turned into another month, and another month, and another month. Rick over time became less responsive, and I was getting more and more frustrated. Eventually he had some account manager lady reach out to me to get in writing everything that I wanted done for the site. Then she provided a quote to Rick (I was CC'd) for numerous thousands of dollars with a timeline of completion at around 2 months out. He then approved the quote in the email thread. Apparently he wasn't bankrolling the project prior, and it was still some side project for his team. I feel a new little bit of relief that progress might finally happen, though that quickly fades after nothing seems to happen for the next few months months.

At this stage, any time I'd bring up any sort of frustration with the pace of things, lack of updates, or no evidence of actual work being done, Rick would ghost me for an additional month or two. I tried calling his cell a couple times, emailing, and many Discord messages. After being ghosted like this a second time I wrote him a scathing Discord message, and ended it with threatening to do my best to embarrass him at his company by bringing my frustrations to his CEO (which Rick had me converse with before, and so I had his phone and email). I was going to make a fuss to anyone that I could get a hold of there, about how I'd made a well-documented contribution to their company and received no compensation in return. I wasn't expecting to get any compensation or anything like that as a result of this approach, but if Rick fucked me, I wanted to at least make him regret giving me the ability to make a scene in front of those important to him.

Rick decided to respond for the first time in I think more than 2 months after this threat, and said "I'll respond to you tomorrow". The next day he asked if I would be willing to meet with a couple of his engineers so that they could give me an update. I said yes. They then emailed me to set up a Zoom, and when I spoke with them they were apologizing profusely that my project had "slipped through the cracks", and asked what they could do to "make it right" with me. I asked that they just get it done, and keep me in the loop on a regular basis. They said that would be no problem, and they did keep me in the loop on a somewhat regular basis, about once every couple weeks. Around the very last moment the site was still living (July of '23), they claimed to have finished planning the upgrade path on a development copy of the site, and had completed a first draft of the new theme. We were about to do another Zoom meeting to review it.

 

Outage

When the site went totally offline this July, the company had posted a bulletin on their site to inform that all of their hosted services are down as the result of a cyber incident in their datacenter, and that they were working with a forensics team to solve the problem and restore services. About 3 weeks later, their website reported that services have been restored for all customers. OC was however still down, so I sent an email to the team that was working on our site. I'll let the email exchange do the talking for this: Screencap

 

Stuck

I tried not to share most of the details that make this situation so screwed, because it rightfully casts Rick in a bad light. Rick is a member of our Discord server, so I didn't want him to read what I say or any following discussion from everyone, and say "fuck it" and delete everything. I was totally at his whim on this; I had no actual hosting agreement with his company, and I had no access to pretty much anything related to the site.

Why couldn't I tell them to stop bothering and give me back the site so that I can host it elsewhere? Well, they hacked the shit out of it to make it work on their hosting environment, and database was ported to some odd format because they don't run mySQL. Getting OC to work with another host would definitely require that I hire someone to do the porting, which would likely be quite an involved process without documentation. So in the meantime, I couldn't totally burn what bridge was left.

I decided to stay the course until they determined whether they could recover the corrupted data, and until they properly revamped the site to work with modern PHP. I then intended to move the site to some other hosting. However I also decided that if they failed to recover the lost data, that I would take the site live with the copy of OC I had from right before this whole mess, which was only slightly older than the backup they apparently had. A couple extra lost months when more than a year was already lost seemed like a small sacrifice to get OC back from this nightmare, in a state where it can easily be moved to a number of other hosting options if needed. As you read in the email exchange, they weren't able to recover the data, and so here we are back in April of '22.

In Closing

I made a big mistake in trusting Rick, but I really don't know that there was much of any other choice. A lot of you wouldn't know his name because he never had a presence on the site, and barely messaged in Discord, but I'd known this guy on and off for longer than OC's been around. He was someone that I thought could be trusted with something which at first seemed so simple. But now we're here, and I'm truly sorry that this impacted so many of you. Sure, livelihoods weren't involved with any of this, but I understand that this has had a significant impact on the Halo modding community at large.

Because of the more than year and a half of lost data, and decline of forums in general, I'm not making an attempt to reinvigorate the site. I fully intend to maintain it as a resource best as I'm able, but in a read-only state.

Thank you very much to all of our community members over the past 11 years. It was one helluva trip. And hey, we were around longer than ModHalo, and were even the last active Halo CE forum! I don't think anyone would've bet on either of those odds.

 

I reached out to Takka about working together in the future but no real intentions or strong interest from him, this will likely be a good case study as to how cloud hosting and storage buckets need to be secured and contracts underwriting need to be made to avoid the kind of problem presented. 

For us I doubt this could happened, we are hosted on a AWS cloud instance and set up with redundancy with the US/EU, the site will load slow for those outside of the US/EU but only by milliseconds and not really noticeable unless someone is on dial-up, at speeds 10mb or less. 

[ADM D Kilkin]

I'm going to presume negotiations didn't go as planned and he or anyone from that community will probably not join us here even though we share so much in common. 

[Accessing]

On 10/12/2023 at 3:29 PM, ADM D Kilkin said:

I'm going to presume negotiations didn't go as planned and he or anyone from that community will probably not join us here even though we share so much in common. 

Yeah he didn't shoo me away or anything but wasn't interesting in anything. I can't blame him though I would be traumatized after what happened. 

If something similar happened to me and this website you would only be able to see it on the good ole web.archive.org database. In realness I just would not work on anyone I could not physically meet or see.