Server Upgrade Feedback

[gnif]

Hi All,

Over the last few days I have been working to identify and resolve a random database deadlock that has been causing some issues here. This issue should now be resolved, and as an added bonus the server throughput should have been improved considerably.

This thread is for people to provide feedback instead of polluting the Server Error Reports thread.

[xrunner]

Just got this error from "Show new replies to your posts" -

Please try again. If you come back to this error screen, report the error to an administrator.

[gnif]

Typical, been working for hours and just as I post this start having issues again

[madires]

Just got this error from "Show new replies to your posts" -

Please try again. If you come back to this error screen, report the error to an administrator.

Over here too.

[BradC]

Probably my fault. I've had one of those days where everything I touch turns to shit.

[tautech]

Hi All,

Over the last few days I have been working to identify and resolve a random database deadlock that has been causing some issues here. This issue should now be resolved, and as an added bonus the server throughput should have been improved considerably.

This thread is for people to provide feedback instead of polluting the Server Error Reports thread.

Seeing an improvement in speed certainly. 

[gnif]

Just got this error from "Show new replies to your posts" -

Please try again. If you come back to this error screen, report the error to an administrator.

This is fixed

[BradC]

Hi All,

Over the last few days I have been working to identify and resolve a random database deadlock that has been causing some issues here. This issue should now be resolved, and as an added bonus the server throughput should have been improved considerably.

This thread is for people to provide feedback instead of polluting the Server Error Reports thread.

Seeing an improvement in speed certainly. 

Yeah, that error message pops up so fast you'd think it was locally cached.




I was looking for a "fingers in ears waving and blowing a raspberry" emoji to indicate a "severe taking of the piss", but I can't seem to find one in the list, so take this as a "I thought it was funny" response.

Just got this error from "Show new replies to your posts" -

Please try again. If you come back to this error screen, report the error to an administrator.

This is fixed

Yes it is. Nice one!

[m k]

Maybe it's time to separate the forum database service.

When the error happened here I lost everything and live domain front, even a shell, would be better.
Panic text is always so...panic.

[gnif]

Maybe it's time to separate the forum database service.

When the error happened here I lost everything and live domain front, even a shell, would be better.
Panic text is always so...panic.

Sorry I do not follow. The database is separate from the forum.

[m k]

The text was something that database is not available.
Forum address had SMF database something and main site had a box with text, database something also.

So my thinking was that one service does them all.

[gnif]

The text was something that database is not available.
Forum address had SMF database something and main site had a box with text, database something also.

So my thinking was that one service does them all.

Ah, no, there is a full stack needed for any dynamic website application like this forum.

1. The HTTP server itself, provides static content like images, but also hands off dynamic requests to another service where required, which brings us to;
2. PHP, this service runs the application itself, in this case SMF, which is written in PHP, however it needs a very fast method of storing and retrieving data, as such we have specialised applications for that, called a database.
3. The Database, or MySQL in our case. This is a service dedicated to the fast storage and retrieval of mass amounts of data. It's highly optimised for this specific purpose and designed to be extremely fast and reliable.

If any one of these three layers breaks down, the forum will stop functioning correctly.
As we are running a clustered setup, there are actually a few extra intermediate layers to deal with failures or high loads.

This forum is one of the largest SMF forums on the internet with an extremely active user base. Keeping it running fast and responsive for so many concurrent users is no easy feat. It doesn't help that the SMF developers (and most forum developers) have little to no experience with forums of this size, as such, scaling it to work in a cluster can become quite tricky.

Edit: If you're talking about running a separate server for the website and a separate one for the forum, it makes little sense to do so from a technical point of view, and would substantially increase Dave's hosting costs.

[Ian.M]

Is there any chance of a more informative error page the next time one of the key components sh!ts the bed?  Maybe with a link to where we can get official status updates.  Even if all you tell us is that you are working on the problem, its still good to know that it isn't another Gorillaservers fire scale disaster!

[PlainName]

Yes, I even resorted to dipping into Twitter to check Dave's page, but he just rabbited on about stuff as if nothing was wrong. Just a note that the system was poorly but being seen to would have been nice - it kind of looked like he has no idea what goes on here!

[thm_w]

From what I remember gnif mentioning, he would receive a notification. If the site is not working at all, its probably a good sign that someone is already aware.
If its down for multiple days, then it might be worth checking for an update.

[gnif]

From what I remember gnif mentioning, he would receive a notification. If the site is not working at all, its probably a good sign that someone is already aware.
If its down for multiple days, then it might be worth checking for an update.

1) Correct, we have active monitoring of the server, if there is a fault that breaks the entire thing I know about it within a few minutes as it sets of alarms.

2) These crashes/outages have been very brief lasting only a few seconds, over the last few days lasting a bit longer as I have been making changes to diagnose the cause. Unless I catch the issue as it happens, I have very little information on what caused the issue, as such they have been intentionally prolonged to give me an opportunity to figure things out.

3) When such an outage occurs, my priority is diagnosis and fixing the issue. Finding a way to update the community outside of the website is not one of my duties, though I always try.

4) Yesterday here was a public holiday, I would not at all be surprised if Dave was away from everything giving priority to his family. His standpoint on this website has always been that it's not mission critical and if it crashes for a few hours or even a day it's not the end of the world. People can live without it for a bit until we can get things running again.

5) Often people want to know what the cause of the crash is, often we can not say due to secure operating procedure.

Rest assured that we have a very good backup strategy for the entire website, so even if the servers were blown up, we have full backups in a remote location that can be used to recover the servers. The backup strategy has been tested and proven to work perfectly as it was used after the GorillaServers (WebNX) fire to restore from after total loss of the servers that host this forum.