Downtime / restore / missing content

[administrator]

In short... a reasonably catastrophic hard drive failure led to about a day of downtime (18 or so hours). There are very good technical reasons that a single hard drive failure should not affect the site, and unfortunate reasons that in this instance it did.

The current site is a restore of my last complete personal backup of it all, made a few weeks ago, hence the age of the messages and missing interim messages.

Unfortunately, the service provider of the server has not been able to provide intermediate daily backups. If such prove available, I will insert that data as soon as it is available. However, I'm not particularly optimistic, so wanted to get things up and running as soon as possible with what we have.

Sorry for the inconvenience -

Jim

[dagies]

You're sorry?

Thank you for your work!!

[NDFAN]

Yes! Thank you for all your hard work!

We really get our money's worth!

[SiouxMD]

Yes! Thank you for all your hard work!

We really get our money's worth!

Speaking of money...who pays for this site?

Jim...do you accept donations?

[ScottM]

As much as I missed the site when it was down, thanks for the hard work on keeping it available to us, Jim.

[siouxnami]

As much as I missed the site when it was down, thanks for the hard work on keeping it available to us, Jim.

It was aweful, I actually worked yesterday. I feel so dirty...

[Kermit]

Jim,

I'd like to add my thanks for your efforts in maintaining this great site. I've run a server myself in the past and I know it can be a real pain. SiouxSports.com is important to me. Add me to the list of those who would be most pleased to contribute financially to this site.

[jimdahl]

Now that I've gotten a few hours sleep, let me give a little additional info for those interested.

The majority of web sites use something called "shared hosting", where you provide content or pay your provider to develop content and the provider hosts that content on a big web server that's actually shared with other sites. The owner of the server keeps up the all the software, you just manage your own content. However, sites that have a lot traffic and custom programming, like this one, use too many resources so don't play well with others so we quit using shared hosting quite a few years ago.

In between shared hosting and having your own staffed datacenter is paying a big data center to keep a server in a rack (or provide a "virtual" server in a larger server) and supply it with power and network connectivity. You may just pay them to make sure its up, or you can pay them to help maintain the operating system and server software. As various faults have happened in the past, I've taken over more of those tasks myself, so now I'm managing pretty much everything (the OS, server software) because I've felt like they weren't providing the level of service I wanted. (Economists might have something to say about comparative vs. absolute advantage at this point, bluntly meaning my time is probably better spent managing the content, regardless).

The one real aspect I haven't been able to take over is daily backups. Dumping the entire database from a remote server in a data center to a machine of mine would involve moving hundreds of megabytes over the Internet to my home. Doing that with high frequency isn't particularly practical, rather you want to perform high frequency backups on a device close to the server. Needless to say, the one part I don't directly control proved the weak link that let us down in this failure. That's why I find it so frustrating.

[redwing77]

There is good in everything, Jim.

For example, the "F#$% the NCAA" thread is gone now. Small bonus, but a bonus nonetheless.

[AZSIOUX]

thanks again for the site and work jim!! i did not know what to do yesterday, then i thought about it and thought, oh yeah i guess i should work

set up a paypal if need be for donations to help you run the site....

[JESUS,family,rutgers]

Have a Blessed Thanksgiving Jim.

Thank you for Sioux Sports.

[HKNTSMN]

Thanks Jim for all your hard work DARN COMPUTERS>...

I just noticed the header information on the problem. I think you do a great job.

I like to keep up with the Sioux. Thanks

JCN

[ND Pride]

This is a great site and we appreciate your work.

[Sioux-cia]

Now that I've gotten a few hours sleep, let me give a little additional info for those interested.

The majority of web sites use something called "shared hosting", where you provide content or pay your provider to develop content and the provider hosts that content on a big web server that's actually shared with other sites. The owner of the server keeps up the all the software, you just manage your own content. However, sites that have a lot traffic and custom programming, like this one, use too many resources so don't play well with others so we quit using shared hosting quite a few years ago.

In between shared hosting and having your own staffed datacenter is paying a big data center to keep a server in a rack (or provide a "virtual" server in a larger server) and supply it with power and network connectivity. You may just pay them to make sure its up, or you can pay them to help maintain the operating system and server software. As various faults have happened in the past, I've taken over more of those tasks myself, so now I'm managing pretty much everything (the OS, server software) because I've felt like they weren't providing the level of service I wanted. (Economists might have something to say about comparative vs. absolute advantage at this point, bluntly meaning my time is probably better spent managing the content, regardless).

The one real aspect I haven't been able to take over is daily backups. Dumping the entire database from a remote server in a data center to a machine of mine would involve moving hundreds of megabytes over the Internet to my home. Doing that with high frequency isn't particularly practical, rather you want to perform high frequency backups on a device close to the server. Needless to say, the one part I don't directly control proved the weak link that let us down in this failure. That's why I find it so frustrating.

Hmm, ok? Thanks for the service. I too didn't realize how much SiouxSports.com is part of my day. Went through a little withdrawal there for a bit. OK, now. Thanks again. Some of the members made a good suggestion, after not getting my SiouxSports fix yesterday, I too am ready to make a donation to this site.

[sprig]

Was the live bookmark functionality lost in the disk failure?

[Taz Boy]

I must say that I'm a bit disappointed to find that my recent post- which analyzed in great detail the statistics and intangibles of the Sioux hockey season thus far, and then extrapolated with a high degree of accuracy, the outcomes and particulars of upcoming contests- was lost in this unfortunate event. I will look to see if I saved a local copy on my machine...

Nope.

taz

[jimdahl]

Update for the techies (others may want to stop reading now):

I just finished setting up a second database which contains a clone of the SiouxSports.com database. The secondary one (hosted on a different machine in a different site) maintains a continuous connection to SiouxSports.com, replicating real-time changes (e.g., your posts!).

If it works well, this means that even if SiouxSports.com died completely, I should be able to retrieve almost perfectly current data from the secondary server. I'll have to monitor the performance of this setup over the next few weeks to determine its long-term viability. Though the preceding step will protect against data loss, I was also somewhat displeased with the length of the downtime. I'm next going to look into ways to provide additional redundancy for the web site itself.