More Uptime Isn't Better

On Friday Night, my company's building did power maintenance. BC Hydro needs to test something (I'm not exactly sure what), but our building needs to shut down all the power for it to happen. They do this testing once a year at least, and sometimes twice.

We have lots of old hardware that is slowly being rotated out. Thanks to UPS's, some of these systems have years of uptime and the hardware is getting old too. the replacement of the old machines is clearly not happening fast enough; whenever they perform these tests, we tend to have a string of hardware issues afterwards. These are almost always hard drives failures.

We have the building supply us with generators to power our server room so we don't need to shut things off, but not all the servers are in there. Some of the development and test machines are seeded around the office.

Today, we've been dealing with a string of dead or dying hard disks. One of our development database servers is having sector errors and we had to replace a drive on it. Some other machines are in the same boat.

Due to a UPS boo-boo, we had a bunch of servers in a rack power off too. Our Scalix machine also decided that one of the drives shouldn't work after the sudden loss of power.

In our 40-50 year old building, these things are inevitable. They just reinforce the fact that every server machine we buy needs to:

• have hot swap hard drives
• use some level of RAID (other than 0, of course)
• have a list somewhere of what needs to be on the server and what should be tested in an unexpected rebooted (on top of a monitoring system)
• Replace old hardware frequently
  

We're getting close on the first 2, but one of the machines isn't hot swappable and needs to be powered down and taken off line to replace the drive. Another system doesn't have redundant drives at all.

The third point is really helpful for the new guys. I've been here a long time and know everything really well. New people don't, and having a page that lists all the key services on a box allows people to help themselves. I definitely need to push to get more of our systems clearly documented.

In my experience, hardware should be replaced in the three year mark of service. Typically, at this point, it's next to impossible to get replacement parts and it's probably cheaper to replace it with a new low end machine anyways. This also fits nicely with support and lease deals, typically.

OCFS2 Looks Interesting

There was a discussion on the CentOS mailing list the other day about Oracle Cluster File System 2. I went and checked it out. Well, I read some docs about it anyway.

It's released under the GPL and it's not really just for Oracle. It looks to be a share nothing type file system layer. Oracle uses it for RAC, I believe. Of course RedHat doesn't ship it -- it would eat into their GFS sales. Oracle does distribute RPM's for RedHat/CentOS though.

I've played around with DRDB for replication/fail over before. It works pretty well, but you definitely feel blind on your secondary machine since you don't have the shared partitions mounted. With MySQL starting to push DRDB for MySQL failover, I wonder if this would work better. You'd still want to use heartbeat to control where MySQL is running, but at least you'd be able to see everything on each node.

It also may be a decent alternative to replication if you're just using MyISAM. This may be good for scaling out data warehouses -- having only one master updating files and many slaves reading those files.

Another interesting use would be for apache document roots. It gets around having to sync each web server node's document root (if you take a share nothing approach), or using an NFS server (if you use NFS).

I haven't really looked into how well OCFS2 handles node additions (I'm guess you have to restart the cluster). Nor what happens when a node gets out of sync.

Sooner or later I'll install and test it out.

MacBook Pro is Crashing

My new MacBook Pro has crashed 5 times now. That's a lot for a brand new laptop in 2007.

It seems to crash when I'm using a VPN connection over WiFi. It seems to be a common problem -- there is a big thread about it in the apple form. Most people say they experience the crash on WPA2 WiFi connections, which is basically what I'm seeing. In particular, the places it crashes in I'm connecting to WRT54G router running OpenWRT.

I'm hoping there is a driver upgrade or something released in the next little bit. If there isn't one by next week, I'll probably give AppleCare a test drive. Some people are telling them to do things like create new users and not run a bunch of software. Seems like a giant hassle to me for something that seems to quite obviously be a driver issue.

It's about time..

It's about time I put one of these together. I've got a few things to post and ideas to publish and this seems like the best place to do it.

I'm starting with blogger cuz it works with my gmail account and I don't have to think about it. I'll have to make sure I can post attachments and stuff so that I can upload scripts and patches.

Since I just got a new MacBook Pro, I figure this can also serve as a log for of what I've had to do to make it work. For instance, I'm using an Intellimouse Explorer for Bluetooth Mouse (it's made it through about 3 laptops now). The side buttons weren't working. I had to install Microsoft IntelliPoint & IntelliType - 6.2 to make it work. I've since configured my trusty forward and back buttons and I can now also get to Expose with a scroll wheel click.

That's enough for now..