Taking the Wordpress Plunge

I've been running on blogger for awhile without a real domain name. A lot of people I know are running wordpress and seem to like it. I set up a hosted wordpress for Alexa a while back. It definitely looks nicer than blogger and has a lot more features. It's a little easier to add plugins, too.

It's not worth switching unless I get a new domain name. So, I'm now proud to present my new blog: The Cult of Gary.

I've imported all of my blogger articles. There's a new Feedburner feed too, so make sure you update your clients if you're following me! I'll leave the blogger site up for a bit, but I will be shutting it down.

I decided to host my own instead of the hosted option. I wanted to be able to have cookies (for google analytics) and patch any bugs. In particular, there's a nasty Safari one.

I have an EC2 host up that's not doing too much, so my blog is running on an EC2!

facter -- where to find puppet variables

I'm really loving puppet. I was able to build an application server from scratch in about 20 minutes. 19 of them were watching progress bars and syslog output. I can magically reconfigure the systems as I go and any work on do for one system can be applied to all of them.

I was hunting around for what variables are available when writing puppet configuration. I found a page that points out puppet uses something called 'facter'. I think facter is packaged with puppet. It's on my systems anyway.

If you call 'facter' you get some nice output about your system. All of those variables are available in your puppet configs.

Timecapsule: every Mac house should have one

About two weeks ago, Alexa's parents were ready to take the Mac plunge. Of course, I offered my help. We decided the 20" iMac was the right fit for them. My brothers in law that are still at home both now have laptops, one of them is a MacBook.

They have had a string of hard drive failures and have almost lost their pictures a couple of times, so I suggested they get a USB hard drive for Time Machine backups. 

They also wanted to put the computer in a more central location, but didn't want all the clutter of having the printer hooked up to it. I suggested that they should also get a wireless print server so they could hide the printer. Their Linksys Wireless-B router was getting a bit dated as well, and I like WRT54GL's, so we were going to get one of those. 

All said and done, we were looking at about $350 in accessories. I suggested that we waited until after Steve Jobs' keynote to order. I figured there may be some iMac upgrades coming. Now I look like a genius because they announced Timecapsule. It's cheaper and less of a pain in the butt than what I was looking to set up for them.

We ordered a Timecapsule and an iMac last night. There's no ETA on Timecapsule yet. I can't wait till it arrives so that I can see how well it works. If it's as good as it seems, I think every multi-mac household should have one. 

Of course, Timecapsule backups don't stop you from losing everything in a fire or theft. I think we're going to set up a smugmug account to get all their pictures 'off site' as well.

Puppet just made me about 3000% more efficient

I've spent the afternoon playing with puppet. It's brilliant and I wish I would have set it up sooner.

My test setup does the following:

• creates a file called /tmp/hello_world
• makes sure that screen and vim-enhanced
• makes sure ntp is installed, running and configured to start (using the default config)
• makes sure the perforce cli is installed
• make sure snmp is configured the way I like it
• /etc/resolv.conf is configured as I expect it

I can create classes that map to my different server rolls. I can make sure key configuration files are up to date -- if I had to change nameservers in /etc/resolv.conf due to IP changes, I'd have to log into every one of my boxes and manually change it. With puppet, it's a variable change and all my systems update.

The following configures your snmp server for you. Stick the following into a pp file:

define snmp_server($community = "public",
$location = "Unknown (edit /etc/snmp/snmpd.conf)",
$contact = "Root (configure /etc/snmp/snmp.local.conf)"
) {

file { "/etc/snmp/snmpd.conf":
ensure => present,
mode => "0644",
owner => "root",
group => "root",
content => template("snmpd.conf.erb"),
}

include snmp_apps
}

class snmp_apps {
package { net-snmp: ensure => installed, }
package { net-snmp-devel: ensure => installed, }
package { net-snmp-utils: ensure => installed, }

service { snmpd:
ensure => true,
enable => true,
subscribe => File["/etc/snmp/snmpd.conf"]
}
}

And this into a snmpd.conf.erb template file in your template directory:

com2sec notConfigUser default
group notConfigGroup v1 notConfigUser
group notConfigGroup v2c notConfigUser
view systemview included .1 <%= community %>
access notConfigGroup "" any noauth exact systemview none none
syslocation <%= location %>
syscontact <%= contact %>

Then configure your systems with:

snmp_server { "not used" : community => 'your_commnity', location => 'Springfield', contact => 'root@apple.com' }


Keep in mind that my configs are for Redhat/CentOS systems only -- but you could easily make these cross-unix compatible.

MySQL Meetup Last Night

It's a strange world. Last night, I attended my first Vancouver MySQL meetup. This morning, my google reader is filled with news that Sun is buying MySQL.

We met at the Milestone's on Robson. At the peak of the evening, there were 8 people out. I think the goal of the meetup was to talk about clustering, but we never got around to it. People were mostly introducing themselves, talking about their backgrounds and what they were working on. 

I suppose that's the problem when you do something like this at a restaurant. This was my second meetup, the first was the Ruby meetup on in December. I think that if you want to have a specific technical discussion you should start with a presentation in a meeting room, then move elsewhere for food. 

That being said, I was happy with how the meetup went, and plan to attend the February meeting. It's a good group with a wide range of experience.

So this is it

Friday was my last day at the old job. It was a very bitter-sweet day. Eight years, two months is a long time to be anywhere in the IT world, especially since it has been my only real day job.

The weirdest part was handing in my key, swipe card and garage door clicker. My keychain is very light now. 

I've been working in that space for 5 years. Where will I park down town? Will I ever see my servers again? What do I do with my guitar?

The IT team gave me a touching card and $100 give certificate. I'm going to miss working with those guys the most. Solving problems with them always gave me endorphin rushes. It will be strange not having a dozen other people within 30 feet to discuss ideas with.

We headed to a pub after work. I was surprised when 25-30 people showed up. A bunch of ex-employees from throughout the ages came out. 

I officially start the new gig Monday. I'll have a few infrastructure things to work on (email, backups, etc). That will be easier than doing it at the old gig, considering there is only a handful of people to deal with it. Then, I think I'm going to be working on build automation and architecture design. 

My plan is to experiment with Puppet and cfengine. I've been making complex kickstart files for a long time. The problem with those is that you only get one shot -- if you change your NTP servers, add an SNMP community or any other global config change you need to log into every system and update them. From what I understand, using puppet or cfengine, this is all automated for you.

It's going to be cool..

My Last EC2 Availability Post

Tomorrow is my last day at the current gig, so this will be my last EC2 uptime report. I won't have access to these graphs/machines any longer.

So far, 100% availability of this EC2 dating back to November 27.

Unfortunately, there have been two other incidences with different EC2 hosted machines in the past couple of weeks. One machine rebooted itself. I'm not sure if it crashed or if there was another issue. It came back up fine, other than a few services that weren't configured to start at boot (that's mostly how we knew it rebooted).

Another machine disappeared for about an hour. It wasn't reachable at all -- ping, ssh, etc. When it came back to life, everything was fine. No change in uptime, so it didn't reboot. It was clustered with another EC2, which was fine during the outage, though it also couldn't access the down machine.

We're still in development, so at any one time, we're currently running 6-8 EC2's. With that rate of failure, be sure to back up your data and keep tabs on them with some sort of remote monitoring.

Where Do I Put My Crontabs?

There's another good discussion forming around where to put your crontabs on the SAGE mailing list.

On linux based systems, there are typically 4 places to stick your crontabs:

1. Where ever crontab -e puts them on your system (per user crons)
2. /etc/crontab
3. /etc/cron.daily, /etc/cron.hourly, /etc/cron.weekly and /etc/cron.monthly
4. /etc/cron.d

My opinion is that crons should go into /etc/cron.d only. They should ideally be grouped into one file per application, or some other logical breakout (database maintenance, health checks, email crons, etc).

Putting them into /etc/cron.d makes it easy to revision control the files and keep track of changes. It's also handy for package management -- you can build a custom RPM or DEB of your app and have it drop any required crontabs into /etc/cron.d.

Crons should never run as human user account. When they are configured this way, they tend to break if someone leaves or an account is shut down for other reasons. Depending on what the cron needs access to, it should run as nobody, apache, and in special cases, root. It's easy to configure which user a cron runs as in these files since each entry in the file requires you to specify which user to run as.

Of course, if your cron doesn't support /etc/cron.d, ignore everything I said :)

LongJump DaaS

LongJump's Database as a Service was featured on TechCrunch today. It appears to be a competitor to SimpleDB:

LongJump’s DaaS provides a complete environment for hosting a database that’s highly secure and web-enabled while simultaneously cutting the costs and easing the hassles for entrepreneurs and developers who would otherwise have to purchase a database server, provision it, address data access and availability issues, manage backup and replication issues, and tackle security and data protection.

They mention that they use Java on MySQL, but it does not appear that they actually expose MySQL API's. According to their sales copy, it's only REST 

My quick scan of their product pages indicate that you need to sign up for LongJump to get access. They charge by the user account as opposed to usage. I don't know if it counts as utility computing then :)

I'll sign up for an account in the next couple of weeks, but on first glance this seems to be targeted at smaller businesses and databases.

Network Monitoring: Auto Discovery vs. Configuration Management

There is some discussion on the SAGE-members list regarding network monitoring solutions. The first post was about Zenoss vs. Nagios. I haven't been involved in the conversation, I've only been lurking. 

I have maintained a few Nagios installs, usually configured via fruity. Before my Nagios/Cacti days, I ran a couple of large mon installs. 

I've been unsuccessful and unsatisfied with Zenoss in the past. I felt it wasn't monitoring what's important to me (mostly raid arrays). I couldn't figure out how to add custom monitoring in. I also had a brief stint with Hyperic, but the free version didn't give me access to the features I wanted.

An interesting comment was added today. Neil Watson said:

I am planning to move to a system that I can configure with my configuration management system instead of auto discovery (probably Nagios as I have used it before).

I've long thought that the division between people who want to use auto discovery and manual configuration has to do with the number of systems you need to maintain. I've been on the manual config side as I've never had more than 100 systems to monitor. I typically write macros to quickly add systems of similar types.

When I was at Scale5x last spring, the OpenNMS people were talking about how many tens of thousands of systems they could monitor with one station and how quickly they could get it up and running. I thought it was cool, but it didn't matter all that much to me.

To me, it's more important to use source control systems to maintain configuration management than using auto discovery. I also want to specify what I monitor -- I'll hand pick what's important for my environment. I also want to know how and why it changes.

I can think of two things that would make me switch camps:

1. I'm in a position where I monitor 200+ machines
2. A monitoring system lets your write customer monitors in perl/ruby/python and the interface includes the ability to add auto discovery code to your monitor

SuperBlade Servers

I came across a press release for a new quad core SuperMicro SuperBlade today. I'm not currently in the market for one of these, but I was shopping around for blade servers about a year ago.

I wonder how long Supermicro will support this platform? I also wonder if they offer support contracts and SLA's for parts. You're putting a lot of eggs into one basket with these chassis'.

Back then, I found that the market was pretty bleak -- the products from Dell and IBM were silly. You might as well buy 1U servers -- they were more powerful and the density was around 1.1 servers/u (of course, you get management and power savings). The IBM machines didn't have hot swap drive bays. The Dell blades were 7U's tall.

HP's c-Class chassis was the exception back then. It was well thought out and had a good density. Since then, they've even come out with a half sized chassis. Their blades were their 1U and 2U servers crammed into a blade and included hot swap drive bays. You could even install a storage module for additional drive attachments.

When I was doing the price comparision, the break even point seemed to be around a half filled chassis -- if you were buying 8 blades, it was cheaper than the equivalent 8 stand alone machines. That included networking modules in the chassis, but no networking for the stand alone machines.

Last year, when I was meeting with the HP reps, they said the c-class chassis was their blade platform for years to come (I'm thinking at least 5 years).

I've been using Supermicro servers for ages. They were always cheaper than the big names and they ran reliably. Recently, I 'grew up' to using HP and Dell servers. I've learned that there are some advantages to buying name brand servers instead. They tend to work better as a unit -- they are well thought out and easy to service.

The biggest issue I ran into with Supermicro was getting replacement parts (like power supplies) for 2 year old Supermicro machines. We've had to throw out a few perfectly fine machines due to lack of parts. With a support contract on a Dell or HP machine, you'll get the part you need.

I won't be in the market for a blade server for a while now, but t least there is another product on the market for comparison when it is time.

List-Unsubscribe

Since I'm leaving my current gig, I'm starting to unsubscribe from all the various mailing lists I'm a part of. This will should make the person who inherits my email address life easier.

I decided awhile ago to only pick up RSS feeds in Google Reader and to no longer subscribe to email lists. Email is for personal communication only. There are a few exceptions (like SAGE). Otherwise, the stuff I'm still subscribed to is left over from before I started using Google Reader.

Most of the mailing lists I subscribe to have List-Unsubscribe headers. That means I only have to view source on mailing list messages to find out how to unsubscribe.

I wish more mail clients supported using these headers. Hotmail sort of has it (but I never really got it working as a sender, so I don't trust it. I also don't use hotmail). You'd think it would be relatively easy to add to Gmail or Mail.app. I'm not the only one to consider it, either.

Google even recommends adding the header:

Provide a 'List-Unsubscribe' header which points to a web form where the user can unsubscribe easily from future mailings (Note: This is not a substitute method for unsubscribing).

I guess I'll be copying and pasting for a bit.