"We're currently experiencing a higher than normal call volumes"

I've noticed that it's really popular to say "we're currently experiencing a higher than normal call volume" before a company drops you on hold.

I've called into many different tech support lines for various reasons and I seem to get this more often than not. Recent offenders include Rogers, Apple and CIBC.

What's the deal? When I call these companies more than once at different times in the day over several days, I get the impression that they don't have enough staff and they are just trying to placate me. They're trying to make it seem like it's my fault that me and my user brethren all broken our phones at the same time.

I would feel much better if they said "there are currently 100 people on hold and 25 reps". At least then I'd know that there is more than 2 reps handling the call volume.

I got my Sysadmin Magazine CD Today

When Sysadmin Magazine closed their doors, they decided to send out their complete publication on CD to all the subscribers.

Mine arrived today. It's pretty neat. The first issue was May/June 1992. I was 11. Some of these ancient articles are awesome looking. A Community-Style Overnight Job Spooler starts off with:

For a small business running a single multi-user UNIX system, processes typically fall into one of two categories: real-time, interactive programs or batch style/background jobs. Interactive programs such as the system shell, editors/word processors, spreadsheets, and data entry systems all vie concurrently for slices of the CPU pie.

Wow. I guess they didn't use screen and vim back then :) You wouldn't want to overwhelm the CPU with a s///.

When I was old enough to get my drivers license (1997), and article entitled New Storage Interfaces states:

Interesting things are occurring with storage technology including the cost per megabyte (MB) dropping, faster drives, and higher bandwidths.

Hah, cost per megabyte.

I wonder what other gems I'll find in here? I've copied it to my hard drive (thanks to the dropping cost per megabyte), so I imagine spotlight will start returning insights into the past.

I shouldn't poke too much fun. I remember when we ran our mutli million dollar website on a 400mhz server with three 36GB SCSI drives and 512MB of memory. We were awesome because it was a $4,000 PC and not a $12,000 Sun box. It was a 4U rackmount chassis that was basically a tower flipped on its side.

EC2 Reliability

I'm not so worried about it myself, but I've integrated some EC2 monitoring into my Nagios system. I'm going to keep an eye on it for a while. I'd like to see how reliable it is. In the few months that I've been running EC2's, I haven't had one crash (but the most uptime I've seen is about 1.5 months) and I haven't had connectivity issues.

So far, I've been monitoring it for about 17 hours with no issues. That's a ping every 5 minutes from a GT data center in Vancouver to wherever the Amazon cloud really lives. I'll try to do a weekly update, or at least monthly.

New Futurama Tomorrow!

This is exciting. The new Futurama movie comes out tomorrow. That's right -- Futurama is back from the dead. With any luck, this will lead to full seasons of the show.

I hope I can hold off till xmas.

More Thought on Confluence Calendars

I've been thinking about my problem with confluence calendars from yesterday.

A cool way to implement this would be to use labels. If you wanted a page to show up in a calendar, you would label it with YYYYMMDD.

When the calendar rendered, it would search for all pages tagged with the label for each day and put a link to that page in the box.

Part of the per calendar configuration would be specifying which spaces and pages to look in. For example:


{garyCalendar:spaces:space1,space2|pages:space1.page1,space1.page2}


Child pages of specified pages would be search.

Now if I was competent in java, I could probably bang this out in a night :(

Confluence vs TWiki

When I first used confluence, I thought it was slow and obtrusive. I thought it had no where near the feature set of TWiki. I didn't know how to do a lot of the things I do in TWiki and WYSIWYG is never as powerful as markup (at least for how I use it).

Then getting non-techie people to use the wiki became my problem. It turns out that TWiki works best for sysadmins and programmers and not so much for artists, writers and business people.

I've figured out how to do a few things with Confluence and I'm really happy with it now. So far:

• I can create service journal pages for my servers
• I can use gliffy to draw diagrams
• I can attach xls files and edit them using Webdav

I still haven't tried to set up knowledge bases yet, but I don't see that being a huge problem. One of the nice things is that confluence has a concept of templates, which makes this easier than it was with TWiki.

The one problem I haven't solved is calendars. The calendar plugin really sucks compared to twiki. In Twiki, I can put a date onto a page and the calendar plugin automatically puts it into the right square for me. As far as I can tell, the confluence calendar plugin is only for ICS and only for well formated calendar events. Being able to map events to a visual calendar automatically is a very powerful tool.

I hope I'm wrong and that someone else has already solved my problem.

Using Checklists

Niclas at Aspiring Sysadmin has an article about using checklists. It's solid advice. I typically call them procedures instead of checklists, but it amounts to the same thing.

I'd just like to add that Wiki's are the perfect tool for creating procedures and checklists.

First, I find it a lot easier to search and find things on a wiki than it is on a file server or my home directory. This is important because chances are you'll be doing it more than once. I've had to migrate data centers 4 times in 8 years and I've been able to recycle my procedures and documentation a few times now.

I've blogged before about creating a page per server. I'll include a procedure common to a server right on it's page. I also like to build 'knowledge bases' (I'll have to blog on this later) right into the wiki. Often times, my server maintenance entries include links to the central procedure, along with any places I had to deviate.

Wiki's typically have formating options for blocks of code, like shell scripts. That's handy for making them readable and copy/paste-able. Word processors typically don't work well for this as they like to mangle special characters. Text documents don't manage your ordered lists and aren't as readable for large procedures. Most wikis also format well for printing -- hard copies are important when you're network is down.

Of course, you can easily update a procedure on a wiki. You can add notes and change steps while you're working through the list if you find inaccurate information. It keeps track of the changes you've made, in case you want to go back to an old one, or at least see what it was.

So far so good, post Archive and Install

It's been almost a week now since I did the Archive and Install, and my laptop is far more stable. I haven't had any crashes and my VPN works fine.

I have noticed that the dock animation is now choppy. I also had to re-install the Cisco VPN client and Parallels, but those are minor issues.

All and all, if you haven't upgraded yet, I recommend doing an Archive and Install. Puts all of your data and apps back into place and you won't have as many issues.

Al Gore helping you to save paper

This is a pet peeve of mine. I go through extra effort to make sure all the printers we buy support double sided printing. When I see people not double siding, I cry a little for the environment.

This morning, somebody had printed out about 200 pages of powerpoint presentations. That was the last straw, so I created this bumper sticker to stick onto printers:

Feel free to duplicate this on your own. It fits nicely onto the top of HP LJ 4100's.

The image of Al Gore came from Yahoo Image search (inside gliffy).

Gmail breaks the 5GB barrier

I just noticed this. I'm using 1248MB of a possible 5061. That's 24%. I really should keep track of how much mail I have -- I think I'm keeping pace or exceeding Google's.

EC2 Or Not?

I have a secret technique for figuring out if a site is hosted on EC2 or not -- I do a PTR lookup on the IP address of the site and see if it's at Amazon. I wrote a perl script to speed the process up yesterday. I showed it to my colleagues, and one of them jokingly suggested I register ec2ornot.com. It was available, so I registered it.

This morning I turned my perl script into a cgi script and EC2 Or Not was born. You can now easily check to see if a site is running on EC2 or not. Well, at least the front end servers.

Don't mind the awesome graphics and layout. I've never been visually oriented.

By the way, my server is also an EC2!

Velocity 2008

O'Reilly have announced a new conference called Velocity. From their front page:

Velocity is the new O'Reilly conference for people building at Internet scale, happening on June 23-24, 2008 at the San Francisco Airport Marriott in Burlingame, California.

Web companies, big and small, face many of the same challenges: sites must be faster, infrastructure needs to scale, and everything must be available to customers at all times, no matter what. Velocity is the place to obtain the crucial skills and knowledge to build successful web sites that are fast, scalable, resilient, and highly available.

This is exciting and close to my current interests with EC2 and other scaling methods. I can't wait to see a list of speakers/sessions.

I've been to several O'Reilly conferences now (MySQL UC 2005, OSCON 2006 and ETel). They are always very well done, and now I need to find someone to send me to this one :)

Just Archived and Installed

I've been having random crashes since I've upgraded. It's been ugly. Often, after I reboot, the system would crash over and over for 45 minutes to an hour. Right when I got through to Applecare, it would be working again.

When I called last week, they had me repair permissions. That didn't do anything. Today, they had me do an Archive and Install. So far, no crashes. We'll see from here.

SAGE-members email list is great

Of all the mailing lists I'm on, sage-members has the greatest solutions:noise ratio. It's basically 1:0. There are about 3 or 4 threads a week with a lot of really great advice.

For instance, this poster seems to have solved his problem. The other threads on the list are also interesting, possibly because they are so varied, as are the people posting responses. I participate in several other lists, like CentOS, Asterisk and MySQL. Those lists are typically noisy with the same questions answered with the same perspectives all the time (that's why FAQ's were born).

The email list alone is worth the SAGE member fees. Now they need an affiliate program so I can get paid for my awesome reviews :)

Google Maps has BC Transit Directions!

I just noticed this while doing some google mapping. No one believes me when I say it takes an hour to take transit to work and an 20 minutes to drive (I drive at 6:30AM, so it's faster than Google says)!

This is awesome because Translink's website is one of the worst I've used. It's not intuitive to find routes, buses, etc and it's slow.

Random Thought of the Day: Hours Developed vs Hours Used

I was talking with a project manager about deploying a beta version of some software today. I don't know if this is a well known fact in the software development world or not, so don't shoot me if it's not original.

In any case, I realized that at some level, the usability of software is related to the number of hours it was developed vs. the number of hours it's been used in production.

If a piece of software took 1000 man hours to build, you probably don't want to be using it until there have been 2000 man hours of use. It takes about that much time for the different use patterns to flush out the bugs that affect them. That's not to say that there aren't still lots of bugs -- it's that they live in less frequently used parts of the software.

I wonder what the hours developed:hours used is like for any given version of Windows at the point it's first service pack is released?

swaks for SMTP testing

I find the best way to test mail servers is to telnet to the port and send a message manually. I can connect to the mail server I want and add time stamps to subjects and other identifiers and headers. It can be time consuming, so sometimes I'll create a text file I can copy and paste with all my commands.

About a year ago I found swaks by John Jetmore (never met him and have no idea who he is). It's a really handy script, because:

• you can tell it what server to connect to, headers to add, who to send the message to and who it is from
• it dumps the output from the mail server to stdout
• the commands are in your bash history, so no copying and pasting