Service testing and alerts with Twitter

October 27, 2007

Have you ever messed with those site alert services that ping your site and email you if the site is down? For free services, they can be pretty handy. We used Montastic, made by the same guys that built Mojo Helpdesk, which we still use. The problem is that they don’t tell you much else, like if your registration page is broken or an SVN operation fails. Plus it would be nice to get IM or SMS alerts instead of emails.

So with a little work we took Mechanize (also available in Python and Perl) and a test framework and made high-level tests of our service’s functionality. For Rails users, it’s a bit like making integration tests, only it runs on the actual site, so it doubles as a more thorough site pinger. The downside is that you don’t really have fixtures, it’s all real data and real interactions. That can make it a bit tricky, but if you can get around that, you’ll have a nice service testing script you can run locally or on another server.

You might want to just put it in the other server’s crontab, but unless you get emailed the cron output, just running the tests is pointless without some kind of alerting. Even if you get cron output, you probably don’t want to see the test results every time it runs. If we have to do our own alerting, we might as well get fancy…

It turns out getting fancy is not all that hard thanks to Twitter. With almost no work at all you can write a wrapper for your tests that grabs the test status line (the one with the dots) and the name of the test that fails and then send them as a direct message to you with Twitter’s super simple web API. Then you get messages like these sent to your IM or even to your phone if you have direct messages enabled for it:

...F...... registration (Registration is broken)

......FF.. svn checkout (SVN checkout is broken)

FFFFFFFFFF homepage (The whole site is down)

Once you have your test script done, you can use a simplified version of our Ruby wrapper script to have your own super awesome free IM and SMS alert system:

require 'net/http'
require 'uri'

username = (your twitter user)
password = (your twitter pass)

alert = (array of twitter users to alert)

ruby_bin = (full path to your ruby, since run by cron)
test_path = (full path to your tests)

url = "http://#{username}:#{password}@twitter.com/direct_messages/new.xml"

output = `#{ruby_bin} #{test_path} 2>&1`
test_stats = output.split("\n")[2]
failure = output.match(/\d\) ((Failure|Error):\n([^\(]+))\(/)
if failure
 failure = failure[3].split('_')
 failure.shift
 failure = failure.join(' ')
 alert.each do |u|
  Net::HTTP.post_form(URI.parse(url),
   {'user'=>u, 'text'=>"#{test_stats} #{failure}"})
 end
end

See? Simple! Now you can get woken up in the middle of the night as soon as something breaks…

-Jeff

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: