Archive for the 'Tips' Category

Consider daylight saving time with cron

November 8, 2007

I know it’s over a week after the fact, but a friend said it’s probably worth mentioning our experience with this recent daylight saving time switch. Ignore this if you don’t observe DST or your servers don’t use it (which might not be a bad idea).

If you have cron scripts that are meant to run only once a day and will run on Sundays (such as daily or weekly schedules), do not run them on or between 1am and 2am. Otherwise you risk them running twice or not at all.

Yes, it may seem obvious, but it’s probably overlooked. We certainly didn’t realize it, but since we just happened to be debugging a weekly cron script running at 1am that Sunday, we caught it in the act of doubling a long running, CPU intensive task.

Happy Wednesday!
-Jeff

Service testing and alerts with Twitter

October 27, 2007

Have you ever messed with those site alert services that ping your site and email you if the site is down? For free services, they can be pretty handy. We used Montastic, made by the same guys that built Mojo Helpdesk, which we still use. The problem is that they don’t tell you much else, like if your registration page is broken or an SVN operation fails. Plus it would be nice to get IM or SMS alerts instead of emails.

So with a little work we took Mechanize (also available in Python and Perl) and a test framework and made high-level tests of our service’s functionality. For Rails users, it’s a bit like making integration tests, only it runs on the actual site, so it doubles as a more thorough site pinger. The downside is that you don’t really have fixtures, it’s all real data and real interactions. That can make it a bit tricky, but if you can get around that, you’ll have a nice service testing script you can run locally or on another server.

You might want to just put it in the other server’s crontab, but unless you get emailed the cron output, just running the tests is pointless without some kind of alerting. Even if you get cron output, you probably don’t want to see the test results every time it runs. If we have to do our own alerting, we might as well get fancy…

It turns out getting fancy is not all that hard thanks to Twitter. With almost no work at all you can write a wrapper for your tests that grabs the test status line (the one with the dots) and the name of the test that fails and then send them as a direct message to you with Twitter’s super simple web API. Then you get messages like these sent to your IM or even to your phone if you have direct messages enabled for it:

...F...... registration (Registration is broken)

......FF.. svn checkout (SVN checkout is broken)

FFFFFFFFFF homepage (The whole site is down)

Once you have your test script done, you can use a simplified version of our Ruby wrapper script to have your own super awesome free IM and SMS alert system:

require 'net/http'
require 'uri'

username = (your twitter user)
password = (your twitter pass)

alert = (array of twitter users to alert)

ruby_bin = (full path to your ruby, since run by cron)
test_path = (full path to your tests)

url = "http://#{username}:#{password}@twitter.com/direct_messages/new.xml"

output = `#{ruby_bin} #{test_path} 2>&1`
test_stats = output.split("\n")[2]
failure = output.match(/\d\) ((Failure|Error):\n([^\(]+))\(/)
if failure
 failure = failure[3].split('_')
 failure.shift
 failure = failure.join(' ')
 alert.each do |u|
  Net::HTTP.post_form(URI.parse(url),
   {'user'=>u, 'text'=>"#{test_stats} #{failure}"})
 end
end

See? Simple! Now you can get woken up in the middle of the night as soon as something breaks…

-Jeff