Peter Marklund's Home

After having fought myself through a couple of Ruby on Rails deployments, it seems there are a lot of little annoying things you need to remember. Some of them are not super critical, at least not in the short run, but they can grow into bigger problems over time. Here is the list:

• Backups and failover. I put this first because I think it is so important. What happens if a hard drive or server crashes? Can you fail over to a different server. How long will this take? Are your backups working? Have you tried doing a recovery? Backing up a MySQL database can be as easy as doing mysqldump from a cron job. You can then use rsync to copy those backup files over to a different server. Remember to also backup any data files that live outside the database. such as media files that you have symlinked under the public directory.
• Documentation. Document how your server is set up and configured. What programs are installed where, where are the config files, how do you restart the servers? This is useful since human memory is short and unreliable and also since we may need to introduce new sys admins and developers to the project.
• Monitoring. You should use an external monitoring server that is entirely external to your system that alerts you by SMS or otherwise when your server is down. You may add to this an internal job that pings your server and restarts your mongrels if they are not responding. You should also monitor disk space and CPU and other parameters of your server. The FiveRuns.com service looks promising when it comes to monitoring.
• Setting the clock. On two recent deployments (Suse and Fedora) I've had issues with the clock being wrong. You can schedule the ntpdate command from a cron job to sync your clock.
• Clearing out old sessions. Set up a cron job to delete old records from the sessions table. I recently worked with a Rails app where this had not been done and the sessions table consequently had millions of rows.
• Clearing out old Capistrano releases. For some reason Capistrano defaults to keeping all releases forever. The fix is easy - add an after_deploy callback that invokes cleanup.
• Log rotation. If you don't rotate your log/production.log file it can quickly grow into hundreds of megabytes or even a few gigabytes. I don't know if that slows your server down, but such log files are not very easy to work with. You can use logrotate to do the rotation, see the Rails Wiki. Here is a sample from etc/logrotate.conf:

  # Rails logs:
    /usr/local/rails_apps/platform/shared/log/*log {
    daily
    missingok
    rotate 100
    compress
    delaycompress
    notifempty
    copytruncate
    create 0666 rails www
  }