Peter Marklund

Peter Marklund's Home

Peter on Software

Lessons Learned in software and web development Subscribe

2007-03-28

Rails Gotcha: No ordering of ActiveRecord after_save callbacks

I am setting up tagging and searching for a demo application and I am combining the plugins acts_as_taggable_on_steroids and acts_as_ferret. I was using the following setup for my users:

  acts_as_ferret :fields => [:login, :email, :bio, :tags_string]

  def tags_string
    tags.map(&:name).join(" ")
  end

However, that didn't work. When making a change to my tags I had to save the user twice for the search index to be updated. Why? Because the acts_as_taggable_on_steroid after_save callback (save_tags) runs after the acts_as_ferret after_save callback (ferret_update). Maybe the callbacks are invoked alphabetically, I don't know. Anyway, there is no explicit ordering and no after_after_save callback...

The workaround in this case was easy because of the virtual attribute tag_list that gets set before the save:

    def tags_string
      Tag.parse(tag_list).join(" ")
    end

Now we are always using the latest tag list when we update the Ferret index. Problem solved.

1 comment(s)

2007-03-26

Rails Extension: ActiveRecord::Base.find(:last)

I'm at the Big Nerd Ranch Ruby on Rails bootcamp in a monastery outside Frankfurt right now having a great time. One of the students asked if you can say ActiveRecord::Base.find(:last), which you can't. The value of the first argument is typically :first, but you can't use :last. Just for educational purposes I decided to change that (it's probably not something I would use in production):

ActiveRecord::Base # Make sure class is loaded, which it probably is by now anyway
module ActiveRecord
  class Base
    def self.find_with_last(*args)
      if args.first == :last
        options = extract_options_from_args!(args)
        find_without_last(:first, options.merge(:order => "#{primary_key} DESC"))
      else
        find_without_last(*args)
      end
    end

    class << self # Needed because we are redefining a class method
      alias_method_chain :find, :last
    end    
  end
end

I guess most interestingly this code illustrates how to alias a static method.

5 comment(s)

2007-03-26

Abandoning Textilize for Comment Formating

I've been using the textilize helper for comment formating in this blog. The textilize helper uses the RedCloth gem, which supports both Textile and Markdown syntax and it's pretty complex code with lots of hairy regular expressions. I noticed that the textilize helper will let arbitrary HTML tags through so it can mess up the HTML of a page. Seems like a vulnerability, and it's even more a problem for me since I like to be able to HTML validate my pages. I decided to stop using textilize for comments and instead use the following simpler and tighter formating:

  # HTML quote, convert line breaks to <br />, and convert URLs to links
  def simple_link_format(text)
    # From http://snippets.dzone.com/posts/show/3654
    url_regex = /(http|https):\/\/[a-z0-9]+([\-\.]{1}[a-z0-9]+)*\.[a-z]{2,5}(([0-9]{1,5})?\/.*)?/ix
    simple_format(h(text).gsub(url_regex, '<a href="\0">\0</a>'))
  end

1 comment(s)

2007-03-26

Capistrano is not just for deployment

I'm appreciating Capistrano more and more. It has such an elegant and powerful API, and you can use it for pretty much any commands you need to execute on one or more of your servers. Here is a simple example. For the development of this weblog I found myself quite often wanting to copy the production data to my development server. Of course at first I was doing this on the command line. The next natural step would have been to write a bash script, but these days I tend to write most of my scripts in Ruby. Rather than stick a small Ruby script under the Rails script directory I'm starting to create Rake tasks under lib/tasks though. One advantage of having scripts as Rake tasks is that you can cagalog them nicely with "rake --tasks". Here is the Rake task:

namespace :db do
  desc "Update the development database with a fresh dump of production"
  task :update_dev do
    host = "deploy@marklunds.com"
    db = "marklunds_production"
    system("ssh #{host} \"mysqldump -u root --set-charset #{db} > /tmp/#{db}.dmp\"")
    system("scp #{host}:/tmp/#{db}.dmp ~/tmp")
    system("mysql -u root marklunds_development > ~/tmp/#{db}.dmp")
  end
end

Looking at this code I realized it needed information in the Capistrano config/deploy.rb file, such as the domain of the production server. Also, what the task really does it execute some commands on a remote server, and that's exactly what Capistrano is built for. So I moved the task over to deploy.rb:

require File.dirname(__FILE__) + '/../config/environment'

...the usual deploy.rb stuff here...

desc "Copy production database to development database"
task :update_dev_db, :roles => :db do
  prod_db = ::ActiveRecord::Base.configurations['production']['database']
  dev_db = ::ActiveRecord::Base.configurations['development']['database']
  run <<-CMD
    mysqldump -u root --set-charset #{prod_db} > /tmp/#{prod_db}.dmp
  CMD
  system("scp #{user}@#{domain}:/tmp/#{prod_db}.dmp ~/tmp")
  system("mysql -u root #{dev_db} < ~/tmp/#{prod_db}.dmp")
end

It seems a bit unclean that I have to source the whole Rails environment from Capistrano and it increases the startup time a bit. The reason I do that is to get the database names from ActiveRecord instead of hard coding them. Notice the flexibility of Capistrano tasks - executing local commands is just as easy as executing remote ones.

1 comment(s)

2007-03-25

Rails Deployment: Running Tests in Production with Capistrano

Murphy teaches us that if something can go wrong it will. Given all the differences there are between my development and production server - operating system, web server, database, gems etc. - I can't really feel comfortable deploying my application unless I have first run my test suite not only in development but also in production. Thanks to the beautiful and powerful API of Capistrano this is a walk in the park to accomplish:

desc "Run the full tests on the deployed app." 
task :run_tests do
 run "cd #{release_path} && RAILS_ENV=production rake && cat /dev/null > log/test.log" 
end

desc "Copy in server specific configuration files"
task :copy_shared do
  run <<-CMD
    cp #{shared_path}/system/voxway.rb #{release_path}/config &&
    cp #{shared_path}/system/database.yml #{release_path}/config
  CMD
end

desc "Run pre-symlink tasks" 
task :before_symlink, :roles => :web do
  copy_shared
  run_tests
end

In my current project I am deploying to two identical Linux servers and since Capistrano runs the tests in parallel on those servers I need to keep two distinct test databases in production - one for each server. Credit for this approach goes to Evan Henshaw-Plath who described it nicely.

2 comment(s)

2007-03-25

Rails Deployment: Check MySQL Charset on Startup

The UTF-8 support got a lot better with the Rails 1.2 release but since Rails has evolved so fast many of the instructions you'll find on the web are dated. Rails will now default the Content-Type of its responses to UTF-8 so you don't need to set that yourself in an after filter.

When it comes to configuring MySQL to use UTF-8 there are at least three pieces to keep track of - the database encoding, the client encoding, and the connection encoding. For my project it was enough to set "default-character-set=utf8" in the MySQL my.cnf config file (takes care of the database part) and set "encoding: utf8" in database.yml (takes care of the connection and client).

A really great idea though is to check that the proper encoding is configured when your server starts up in production on your various servers and abort if it isn't. Credit goes to Mike Clark and the Pragmatic Programmers for sharing the following useful code snippet:

if RAILS_ENV == "production"
%w(character_set_database character_set_client character_set_connection).each do |v|
  ActiveRecord::Base.connection.execute("SHOW VARIABLES LIKE '#{v}'").each do |f|
    unless f[1] == "utf8"
      puts "ERROR: MySQL database isn't properly encoded!"
      puts "Kindly set your #{f[0]} variable to 'utf8'."
      RAILS_DEFAULT_LOGGER.error("MySQL database isn't properly encoded!")
      exit 1
    end
  end
end
end

This is yet another example of failing early, one of my favorite design principles, and it seems to have become a running theme in this blog...

1 comment(s)

2007-03-24

Rails Testing: Making assert_select XML safe

I've been using the new assert_select command extensively in my tests lately and it's a wonderfully powerful tool - a huge improvement over assert_tag. The only annoyance has been the warning messages "ignoring attempt to close form with link" that have been polluting my test output. The other day I decided to track the source of the messages and I discovered that others had already complained about them. I found that the root cause of the problem lies in the HTML::Document and the HTML::Tag classes that ship with Rails. More specifically the problem is in the HTML::Tag#childless? method that auto-closes HTML tags such as img, br, and hr.

What do I mean by auto-closing. Well, in XML you have to close all tags, either with a trailing slash, or with a closing tag. In HTML however, we use tags such as <br> and <link>, usually without a trailing slash. The HTML::Tag class takes those special tags into account and understands that they are auto-closing (childless) without the trailing slash. This becomes a problem if you have an XML document that contains <link>some content</link> since the parser will think the first tag is self-closing and the second tag will be a mismatch. Not only does this cause warning or error messages, but it breaks assert_select for those specific tags.

I have submitted a patch that makes HTML::Document enter XML mode if the document has an <?xml declaration. Jamis has posted about his approach which he calls assert_xml_select. However, since all my documents are XML (XHTML or VoiceXML in my current project) I chose the following approach instead which allows me to use assert_select:

if RAILS_ENV == 'test'
  module ActionController #:nodoc:
    module TestProcess
      # Work around Rails ticket http://dev.rubyonrails.org/ticket/1937
      def html_document
        @html_document ||= HTML::Document.new(@response.body, true, true)
      end
    end
  end
end

Setting the first boolean argument to true makes the parser error out if there is an unclosed or mis-matched tag. I want my documents to be valid and I like to fail early.

2 comment(s)

2007-03-20

Rails Tip: Declare your Gem versions

Many Rails applications have external dependencies in the form of RubyGem libraries. The problem with such external dependencies is that you need to make sure that you have the right gem versions installed across all your servers, i.e. development, staging, and production. The probably easiest way around this is to freeze gems into your source tree. For rails you can use the Rake task "rails:freeze:gems" and you can freeze other gems by using Rick Olsen's Gems plugin.

At the end of the day there may still be external gems that you depend on, maybe because they need to be compiled, or maybe because you want to avoid your application source tree growing too big. You can fix the version of Rails using the RAILS_GEM_VERSION constant in environment.rb. For other gems you can fix the versions by adding statements like the following to your environment.rb:

gem "RedCloth", "3.0.4"

The gem command specifies that we will be using a gem of a certain version and it will attempt to add that gem to the load path (activate it), but not require it. If the gem version is missing you will be alerted on server startup.

So how do you know which gems you are depending on? One way might be to look at the load path. The following expression will return a list of Gems that have been activated so far in your application:

$:.map { |path| path[/^#{ENV['GEMS_PATH']}\/([^\/]+)/, 1] }.compact.uniq

You can inquire about a certain gem on your system using "gem list" and "gemwhich":

$ gem list|grep RedCloth
RedCloth (3.0.4, 3.0.3)
    RedCloth is a module for using Textile and Markdown in Ruby. Textile

$ gemwhich RedCloth
/usr/local/lib/ruby/gems/1.8/gems/RedCloth-3.0.4/lib/RedCloth.rb

Also, don't forget that if the gem comes with RDoc you can access that documentation by invoking gem_server on the command line.

1 comment(s)

2007-03-20

RubyGems: LoadError in nested require no longer causes silent failure with Ruby 1.8.5

I had a Rails application where I was trying to reference a helper that didn't exist. What happens in Rails when you try to use a class that you haven't required is that the Ruby callback method const_missing is invoked. Behind the scenes Rails will then try to require the class for you. What was happening was that Rails was trying to require the helper file, but the helper file didn't exist, and so that failed. The problem wasn't so much that it failed, but that it failed silently - there was no error message to be seen anywhere.

After having painstakingly tracked down the problem it turned out that it had to do with the Ruby require method and how RubyGems (I was using Rails through RubyGems) overrides the require method. RubyGems redefines the require method so that it first invokes the original Ruby require method, and if a LoadError is thrown, it will attempt to find the appropriate Gem, add it to the load path, and then invoke the Ruby require method again.

I posted about this silent failure on the RubyGems mailinglist and Eric Hodel suggested I should file a bug if I could reproduce the problem without Rails. Now several weeks later, and after an upgrade to Ruby 1.8.5, it turns out I can no longer reproduce the bug. The reason is that the Ruby require method behaves slightly differently with respect to LoadErrors in Ruby 1.8.5.

This may never be relevant to you, but let me just briefly try to explain the details of what is going on. Suppose you have a file called test.rb with a single line that says:

  require 'this_file_does_not_exist'

Then with Ruby 1.8.4 you would get:

$ irb
irb(main):001:0> require 'test'
LoadError: no such file to load -- this_file_does_not_exist
        from ./test.rb:1:in `require'
        from ./test.rb:1
        from (irb):1
irb(main):002:0> require 'test'
=> false

As you can see the second time you try to require the file the require method returns false, i.e. no LoadError is thrown. This indicates that the file has already been loaded. However it doesn't tell us that the loading failed. This absence of a second LoadError is what yields a silent failure if test.rb lives in an activated gem. The reason is that the first LoadError is caught by the RubyGems require method.

With Ruby 1.8.5 however a LoadError will be thrown everytime you do "require 'test'" and thus the LoadError will no longer be hidden by RubyGems. I don't know why they changed the contract of the Ruby require method, they might have had other reasons, but my suspicion is it was to avoid some kind of silent failures.

2007-03-02

Rails Gotcha: ActiveRecord::Base#update_attribute skips validation

It's something others have been bitten by. Although we should know better (why didn't we read the API docs carefully?) we naivley thought we could safely update a single attribute with the update_attribute method. When you invoke my_precious_object.update_attribute(:important_attribute, erroneous_value) then Rails will happily skip validations and save all attributes to the database. This is unlike the sister method update_attributes which updates several attributes and does use validation. I'm sure validation is skipped for a reason but I'm not sure what that reason is. In particular I'm not sure if that reason is strong enough to warrant the inconsistency and the risk of skipping validations.

I for one thought all public methods that create and update an ActiveRecord object were supposed to use validations. That would have seemed natural and even essential. Especially since most Rails apps run on MySQL which doesn't have a great track record when it comes to data integrity.

As far as I can see there is no mention in the Rails book of this validation glitch. I think I should pass a note to Dave Thomas about that.

If you look under the hood of Rails, it turns out update_attribute is equivalent to:

  object.attribute = value
  object.save(false)

The update_attribute method is first defined in the Base class and then overridden by the mixed in Validations module. To see the definitions of the save, update_attribute, and update_attributes methods you would be looking in vain in the Base class, in fact, the code you see there is never used, which feels a little bizarre. Fortunately there are comments in there pointing us to the Validations module.

On a side note, notice that unlike in Java, there is no method overloading on argument signatures in Ruby, so a method is identified solely by its name. When you redefine a method you can effectively change its argument list if you like, although this is probably not a good idea in most cases. The Validations module uses this ability though for the save method to add an optional boolean argument that determines if validation should be performed or not. This in turn is what the redefined update_attribute method makes use of. Questions? :-)

10 comment(s)

2007-03-02

Nokia N70 - 11 Clicks Away From Usability

I've had the privilege and misfortune of using a Nokia N70 3g camera phone over the last couple of days. My first impression of the phone has been mostly a series of frustrations but most importantly a lesson in the importance of usability and delivering on the basics. To quote 37Signals:

The basics are the secrets of business. Execute on the basics beautifully and you’ll have a lot of customers knocking at your door. Cool wears off, usefulness never does.

Well, if 37Signals had consulted for the Nokia N70 development team I'm sure its UI and featureset would have been radically different.

To use a phone I need to know how to turn it on and off, how to disable the keyboard to put the phone in my pocket, how to put the phone in silent mode, how to make normal old fashioned phone calls, how to store friends phone numbers, and how to send SMS (including how to turn the auto completion on and off and write special chars like punctuation). That's about it. That's the full list of essential features that I need to master in order to pick up a phone and get by with it in my everyday life.

There is too much noise in the N70 UI, the essential features are not prominent and accessible enough. All else being equal, the more features you add, the more noise, the less obvious the basics will be, and the lower the usability. The user manual mirrors the N70 UI in being unstructured and not highlighting the basics. For example, I would have expected the first chapter to tell me what's in the package, have a picture of the phone and explain essential buttons and functions, how to turn the phone on and off etc. The Nokia N70 manual doesn't care for this approach but instead dives straight into advanced features such as how to configure the camera.

Another reason for low usability is not following conventions. The N70 phone is in between a Windows desktop UI and the UI of a second generation phone (remember the phones that didn't have cameras and internet?). It has its own set of conventions and before you have loaded those into your brain you are likely to be left confused and frustrated. Clicking the "select" button placed to the left on the phone is the equivalent of the right mouse button in Windows. Ok, so left and right are confused, we can live with that I suppose. Well, hold on, that analogy doesn't even hold because some of the options you get when you press "select" are not related to that particular item but to the whole list of items...

Putting the phone in silent mode is something I need to do often and its about 11 clicks on the Nokia N70. At the same time the desktop (start screen) is mostly empty and as far as I can tell not configurable, and several buttons on the start page are left unused.

Let's leave this ramble on a positive note - taking pictures with the phone is easy! Just slide open the camera and press the button. I figured that out on the first try. Except, the annoying thing about the camera is that it slides open by mistake and is left in camera mode in my pocket. This might explain why the battery drains so quickly, or maybe that is because of the large screen and all the software. Oh well, I try to be positive, I really do...

10 comment(s)

2007-01-31

Swedish Rails Article and Course

I was very pleased to see my Ruby on Rails article published yesterday by the IT education company Informator here in Stockholm. The article is in swedish and has a title along the lines of "Rails Brings Back the Joy of Working". It was published in the slick offline paper called "Format" that Informator distributes and is also available online.

I there is enough interest (fingers crossed) I will be giving the Ruby on Rails course for Informator here in Stockholm later this year. Needless to say, I'm very excited about that opportunity!

4 comment(s)

2007-01-06

Rails Tip: Use the Unit Tests as Documentation

If you are looking for authoritative answers on how to use the Rails API, look no further than the Rails unit tests. Those tests provide excellent documentation on what's supported by the API and what isn't, how the API is intended to be used, the kind of use cases and domain specific problems that drove the API, and also what API usage is most likely to work in future versions of Rails.

Let me give you a concrete example. In my code I had the equivalent of a Tag class connected through a Tagging join model with two different classes, say Post and Comment. What I was trying to do was say in the Tag class something like "has_many :posts, :through => :taggings, :source => :taggable" or maybe "has_many :taggables, :through => :taggings". However, Rails refused to let me do this and kept throwing an ActiveRecord::HasManyThroughAssociationPolymorphicError. At this point I wasn't sure if I was misusing the API or if I had come across a Rails bug (although I knew that the former was more likely).

I turned first to the most authoritative documentation such as the API doc, the Agile Web Development with Rails book, and the Rails Recipes book (Recipe 22: "May-to-Many Relationships with Extra Data", and Recipe 23: "Polymorphic Associations - has_many :whatevers"). However, none of that documentation seemed to address my specific problem. Through Google I found the wiki page HowToUsePolymorphicAssociations which did address my problem. However, the solution there seemed clumsy, and I didn't know how authoritative and up-to-date it was.

It's usually a good idea to debug a problem by following the stack trace into the Rails code and try to figure out what's going on right there. However, in this particular case I found that reading the clean but complex ActiveRecord code didn't give me any easy answers. It then hit me that what I really should be doing was read the ActiveRecord unit tests. Luckily for us application developers David and the rest of the core team has provided us with a nice suite of unit tests. Once I looked at the join model test case the whole picture cleared up - the fog lifted and there was bright sunlight. Laid out right there in front of me were excellent and authoritative examples of how to use ActiveRecord associations with classes such as Tag, Tagging, Post, Comment, Category, and Categorization. I also found the particular assertion that said that the ActiveRecord::HasManyThroughAssociationPolymorphicError should be raised in my case...

One could speculate that Rails really should provide a way to do has_many through a polymorphic association, and surely there is a way to hack Rails into doing something like that. What's important though is that now I know that the approach is not officially supported or intended and that's fine with me. I like to stay on the unit tested golden Rails path and I recommend you to do the same.

1 comment(s)

2007-01-04

Good and Bad Programmer Practices

As programmers we have a tendency to take shortcuts that come back and bite us eventually. It is very popular to write overly complex system that seem sophisticated and make us feel smart and special. The irony is that a lot of times we think our code is so simple and obvious that it doesn't need automated tests. Here is a list of programming practices to consider:

In summary then, keep your code clean, simple, conventional, and well tested and listen to your users. Those recommendations may sound obvious, in practice though they often contradict our instincts as programmers and require discipline and courage to stick to.

2006-12-27

New Rails Plugin for HTML Validation: Html Test

Update on 2009-04-16: The source code has been moved from Google Code to GitHub

Update on 2007-12-12: I have renamed the plugin from "Http Test" to "Html Test" and moved it to a new home at Google Code. Sorry about the inconvenience that this may cause those of you who have the plugin installed and want to update it. The change is essentially a namespace change from Http:: to Html:: so it should be manageable. Thanks.

As I wrote earlier I like to monitor my production sites with HTTP tests that validate their HTML and check for broken links and images. I think that serves as a good baseline check, a smoke test if you like, that the site is not fundamentally broken.

It used to be that my HTTP tests were implemented in Perl using WWW::Mechanize. Now that we have the wonderful Ruby and Ruby on Rails at our fingertips that felt like a real shame. I decided to implement my HTTP tests in Ruby and once I had gotten that far, going the extra mile to package my scripts as a Rails plugin seemed natural. The plugin is called http_test and it's available at Google Code.

So far, what the plugin does is mostly provide support for HTML validation, and to some extent link checking. It provides a command line script that you can use to conveniently HTML validate your site, either in production, staging, or development, as you please. You give the script a URL and it will fetch and validate the page over HTTP. It will also follow links and validate those pages. The script is not a web spider though. It only follows links one step away from the start page.

The other thing you can use the plugin for is to validate pages in your controller and integration tests. The plugin provides the assertion trio assert_tidy, assert_w3c, and assert_xmllint. There is also the catch all assertion assert_validates that can invoke all three validators for you.

What I like to do is setup my application controller to validate all my requests in the test environment. That way you know that all pages that you request in your functional and integration tests validate. The plugin makes this easy. All you have to do is add the following lines to your test_helper.rb:

    ActionController::Base.validate_all = true
    ActionController::Base.validators = [:tidy]

To be able to use Tidy the http_test plugin relies on code in the rails_tidy plugin, so credit to Damien Merenne for providing that. Credit also goes to Scott Baron for the W3C validation, and to Maik Schmidt, author of "Enterprise Integration with Ruby", for the xmllint validation code.

I hope the Html Test plugin will be useful to others in their HTML validation and link checking pursuits. Please let me know what works and what doesn't and how I can improve. Thanks!

7 comment(s)

2006-12-12

Hype and the Rise and Fall of Programming Languages

Steve Yegge has written an article on Ruby as a competitor to Python, the Rails hype, and the rise and fall of programming languages. The article touched me deeply. Thanks to Olle Jonsson for sending the link. The article is almost a year old but should still be relevant. I especially like the last section on Ruby:

"The worldwide Ruby culture is the warmest and friendliest I've seen in my long history with programming languages. And Ruby is a sweet language. Other people seem to agree, and are taking steps to market it, which is getting them labeled as "hyper-enthusiasts" by the Sour Grapes camp. It appears to me that Ruby is doing what I wanted Python to do a few years ago, so I've finally learned Ruby and have switched most of my development over to it.

After all, both languages have a long way to go before they catch up with Java in terms of tools, IDEs, books, performance, threading stability, and tons of other stuff. I wanted to make a reasonably educated bet, and choose the language I think is going to be bigger, so it'll work well for me, and so I won't have to fight so hard to use it in my job.

It wasn't hard to learn Ruby. In fact after a few days with it, Ruby felt as comfortable as languages I'd been using for years. I've really been enjoying it. It has a few warts; all languages do. No biggie. It looks as if Matz is intent on fixing them in Rite.

I don't know if I like it more than Python and Scheme. I like it at least as much as those languages, certainly. But Ruby's my favorite (as in "preferred") language now because I can see the trajectory it's on, especially now with Rails, and I believe it's going to be the Next Big Thing -- a Java-like phenomenon. So did Bruce Tate when he wrote "Beyond Java". So do James Duncan Davidson, Dave Thomas, Martin Fowler, and many other people who are a heck of a lot smarter than me. You'd think they're on to something big, wouldn't you? I do.

Java-like worldwide adoption really matters. Without that level of mass-market adoption, Ruby won't get the tools, stability, and CPAN-like library selection that it needs in order to compete with Java and Perl. It's a chicken-and-egg problem that all languages face, and Ruby stands a chance of succeeding where Smalltalk, Python, and other great languages have (to date) failed.

I see Rubyists worrying that Rails is stealing the show. Geez, folks, LET it steal the show. Talk about a free ticket for Ruby success. Java Applets were a way to get Java in front of a million or so programmers, ultimately allowing the Java platform to succeed in all sorts of domains that it might never have seen without the initial "killer app" of Applets.

We live in a world where culture matters, economics matter, and marketing hype matters. They are very real forces that directly affect our quality of life as programmers. You ignore them at your peril, a lesson learned by so many almost-forgotten languages that were stomped by marketing hurricanes like Java and Perl.

I really wanted Python to succeed, and I still wish them the best, but I think they're ignoring marketing. I really want Ruby to succeed, so I get a bit miffed when I hear famous people like Bruce Eckel making uninformed generalizations about both Ruby and the folks who are working hard to make it successful. I think Pythonistas should be focusing on doing the same -- working to make Python successful. I do think it will take a minor cultural adjustment on their part. And they need to start accepting hype as a natural part of the world we live in, a requirement for cutting through the noise. But I think they can do it. "

2006-12-08

MySQL Gotchas and a Deadly Sin

Coming from mostly a PostgreSQL and Oracle background I experienced some major friction working with MySQL today.

The first obstacle to overcome was the syntax for foreign keys. I started out using

  create table test (
    service_id integer NOT NULL references services(id) on delete cascade
  );

MySQL 5.0.21 on OSX swallowed this syntax happily with no complaints. Only by running Rails unit tests did I discover that to actually create a foreign key in MySQL (and not just look like you are) you need to write:

  create table test (
    service_id integer NOT NULL,
    foreign key (service_id) references services(id) on delete cascade
  );

When I then actually had working foreign keys the ordering of table create and drop statements became very sensitive. If table A has a foreign key to table B then table B must be created before table A. Also, MySQL lacks the DROP TABLE CASCADE feature so the order of drop table statements needs to be the reverse of the create table statements. In this case (apparently for porting reasons) you can still write DROP TABLE CASCADE but MySQL will just ignore the CASCADE command. This, just like the foreign key example above, is a form of silent failure that I think in general is the root of much frustration among developers. It's an anti-pattern.

Now here comes the grand finale and my worst discover when it comes to MySQL today - the implementation of NOT NULL. You see in this case, the NOT NULL instruction is not ignored by MySQL, but arguably it's not fully implemented either (see a good discussion on this here). It turns out that if you try to insert NULL into a NOT NULL column MySQL throws an exception. This is to be expected and is exactly what we want. However, if you instead try to update the column to NULL MySQL happily sets the column to the empty string...

CREATE TABLE peter (
  name varchar(100) NOT NULL
) ENGINE=InnoDB;

mysql> insert into peter (name) values (NULL);
ERROR 1048 (23000): Column 'name' cannot be null

mysql> insert into peter (name) values ('foobar');
Query OK, 1 row affected (0.06 sec)

mysql> update peter set name = NULL;
Query OK, 1 row affected, 1 warning (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 1

mysql> select count(*) from peter where name = '';
+----------+
| count(*) |
+----------+
|        1 | 
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from peter where name is null;
+----------+
| count(*) |
+----------+
|        0 | 
+----------+
1 row in set (0.00 sec)

I'm at a loss of words...

3 comment(s)

2006-12-03

Usability on the Web and Embracing Simplicity

I just booked flight tickets for the christmas holiday and it got me thinking of usability. People depend on websites so much these days and it's a shame to let them suffer from poor usability. Of course, usability has improved quite a bit over the last couple of years. Indeed SEB finally uses a date widget, and I think even suggests a sensible default (the next working day), when you do payments through their online bank. For the longest time they left the user with a bare bones YYYYMMDD text input widget.

Booking tickets on the web is often a series of frustrations for me. This is in spite of the fact that I am a skilled web user. For example a form won't submit and the validation error is hardly visible. The user is left wondering if the form submitted at all, if there was a validation error, and if so, where?

When I am booking tickets I am almost always in a hurry. Buying a ticket shouldn't take more than a few minutes, and it should be a simple and streamlined process. You want to reduce complexity, minimize the number of options and thus the likelihood of the user being confused.

When ordering my tickets I chose to become member of a frequent flyer program. I checked my email and found about five different confirmation emails from the airline. One of them contained my account number which I eventually found, only I couldn't use that number to log in, I had to figure out myself that I needed to strip the first two characters (leaving only digits) for it to work. I then had to dig out my password, I think from a different email. Once I had logged in I was offered to change my password. Only I couldn't use my standard password that I have memorized, I was forced to choose a new four digit password that I may end up forgetting.

In the midst of usability shortcomings one can of course always take comfort in the fact that online bookings are still better than their offline alternatives. We still have a long way to go with web usability, and as usual, I think most of the answer lies in embracing simplicity. It has struck me again and again through my different endeavours - programming, writing, UI design - that simplicity is key. With simplicity comes focus and a clear goal. Embracing simplicity is something most people can agree with in principle, but putting it to practice takes courage. It is all too easy to go with the main stream complex solution, it seems so much more sophisticated and professional and makes you feel competent and special. After all, you are one of the chosen few who can understand the solution. Therein lies the problem.

1 comment(s)

2006-11-27

Third Place in Salsa Competition

There is a quote by Woody Allen that rings more and more true to me:

Eighty percent of success is showing up

Last friday me and my girlfriend Janne were able to win third price at a Salsa competition at La Isla here in Stockholm. Thanks to the encouragement of Pilar and Ociel at Stockholm Salsa dance we decided to give the competition a try and it was great fun! My friends Håkan and Mimmis won the silver, and a very professional and choreographed couple from Uppsala took home the gold. We were very pleased to walk home with the trophies and 1000 SEK though...

There is a huge difference between social dancing and show dancing, and as my friend Håkan said, if you want to be a successful show dancer you need to learn to dance outwards (with the audience) rather than inwards (with yourself and your partner and the music).

2 comment(s)

2006-11-15

Teaching Rails at Great Works

I've started teaching Ruby on Rails at Great Works and I'm very excited about it! It's a three evening course given to a small group of up to 10 people. We use a lot of hands on excercise rather than lecturing and the goal is to build a small Rails application. The course participants are designers rather than programmers so we needed to lay down some object orientation and programming basics before we could take on Rails. Despite of this obstacle we were able to make a lot of progress by using Locomotive for installation and scaffolding for UI generation. You can check out the slides here.

I genuinely enjoy teaching and am now looking for more opportunities to help companies learn and get started with Rails quickly. Get in touch if this sounds interesting.

2006-11-15

Stockholm Rails Meetup Tonight was Awesome

We had a fantastic turnout and great atmosphere at the Rails Meetup at Valtech here in Stockholm tonight. There was a great presentation by Ola Bini on web services and Rails, and one by Christian and Albert at Adocca about caching and a bunch of plugins that they have realeased at RubyForge. All very exciting.

I gave a presentation on the Media Contact Manager system that I have built for Green Media Toolshed.

2006-11-14

Rails: File Uploading and Storage

File uploading and storage is no big deal, right? I mean with Ruby it can be as easy as:

  def create
    File.open("/Users/db/_dev/rails/videos/swing.mov", "w") do |f|
      f.write(@params["video"].read)
    end
  end

Don't forget to set :multipart => true on your form though. The easiest (and most naive?) approach to uploading files is probably to use the File Column plugin. The Rails Wiki has a good page that discusses file uploads in more depth. The file system base storage approach that I've used in my Rails app is similar to what's described here. I use the after_save and after_destroy callbacks to create and delete the file in the file system - an approach also described in this Rails manual. Johan Sorensen has some code for doing buffered file uploads that might be useful for huge files.

This page is quite impressive. It has everything you ever wanted to know about the input type="file" form widget, including why browsers will ignore the value attribute if you try to set it. An annoyance is that if the user gets validation errors on a form containing a file_field widget she will have to re-select the file to upload.

The classic controversy is whether to store files in the database or in the filesystem. If you store them in the database you have a more unified and consistent storage solution. You keep all your data in one place, in one format, and the only data you need to back up is your database. On the other hand, putting files in the file system could potentially offload your database and make the database dumps smaller. Also, databases were not really intended to store files, that's what file systems are for. Personally I'm ambivalent on this issue, but due to some incidents with file system storage recently I'm now leaning more in favor of storing files in the database.

2 comment(s)

2006-11-11

Rails script/console - Did you know it could do all this?

Yesterday someone pointed out some features of the Rails console that I had missed out on. Here are some objects and commands to be aware of:

4 comment(s)

2006-11-09

Rails Patch: Rake :test should not fail silently

Swallowing exceptions and failing silently has to be one of the worst sins you can commit as a programmer and I was surprised to run into that issue with the Rake :test task. The Rake :test task (the default task) will in turn invoke the tasks test:units, test:functionals, and test:integration. Any exceptions raised within these sub tasks are not exposed directly but are merely noted as an overall test failure - obscuring the root cause of the failure. This is obviously not very debugging friendly if you have an exception raised somewhere deep down in the Rake task dependency hierarchy. I submitted a patch to remedy the problem.

3 comment(s)

2006-11-09

Rails QA: Watch Out for Duplication in your Fixtures

I had a long debugging session due to duplicated record keys in one of my fixture files. It turns out the YAML parser will not complain if you have duplicated record keys (the keys on the top level) or duplicated column keys. Eventhough the YAML specification says that map keys should be unique the YAML parser will happily overwrite any existing value for a key if it encounters that key again.

Under the motto of crashing early and avoiding silent failures I came up with the following unit test:

# Check for duplication in Fixture files, i.e. that:
#
# - keys of records are unique
# - column names are unique
# - id values are unique
#
# The idea is that this test can complent the syntax checking of the YAML
# parser and help avoid debugging nightmares due to duplicated records or columns.
# What the YAML parser will do when a record or column key is duplicated is it
# will just use the last one and let that overwrite the earlier ones.
# Before writing this script I tried using the Kwalify YAML validator but I
# couldn't quite coerce it into doing what I wanted.
class FixtureTest < Test::Unit::TestCase
  def test_fixtures
    fixture_file_paths.each do |file_path|
      initialize_variables(file_path)
      fixture_contents(file_path).each do |line|
        next if skip_line?(line)
        if is_record?(line)
          assert_record_not_dupe(line)
        elsif is_column?(line)
          assert_column_not_dupe(line)
          assert_id_not_dupe(line) if is_id?(line)
        end # End if statement
      end # End line loop
    end # End fixture file loop
  end

  private
  def initialize_variables(file_path)
    @file_path = file_path
    @record_keys = []
    @column_keys = []
    @ids = []   
  end

  def fixture_file_paths
    Dir["#{Test::Unit::TestCase.fixture_path}/**/*.yml"]
  end

  def fixture_contents(file_path)
    ERB.new(IO.read(file_path)).result
  end

  # Skip YAML directive, comments, and whitespace lines
  def skip_line?(line)
    line =~ /^\s*$/ || line =~ /^\s*#/ || line =~ /^---/    
  end

  # A record has no indentation (leading white space)
  def is_record?(line)
    line =~ /^\S/
  end

  # A column line has some indentation (should be same as the first column)
  # and a colon
  def is_column?(line)
    return false if line !~ /^\s+[a-zA-Z_]+:/

    # Looks like a column - check the indentation level as well
    indent_level = line[/^(\s+)/].length
    if @column_indent
      # If the indentation level is different from the first column this may
      # not be a column, it could be nested data of some sort
      return @column_indent == indent_level
    else
      # This is the first column so remember its indentation level
      @column_indent = indent_level
      return true
    end
  end

  def is_id?(line)
    column_key(line) == "id"
  end

  def column_key(line)
    line[/^\s+([^:]+):/, 1]
  end

  def assert_record_not_dupe(line)
    record_key = line[/^(?:- )?([^:]+):/, 1]
    assert !@record_keys.include?(record_key),
      "Record key #{record_key} in fixture file #{@file_path} " +
      "is duplicated on this line: #{line.chomp}"
    @record_keys << record_key
    @column_keys = []
    @column_indent = nil
  end

  def assert_column_not_dupe(line)
    assert !@column_keys.include?(column_key(line)),
      "Column #{column_key(line)} for record #{@record_keys.last} is duplicated " +
      "in file #{@file_path} on this line: #{line.chomp}"    
    @column_keys << column_key(line)
  end

  def assert_id_not_dupe(line)
    id_value = line[/^\s+id:\s*(\S+)/, 1]
    assert !@ids.include?(id_value),
      "Value for id column duplicated in file #{@file_path} on this " +
      "line: #{line.chomp}"
    @ids << id_value
  end
end

2 comment(s)

2006-08-24

Rails Recipe: Checking for Broken Links in Integration Tests

Having a way to automatically check for broken links in your application can be quite valuable and there is a simple way to do that with integration tests. Here is the DSL methods that I used for that purpose along with some sample usage:

  module TestingDSL
    def visits_start_page      
      get_with_links("/mcm") do
        assert_tag :tag => 'a', :attributes => {:href => '/mcm/contacts/search_form'}
      end
    end

    def get_with_links(*args)
      request_with_links(:get, *args)
    end

    def post_with_links(*args)
      request_with_links(:post, *args)
    end

    def request_with_links(method, *args)
      self.send(method, *args)
      yield if block_given?
      assert_response :success
      check_links            
    end

    def check_links
      find_all_tag(:tag => 'a').each do |anchor|
        url = anchor.attributes['href']
        if check_url?(url)
          get(url)
          assert_response_code
        end
      end      
    end

    def check_url?(url)
      [/mailto:/, /javascript:/, /user\/logout$/, /:\/\//, /^\#$/].each do |skip_pattern|
        return false if url =~ skip_pattern
      end
      true
    end
    
    def assert_response_code
      assert [200, 302].include?(@response.response_code), 
        "Invalid response code #{@response.response_code} for path #{@request.path} #{@response.body}"
    end
  end

I've found this approach to link checking helps me find bugs that other tests don't find. The only downside is that it can significantly increase the number of requests in your integration tests and thus the test time. There are ways to work around that though, you don't have to check links on every page request, and the link checker could be extended to not request the same URL twice.

3 comment(s)

2006-08-23

Rails Workaround: Preserving Nested Hash Params With List Values Across Requests

Suppose you have an HTML search form with a multi-valued select with the name "person[personal][interests][]". The trailing brackets are there to indicate to rails that this HTTP parameter should be converted to a Ruby list. Now, suppose the form submits to the search action in the contacts controller. Our params hash will then contain:

    params = {
      :controller => 'contacts',
      :action => 'search',
      :person => {
        :personal => {
          :interests => ['music', 'chess']
        }
      }
    }

So far so good. Now suppose you have pagination links on the search results page and also that you want to provide a link or button back to the search form for refining the search criteria. Here come the bad news - url_for which is used to create links doesn't support nested hash parameters and also doesn't support list values (there are some patches already to address at least the nesting problem). Here is what url_for produces in this case:

    > puts CGI.unescape(url_for(params))
    > /contacts/search/?person=personalinterestsmusicchess

Clearly url_for is inadequate in this case. I have written a set of helper methods to help save the day:

  def flatten_hash(hash = params, ancestor_names = [])
    flat_hash = {}
    hash.each do |k, v|
      names = Array.new(ancestor_names)
      names << k
      if v.is_a?(Hash)
        flat_hash.merge!(flatten_hash(v, names))
      else
        key = flat_hash_key(names)
        key += "[]" if v.is_a?(Array)
        flat_hash[key] = v
      end
    end
    
    flat_hash
  end
  
  def flat_hash_key(names)
    names = Array.new(names)
    name = names.shift.to_s.dup 
    names.each do |n|
      name << "[#{n}]"
    end
    name
  end
  
  def hash_as_hidden_fields(hash = params)
    hidden_fields = []
    flatten_hash(hash).each do |name, value|
      value = [value] if !value.is_a?(Array)
      value.each do |v|
        hidden_fields << hidden_field_tag(name, v.to_s, :id => nil)          
      end
    end
    
    hidden_fields.join("\n")
  end

Here is the output of my helpers for our example hash above:

  > puts CGI.unescape(url_for(flatten_hash(params)))
  > /contacts/search?person[personal][interests][][]=music&person[personal][interests][][]=chess

  > puts hash_as_hidden_fields(params)
  > <input name="action" type="hidden" value="search" />
  > <input name="controller" type="hidden" value="contacts" />
  > <input name="person[personal][interests][]" type="hidden" value="music" />
  > <input name="person[personal][interests][]" type="hidden" value="chess" />

Hope this will save other Rails programmers some time and frustration.

6 comment(s)

2006-08-18

Rails Recipe: HTML Validation

In this howto I'll show a simple approach to HTML validation that I use in my current Rails application. For me, HTML validation is a way to achieve wide browser compatibility, and to do a baseline check for correct rendering and UI brokenness. By putting ampersands and less-than signs in my test fixtures I can use HTML validation tests to check that I haven't forgotten any HTML quoting in my templates, i.e. that I haven't forgotten to use the following constructs:

<%=h some_variable %>
<%= link_to h(some_variable) ... %>

Two common tools for HTML validation are the W3C Validator and Tidy and since I've found them to be complementary I've decided to use both. Tidy warns about empty tags which the W3C validator doesn't. On the other hand Tidy sometimes misses obvious errors such as missing paragraph end tags.

The approach that I came up with for HTML validation was to do it in an after_filter, but only when tests are run, so I added the folling to my test_helper.rb:

# HTML validate the response of all requests
require File.join(File.dirname(__FILE__), '..', 'app', 'controllers', 'application')
class ApplicationController
  after_filter :assert_valid_markup

  def status_code
    @response.headers['Status'][0,3].to_i
  end

  def assert_valid_markup
    return if RAILS_ENV != 'test'
    return if !(status_code == 200 &&
      @response.headers['Content-Type'] =~ /text\/html/i && @response.body =~ /<html/i)

    assert_tidy

    # Going to the W3C validator over HTTP is a bit slow so we make this optional
    return if !ENV['HTML_VALIDATE']
    assert_w3c_validates
  end

  def assert_tidy
    tidy = RailsTidy.tidy_factory
    tidy.clean(@response.body)

    unless tidy.errors.size.zero?
      message = ("-" * 40) + $/
      i = 1
      @response.body.each do |line|
        message << sprintf("%4u %s", i, line)
        i += 1
      end
      message << ("-" * 40) + $/
      message << tidy.errors.join($/)
    end
    raise "Tidy detected html errors in response body: #{$/} #{message}" unless tidy.errors.size.zero?
    tidy.release
  end
 
  def assert_w3c_validates
    require 'net/http'
    print "Querying W3C XHTML validator ... "
    response = Net::HTTP.start('validator.w3.org') do |w3c|
      query = 'fragment=' + CGI.escape(@response.body) + '&output=xml'
      w3c.post2('/check', query)
    end
    raise response.body if response['x-w3c-validator-status'] != 'Valid'
    print response['x-w3c-validator-status']   
  end
end

As you can see from the code above, I only do the time consuming HTTP request to the W3C validator if the environment variable HTML_VALIDATE is set. This way I can easily turn off W3C validation, the obvious risk here being that it always stays turned off. Possible solutions include running the tests with full HTML validation nightly and to install the W3C validator locally.

In the code above you can also see that I use the excellent assert_tidy command from the RailsTidy plugin, so installing that plugin along with the Tidy library itself is a prerequisite for the code to work.

Great tools that I use for manual HTML validation include the Web Developer Extension for Firefox with its W3C validation capability for local HTML and the Safari Tidy plugin that allows you to see for every loaded page any Tidy errors and warnings.

When selecting a DOCTYPE to validate against I was choosing between XHTML 1.0 strict and transitional and I was convinced by certain experts that strict was the way to go.

Doing HTML validation is not without its frustrations of course. For example I had to work around the fact that in XHTML 1.0 strict, form elements (input, select etc.) need to be inside a p, div, or fieldset tag. Also, Tidy requires tables to have the summary attribute. Those are just small annoyances though and I haven't come across any bigger stumbling blocks yet. All in all I'm very happy about my validation efforts and I have a lot more confidence in the UI of my Rails application now that I'm validating its markup automatically in my controller and integration tests.

7 comment(s)

2006-08-17

Rails Custom Validation

Suppose you have validation code for a certain type of data that you want to reuse across your ActiveRecord models. What is the best approach to doing this in Rails? As you probably know, Rails comes with a number of validation macros such as validates_presence_of, validates_format_of, and validates_uniqueness_of etc. One approach is to write your own validation macro that wraps one of the Rails validation macros. The validates_as_email plugin shows us how:

module ActiveRecord
  module Validations
    module ClassMethods
      def validates_as_email(*attr_names)
        configuration = {
          :message   => 'is an invalid email',
          :with      => RFC822::EmailAddress,
          :allow_nil => true }
        configuration.update(attr_names.pop) if attr_names.last.is_a?(Hash)

        validates_format_of attr_names, configuration
      end
    end
  end
end

The RFC822::EmailAddress constant in the code above is a huge regexp that I'm not going to list here... A macro that wraps validates_format_of works well as long as your validation can be done with a regexp. What about validations that are too unwieldly or impossible to do with a single Regexp? Suppose you want to validate phone numbers to make sure they have 10 digits and only contain the characters 0-9()/-.+. Using validates_format_of as a starting point we could write our own macro:

      def validates_as_phone(*attr_names)
        configuration = {
          :message => 'is an invalid phone number, must contain at least 5 digits, only the following characters are allowed: 0-9/-()+', 
          :on => :save 
        }

        validates_each(attr_names, configuration) do |record, attr_name, value|
          n_digits = value.scan(/[0-9]/).size
          valid_chars = (value =~ /^[+\/\-() 0-9]+$/)
          if !(n_digits > 5 && valid_chars)
            record.errors.add(attr_name, configuration[:message])
          end
        end
      end

In the general case we could reduce duplication by hacking validates_format_of and make its :with option accept Proc objects:

      def validates_format_of(*attr_names)
        configuration = { :message => ActiveRecord::Errors.default_error_messages[:invalid], :on => :save, :with => nil }
        configuration.update(attr_names.pop) if attr_names.last.is_a?(Hash)
        
        raise(ArgumentError, 
          "A regular expression or Proc object / method must be supplied as the :with option of the configuration hash") \
            unless (configuration[:with].is_a?(Regexp) || validation_block?(configuration[:with]))

        validates_each(attr_names, configuration) do |record, attr_name, value|
          if validation_block?(configuration[:with])
            value_valid = configuration[:with].send('call', value)
          else
            value_valid = (value.to_s =~ configuration[:with])
          end
          record.errors.add(attr_name, configuration[:message]) unless value_valid
        end
      end
      
      def validation_block?(validation)
        validation.respond_to?("call") && (validation.arity == 1 || validation.arity == -1)
      end

Using our new validates_format_of could look like:

  validates_format_of :phone, :fax, :with => Mcm::Validations.valid_phone


  module Mcm
    class Validations
      def self.valid_phone
        Proc.new do |number|
          return_value = false
          if !number.nil?
            n_digits = number.scan(/[0-9]/).size
            valid_chars = (number =~ /^[+\/\-() 0-9]+$/)
            return_value = (n_digits > 5 && valid_chars)
          end
          return_value
        end
      end    
    end
  end

An issue with this approach is that the validation proc object may want to determine what the error message should be. What we could do is adopt the convention that if the Proc object returns a string, then that is the error message.

After all this touching of interal Rails method (potentially making us vulnerable to upgrades), we arrive at the safer and simpler approach that Rails provides for custom validations, namely to implement the validate method:

  # In the ActiveRecord class to be validated
  def validate
    Mcm::Validations.validate_phone(self, 'phone', 'fax')
  end

  # In lib
  class Mcm::Validations
    def self.validate_phone(model, *attributes)
      error_message = 'is an invalid phone number, must contain at least 5 digits, only the following characters are allowed: 0-9/-()+'
      attributes.each do |attribute|
        model.errors.add(attribute, error_message) unless Mcm::Validations.valid_phone?(model.send(attribute))
      end
    end    

    def self.valid_phone?(number)
      return true if number.nil?

      n_digits = number.scan(/[0-9]/).size
      valid_chars = (number =~ /^[+\/\-() 0-9]+$/)
      return n_digits > 5 && valid_chars
    end  
  end

  # In config/environemnt.rb (not sure why this is needed)
  require 'validations'

We can make the validate method approach more Rails like by using a mixin instead:

  # In the ActiveRecord class to be validated
  def validate
    validate_phone('phone', 'fax')
  end

module Mcm::Validations
  def validate_phone(*attributes)
    error_message = 'is an invalid phone number, must contain at least 5 digits, only the following characters are allowed: 0-9/-()+'
    attributes.each do |attribute|
      self.errors.add(attribute, error_message) unless valid_phone?(self.send(attribute))
    end
  end    

  def valid_phone?(number)
    return true if number.nil?

    n_digits = number.scan(/[0-9]/).size
    valid_chars = (number =~ /^[+\/\-() 0-9]+$/)
    return n_digits > 5 && valid_chars
  end  
end

class ActiveRecord::Base
  include Mcm::Validations
end

Which looks clean and I'm reasonably happy with this last approach. Finally...

18 comment(s)

2006-08-16

Rails Recipe: A Timezone Aware Datetime Picker

NOTE! Only follow this recipe if you are using an earlier version of Rails than 2.1. As of Rails 2.1 timezone support is built into Rails in a way that makes it much easier to create timezone aware applications.

Quite often in web applications we display dates and times and also have users input them and store them in our databases. If your users are spread across the globe you really want to be able display times in the users own timezone. How do we accomplish this with Rails?

First off, as Scott Baron has pointed out, you want to grab a copy of the TZInfo Ruby timezone library. We need TZInfo since it can deal with daylight savings time (summer/winter hourly adjustments), something that the Timezone library that ships with Rails is not able to do. Installation is as simple as downloading the latest tgz file from RubyForge, extracting it into vendor/tzinfo, and requiring the library from config/environment.rb. We also add some new settings:

# In config/environment.rb
ActiveRecord::Base.default_timezone = :utc # Store all times in the db in UTC
require 'tzinfo/lib/tzinfo' # Use tzinfo library to convert to and from the users timezone
ENV['TZ'] = 'UTC' # This makes Time.now return time in UTC

The strategy we are adopting here is that times in the UI when shown to or entered by the user are in the users own timezone. In the application on the other hand, i.e. in Ruby code and in the database, times are always kept in the UTC timezone. Our job then becomes to allow the user to select a timezone, and then to convert back and forth between this timezone and the UTC timezone as needed. To store the user timezone we can add a time_zone string column to the users table and use the composed_of macro just like Scott Baron describes:

class User < ActiveRecord::Base
  composed_of :tz, :class_name => 'TZInfo::Timezone', 
              :mapping => %w(time_zone time_zone)
end

Then add a timezone select on a "Set Timezone" preference page that the user can access:

<% # Select using TZInfo timezone names such as "Europe - Amsterdam"
   # In the controller on submit you can then do 
   # @user.tz = TZInfo::Timezone.new(params[:user][:timezone_name])
<%= time_zone_select 'user', 'timezone_name', TZInfo::Timezone.all.sort, :model => TZInfo::Timezone %>

If you prefer the timezone names of the Rails Timezone class you can use them instead and then convert to a TZInfo timezone object:

<% # Select using the Rails Timezone names such as "(GMT+01:00) Amsterdam". Will require a
   # conversion to the TZInfo timezone on submit. %>
<%= time_zone_select 'user', 'timezone_name' %>

# Helper method in the controller
def tzinfo_from_timezone(timezone) 
  TZInfo::Timezone.all.each do |tz|
    if tz.current_period.utc_offset.to_i == timezone.utc_offset.to_i
      return tz
    end
  end
  return nil   
end

# On submit in the controller we convert from Rails Timezone to TZInfo timezone via the UTC offset
# and store the user timezone in the database.
@user.tz = tzinfo_from_timezone(TimeZone.new(params[:user][:timezone_name])
@user.save

Now when displaying times in the UI we can consistently convert them to the users timezone with a helper like this:

  def format_datetime(datetime)
    return datetime if !datetime.respond_to?(:strftime)
    datetime = @user.tz.utc_to_local(datetime) if @user
    datetime.strftime("%m-%d-%Y %I:%M %p")
  end

To have the user enter a date and a time in a user friendly fashion you can install the bundled_resource plugin and use its JavaScript based calendar date picker. When displaying the date to the user in an HTML form we use @user.tz.utc_to_local to convert from UTC to the users timezone, and when receiving a date from a form submit we convert back to UTC with @user.tz.local_to_utc:

# The new action:
def new
  @email = BulkEmail.new
  @email.schedule_date = @user.tz.utc_to_local(Time.now) # Default schedule date in local time
end

# new.rhtml
<%= dynarch_datetime_select('email', 'schedule_date', :select_time => true) %>

# The create action that the new form submits to
def create
  @email = BulkEmail.new(params[:email])
  # Convert the local schedule date from the form to UTC time
  @email.schedule_date = @user.tz.local_to_utc(@email.schedule_date)
  if @email.save
    ...
  else
    ...
  end
end

# The edit action
def edit
  @email = BulkEmail.find_by_id_and_group_id(id, session[:group_id])
  # Show scheduled date in local time
  @email.schedule_date = @user.tz.utc_to_local(@email.schedule_date) 
end

# The update action that edit submits to
def update
  @email = BulkEmail.find_by_id_and_group_id(id, session[:group_id])
  @email.attributes = params[:email]
  @email.schedule_date = @user.tz.local_to_utc(@email.schedule_date)

  if @email.save
    ...
  else
    ...
  end  
end

Apparently there is also a Ruby on Rails TZInfo plugin that I discovered only now as I was writing this post and I haven't looked into using it yet. A very helpful page when dealing timezones is the timeanddate.com World Clock.

Testing of the functionality described here is left as an exercise for the reader...

124 comment(s)

2006-08-16

Rails Recipe: CSV Export

A common requirement from customers is the ability to export tabular data to a CSV file that can be imported into Excel. Ruby on Rails uses the standard Ruby CSV library to import test fixtures that are in CSV format. However, there is a runner up CSV library called FasterCSV which as the name suggests is faster and also has a much cleaner API. I was hesitant to use FasterCSV in production since it is still in Beta but after finding the discussion about including FasterCSV in the Ruby standard library I decided to give it a try.

The first step was to download the latest FasterCSV tgz file and extract into RAILS_HOME/vendor/fastercsv. I then added the following line to config/environment.rb:

require 'fastercsv/lib/faster_csv'

The action that exports the CSV data looks roughly like the following:

  def csv
    @list = @group.lists.find(params[:id])

    csv_string = FasterCSV.generate do |csv|
      csv << ["Type", "Name", ... more attribute names here ... ]

      @list.records.each do |record|
        csv << [record['type'],
                record['name'],
                ... more values here ...]
      end
    end

    filename = @list.name.downcase.gsub(/[^0-9a-z]/, "_") + ".csv"
    send_data(csv_string,
      :type => 'text/csv; charset=utf-8; header=present',
      :filename => filename)
  end

In the corresponding controller test I check that the CSV response can be parsed with the other CSV parser, that there is a header row with expected attributes, that the number of data rows is correct, and finally that one of the rows has the correct values:

  def test_csv
    get_success :csv, :id => lists(:peterlist).id

    reader = CSV::Reader.create(@response.body)

    header = reader.shift
    ['Type', 'Name', ... more attributes here ...].each do |attribute|
      assert header.include?(attribute)
    end

    rows = []
    reader.each do |row|
      rows << row
    end
    
    assert_equal rows.size, lists(:peterlist).size
    item = lists(:peterlist).items.first
    item_row = rows.select { |row| row[1] == item.contact.name }.first
    assert_equal item_row[0], 'Contact'
    assert_equal item_row[1], item.name
    ... assertions for the other attribute values here ...
  end

To see what CSV export from a controller looks like with the Ruby CSV library - check out this blog post.

15 comment(s)

2006-07-18

This Might Cost You a Little...

I just received an invoice from my dentist for 900 SEK (about a 100 EUR) for some unspecified service. I called them and it turned out I was paying 900 SEK for being 15 minutes late. What happened was I over slept and four minutes into the appointment when I was rushing out of my apartment to get on my bike my dentist calls (from Folktandvården, Eastmaninstitutet). I apologize and tell her I can be at her clinic in 7 minutes on my bike. She says nah, I'm not sure it's worth you coming in now that you're late, we better schedule a new time for you. I say ok, given that I didn't seem to have a choice. We book a new time and then at the end of the conversation she mentions that "oh, this might cost you a little" (the exact words in swedish were "det här kommer kanske att kosta lite").

Interesting points of comparison here is that once before when I missed an apointment I paid 300 SEK. My mom says she has never had to pay for missed appointments at her dentist. Last time I visited the dentist and they examined my teeth I paid 500 SEK, which right now seems pretty cheap, especially given that that time they actually performed a service for me.

2006-07-04

The Weepies Make me Weep with Joy

I just came home from a concert with The Weepies (a folk/pop duo from the US) at Debaser here in an exceptionally warm and sunny Stockholm.

The Weepies made me all misty eyed, showering beautiful harmonies of vocals and guitars over a small audience in an intimate concert setting. I loved the fact that about half of the songs were new to me since they were drawn from the solo CDs of Deb Talan and Steve Tannen that preceeded The Weepies. They also played a new song about an "Old Coyote" to be released soon on Itunes...

I wish there would have been more people to cheer Deb and Steve and Meghan Toohey (guitar) on. The band was so humble and appreciative of the few of us who had shown up though that it didn't seem to be a problem. Afterwards I stayed around and chatted a bit with Deb, Steve, and Meg and they are absolutely lovely, warm, and down to earth people.

I discovered The Weepies by chance last christmas holiday thanks to a "Staff Favorites" recommendation in Itunes Music Store. In a way The Weepies made me rediscover and re-appreciate pop music, and that's no mean feat. Over the last years I've been so busy listening to and dancing to salsa music (and some jazz and tango on the side) that I haven't been able to keep up with mainstream pop much.

Do check out The Weepies on Itunes Music Store and don't miss the cute story of how they met. It's almost too perfect to be true.

The Weepies will make it big. I just know it. Next time they play in Stockholm they will probably be way too popular for me to be able to go and talk to them after the concert, unfortunately. Thanks for making it all the way here to play - and welcome back!

2006-06-07

Rails Code Generation Considered Useful

I just checked out Ryan Daigle's excellent What's New in Rails blog. There are a lot of great findings there for anyone like me who hasn't monitored closely the changes to the Ruby on Rails source code. In the post about the observer generator Ryan writes:

"On a side note, I feel like the generator utility has been getting a bad rap recently from people that frown upon auto-generation of code. Fair enough, but think of them more as best practices templates and learning tools rather than some sort of coding robot. They're really great tools for seeing how things should be done, and how things are expected to be done. If you don't want to use them, don't - but if you're not quite sure how to go about creating an observer, or controller, or model they're of great help."

Ryan puts it very well and I couldn't agree more. I've found the scaffold generator very useful in the project I'm working on now, and it's not for prototyping I'm using it, but for providing a skeleton for the production code I'm going to write.

Another advantage of the code generation in Rails is that if you use it your code will probably follow conventions (naming of controller actions for example) more closely and this will make the code easier to read and maintain for other developers. The generated code also contains best practices (such as the verify :method => :post for certain actions in controllers) that an average Rails programmer (or even a good Rails programmer) may not know about or may forget to use otherwise.

2006-05-10

Presentations from the Rails Recipes Meetup

The Rails Recipes Meetup yesterday was a great success with over 20 participants and four interesting presentations. I would like to thank Xlent for hosting us and all the people who showed up and contributed to making the evening so nice.

If you are interested in coming to the next meetup (scheduled to happen in June or July), please signup for the swedish Rails mailing list at rails.se. Please also suggest presentation topics on that mailing list.

Here are two of the presentations from the meetup:

See you at the next meetup!

1 comment(s)

2006-04-29

Ruby on Rails Tips and Tricks

I've started a wiki page with some Ruby on Rails tips and tricks that I pick up from various weblogs and books. Hope it might be useful to others as well.
2006-04-26

Starting a Startup - Hard but Doable

Here are some encouraging words from Paul Graham's How to Start a Startup article (which will make you scroll through eternity):

"You need three things to create a successful startup: to start with good people, to make something customers actually want, and to spend as little money as possible. Most startups that fail do it because they fail at one of these. A startup that does all three will probably succeed.

And that's kind of exciting, when you think about it, because all three are doable. Hard, but doable. And since a startup that succeeds ordinarily makes its founders rich, that implies getting rich is doable too. Hard, but doable."

Interesting to see the Web 2.0 Startup Toplist as well for a list of successful startups.

From Paul Graham's Y Combinator:

"Y Combinator relies on certain premises: that open-source software and falling hardware prices means that tech start-ups are cheap to finance; that large companies are no longer at the forefront of innovation; and that mature technology companies find it cheaper to buy than to build."

2006-04-26

Rails Recipes Meetup in Stockholm

Each presentation is up to 20 minutes long followed by a 10 minute discussion. The presentations should be in the Rails tradition of "show don't tell". We prefer short presentations with code samples from recipes applied to real problems over the traditional type lengthy and theoretical Powerpoint presentations.

"Now, I know Rails well enough that over time I may have stumbled on some of the solutions, but that's not the point. With a collection of great recipes at hand - In-Place Form Editing, Live Search, RJS Templates, Polymorphic Associations, Tagging, Syndicate Your Site with RSS, and Sending Email with Attachments to name a few - I can quickly give my app the functionality today's web users have come to expect, then move on to the features that really set it apart from the crowd."

2006-04-07

Quote of the Day

Thanks Jarkko for highlighting the following nice quote from the article Two more reasons why so many tech docs suck which argues the case of getting your users excited and using use cases for technical documentation:

One of the best techniques for creating and keeping interest is to make non-reference docs use-case driven. That way, each topic is framed by a context that matches something the user really wants to do. That way, each little sub-topic shows up just-in-time, instead of appearing to be there just-in-case.

It makes a lot of sense to me to organize technical documentation by the readers goals, i.e. to use the howto or cookbook approach. The Perl Cookbook is a great example of how this approach can be successful. In that instance the use-cases are on a very low level such as "How do I iterate over an array" but the approach should also work on a higher level, i.e. "How do I setup a weblog". High level use-cases like that are probably best broken down into smaller use-cases.

2006-03-31

Quote of the Day

I was catching up with a lot of Rails related posts in my RSS reader and found DHH's bashing of James McGovern which was pretty entertaining. I also like this quote in the comments:

First they ignore you
Then they laugh at you
Then they compete with you
Then you win

I also learned today that Dreamhost upgraded to Rails 1.1. Apparently my homepage survived that upgrade. I still don't have Rails in the vendor directory but I do have web tests set up so I don't have to worry too much.

2006-03-21

Rails Meetup @ Akkurat

The Akkurat pub

Tonight a bunch of hardcore geeks here in Stockholm met up at the excellent Akkurat pub to chat about Ruby and Rails. More than Rails though, the conversation seemed to drift towards ABC 80, Atari, and Amiga nostalgia which was great fun. I am very positively surprised by the atmosphere and service at Akkurat by the way and I now consider it to be my favorite pub here in Stockholm. The next time I need to try their special "Heaven" and "Hell" beers.

I've thoroughly enjoyed both the Rails meetups that we've had so far. The first one was held at The Dancing Dingo, the second one tonight at Akkurat, and the third one we are planning to have in a couple of weeks at Saddle and Sabre. To signup for notifications about future meetups in Stockholm make sure to get on the swedish Rails mailing list which you'll find at rails.se.

2006-03-19

Sunday Brunch with a Touch of Flair

Stylish lobby of Hotel Amaranten

I enjoyed an exotic brunch today with my girlfriend at the flairful Amaranten hotel here on Kugsholmen in Stockholm. As you can see, the hotel lobby and restaurant is stylish and has a nice ambience. The buffet is offered at 160 SEK and is high quality with a good selection of meat such as chicken wings, spare ribs, bacon, and a number of more exotic dishes that I don't even know the name of :-) Some of the Cajun specialties didn't particularly speak to our taste but all in all it was quite good. On the minus side was the fact that the place was almost empty and one of the waitresses was not so service minded.

The atmosphere was enhanced by a house Jazz band - a trio on bass, drums, and piano. They performed professionally albeit somewhat uninspired, which is understandable given the lack of an engaged audience. We tried to cheer the band on by taking the initiative to applaud and I think this was appreciated.

I seem to have taken up an interest in luxury hotels lately - there is something fascinating about them. A couple of years ago I stayed with my girlfriend at the Grand hotel here in Stockholm - a very memorable and highly recommended experience. Recently I read in the paper about how the new Hotel Rival (owned by famed Benny Anderson of Abba) in Stockholm came in highest in scandinavia (at place ninety something) in a listing of the most luxurious hotels in the world. This sparked my interest and I have now booked a brunch there for the second of April and will spend the night with my girlfriend in one of their "de luxe" rooms. It's no exaggeration to say that I look forward to that...

2006-03-07

Less is More - The 37Signals Mantra

I remember seeing a presentation by Jason Fried of 37Signals at Reboot in Copenhagen a couple of years ago. The presentation was about defensive web design and it was very professional and convincing. It seems now though, Jason has become not only an authority on web design but also on the general process of how to successfully build websites. I listened to Jason's interview at IT conversations a while back and it really impressed me. I felt like listening to the speech several times and write down all the important points that Jason is making. However, 37Signals beat me to that task and for only $19 you can have the Getting Real book which essentially covers and expands on the same philosophy and guidelines that are in the speech. The book notwithstanding, here are my own sketchy notes from the IT conversations interview:

Reduce mass. Physics tells us that the higher the mass of a moving object the more difficult it is to change its direction. Mass can be too many people, hardware, software lockin, and long term contracts. Ability to change is very important in any product development.

Embrace constraints. Examples of constraints are hardware, people, and money. Constraints is where creativity happens. Let people use the product and look for real patterns. Start with less and your more with be a lot more clear down the road. The initial release should be half the product. Everything is beta these days but its time to have confidence, public is public. Keep iterating and make what you have rock solid, bullet proof, and great

Getting real. Use few documents, specs and diagrams. Build top down - build the interface first.

Managing debt. Feel the hurt. The people who built the product should support it. Chefs become waiters once in a while.

Finding people. Look for people who are:

2006-03-07

My First Acoustic Instrument

I just had to post a picture of my nice new LP Bongo drums:

My Bongo Drums

It was in Copenhagen that I regularly started dancing salsa and through the dancing I developed an interest in the music. On the salsa mailing list in Copenhagen there came an invitation from Jakob Johansen to learn how to play in a salsa band. As a child I always wanted to play the drums but my dad wasn't happy about the idea of having drums in the house so that dream never materialized. However, in Copenhagen I jumped on the opportunity and by chance I ended up playing Timbales in the band. I was the only one in the band who was a total beginner but it worked out ok anyway and I throughly enjoyed it.

Once in Stockholm last year I was looking for the opportunity to continue playing in a salsa band. I found Pygge and Alternativa Musikskolan in Sollentuna where I took private lessons in Timbales. I decided to switch to conga drumming and pursued that for a while. Eventually I found a salsa band that I could play in. However, it turned out they alreay had a conga drummer so I had to learn the bongos instead. I find bongos easier to play than the congas and they are great for soloing. I think my new bongo drums look great - now I just have to learn how to play them...

2 comment(s)

2006-01-19

Financial Freedom vs Liking What you Do

Paul Graham writes that:

"Most people would say, I'd take that problem. Give me a million dollars and I'll figure out what to do. But it's harder than it looks. Constraints give your life shape. Remove them and most people have no idea what to do: look at what happens to those who win lotteries or inherit money. Much as everyone thinks they want financial security, the happiest people are not those who have it, but those who like what they do. So a plan that promises freedom at the expense of knowing what to do with it may not be as good as it seems."

Whichever route you take, expect a struggle. Finding work you love is very difficult. Most people fail. Even if you succeed, it's rare to be free to work on what you want till your thirties or forties. But if you have the destination in sight you'll be more likely to arrive at it. If you know you can love work, you're in the home stretch, and if you know what work you love, you're practically there.

2006-01-01

Lessons from Adam Bosworth

Adam Bosworth's article Learning from THE WEB is well worth reading (thanks Branimir for pointing me to it). A surprising number of the lessons have to do with simplicity, for example the KISS one:

"KISS. Keep it (the design) simple and stupid. Complex systems tend to fail. They are hard to tune. They tend not to scale as well. They require smarter people to keep the wheels on the road. In short, they are a pain in the you-know-what. Conversely, simple systems tend to be easy to tune and debug and tend to fail less and scale better and are usually easier to operate. This isn't news. As I've argued before, spreadsheets and SQL and PHP all succeeded precisely because they are simple and stupid - and forgiving."

Other lessons such as "It is acceptable to be stale much of the time" and "The wisdom of crowds works amazingly well" ring very true to me. His critique of database vendors inability to learn from the web is thought provoking.

1 comment(s)

2005-12-16

How I Test my Website

Introduction

This is an article that talks about the motivation for testing and how I went about writing automated tests for my personal website. It is best suited for someone with at least some familiarity with Rails but who hasn't yet explored its testing capabilities. There is also a section on web testing that is independent of Rails.

I learned my lesson in Berlin

I still remember clearly the somewhat embarrassing mistake that sparked my interest in software testing. I was working as an intern for a research institute in Berlin and my task was to program an image processing algorithm in C. Once the program was finished I was stalled for a long time, maybe weeks, puzzled over the unexpected outputs of my program. My supervisor was starting to lose patience with my lack of progress. Eventually it occurred to me that the underlying algorithm that I had implemented was broken due to a trivial mistake. All along I had simply assumed that it was working without conducting any thorough tests to prove my assumption. That's when it dawned on me what a challenging and important problem software testing is.

The miserable past of web testing

A lot of developers see testing as a dull or even unnecessary activity. The level of testing has been particularly poor in web development, a sector of the software industry notorious for being hackish and low in quality. Granted, the cost of a broken web page is typically not as big as a bug in banking software. However, a broken page on a high volume public site or an internet banking site can be very costly indeed.

With Agile methodologies such as Extreme Programming the idea is to make changes, refactor, and release often. Good automated test coverage is almost a prerequisite for this style of development. This is because for each change that we make we need to make sure that we didn't break existing functionality. In a system with high coupling the bug caused by a change can arise in a seemingly unrelated part of the system. Of course, manual testing is not really an option in agile development as it would be too expensive and slow down the development and release cycle too much.

There are benefits to testing other than just finding bugs that are worth mentioning. One is that testing forces you to focus on and define the interface (aka contract) of your application or API. This is the reason why test driven development can be so valuable since defining the contract of your code early on provides a good direction and goal for your programming - it serves as a specification. Also, when you are writing tests you will often find yourself reviewing your code and this review in and of itself can reveal valuable improvements.

Rails shows the way

Most of the hype surrounding Rails has to do with its productivity and the surprisingly few lines of code needed to build a typical website. A nice side effect of having a small code base is that you have fewer lines of code to debug. Testing in Rails is also helped by its strict MVC (Model-View-Controller) structure. Last but not least, Rails comes shipped with a tightly integrated testing framework that allows testing both on the controller and model levels.

Rails has an interesting code statistics tool invoked by "rake stats" that counts the lines of code (LOC) and the ratio of production to test code. Here is the output for my website:

+----------------------+-------+-------+---------+---------+-----+-------+
| Name                 | Lines |   LOC | Classes | Methods | M/C | LOC/M |
+----------------------+-------+-------+---------+---------+-----+-------+
| Helpers              |    19 |    18 |       0 |       1 |   0 |    16 |
| Controllers          |   260 |   210 |       6 |      37 |   6 |     3 |
| Components           |     0 |     0 |       0 |       0 |   0 |     0 |
|   Functional tests   |   340 |   253 |      10 |      45 |   4 |     3 |
| Models               |   122 |    91 |       4 |      12 |   3 |     5 |
|   Unit tests         |   191 |   150 |       3 |      17 |   5 |     6 |
| Libraries            |     0 |     0 |       0 |       0 |   0 |     0 |
+----------------------+-------+-------+---------+---------+-----+-------+
| Total                |   932 |   722 |      23 |     112 |   4 |     4 |
+----------------------+-------+-------+---------+---------+-----+-------+
  Code LOC: 319     Test LOC: 403     Code to Test Ratio: 1:1.3

Unit testing

A unit test in Rails tests a single model class. For my website I have three model classes - Post, Category, and User. There is a one-to-one mapping in Rails between model classes and database tables. The model classes have methods for the usual CRUD (create, read, update, delete) operations and all of these operations interact with the database.

When you write tests that depend on database data it will simplify your life a lot if the tests have control over that data. This is one of the areas where the Rails testing framework shines. All Rails tests are executed against a separate test database into which data is loaded with the help of fixtures. Each database table has a corresponding fixture YAML file. YAML is a concise and yet readable text data format that is quite handy to work with. The YAML files can contain embedded ruby code and so can be dynamic with loops and variables.

Here are some examples of what my unit tests cover:

Functional testing

In Rails functional tests are applied at the controller level. Each controller class has a corresponding controller test class. The controller test executes actions on its controller, makes assertions about the values of the member variables in the controller, about the type of response (success, redirect etc.), and about the HTML output generated by the view. The controller test also has access to the request and response objects, to the session, and to the flash object used to display status messages to the user. All in all this makes controller tests very powerful.

Here are some functional tests I wrote for my website:

One minor limitation of controller tests to be aware of is that they don't include the URL-action mappings (the routes). One could probably test those separately somehow but I haven't seen an example of that yet. The URL mappings are covered by the web tests though, see below.

There is potential for overlap between unit and functional testing. Thorough testing at the model level decreases (but does not eliminate) the need for testing at the controller level. It is clearly an advantage if the controller tests can rely on the underlying model to be working correctly so that they can focus on the controller specific logic.

Web testing and Monitoring

I use web testing for pre-launch and post-launch acceptance testing and for monitoring of my website. This means I run the web tests just before a site upgrade, immediately after the upgrade, and then regularly every day against the production site. The web tests execute the entire system including the production database. The fact that the production database is used is important since it is not uncommon for bugs to depend on database data. The Rails unit and functional tests can provide good test coverage but the web tests are a necessary complement and are the ultimate check that the system is working at the point of release and that it keeps working in production.

I have divided my web tests into two parts. I have one "ping" test that I run frequently and that just makes sure that the front page works and the site is up. The other test is a smoke test that I run twice a day. For all pages on my site it checks that:

If any of the web tests fail I am alerted by email. I am also alerted by email if an error occurs in the code (i.e. an exception is thrown). With this setup I can rest assured that if major functionality on the site breaks down I will know about it and have a good chance of taking action quickly.

Web testing is not provided by the Rails framework and I have chosen to rely on Perl scripts for those tests. My web tests currently don't do much HTML parsing, but if there was a need for that Perl has its simple yet powerful look_down method. However, if there were similar APIs available in Ruby I would probably switch to Ruby as I am more comfortable with that language.

What's Missing?

There should be some kind of automatic test that verifies that site backups are working. A way to do that would be to have a mirror site in a different location that is updated nightly with the database backup from the production site. One could then run the web tests against the mirror site as well. One would also need to verify, through a web test or otherwise, that no data is missing in the backup. If the production site is not changed during the recreation of the mirror site a sufficient test would be to check if the two sites are identical.

If a website makes heavy use of JavaScript and Ajax then an in-browser testing tool such as Selenium may be useful for acceptance testing and browser compatibility testing.

Last, but not least, if you are expecting a lot of traffic you should look at load testing with tools such as Apache Bench (ab), or siege. Basic usage of these tools is shown in the Rails book. Rails also has the concept of performance tests that make it easy to load huge quantities of data and then repeatedly execute some critical action and measure its average execution time.

Conclusion

The level of testing that I have described here isn't comprehensive and really ought to serve as a baseline for pretty much any serious website. Unfortunately, this is far from the case in the industry today. With the advent of Rails, testing is more readily available to web developers than ever before. Let's remind ourselves to take full advantage of that.

1 comment(s)

2005-12-06

The Threat of New Technologies

In the midst of one of the many Java/Rails flamewars (haven't we had enough of them already?) I found a thought provoking comment by the creator of Tapestry on Rails and the resistance to technology change among developers:

"It's natural to be a little scared of new developments. It's painful to think that your exclusive, hard-won skills may be even a little bit obsolete. When I see Flex, Laszlo or Rails, my stomach does a little flip-flop in terms of all the effort I've put into Tapestry. But the reality is that *all* of these approaches are transitory (I'm sure we'll all be doing something quite different in five years), and we should embrace change, learn from other's efforts, and create even better frameworks and applications."

"For example, there may come a time when much Tapestry work is done using Trails (a Tapestry/Rails/Spring/Hibernate hybrid). Trails may never have offer quite the same developer productivity as Rails, but it is still a gateway into full-blown Tapestry and J2EE development."

Openness to new programming languages and toolkits seems to be a common characteristic among leaders in the programming community. The best example of this is probably the Pragmatic Programmers who came up with the now famous advice to learn one new programming language a year. I can't help but wonder that maybe if Dave Thomas hadn't picked up and documented Ruby, David Heinemeier Hansen would not have discovered it either and Rails would not have been a reality today.

I have been impressed that so many Java heavy weights (David Geary/JSF, James Duncan Davidson/Ant, Bruce Tate etc.) have had the courage to step up and recommend Rails. Personally I'm certainly ready and well equipped to make the transition to Rails and the urge to do so is bigger than ever now that I am back working in the less productive and overly complicated Java world.

2005-11-27

I'm on Rails Now

I was finally able to put my new homepage live late last night after too many hours of programming (why do things always take longer than we expect?). About two weeks ago I caught a cold and I spent several days alone in my apartment. The upside of being sick is that I've had time to work very intensely on my homepage (when I probably should have been resting)...

This is the third incarnation of my homepage. A couple of years ago I started out with an OpenACS site using Lars Pind's weblog software. Since OpenACS hosting is hard to come by and that technology had little momentum I moved to dreamhost.com and Wordpress which both have worked out very, very nicely for me. At that point I didn't know the Rails revolution was coming...

When I was working for Collaboraid with Lars last year we had David Heinemeier Hansson (the creator of Rails) come visit us in our office and tell us about his toolkit. I remember how we were shooting all kinds of tough questions at him of the type "ok, that sounds good, but can Rails do this or that particular thing that we need". David exuded confidence and would patiently and immediately answer each and every one of our questions like "sure you can do that, or in fact, you can do something even better, let me show you...". We were all very convinced. At that time though, I didn't have the time or energy to pick up a new web technology and I was somewhat deterred by the fact that Rails was still such a small niche. Having lived in the OpenACS niche for so long I was very impatient to find my way back into the mainstream.

Rails stayed in the back of my mind though and when I came to Stockholm to work as a Java developer at Pricerunner I noticed how people were talking about Rails. I then realized just how big the technology must have gotten. Over the past months I've read countless articles about Rails and it's really mind boggling to see how Rails had litterally taken the world by storm. I didn't see that coming, at least not at the rate that it's been happening. There is a myriad of open source web toolkits out there competing for attention and people like me who have worked with OpenACS know how hard it is to get a toolkit using an unpopular programming language into the mainstream (there are other more compelling reasons why OpenACS never entered the mainstream, but that's a different story).

When I had realized that the Rails way is really the future of web development, and also had experienced first hand how wonderful the Ruby language is, I knew I had to learn these technologies. I figured that what better way to do that than to build my own homepage using them? I started by trying Typo - the Rails weblogging software that all the other cool Rails hackers seem to be using. However, at the time that I tested it I was kind of put off by some bugs that it had. It seemed awfully immature coming from Wordpress. Also, Rails is not a framework for plugging in and reusing applications (like OpenACS is). Rather, it is a framework for building applications from scratch in a very fast and elegant fashion.

Of course, building any application, even if it's something seemingly simple as a weblog, is going to take a lot of work and thought if you want the result to be top class. What's fascinating is that Rails makes it feasible (because of its productivity) to build from scratch in a lot of cases where previously application reuse was the only realistic option.

I'm really thrilled about web development again know that I know Rails. I'll write up some details about how I built my homepage later on.

My old homepage is still accessible and now lives at old.marklunds.com.

1 comment(s)

2005-11-25

Java is not Always the Best Tool? Really?

I really like the article Ruby the Rival over at OnLamp.com, and especially this part by James Duncan Davidson (this has been said many times before - when all you have is a hammer everything looks like a nail):

"And I think that's the real win from the recent attention on Ruby on Rails and the breakaway from viewing the world with Java-colored glasses. It's not that Ruby on Rails is going to be the next Java. Far from it. It's that Ruby on Rails helps to break this idea that there is "One True Way." There's not. There are many different ways to solve a problem. And really, none of them is the clear-cut winner. There's just places where one solution has advantages."

» Next page