The last few months have been pretty exciting for BoxCycle. We are seeing more visits, more listings, and more orders and we are refining the process to help us keep up and provide the best possible experience for buyers and sellers.

Unfortunately, the last few months have also been maddening. Just as traffic improved, website performance became slow and unreliable. We tried a number of things to improve speed, but, ultimately, the problem came down to deteriorating performance of our web host, HostingRails. HostingRails has worked quite well when we setup BoxCycle, but, since being taken over by JumpLine in 2010, seems to have lost their Rails expertise and their performance focus. As time passed, the site performed poorer and poorer and JumpLine’s response gave little hope that it would get better.

Switching providers is always scary, but in mid-May we pulled the plug and moved over to WebFaction which is known for its modern setup, less loaded servers, and a focus on performance. The transition was nerve-racking, but went through without a hitch.

We also implemented a number of improvements, both before and after the transition, to help with performance:

• Switched from Apache/Mongrel to Nginx/Passenger
• Implemented many server-side best practices such as GZip
• Added Expires headers to static assets and made sure that they are served by Nginx without touching Rails
• Implemented Rails fragment caching on some heavily used components
• Modified deployment so last modified dates are only updated on changed files
• Upgraded to the latest version of the Twilio API allowing us to setup Failover for outgoing calls to better deal with server timeouts
• Started using asset_host in Rails to split requests for static files between several domains
• Updated BoxCycle to heed more of the recommendations made by tools like YSlow and Google PageSpeed
• Upgraded Rails to 2.3.11 (made possible by switching to Passenger since Mongrel has problems above 2.3.5 and we don’t yet have a good reason to migrate to Rails 3)

There are certainly more things we can do, but fundamentally, we aren’t yet at a point where optimizing performance should be our highest priority. There is plenty of work on our plate that more directly impacts people’s ability to buy and sell used boxes.

Unfortunately, we ended up on a WebFaction server that was under quite a bit of load and despite all the work we were stuck with a less intense version of the same problem. Talk about frustrating. To their credit, WebFaction was prompt with action and communication and worked with us to identify and remove issues causing load. After a few weeks of effort, things improved, but were still not acceptable. We requested to be transferred to another server. Again, to their credit, WebFaction was very accommodating and setup our account on a server with lowest average load.

So once again, we rolled up our sleeves and transferred servers. Transferring within the same host is easier, and, since WebFaction supports multi-server setups, it is easier still. But it was still nerve-racking.

The transition is now over and the difference is spectacular. BoxCycle feels super fast. I, for one, am just relieved to have this issue over with.

BoxCycle is a Rails application that just switched to Git from SVN for version control and uses Capistrano to help automate deployment.

Problem: Deploy Updates Last Modified Dates on All Files

As shocking as it may seem, when you deploy a new version of your app, there is a good chance that Last Modified dates of all files will be updated to the date of deployment even though most of those files did not change.

Why Are Last Modified Dates Important?

In a word, caching.

Last Modified dates are used for Last-Modified and Etag headers which are used by the server to tell the browser if it can use the file already stored in the cache or needs to download the file again. They are also used by Rails for generating cache-busters – the query parameter it adds to images, javascript, and css. Cache-busters allow you to set Expires headers far in the future for files that don’t change often without worrying that you won’t be able to to update them should they change.

Expires headers are crucial since they allow the browser to use files in the cache without even asking the server if the files are still current (like it does with Last-Modified or ETag headers). Their importance is clear with Rails enabling cache-busters by default even though Expires headers require server configuration and won’t be enabled by many users. Rails cache-busters carry a decent performance hit since they must access the file system to get the modified date, but are still enabled by default since making setting Expires headers easier is deemed worth it.

But many default deployment processes will result in Last Modified dates being updated for unchanged files which means that users’ browser have to reload all static assets after every deploy significantly reducing the benefit of Expires headers as well as Last-Modified and Etag headers

Cause: Commands like cp, checkout, and clone Modify Dates

Odds are that your deployment process copies files or uses git clone or svn checkout. If it does, modified dates of all files will be updated.

cp offers options to retain modified dates. From everything I could find, SVN does not store modified dates in the repository and so there is no simple way of maintaining dates after you commit. Git seems to retain modified dates and this was the reason we switched to it, hoping this would solve this problem. While it may retain the dates, it did not solve the problem, since git clone sets new file dates. There may be a way to configure git clone to not overwrite the modified dates, but I didn’t find it before finding another solution.

Solution: Use :deploy_via, :remote_cache and set :normalize_asset_timestamps, false

In Capistrano we originally used :deploy_via, :checkout. With :remote_cache, Capistrano loads the latest version into a shared/cached-copy directory and fetches only the updates which allows the dates of unmodified files to remain on subsequent deploys. :remote_cache definitely works with Git and will probably work with SVN.

You may still have a problem since Capistrano may set all the dates for you. To make sure this doesn’t happen set :normalize_asset_timestamps, false in your deploy file.

Since the last post, most of the updates to BoxCycle have been small usability improvements and internal process improvements.  The most interesting customer facing feature we’ve added is A+ Sellers.  I have a well-developed idea of a rating system BoxCycle will use in the future to relay seller performance to buyers, giving them additional information relevant during order placement.  However, I realized that this system, although important, is probably not the highest priority.

At this time, we can achieve much of the benefit by simply adding a designation for our best sellers – this is exactly what A+ accomplishes.  It highlights sellers who’ve processed many orders reliably, confirmed appointments quickly, and otherwise dealt with customers in an exemplary manner.  The designation should go a long way in telling customers concerned with uncertainty who their best bet is and, as a result, rewarding high seller performance.

Time Zone Support

I am very excited about the update released today.  BoxCycle now has full time zone support.  To end users used to dealing with a single time zone this may seem like a non-event.  Programmers who’ve never dealt with time zones may think it’s trivial.  But time zones are more difficult to deal with than it first appears and have a been a constant barrier, requiring ugly workarounds that never fully solve the problem.

The scary part about this change is that it’s so all-encompassing.  Although visually you’ll see only minor changes, nearly every section of our site might be impacted and every date in our database had to be updated.  I don’t doubt we’ll run into issues, but this was a required step in the site’s development and I am very happy to have the bulk of the work behind us.  A number of annoying, odd behaviors regarding time were immediately eliminated and several exciting features on the roadmap are now a big step closer to reality.  Woohoo!

Hopefully, this will save some Rails programmers a bit of searching.

Permission denied – activesupport-2.3.2 or activesupport error when trying to run rake rails:freeze:gems?  For solution refer to http://www.ruby-forum.com/topic/184589 . 

In a nutshell, you need to edit line 28 of rubylibrubygems1.8gemsrails-2.3.2libtasksframework.rake to include sleep(5) between the Gem::GemRunner line and the mv line.

I haven’t done a programming related blog entry until this one.  Why?  First, I want to keep this blog focused on BoxCycle and environmental issues.  Second, I haven’t had much time, as is evident from lack of posts on environmental issues.  Finally, I haven’t run into much that I couldn’t find an answer to with a few searches and I am not fond of duplicating effort just to get a post up.  This time I think I have something to contribute to the Ruby on Rails community.

The issue is updating plugins.  Initially, I simply did ‘plugin install’ and checked them into my svn repository not thinking much of it.  My guess is, this is what most beginners to Subversion do.  In passing I saw things about svn:externals, but figured it was advanced stuff and I’d sort it out later.

When it came time to update some of the plugins I did ‘plugin install’ again, and didn’t think much of it, again.  That is, until I went to checkin the project.  Subversion refused.  It took me a while to figure out what’s going on and a lot of trial and error to be able to check in the project at all.  I now know exactly how to do it:

1. Delete the directory containing the plugin
2. Checkin changes
3. Install the new plugin
4. Checkin changes

Quite annoying.  Even worse if you made changes to the plugin code.  Why is this necessary?

Subversion keeps state in hidden .svn directories in your local copy.  It can’t checkin directories that are missing .svn folders because it can’t figure out their state.  ‘plugin install’ removes the directory tree when it reinstalls the plugin.  The only thing you can do is first checkin a version without the directory and then add the directory back in the next commit.  Some good links on the topic: http://railsforum.com/viewtopic.php?id=24882 and http://stackoverflow.com/questions/119006/what-should-i-do-with-the-vendor-directory-with-respect-to-subversion .

Once I figured out the issue, I understood svn:externals and found tools like Piston to make management easier.  However, by then, most of the Rails community has moved to Git version control which doesn’t work well with svn:externals or Piston.  Piston has been promising to make a Git compatible version, but for much of BoxCycle’s existence it hasn’t occurred.  Plugin updates have been a huge thorn in my side.

With large changes to Rails 2.3.2, most plugins needed to be updated.  I was very excited to see that Piston 2.0.2 has finally been released and ‘gem install piston’ actually installed the right version, which has not been the case for me with 1.9.x releases.  However, I did run into installation issues on Windows which took me a while to resolve.  Sharing the solution is my primary motivation for this post.

So keep the following in mind when you install Piston 2.0.2 on a Windows XP machine (I presume Vista is similar):

• After ‘gem install piston’ type in the ‘piston’ command.  If you get errors, read them carefully.  Dependencies don’t seem to be well defined for the Piston gem and I had to install a number of gems separately before the ‘piston’ command displayed the usage help screen.
• You need to have Git installed for Piston to work with Git repositories.  I installed  MSysGit . 
• With Windows you are likely using InstantRails which runs in it’s own isolated environment.  This means that Piston will not be aware that MSysGit is installed.  You need to update the InstantRails PATH to include Git.  Edit c:InstantRailsuse_ruby.cmd to have the PATH line include c:Program FilesGitcmd (or whatever your path to Git is).   Also update the use_ruby.cmd template in conf_files directory.

Once I got Piston working I deleted existing plugins, hopefully for the last time, and did a checkin.  Then I used ‘piston import’ to install new versions under Piston’s management.  I am quite relieved to not have to dread updating plugins again!