interprogramming articles

Code Transition to Github

2012-07-19T16:11:00Z

I’ve finally taken time to migrate all my public code bases over to github. Effective immediately, the subversion repository is deprecated and will likely disappear soon.

Links to everything can be found on the projects page.

New and shiny site design

2012-07-14T22:30:00Z

Once again it has become time to refresh the site design. Looking back, it appears the last site refresh was in July of 2007—I’m not sure if 5 years on the same design demonstrates longevity and consistency, or if it just means I’m horribly overdue.

Regardless, now everything is nice and shiny. One of the goals of the new design was to do my best to leverage HTML5 and CSS3, including new HTML5 tags and use of many of the newer CSS features. As a consequence, the only image file is the background. Everything else is CSS effects along with a couple CSS fonts.

I also used the refresh as an excuse to experiment with responsive design. I haven’t had a chance to test a wide array of small-screen devices yet, but on the ones I have access to (various iDevices), the site is usable. I’d love to hear your experience, good or bad, on any kind of smaller-screen device.

Other changes include numerous updates to the software stack (Rails 3.2, etc.), better syntax highlighting (now with line numbers!), and improved (in theory) spam filtering.

Catching up

2012-06-09T00:29:00Z

It’s clearly time that I catch up with the world around me. It’s curious how much can pass you by while you’re consumed with other priorities. Now that some of those have passed, I’m rediscovering several things that got pushed aside.

Among them is hopefully writing at least a little more often here. Apparently, that won’t be a difficult bar to jump over—it’s been 2 1/2 years since my last post. Ack!

Apart from writing here, the process of catching up looks like it will include a bunch of software upgrades (Rails 3 a part of that), a new design for here, and migrating a bunch of stuff to git.

Stay tuned.

Faster alternatives to ActiveRecord::Base.to_xml (Rails Performance Series)

2009-11-13T16:09:00Z

This is part 5 of my ongoing series on Ruby on Rails performance. Today’s topic is a bit more complicated that some of the previous parts.

I’m going to explore an alternative to using the method ActiveRecord::Base.to_xml.

Before I get into that, I probably should explain why. After all, ActiveRecord::Base.to_xml is really easy to use. The alternative I’m going to demonstrate isn’t very easy to use.

The problem is that ActiveRecord::Base.to_xml can be really slow—even after making the optimizations I’ve explored in the previous parts to this series. For fairly simple AR objects, this won’t be a problem. However, when you have a deep object hierarchy and need substantial portions of it to be included in the XML response, it becomes a problem.

One instance where I encountered this was a nested has_many tree. That is, the object tree looks something like this:

ParentObject has_many ChildObjects
Each ChildObject has_many OtherObjects

The XML output is often 100k or more, and representing a few dozen or more total objects.

With that scenario, the ActiveRecord::Base.to_xml call looks something like this:

@parent_object.to_xml(:include=>{:child_objects=>{:include=>:other_objects}})

The alternative is definitely more work:

h = {:attr1=>parent_object.attr1, :attr2=>parent_object.attr2} # ... etc

h[:child_objects] = parent.child_objects.map do |child|
  h2 = {:attr1=>child.attr1, :attr2=>child.attr2} # ... etc

  h2[:other_objects] = child.other_objects.map do |other|
    h3 = {:attr1=>other.attr1, :attr2=>other.attr2} # ... etc
  end

  h2
end

h.to_xml(:root=>"parent_object")

Note that we’re actually still using a form of to_xml — it’s just Hash.to_xml instead.

There are some ways this could be streamlined, such as using parent.attributes to pull the list of attributes, or even an array of wanted attributes. I skipped that here to make the example easier to understand.

This method does have a couple benefits. One is that you can use more intelligent logic in regard to selecting the child_objects to include. For example: parent.child_objects.all(:limit=>10, :order=>'id desc') if you wanted just the last ten child objects.

Beyond that, and the focus of this posting, I found that the above resulted in a 2-3x speed increase. While I haven’t dug really far into the matter, it appears that the Rails magic that figures out what associations to include and the associated reflection has a heavy overhead.

I still use ActiveRecord::Base.to_xml much of the time, but the above option is nice for frequently requested actions with large, deep object trees.

Measuring ActiveResource Response Times (Rails Performance Series)

2009-10-26T19:14:00Z

This is a quick one, but if your Rails app uses ActiveResource as a client, it can be really useful to know something about each ARes request.

ActiveResource actually has some built-in logging of what requests are made, the response size, and how long it took. But, it’s not enabled. The good news is, it’s easy to turn it on.

I like to make ActiveResource’s logging match the rest of my environment — that is, it will log when in development mode and not in production mode.

Add the following to a file in config/initializers/:

ActiveResource::Base.logger = ActionController::Base.logger

Obviously, if you want it to go to a different log file, or function at a different log level, you can create a new Logger instance instead.

The logging output will look something like this:

GET http://api-server.com/resources.xml
--> 200 OK (1526 69ms)

Rails XML Parsing (Rails Performance Series)

2009-10-21T15:08:00Z

In part 3 of my Ruby on Rails Performance Series we’ll look at how to improve the parsing speed of XML content.

Rails will automatically parse XML in two key places:

1. When XML content is sent to a controller/action pair via POST or PUT. (Your app is an API server.)
2. When XML content is received from an ActiveResource request. (Your app is an API client.)

The default XML parser is REXML. It’s written in pure Ruby which makes it very portable. And slow. Very slow.

So, we need to replace REXML with something faster. The key is the libxml-ruby gem which is just a binding to the very fast libxml2 C library.

libxml-ruby has a different set of methods from REXML and is not drop in compatible. If you’re planning on moving your own REXML parsing code to libxml-ruby, it will take some work. However, it’s worth it. In one particular project, I had about a 100x speed improvement!

Rails, as it often does, makes the switch to libxml-ruby super easy. First, install the gem (requires a functioning compiler, the libxml2 library, and headers for libxml2):

gem install libxml-ruby

Then, just add a new file to config/initializers/. Let’s call it config/initializers/xml_config.rb. In that file, put this:

ActiveSupport::XmlMini.backend = 'LibXML'

This was only added in recent versions of Rails, I believe in 2.3.

Restart your Rails app server(s) and off you go!

Ruby XML Generation (Rails Performance Series)

2009-10-14T15:51:00Z

The standard way to build XML in Ruby is with the Builder gem.

Builder iterates over each character in the output text to determine if it needs to escape it. When generating larger XML documents, this can be quite slow.

Thankfully, the solution is already available via the fast_xs gem. This gem does compile an extension, so you’ll need a working compiler.

gem install fast_xs

Then just require it after Builder:

require 'builder'
require 'fast_xs'

It will be used automatically by Builder going forward.

If your app is built on Rails, it’s even easier. Just install the gem—that’s it! Recent versions of Rails will auto-detect the existence of fast_xs and enable it.

Ruby on Rails Performance Series: Intro and all about YAML

2009-10-07T17:32:00Z

Lately I’ve been doing a lot of API type work in Rails — both client and server. It mostly uses Rails’ default XML generation and consumption.

Along the way I’ve discovered that the default performance of XML and related things on Ruby and Rails is, shall we say, less than amazing. On the bright side, there are a number of ways to improve matters.

So, I’m going to do a series on improving the performance of Rails’ API-related stack.

I’ll start off with serialized columns in ActiveRecord.

Serialized columns are a way to store arbitrary information into a single TEXT column in your database. They probably shouldn’t be used too often, but they are useful at times.

YAML is the storage format for these serialized columns. They are defined like this:

serialize :some_column, Hash

Easy enough. The column some_column will be a native Ruby hash and yet will be serialized into a single TEXT column in the database. As is so often the case with Rails, it just works.

The serialization format is YAML. If you use Rails’ built-in to_xml methods to turn your ActiveRecord object into XML, it will also use YAML to serialize that particular column into a string and then pass that string to the XML generator.

This is at least consistent and works quite well. It’s just slow.

The core problem is that YAML does not perform particularly well. There are others online who have documented this in sufficient detail so I won’t do so here.

If your use of objects with serialized columns, either directly to the DB or via an API is low volume, the performance doesn’t particularly matter. On the other hand, it may be a hidden performance drain if you have a higher level of activity.

In my tests, converting a Hash to YAML takes between 4 and 5 times as long as converting YAML back to a Hash.

In the case I’ve been working on, the database records are read often but written rarely. However, API traffic is much higher and performance there is critical.

Because, performance at the DB layer isn’t a big bottleneck here based on actual usage patterns, the decision was to leave the database records alone and continue to use the default serialization mechanism there.

However, using YAML inside the API’s XML output had to go. Some API responses have 100 or more individual instances of the hashes that are currently being serialized with YAML. The next question was, what does it get changed to?

There are a few options: a) JSON, b) Marshal, c) nested XML, and d) some other custom serialization.

Each has pros and cons. Interestingly, all are faster than YAML. JSON is around 3x faster but doesn’t have quite the richness of object mapping. Marshal is really fast (about 9x over YAML) but has an almost binary-like output and isn’t very portable. Nested XML may or may not map into something desirable on the client side and may not be much faster. Custom formats may also be limited in object representation and take the most work to implement, both client and server.

If you had a simple array of strings, a custom format like “one,two,three” would be fairly straight forward. Complex objects, like entire referenced models would make this unworkable though.

So what’d I end up using? In this case, JSON. It’s much faster, easy to work with, and the data that needs to be transmitted is in a very simply format: all key to numeric value pairs.

Unfortunately, this can be a bit messy in both the client and server (although not difficult).

On the server, the attribute needs to be converted to JSON before to_xml sees it. The decision was to create an alternate version of the attribute (“options”) that was already in JSON.

def options_json
  options.to_json
end

Then you’ll need a couple options added to to_xml:

to_xml(:methods=>:options_json, :except=>:options)

This tells to_xml to skip the standard options rendering and include your custom method.

On the client side, in your model class inherited from ActiveResource::Base, add:

def options
  @options ||= ActiveSupport::JSON.decode(attributes["options_json"] || "{}")
end

As noted above, some API response here had dozens of objects that needed to be serialized. Moving to JSON over YAML reduced those API calls by about 25% (roughly 4 seconds to 3 seconds).

ebay + Paypal : privacy failures and unfriendly websites

2009-08-09T19:25:00Z

I bought something on ebay this morning (a 60-year old book). That’s not particularly exciting. But in the process of trying to pay for it, I had several observations.

1. Finding and buying the item on ebay was fairly straight-forward. I’m not sure why the item didn’t show up in the main search results, but several acceptable options did show up in the “Here’s what people have in their stores” section. Good enough.

2. ebay continues to try to force everything through Paypal. I was going to do that this time anyway, but the lack of choice is below average.

3. ebay and Paypal are still run as separate sites. So when I click on checkout from ebay, it sends me over to Paypal. That’s the theory anyway. Instead I get a giant error message telling me that Paypal has experienced an error and that I should wait and try again.

So I wait. And try again. And it fails again.

I wait some more. Same error. “Odd,” I think.

Months ago (years maybe) I started configuring my web browsers to only allow cookies back to the source domain. This is mostly a privacy protection attempt as it keeps volumes of advertising cookies from being accepted.

On occasion some site exhibits bad behavior by relying on these cross-domain cookies. Wondering if ebay/Paypal is one of these errant sites, I dig into my browser’s cookies.

Sure enough, cookies that are shared between ebay.com and paypal.com have been refused. I enable just the select cookie and try the checkout again. Now it works.

While reviewing all the cookies, there were more than a dozen various advertising cookies that ebay also tried to set. This merely justifies my general decision to block such things.

The failure here is really two-fold. One is not having a better way to hand off the session data between domains — something in the url would seem appropriate enough.

The second is having an unhelpful, misleading error message.

4. Lastly, Paypal still, after all these years, refuses to let me set a credit card as my default payment method, instead forcing me to go through several clicks and multiple pages to override it for each purchase.

I’m reminding yet again why I buy from ebay and use Paypal so infrequently. It’s just a hassle.

Disabling Rails asset timestamps

2009-06-26T02:34:00Z

One of the common questions in Rails land is how to disable the funny numbers at the end of images and CSS stylesheets. Let’s explore what exactly those numbers are, why they’re sometimes helpful, and how to turn them off when they’re not.

Those numbers are timestamps — the number of seconds since January 1, 1970. The time is the last time the image or stylesheet was last modified.

So if the time is June 26, 2009 at 02:39:24, then the timestamp would be 1245983964.

Great! Now, why do we care?

Rails appends these timestamps to cause web browsers to think the files are unique. This causes a browser to not use its cache and to reload the file. That is, even those these reference the same file, the browser thinks they are different.

/stylesheets/styles.css?1245981180
/stylesheets/styles.css?1245983964

Let’s backtrack just a little. Web browsers, in an effort to show you a page faster, will keep a copy of recently loaded files either in memory or on your hard disk. If it is just told to load /stylesheets/styles.css and you change the CSS file, the web browser might use the old copy in the cache rather than load the new version of the file.

However, by appending these timestamps on the end, when the file changes, the browser will also think it’s a different file and will always load the new copy.

So it’s really a useful feature most of the time.

Sometimes it can get in the way (or you may just think it’s ugly and aren’t concerned about browser caches. For those instances, you can turn it off.

ENV['RAILS_ASSET_ID'] = ''

You should add a new file in your config/initializers/ directory to put that in. Let’s call it no_timestamps.rb. So, in config/initializers/no_timestamps.rb, just add the one line above. Then, restart your Rails server.

That’s it! No more appended timestamps.

Updated 11/12/2009: As noted in the comments, I had incorrectly suggested using cache_asset_timestamps=false. Corrected the above to show the proper way to do this.