interprogramming articles

List of ActiveJob queue backends

2020-08-26T04:12:00Z

During some recent research, I made a list of all the Rails/ActiveJob compatible queue backends I could find.

I started with the list from Rails, which is just the queues with adapters built-in to Rails, then added anything else I could find that wasn’t obviously dead (mostly determined by lack of commits for an extended time).

Additions and corrections are welcomed.

	Async	Queues	Delayed	Priorities	Timeout	Retries	Job storage	Exec mode
Backburner	Yes	Yes	Yes	Queue	Job	Global	beanstalkd	inprocess, fork, thread
Delayed Job	Yes	Yes	Yes	Job	Global	Global	activerecord, mongoid	fork
Faktory Worker	Yes	Yes	Yes	Queue	No	Job	faktory (embedded redis)	thread
Fliess	Yes	Yes	Yes	Job	Job	Job	activerecord	thread
Gearman Ruby	Yes	Yes	?	?	?	?	gearman	single worker
GoodJob	Yes	Yes	Yes	Job	Job	Job	postgres	inprocess, thread
Google Cloud Pub/Sub	Yes	Yes	?	?	?	?	gcp	thread
KubeQueue	Yes	No	Yes	No	Job	Job	kubernetes (etcd)	k8s job
Que	Yes	Yes	Yes	Job	No	Job	postgres	thread
queue_classic	Yes	Yes	Yes	No	No	No	postgres	fork, thread
Resque	Yes	Yes	Yes (Gem)	Queue	Global	Yes	redis	fork
Shoryuken	Yes	Yes	Yes	Queue	No	Job	aws sqs	thread
Sidekiq	Yes	Yes	Yes	Queue	No	Job	redis	thread
Sneakers	Yes	Yes	No	Queue	Queue	No	rabbitmq	thread
Advanced Sneakers	Yes	Yes	Yes	Queue	Queue	No	rabbitmq	thread
Solid Queue	Yes	Yes	Yes	Job	Job	Job	activerecord	thread
Sqewer	Yes	No	Yes	No	No	Global	aws sqs	thread
Sucker Punch	Yes	Yes	Yes	No	No	No	memory	inprocess
Workhorse	Yes	Yes	Yes	Job	No	Global	activerecord	thread
Active Job Async	Yes	Yes	Yes	No	No	No	memory	inprocess
Active Job Inline	No	Yes	No	No	No	No	memory	inline

There are also gems for running background jobs that are not ActiveJob compatible. Those are not listed above.

TLS certificate verification error on AWS RDS

2019-09-01T00:55:00Z

Recently, when upgrading the Docker base image for a Ruby on Rails app, the app suddenly started raising SSL error: certificate verify failed exceptions when attempting to connect to Postgres. Unfortunately, the error message doesn’t describe why verification failed.

The base image was being upgraded from Debian 9 (Stretch) to Debian 10 (Buster). The Postgres 11 instance (which started life on 9.x, so it’s older) is hosted by AWS RDS and configured to require TLS connections.

It turns out that AWS RDS only supports TLS 1.0. However, Debian 10’s openssl config defaults to requiring TLS 1.2+. The result is the certificate verification error above.

The quick fix is to modify /etc/ssl/openssl.cnf (as this is what changed between Debian 9 and 10) to allow TLS 1.0. Near the bottom of the file, change

MinProtocol = TLSv1.2

MinProtocol = TLSv1.0

Or, run this simple one liner:

sed -i -e 's/TLSv1.2/TLSv1.0/' /etc/ssl/openssl.cnf

The better fix is to upgrade/replace the Postgres server to support TLS 1.2+ since TLS 1.0 and 1.1 are being deprecated.

Rails 6: How to use multi-environment credentials

2019-05-12T22:26:00Z

Rails 6.0 adds support for multi-environment credentials. Here’s a quick summary of how it works:

By default, behavior is the same as Rails 5.2. To configure credentials for another enviornment, just add --environment <env>.

Available commands

rails credentials:show [--environment <env>]
rails credentials:edit [--environment <env>]
rails credentials:help

Example: rails credentials:edit --environment production

(Hint: RAILS_ENV=<env> rails credentials:show won’t work. --environment must be used instead.)

How Rails behaves

Rails continues to use the 5.2-compatible files (config/master.key & config/credentials.yml.enc) in the absence of an environment-specific file.

If an environment-specific file is available, then that is used instead (and the default file is ignored – they are not merged). This is convenient when you want multiple environments, eg: development and test, to both use the same file. In this example, use the default file for those two environments, and then provide an environment-specific file for production only.

For reference, Rails will look in the following places for the master key for decryption:

ENV['RAILS_MASTER_KEY']
config/credentials/<env>.key
config/master.key

And it will look for the actual encrypted credentials at these locations:

config/credentials/<env>.yml.enc
config/credentials.yml.enc

Replacing the Rails 5.1 asset pipeline with webpacker 3

2017-11-16T21:41:00Z

My last post was about using webpacker v3 on Rails 5.1.

This time I want to explore how to replace the traditional sprockets-based asset pipeline completely.

Our goal will be to add Bootstrap v4 and jQuery v3 to an app, handled entirely through webpacker. We won’t even install the sprockets related gems.

We’ll also enable rails-ujs, turbolinks, and even images using webpacker.

First steps

Let’s start with a new app:

rails new look_ma_no_sprockets --skip-spring --skip-sprockets --webpack

Notice that Gemfile is rather sparse and leaves out all the sprockets related gems.

Next, create a controller, skipping the default assets that would be for sprockets:

rails g controller home index --no-assets

# We'll need these in a minute:
mkdir -p app/javascript/stylesheets app/javascript/images

And make the new route the default in config/routes.rb:

Rails.application.routes.draw do
  root 'home#index'
end

(See app at this stage)

Plumbing in webpacker

Now let’s adjust our layout file to use webpacker assets.

In app/views/layouts/application.html.erb, replace the default stylesheet and javascript tags with their webpacker equivalents:

<head>
  <title>LookMaNoSprockets</title>
  <%= csrf_meta_tags %>

  <%= stylesheet_pack_tag 'application', media: 'all', 'data-turbolinks-track': 'reload' %>
  <%= javascript_pack_tag 'application', 'data-turbolinks-track': 'reload' %>
</head>

We’ll be creating all stylesheets in app/javascript/stylesheets/. Yes, it’s a bit of an odd place, but it keeps everything for webpacker in app/javascript/, which is especially helpful if you decide to use both sprockets and webpacker.

Create app/javascript/stylesheets/application.scss:

.webpacker-test {
  color: blue;
}

Add this line to the bottom of app/javascript/packs/application.js:

require.context('../stylesheets/', true, /^\.\/[^_].*\.(css|scss|sass)$/i)

FYI, the true in the second argument is whether to search all subdirectories. We’ll leave it as true here, but know that you can restrict which directories are searched if you need to.

Let’s also replace app/views/home/index.html.erb with a few things to help us check our progress:

<h1>Home</h1>

<div class="webpacker-test">
  I will be blue if webpacker's CSS is working.
</div>

<div class="bootstrap-test text-success">
  I will be green if bootstrap is working.
</div>

<div class="jquery-test">
  jQuery is not yet working.
</div>
<script>
  $(function(){
    $(".jquery-test").html("jQuery is working.");
  });
</script>

Once this is in place, go ahead and fire up rails server and check your work at http://localhost:3000/. You should see the “blue” test pass, but not the other two (yet).

Your browser’s console log should also show “Hello World from Webpacker”. (If you don’t see it, make sure you’re showing all logs, not just errors.)

(See app at this stage)

Adding jQuery and Bootstrap

Now let’s get Bootstrap going. Since Bootstrap depends on jQuery and popper.js, we’ll handle those too.

As of this writing, Bootstrap 4 is at beta 2. Once v4 is fully released, you can drop the @version part off the end.

yarn add jquery popper.js bootstrap@4.0.0-beta.2

cp node_modules/bootstrap/scss/_variables.scss app/javascript/stylesheets/
ruby -pi -e 'gsub(/ !default/, "")' app/javascript/stylesheets/_variables.scss

The last two lines above prepare a copy of Bootstrap’s variables for your app to customize. We won’t customize it here, but it’ll be ready for you later.

Next, add the following three lines to the top of app/javascript/stylesheets/application.scss. These will instruct our app to use Bootstrap’s CSS.

@import '~bootstrap/scss/functions';
@import 'variables';
@import '~bootstrap/scss/bootstrap';

We also need to import jquery and bootstrap in our app/javascript/packs/application.js. Here’s the complete new file:

import $ from 'jquery'
global.$ = $
global.jQuery = $

import 'bootstrap'

require.context('../stylesheets/', true, /^\.\/[^_].*\.(css|scss|sass)$/i)

Finally, we need to help jQuery and popper.js be seen by other .js files you’ll write. Replace config/webpack/environment.js with the following:

const { environment } = require('@rails/webpacker')

const webpack = require('webpack')
environment.plugins.set(
  'Provide',
  new webpack.ProvidePlugin({
      $: 'jquery',
      jQuery: 'jquery',
      'window.jQuery': 'jquery',
      Popper: ['popper.js', 'default'],
  })
)

module.exports = environment

If you reload the page in your browser, you should now see some green text indicating Bootstrap’s CSS is working. The jQuery message should also have changed to show it is also working.

(See app at this stage)

rails-ujs and turbolinks

If you want to use rails-ujs (and you probably do, since Rails 5.1 now encourages remote/ajax forms), we now install it via yarn too.

yarn add rails-ujs

Then, to app/javascript/packs/application.js, add:

import Rails from 'rails-ujs'
Rails.start()

(See app at this stage)

Adding turbolinks is similar.

yarn add turbolinks

In app/javascript/packs/application.js, add:

import Turbolinks from 'turbolinks'
Turbolinks.start()

(See app at this stage)

Images

Images work a bit like stylesheets, in that they’ll be placed in app/javascript/images/.

Add this to app/javascript/packs/application.js, right after to the stylesheets line:

require.context('../images/', true, /\.(gif|jpg|png|svg)$/i)

Using images (or other similar assets) within Rails views looks a little bit different from before. Use asset_pack_path() inside the main asset helper, like this:

<%= image_tag(asset_pack_path('images/abc.svg')) %>

(See app at this stage)

Wrapping it up

That should be about everything. If you find a better way to do some part of this, please add a comment below.

For that matter, I’d love to hear how it goes for you.

Using Webpacker 3 on Rails 5.1

2017-11-08T23:27:00Z

I recently started a new Rails 5.1 project. I wanted to try out webpacker, which is now at version 3. Turns out, the webpacker team has been working hard to simplify things and they’ve accomplished quite a bit. Most posts online talk about v1 or v2, and they include a lot more steps than are now necessary.

My goal here is to show how to use webpacker alongside the traditional asset pipeline (sprockets). The next post will show how to completely replace the asset pipeline with webpacker.

For reference, I’m working with ruby 2.4.2, rails 5.1.4, webpacker 3.0.2, node 8.7.0, and yarn 1.2.1.

Let’s dive into the now-simplified webpacker world.

Installing node and yarn

First things first. You’ll need a functioning copy of node and yarn.

On macOS, these are both easily installed using homebrew.

brew install node yarn

Obviously you’ll need ruby and rails installed too. Since that’s well covered elsewhere, I won’t repeat it here.

Creating the app

Rails and webpacker provide automatic integration with a handful of popular javascript frameworks. Just for fun, we’ll choose react this time around.

rails new webpacker3_example --skip-spring --webpack=react

If you had all the prerequisites in place, not only was bundle install run automatically, but so was yarn install.

(See app at this stage)

Rendering a React component

Now let’s create a controller that will render the default react component.

rails g controller home index

In config/routes.rb, make the new route the default:

Rails.application.routes.draw do
  root 'home#index'
end

By default, webpacker 3 is now integrated into the rails server process and will automatically compile all packs inside app/javascript/packs/. There is no longer any need to run an external webpack-dev-server process. All you need to do is reference one or more .js files inside packs/.

Let’s do that now by adding the following line inside the <head> section of our primary layout at app/views/layouts/application.html.erb:

<head>
  <!-- [existing html tags] -->
  <%= javascript_pack_tag 'hello_react' %>
</head>

Let’s test it by going to http://localhost:3000/ in our browser. You should see:

Home#index

Find me in app/views/home/index.html.erb

Hello React!

If “Hello React!” is at the bottom, then everything’s working properly!

And that’s it. Nothing more is required any more. webpacker takes care of it all.

(See app at this stage)

Next steps

From here, if you need to install a new js package, use yarn add <package_name>.

If you (or a colleague) are pulling your code onto a fresh box, you may need to run yarn install to install all the node packages. Basically, anytime you need to run bundle install, plan to run yarn install next. This is true on your dev box and on production servers or containers.

When going into production, the webpack assets are now automatically compiled as part of the traditional rails assets:precompile hook.

So there you have it. Yes, it really is that straightforward now to get up and running with webpacker 3 and rails 5.1.

Benchmarking GeoIP gems for Ruby

2017-10-10T03:55:00Z

On a recent project, I had need again to use GeoIP (by the lovely people at MaxMind). I decided to take some time to benchmark a handful of various GeoIP gems, both for GeoIP v1 (legacy) and the newer GeoIP v2.

I decided to benchmark the extraction of 3 pieces of data, using the GeoIP Lite City database:

2-character ISO country code (eg: US)
Full region name (eg: Colorado)
Timezone (eg: America/Denver)

Entries are labeled according to the installed gem name. Gems with “1” use the v1 GeoIP database. Those with “2” use the v2 database.

Gems labeled with “rb” are pure ruby. Those with “c” have C extensions.

3 of the tested gems do not return a timezone. Those 3 libraries turned out to be the 3 fastest, strongly suggesting that retrieving a more complete set of data from the GeoIP dataset does come with increased overhead. These 3 gems are labeled with “-tz” in the results below.

Lastly, the geoip gem has the option of preloading the entire GeoIP database into memory. It was benchmarked without and with preloading.

Without further ado, here are the results, based on 100,000 iterations (ordered by real time):

user     system      total        real
geoip-c (1;c) -tz           0.520000   0.010000   0.530000 (  0.533592)
geoip2_compat (2;c) -tz     0.520000   0.020000   0.540000 (  0.534711)
mmdb (2;c) -tz              0.700000   0.010000   0.710000 (  0.725789)
hive_geoip2 (2;c)           3.470000   0.010000   3.480000 (  3.495051)
geoip (1;rb;preload)        3.580000   0.000000   3.580000 (  3.583469)
geoip (1;rb)                4.780000   1.740000   6.520000 (  6.542248)
maxmind_geoip2 (2;c)       12.420000   4.670000  17.090000 ( 17.092035)
maxminddb (2;rb)           27.730000   0.030000  27.760000 ( 27.781552)

As noted earlier, the 3 fastest libraries (geoip2_compat, geoip-c, and mmdb) all have C extensions and all skip timezone loading.

The next fastest, and also the overall fastest to return a full response from the database, is hive_geoip2. The API for this gem is rather basic, but it’s arguably a very worthwhile tradeoff given its high performance.

Right behind it is the preload-flavor of the pure Ruby geoip gem, which is quite impressive. Equally impressive is that the non-preload flavor of the same gem comes in right behind, taking only about twice as long to run.

maxmind_geoip2 is just slow. Digging into the code, it appears to open and close the database file descriptor on every lookup, resulting in a substantial performance penalty.

maxminddb has a friendly API to work with, but I wonder if that’s perhaps the cause of its slower performance. That said, unless you’re processing a rather high rate of GeoIP lookups, this one is still worth a look.

If you want a pure ruby solution, both geoip for GeoIP v1 and maxminddb for GeoIP v2 are both viable for modest volumes of lookups.

For higher volumes, definitely use a gem with a C extension. If you need timezones, then hive_geoip2 is your best bet. If not, then you have several choices. Do remember that you’ll also need the underlying C library and headers installed before installing the gem. That’ll be libgeoip for v1 and libmaxminddb for v2.

I’ve published the benchmark code if you’re interested. There are a couple of gem namespace conflicts. To run the benchmarks, set PASS==1 first and install the matching gems. Then set PASS==2 and install those gems, being careful to uninstall geoip. If you want to return to pass 1, uninstall geoip-c.

Using Mongoid with Rails 4

2013-07-13T18:11:00Z

Mongoid 3 is only compatible with Rails 3.x. Rails 4 requires Mongoid 4.

However, as of this writing, Mongoid 4 is still in development. I’ve had success using the master branch though.

In your Gemfile:

gem 'mongoid', github: 'mongoid/mongoid'

# and if you use Sidekiq + Kiqstand:
gem 'kiqstand', github: 'mongoid/kiqstand'

Upgrading to Rails 4.0

2013-07-11T14:47:00Z

I’m busy upgrading several apps to Rails 4.0. As is to be expected, there are a few gotchas here and there. Most are documented somewhere… deep in a changelog… mixed in with dozens of other things that probably won’t bite you.

I’ve been posting some of my notes along the journey. This particular post will serve as a central reference point, a table of contents of sorts, and will just contain links to other posts. I’ll try to keep it updated as I add new posts.

Overview

See the official Ruby on Rails article to start.

Preparing a Rails 3.2 app to be upgraded

Models and datastores

Using Mongoid

Assets

Deployment

Missing assets in Rails 4.0 (part 2)

2013-07-09T17:00:00Z

In addition to the asset path changes covered in part 1, there are some changes in how processed asset files are exposed to web clients.

In this case, the problem shows up with asset files that are referenced by name without the md5 fingerprint. Previously, plain-named asset files were also exposed to the webserver. Now they don’t seem to be.

The most likely culprit here are files referenced from within a CSS file, such as:

src: font-url('custom-webfont.eot');

background: url('background.png');

In this case, you’ll need to use the asset_path helper like so:

src: url('<%= asset_path('custom-webfont.eot') %>');

background: url('<%= asset_path('background.png') %>');

You may need to rename your css file to enable ERB parsing, eg: design.css becomes design.css.erb.

Another possibility are hard-coded asset references in your view templates somewhere. Once again, change such references to use the asset_path helper.

Missing assets in Rails 4.0 (part 1)

2013-07-08T16:04:00Z

The asset pipeline was introduced in Rails 3.1. It continues to exist in Rails 4 and has seen a number of changes.

In one key change, Sprockets (the brains behind the asset pipeline) now looks in fewer places for asset files.

In Rails 3, Sprockets looked in app/assets, lib/assets, and vendor/assets for assets. Importantly, this applied not only to your main app, but also to any asset-providing gems, such as font-awesome-rails.

In Rails 4, Sprockets only looks in app/assets (both in your app and in gems). If you have any assets that aren’t working properly (most likely fonts, images, or CSS files not loading), first check for updated versions of any gems.

If it’s your own gem or a gem that hasn’t been updated, it may be as simple as moving lib/assets or vendor/assets to app/assets.