Category Archives: rails

Rails, Active Record & Postgres – Optimizing Deletions

Every Rails project I’ve worked on has used Active Record. ORMs like Active Record have many benefits. They abstract you away from the database. They make querying data elegant and simple (in most cases).

However, sometimes Active Record is not more efficient than Postgres.

Let’s look at an example. Say our Postgres database has the following schema. Our top-level table is a building. The building has many apartments. An apartment has many rooms. Each room has multiple pieces of furniture, and each piece of furniture has several materials. Your models might look something like this:

class Building < ActiveRecord::Base
  has_many :apartments, dependent: :destroy
class Apartment < ActiveRecord::Base
  has_many :rooms, dependent: :destroy
class Room < ActiveRecord::Base
  has_many :furnitures, dependent: :destroy
class Furniture < ActiveRecord::Base
  has_many :materials, dependent: :destroy
class Materials < ActiveRecord::Base

Now, let's say we want to remove a building. This building has 100 apartments, each apartment has 10 rooms, each room has 15 pieces of furniture, and each piece of furniture has five materials.

Using Active Record

Removing the building with Active Record is simple. You just need to call building.destroy. Because we set up all of those dependent destroys, Active Record will take care of removing all of the related records.

Active Record is so nice that it will also make sure to call the delete hooks on any object that has them. It does this by loading each object into memory and calling object.destroy on it. This transaction will create 91,001 objects. It takes my development machine about 77 seconds to run this scenario. So, it's not very efficient. Let's try this a different way.

Using Postgres

Databases are built to handle operations on large data sets. So, why not leverage the database for its purpose? Rails 4+ supports foreign keys. Let's add a migration to add foreign keys to our tables. It might look like this:

class CreateForeignKeys < ActiveRecord::Migration
  def change
    add_foreign_key :apartments, :buildings, on_delete: :cascade
    add_foreign_key :rooms, :apartments, on_delete: :cascade
    add_foreign_key :furnitures, :rooms, on_delete: :cascade
    add_foreign_key :materials, :furnitures, on_delete: :cascade

Now, when we call building.destroy, instead of creating all of those objects in memory, Postgres will handle destroying all of the related records. It takes my machine 45 seconds to run the same operation using cascade deletes.


Active Record is a powerful tool. Because of this, we may shy away from letting the database handle work for us. But there are occasions where the database is the correct answer.

Some will argue that putting optimizations in the database causes opaqueness, and new developers rolling onto a project may not understand why all apartments disappear when a building is removed. I would argue that one of the first things developers should do when joining a project is to become familiar with the project schema. If they do, they will understand why and how the deletion happens.

Let your database do what it was built to do!

The post Rails, Active Record & Postgres – Optimizing Deletions appeared first on Atomic Spin.

Embedding Ember in Existing Apps

My current project, an online student assessment tool, involves adding features to a Rails app built in the 2011/2012 Rails 3 era. The app was also built using Backbone.js, a custom templating language, and raw JSON responses. It was developed using good development and design practices (such as TDD, SRP, etc.), but many of these technologies have been superseded by modern frameworks such as Ember.js or Angular. Active development on this project was not going to be easy or fun.

ember-cli-rails helps solve many of these problems, so we decided to carve off a chunk of the app to convert to Ember.

What is ember-cli-rails?

ember-cli-rails is a library for merging the two worlds of Rails and Ember. It allows you to embed Ember.js apps (even pre-existing ones) within Rails apps in a clean and simple way.

Here are some of the benefits of using Ember CLI Rails (from their website):

  • Keep up to date with latest npm modules and EmberCLI addons via ember command
  • Inject Rails-generated content into your EmberCLI app
  • Avoid cross-origin woes
  • Enable easier end-to-end testing

This is a great tool for greenfield development, but we’re not always fortunate enough to be doing greenfield development in the tech stack of our choosing.

Why Use a Hybrid Approach?

Luckily, we have a nice, mostly isolated section (student assessments) that would benefit from conversion to Ember. Here are some of the benefits of this approach:

  • Helps update older technology
  • Improves user experience through a single-page application
  • Speeds up development of new features
  • Uses existing styles and site-wide JavaScript
  • Integrates with Rails’ existing asset pipeline, including finger-printing
  • Easily adds more Ember mount points to convert the site in lower-risk chunks

Bringing in EmberCLI Rails

The installation and setup was very straightforward. For a basic setup, all you have to do is follow the README. Here are a few other things I did to make ember-cli-rails play nicely with my pre-existing site:

  • Upgrade ActiveModelSerializers from 0.8 to 0.10
  • Create new serializers that adhere to the expected JSON API
  • Revise and create new API endpoints using the new serializers
  • Manually convert JST templates from Haml to Handlebars

Moving the Embed Ember.js Project Forward

Please note, this plan is not for the faint of heart. We’re able to mitigate some of the risk by updating one part of the app at a time, but it’s still a partial rewrite. We will be investing weeks of the team’s time to convert the current working feature set before we can reap any of the benefits.

However, it is worth the risk to update our technology, improve the user experience for students, and speed up future development. ember-cli-rails has done a great job of sweating the details, getting out of my way, and letting me spend time building the features I want. As the project progresses, I’m sure I’ll be posting updates of our wins and losses, but I’m excited to see where this takes us.

The post Embedding Ember in Existing Apps appeared first on Atomic Spin.

Uploading Files in Rails Using Paperclip and Active Admin

I recently came across a situation where I needed to be able to upload a file to a Rails server with Active Admin. I did a quick search on Google and found this post by Job, a fellow Atom.

Our use cases were a little bit different, though. He was storing the file contents directly in the database, whereas I needed to be able to uplaod a firmware image file to the server’s filesystem, parse the file name, and perform some validations on the file. I decided to use the Paperclip gem to manage the file processing and storage. Using Job’s advice on Active Admin file uploads, I expanded his example to incorporate Paperclip.

What is Paperclip?

Paperclip, as its name suggests, is a file attachment library for ActiveRecord. It is designed to treat files much like other attributes, and it provides a whole slew of built-in extensions (e.g. validations, pre/post processing callback hooks, security enhancements, etc.). The setup is very simple, and getting the basics up and running takes only a few minutes. For the sake of this example, we will set up the ActiveRecord model, create a migration, and get rolling with the Active Admin file upload.

Install the Paperclip Gem

Installing the gem is easy. Just add

gem 'paperclip'

to your Gemfile and run

bundle install

Create a Migration

If you have an existing model where you want to add Paperclip support, then you simply need to create a migration. In this case, I already had a table and a corresponding model for my firmware image, so I just needed to add a migration to add the columns that Paperclip requires. The easiest way to do this is with the Rails migration generator:

rails generate paperclip firmware image

This will automatically create a migration file that looks like this:

class AddImageColumnsToFirmware < ActiveRecord::Migration
  def up
    add_attachment :firmware, :image
  def down
    remove_attachment :firmware, :image

The add_attachment helper will automatically create the following columns in your table:

  • image_file_name
  • image_content_type
  • image_file_size
  • image_updated_at

Technically, only the *_file_name column is required for Paperclip to operate, so you could throw away the others if you don’t need them.

Paperclip Your Models

The true beauty of Paperclip is how well it integrates with ActiveModel. To add Paperclip support to your model, start by adding the following line to your model class:

class Firmware < ActiveRecord::Base
  has_attached_file :image
  # more stuff to come

With that one line, Paperclip is now integrated with your Rails model, and it will automatically handle your file “attachments” just like any other Rails attribute! There are, of course, plenty of other options that you can add to the has_attached_file attribute (e.g. specifying a file path, style type, etc.), but I won’t go into that right now. For our purposes, the defaults should be just fine.


Paperclip also makes it really easy to perform powerful validations on the file. If any validation fails, the file will not be saved to disk and the associon will be rolled back, just like any other ActiveRecord validation. There are built-in helpers to validate attachment presence, size, content type, and file name.

In our case, we really just need to validate the file name to ensure that it had the proper format and also that it was a unique file name. The following validation did the trick:

validates_attachment_file_name :image, :matches => [/_d+_d+_d+.bin$/]
validates_uniqueness_of :image_file_name # this is a standard ActiveRecord validator

Additional Processing

I also wanted to be able to grab the firmware version out of the file name. The best way to do this is with a before_post_process callback in the model, like this:

before_post_process :parse_file_name
def parse_file_name
  version_match = /_(?d+)_(?d+)_(?d+).bin$/.match(image_file_name)
  if version_match.present? and version_match[:major] and version_match[:minor] and version_match[:patch]
    self.major_version = version_match[:major]
    self.minor_version = version_match[:minor]
    self.patch_version = version_match[:patch]

Before the file is saved, but after the filename is validated (so we can be sure it has the proper formatting), we extract the major, minor, and patch numbers from the filename and save them in our database.

Configure Active Admin for File Upload

Now that Paperclip is all set up and wired into our Rails model, we need to actually set up the file upload piece in Active Admin. I won’t go into much detail, since I relied on Job’s post as a reference. Basically, all we need to do is:

  1. Define the Index page contents

    Enumerate which columns will be displayed. If any special decoration is required, such as customized column title, sort properties, or row formatting, they can be specified easily. For example, I wanted a link to download the firmware image, so I added a link_to in the “Image” column. Note that the file path is stored in the image.url attribute.

  2. Specify which parameters may be changed

    Use the permit_params method to whitelist any attributes.

  3. Create the upload form

    Use the f.input :image, as: :file syntax to automatically create a file upload field in Active Admin.

The code snippet below is the Active Admin page, which allows the user to create, view, edit, and delete a firmware image.

ActiveAdmin.register Firmware do
  permit_params :image
  index do
    column 'Image', sortable: :image_file_name do |firmware| link_to firmware.image_file_name, firmware.image.url end
    column :image_file_size, sortable: :image_file_size do |firmware| "#{firmware.image_file_size / 1024} KB" end
    column "Version" do |firmware| "#{firmware.major_version}.#{firmware.minor_version}.#{firmware.patch_version}" end
    column :created_at
  form do |f|
    f.inputs "Upload" do
      f.input :image, required: true, as: :file

And that’s it! We now have a fully functional file-upload implementation using Active Admin backed by Paperclip. It’s really quite simple, and it only took a few minutes to get it set up and running.

The post Uploading Files in Rails Using Paperclip and Active Admin appeared first on Atomic Spin.

Waiter, There’s a WordPress in My Web App!

If you’ve ever been a part of developing custom software, you’ve probably seen some features turn out to be much more complicated than anticipated. Sometimes, it’s due to unforeseen technical constraints. Other times, it’s a case of not fully understanding the nature of the feature—a situation that led me to an unexpected use for WordPress.

We were developing a new version of a web app with one major activity, plus a few content pages off to the side. We had de-prioritized these content pages, estimating that they would be very quick to implement with some basic HTML to hold the content and a little CSS to make it look decent. The problem was, we didn’t understand the dynamism involved in the content pages.

While these pages appear static from the user’s perspective, the content actually depends on  what type of user you are—including which user group you’re affiliated with and which language you speak. One of the user groups has a sub-group for Spanish speakers, so all pages for that subgroup need to show the content in Spanish, not English. The cherry on top of the complexity sundae was that the customer wanted to be able to edit this content, and really wanted a WYSIWYG editor as opposed to editing the HTML.

By the time we properly understood these requirements, we had already spent most of the project time and budget on the major activity in the app, so we had little ability to devote more resources to the feature. Moreover, even with triple our initial budget, building a proper CMS was out of the question. So what could we do? Luckily, the designer on our project had heard of a little thing called WP-API.

Getting to Know WP-API

WordPress is far and away the most popular open-source Content Management System. It supports a mind-boggling number of plugins, including one called WP-API which allows access to posts via a JSON interface. Because WP-API lets you specify filters when you ask for posts, we realized we could use this feature to leverage the power of WordPress, a neat solution which would provide the dynamic content we needed and the content management experience the customer wanted.

Our approach has been referred to as “headless CMS.” Essentially, WordPress lives as a separate service on our server, accessible only to our app’s backend (using Rails) and to computers behind the customer’s firewall. When the client (using Ember) requests content from the backend, the backend then requests content from the WordPress site based on the current user and the type of content requested. This image provides a fairly good approximation of what’s happening:

WordPress API

When creating posts in WordPress Admin, the customer specifies categories which indicate the type of content and the intended user. Consequently, our search can identify the right post given the type of content and the current user.

Handling Content Requests

For technical reasons, I was not able to add all of the filters I wanted into the query, so I had do some filtering on our backend based on the list of matching posts. After that point, I had HTML ready to send to the client for rendering—almost.

If the content contained images, documents, or links to other posts, I still had to do something to make those resources available to the site. I couldn’t use the links generated by the WordPress WYSIWYG editor. Firstly, the URL isn’t accessible to the outside world, and secondly, even if it were, the content is supposed to be gated according to user group. So I had to rewrite the URLs to point back through our backend, and then whitelist those URLs for later retrieval by the current user. Additionally, if a post linked to another post, I had to rewrite that link to point to a general “content holder” route in Ember. Here’s a high-level view of what’s going on:

def load_content(ctx)
  json = WordPressService.retrieve_posts ctx
  if json.present?
    posts = WordPressService.parse json
    post = WordPressService.find_post ctx.current_user_group, posts
    if post.present?
      rewritten_post = WordPressService.translate_content_and_links post.raw_content
      WordPressService.whitelist_urls ctx.session_access, rewritten_post.urls_to_whitelist
      return rewritten_post.to_h

With that code in place, I could use the returned HTML in the Ember app, knowing that users could see it, images and all. Links to files, and even links to other blog posts, worked. The content loaded fairly quickly, and we could easily add caching later if need be.

Streamlining Client-Side Links

So far, the content worked, but the experience of going from one WordPress post inside our app to another wasn’t very smooth. Because Ember couldn’t detect that <a href=''> should transition via an Ember route change, it reloaded the whole site whenever we clicked a link. To fix this, I added a click handler in the “content holder” component to hijack the link when necessary:

var matchContent = window.location.origin + '/content/';
if ( && {
  var path =;
  path = path.match(/[w-]*/)[0];
  this.transitionTo('content', path);

Putting It All Together

WordPress in a web app? Absolutely. WP-API enabled us to deliver the feature to our customer, on time and wrapped in a CMS experience that surpassed their previous system. To me, it really shows the power of combining tools through the polyglot approach. If you keep your eyes (and your mind) open for creative solutions, you may just find a simple way to solve a complex problem.

The post Waiter, There’s a WordPress in My Web App! appeared first on Atomic Spin.

Script Away Your Annoyances – Removing Manual Project Setup Tasks

Don't make me think.
Don’t make me think. Image credit : Beth Tourek.

In recent years I’ve come to deeply appreciate the phrase “don’t make me think.” I will forget how to perform a setup step approximately 7 seconds after I do it, let alone 3 years after I’ve done it (and I know I’m not alone!).

Thus starting to work on an existing project will often give me anxiety—not because I’m worried about learning a new domain or existing system, but because getting the dang thing running can sometimes be the hardest part!

Of course, the best way to mitigate this pain is to codify setup tasks. Why rely on my memory when the computer can do something for me? In other words, don’t make me think. Below are two examples of where I scripted away my annoyances at setting up some projects.

Setting up YouTube Playlists

I worked on a Rails project that integrated with YouTube playlists. Our system created playlists, added, and removed videos from them via the YouTube API. Each system that interacted with YouTube—be it a production instance or local test instance—requires several API tokens in order to interact with YouTube. The tokens are read from the config/system_config.yml file at runtime.

After setting this up once locally, I knew I’d never want to do it by hand again. Who wants to create this file, go to YouTube, create a bunch of playlists with specific names, dig out the tokens, and then paste them back into the file? It’s boring, repetitive, and error prone. Thus I created a Rake task to generate config/system_config.yml on new machines.

I’m not going to paste the implementation here, as I am not here to discuss writing Rake tasks or the YouTube API. The point is that I used a script to replace my memory and opportunity for human error.

Generating Rails’s secrets.yml

Another Rails project I fired up required three generated tokens to be in the config/secrets.yml file. Thankfully the README documented how to generate the secrets—run rake secret, copy the output, and paste into the YAML file. But who wants to do that three times every time you setup the local repository?

In this case, I created another Rake task named generate:secrets that:

  1. Checks for the presence of an existing config/secrets.yml and aborts if it’s already there.
  2. Generates three secrets using the same library rake secret does. The secrets go into a Ruby hash.
  3. The hash is serialized to config/secrets.yml as YAML.

Script Away Your Annoyances

Now I don’t need to think about setup again in either project. As long as I can copy and paste a setup command from a README, I’ve saved my brain cycles for more valuable problems. I hope that this post has inspired you to script away your setup annoyances, whatever they may be!

Other posts in this series:

The post Script Away Your Annoyances – Removing Manual Project Setup Tasks appeared first on Atomic Spin.

Managing Complex Permissions in CanCanCan for Rails

We’re using the CanCanCan authorization gem to control access to resources in our current Rails project. It’s a great way to get started and covers a lot of ground with its basic and extended syntaxes, and it has decent support for adding more complex permissions checks that may not fit into more conventional patterns.

It’s very easy to express rules, like granting users the ability to manage their own User record:

can :manage, User, id:

Indirect resource relationships can be traversed intuitively, say, to restrict WorkSchedule management solely to the owner of a Location:

can :manage, WorkSchedule, work_schedule: { location: { user_id: } }

However, when it came time for us to support dynamic, domain-level permissions for the application, we needed slightly more complex, tailored code to get answers. For example, if we want to know whether a given User is permitted to manage pricing data when setting up Promotions at a given Location, we can figure that out by retrieving the relationship between user and location, and querying that relationship for the proper permission:

relation = LocationUser.where(user: user, location: location)

relation = if relation.nil?

This code answers the compound question: “Does this user have a managing relationship with the given Location, and if so, is she permitted to manage pricing?”

Some problems with this code:

  • It’s big, clunky, and repetitive. This is shorter, but not better:
    Manager.where(user: user, location: location) ||
  • It exposes details unnecessarily: users and locations are linked via a Manager class, how to find Managers properly in the database, and that permissions are accessed directly on the Manager objects.
  • Requires careful reading to determine the intent and function.

Contrast with the following equivalent code:


This code is concise and clear: “Is the user allowed to manage pricing at the promotion’s location?” We’re also decoupled from the details of the LocationUser relationship.

Extend the User model by implementing the #managing method:

class User < ActiveRecord::Base

def managing(location)
managed_location(location) or
# Helper method:
def managed_location(location)
Manager.where(user: self, location: location).first

Leverage #managing in your Ability file when expressing complex permissions that don’t fit the mold of straightforward user-resource access control:

class Ability

include CanCan::Ability
def initialize(user)
can [:show,:update], StoreHours do |store_hours|
currently = user.managing(store_hours.location)
currently.can_open_store? or currently.can_set_schedules?
can :manage, Promotion do |promotion|

Though we first used this technique to improve the code in our Ability class, the ideas and code are not bound to CanCanCan, and stand alone as a neat little trick to improve our code in any context.

The post Managing Complex Permissions in CanCanCan for Rails appeared first on Atomic Spin.

Deploying from Git with Capistrano

Justin and I provide operational support to the SME Toolkit project, an education portal for small to medium sized enterprises in developing countries sponsored by the IFC (which is the private sector development branch of the World Bank Group).

Recently, the source code for the Rails-based web application was migrated from Subversion to Git. This also changed how we deploy the application. Previously, we deployed a snapshot of code from a tarball placed on a bastion server. With a few changes to our Capistrano configuration, we are able to deploy directly from the source code repository.

Basic Deployment

First we switch from

set :scm, :none
set :scm, :git

and from

set :repository, "/path/to/unzipped/snapshot"
set :repository, "git@server:project/repository.git"

Then we’ll also want to use set :deploy_via, :remote_cache so that we’ll only need to pull down the commits since the last deploy each time rather than cloning the repository and pulling down the entire history every time.

Now we can deploy using a command like:
cap stage_name deploy -S branch=master

Deploying a Specific Tagged Version

But what if we want to deploy a specific tagged version? We should be able to use cap stage_name deploy -S tag=3.2.1.

Unfortunately, it’s the ‘branch’ variable that’s used as a ref by Capistrano internally to resolve the revision deployed. This is not well documented, but see: lib/capistrano/recipes/deploy/scm/base.rb:77, and lib/capistrano/recipes/deploy/scm/git.rb:119, and lib/capistrano/recipes/deploy.rb:L29.

The simplest way to address this is just to set :branch, tag in config/deploy.rb.
In order to run other tasks without specifying a tag however, we should probably qualify that with a conditional: set :branch, tag if exists?(:tag).

Updating the Version Number

There’s one last thing to be updated: the version number. Before, the developers would populate the REVISION file with a version number when creating a release tarball. Now the REVISION file is populated by Capistrano, with the SHA of the commit being deployed.

If we substitute a new file, say BUILD_VERSION for REVISION in environment.rb, then we can again control of the version number displayed in various places throughout the application. Rather than a long seemingly meaningless string of letters and numbers, we’d like to include the date the code was deployed and the semantic version number of the code.

We can do this with a small Capistrano task like:

  task :build_version, :except => { :no_release => true } do
    deploy_date ='%F')
    build_version = "#{deploy_date} #{tag}.build+#{current_revision}"

If we put this task in the deploy namespace, we can ensure that it’s run at the end of each deployment with something like: after "deploy:create_symlink", "deploy:build_version"

Now when we deploy, the date and the tag are prepended to the SHA for a much more meaningful build version.

The post Deploying from Git with Capistrano appeared first on Atomic Spin.

Beyond Bundler: A Configuration Management Starter Kit

Configuration management or “infrastructure as code” can provide a common language for application developers and operations specialists alike to describe the infrastructure requirements of an application. By capturing these requirements in code, bootstrapping becomes a repeatable process, and insights from operations teams supporting the application in a production environment can be fed back to the developers in a virtuous cycle.

As an example of what this might look like with some current tools, I’ve created a starter kit for using vagrant, veewee, and a bit of puppet to automate the building of virtualized infrastructure for a Rails 3 application. The end result is a VirtualBox virtual machine described in code (from a Veewee basebox definition of the basic virtual hardware to a Puppet manifests describing the necessary packages and bootstrapping). This means that down the road, an environment in which your application will run can be repeatedly built and all of the steps of that process are both visible and modifiable, with changes captured in source control.

Find it on GitHub, here.

There are a few things I haven’t finished wiring together as of this writing, but it should be enough to see how the main pieces fit together.

The project makes as few assumptions as possible about your environment. It assumes that you have a recent version of VirtualBox installed, RVM installed, and ruby-1.9.3-p125 installed via RVM – from there the project rvmrc and bundler should take care of the remaining dependencies.

To run it (build a new VM from scratch and deploy your app to it), you’ll want to run the following commands:
vagrant basebox build demo-centos-box
vagrant basebox validate demo-centos-box
vagrant basebox export demo-centos-box
vagrant up
cap environment:vagrant deploy:setup
cap environment:vagrant deploy

As of this writing, I haven’t added the hooks to actually launch the application server, but you can start WEBBrick by hand like so:
vagrant ssh
…and then from within the VM:
cd sites/demo-app/current && RAILS_ENV=production bundle exec rails s

Then, in your browser visit Tada!

This was a weekend project, and it’s likely that I’ve overlooked some things, but I plan to continue honing it. Let me know in the comments (or in pull requests) what’s still broken. One of the major motivations of this approach is getting past the issue of “works for me” – so if it doesn’t work for you, I want to know! Thanks.

The post Beyond Bundler: A Configuration Management Starter Kit appeared first on Atomic Spin.