Category Archives: ruby

Uploading Files in Rails Using Paperclip and Active Admin

I recently came across a situation where I needed to be able to upload a file to a Rails server with Active Admin. I did a quick search on Google and found this post by Job, a fellow Atom.

Our use cases were a little bit different, though. He was storing the file contents directly in the database, whereas I needed to be able to uplaod a firmware image file to the server’s filesystem, parse the file name, and perform some validations on the file. I decided to use the Paperclip gem to manage the file processing and storage. Using Job’s advice on Active Admin file uploads, I expanded his example to incorporate Paperclip.

What is Paperclip?

Paperclip, as its name suggests, is a file attachment library for ActiveRecord. It is designed to treat files much like other attributes, and it provides a whole slew of built-in extensions (e.g. validations, pre/post processing callback hooks, security enhancements, etc.). The setup is very simple, and getting the basics up and running takes only a few minutes. For the sake of this example, we will set up the ActiveRecord model, create a migration, and get rolling with the Active Admin file upload.

Install the Paperclip Gem

Installing the gem is easy. Just add

gem 'paperclip'

to your Gemfile and run

bundle install

Create a Migration

If you have an existing model where you want to add Paperclip support, then you simply need to create a migration. In this case, I already had a table and a corresponding model for my firmware image, so I just needed to add a migration to add the columns that Paperclip requires. The easiest way to do this is with the Rails migration generator:

rails generate paperclip firmware image

This will automatically create a migration file that looks like this:


class AddImageColumnsToFirmware < ActiveRecord::Migration
  def up
    add_attachment :firmware, :image
  end
  def down
    remove_attachment :firmware, :image
  end
end

The add_attachment helper will automatically create the following columns in your table:

  • image_file_name
  • image_content_type
  • image_file_size
  • image_updated_at

Technically, only the *_file_name column is required for Paperclip to operate, so you could throw away the others if you don’t need them.

Paperclip Your Models

The true beauty of Paperclip is how well it integrates with ActiveModel. To add Paperclip support to your model, start by adding the following line to your model class:


class Firmware < ActiveRecord::Base
  has_attached_file :image
  # more stuff to come
end

With that one line, Paperclip is now integrated with your Rails model, and it will automatically handle your file “attachments” just like any other Rails attribute! There are, of course, plenty of other options that you can add to the has_attached_file attribute (e.g. specifying a file path, style type, etc.), but I won’t go into that right now. For our purposes, the defaults should be just fine.

Validations!

Paperclip also makes it really easy to perform powerful validations on the file. If any validation fails, the file will not be saved to disk and the associon will be rolled back, just like any other ActiveRecord validation. There are built-in helpers to validate attachment presence, size, content type, and file name.

In our case, we really just need to validate the file name to ensure that it had the proper format and also that it was a unique file name. The following validation did the trick:


validates_attachment_file_name :image, :matches => [/_d+_d+_d+.bin$/]
validates_uniqueness_of :image_file_name # this is a standard ActiveRecord validator

Additional Processing

I also wanted to be able to grab the firmware version out of the file name. The best way to do this is with a before_post_process callback in the model, like this:


before_post_process :parse_file_name
def parse_file_name
  version_match = /_(?d+)_(?d+)_(?d+).bin$/.match(image_file_name)
  if version_match.present? and version_match[:major] and version_match[:minor] and version_match[:patch]
    self.major_version = version_match[:major]
    self.minor_version = version_match[:minor]
    self.patch_version = version_match[:patch]
  end
end
  

Before the file is saved, but after the filename is validated (so we can be sure it has the proper formatting), we extract the major, minor, and patch numbers from the filename and save them in our database.

Configure Active Admin for File Upload

Now that Paperclip is all set up and wired into our Rails model, we need to actually set up the file upload piece in Active Admin. I won’t go into much detail, since I relied on Job’s post as a reference. Basically, all we need to do is:

  1. Define the Index page contents

    Enumerate which columns will be displayed. If any special decoration is required, such as customized column title, sort properties, or row formatting, they can be specified easily. For example, I wanted a link to download the firmware image, so I added a link_to in the “Image” column. Note that the file path is stored in the image.url attribute.

  2. Specify which parameters may be changed

    Use the permit_params method to whitelist any attributes.

  3. Create the upload form

    Use the f.input :image, as: :file syntax to automatically create a file upload field in Active Admin.

The code snippet below is the Active Admin page, which allows the user to create, view, edit, and delete a firmware image.


ActiveAdmin.register Firmware do
  permit_params :image
  
  index do
    selectable_column
    id_column
    column 'Image', sortable: :image_file_name do |firmware| link_to firmware.image_file_name, firmware.image.url end
    column :image_file_size, sortable: :image_file_size do |firmware| "#{firmware.image_file_size / 1024} KB" end
    column "Version" do |firmware| "#{firmware.major_version}.#{firmware.minor_version}.#{firmware.patch_version}" end
    column :created_at
    actions
  end
  form do |f|
    f.inputs "Upload" do
      f.input :image, required: true, as: :file
    end
    f.actions
  end
end

And that’s it! We now have a fully functional file-upload implementation using Active Admin backed by Paperclip. It’s really quite simple, and it only took a few minutes to get it set up and running.

The post Uploading Files in Rails Using Paperclip and Active Admin appeared first on Atomic Spin.

Interacting with the Puppet CA from Ruby

I recently ran into a known bug with the puppet certificate generate command that made it useless to me for creating user certificates.

So I had to do the CSR dance from Ruby myself to work around it, it’s quite simple actually but as with all things in OpenSSL it’s weird and wonderful.

Since the Puppet Agent is written in Ruby and it can do this it means there’s a HTTP API somewhere, these are documented reasonably well – see /puppet-ca/v1/certificate_request/ and /puppet-ca/v1/certificate/. Not covered is how to make the CSRs and such.

First I have a little helper to make the HTTP client:

def ca_path; "/home/rip/.puppetlabs/etc/puppet/ssl/certs/ca.pem";end
def cert_path; "/home/rip/.puppetlabs/etc/puppet/ssl/certs/rip.pem";end
def key_path; "/home/rip/.puppetlabs/etc/puppet/ssl/private_keys/rip.pem";end
def csr_path; "/home/rip/.puppetlabs/etc/puppet/ssl/certificate_requests/rip.pem";end
def has_cert?; File.exist?(cert_path);end
def has_ca?; File.exist?(ca_path);end
def already_requested?;!has_cert? && File.exist?(key_path);end
 
def http
  http = Net::HTTP.new(@ca, 8140)
  http.use_ssl = true
 
  if has_ca?
    http.ca_file = ca_path
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER
  else
    http.verify_mode = OpenSSL::SSL::VERIFY_NONE
  end
 
  http
end

This is a HTTPS client that uses full verification of the remote host if we have a CA. There’s a small chicken and egg where you have to ask the CA for it’s own certificate where it’s a unverified connection. If this is a problem you need to arrange to put the CA on the machine in a safe manner.

Lets fetch the CA:

def fetch_ca
  return true if has_ca?
 
  req = Net::HTTP::Get.new("/puppet-ca/v1/certificate/ca", "Content-Type" => "text/plain")
  resp, _ = http.request(req)
 
  if resp.code == "200"
    File.open(ca_path, "w", Ob0644) {|f| f.write(resp.body)}
    puts("Saved CA certificate to %s" % ca_path)
  else
    abort("Failed to fetch CA from %s: %s: %s" % [@ca, resp.code, resp.message])
  end
 
  has_ca?
end

At this point we have the CA and saved it, future requests will be verified against this CA. If you put the CA there using some other means this will do nothing.

Now we need to start making our CSR, first we have to make a private key, this is a 4096 bit key saved in pem format:

def write_key
  key = OpenSSL::PKey::RSA.new(4096)
  File.open(key_path, "w", Ob0640) {|f| f.write(key.to_pem)}
  key
end

And the CSR needs to be made using this key, Puppet CSRs are quite simple with few fields filled in, can’t see why you couldn’t fill in more fields and of course it now supports extensions, I didn’t add any of those here, just a OU:

def write_csr(key)
  csr = OpenSSL::X509::Request.new
  csr.version = 0
  csr.public_key = key.public_key
  csr.subject = OpenSSL::X509::Name.new(
    [
      ["CN", @certname, OpenSSL::ASN1::UTF8STRING],
      ["OU", "my org", OpenSSL::ASN1::UTF8STRING]
    ]
  )
  csr.sign(key, OpenSSL::Digest::SHA1.new)
 
  File.open(csr_path, "w", Ob0644) {|f| f.write(csr.to_pem)}
 
  csr.to_pem
end

Let’s combine these to make the key and CSR and send the request to the Puppet CA, this request is verified using the CA:

def request_cert
  req = Net::HTTP::Put.new("/puppet-ca/v1/certificate_request/%s?environment=production" % @certname, "Content-Type" => "text/plain")
  req.body = write_csr(write_key)
  resp, _ = http.request(req)
 
  if resp.code == "200"
    puts("Requested certificate %s from %s" % [@certname, @ca])
  else
    abort("Failed to request certificate from %s: %s: %s: %s" % [@ca, resp.code, resp.message, resp.body])
  end
end

You’ll now have to sign the cert on your Puppet CA as normal, or use autosign, nothing new here.

And finally you can attempt to fetch the cert, this method is designed to return false if the cert is not yet ready on the master – ie. not signed yet.

def attempt_fetch_cert
  return true if has_cert?
 
  req = Net::HTTP::Get.new("/puppet-ca/v1/certificate/%s" % @certname, "Content-Type" => "text/plain")
  resp, _ = http.request(req)
 
  if resp.code == "200"
    File.open(cert_path, "w", Ob0644) {|f| f.write(resp.body)}
    puts("Saved certificate to %s" % cert_path)
  end
 
  has_cert?
end

Pulling this all together you have some code to make keys, CSR etc, cache the CA and request a cert is signed, it will then do a wait for cert like Puppet does till things are signed.

def main
  abort("Already have a certificate '%s', cannot continue" % @certname) if has_cert?
 
  make_ssl_dirs
  fetch_ca
 
  if already_requested?
    puts("Certificate %s has already been requested, attempting to retrieve it" % @certname)
  else
    puts("Requesting certificate for '%s'" % @certname)
    request_cert
  end
 
  puts("Waiting up to 120 seconds for it to be signed")
  puts
 
  12.times do |time|
    print "Attempting to download certificate %s: %d / 12r" % [@certname, time]
 
    break if attempt_fetch_cert
 
    sleep 10
  end
 
  abort("Could not fetch the certificate after 120 seconds") unless has_cert?
 
  puts("Certificate %s has been stored in %s" % [@certname, ssl_dir])
end

Accessing Google Sheets with Ruby and Google Apps Script

For a recent project, I wrote a Ruby script to upload my team’s hours to a Google Spreadsheet. From time to time, I would have to manually edit the hours on the spreadsheet, so I left a note on those cells to remind myself that I changed the values. I needed my script to see these notes and know not to overwrite these cells. Unfortunately, the gem I usually use did not support access to these notes. I decided to look into how I could build my own Google Sheets API that my Ruby script could use, giving me a bit more flexibility for any other Ruby scripts I might create.

Creating an API

I ended up using Google Apps Script to create a fairly simple API for this task. This scripting language is similar to JavaScript, but it has access to various Google services, such as Google Docs and Google Sheets.

Once I created a new Google Apps Script file, I made the following endpoint:


function getCellNote(url, sheetIndex, row, column) {
  var ss = SpreadsheetApp.openByUrl(url);
  var sheet = ss.getSheets()[sheetIndex];
  var cell = sheet.getRange(row, column);
  return cell.getNote();
}

I used SpreadsheetApp to access the spreadsheet via url. Since the spreadsheet contains multiple sheets, this let me specify which particular sheet I want to edit. Within that sheet, I can also specify the row and column of the cell where I left my note.

After coding the API, I selected “Deploy as API Executable…” from the Publish menu. When you do this, a modal appears and guides you through deploying the API.

Authorizing

Next, I needed to enable my script and gather the proper credentials to access the endpoint. Detailed instructions can be found on Google’s quickstart guide, but there were three things I needed to do:

  1. Access the Developers console from the app script using the Resources menu.
  2. Turn on the API from the “Enable API” tab.
  3. Create credentials for the app and download the client_secret.json.

For the Ruby side, I needed to install the Google API Client gem:

gem install google-api-client

In order to access a user’s sheet on their Google Drive, the script has to gather the user’s credentials. Google provides Ruby code for this in their quickstart guide. I put this same code below, with a few changes to allow access to Google Spreadsheets as well as Google Drive.


require 'google/apis/script_v1'
require 'googleauth'
require 'googleauth/stores/file_token_store'
require 'fileutils'
OOB_URI = 'urn:ietf:wg:oauth:2.0:oob'
CLIENT_SECRETS_PATH = 'path/to/client_secret.json'
CREDENTIALS_PATH = File.join(Dir.home, '.credentials',
                             "my-app-credentials.yaml")
SCOPE = ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/spreadsheets']
def authorize
  FileUtils.mkdir_p(File.dirname(CREDENTIALS_PATH))
  client_id = Google::Auth::ClientId.from_file(CLIENT_SECRETS_PATH)
  token_store = Google::Auth::Stores::FileTokenStore.new(file: CREDENTIALS_PATH)
  authorizer = Google::Auth::UserAuthorizer.new(client_id, SCOPE, token_store)
  user_id = 'default'
  credentials = authorizer.get_credentials(user_id)
  if credentials.nil?
    url = authorizer.get_authorization_url(base_url: OOB_URI)
    puts "Open the following URL in the browser and enter the " +
         "resulting code after authorization"
    puts url
    code = gets
    credentials = authorizer.get_and_store_credentials_from_code(
      user_id: user_id, code: code, base_url: OOB_URI)
  end
  credentials
end

This prompts the user of the script with a url to follow and verify a Google account. A few key things to note:

  • The client_secret.json generated from the Developers Console should be stored somewhere convenient. I put mine in a creds folder in the same directory as my script.
  • SCOPE defines what we need authorization for. In this case, I want access to both Google Drive and Google Spreadsheets, so I added the appropriate Google API urls to that list.

Using the API Endpoints

After all that setup, it’s finally time to use the endpoints. In the same Ruby file as the authorization method, I added these four lines to specify which API the script should access:


SCRIPT_ID = 'MDsPuc46EcIuUIt77LQ6LC3Ac4yVd_8hJ'
SERVICE = Google::Apis::ScriptV1::ScriptService.new
SERVICE.client_options.application_name = "My Application Name"
SERVICE.authorization = authorize

(I got the SCRIPT_ID from the url of my Google Apps script.)

Finally, I could use the endpoint defined in the app script. Here’s the method I used to wrap the request to the endpoint:


def get_cell_note(spreadsheetUrl, sheetNum, cell)
  request = Google::Apis::ScriptV1::ExecutionRequest.new(
    function: 'getCellNote',
    parameters: [ spreadsheetUrl, sheetNum, cell ]
  )
  begin
    response = SERVICE.run_script(SCRIPT_ID, request)
    if response.error
      # Handle the error...
    else
      return response.response['result']
    end
  rescue Google::Apis::ClientError
    # Handle the error...
  end
end

Keep in Mind

A few things to note here:

  • function corresponds to the name of the function found in the Apps script.
  • parameters corresponds to the parameters for the function in the Apps script, so remember to keep the same order!
  • Make sure to handle the errors. I left the implementation details out for them, but it might be nice to either notify the user or log the issue, depending on your situation.

And that’s it! I can now access the notes on my Google Spreadsheets. By setting up this API, I can also create other endpoints, allowing my Ruby scripts to access all of my Google docs.

The post Accessing Google Sheets with Ruby and Google Apps Script appeared first on Atomic Spin.

Monadt – Algebraic Data Types and Monads in Ruby, Part 2: Monads

In yesterday’s post, I introduced monadt, a gem that adds algebraic data types (ADTs) and monads to Ruby. Today I’m going to dive into how monadt provides monad support, specifically the imperative-looking syntactical sugar you get in languages like Haskell and F#.

I’m not going to cover how monads work in this post, but I will suggest reading either (or both) of the following:

  1. Learn You a Haskell: A Fistful of Monads
  2. F# for Fun and Profit: Dr. Frankenfunctor and the Monadster

Just the Basics

Monadt defines an API for defining and using monads in Ruby, and it includes several common monads out of the box: Maybe, Either, Reader, State, Async, AsyncEither, and ReaderStateEither.

Let’s take a closer look at the AsyncEither monad. Briefly, the Either data structure represents having one of two values: Left X or Right Y. In a monadic context, Either is usually used for the choice between a success value and an error value. In monadt, Either is defined as follows using monadt’s ADT syntax:


class Either
  Left = data :left
  Right = data :right
end

When you use Either in a monad, you perform a series of calculations, opting out early if any of the steps returns an error value.

The Async monad is designed to streamline performing sequences of asynchronous operations. Basically, it lets you write imperative-looking asynchronous code instead of using a bunch of continuation blocks.

Putting these concepts together, we can use an AsyncEither monad, which asynchronously retrieves either a success value or a failure value. If any asynchronous step fails, the failure value at that point will be returned.

To see this in practice, imagine a method get_json in Ruby that returns a Fiber which will return the JSON contents of a URL as a Hash (when accessed with Fiber.resume). Given such a method, we might construct a sequence of operations as follows:


def get_bookface_spouse_data(user)
  AsyncEither.bind (get_url "www.mydata.com/user-info?uid=#{user}") do |profile|
    AsyncEither.bind (get_url "www.bookface.com/profile/#{user_info[:bookface_id]}") do |spouse_id|
      AsyncEither.bind (get_url "www.mydata.com/user-info?bfid=#{profile[:spouse_id]}") do |spouse_data|
        AsyncEither.return data
      end
    end
  end
end

If the request to mydata.com for uid user fails, get_url will return Left "url could not be retrieved #{message}". Consequently, get_url will not be called the second time (using the bookface.com URL) and the method will complete with the Left value.

However, if the first call succeeds, get_url will return Right profile where profile will be passed into the block given to AsyncEither.bind() (where the block has the signature T -> AsyncEither<U>, that is T -> Fiber<Either<U>>). The block given to AsyncEither.bind() in our example extracts the value from key :bookface_id in the user_info to retrieve the bookface profile data.

If this fails, then again you will short-circuit early with a Left value containing error information. If it succeeds, the user’s bookface profile will be used to retrieve their spouse’s ID, which is then used to locate the stored data for a spouse.

While the above code clearly demonstrates the way nested bind() calls work, I find it hard to read. Haskell and F# provide syntactical sugar to make the code look more imperative, which greatly improve readablity. Fortunately, there is a way to emulate this syntactical sugar in Ruby:


def get_bookface_spouse_data2(user)
  Monad.async_either do |m|
    user_info = m.bind (get_url "www.mydata.com/user-info?uid=#{user}")
    profile = m.bind (get_url "www.bookface.com/profile/#{user_info[:bookface_id]}")
    spouse_data = m.bind (get_url "www.mydata.com/user-info?#bfid={profile[:spouse_id]}")
    m.return spouse_data
  end
end

get_bookface_spouse_data2() is equivalent to get_bookface_spouse_data(), but it looks imperative. We’ll take a look at how this works later, but first let’s see how to define your own monadic types.

Defining Additional Monads

To define your own monad, define bind() and return() as class methods:


class MyMonad
  class << self def bind(ma, &blk) # blk is a lambda/proc of signature a -> mb
      # use ma and blk to return something of type mb
    end
    
    def return(a)
      # turn a into ma
    end
  end
end

Then you can call Monadt::Monad.do_m()


Monadt::Monad.do_m(MyMonad) do |m|
  x = m.bind method1(5)
  y = m.bind method2("hello", x)
  m.return (x + y)
end

do_m() takes as its argument a Class which is presumed to have bind() and return() defined. It also takes a block which is the monadic context to execute. The input argument to the block is a special object that knows how to call bind() and return(). The method Monad.async_either() used in our example is just a convenience method that calls Monad.do_m(AsyncEither).

Under the Hood

The do_m() method uses Ruby’s Fiber and a pattern I call “yield abuse” in order to mimic the monadic syntactical sugar you see in Haskell or F#. Fibers let you pass control back and forth between the monad implementation code and the user code, with the monad implementation code having the option to opt out early. By stringing the Fiber block execution through a series of recursive lambdas, we can create a code flow that looks like standard monadic execution. You can pull off the same trick in JavaScript.

Function-Style Functors

Many of the standard monad types have functions as the monad/functor type that wraps the interior type. So for example, the state monad type m a is traditionally a function that takes in a state, and returns a tuple of a generated value and an updated state: state -> output * state. This is going to look pretty nasty in Ruby unless you find a way to turn normal methods into Procs (the closest thing Ruby has to a function object). Fortunately there is the funkify gem, which I’ve written about before. This handy gem will let you turn standard Ruby methods into Procs that can be partially applied, e.g. so you can have a method


def add(x,y)
  x + y
end

and partially apply it like so:


add_two = add(2) # add_two is a Proc taking one argument

Using funkify, you can generate pretty decent-looking state monad blocks:


def state_func_1(arg, state)
  #...
  [value_1, updated_state_1]
end
def state_func_2(arg1, arg2, state)
  # ...
  [value_2, updated_state_2]
end
Monad.run_state(initial_value) do |m|
  x = m.bind (state_func_1 arg1)
  y = m.bind (state_func_2 arg2, x)
  m.return x * y
end

No Multiverses…Yet

One significant limitation of the current version of monadt is that the List monad is not supported by do_m(). The list monad requires the ability to retry bind functions for different inputs, which would require a partial rewind through the Fiber block. Unfortunately, this is not supported by Ruby. My coworker Job and I have some ideas of how to get around this, but we haven’t implemented it yet.

Conclusion

I was pleasantly surprised at how well monadic control flow can be expressed in Ruby. The dynamic typing and monkey patching, fraught with danger though they may be, actually make it possible to implement complex features like typeclasses in Ruby. And Fibers give you a lot of flexibility in how a code block executes. I encourage you to consider monadt for projects that contain structured data you might otherwise express loosely as a Hash or an Array. And even if monadt doesn’t make sense to include in your particular project, I encourage you to dig deeper into the control flow features of Ruby and see how you might be able to leverage them to streamline the logic in your application. Happy (slightly more functional) programming!

The post Monadt – Algebraic Data Types and Monads in Ruby, Part 2: Monads appeared first on Atomic Spin.

Monadt – Algebraic Data Types and Monads in Ruby, Part 1: ADTs

Functional programming is elegant and expressive. I’ve written before about my love of partial application, and how the funkify gem can be used to bring the power of partial application to your Ruby code. But partial application is just one of the powerful idioms from functional languages that I’d like to borrow in object-oriented languages. I’m also pretty into algebraic data types and monads.

So, continuing my pattern of adding functional concepts to object-oriented languages whether they like it or not, I recently created the monadt gem which adds support for using algebraic data types and monads to Ruby. This library gives you these powerful features while attempting to be as readable and Ruby-like as possible.

In this post, I’m going to explain how the monadt library provides algebraic data types. I will cover its use of monads in a future post.

Algebraic Data Types

An algebraic data type (or discriminated union), is like a C union which also maintains which value of the union should be used. Alternatively, you can think of ADTs like enums with data values attached to the different enum values. So for example, if I wanted to create an ADT that captures the state of a car’s gear and velocity, it might look like this:


type GearSpeed =
  | Park
  | LowGear of int * double
  | Drive of double

where Park has no additional data (we are in the parked gear and not moving), LowGear tracks both the forced low gear and our current velocity, and Drive indicates that we are in the regular drive gear at a certain speed. Some possible values:


Park              // parked
Drive 45.0        // drive, 45 mph
LowGear (1, 10.0) // first gear, 10 mph
Drive 55.2        // drive, 55.2 mph
LowGear (2, 20.0) // second gear, 20 mph

What’s so powerful about ADTs is that you can use them to expressively define all possible states. I cannot create a GearSpeed without specifying one of the three cases, and I cannot specify one of the cases without providing its required values. Additionally, languages with ADTs define powerful pattern matching semantics to quickly identify which case you are in and extract the values.


match aGearSpeed with
| Park -> "stopped"
| LowGear (gear, speed) -> sprintf "%d gear, %f mph" gear speed
| Drive speed -> sprintf "drive, %f mph" speed

ADTs also allow you to specify default cases, and they check at compile time to make sure you have covered all the cases when you branch.


let isStopped aGearSpeed =
  match aGearSpeed with
  | Park -> true
  | _ -> false
// will not compile, missing Park case
let getSpeed aGearSpeed =
  match aGearSpeed with
  | LowGear (_, speed) -> speed
  | Drive speed -> speed

monadt

So where does Ruby come in? As a dynamically typed language, we can’t formally restrict the types associated with data, and we can’t guarantee all cases have been covered at compile time. However, we can define ADT such that pattern matching will indicate which case we are and easily extract the correct values.

The monadt library (get it?) builds on the code from this blog post by fellow Atom Job Vranish, but it uses functional/functor map style pattern matching, instead of taking a lambda intended to have side effects.

Here’s an example:


class GearSpeed
  Park = data
  LowGear = data :gear, :speed
  Drive = data :speed
end

This code declares an ADT GearSpeed with the same meaning as the original definition in F#. It has three different values: Park, LowGear, and Drive. Park has no associated data, LowGear has two pieces of associated data (“gear” and “speed”), and Drive has one (“speed”). We can’t restrict the types of those values, but we can name them for future refernce.

You can create new instances with the standard Ruby constructor for the value type:


GearSpeed::LowGear.new 3, 21.0  # gear = 3, speed = 21.0 mph

Given an ADT value, we can do pattern matching using monadt’s match() and with() functions:


match val,
  with(GearSpeed::Park) { "stopped" },
  with(GearSpeed::LowGear) { |gear, speed| "#{gear} gear, #{speed} mph" },
  with(GearSpeed::Drive) { |speed| "drive, #{speed} mph" }

The above code takes an ADT value and pattern matches by type, taking a lambda which will yield the values associated with the particular ADT case. You can also do default matching, using the “Default” type in monadt instead of the _ wildcard (as you would in F# or Haskell):


match val,
  with(GearSpeed::Park) { true },
  with(Default) { false }

Decorating ADTs

But wait, there’s more! To handle other common usages, and to shorten the length of code to create ADT values, monadt includes a method decorate_adt().


decorate_adt GearSpeed

Calling this method on an ADT type adds the following:

  1. constructor functions based on the lowercase ADT value name, attached to the parent type:
    
    GearSpeed.drive 50.0 # same as GearSpeed::Drive.new 50.0
    GearSpeed.park # same as GearSpeed::Park.new
    GearSpeed.low_gear 2, 10.5 # same as GearSpeed::LowGear.new 2, 10.5
    
  2. is_###? functions for each case (same lowercase naming as the constructor):
    
    gearspeed_value.is_park?
    gearspeed_value.is_drive?
    gearspeed_value.is_low_gear?
    
  3. sensible default to_s()
    
    gearspeed_value.to_s # "Drive(36.0)", "Park", or "LowGear(2, 12.5)"
    

Moving on to Monads

That covers the basics of using ADTs with monadt. In my next post, I’ll talk about how monadt uses Ruby Enumerators to simulate the syntactic sugar associated with monads in Haskell or F#, particularly focusing on Maybe and Either monads—two types that can be implemented as ADTs.

The post Monadt – Algebraic Data Types and Monads in Ruby, Part 1: ADTs appeared first on Atomic Spin.

     

Windows Socket Functions in Ruby Extensions

If you want to use Windows socket functions (such as WSARecv, and WSASend) inside a Ruby extension on Windows, you may have your work cut out for you. Here’s a helpful work-around I discovered during a recent project.

Spotting the Problem

I was working on an improvement to the rENet library, which is a Ruby binding to the excellent ENet library. Previously, building the extension required preinstalling the ENet library on your system. But ENet is a nice, small library with no other dependencies, so I thought I’d try to build it into the extension. This would be much more convenient to users of the library, as the Ruby extension builder would build ENet automatically without requiring pre-installation.

My first attempt worked great on Linux and OSX systems, but to my bafflement, it failed in spectacular ways on Windows systems. After a little debugging, I found that most operations would work fine, but some would give:

WSAERROR 10038 : (WSAENOTSOCK): Socket operation on non socket.

What was especially strange is that, with no code changes, everything would work great if I just linked to an ENet library that was built externally—not by the Ruby extension builder. It appeared that the Ruby extension builder was messing up some, but not all, of my socket operations.

Some googling and the headers in the extension builder source hinted that Ruby was creating its own wrappers for socket functions. But I had to dig all the way down into the Ruby source code to figure out what was actually going on and how I could fix it.

Finding the Cause

Internally, Ruby wraps the Windows socket functions with its own functions. These wrappers replace the normal socket handles with a homemade “file descriptor,” which is managed internally by Ruby.

For an example of what I’m talking about, search for: rb_w32_socket here.

I’m assuming this is done to make the internal APIs more consistent across OSs. For example, it lets you use select() to block on either a file or socket IO event in the same call. This is a common technique on POSIX systems, but is normally not possible on Windows.

The problem arises when this fake socket handle gets mixed in with an unwrapped function that expects a real one, or vice versa. That’s what caused my WSAENOTSOCK error. The function was expecting a real socket, but got the fake handle instead.

Unfortunately, not only does Ruby wrap the socket functions; it also replaces the symbols for the wrapped functions at the linker level so it is nearly impossible to use the native Windows socket functions.

Choosing the Best Solution

Once you understand what’s going on, there are two potential solutions to this problem.

  1. Make your own wrappers for the functions that are causing problems.
  2. Use an ugly hack to go around Ruby and use the native functions.

If your extension needs to share socket handles with other Ruby code, you have no choice and must use Option 1. Unfortunately, this is the harder option. You’re pretty much going to have to dig into the Ruby source and try to copy what they are doing, but for your own wrapper. This might be a good place to start digging around for something close that you can copy.

If your extension does not need to share socket handles with other Ruby code and is going to manage the sockets completely internally, you can use either option. However, Option 2 is probably easier and less error-prone.

Here’s the code I used to get the actual Windows socket syscalls, instead of the wrapped ones provided by Ruby:


bindptr        = (LPFNBIND)       GetProcAddress(GetModuleHandleA("ws232.dll"), "bind");
getsocknameptr = (LPFNGETSOCKNAME)GetProcAddress(GetModuleHandleA("ws232.dll"), "getsockname");
listenptr      = (LPFNLISTEN)     GetProcAddress(GetModuleHandleA("ws232.dll"), "listen");
socketptr      = (LPFNSOCKET)     GetProcAddress(GetModuleHandleA("ws232.dll"), "socket");
ioctlsocketptr = (LPFNIOCTLSOCKET)GetProcAddress(GetModuleHandleA("ws232.dll"), "ioctlsocket");
setsockoptptr  = (LPFNSETSOCKOPT) GetProcAddress(GetModuleHandleA("ws232.dll"), "setsockopt");
getsockoptptr  = (LPFNGETSOCKOPT) GetProcAddress(GetModuleHandleA("ws232.dll"), "getsockopt");
connectptr     = (LPFNCONNECT)    GetProcAddress(GetModuleHandleA("ws232.dll"), "connect");
acceptptr      = (LPFNACCEPT)     GetProcAddress(GetModuleHandleA("ws232.dll"), "accept");
shutdownptr    = (LPFNSHUTDOWN)   GetProcAddress(GetModuleHandleA("ws232.dll"), "shutdown");
closesocketptr = (LPFNCLOSESOCKET)GetProcAddress(GetModuleHandleA("ws232.dll"), "closesocket");
selectptr      = (LPFNSELECT)     GetProcAddress(GetModuleHandleA("ws232.dll"), "select");

Then, simply replace all the calls to functions such as bind and select to calls on appropriate function pointer, such asbind_ptr and select_ptr.

In Ruby 2.0 or higher, you’ll probably need to add a +#define INCL_WINSOCK_API_TYPEDEFS 1 to gain visiblity to the pointer types. It’s a super-gross solution, but it happens to work quite well.

The post Windows Socket Functions in Ruby Extensions appeared first on Atomic Spin.

Using ruby mocha outside of unit testing frameworks

I find myself currently writing a lot of orchastration code that manages hardware. This is very difficult because I like doing little test.rb scripts or testing things out in irb or pry to see if APIs are comfortable to use.

The problem with hardware is in order to properly populate my objects I need to query things like the iDRACs or gather inventories from all my switches to figure out where a piece of hardware is and this take a lot of time and requires constant access to my entire lab.

Of course my code has unit tests and so all the objects that represents servers and switches etc are already designed to be somewhat comfortable to load stub data for and to be easy to mock. So I ended up using rspec as my test.rb environment of choice.

I figured there has to be a way to use mocha in a non rspec environment, and turns out there is and it’s quite easy.

The magic here is line 1 and line 5, including Mocha::API will extend Object and Class with all the stubbing and mocking methods. I’d avoid using expectations and instead use stubs in this scenario.

At this point I’d be dropped into a pry shell loaded up with the service fixture in my working directory where I can quickly interact with my faked up hardware environment. I have many such hardware captures for each combination of hardware I support – and the same data is used in my unit tests too.

Below I iterate all the servers and find their mac addresses of the primary interfaces in each partition and then find all the switches they are connected to. Behind the scenes in real life this would walk all my switches looking for the port each mac is connected to and so forth, quite a time consuming operation and would require me to dedicate this lab hardware to me. Now I can just snapshot the hardware and load up my models later and it’s really quick.

I found this incredibly handy and will be using it pretty much all the time now, so thought it worth sharing :)

6 Months of Avoiding Rails Controllers with DDC

My distaste for unnecessary Rails controllers is no secret. That’s why I wrote the ddc gem (Data Driven Controllers).

When DDC was released, it got mixed reviews in the comments section, so I thought I’d post a follow up with my results thus far.

Example


DDC::ControllerBuilder.build :monkeys, 
  actions: {
    show: {
      context: 'context_builder#user_and_id',
      service: 'monkey_service#find'
    },
    index: {
      context: 'context_builder#user_and_id',
      service: 'monkey_service#find_all'
    },
    update: {
      context: 'context_builder#monkey',
      service: 'monkey_service#update'
    },
    create: {
      context: 'context_builder#monkey',
      service: 'monkey_service#create'
    }
  }

Review

My current project’s Rails API has about 20 controllers, almost all of which are mere data describing how to glue the Rails pieces together. The overall approach has held up well. With decent naming conventions and DDC, I barely ever have to think about Rails controllers. When I did need a custom bit of something, DDC let me define the methods that actually required some thought. So far, working with DDC has been great: easy to use with very little mental overhead.

New Features

There have been a few features added along the way to round out the DDC API:

  1. :render_opts can now be specified. These options let you control which serializer will be used to render domain specific content to JSON. I actually haven’t needed this feature, but other projects using DDC have.
  2. :context will now take an optional array of function strings. The functions will be called in order, being passed the previous functions results and outputting its own. (Like building an onion from the inside-out.)
  3. DDC has been a great tool in my tool belt. I’d really like to see Rails find a clean way to make controllers optional the way Ember.js does.

    I’m curious what others are doing to streamline the building of Rails controllers and Rails sites in general. Leave a comment below.

    The post 6 Months of Avoiding Rails Controllers with DDC appeared first on Atomic Spin.

RSchema-Hamster: Schemas for Shaping Immutable Data

RSchema-Hamster is a Ruby gem that lets you use Hamster’s persistent data structures to define and validate the shape of your data via RSchema.

What for?

In order to clarify my designs, reduce mistakes, and leave code easier to change than when I found it, I:

  • Strive to program with immutable data
  • Learn to apply ideas from type-driven development

Using RSchema-Hamster, I can define a schema such as:

require 'rschema_hamster'
Name = String
PageNumber = Integer
Appearances = Hamster.vector(
  Hamster.hash(character: Name, page: PageNumber)
)

Build a data structure like this:

guide = Hamster.vector(
  Hamster.hash(character: "Arthur", page: 1), 
  Hamster.hash(character: "Zaphod", page: 98)
)

And assert the data conforms to the schema like this:

RSchema.validate!(Appearances, guide)

Immutable Data Structures via Hamster

I prefer to code using primarily immutable values because I’ve come to understand just how careful I need to be when mutating program state: plan for it, isolate it, and generally avoid changing internal object state unnecessarily.

Two up-front challenges have been to change my design habits to favor stateless, functional code (difficult, took lots of practice) and to find a powerful persistent data structures tool (not difficult; for Ruby, I picked Hamster).

The Hamster Ruby gem provides a set of persistent collections to supplement or replace Ruby’s mutable collections, such as Hash, Vector, lazy Lists, and so forth. It looks like this:

ford = Hamster.hash(name: "Ford Prefect", gender: :male)
# => Hamster::Hash[:name => "Ford Prefect", :gender => :male]
arthur = ford.put(:name, "Arthur Dent")
# => Hamster::Hash[:name => "Arthur Dent", :gender => :male]
# (ford remains unchanged)

Getting Lost

As I start moving back to using generic collections like Hashes, Vectors, and Sets to build my data structures, I’m getting a little nervous about inconsistency and accidental misuse. Immutable or not, these collections don’t care what you store in them and will give you no hint as to their intended use. There’s a lot of room for mistakes; this is why I had a hard time letting go of my descriptively-named classes and their attr_accessor declarations.

(A couple years back, Patrick Bacon and I created Hamsterdam as a first step toward alleviating this concern, using immutable record-like objects as a means of defining and documenting attributes.)

Schema-based Definition and Validation via RSchema

Whereas some functional languages like Haskell and F# have very expressive type systems for describing data, dynamic languages like Clojure and Ruby require me to take matters into my own hands. I discovered Tom Dalling’s RSchema (a Ruby implementation of Prismatic’s Schema for Clojure), and started tinkering with ad-hoc schema structures to help clarify and test some of my existing code as I refactored it. I now felt like I could start taking some queues from the Type-Driven crowd.

In RSchema, a schema is itself data, a value that describes the shape of other values:

OrderRow = {
  order_id: Integer,
  order_totals: Totals
}
row = { 
  order_id: 5, 
  order_totals: a_valid_totals_structure 
}
RSchema.validate! OrderRow, row

Read: An OrderRow is a Hash whose order_id key refers to an integer value, and whose order_totals key refers to a Totals object, presumably a Ruby class or perhaps another schema object.

It may not be compile-time type-checking, but it sure is expressive, and it gives me the means to programmatically connect variables to their intended types. I can easily discover and understand the data’s shape by reading the code, and at runtime (especially during automated unit and integration testing) assert that the data conforms. Executable documentation, y’see.

So if I want to insist that OrderRow is an immutable Hash, I want to be able to write:

OrderRow = Hamster.hash(
  order_id: Integer,
  order_totals: Totals
)

However, the above schema isn’t valid. In order for an object to be treated as a descriptive instances (in this case a Hash containing sub-schemas), the object must implement a method called “schema_walk”, and Hamster structures don’t.

Extending Hamster to Support RSchema

RSchema defines schemas in terms of classes, Hashes, and Arrays, plus a few configurable schemas available via its DSL, such as “enum,” “maybe,” and an optional Hash-key decorator. Ruby’s Hash and Array classes implement their respective “schema_walk” such that their instances may be used to describe internal structure.

In order to use Hamster instances to describe structure, I provided implementations of “schema_walk” to Hamster::Hash, Hamster::Vector and so on, plus a few generic schemas to give parity to RSchema’s “hash_of” and “set_of” DSL methods (“hamster_hash_of” and “hamster_set_of” respectively).

The big bonus: schemas are designed to be nestable in RSchema. Because of this, schemas I define using Hamster’s types may still contain (or be contained by) Ruby’s Hash and Array structures, as well as utilize (or be used by) RSchema’s “enum,” “maybe,” and “_?” (optional hash key) schema helpers, without any extra implementation.

Here’s an abridged real-world example of how I was able to untangle a document-like structure that was already on its way to generating headaches in a Rails app I’m working on:

require 'rschema_hamster'
module OrderReport::Schema
  Name    = String
  Id      = Integer
  Dollars = BigDecimal
  Totals = Hamster.hash(
    gross: Dollars,
    tax:   Dollars,
    fee:   Dollars,
    net:   Dollars,
  )
  OrderRow = Hamster.hash(
    order_id:     Id,
    order_number: Name,
    order_totals: Totals,
  )
  MarketBlock = Hamster.hash(
    market_id:        Id,
    market_name:      Name,
    account_dropdown: RSchemaHamster.schema {
      hamster_hash_of(Name => Id)
    },
    order_rows:       Hamster.vector(OrderRow),
    market_totals:    Totals
  )
end
def self.dollars(str); BigDecimal.new(str); end
market_block = Hamster.from(
  { 
    market_id: 42,
    market_name: "The Restaurant at the End of the Universe",
    account_dropdown: {
      "Hotblack Desiato" => 1, 
      "Zaphod Beeblebrox" => 3
    },
    order_rows: [
      { order_id: 101, order_number: "MILLIWAYS-00101", order_totals: { gross: dollars("120"), tax: dollars("14.4"), fee: dollars("20"), net: dollars("85.6") } },
      { order_id: 102, order_number: "MILLIWAYS-00102", order_totals: { gross: dollars("3030"), tax: dollars("363.6"), fee: dollars("505.10"), net: dollars("2161.3") } },
    ],
    market_totals: { gross: dollars("3150"), tax: dollars("378"), fee: dollars("525.10"), net: dollars("2246.9") }
  }
) 
RSchema.validate!(OrderReport::HamsterSchema::MarketBlock, market_block)

The post RSchema-Hamster: Schemas for Shaping Immutable Data appeared first on Atomic Spin.

Building an Infinite Procedurally-Generated World

I had a lot of fun writing my last blog post: All Work & No Play – Taking Time to Code for Fun. In it I talked about writing fun code that keeps you interested in programming and keeps you creative. I used the example of writing a 2D procedurally-generated, infinite world. In this post, I am going to explain details of how that example works.

proc_terrain_small

Bring the Noise

To build our terrain, we need something better than just randomly selecting a tile for each set of coordinates. Since we want the world to be infinite, we can’t design it by hand: that’s where Perlin noise comes in. Perlin noise is an algorithm for generating organic looking, multidimensional noise quickly (at least for 2D). You’ll want to know a few terms when playing with Perlin noise:

Octaves
The number of subsequent generations to run, usually double the frequency and half the amplitude.
Persistence
How to weight the additional octaves when adding them.

These can be fun to play with when generating terrain; they will make it smooth and flat or bumpy and spiky. Further explanation can be found here.

Homework assignment: try merging two noise datasets that have been generated with different octave / persistence values.

From Noise to Tiles

Once you have access to random 2D data, it’s pretty straight forward to convert that noise into usable data. Simply set up thresholds for each tile type you want to support, eg:

  • water if < 0.3
  • grass if >= 0.3 and <= 0.6
  • mountain if > 0.6

I recommend leaving the upper and lower cases open. If they are not left open, you may end up with holes in your map for unexpectedly high or low values. An alternative is to clamp or scale the noise values when you generate them.

Going Beyond Tiles

An infinite world of terrain isn’t all that interesting for very long without something in it. We want to add things like trees, caves, and towns to our game. We can simply use the same random number generator (RNG) that we used to generate our noise to determine when to place and how to build objects. We have to be careful though. If we use a single RNG for the whole world, the order in which we discover things will change the how they are generated.

chunks_small

The trick to deterministically creating the world based on a seed is to break it up into chunks (see screenshot above). A chunk is simply a range of tiles in the world. We keep 9 chunks loaded at a time. Each chunk is 50×50 grid of tiles. When the player enters a place with no chunk, we generate a new one. The first thing we do is seed a new RNG for that chunk with the global seed and the coordinates for that chunk. This allows us to throw away the chunk when the player leaves it, but know that it will be recreated the same when the player comes back. Just make sure your code is deterministic and that it only pulls random values from the chunk’s RNG and things should work out well.

The source for this project can be found on github. Please leave a comment if you are doing something similar and want an excuse to show off.

UPDATE:
Example uses tileset from Oryx Design Lab. Thanks!

The post Building an Infinite Procedurally-Generated World appeared first on Atomic Spin.