Category Archives: ruby

Monadt – Algebraic Data Types and Monads in Ruby, Part 2: Monads

In yesterday’s post, I introduced monadt, a gem that adds algebraic data types (ADTs) and monads to Ruby. Today I’m going to dive into how monadt provides monad support, specifically the imperative-looking syntactical sugar you get in languages like Haskell and F#.

I’m not going to cover how monads work in this post, but I will suggest reading either (or both) of the following:

  1. Learn You a Haskell: A Fistful of Monads
  2. F# for Fun and Profit: Dr. Frankenfunctor and the Monadster

Just the Basics

Monadt defines an API for defining and using monads in Ruby, and it includes several common monads out of the box: Maybe, Either, Reader, State, Async, AsyncEither, and ReaderStateEither.

Let’s take a closer look at the AsyncEither monad. Briefly, the Either data structure represents having one of two values: Left X or Right Y. In a monadic context, Either is usually used for the choice between a success value and an error value. In monadt, Either is defined as follows using monadt’s ADT syntax:

class Either
  Left = data :left
  Right = data :right

When you use Either in a monad, you perform a series of calculations, opting out early if any of the steps returns an error value.

The Async monad is designed to streamline performing sequences of asynchronous operations. Basically, it lets you write imperative-looking asynchronous code instead of using a bunch of continuation blocks.

Putting these concepts together, we can use an AsyncEither monad, which asynchronously retrieves either a success value or a failure value. If any asynchronous step fails, the failure value at that point will be returned.

To see this in practice, imagine a method get_json in Ruby that returns a Fiber which will return the JSON contents of a URL as a Hash (when accessed with Fiber.resume). Given such a method, we might construct a sequence of operations as follows:

def get_bookface_spouse_data(user)
  AsyncEither.bind (get_url "{user}") do |profile|
    AsyncEither.bind (get_url "{user_info[:bookface_id]}") do |spouse_id|
      AsyncEither.bind (get_url "{profile[:spouse_id]}") do |spouse_data|
        AsyncEither.return data

If the request to for uid user fails, get_url will return Left "url could not be retrieved #{message}". Consequently, get_url will not be called the second time (using the URL) and the method will complete with the Left value.

However, if the first call succeeds, get_url will return Right profile where profile will be passed into the block given to AsyncEither.bind() (where the block has the signature T -> AsyncEither<U>, that is T -> Fiber<Either<U>>). The block given to AsyncEither.bind() in our example extracts the value from key :bookface_id in the user_info to retrieve the bookface profile data.

If this fails, then again you will short-circuit early with a Left value containing error information. If it succeeds, the user’s bookface profile will be used to retrieve their spouse’s ID, which is then used to locate the stored data for a spouse.

While the above code clearly demonstrates the way nested bind() calls work, I find it hard to read. Haskell and F# provide syntactical sugar to make the code look more imperative, which greatly improve readablity. Fortunately, there is a way to emulate this syntactical sugar in Ruby:

def get_bookface_spouse_data2(user)
  Monad.async_either do |m|
    user_info = m.bind (get_url "{user}")
    profile = m.bind (get_url "{user_info[:bookface_id]}")
    spouse_data = m.bind (get_url "{profile[:spouse_id]}")
    m.return spouse_data

get_bookface_spouse_data2() is equivalent to get_bookface_spouse_data(), but it looks imperative. We’ll take a look at how this works later, but first let’s see how to define your own monadic types.

Defining Additional Monads

To define your own monad, define bind() and return() as class methods:

class MyMonad
  class << self def bind(ma, &blk) # blk is a lambda/proc of signature a -> mb
      # use ma and blk to return something of type mb
    def return(a)
      # turn a into ma

Then you can call Monadt::Monad.do_m()

Monadt::Monad.do_m(MyMonad) do |m|
  x = m.bind method1(5)
  y = m.bind method2("hello", x)
  m.return (x + y)

do_m() takes as its argument a Class which is presumed to have bind() and return() defined. It also takes a block which is the monadic context to execute. The input argument to the block is a special object that knows how to call bind() and return(). The method Monad.async_either() used in our example is just a convenience method that calls Monad.do_m(AsyncEither).

Under the Hood

The do_m() method uses Ruby’s Fiber and a pattern I call “yield abuse” in order to mimic the monadic syntactical sugar you see in Haskell or F#. Fibers let you pass control back and forth between the monad implementation code and the user code, with the monad implementation code having the option to opt out early. By stringing the Fiber block execution through a series of recursive lambdas, we can create a code flow that looks like standard monadic execution. You can pull off the same trick in JavaScript.

Function-Style Functors

Many of the standard monad types have functions as the monad/functor type that wraps the interior type. So for example, the state monad type m a is traditionally a function that takes in a state, and returns a tuple of a generated value and an updated state: state -> output * state. This is going to look pretty nasty in Ruby unless you find a way to turn normal methods into Procs (the closest thing Ruby has to a function object). Fortunately there is the funkify gem, which I’ve written about before. This handy gem will let you turn standard Ruby methods into Procs that can be partially applied, e.g. so you can have a method

def add(x,y)
  x + y

and partially apply it like so:

add_two = add(2) # add_two is a Proc taking one argument

Using funkify, you can generate pretty decent-looking state monad blocks:

def state_func_1(arg, state)
  [value_1, updated_state_1]
def state_func_2(arg1, arg2, state)
  # ...
  [value_2, updated_state_2]
Monad.run_state(initial_value) do |m|
  x = m.bind (state_func_1 arg1)
  y = m.bind (state_func_2 arg2, x)
  m.return x * y

No Multiverses…Yet

One significant limitation of the current version of monadt is that the List monad is not supported by do_m(). The list monad requires the ability to retry bind functions for different inputs, which would require a partial rewind through the Fiber block. Unfortunately, this is not supported by Ruby. My coworker Job and I have some ideas of how to get around this, but we haven’t implemented it yet.


I was pleasantly surprised at how well monadic control flow can be expressed in Ruby. The dynamic typing and monkey patching, fraught with danger though they may be, actually make it possible to implement complex features like typeclasses in Ruby. And Fibers give you a lot of flexibility in how a code block executes. I encourage you to consider monadt for projects that contain structured data you might otherwise express loosely as a Hash or an Array. And even if monadt doesn’t make sense to include in your particular project, I encourage you to dig deeper into the control flow features of Ruby and see how you might be able to leverage them to streamline the logic in your application. Happy (slightly more functional) programming!

The post Monadt – Algebraic Data Types and Monads in Ruby, Part 2: Monads appeared first on Atomic Spin.

Monadt – Algebraic Data Types and Monads in Ruby, Part 1: ADTs

Functional programming is elegant and expressive. I’ve written before about my love of partial application, and how the funkify gem can be used to bring the power of partial application to your Ruby code. But partial application is just one of the powerful idioms from functional languages that I’d like to borrow in object-oriented languages. I’m also pretty into algebraic data types and monads.

So, continuing my pattern of adding functional concepts to object-oriented languages whether they like it or not, I recently created the monadt gem which adds support for using algebraic data types and monads to Ruby. This library gives you these powerful features while attempting to be as readable and Ruby-like as possible.

In this post, I’m going to explain how the monadt library provides algebraic data types. I will cover its use of monads in a future post.

Algebraic Data Types

An algebraic data type (or discriminated union), is like a C union which also maintains which value of the union should be used. Alternatively, you can think of ADTs like enums with data values attached to the different enum values. So for example, if I wanted to create an ADT that captures the state of a car’s gear and velocity, it might look like this:

type GearSpeed =
  | Park
  | LowGear of int * double
  | Drive of double

where Park has no additional data (we are in the parked gear and not moving), LowGear tracks both the forced low gear and our current velocity, and Drive indicates that we are in the regular drive gear at a certain speed. Some possible values:

Park              // parked
Drive 45.0        // drive, 45 mph
LowGear (1, 10.0) // first gear, 10 mph
Drive 55.2        // drive, 55.2 mph
LowGear (2, 20.0) // second gear, 20 mph

What’s so powerful about ADTs is that you can use them to expressively define all possible states. I cannot create a GearSpeed without specifying one of the three cases, and I cannot specify one of the cases without providing its required values. Additionally, languages with ADTs define powerful pattern matching semantics to quickly identify which case you are in and extract the values.

match aGearSpeed with
| Park -> "stopped"
| LowGear (gear, speed) -> sprintf "%d gear, %f mph" gear speed
| Drive speed -> sprintf "drive, %f mph" speed

ADTs also allow you to specify default cases, and they check at compile time to make sure you have covered all the cases when you branch.

let isStopped aGearSpeed =
  match aGearSpeed with
  | Park -> true
  | _ -> false
// will not compile, missing Park case
let getSpeed aGearSpeed =
  match aGearSpeed with
  | LowGear (_, speed) -> speed
  | Drive speed -> speed


So where does Ruby come in? As a dynamically typed language, we can’t formally restrict the types associated with data, and we can’t guarantee all cases have been covered at compile time. However, we can define ADT such that pattern matching will indicate which case we are and easily extract the correct values.

The monadt library (get it?) builds on the code from this blog post by fellow Atom Job Vranish, but it uses functional/functor map style pattern matching, instead of taking a lambda intended to have side effects.

Here’s an example:

class GearSpeed
  Park = data
  LowGear = data :gear, :speed
  Drive = data :speed

This code declares an ADT GearSpeed with the same meaning as the original definition in F#. It has three different values: Park, LowGear, and Drive. Park has no associated data, LowGear has two pieces of associated data (“gear” and “speed”), and Drive has one (“speed”). We can’t restrict the types of those values, but we can name them for future refernce.

You can create new instances with the standard Ruby constructor for the value type: 3, 21.0  # gear = 3, speed = 21.0 mph

Given an ADT value, we can do pattern matching using monadt’s match() and with() functions:

match val,
  with(GearSpeed::Park) { "stopped" },
  with(GearSpeed::LowGear) { |gear, speed| "#{gear} gear, #{speed} mph" },
  with(GearSpeed::Drive) { |speed| "drive, #{speed} mph" }

The above code takes an ADT value and pattern matches by type, taking a lambda which will yield the values associated with the particular ADT case. You can also do default matching, using the “Default” type in monadt instead of the _ wildcard (as you would in F# or Haskell):

match val,
  with(GearSpeed::Park) { true },
  with(Default) { false }

Decorating ADTs

But wait, there’s more! To handle other common usages, and to shorten the length of code to create ADT values, monadt includes a method decorate_adt().

decorate_adt GearSpeed

Calling this method on an ADT type adds the following:

  1. constructor functions based on the lowercase ADT value name, attached to the parent type: 50.0 # same as 50.0
    GearSpeed.park # same as
    GearSpeed.low_gear 2, 10.5 # same as 2, 10.5
  2. is_###? functions for each case (same lowercase naming as the constructor):
  3. sensible default to_s()
    gearspeed_value.to_s # "Drive(36.0)", "Park", or "LowGear(2, 12.5)"

Moving on to Monads

That covers the basics of using ADTs with monadt. In my next post, I’ll talk about how monadt uses Ruby Enumerators to simulate the syntactic sugar associated with monads in Haskell or F#, particularly focusing on Maybe and Either monads—two types that can be implemented as ADTs.

The post Monadt – Algebraic Data Types and Monads in Ruby, Part 1: ADTs appeared first on Atomic Spin.


Windows Socket Functions in Ruby Extensions

If you want to use Windows socket functions (such as WSARecv, and WSASend) inside a Ruby extension on Windows, you may have your work cut out for you. Here’s a helpful work-around I discovered during a recent project.

Spotting the Problem

I was working on an improvement to the rENet library, which is a Ruby binding to the excellent ENet library. Previously, building the extension required preinstalling the ENet library on your system. But ENet is a nice, small library with no other dependencies, so I thought I’d try to build it into the extension. This would be much more convenient to users of the library, as the Ruby extension builder would build ENet automatically without requiring pre-installation.

My first attempt worked great on Linux and OSX systems, but to my bafflement, it failed in spectacular ways on Windows systems. After a little debugging, I found that most operations would work fine, but some would give:

WSAERROR 10038 : (WSAENOTSOCK): Socket operation on non socket.

What was especially strange is that, with no code changes, everything would work great if I just linked to an ENet library that was built externally—not by the Ruby extension builder. It appeared that the Ruby extension builder was messing up some, but not all, of my socket operations.

Some googling and the headers in the extension builder source hinted that Ruby was creating its own wrappers for socket functions. But I had to dig all the way down into the Ruby source code to figure out what was actually going on and how I could fix it.

Finding the Cause

Internally, Ruby wraps the Windows socket functions with its own functions. These wrappers replace the normal socket handles with a homemade “file descriptor,” which is managed internally by Ruby.

For an example of what I’m talking about, search for: rb_w32_socket here.

I’m assuming this is done to make the internal APIs more consistent across OSs. For example, it lets you use select() to block on either a file or socket IO event in the same call. This is a common technique on POSIX systems, but is normally not possible on Windows.

The problem arises when this fake socket handle gets mixed in with an unwrapped function that expects a real one, or vice versa. That’s what caused my WSAENOTSOCK error. The function was expecting a real socket, but got the fake handle instead.

Unfortunately, not only does Ruby wrap the socket functions; it also replaces the symbols for the wrapped functions at the linker level so it is nearly impossible to use the native Windows socket functions.

Choosing the Best Solution

Once you understand what’s going on, there are two potential solutions to this problem.

  1. Make your own wrappers for the functions that are causing problems.
  2. Use an ugly hack to go around Ruby and use the native functions.

If your extension needs to share socket handles with other Ruby code, you have no choice and must use Option 1. Unfortunately, this is the harder option. You’re pretty much going to have to dig into the Ruby source and try to copy what they are doing, but for your own wrapper. This might be a good place to start digging around for something close that you can copy.

If your extension does not need to share socket handles with other Ruby code and is going to manage the sockets completely internally, you can use either option. However, Option 2 is probably easier and less error-prone.

Here’s the code I used to get the actual Windows socket syscalls, instead of the wrapped ones provided by Ruby:

bindptr        = (LPFNBIND)       GetProcAddress(GetModuleHandleA("ws232.dll"), "bind");
getsocknameptr = (LPFNGETSOCKNAME)GetProcAddress(GetModuleHandleA("ws232.dll"), "getsockname");
listenptr      = (LPFNLISTEN)     GetProcAddress(GetModuleHandleA("ws232.dll"), "listen");
socketptr      = (LPFNSOCKET)     GetProcAddress(GetModuleHandleA("ws232.dll"), "socket");
ioctlsocketptr = (LPFNIOCTLSOCKET)GetProcAddress(GetModuleHandleA("ws232.dll"), "ioctlsocket");
setsockoptptr  = (LPFNSETSOCKOPT) GetProcAddress(GetModuleHandleA("ws232.dll"), "setsockopt");
getsockoptptr  = (LPFNGETSOCKOPT) GetProcAddress(GetModuleHandleA("ws232.dll"), "getsockopt");
connectptr     = (LPFNCONNECT)    GetProcAddress(GetModuleHandleA("ws232.dll"), "connect");
acceptptr      = (LPFNACCEPT)     GetProcAddress(GetModuleHandleA("ws232.dll"), "accept");
shutdownptr    = (LPFNSHUTDOWN)   GetProcAddress(GetModuleHandleA("ws232.dll"), "shutdown");
closesocketptr = (LPFNCLOSESOCKET)GetProcAddress(GetModuleHandleA("ws232.dll"), "closesocket");
selectptr      = (LPFNSELECT)     GetProcAddress(GetModuleHandleA("ws232.dll"), "select");

Then, simply replace all the calls to functions such as bind and select to calls on appropriate function pointer, such asbind_ptr and select_ptr.

In Ruby 2.0 or higher, you’ll probably need to add a +#define INCL_WINSOCK_API_TYPEDEFS 1 to gain visiblity to the pointer types. It’s a super-gross solution, but it happens to work quite well.

The post Windows Socket Functions in Ruby Extensions appeared first on Atomic Spin.

Using ruby mocha outside of unit testing frameworks

I find myself currently writing a lot of orchastration code that manages hardware. This is very difficult because I like doing little test.rb scripts or testing things out in irb or pry to see if APIs are comfortable to use.

The problem with hardware is in order to properly populate my objects I need to query things like the iDRACs or gather inventories from all my switches to figure out where a piece of hardware is and this take a lot of time and requires constant access to my entire lab.

Of course my code has unit tests and so all the objects that represents servers and switches etc are already designed to be somewhat comfortable to load stub data for and to be easy to mock. So I ended up using rspec as my test.rb environment of choice.

I figured there has to be a way to use mocha in a non rspec environment, and turns out there is and it’s quite easy.

The magic here is line 1 and line 5, including Mocha::API will extend Object and Class with all the stubbing and mocking methods. I’d avoid using expectations and instead use stubs in this scenario.

At this point I’d be dropped into a pry shell loaded up with the service fixture in my working directory where I can quickly interact with my faked up hardware environment. I have many such hardware captures for each combination of hardware I support – and the same data is used in my unit tests too.

Below I iterate all the servers and find their mac addresses of the primary interfaces in each partition and then find all the switches they are connected to. Behind the scenes in real life this would walk all my switches looking for the port each mac is connected to and so forth, quite a time consuming operation and would require me to dedicate this lab hardware to me. Now I can just snapshot the hardware and load up my models later and it’s really quick.

I found this incredibly handy and will be using it pretty much all the time now, so thought it worth sharing :)

6 Months of Avoiding Rails Controllers with DDC

My distaste for unnecessary Rails controllers is no secret. That’s why I wrote the ddc gem (Data Driven Controllers).

When DDC was released, it got mixed reviews in the comments section, so I thought I’d post a follow up with my results thus far.

Example :monkeys, 
  actions: {
    show: {
      context: 'context_builder#user_and_id',
      service: 'monkey_service#find'
    index: {
      context: 'context_builder#user_and_id',
      service: 'monkey_service#find_all'
    update: {
      context: 'context_builder#monkey',
      service: 'monkey_service#update'
    create: {
      context: 'context_builder#monkey',
      service: 'monkey_service#create'


My current project’s Rails API has about 20 controllers, almost all of which are mere data describing how to glue the Rails pieces together. The overall approach has held up well. With decent naming conventions and DDC, I barely ever have to think about Rails controllers. When I did need a custom bit of something, DDC let me define the methods that actually required some thought. So far, working with DDC has been great: easy to use with very little mental overhead.

New Features

There have been a few features added along the way to round out the DDC API:

  1. :render_opts can now be specified. These options let you control which serializer will be used to render domain specific content to JSON. I actually haven’t needed this feature, but other projects using DDC have.
  2. :context will now take an optional array of function strings. The functions will be called in order, being passed the previous functions results and outputting its own. (Like building an onion from the inside-out.)
  3. DDC has been a great tool in my tool belt. I’d really like to see Rails find a clean way to make controllers optional the way Ember.js does.

    I’m curious what others are doing to streamline the building of Rails controllers and Rails sites in general. Leave a comment below.

    The post 6 Months of Avoiding Rails Controllers with DDC appeared first on Atomic Spin.

RSchema-Hamster: Schemas for Shaping Immutable Data

RSchema-Hamster is a Ruby gem that lets you use Hamster’s persistent data structures to define and validate the shape of your data via RSchema.

What for?

In order to clarify my designs, reduce mistakes, and leave code easier to change than when I found it, I:

  • Strive to program with immutable data
  • Learn to apply ideas from type-driven development

Using RSchema-Hamster, I can define a schema such as:

require 'rschema_hamster'
Name = String
PageNumber = Integer
Appearances = Hamster.vector(
  Hamster.hash(character: Name, page: PageNumber)

Build a data structure like this:

guide = Hamster.vector(
  Hamster.hash(character: "Arthur", page: 1), 
  Hamster.hash(character: "Zaphod", page: 98)

And assert the data conforms to the schema like this:

RSchema.validate!(Appearances, guide)

Immutable Data Structures via Hamster

I prefer to code using primarily immutable values because I’ve come to understand just how careful I need to be when mutating program state: plan for it, isolate it, and generally avoid changing internal object state unnecessarily.

Two up-front challenges have been to change my design habits to favor stateless, functional code (difficult, took lots of practice) and to find a powerful persistent data structures tool (not difficult; for Ruby, I picked Hamster).

The Hamster Ruby gem provides a set of persistent collections to supplement or replace Ruby’s mutable collections, such as Hash, Vector, lazy Lists, and so forth. It looks like this:

ford = Hamster.hash(name: "Ford Prefect", gender: :male)
# => Hamster::Hash[:name => "Ford Prefect", :gender => :male]
arthur = ford.put(:name, "Arthur Dent")
# => Hamster::Hash[:name => "Arthur Dent", :gender => :male]
# (ford remains unchanged)

Getting Lost

As I start moving back to using generic collections like Hashes, Vectors, and Sets to build my data structures, I’m getting a little nervous about inconsistency and accidental misuse. Immutable or not, these collections don’t care what you store in them and will give you no hint as to their intended use. There’s a lot of room for mistakes; this is why I had a hard time letting go of my descriptively-named classes and their attr_accessor declarations.

(A couple years back, Patrick Bacon and I created Hamsterdam as a first step toward alleviating this concern, using immutable record-like objects as a means of defining and documenting attributes.)

Schema-based Definition and Validation via RSchema

Whereas some functional languages like Haskell and F# have very expressive type systems for describing data, dynamic languages like Clojure and Ruby require me to take matters into my own hands. I discovered Tom Dalling’s RSchema (a Ruby implementation of Prismatic’s Schema for Clojure), and started tinkering with ad-hoc schema structures to help clarify and test some of my existing code as I refactored it. I now felt like I could start taking some queues from the Type-Driven crowd.

In RSchema, a schema is itself data, a value that describes the shape of other values:

OrderRow = {
  order_id: Integer,
  order_totals: Totals
row = { 
  order_id: 5, 
  order_totals: a_valid_totals_structure 
RSchema.validate! OrderRow, row

Read: An OrderRow is a Hash whose order_id key refers to an integer value, and whose order_totals key refers to a Totals object, presumably a Ruby class or perhaps another schema object.

It may not be compile-time type-checking, but it sure is expressive, and it gives me the means to programmatically connect variables to their intended types. I can easily discover and understand the data’s shape by reading the code, and at runtime (especially during automated unit and integration testing) assert that the data conforms. Executable documentation, y’see.

So if I want to insist that OrderRow is an immutable Hash, I want to be able to write:

OrderRow = Hamster.hash(
  order_id: Integer,
  order_totals: Totals

However, the above schema isn’t valid. In order for an object to be treated as a descriptive instances (in this case a Hash containing sub-schemas), the object must implement a method called “schema_walk”, and Hamster structures don’t.

Extending Hamster to Support RSchema

RSchema defines schemas in terms of classes, Hashes, and Arrays, plus a few configurable schemas available via its DSL, such as “enum,” “maybe,” and an optional Hash-key decorator. Ruby’s Hash and Array classes implement their respective “schema_walk” such that their instances may be used to describe internal structure.

In order to use Hamster instances to describe structure, I provided implementations of “schema_walk” to Hamster::Hash, Hamster::Vector and so on, plus a few generic schemas to give parity to RSchema’s “hash_of” and “set_of” DSL methods (“hamster_hash_of” and “hamster_set_of” respectively).

The big bonus: schemas are designed to be nestable in RSchema. Because of this, schemas I define using Hamster’s types may still contain (or be contained by) Ruby’s Hash and Array structures, as well as utilize (or be used by) RSchema’s “enum,” “maybe,” and “_?” (optional hash key) schema helpers, without any extra implementation.

Here’s an abridged real-world example of how I was able to untangle a document-like structure that was already on its way to generating headaches in a Rails app I’m working on:

require 'rschema_hamster'
module OrderReport::Schema
  Name    = String
  Id      = Integer
  Dollars = BigDecimal
  Totals = Hamster.hash(
    gross: Dollars,
    tax:   Dollars,
    fee:   Dollars,
    net:   Dollars,
  OrderRow = Hamster.hash(
    order_id:     Id,
    order_number: Name,
    order_totals: Totals,
  MarketBlock = Hamster.hash(
    market_id:        Id,
    market_name:      Name,
    account_dropdown: RSchemaHamster.schema {
      hamster_hash_of(Name => Id)
    order_rows:       Hamster.vector(OrderRow),
    market_totals:    Totals
def self.dollars(str);; end
market_block = Hamster.from(
    market_id: 42,
    market_name: "The Restaurant at the End of the Universe",
    account_dropdown: {
      "Hotblack Desiato" => 1, 
      "Zaphod Beeblebrox" => 3
    order_rows: [
      { order_id: 101, order_number: "MILLIWAYS-00101", order_totals: { gross: dollars("120"), tax: dollars("14.4"), fee: dollars("20"), net: dollars("85.6") } },
      { order_id: 102, order_number: "MILLIWAYS-00102", order_totals: { gross: dollars("3030"), tax: dollars("363.6"), fee: dollars("505.10"), net: dollars("2161.3") } },
    market_totals: { gross: dollars("3150"), tax: dollars("378"), fee: dollars("525.10"), net: dollars("2246.9") }
RSchema.validate!(OrderReport::HamsterSchema::MarketBlock, market_block)

The post RSchema-Hamster: Schemas for Shaping Immutable Data appeared first on Atomic Spin.

Building an Infinite Procedurally-Generated World

I had a lot of fun writing my last blog post: All Work & No Play – Taking Time to Code for Fun. In it I talked about writing fun code that keeps you interested in programming and keeps you creative. I used the example of writing a 2D procedurally-generated, infinite world. In this post, I am going to explain details of how that example works.


Bring the Noise

To build our terrain, we need something better than just randomly selecting a tile for each set of coordinates. Since we want the world to be infinite, we can’t design it by hand: that’s where Perlin noise comes in. Perlin noise is an algorithm for generating organic looking, multidimensional noise quickly (at least for 2D). You’ll want to know a few terms when playing with Perlin noise:

The number of subsequent generations to run, usually double the frequency and half the amplitude.
How to weight the additional octaves when adding them.

These can be fun to play with when generating terrain; they will make it smooth and flat or bumpy and spiky. Further explanation can be found here.

Homework assignment: try merging two noise datasets that have been generated with different octave / persistence values.

From Noise to Tiles

Once you have access to random 2D data, it’s pretty straight forward to convert that noise into usable data. Simply set up thresholds for each tile type you want to support, eg:

  • water if < 0.3
  • grass if >= 0.3 and <= 0.6
  • mountain if > 0.6

I recommend leaving the upper and lower cases open. If they are not left open, you may end up with holes in your map for unexpectedly high or low values. An alternative is to clamp or scale the noise values when you generate them.

Going Beyond Tiles

An infinite world of terrain isn’t all that interesting for very long without something in it. We want to add things like trees, caves, and towns to our game. We can simply use the same random number generator (RNG) that we used to generate our noise to determine when to place and how to build objects. We have to be careful though. If we use a single RNG for the whole world, the order in which we discover things will change the how they are generated.


The trick to deterministically creating the world based on a seed is to break it up into chunks (see screenshot above). A chunk is simply a range of tiles in the world. We keep 9 chunks loaded at a time. Each chunk is 50×50 grid of tiles. When the player enters a place with no chunk, we generate a new one. The first thing we do is seed a new RNG for that chunk with the global seed and the coordinates for that chunk. This allows us to throw away the chunk when the player leaves it, but know that it will be recreated the same when the player comes back. Just make sure your code is deterministic and that it only pulls random values from the chunk’s RNG and things should work out well.

The source for this project can be found on github. Please leave a comment if you are doing something similar and want an excuse to show off.

Example uses tileset from Oryx Design Lab. Thanks!

The post Building an Infinite Procedurally-Generated World appeared first on Atomic Spin.

Some travlrmap updates

Been a while since I posted here about my travlrmap web app, I’ve been out of town the whole of February – first to Config Management Camp and then on holiday to Spain and Andorra.

I released version 1.5.0 last night which brought a fair few tweaks and changes like updating to latest Bootstrap, improved Ruby 1.9.3 UTF-8 support, give it a visual spruce up using the Map Icons Collection and gallery support.

I take a lot of photos and of course often these photos coincide with travels. I wanted to make it easy to put my travels and photos on the same map so have started adding a gallery ability to the map app. For now it’s very simplistic, it makes a point with a custom HTML template that just opens a new tab to the Flickr slideshow feature. This is not what I am after exactly, ideally when you click view gallery it would just open a overlay above the map and show the gallery with escape to close – that would take you right back to the map. There re some bootstrap plugins for this but they all seem to have some pain points so that’s not done now.

Today there’s only Flickr support and a gallery takes a spec like :gallery: flickr,user=ripienaar,set=12345 and from there it renders the Flickr set. Once I get the style of popup gallery figured out I’ll make that pluggable through gems so other photo gallery tools can be supported with plugins.

As you can see from above the trip to Spain was a Road Trip, I kept GPX tracks of almost the entire trip and will be adding support to show those on the map and render them. Again they’ll appear as a point just like galleries and clicking on them will show their details like a map view of the route and stats. This should be the aim for the 1.6.0 release hopefully.

Some travlrmap updates

Last weekend I finally got to a point of 1.0.0 of my travel map software, this week inbetween other things I made a few improvements:

  • Support 3rd party tile sets like Open Streetmap, Map Quest, Water Color, Toner, Dark Matter and Positron. These let you customise your look a bit, the Demo Site has them all enabled.
  • Map sets are supported, I use this to track my Travel Wishlist vs Places I’ve been.
  • Rather than list every individual yaml file in a directory to define a set you can now just point at a directory and everything will get loaded
  • You can designate a single yaml file as writable, the geocoder can then save points to disk directly without you having to do any YAML things.
  • The geocoder renders better on mobile devices and support geocoding based on your current position to make it easy to add points on the go.
  • Lots of UX improvements to the geocoder

Seems like a huge amount of work but it was all quite small additions, mainly done in a hour or so after work.

Travlrmap 1.0.0

As mentioned in my previous 2 posts I’ve been working on rebuilding my travel tracker app. It’s now reached something I am happy to call version 1.0.0 so this post introduces it.

I’ve been tracking major travels, day trips etc since 1999 and plotting it on maps using various tools like the defunct Xerox Parc Map Viewer, XPlanet and eventually wrote a PHP based app to draw them on Google Maps. During the years I’ve also looked at various services to use instead so I don’t have to keep doing this myself but they all die, change business focus or hold data ransom so I am still fairly happy doing this myself.

The latest iteration of this can be seen at It’s a Ruby app that you can host on the free tier at Heroku quite easily. Features wise version 1.0.0 has:

  • Responsive design that works on mobile and PC
  • A menu of pre-defined views so you can draw attention to a certain area of the map
  • Points can be catagorized by type of visit like places you've lived, visited or transited through. Each with their own icon.
  • Points can have urls, text, images and dates associated with them
  • Point clustering that combines many points into one when zoomed out with extensive configuration options
  • Several sets of colored icons for point types and clusters. Ability to add your own.
  • A web based tool to help you visually construct the YAML snippets needed using search
  • Optional authentication around the geocoder
  • Google Analytics support
  • Export to KML for viewing in tools like Google Earth
  • Full control over the Google Map like enabling or disabling the street view options

It’s important to note the intended use isn’t something like a private Foursquare or Facebook checkin service, it’s not about tracking every coffee shop. Instead it’s for tracking major city or attraction level places you’ve been to. I’m contemplating adding a mobile app to make it easier to log visits while you’re out and about but it won’t become a checkin type service.

I specifically do not use a database or anything like that, it’s just YAML files that you can check into GitHub, easily backup and hopefully never loose. Data longevity is the most important aspect for me so the input format is simple and easy to convert to others like JSON or KML. This also means I do not currently let the app write into any data files where it’s hosted. I do not want to have to figure out the mechanics of not loosing some YAML file sat nowhere else but a webserver. Though I am planning to enable writing to a incoming YAML file as mentioned above.

Getting going with your own is really easy. Open up a free Heroku account and set up a free app with one dynamo. Clone the demo site into your own GitHub and push to Heroku. That’s it, you should have your own up and running with place holder content ready to start receiving your own points which you can make using the included geocoder. You can also host it on any Ruby based app server like Passenger without any modification from the Heroku one.

The code is on GitHub ripienaar/travlrmap under Apache 2. Docs for using it and configuration references are on it’s dedicated gh-pages page.