Syndication of blogs and tweets by users of the Freenode ##infra-talk IRC channel
When you're stuck on a problem, or not even stuck but you just want to boost your creative/out-of-the-box thinking...
Take a shower. When I'm thinking about a problem and I take a shower, the ideas and thoughts just start popping up, one after each other, or sometimes even two at the same time. It's amazing. And it works every time.
::Read more
Gastro has been relaunched at gotgastro.com!
DNS issues should be all fixed, so go nuts and share the site.
I pushed two new features on Gastro last night: GeoRSS support, and a JSON export.
This should make it a lot easier to take the data i've scraped and reuse it. Feel free to grab it and play with it as you will!
You should be able to take the GeoRSS feed and pipe it straight into Google Maps (paste the feed url into the search bar), however GMaps doesn't seem to like fetching the feed.
It might be something to do with a reported DNS issue. If anyone has a free second to do a dig and see if anything is boned I would be very appreciative.
The new release also has a bunch of performance improvements (i've basically halved the app load time), and the info window on the map points display the restaurant names.
Simon wanted to know the tech details, so here they are:
The mashup is a collection of small Ruby apps.
There's a scraper, which handles caching copies of the food authority site locally, extracting meaningful data from the cached copies, geocoding against the extracted address data, and writing that data out in an easy-to-parse format (YAML).
The geocoding is done against Google's geocoding service, using the ym4r library. I serialise the store the lat/lng data back to the YAML.
The website is written in Merb (though I am considering rewriting it in nanoc), and uses an in-memory Sqlite3 database that's created from the YAML data every time the app boots. This means the app takes about 10 seconds to boot, but access to all the data is pretty damn quick once the app is up.
I'm caching the JSON/RSS/HTML pages just in case it gains momentum and my server gets hammered. I'm using the standard Merb::Cache caching mechanism, which generates new page fragments every 30 minutes.
The app is running under 2 Mongrels with apache2 + mod_proxy_balancer and it seems to cope ok. I've had 70 unique visits in the last 24 hours so i'm not particularly worried about load.
If you want to check out the code, you can grab a copy of my repo with a
$ bzr branch http://holmwood.id.au/~lindsay/code/gastro
There are a bunch of rough edges (no documentation, notice urls aren't serialised yet, ~70 addresses can't be geocoded, and geocode data can be easily overwritten), but interested folk might like a stickybeek.
Spread the word if you like it! New features are pending. :-)
The adventure at Netlog didn't work out entirely, so I'm looking for a new challenge!
My new ideal (slightly utopic) job would be:
To get a detailed overview of my interests and skills, I refer to:
Recently I started thinking about writing my own automatic installer that would set up my system exactly the way I want.
(See http://dieter.plaetinck.be/rethinking_the_backup_paradigm_a_higher-level...)
I looked at the official Arch install scripts to see if I could reuse parts of their code, but unfortunately the code was just one big chunk of bash code with the main program and "flow control" (you must first do this step, then that), UI-code (dialogs etc) and backend logic (create filesystems, ...) all mangled up and mixed very closely together.
Functionality-wise the installer works fine, but I guess the code behind it is the result of years of adding features and quick fixes without refactoring, making it impossible to reuse any of the code.
So I started to write AIF: the Arch Linux Installation Framework (actually it had another name until recently), with these 3 goals in mind:
::Read more
I recently had to rename a repo on my Github account. Github has made this very easy but it's just one side of the issue. Obviously you must also update any references to this remote in other clones, otherwise pushes, fetches etc won't work anymore.
You can do this in two ways:
That's it! All will work fine again.
Weird as it might sound, I've never bothered to listen to Muse songs.. until now. Some people have recommended the band to me so I really had to stop ignoring this band someday. And wow.. what have I been missing al that time :/
Songs like Butterflies and Hurricanes and Citizen Erased are among the most beautiful songs I've ever heard now.
I was reading the FiveRuns blog post on automating your Rails server configuration, and had a peek at the script they included to setup a machine.
There are some fundamental problems with it.
What happens when the machine can't reach RubyForge to download the RubyGems tarball? The rest of the script will continue to run and error out horribly, leaving you to wade through a screenful of error output to work out what died.
What about when you need to update your production systems with new packages or configuration files? You'll probably log into all of them and apply the update (or maybe you'll write a script to do it). But what about new machines that you roll out? Do you update the install script with the new tasks? Do you use the old install method and then apply the new configuration?
These problems are vaguely manageable with a small number of machines, but as soon as you grow beyond 3 servers it becomes tedious and painful - deployment is hard.
I learnt this the hard way: I started out my sysadmin career building systems like this. The first time I did this I thought it was great. I wrote a ~400 line provisioning script that would set up a PXE booted machine with Samba/Apache/OpenLDAP. This worked exceptionally well in my test environment. Production was another story.
I had inadvertently optimised my deployment for provisioning, not for maintenance.
When you consider the lifecycle of a system, < 0.001% of it's life is spent being provisioned. The other 99.999% is the server sitting there serving requests, having new software deployed to it, or having it's configuration tweaked.
Putting all the logic into the provisioning process is setting yourself up for failure. To pull it off, you have to know exactly what the machine is going to be doing for its entire lifecycle. Prescience is not a common trait amongst humans.
So the project requirements change, and the time comes to add another service to the system. How do you manage this? ssh-in-a-for-loop is the obvious response, but then you'd be perpetuating the problem you created for yourself in the first place. And what about new systems? Oh the pain!
Unfortunately there's no easy way out of this mess. You can start from scratch, or push through and waste countless hours of your life maintaining a thousand houses of cards. And you can only start from scratch so many times before it gets equally tedious and fragile.
Configuring your systems with magical scripts is not maintainable. Competent sysadmins around the world have a responsibility to stamp out deployment techniques of this ilk.
This is how some people use Capistrano, and it gives me the willies. Invoking your distro's package manager from Capistrano is no better than doing it from a shell script. Use the tool for what it's built for: automating the deployment of your application.
So what do I recommend? Configuration management is a much better solution to this problem. Instead of maintaining separate procedures for provisioning servers and doing change management, you can merge them into one.
Puppet is my configuration management system of choice, and I use it extensively across all the infrastructure I manage. I do this for three reasons:
Combining Puppet with a distributed version control system is a solid one-two punch to your provisioning and maintenance woes.
Hopefully i've convinced you to use Puppet. The first question you probably have is "how do I use puppet to manage $app_built_with_my_framework_of_choice?".
Just like I recommend not using Capistrano to configure your servers, I recommend using Puppet to manage the infrastructure around your application. Your Puppet manifests should manage web + database + mail servers, monitoring, system package repositories, system utilities, and the standard library of the language your app is written in.
This makes application the side of deployment really simple: for Ruby folk, I recommend bundling all your gems with your application. You should expect the ability to plop your app on any machine that has no more than the Ruby standard library and interpreter and have the app run.
The end result is configuration that can be applied to a new machine in 10 minutes with Puppet, and a seamless app deployment in less than 5 using Cap or Vlad.
When you need to scale further down the line, you can simply apply that same configuration to a new node. Making changes across all your machines becomes an order of magnitude easier - update the manifests, push to your Puppetmaster, have the machines update themselves.
What could be simpler?
