Thinking Abstractly: Why Programmers Don’t Write Binary Code

When you imagine a computer programmer at work, what comes to mind? Science fiction movies like The Matrix might give you the impression that programmers read and write endless streams of 1s and 0s, communicating with the computer in a digital dialect unintelligible to most humans.

But in reality, computer programming isn’t about the 1s and 0s, any more than painting is about the blobs of oil and pigment. When a painter works, their mind is focused on expressing an artistic vision for audiences to appreciate. Likewise, when computer programmers work, our minds are focused on expressing instructions for computers to perform. The 1s and 0s (binary code) are just the most raw, basic form of our medium, and we very rarely work in it directly.

Just like painters have invented new tools and techniques over the centuries, computer scientists have been working for decades to invent new ways of expressing instructions for computers. Where painters develop new brushes, paint formulas, and artistic movements, we invent and improve operating systems (like Unix, Windows, and Mac OS), programming languages (like Fortran, Java, and Ruby), and paradigms (like object-oriented programming and functional programming). Each new advance is based on advances of years past, as we build up layers of abstraction, allowing us to express instructions more easily, effectively, and economically.

Abstraction is a fundamental aspect of computer science and a valuable technique used by programmers every day, but it’s easy to understand even if you aren’t a computer scientist. To get a grasp on the concept of abstraction layers, let’s explore a common scenario unrelated to computers: walking to a restaurant.

An Everyday Example: Walking to a Restaurant

Suppose you want to meet some friends for lunch. They are waiting for you at the restaurant down the street and around the corner. What actions would you perform to get there?

If we think about it at a low level of abstraction, we could say that you perform some actions like these:

  1. Flex the muscles in your abdomen to lean forward slightly.
  2. Flex the muscles in your right leg to move it forward until it is in front of you.
  3. Extend your right leg until your right foot is on the ground.
  4. Extend your left leg to shift your balance onto your right leg.
  5. Flex the muscles in your left leg to move it forward until it is in front of you.
  6. Extend your left leg until your left foot is on the ground.
  7. Extend your right leg to shift your balance onto your left leg.
  8. Check whether you have reached the street corner. If you have, turn 90 degrees, then continue to action 9. Otherwise, go back to action 2.
  9. Perform actions 2 through 8, then go to action 10.
  10. Check whether you have reached the restaurant. If you have, stop. Otherwise, go back to action 9.

Even though you do flex your muscles to bend and extend your legs when you walk, it would be very difficult to maintain a smooth stride if you had to think about each muscle movement individually. Thankfully, we can think more abstractly. Instead of specifying such minor details, let’s describe the actions a little more broadly:

  1. Step forward with your right leg.
  2. Step forward with your left leg.
  3. Perform actions 1 and 2 until you are at the street corner.
  4. Turn 90 degrees.
  5. Perform steps 1 and 2 until you are at the restaurant.

We can consider this an additional layer of abstraction, because the ability to step forward is built on top of the ability to flex your muscles to move your legs. If we assume that you already know how to step forward by flexing your muscles, then there is no reason to write out the instructions in so much detail.

This new layer of abstraction makes the instructions easier to understand, and you could probably follow these instructions without looking like a real-life version of QWOP. But it’s still pretty awkward to think about each individual step you take. Let’s add another layer of abstraction:

  1. Walk to the street corner.
  2. Walk to the restaurant.

Now we’re getting somewhere! This is another layer because the ability to walk to a certain location is built on top of the ability to take steps. As you can see, there can be many abstraction layers built on top of each other.

Those instructions are pretty good, but we can go up to an even higher level of abstraction:

  1. Go to the restaurant.

This action is so abstract that we have actually just expressed our overall intention or goal. Even though you might accomplish this action by performing all those lower-level actions, there are definite benefits to thinking about actions and instructions (including computer code) at a higher level of abstraction.

Why Think Abstractly?

Computer programmers think and write instructions abstractly because it allows us to:

  • Express instructions more succinctly and understandably. We have already seen how abstraction can make instructions more succinct and understandable. Compare the one simple instruction (“Go to the restaurant.”) to the long list of complex instructions we started with. If you already know how to walk, it’s unnecessary and counterproductive to go into all that detail. In computer programming, if the programming language you use already provides the ability to calculate the sum of a table of numbers, for example, it’s usually a waste of time to write your own code to do that same thing.
  • Eliminate distracting details. If you try to consciously control each muscle while walking, you’ll be so preoccupied with those little actions that you can’t think about anything else. Likewise, if programmers had to write directly in binary code, we would have a hard time thinking about anything except the immediate task at hand. By thinking abstractly and allowing the lower-level systems (the abstraction layers built up over the years) to take care of the details, we free up our mind to think about the overall structure and goals of the software we are creating.
  • Avoid needless repetition. Notice how in the first walking instructions above, I had to repeat the instructions for each leg. Actions 2-4 are almost exactly the same as actions 5-7, except that right and left have been switched. As we added abstraction layers, the instructions became less and less repetitive. In computer programming, we have a principle known as DRY: Don’t Repeat Yourself. The fewer places you repeat some instruction, the easier it is to read and write, and the less work it takes to fix a bug or make the code more efficient later.
  • Use alternative solutions and implementations. Suppose you were late for lunch with your friends, so you decided to ride a bicycle instead of walking. If we are thinking at the lowest level, we would have to totally replace many complex actions. But the higher we go, the fewer changes we would have to make. At our highest level of abstraction, we don’t require any changes at all: riding a bicycle is a valid way to perform the abstract action, “Go to the restaurant.” In computer programming, this type of abstraction allows one application to work on many different computers, even though each computer might have a different CPU, type of memory, etc. The application relies on the abstraction provided by the programming language, which in turn relies on the abstraction provided by the operating system, and so on down, layer by layer.

Of course, there are times when you need to give low-level instructions, such as when a certain action is not obvious or must be done in an unusual way. Likewise, programmers sometimes need to dive down into a lower level of abstraction to implement some low-level details that aren’t already provided, or to do something in a special way. But when we do that, it is useful to think of it as creating a new layer of abstraction upon which the higher-level instructions will rely.

Abstractions All Around

Computer programming is certainly not the only area where abstraction layers exist. As we saw earlier, abstraction allows us to think about and express everyday actions, like going to a restaurant, without being distracted by little details.

Some other examples of abstraction layers in our everyday lives are:

  • Workplace management, where a manager can delegate some task to an employee, without having to specify every detail of how the work is done.
  • Government, where each state or province can have different local laws, which in many cases do not affect how the national government operates.
  • Making a purchase at a store, where the staff doesn’t really care whether you pay with cash or credit card, as long as you complete the abstract action of paying.
  • Even the concept of money is an abstraction that eliminates the distracting details of a barter system. (Next time you pay your electric bill, thank abstraction layers that you don’t have to negotiate in terms of goats and potatoes!)

Think about your own line of work and everyday activities. What layers of abstractions do you use and rely on? Have you ever devised an abstraction layer to simplify some task or process? Tell us about it in the comments.
 

self-contained applications

How much configuration do your web servers need to just run your application? Do they need to have Apache and Passenger installed? Or maybe your app is a WAR that runs in Tomcat?

Even if you use configuration management to automate all of this, it’s still quite a lot of moving parts just to get your application running.

It also makes upgrading hard. Let’s say you run into some obscure Tomcat bug. It’s so obscure that you can’t even find a package of Tomcat with the fix you need so you have to build your own package, add it to a repository, then upgrade the package everywhere, at the same time ensuring the application keeps working properly. And sure maybe you could automate and improve all of this but it’s still a bit clunky and really what you’ve effectively done is move a lot of application logic into the infrastructure. Okay, you might not have business rules as such but you’re still moving the part that handles request into the infrastructure when this really belongs with the app.

The solution to all of this is to make your application as self-contained as possible. If you’re building Java web applications, why not embed Jetty into your application? The entry point of your application then simply becomes a good old main method. If you need to upgrade Jetty for some reason, then you just need to bump up the dependency version in your build file, and hey presto you have upgraded and can move this through your continous delivery pipeline as usual. You can then go a step further and package this as a RPM or DEB and deployment also becomes signfiicantly easier, just a case of installing a package.

Sending PGP HTML Encrypted e-mail with PHP

While adding the PGP HTML Report feature to WeSunSolve, I first successfully crypted the content of the HTML report to be sent to the user with PGP key. I would have thought that this was gonna be the hardest part, that was without thinking about MIME and HTML support of PGP encrypted mails.

Here is how I finally ended up by creating HTML PGP encrypted Mails using PHP which can be opened using (at least) claws-mail and thunderbird with proper rendering of the HTML report:

Content of the clear message

To: test@test.com
Subject: My HTML crypted report
X-PHP-Originating-Script: 1000:mlist.obj.php
From: "We Sun Solve" 
Reply-to: admin@wesunsolve.net
X-Sender: WeSunSolve v2.0
Message-ID: <1335717276@wesunsolve.net>
Date: Sun, 29 Apr 2012 18:34:36 +0200
MIME-Version: 1.0
Content-Type: multipart/encrypted;
 protocol="application/pgp-encrypted";
 boundary="------------enig029BFFF948226050D5D90E10F"

This is an OpenPGP/MIME encrypted message (RFC 2440 and 3156)
--------------enig029BFFF948226050D5D90E10F
Content-Type: application/pgp-encrypted
Content-Description: PGP/MIME version identification

Version: 1

--------------enig029BFFF948226050D5D90E10F
Content-Type: application/octet-stream; name="encrypted.asc"
Content-Description: OpenPGP encrypted message
Content-Disposition: inline; filename="encrypted.asc"

-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.10 (GNU/Linux)

****SNIPPED PGP CRYPTED BASE64 MESSAGE ****
-----END PGP MESSAGE-----

--------------enig029BFFF948226050D5D90E10F--

Content of the PGP encrypted part

Content-Type: multipart/alternative;
 boundary="------------F983FADF500537B8AFDC5E483"

This is a multi-part message in MIME format.
--------------F983FADF500537B8AFDC5E483
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

You need to have a MUA capable of rendering HTML to read
the WeSunSolve emails.

You can consult the website http://wesunsolve.net if you
are not able to read this email, the information sent to you
should also be on the website...

--------------F983FADF500537B8AFDC5E483
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

This is the report in cleartext!

--------------F983FADF500537B8AFDC5E483--

Code Used

$pgpmime = '';
$mime = '';
$headers = '';
$dest = 'test@test.com';
$subject = 'My HTML crypted report';
$clearContent = '<html><p>This is the report in cleartext!</p></html>';
$clearText = 'This is the text version of the report';
/* Prepare the crypted Part of the message */
$bound = '------------'.substr(strtoupper(md5(uniqid(rand()))), 0, 25);
$pgpmime .= "Content-Type: multipart/alternative;\r\n boundary=\"$bound\"\r\n\r\n";
$pgpmime .= "This is a multi-part message in MIME format.\r\n";
$pgpmime .= "--$bound\r\n";
$pgpmime .= "Content-Type: text/plain; charset=utf-8\r\n";
$pgpmime .= "Content-Transfer-Encoding: quoted-printable\r\n\r\n";
$pgpmime .= $clearText."\r\n\r\n";
$pgpmime .= "--$bound\r\n";
$pgpmime .= "Content-Type: text/html; charset=\"utf-8\"\r\n";
$pgpmime .= "Content-Transfer-Encoding: quoted-printable\r\n\r\n";
$pgpmime .= $clearContent."\r\n";
$pgpmime .= "--$bound--\r\n";
$content = GPG::cryptTxt($pgpkey, $pgpmime);
/* Make the email's headers */
$headers = '';
$headers = "From: $from\r\n";
$headers .= "Reply-to: ".$config['mailFrom']."\r\n";
$headers .= "X-Sender: WeSunSolve v2.0\r\n";
$headers .= "Message-ID: <".time()."@".$_SERVER['SERVER_NAME'].">\r\n";
$headers .= "Date: " . date("r") . "\r\n";
$bound = '------------enig'.substr(strtoupper(md5(uniqid(rand()))), 0, 25);
$headers .= "MIME-Version: 1.0\r\n";
$headers .= "Content-Type: multipart/encrypted;\r\n";
$headers .= " protocol=\"application/pgp-encrypted\";\r\n";
$headers .= " boundary=\"$bound\"\r\n\r\n";
/* And the cleartext body which encapsulate PGP message */
$mime = '';
$mime .= "This is an OpenPGP/MIME encrypted message (RFC 2440 and 3156)\r\n";
$mime .= "--$bound\r\n";
$mime .= "Content-Type: application/pgp-encrypted\r\n";
$mime .= "Content-Description: PGP/MIME version identification\r\n\r\n";
$mime .= "Version: 1\r\n\r\n";
$mime .= "--$bound\r\n";
$mime .= "Content-Type: application/octet-stream; name=\"encrypted.asc\"\r\n";
$mime .= "Content-Description: OpenPGP encrypted message\r\n";
$mime .= "Content-Disposition: inline; filename=\"encrypted.asc\"\r\n\r\n";
$mime .= $content."\r\n";
$mime .= "--$bound--";
mail($dest, $subject, $mime, $headers);

Automate Everything!

Oh how I cheered when I read Tom Blomfield’s Automate Everything post! I’ve lost count of the number of times I’ve been subjected to some Byzantine business process and thought that surely there must be a better way?! What’s weird is how many people don’t feel this way. These are people who’ll do the same thing a hundred times and just put up with it like it makes perfect sense to them to do the same thing a hundred times! The thought that there could be a better way doesn’t enter their head.

It seems to me that these are people who interact with the world in a fundamentally different way from the way that I do. That doesn’t make them wrong – just different (and maybe a little sadomasochistic).

I have often thought it would be cool to have some putative automater person whose sole job is go round and improve all these sort of silly processes and do things like making the coffee machine report outages to IRC (and other such little things that bring joy and happiness) but somehow I doubt this idea will catch on any time soon.

Logging in Linux

Logging in Linux is something that’s always bothered me. I hate this stuff:

james@lucinda:/var/log$ ls -l dmesg*
-rw-r----- 1 root adm 45822 Feb  4 08:41 dmesg
-rw-r----- 1 root adm 45822 Jan 30 00:58 dmesg.0
-rw-r----- 1 root adm 12264 Jan 30 00:28 dmesg.1.gz
-rw-r----- 1 root adm 12266 Jan 30 00:19 dmesg.2.gz
-rw-r----- 1 root adm 12271 Jan 29 20:02 dmesg.3.gz
-rw-r----- 1 root adm 12130 Jan 29 19:48 dmesg.4.gz

Old logs are the in same directory as current logs which is just noisy but more importantly there is also a whole slew of problems with log rotation, compression, keeping arcane syslog configuration updated as things change, running out of space if you mess up etc. And let’s face it, having logs sitting on a server like this is next to useless – you want them pushed to a central location so that you can keep track of what’s going on.

With this mind, I finally got round to trying out an idea I had a while ago about how to clean this stuff up. I now have this:

james@lucinda:/var/log$ ls -l dmesg*
prw-r--r-- 1 root root 0 Feb  5 14:05 dmesg

My logs are now named pipes! From an application’s perspective nothing has changed – the log file is the same place and looks and acts like a file. Of course, this isn’t very useful if nothing is connected to the other end of the pipe, so I wrote a little ruby application called Monsoon that sits and watches named pipes in directory that you configure and every time something is sent down one of these pipes, it packages it up into a json fragment as follows:

    {
        "time": "Sun Feb 05 14:05:41 +0000 2012",
        "message": "test message",
        "origin": {
            "route": "/var/log/dmesg",
            "hostname": "lucinda"
        }
    }

and sends off to a TCP server socket that you configure – but this of course could be sent off via stomp or HTTP instead – where you can do something useful with it (put into a database, logstash, SOLR, and so on) It’s then just a case of ensuring that whatever is storing the logs has enough disk space instead of your whole web cluster or whatever (though obviously you should still monitor that too).

I’m quite pleased with the results so far. I’m going to test some more and see how things go.

Logging in Linux

Logging in Linux is something that’s always bothered me. I hate this stuff:

james@lucinda:/var/log$ ls -l dmesg*
-rw-r----- 1 root adm 45822 Feb  4 08:41 dmesg
-rw-r----- 1 root adm 45822 Jan 30 00:58 dmesg.0
-rw-r----- 1 root adm 12264 Jan 30 00:28 dmesg.1.gz
-rw-r----- 1 root adm 12266 Jan 30 00:19 dmesg.2.gz
-rw-r----- 1 root adm 12271 Jan 29 20:02 dmesg.3.gz
-rw-r----- 1 root adm 12130 Jan 29 19:48 dmesg.4.gz

Old logs are the in same directory as current logs which is just noisy but more importantly there is also a whole slew of problems with log rotation, compression, keeping arcane syslog configuration updated as things change, running out of space if you mess up etc. And let’s face it, having logs sitting on a server like this is next to useless – you want them pushed to a central location so that you can keep track of what’s going on.

With this mind, I finally got round to trying out an idea I had a while ago about how to clean this stuff up. I now have this:

james@lucinda:/var/log$ ls -l dmesg*
prw-r--r-- 1 root root 0 Feb  5 14:05 dmesg

My logs are now named pipes! From an application’s perspective nothing has changed – the log file is the same place and looks and acts like a file. Of course, this isn’t very useful if nothing is connected to the other end of the pipe, so I wrote a little ruby application called Monsoon that sits and watches named pipes in directory that you configure and every time something is sent down one of these pipes, it packages it up into a json fragment as follows:

    {
        "time": "Sun Feb 05 14:05:41 +0000 2012",
        "message": "test message",
        "origin": {
            "route": "/var/log/dmesg",
            "hostname": "lucinda"
        }
    }

and sends off to a TCP server socket that you configure – but this of course could be sent off via stomp or HTTP instead – where you can do something useful with it (put into a database, logstash, SOLR, and so on) It’s then just a case of ensuring that whatever is storing the logs has enough disk space instead of your whole web cluster or whatever (though obviously you should still monitor that too).

I’m quite pleased with the results so far. I’m going to test some more and see how things go.

Realer World Haskell

A Standardized, general-purpose purely functional programming language, with non-strict semantics and strong static typing for.. shell scripting? Yup.

Don Stewart’s presentation on  Practical Haskell Programming really opened my eyes.

Don shows how even the most throwaway scripts, in this case a script to toggle a value in a sysctl, can benefit from a type-safe language with generic, reusable abstractions.

Very nice indeed. I do wonder how practical it would be to write code like this for every little thing all the time though!

 

Realer World Haskell

A Standardized, general-purpose purely functional programming language, with non-strict semantics and strong static typing for.. shell scripting? Yup.

Don Stewart’s presentation on  Practical Haskell Programming really opened my eyes.

Don shows how even the most throwaway scripts, in this case a script to toggle a value in a sysctl, can benefit from a type-safe language with generic, reusable abstractions.

Very nice indeed. I do wonder how practical it would be to write code like this for every little thing all the time though!

 

PHP Considered Harmful

I know what you’re thinking. “Not another anti-PHP blog post!” But instead of complaining about specific deficiencies in the language, or how I think PHP encourages you to be a bad programmer or whatever, I want to talk about a fairly eye-opening conversation I just had with a friend. We were talking about possible optimizations for some of our Project Euler solutions. Since I don’t have enough of a background in number theory to come up with many of those kinds of tricks, I suggested that threading might be a simple way to make a big difference for some problems.

Friend: “I don’t know what you’re talking about.”

Well, for example, instead of iterating through a huge range of numbers with one giant loop, you could break the range up into several smaller ranges (according to the number of cores on your machine), then run your algorithm several times in parallel and just sum up your results at the end.

Friend: “Haha, I still have no idea what you’re talking about.”

And then it dawned on me; my friend doesn’t have a formal computer science or engineering background, and he’s been working with PHP (a language with no concept of concurrent programming) almost exclusively for about a decade now. He’s a smart guy, and not what I would describe as a stereotypical “bad PHP programmer,” but the fact that he hasn’t spent much time with any other languages means he’s never been exposed to this very fundamental concept in computer science. Now, I’m fully aware that most web developers don’t normally have to deal with concurrent programming at all, but I still think it’s something every programmer should have some rudimentary knowledge of. And I’m willing to bet that if my friend had been working with Python or Ruby all these years, he would have at least seen some mention of threads in library documentation or someone else’s code.

But maybe the point isn’t just that PHP is harmful. Maybe it’s that spending all your time working with any one language is harmful. This conversation made me wonder what kinds of things I might be missing out on by spending all of my time working with Ruby and Python!

Project Euler

I’ve been working on Project Euler problems in my spare time for the last few weeks. I don’t really know what a “good” score is, but here’s my ranking so far:

I think one of the reasons these problems are so much fun to solve is because no single algorithm is necessarily more “correct” than any other (assuming you get the correct answer at the end). Consequently, there are widely varying philosophies when it comes to how they should be solved. Some people go for the most efficient algorithms, using all kinds of arcane bit-shifting tricks in low-level languages like C and assembly. Other people are only concerned with programmer productivity, opting for simple brute-force solutions that may take several hours to finish.

My personal goal has been to come up with elegant and reasonably efficient algorithms in as few lines as possible. Nice looking code almost always wins out over efficiency for me, but if my solution takes more than 30 seconds to finish, then I know I have some serious optimization to do. For a non-math guy, I think I’ve been doing pretty well so far, considering that most of my solutions finish in a fraction of a second, and the average across all my solutions is currently less than 3 seconds.

Which brings me to my next point. As a person who went through school absolutely dreading my next math class, I find it kind of amazing that I’ve been willing to spend hours solving these problems in my spare time. This tells me that there’s something seriously wrong with the way math is currently being taught. I suspect Bret Victor is on to something when he says “math needs a new interface.”