Not long ago, I had an interesting lunchtime discussion with Job Vranish. I don’t recall exactly how we got on the topic, but we were discussing how mesmerizing siphons are. As the discussion progressed to imagining a fluid-based computer (that’s another story…), I remembered something I had seen as a child at the Indianapolis Children’s museum: an immense clock that kept time with the flow of water through glass tubes.

Later I did some research into what I discovered is one of the largest water clocks ever made by the French physicist-turned-artist, Bernard Gitton.

Gitton’s water clock begins, as many clocks do, with a pendulum. This pendulum is attached to a small scoop into which a constant stream of water flows. As the pendulum swings, the scoops spills its contents into another vessel at regular intervals.

These “dollops” of water must then be transformed into the displayed minute and hour values along the sides of the clock.

First, the dollops drain from the collection vessel down a piece of glass tubing where they settle into the “trap” of an S-shaped siphon. The siphon is calibrated such that multiple dollops collect before the siphon triggers, suctioning the entirety of the contents down into the next component. These siphons are triggered when the fluid reaches the height of the top of the S-shaped curve and begins to be pulled downward by gravity. Once this process begins, the fluid continues to drain from the siphon until it is empty.

In the case of Gitton’s water clock, the fluid drains into a series of additional siphons, calibrated to hold increasing volumes of water before triggering. These siphons act as frequency dividers for the initial signal provided by the pendulum. (i.e. If a dollop comes every two seconds, and the first siphon can hold three dollops, then the first siphon will drain only every 6 seconds, the second siphon in the series may hold several of these larger dollops, dividing the frequency still more, and so forth.)

The last siphon in the series has a very interesting property – the low pressure created by draining of the vessel is “captured” by a sort of “vacuum locking” device that keeps the “vacuum” from “traveling” back up the input channel. Instead, the lower pressure is carried by another piece of tubing over to a vessel with a siphon on the brink of draining. (Just how this balance is calibrated is not clear to me.) When the “vacuum locked” siphon triggers and transfers this lower pressure to the other vessel, it triggers that siphon and drains the fluid into the minutes counter.

This minutes counter is column of stacked globes that contain the colored fluid. The height of the fluid in the column fills a number of the globes corresponding to the minutes past the hour. Alongside the column of minute-globes is another S-shaped siphon that triggers just before the hour, draining the entire column and triggering another vessel to drain into the hours column. This process takes several minutes, during which the observer has the distinct impression that time itself is being lost down the drain. The Greek idiom used to refer to water clocks is “Clepsydra”, which translates more literally to “water thief”. Gitton calls these clocks (of which there are several around the world), Time-Flow Clocks or “Horloges à voir le temps couler” (roughly translated: clocks for viewing the passing of time).

If you’re as fascinated by this device as I am, you can read more at this page, put together by David M. MacMillan, where you can find a much more detailed technical explanation along with a number of helpful diagrams. I also recommend watching the fantastic nine minute documentary, Bernard Gitton – Physics and Art, done by Michael Magee. Another explanation of the water clock, produced by The Children’s Museum of Indianapolis, can be viewed on YouTube. There are a few other videos on YouTube of these water clocks in operation, but most of them are not very clear. You can get a better sense of what is happening by watching this intricately animated illustration created by Roland Barth. Additionally, Gitton’s studio has a small webpage with contact information at www.bernard-gitton.com. (You may need to scroll sideways to reach the content in the frames on the Gitton site.)

What beautiful artifacts inspire you? If you have children, do you try to expose them to things that inspire? What feats of engineering are still impressed upon your memory from childhood?

• At the installer, choose "Repair your computer"
• Choose "troubleshoot"
• Choose "advanced options"
• Choose "command prompt"
• Run diskpart.

list disk

# now pick your disk
select disk 0
clear
create partition
format fs=ntfs compress quick
active

Some of you might have noticed that I`m cochairing the devops track for DrupalCon Munich,
The CFP is open till the 11th of this month and we are still actively looking for speakers.

We're trying to bridge the gap between drupal developers and the people that put their code to production, at scale.
But also enhancing the knowledge of infrastructure components Drupal developers depend on.

We're looking for talks both on culture (both success stories and failure) , automation,
specifically looking for people talking about drupal deployments , eg using tools like Capistrano, Chef, Puppet,
We want to hear where Continuous Integration fits in your deployment , do you do Continuous Delivery of a drupal environment.
And how do you test ... yes we like to hear a lot about testing , performance tests, security tests, application tests and so on.
... Or have you solved the content vs code vs config deployment problem yet ?

How are you measuring and monitoring these deployments and adding metrics to them so you can get good visibility on both
system and user actions of your platform. Have you build fancy dashboards showing your whole organisation the current state of your deployment ?

We're also looking for people talking about introducing different data backends, nosql, scaling different search backends , building your own cdn using smart filesystem setups.
Or making smart use of existing backends, such as tuning and scaling MySQL, memcached and others.

So lets make it clear to the community that drupal people do care about their code after they committed it in source control !

Please submit your talks here

Here is how I finally ended up by creating HTML PGP encrypted Mails using PHP which can be opened using (at least) claws-mail and thunderbird with proper rendering of the HTML report:

Content of the clear message

To: test@test.com
Subject: My HTML crypted report
X-PHP-Originating-Script: 1000:mlist.obj.php
From: "We Sun Solve" 
Reply-to: admin@wesunsolve.net
X-Sender: WeSunSolve v2.0
Message-ID: <1335717276@wesunsolve.net>
Date: Sun, 29 Apr 2012 18:34:36 +0200
MIME-Version: 1.0
Content-Type: multipart/encrypted;
 protocol="application/pgp-encrypted";
 boundary="------------enig029BFFF948226050D5D90E10F"

This is an OpenPGP/MIME encrypted message (RFC 2440 and 3156)
--------------enig029BFFF948226050D5D90E10F
Content-Type: application/pgp-encrypted
Content-Description: PGP/MIME version identification

Version: 1

--------------enig029BFFF948226050D5D90E10F
Content-Type: application/octet-stream; name="encrypted.asc"
Content-Description: OpenPGP encrypted message
Content-Disposition: inline; filename="encrypted.asc"

-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.10 (GNU/Linux)

****SNIPPED PGP CRYPTED BASE64 MESSAGE ****
-----END PGP MESSAGE-----

--------------enig029BFFF948226050D5D90E10F--

Content of the PGP encrypted part

Content-Type: multipart/alternative;
 boundary="------------F983FADF500537B8AFDC5E483"

This is a multi-part message in MIME format.
--------------F983FADF500537B8AFDC5E483
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

You need to have a MUA capable of rendering HTML to read
the WeSunSolve emails.

You can consult the website http://wesunsolve.net if you
are not able to read this email, the information sent to you
should also be on the website...

--------------F983FADF500537B8AFDC5E483
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

This is the report in cleartext!

--------------F983FADF500537B8AFDC5E483--

Generally I am very glad I started this process, I am actually continuing to use it today which is probably the longest I’ve ever used any TODO or reminder application. It really is good to be able to hack your own workflow and have a tool you can adapt to your needs rather than try to fit the mold of some off the shelf tool.

The time tracking feature has proven very valuable – as I work on my various projects I will track time worked with gwtf and review my mental estimate at the end of adding a feature. In time I am sure this will improve my ability to provide accurate time estimates while coding something for work etc.

The big thing I’ve added is a reminder system that can send notifications via email, boxcar or notifo (community contributed method). The image show my iPhone receiving a push message. All items can have a reminder date and in line with the Unix CLI approach this is done using your system at(1) command. Each notification can go to multiple recipients so I get email and push notifications.

% gwtf new this is a test --remind="now + 1 week" --done --ifopen
Creating reminder at job for item 30: job 46 at 2012-03-13 20:09
   30                 this is a test

Here I will get a reminder a week from now after reminding the item will be marked as done. The reminder will only be sent if item has not already been closed.

Building on this I added a special project called reminders that does not show up in the normal list output, this project is where simple one-off reminders (and soon repeating ones) go. Being hidden from the list output means I can have many of them without feeling like I have a huge TODO backlog since these aren’t strictly TODO items.

% gwtf remind --at="now +1 hour" do something
Creating reminder at job for item 84: job 66 at 2012-04-10 15:11
   103 L   2012-04-13 do something

Items now have due dates and there is a notification method that will email you all due and overdue tasks. I find if my todo list get mailed to me every day I get blind to it real quick, by only mailing due and overdue items they stand out and I pay them attention.

Various item list commands have been added. When I log into my shell I get the following, the colors show me that the projects have items due soon, it would go red when they are overdue.

I’ve added an overview mode to the list command that shows all projects and their open items, again color coded as above.

I’ve also augmented all the date processing using the excellent Chronic gem. This means anywhere I need a date or time specification I can use natural language dates. For example the due date specification can be as simple as –due=”next week” which would end up being due on next Wednesday. I can be more specific like –due=”7pm next tuesday” etc. I really like this mode of date input since I almost never know what the date is anyway it’s a big challenge to type in full dates for this kind of system.

That’s the big ticket items but there has been a ton of small tweaks. Overall I’ve done 16 releases of the Gem and it’s been downloaded 1600+ times from rubygems.org. I put a little website up for it using the new GitHub site system with full documentation etc.

I am not sure who the 1600 downloaders are, I am certainly not developing this with other peoples needs in mind but hopefully someone is gaining value from it.

Till now I’ve just done puppet runs via MCollective, basically I’d edit some puppet files and after comitting them just send off a puppet run with mcollective, supplying filters by hand so I only trigger runs on the appropriate nodes.

I started looking into git commit hooks to see if I can streamline this. I could of course just trigger a run on all nodes after a commit, there is no problem with capacity of masters etc to worry about. This is not very elegant so I thought I’d write something to parse my git push and trigger runs on just the right machines.

I’ll show a simplified version of the code here, the full version of the post-receive hook can be found here. I’ve removed the parse_hiera, parse_node and parse_modules functions from this but you can find them in the code linked to. To use this code you will need MCollective 2.0.0 that is due in a few days.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

#!/usr/bin/env ruby
 
require 'rubygems'
require 'grit'
require 'mcollective'
 
include MCollective::RPC
 
@matched_modules = []
@matched_nodes = []
@matched_facts = []
 
# read each git ref in the push and process them
while msg = gets
  old_sha, new_sha, ref = msg.split(' ', 3)
 
  repo = Grit::Repo.new(File.join(File.dirname(__FILE__), '..'))
 
  commit = repo.commit(new_sha)
 
  case ref
    when %r{^refs/heads/(.*)$}
      branch = $~[1]
      if branch == "master"
        puts "Commit on #{branch}"
        commit.diffs.each do |diff|
          puts "    %s" % diff.b_path
 
          # parse the paths and save them to the @matched_* arrays
          # these functions are in the full code paste linked to above
          case diff.b_path
            when /^hieradb/
              parse_hiera(diff.b_path)
            when /^nodes/
              parse_node(diff.b_path)
            when /^common\/modules/
              parse_modules(diff.b_path)
            else
              puts "ERROR: Do not know how to parse #{diff.b_path}"
          end
        end
      else
        puts "Commit on non master branch #{branch} ignoring"
      end
  end
end
 
unless @matched_modules.empty? && @matched_nodes.empty? && @matched_facts.empty?
  puppet = rpcclient("puppetd")
 
  nodes = []
  compound_filter = []
 
  nodes << @matched_nodes
 
  # if classes or facts are found then do a discover
  unless @matched_modules.empty? && @matched_facts.empty?
    compound_filter << @matched_modules << @matched_facts
 
    puppet.comound_filter compound_filter.flatten.uniq.join(" or ")
 
    nodes << puppet.discover
  end
 
  if nodes.flatten.uniq.empty?
    puts "No nodes discovered via mcollective or in commits"
    exit
  end
 
  # use new mc 2.0.0 pluggable discovery to supply node list
  # thats a combination of data discovered on the network and file named
  puppet.discover :nodes => nodes.flatten.uniq
 
  puts
  puts "Files matched classes: %s" % @matched_modules.join(", ") unless @matched_modules.empty?
  puts "Files matched nodes: %s" % @matched_nodes.join(", ") unless @matched_nodes.empty?
  puts "Files matched facts: %s" % @matched_facts.join(", ") unless @matched_facts.empty?
  puts
  puts "Triggering puppet runs on the following nodes:"
  puts
  puppet.discover.in_groups_of(3) do |nodes|
    puts "   %-20s %-20s %-20s" % nodes
  end
 
  puppet.runonce
 
  printrpcstats
else
  puts "ERROR: Could not determine a list of nodes to run"
end

The code between lines 14 and 46 just reads each line of the git post-receive hook STDIN and process them, you can read more about these hooks @ git-scm.com.

For each b path in the commit I parse its path based on puppet module conventions, node names, my hiera structure and some specific aspects of my file layouts. These end up in the @matched_modules, @matched_nodes and @matched_facts arrays.

MCollective 2.0.0 will let you supply node names not just from network based discovery but from any source really. Here I get node names from things like my node files, file names in iptables rules and such. Version 2.0.0 also supports a new query language for discovery which we use here. The goal is to do a network discovery only when I have non specific data like class names – if I found just a list of node names I do not need to do go out to the network to do discovery thanks to the new abilities of MCollective 2.0.0

In lines 48 to 90 I create a MCollective client to the puppetd agent, discover matching nodes and do the puppet runs.

If I found any code in the git push that matched either classes or facts I need to do a full MCollective discover based on those to get a node list. This is done using the new compound filtering language, the filter will look something like:

/some_class/ or some::other::class or fact=value

But this expensive network wide discovery is only run when there are facts or classes matched out of the commit.

Line 72 will supply the combined MCollective discovered nodes and node names discovered out of the code paths as discovery data which later in line 85 will get used to trigger the runs.

The end result of this can be seen here, the commit matched only 5 out of my 25 machines and only those will be run:

$ git push origin master
Counting objects: 13, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (7/7), 577 bytes, done.
Total 7 (delta 4), reused 0 (delta 0)
remote: Commit on master
remote:     common/modules/mcollective/manifests/client.pp
remote:
remote: Files matched classes: mcollective::client
remote:
remote: Triggering puppet runs on the following nodes:
remote:
remote:    node1                node2            node3
remote:    node4                node5
remote:
remote: 5 / 5
remote:
remote: Finished processing 5 / 5 hosts in 522.15 ms
To git@git:puppet.git
   7590a60..10ee4da  master -> master

• Added wiki to hold the documentation;
• Added the monitoring of multiple IPS Repositories;
• User list now allows the user to load multiple patches at once;
• Added patch timeline
• Added CVE list affecting Solaris packages
• Added patch link to CVE when issue is fixed
• Users can now download a ZIP containing all README of user patch list
• SSL Signed certificate added to wesunsolve.net https domain
• User login goes over SSL by default
• Added support for SRV4 Packages link to patches
• Modified the structure of patch level to link SRV4 packages
• Added user setting for API access
• Added function to API to allow server registration and adding a patch level
• Added patch/security report based on patch level and PCA execution
• Added mail report for patch/security based on a patch level and patchdiag.xref automatic selection
• We added a logo to our Wiki! (thanks to Dagobert Michelsen for the logo ;-)

Full listing of changes made can be found here

Patch level report (Using PCA)

PCA has been integrated into WeSunSolve so you can generate patch report on any server registered into your account where at least one patch level is defined.
The report which is created by WeSunSolve is based on the information you are entering when adding a server's patch level: showrev-p.out and pkginfo-l.out.
Theses two files are generated while running the Explorer or simply gathered by hand with the two corresponding commands. (respectively: /usr/bin/showrev -p and /usr/bin/pkginfo -l).

You can see there a full example of such generated report.
To generate a report like this, you must Add a server and an associated patch level, you can achieve this by following steps pointed in the documentation.

Please, give us feedback if you feel something is missing inside this report!

Mail reports

You can also get the previous report being sent to you by mail regularly, everything can be configured to fit your needs... You can:

• Choose the server and the patch level on which the report will be generated;
• Choose the interval between two reports being sent to you: every day, every week, every month ?
• You can decide which patchdiag.xref delay you want to have, this is the best if you always want to have a delay between what's out and what you will actually install.

This way, you can get a report of what patches are to be installed on your server based on an up-to-date baseline every day...

To create a report, simply follow the steps at our documentation.

API Access

As of now, you can enable the API access inside your panel and take advantage of the function we have recently implemented, like:

• Add a server easily;
• Upload a patch level directly from command line;

Least known features: Window size

If you are browsing WeSunSolve regularly, you can greatly enhance your browsing by fitting the size of the website to your resolution.
We've implemented three size of screen:

• 960px
• 1200px
• 1600px

Like it? Spread it!

Please, if you do like WeSunSolve, spread it over your fellow sysadmin! Write a blog post 'bout it and send it over to get a backlink :-)

You found a cool way of doing something with WeSunSolve that spared you hours of work? Please, tell us how! Don't hesitate to write a Howto on our wiki...

Finally, if you want to thank me personally, you can simply connect through LinkedIN and let a little recommendation on the WeSunSolve job...

A recent example of the usefulness of Vagrant is the new packaging and testing work undertaken in the Sensu project. The project set out to build a new set of native OS packages with a goal of making Sensu easy to deploy on a variety of platforms and without a lot of the friction that sometimes accompanies Ruby apps. As part of the packaging effort we needed a simple mechanism to build native packages on the relevant platforms, ie: .deb's on debian and .rpm on redhat/centos.

We ended up using a combination of Vagrant and some homegrown tools such as Bunchr.

You can see the work in these 2 repos:

• http://github.com/sensu/sensu-build
• http://github.com/joemiller/sensu-tests

Both codebases contain a para-vagrant.sh script that is used in place of the normal vagrant up to kick off parallel provisioning tasks. sensu-tests is the more interesting example as it runs a set of rspec tests against Sensu across 14 VM's and this will likely grow to encompass other OS's in the future. The tests are executed as Vagrant provisioners (a combo of Chef and shell to call rspec).

The simplest way to use multi-VM's with Vagrant is the typical vagrant up. However, this will boot and run the provisioning tasks sequentially on each VM. With 14 VM's to test, this process can take a long time.

Can we speed this up? Yes. In fact we were able to reduce the time taken to run the sensu-build tasks from about 33 minutes to 12 minutes, and reduced sensu-tasks from almost 90 minutes to 15!

Here was the first attempt at a parallelization script:

#!/bin/sh
 
# concurrency is hard, let's have a beer
 
MAX_PROCS=4
 
parallel_provision() {
    while read box; do
        echo "Provisioning '$box'. Output will be in: $box.out.txt" 1>&2
        echo $box
    done | xargs -P $MAX_PROCS -I"BOXNAME" \
        sh -c 'vagrant provision BOXNAME >BOXNAME.out.txt 2>&1 || echo "Error Occurred: BOXNAME"'
}
 
## -- main -- ##
 
# start boxes sequentially to avoid vbox explosions
vagrant up --no-provision
 
# but run provision tasks in parallel
cat <<EOF | parallel_provision
centos_5_64
centos_5_32
ubuntu_1004_32
ubuntu_1004_64
EOF

There are 2 steps to the process:

• Sequentially boot each VM listed in the Vagrantfile, but without running provisioners (vagrant up --no-provision). The reason we do the bootups sequentially is to avoid potential kernel panics that VirtualBox is prone to do, at least on OSX.

• Feed a list of VM's into xargs to execute $MAX_PROCS vagrant provision $BOXNAME processes in parallel. xargs will manage the concurrency for us.

The final script we used with the sensu-tests repo is a little different than the above example, which can be viewed here: para-vagrant.sh. This script reduced the time it takes to run the full test suite from over 90 minutes to 15 minutes. The differences in this final version of the script:

• uses GNU parallel instead of xargs for better handling/grouping of the output from the vagrant provision processes.

• reads the list of VM's from a JSON file (the Vagrantfile also uses this JSON file). To add/remove new VM's, you only need to edit one file now.

Future?

It should be possible to make the process go even faster by parallelizing the bootup phase, but I have not tried this yet because I'm worried about VirtualBox stability.

Also, I'm sure there's a more polished script/utility that can be written to parallelize any multi-VM Vagrantfile.

Alternatives?

As @vvuksan correctly points out, doing such short tasks with Vagrant is probably always going to be a little slow due to the overhead in spinning up VM's, so a possible alternative maybe LXC. There's already a Vagrant-like tool for LXC called toft. LXC should work fine for workflows that include only various Linux distributions, but won't help if you need to test other OS's like FreeBSD or Solaris.

Unfortunately, I couldn’t find any current packaging for logstash. However, the writer of Logstash (Jordan Sissel, aka whack) writes another utility called FPM (F’n Package Management) which allows you to easily create OS packages from a directory structure. So I forked the elasticsearch init script which is included with elasticsearch’s deb package, making one init script for each of the three moving parts in Logstash.

I put the default configs and /etc/default/logstash-{web,shipper,indexer} in a directory, and a couple of pre and post install scripts. Then, I downloaded the latest stable logstash all in one jar. I named this jar logstash.jar and put it in logstash_packaging/usr/share/logstash/.

I then used fpm to build a package like so:

1

fpm -n logstash -v 1.1.0.1yoursite1 -d grok -d default-jre -a all -C logstash_packaging/ -m "The Systems Group systems@yoursite.com" -t deb -s dir --pre-install ./logstash.preinstall --post-install ./logstash.postinstall --description "Logstash Open Source Log Management" --url 'http://www.logstash.net/'

Remember if you need to re-package, only bump the number after your sitename.

You can find the packaging files here: GitHub.

Now onto the actual deployment. As usual, I use Puppet for this. Basically I have a parameterized class that you feed a ‘role’ to. This is an array of which logstash agents you’ll be running. Depending on this role, certain services are enabled and started up, and their configs served out.

That code can be found here: GitHub.

A couple of small caveats. I put in an elasticsearch parameter, which should work. Just be wary that if you use an external elasticsearch server, you must use the version your logstash jar is compiled against. This is because the REST API was not fast enough, so Jordan had to use the Java messaging.

Also, be wary that elasticsearch discovery is done using multicast. So if you are firewalling, you may notice that changing your default policy to ACCEPT during startup of logstash alleviates the issue, which will probably manifest in you seeing the indexer see messages, but nothing appearing in elastisearch.

As theses delays are quite common when the SSHd is configured by default, I quickly added theses lines to remove GSSAPI and DNS common issues:

/etc/ssh/sshd_config

LookupClientHostnames no
VerifyReverseMapping no
GSSAPIAuthentication no

Although, theses settings didn't fixed the problem.

I added some verbosity to both ssh client and server and tracked down the delay to happen at this stage of the connection:

On the client:

$ ssh -v -p 2222 s11box -l adminifm
 OpenSSH_5.8p1-hpn13v10lpk, OpenSSL 1.0.0c 2 Dec 2010
debug1: Reading configuration data /home/wildcat/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to admblockum04 10.2.12.155 port 2222.
debug1: Connection established.
debug1: identity file /home/wildcat/.ssh/id_rsa type -1
debug1: identity file /home/wildcat/.ssh/id_rsa-cert type -1
debug1: identity file /home/wildcat/.ssh/id_dsa type 2
debug1: identity file /home/wildcat/.ssh/id_dsa-cert type -1
debug1: identity file /home/wildcat/.ssh/id_ecdsa type -1
debug1: identity file /home/wildcat/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version Sun_SSH_2.0
debug1: no match: Sun_SSH_2.0
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1-hpn13v10lpk
debug1: SSH2_MSG_KEXINIT sent
*** HANG ***

And on the server:

 debug1: Reloading X.509 host keys to avoid PKCS#11 fork issues.monitor debug1: reading the context from the   child
 debug1: use_engine is 'no'
 
 debug1: list_hostkey_types: ssh-rsa,ssh-dss
 debug1: My KEX proposal before adding the GSS KEX algorithm:
 debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-  group1-sha1
 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
 debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour128,arcfour256,arcfour
 debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour128,arcfour256,arcfour
 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
 debug2: kex_parse_kexinit: none,zlib
 debug2: kex_parse_kexinit: none,zlib
 debug2: kex_parse_kexinit: de-DE,en-US,es-ES,fr-FR,it-IT,ja-JP,ko-KR,pt-BR,zh-CN,zh-TW,i-default
 debug2: kex_parse_kexinit: de-DE,en-US,es-ES,fr-FR,it-IT,ja-JP,ko-KR,pt-BR,zh-CN,zh-TW,i-default
 debug2: kex_parse_kexinit: first_kex_follows 0 
 debug2: kex_parse_kexinit: reserved 0 
 *** HANG ***

Adding truss of the server process is helping us a lot:

  17477:  so_socket(PF_INET, SOCK_STREAM, IPPROTO_IP, 0, SOV_DEFAULT) = 3
  17469:  pollsys(0x080451C0, 1, 0x00000000, 0x00000000) (sleeping...)
  17477:  connect(3, 0x08047030, 16, SOV_DEFAULT) (sleeping...)

The lock is happening just after the connect() syscall. We can now check the pfiles of this process together with a netstat to identify which connection is causing trouble to be established:

  root@admblockum04:/local/home/ucc# pfiles 16585
  16585:  /usr/lib/ssh/sshd -f /etc/ssh/sshd_config -p 2222 -ddd -D
  Current rlimit: 256 file descriptor
    0: S_IFCHR mode:0620 dev:532,0 ino:3502445063 uid:60004 gid:7 rdev:221,8
      O_RDWR|O_NOCTTY|O_LARGEFILE
     /dev/pts/8
      offset:43267
   1: S_IFCHR mode:0620 dev:532,0 ino:3502445063 uid:60004 gid:7 rdev:221,8
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/8
      offset:43267
   2: S_IFCHR mode:0620 dev:532,0 ino:3502445063 uid:60004 gid:7 rdev:221,8
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/8
      offset:43267
   3: S_IFSOCK mode:0666 dev:540,0 ino:34566 uid:0 gid:0 size:0
      O_RDWR
        SOCK_STREAM
        SO_SNDBUF(49152),SO_RCVBUF(131072)
        sockname: AF_INET 127.0.0.1  port: 55867
        congestion control: newreno
   4: S_IFSOCK mode:0666 dev:540,0 ino:5714 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK
        SOCK_STREAM
        SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(128872)
        sockname: AF_INET6 ::ffff:10.2.12.155  port: 2222
        peername: AF_INET6 ::ffff:10.2.60.1  port: 43575
        congestion control: newreno
   5: S_IFIFO mode:0000 dev:529,0 ino:70783 uid:0 gid:0 size:0
     O_RDWR
    6: S_IFIFO mode:0000 dev:529,0 ino:70783 uid:0 gid:0 size:0
      O_RDWR
   8: S_IFIFO mode:0000 dev:529,0 ino:70784 uid:0 gid:0 size:0
      O_RDWR FD_CLOEXEC

 # netstat -an|grep 55867
 127.0.0.1.55867      127.0.0.1.30003          0      0 131072      0 SYN_SENT

The port 30003 is the default port of tcsd daemon, which is managing physical cryptography (through /dev/tpm). If there is no hardware crypto devices, this daemon is disabled. It seems though that cryptoadm is linking tpm crypto mechanism by default, enabling ssh to trying to access this daemon.

Workaround found (just to confirm slowliness is caused by tcsd):

Run this command on the server:

 # nc -e 'cat /dev/null' localhost 30003

and try to ssh the box, it should be fast.

Permanent workaround:

Simply remove the pcks11_tpm provider from the crypto framework:

cryptoadm disable provider=/usr/lib/security/\$ISA/pkcs11_tpm.so mechanism=all
cryptoadm uninstall provider=/usr/lib/security/\$ISA/pkcs11_tpm.so

Other references:

• Illumos Issue 1983
• Glassfish issue 1553