Backing up OS X onto Linux. Finally.

I've tried all sorts of black voodoo magic to make this work, but finally I have something repeatable and reliable. Also, please excuse the horrible formatting of this post.

History

I started with Time Machine talking to my Linux box with netatalk. This worked fine until one day Time Machine told me it needed to back up everything all over again.

Then it started to tell me this almost weekly.  Apparently when this happens, the existing backup is corrupt and all that precious data is at worst irretrievable or at best tedious to retrieve. These are not attributes I want associated with my backup solution.

Then I did an rsync over ssh to my Linux box.  This is fine exception that it lacks all the special permissions, resource forks, acls, etc, etc that are hidden in a Mac filesystem.

Then I tried SuperDuper! backing up to a directory served up via netatalk mounted via afp:// on OS X.  This worked, but was mind numbingly slow. Also, it would mean I'd have to pay for a tool if I wanted to do incremental backups. This gets expensive as I also back up a few friends OS X laptops on my Linux file server.

I tried SuperDuper! backing up over Samba, but hdiutil create apparently doesn't work over Samba. Workarounds all needed the purchased version of SuperDuper!.

There's *another* work around for SuperDuper! where I can use MacFUSE and sshfs, but the MacFUSE author has abandoned the project and recommends people not to use it.

Sheesh.

The Solution

Ultimately, the goal is to make a sparsebundle HFS+ disk image, put it on a Samba mounted share and rsync my data over to it. You'd be surprised how many niggly bits there are for this.

Install Rsync

First, I grabbed the 3.1.x version of rsync from Homebrew - install Homebrew as per the directions there, then run:

brew install https://raw.github.com/Homebrew/homebrew-dupes/master/rsync.rb

If you've been digging through voodoo magic, then you'll be happy to hear this version of rsync has the all the rsync patches you'll read about (like --protect-decmpfs).

Samba

Nobody needs another out of date blog entry explaining how to setup Samba. Follow some other guide, make sure Samba starts automatically and use smbpasswd to create an account.  

I recommend using the name of the machine being backed up as the account name. I'm calling that machinename for the rest of this post.

Make sure you can mount this share on OS X via smb:// ( Finder > Go > Connect to Server... ).  Make sure you can 1) create a file, 2) edit and save the file, 3) delete the file.  I'm going to assume you've mounted this share at /Volumes/Machinename

Backup Into

Now lets make something for us to backup into.  Figure out how big the disk is on the source machine (we'll assume 100g) then run:

hdiutil create /tmp/backup.sparsebundle -size 100g -type SPARSEBUNDLE -nospotlight -volname 'Machinename-Backup' -verbose -fs 'Case-sensitive Journaled HFS+'

Yes, you're creating it in /tmp, this is to work around hdiutil create not liking Samba.

Next you'll want to copy this sparse bundle onto your Samba share:

 cp -rvp /tmp/backup.sparsebundle /Volumes/machinename

This will copy a bunch of files and should be successful without any warnings.  Now lets mount this sparse bundle:

hdiutil attach /Volumes/machinename/backup.sparsebundle -mount required -verbose

You should now have /Volumes/Machinename-Backup mounted on your system. Fun story, OS X recognizes that this disk image is hosted off the machine, so it mounts this disk image with "noowners" (see mount man page). That's going to be a problem for our backup, so we need to tell OS X it's okay to use userids normally:

sudo diskutil enableOwnership /Volumes/Machinename-Backup

Preparing Rsync

There are a handful of files which recommend to be excluded:

.DocumentRevisions-*/
.Spotlight-*/
/.fseventsd
/.hotfiles.btree
.Trashes
/afs/*
/automount/*
/cores/*
/private/var/db/dyld/dyld_*
/System/Library/Caches/com.apple.bootstamps/*
/System/Library/Caches/com.apple.corestorage/*
/System/Library/Caches/com.apple.kext.caches/*
/dev/*
/automount
/.vol/*
/net
/private/tmp/*
/private/var/run/*
/private/var/spool/postfix/*
/private/var/vm/*
/private/var/folders/*
/Previous Systems.localized
/tmp/*
/Volumes/*
*/.Trash
/Backups.backupdb
/.MobileBackups
/.bzvol
/PGPWDE01
/PGPWDE02

Store this in a file somewhere. I stored mine as exclude-from.txt in /Volumes/Machinename

Okay, now we're ready to run rsync. I think the correct arguments to rsync are: -aNHAXx --nfs-compression --protect-decmpfs --itemize-changes --super --fileflags --force-change --crtimes

So, we run:

rsync all-those-args-above --exclude-from=/Volumes/Machinename/exclude-from.txt / /Volumes/Machinename-Backup/

When The Backup Is Done

This will take a little while. When it's done, you can then bless your backup so it could be booted:

sudo bless -folder /Volumes/Machinename-Backup/System/Library/CoreServices

Then you can umount your backup:

hdiutil detach /Volumes/Machinename-Backup

Periodically, and after your first run, you should compact down your sparsebundle disk image:

hdiutil compact /Volumes/Machinename/backup.sparsebundle -batteryallowed

You can now log into your Linux server and tar up the backup (apparently XZ > bzip2 for compression size).

 tar Jcvf machnename-backup.tar.xz backup.sparsebundle

Depending on the size of that tarball, you could upload it to Google Drive, Drop Box, etc. Before you do, you'll probably want to encrypt it. I used OpenSSL:

openssl aes-256-cbc -a -salt -in machinename-backup.tar.xz -out machinename-backup.tar.xz.aes-256-cbc

Many of these steps will take *hours* if you have a lot of data, so you may consider just backing up parts of your system more frequently and doing your whole system once every-so-often.

Why This Is Nice

One thing I really love about this setup is that each piece of the puzzle does one thing and does it well. If Samba is too slow, I could go back to netatalk. The sparsebundle disk image hosts HFS+ properly with all its OS X-specific voodoo and rsync's job is to copy files. If there's a better file copier, I could drop that in.

Conclusion

I left a lot out. I know. I'm kind of expecting you to have a rough idea of how to get around OS X and Linux, figure out how to put most of the above in a shell script, decide when to do backups, how to store those tarballs, etc. Hopefully though this will help someone that just needs some of the key ingredients to make it work.  

Navigation 2.0: The Missing Features

I recently found myself driving a car in an unfamiliar city relying heavily on Google Navigation on my phone to get me from point A to point B (and back to point A again). From this two week experience I found some things in the Navigation experience I'd really love to have. Maybe other Navigation apps have this and I've just been googleblind.

Not Looking At The Phone

Most of the time I'm driving, I'm relying on Navigation to tell me what's going on. You're probably the same. Some simple options would make me a lot more comfortable. These also apply to when I'm on a motorbike and my phone is locked away in a pocket.

1. Don't tell me compass directions: If I'm in a new city, in a new area telling me "Head East" is useless. Especially at night where I can't use the Sun to figure out which way East is.
2. Give me a hint which lane I should eventually be in, "Travel three kilometers, eventually turning left."
3. If I'm driving for thirty minutes on a highway, periodically remind me that you haven't crashed or suddenly run out of battery power. Even a little beep every 10 minutes (or % of a leg) would be nice. Maybe even tell me what % of battery is left.
4. Along those lines, since Navigation knows how fast I'm going and how far I have to go, telling me "Continue for about 15 minutes." is more handy than "continue for 23 kilometers".
5. On navigation start, tell me something useful; even "Route calculated" or "Estimated twenty minutes of travel time."
6. If I'll have less than 10% battery life by the calculated arrival time, tell me this up front and give me the option to just read the list without actual navigation.
7. It may annoy you, but I like to know when Navigation has changed its mind about where we're going. Telling me that a route has been recalculated is re-assuring.

Tell Me Less

There are some things where being quieter makes more sense:

1. Cut me some slack in a parking lot. Unless you're going to tell me exactly how to get out of the parking lot, keep quiet until I'm close to an exit. Calculate some routes of the closest few exits and get ready to tell me "Turn left here." or "Turn right here.". Telling me to "Head North to 11th Ave South West going to 23rd Street East going East." is just distracting while I'm trying to navigate the maze of grocery carts and SUVs.
2. Some cities have Main St. North, West, East, South or North East / North West / etc.  If I'm on "Main St North West" turning onto "1st Street North West", don't say "Northwest" both times.  In fact, unless I'm close to a boundary where it's important to know which cardinal direction of the street I'm on, simply saying "Turn left onto First Street." is fine.

Nicer Touches

And there are a few nice things I bet wouldn't be too hard.

1. Traffic Lights. I'm sure most cities have a database of which intersections have traffic lights. Telling me "At the lights, turn left onto Third street." trumps "In six hundred and fifty meters turn left onto Third street.".
2. Let me mark certain areas as "well known" areas and silence turn by turn directions if my destination is in that area unless I ask for them to be re-enabled.
3. All major highways in Canada have numbered exits. They're numbered based on kilometer distance. If I'm at exit 35, and I'm going to exit 25, I know there's ten kilometers to go. When I get on a highway, let me know the exit number I'm going to be exiting on.
4. If I'm going to a destination that has a Google Place (or whatever they're called now), and it seems that they'll be closed or closing within an hour of the calculated arrival, give me an option to call the place.
5. Let me tell Navigation how comfortable I am with a given city or area. If I don't know it very well, give me notifications earlier and take me on possibly longer but easier to navigate routes.

New Plugin: licensing-maven-plugin

Ever wanted to know what licenses your dependencies (and their dependencies) are using? Maybe you work for a company that wants to sell their source code so you're wanting to avoid the GPL (and AGPL)? I've got the plugin for you!

I mentioned a while back that the build process at my day job had been declared bankrupt. Well, it's doing much better now; what used to be a multi-day process where you were never quite sure if it was working 100% now takes less than an hour (including data population). We're quite happy about that part.

Along the way we started looking at some of the other bits that fit more into release management; one of them was a "licensing report". This report listed most of our dependencies and which open source license they were in. Instead of hacking at the old scripts, we decided to let Maven take over and handle providing licensing and dependency information.

So with a rough idea of what we wanted to do, I put together the Licensing Maven Plugin. It has a few handy features:

1. Transitively aggregate licensing information of dependencies of child modules in multi-module reactors (or in English, it works the way you would expect it to on multi-module builds).
2. Coalesce license names (so "Apache License, Version 2.0", "Apache 2.0" and "ASLv2.0" can all be reported as "The Apache Software License, Version 2.0").
3. Fail builds if a dependency is only available under a disliked license.
4. Exempt artifacts under disliked licenses from failing the build.
5. Manually declare licenses for dependencies that fail to provide their own.

It's hosted on central, so give it a whirl:

mvn org.linuxstuff.maven:licensing-maven-plugin:1.5:aggregate

You'll see a truckload of warnings go by, and when it stops, you'll have a target/aggregated-third-party-licensing.xml file (yes, I know it's not nicely formatted yet).

If you'd like some more details, checkout the licensing-maven-plugin README.

Hello Developers, look at your code.

Hello developers, look at your code, now back to me, now back at your code, now back to me. Sadly, your code isn't me, but if it stopped collecting tech debt and you started making time for technical quality it could smell like me. Page down, back up. Where are you? You're at an interface with the code that your code could smell like. What's in your hand, back at me. I have it, it's an oyster with two different implementations to that thing you have an API for. Look again, the implementation is now secure and scalable. Anything is possible when your code has no smells. I'm on a horse.

A Patentable Idea

A quick post that if someone were to come up with a "USB Charger Condom" that I could plug into those free charging stations (or into my phone) which acted like an application layer proxy for my phone and prevented the charging station for giving anything more than power and not access anything on my phone that I'd gladly use one, and I'd make everyone I know use one too.

Real Reasons Developers Leave Jobs

While I'm still working on improving my commenting skills, I've also started trying to ponder how to write software in company with high developer turnover.

You see, I've always said that software reflects the team that writes it.  If you have a team of people who are seriously experienced at writing stored procedures and OSGi, you can expect that the software they write will use that.  On the flip side, a team who knows JavaScript and CSS, asked to write the same software will write something completely different. So, given an environment that has higher turnover, how does someone design a system that embraces and survives?

While I was researching about turnover, I found a survey on why developers left their current positions (or rather, why they would leave their current position).

The survey is a bit older, but the list is pretty fascinating:

1. Higher Salary
2. More Experience
3. More Interesting Work
4. Need for a Change
5. More Challenge
6. Greater Chance of Promotion
7. Better Location
8. Present Company Getting too Large
9. More Control over own Work
10. Keep Up With Technical Knowledge
11. Run Own Company
12. More Flexible Working Arrangements
13. Greater Structure in Work
14. Go to More Successful Company
15. Difficulties with Immediate Boss
16. Friendlier Company
17. Opportunity for Equity Stake
18. Go to Less Commercial Company
19. Go to More Commercial Company

Write Better Comments by Tomorrow Morning

I've spent some time recently trying to improve my code comments.  Most of us just write comments without much thought and carry on with the task at hand, however after a couple of weeks of iterative improvement I have some really simple tips that will make you write better comments tomorrow.

Tip 1: Delete Bad Comments

If you're working on a large code base, you'll invariably find some comment that's questionable, but sometimes you'll find comments that are even worse, and those comments simply need to go.  To me, the two kinds of comments that fit in this category are:

1. Machine generated comments: These tend to be blatant and useless.  "The getFoo method gets Foo." or "The isNotEnabled method returns the value of notEnabled.".  If you're using Eclipse, the shortcut Ctrl-D can be used to delete a line and these comments are ripe for the killing.
2. Lies: Over time code is refactored, shaped, copy-pasted, "fixed", outsourced, insourced, sold, bought until it eventually winds up in front of you.  During all this ebb and flow, some comments don't keep up with the code.  If the method signature you're looking at is "public List obtainShipments(Locale locale, Country country);" and the comment is "This method will return true if the system can ship to the given country." then you know it's wrong.   If this is not code you're currently cleaning up, then delete it.  

Wait, this sounds harsh.  You should spend the time to rewrite it as a proper comment, right?  Wrong, and here's why.

If you've been a developer for a while, you know the feeling of being "in the zone".  That time where the entire system you're working on is in your head and you can think of side-effects and how everything works.  You're one with the matrix. If you're not working on the methods with the generated comments and lies, then you're not going to add any value trying to write comments and it's going to pull you out of the zone; so don't.

Also, the comments that you see there will leave you (and others) with the feeling that comments are there to help you.  You've probably scanned a class before and thought "Uh-oh, there's almost no comments here, this could be tricky."  With the above two types of comments in that class it gives you a false sense of security.  You've got source control.  Bad comments are worse than no comments, delete them and move on.

Tip 2:  Explain the WTF's

Sometimes the code you're working on has something weird and counter-intuitive about it.  Things that you totally understand, but others may not.  Usually they're the result of bugs or missing features elsewhere in the system.  When you're writing code that works around a bug, go find the issue # in that project's issue tracker and add the full URL in a comment.  Also explain what you would rather the code do if the bug weren't there.  If there isn't an existing bug then file one.  You may discover you misunderstood some functionality or that it's fixed in a newer version.  

If you can't upgrade the library you depend on, mention that the bug is fixed in a newer version in your comment. Also, if you use a dependency management system like Maven or Ivy, add a comment near the dependency "Version 2.1.2 will fix BUG-7509 and then we can remove the workaround in SuperFancyImpl". This tiny investment of time will help the next developer who looks at the code.

Tip 3:  Describe Relationships

More than the other two tips, this tip takes your comments to the next level.

A developer will stumble across your code, following data flow.  They'll want to know why your code is here and how it interacts with other code, so tell them!

Lets imagine a restaurant simulator.   As a developer you find the "Table" class.  Comments like "A table in a restaurant." or "An implementation of a Table." are good Ctrl-D candidates.  Instead, here's something a little better:

"Tables represent a location in a restaurant where customers are served food.  Restaurant Customers attach themselves to a Table which is then served by a number of TableWorkers including the Server Employee and TableCleaner.  It is often allocated by the Reservation system, check CustomerSeater for how Customers are seated.  Constructed Tables usually come from the TableFactory and are always added to the RestaurantBuilding."

Let's take a look at the recipe this comment follows:

1. Who Am I: If this class represents a business object or business activity, then copy and paste the description from any documentation or planning sessions you've had.  If there isn't any, then write a note to yourself about why you decided you wanted to make a new class.
2. My Close Family: This class describes very simply how it interacts with other classes around it, either in the same package or members that would reference it.
3. Bend Encapsulation: In the comment we mention TableCleaner as an implementation of TableWorker.  Chances are the Table class doesn't have any code reference to TableCleaner, but our comment does.  This can be an "Aha!" moment for a developer.  Eclipse is smart enough to dig around in comments when you're refactoring so it's not as dangerous as you might think.
4. Use "usually" and "often": Should Table know that it comes from a TableFactory? No.  Should a developer know this? Absolutely.  By saying "usually", we hint to the developer that this is probably the common case, but because of the separation we can't say definitively this will be true.

The last two ingredients are what makes this comment a cut above the rest. It helps guide me up and beyond what most other comments would help me with.

What If I Have To Have Comments?

You might wondering what to do if you have a tool that enforces comments in your system and will fail your CI build if you start getting Ctrl-D happy.  It's pretty simple, disable the rules that force you to write comments.  

At my work place we have rules that all methods must have comments, and comments must end with a period for proper grammar.  Want to know what comment I've seen more times than I'd like to admit?

 /**

  * .

    * @return boolean .

    */

Yup.  The problem with enforcing comments is you're enforcing their existence, not their content and simply having a comment isn't ever going to be enough.  The code metric tools don't enforce quality, just quantity.  So ask people around you if it's more important to have lots of comments or if it's important to have better comments.  Maybe for your workplace, quantity is better...

 

That's It!

I'm still experimenting with other techniques, but so far these have had a great impact in my development. Other tips and ideas are welcome!

Technical Debt

Our build came up recently in a planning session at work. It's a standard big ball of mud, with hidden knowledge all over. It's frail, inconsistent and unhelpful, it's complicated to work with and almost nobody ever gets their local build past "good enough for now" to "working well".

This has been an issue for a while and it hadn't been given the proper attention to be improved. Every change had been reactionary to various needs and it had changed hands so many times that any beliefs or design philosophies it had were long lost.

One developer estimated that wrangling the build system alone accounted for 30% of their development time (that's three days of a two week iteration).

When asked how much effort was required to fix it, the answer was "You can't."

Taking Out Your Technical Loan

While I wasn't originally, I'm now a fan of the term "technical debt". It brings with it a handful of related concepts and terminology that make discussions easier. In this case, compound interest, and bankruptcy.

It all starts when you first make a big investment; probably your first release. Fixing up the last minute bugs before release, cutting corners in the name of getting to market, making "just for now" trade offs going against technical soundness. Each of these things add to your technical debt. There's nothing wrong with this, it's expected.

But then your debt starts adding up pretty quickly:

• Changes in business needs for the system
• Changes in company direction
• Changes in team composition, staff sizing
• Cutting corners on in-code documentation
• Industry changes
• Not keeping up with underlying frameworks
• Not following proper language techniques
• Quick security patches
• Scalability needs
• Blindly meeting code metrics
• Over reliance on under-skilled developers
• Not testing on up to date integration points (databases, operating systems, etc)
• Quick and dirty bug fixes
• Minor improvements

Any of these things can put you further into debt, and while you're in debt you'll need to pay compound interest on your technical debt. What was one day of work to fix this iteration can become two days of work a few iterations from now and a week of work next release. Leave it longer and suddenly it's a month of work two years from now.

How does this happen?

Lots of ways.

One of the beautiful things about refactoring is the way it takes into consideration the "context" that a developer has gained in immediately working with the code. Their short-term memory is filled with nuances and intents and allows them to glide smoothly through code. They can envision the refactoring and perform it optimally.

As time passes, they lose some of the context. Surrounding code begins to change, bug fixes are introduced, and other developers build on what was written instead of what was intended. More time passes and other code is refactored, team members come and go or the pattern gets copied elsewhere in your code base.

A few iterations later, there's a hierarchy of dependencies built on this code and what was once a simple refactoring has become a costly code spill cleanup effort.

Eventually the frameworks become out of date and coding styles fall out of style. Instead of wanting to clean up the code, new developers will work around it, introduce a functional duplicate, wrap it or simply make their changes as a series of hacks.

If code quality drops too low, some say morale will follow and keeping more than a handful of experienced developers will become a challenge; attracting developers eager to land their first job, negative velocity developers or those that simply don't care.

This affects your ability to pay off your debt.

Ways To Pay

Every organization has different priorities, goals and principles. As such, calculating how or where to pay for your technical debt varies wildly. Of course, the rule of thumb is "Pay off debt that is costing you the most." There are a handful of ways to do this:

• Research and Analysis: Work to establish what business functionality should be delivered next then prototype any areas that are known hot spots or developers do not have experience with. With the gained experience determine if the code in question is deeply in debt or if it can afford the new feature.
• Raise Rates: When asked to estimate a new feature in an area that has high technical debt, focus on estimates which include needed code revitalization. Just as it's the business duty to plan features customers want, it's a developer's responsibility manage risk ensuring it's done technically correct.
• Foreign Exchange: Exchange the technical currency into salaries as an incentive to keep developers. Many larger, non-technical firms follow this route. It's like buying carbon credits; the debt is still there, but developers are rewarded for tolerating it.
• Creative Accounting: Buffer estimates and deliverables with needed cleanups. Once someone knows an area of code well, perform small improvements under other ledgers.
• Get a Second Job: One of the most dire methods is to band together and either work nights or through lunchtime to make the needed improvements.
• Bankruptcy: Sometimes there's honestly no other choice other than to start over.

Back To The Build

Our build system had gone bankrupt. It had gained such complexity that we could no longer service the interest on the debt.

In our case, this opened the door to switching build tools, cleaning up packaging and performing much needed work on the development environment. It's exciting, as it will not only boost the productivity of each developer, but it hopes to also be something that isn't always dreaded; so developers can focus on solving their problems more effectively and not how to make sure everything builds properly.

And, if the payments are made on it on a regular basis, serve our needs well for years to come.

Who Owns Your Code?

At work, our marketing team is in control of deciding where development spends time on products. This is an effective way to ensure that our product meets the needs of the market we're in.

In fact, I would say we have an effective backlog of "big features" that are desired by our target market. We're only limited by the resources we have on hand to deliver these features. Therein lies the challenge; almost every feature is done on a tight timeline, and once it's in a non-laughable state, it's released and the team goes on to the next big feature. No time is really spent revisiting the overall design of the product, or just bringing its dependencies up to date.

In a previous workplace, the development team had strong say on what was worked on. This was an effective way to ensure the product was relatively nice to work on and its technology up to date. New features were strongly scrutinized and often there was a focus on picking newer technologies that made it easier to develop and service. The challenge was we'd fall behind on bolting on new business features from time to time.

In yet another workplace, our SEO team had the strongest say in what happened. Our main products were publicly accessible websites, but I feel like this was a pretty unique situation. Many technical decisions had to be discussed with the SEO team (which CMS should be chosen, which captcha system should be employed, etc..) and everything from character set encodings, performance trade offs and cache expiry times needed to have their approval. Our challenge there was that the success of some SEO practices are not easily measured, it was difficult to assign a ROI on particular decisions.

Who owns your product? How does that shape it?

Negative Velocity Developers

I've worked on a handful of projects over my career where there's been a developer that I've started to coin as "negative velocity".

Their positive impact on projects is best when they're on vacation or spinning their wheels in meetings. Everything they touch could instantly earn an @Deprecated annotation, and often when they're focused on their next tasks, the rest of the team is left to either throw out and rewrite, duplicate in parallel or at best refactor the work they've done.

Some of the best code they write is copy and pasted from other code, some of the worst is of their own design. Often the team works hard to try to minimize the impact of the developer, but they simply just don't understand what they're doing.

Agile projects make the results of this developer even more vicious as they get to choose what area of the project they will molest next. Also, many teams are "agile but" where the team follows some practices, they are not empowered to remove that developer from the team.

Often those that make employment changing decisions don't have the tools to validate such a claim. So what's one to do with a negative velocity developer? How does one capture enough information to prove to someone non-technical that their skill sets would best be used elsewhere?

Note: This is not a current situation for me.

What Your Hiring Process Says About You

What's your hiring experience like? How many "hoops" do you make candidates go through? Why do you make them go through them? In this post I'll recount some hiring processes I experienced and chat a bit about what (as a candidate) they said about the company. At the end I have a potential checklist.

1. Longest: Archiving Company

As a referral, I was spared an initial phone screen and instead was brought in person to meet an operations manager. His questions were non-technical and focused along the "How well do you deal with conflict? How well do you deal with challenging people to work with? Have you been in an environment where people were difficult to get along with?". We ran over time and I was asked to come back to talk to a project manager.

I returned a few days later and the interviewer was reading off situational HR questions, all with a tilt toward conflict resolution.

A few days later, I was asked to complete an assignment - write a service with certain features, complete with documentation and tests. Unfortunately I received the assignment on a Friday and was told it had to be completed before Monday morning. This was a surprise and a bit of a disappointment as it conflicted with birthday plans I had that weekend.

A few days later I was brought in for a third interview with a developer, project manager and product manager. Very few questions were asked about my assignment and the developer asked me some technical questions (some were unrelated to the problem domain they were in). One was my own question that my previous co-worker had brought with him to the company! Further general questions about process followed from their project manager and product manager.

As the interview was closing up, I discovered they'd like me to write two tests: a number matching test and an IQ test. Both timed, both while a loud lunch party was happening through the adjacent wall. The number matching was a doubled-sided, double-columned list. Each entry had a hand written number and a printed number; I was to mark an X on the ones that differed. The IQ test was pretty standard with tricky logic, English and math questions.

Two days later I received a call from their receptionist saying they were no longer interested. Naturally I contacted my reference to find out more. He responded that as far as he knew, they were still discussing it. He confirmed this later, it was a mistake on their part.

The next day he contacted me with their decision; they were no longer interested. Infact, he said that it was a mistake to even begin the interview process with me as they weren't looking for me.

(Had they been interested, I would have been asked back for an additional interview with the CEO.)

Notes

Needless to say I was disappointed in this process and if it wasn't for the initial reference, I would have given up on this path much sooner.

Explain the workflow to the candidate. Before I started the process I was told it was similar to a previous company I worked at. It was most definitely not. Not knowing the steps in the hiring workflow, a candidate has no idea that he's about to invest a dozen hours.

Screen for culture. Every organization has a different "culture", and it's important to double check that a person will fit in with the existing employees (or if they'll shake things up in a good way!)

Your questions will reflect what a candidate thinks is important to you. When three separate employees ask questions about navigating difficult people, constantly changing requirements and micromanagement, this is what your candidate will think your place is like. If you ask questions about networking protocols or multi-threaded programming, then your candidate will feel the position will involve both. When I asked "Do you use the technology you're asking me about?", I was surprised their answer was "Not really.".

Sell the company to the candidate. I was walked around the development area on two occasions and got the feel of the environment. I was told about some of the unique perks of the company too. This kept me interested.

Involve only the right people. Out of the five people who interviewed me, one was for technical purposes and the other I would potentially be reporting to. The other three were brought in "just to meet me" and "had no real serious questions to ask". As a candidate, this tells me your decision making process requires people who may be uninvolved with a decision to approve.

Limit an interview time. After about 90 minutes, my ability to continue to answer a barrage of questions begins to diminish. After two hours, I really felt that writing a surprise number-matching and IQ test was quite unfair.

Hiring processes should be fail-fast. As soon as an organization realizes it's not interested, the process should stop. Clearly this one suffered from poor internal communication and lack of direction. Consuming not only my time, but employee time too.

The "assignment" was an interesting project, but given the investment I made and how little it was referenced, I was quite disappointed to waste my time on it. I got the feeling the developer hadn't had time to read it, or that this was just an extra step he disagreed with. The length of this process gave me a lot of experience in what not to do. Had I been accepted, refactoring the hiring workflow was going to be a personal side project.

Shortest: Payment Company

Reading up about the company, I figured that I would be a good fit. I emailed a simple cover letter with some specifics of where I'd fit in. Their CTO responded back and we set up an in-person interview.

The interview with the CTO went well, but he was hoping to have me fill a different role than I was looking for. I met after with two of their developers who were prepared to ask me questions relating to the other role and were ill-prepared to ask me Java related questions.

The next day I heard back from the CTO; his team didn't feel I was Java-savvy enough.

Notes

Short, and sweet. Within a few days this company came and left my radar.

Know what you're looking for. Before bringing in candidates, clearly define the role you're looking for. If you're not sure (it happens sometimes), then spend time defining what you're not looking for.

Be prepared. Having stock questions to get a conversation going is fine, but if you haven't read the candidate's resume and don't know what to expect, you'll be ill-prepared to ask the right questions to find out their skills, and most likely opt not to hiring them simply because of this.

They did an excellent job explaining the technology, where they were going and what challenges were ahead for them. The developers I chatted with were interesting, but since they had a "systems guy" and not a "Java guy" mindset, they didn't know what to ask.

Accepted: eCommerce Company

Did some research to discover they'd been looking for someone to fill this position for a while, skipped their online application form and found the email address for hiring. I had customized my cover letter to ensure they realized I was looking for a specific role and highlighted how my experience meshes with what I was looking for.

I received a call back from their hiring team and we set up a phone screen. During the phone screen we discussed what I was looking for, salary expectations, and more about what the company does. From there I was told there was an online Java quiz to do.

The quiz was timed and fairly wide reaching.

I heard back a couple days later that they would like to schedule an in-person technical and HR interview. I asked if this was the interview or if there were multiple steps. At the time I was told that this was the interview, so I fit it in the day before I left on a trip.

I met with a developer there who knew his stuff, and he asked me a stock set of questions and asked me to clarify something from the quiz. Apparently I had got one question wrong but they were still interested. After, I met with the internal recruiter and we talked more about the company and where things were. She let me know that there were two more steps: a refactoring exercise and a second interview.

My first interview went well and I received the assignment while on my trip. Once I returned I sent back my answer and a couple days later I was asked in for a second interview. The assignment took probably an hour or two.

This interview I met with their development manager and we discussed my solution, my decisions, alternatives, drawbacks and technology in general. Later I met with their CTO who asked me some higher level questions.

Then I waited. After what seemed unusually long, I was asked to provide references. After I did, it took an additional week to check them; mid-week I was told why. The company was making some internal changes and this role may not be exactly what it initially was. After discussing it further, I was still on board and was asked for a third interview - more of a just a chat - with the person who would signoff on the hiring paperwork.

I was in the next day and chatted with the VP in question for approximately 10 minutes and he was sold; an offer was made later that day.

Notes

Internal political changes made this hiring process a lot longer than it should have been, and being accidentally mislead about the hiring process was a bit of a disappointment.

Funnel Requests. A month prior to emailing them directly, I had filled out an online application form on their site and never heard from them. Since I felt I was a strong potential, I skipped that step when I later saw the role was still unfilled. I still wonder where my first application went.. Make sure that all different interfaces candidates can apply through are given equal priority.

Have Discussions. In a previous company, I would bring whatever problem I was currently tackling to the interview and ask the candidate, providing as much background as I could. We'd work through it together and see where we got. Having a refactoring assignment to chat about simulated this and turned the interview from Q&A to more of a free flowing discussion of ideas.

Summary

The eCommerce Company with their phone screen and online quiz had a "fail-fast" process, and I met only with relevant people. They mostly knew what they were looking for, but internal change made the hiring more challenging. I only answered a handful of direct "fit" questions. Everyone I chatted with was open about the challenges they were facing but also quick to highlight some of the great things they really liked. I discovered that both the quiz and the assignment came from existing weaknesses in their team and code base.

The Checklist

• We explain our complete hiring workflow to candidates up front and keep them informed of their progress.
• We will stop the hiring process as soon as we feel the answer would be "no". Our process is optimized to make this happen as soon as possible.
• We limit the time an interview will take, but will cut it short if the answer is a definite "yes" or "no".
• All potential candidates which enter our hiring system are treated equally, regardless of source.
• We ensure candidates will be a good fit with the rest of the people here, and we feel we can do this mostly by discussing relevant topics with them.
• Our interviewers always take time to prepare before an interview.
• Our questions are relevant to the company, technology and role we're looking to fill.
• We know the specific skills needed to succeed in the role we're looking to fill.
• Our hiring process includes conversations with candidates similar to the ones that we have internally already.
• We involve the fewest people necessary, each person adds unique value to the process.
• We're honest about our challenges, and we mention the great things about our company throughout the hiring process.

Winner of the Worst Java API Award Goes To.....

One of my Java interview questions revolves around the best/worst Java APIs a candidate has worked with.  Tiago Fernandez has thought to take this question to the next level and posted a poll.  While he had less than 100 respondents, it meshes pretty closely with my experience, so I would say that the results are pretty accurate.  Check out the winner of the worst Java API.  Now, how long will we have to wait for JSR-310?

My Java Developer Interview Questions

Here it is! Many people have asked for this list - it's one I built, borrowed and tweaked over the years. I place a lot of value in "I don't know" over inventing answers as that's what I'd expect a colleague to do.  

It's really important to tailor questions to each candidate.  I'm a strong believer that a smart, experienced developer can pickup new frameworks, APIs, etc, so a lot of these questions are not Java specific.

Know the answers to any of these? :)

Difference Questions


These are really great kinds of questions, candidates can talk approach the answer however they feel comfortable and aren't required to give "definitions" of things.

1. What's the difference between TCP and UDP? (if they know, follow up asking about UDP vs ICMP)
2. What's the difference between Bandwidth and Latency?
3. What's the difference between an Inner Join and an Outer Join?
4. What's the difference between NoClassDefFoundError  and ClassNotFoundException?
5. What's the difference between SAX and DOM?
6. What's the difference between a Thread and Process?
7. Order from slowest to fastest: register read, disk seek, context switch, read from main memory.

Design Questions

1. Let's say you're reviewing (or writing) a simple system that requests an email address, a password and comments (in a textarea) then stores those values in a database. Pretending performance isn't an issue, what kinds of things would you ensure in place before it went live?Sometimes candidates would need some assistance on topics (security, usability, database schemas).
2. Let's say McDonald's has asked you to develop their "N millions served" counter for their main website.  This value comes from a slower system (via HTTP).  Talk a bit about how you would design this.  
3. How do you traverse a cyclic graph?

Java Specific Questions

1. What is autoboxing? Had any problems with it?
2. Every worked with JMS topic/queue, clustered?
3. How are Servlets and JSPs different?
4. What does 'final' mean?  Ever worked with a class marked final? (bonus points if they laugh on the second question)
5. What does 'static' mean?
6. List and describe different access modifiers.
7. How would you diagnose a memory leak?

Random Grab-Bag Questions

A bit of systems, object-orienteering, and such.

1. How do you check error conditions in bash? (If they list shell scripting)
2. What's the difference between lazy and greedy regular expressions?
3. What's a context switch?
4. Ever worked on an "AJAX"y system?  What challenges did you run into? (a little bit dated now I guess!)
5. What's a deadlock?  How do you prevent deadlocks?
6. What is Polymorphism?
7. What are the characteristics of a singleton?

General Opinion/Discussion Questions

1. What makes a good build tool?  What makes a bad build tool?
2. What's your favourite/least favourite API/package? Why?
3. Have you worked on a high-volume system before?  What is "high volume" to you?

What have you been asked before?

97 Things Every Programmer Should Know

Short on the heels of the now-lost 97 Things Every Software Architect Should Know comes a similar list for developers.  It covers a wide variety of ideas from ethics and requirements to code formatting and testing.  You can buy it from O'Reilly or check out the 97 things online.

Four Easy and Practical Tips When Visiting Vancouver for the 2010 Winter Olympics

I've lived in Vancouver for a while now, and realize that you (and 2.3 million of your closest friends) may be coming to visit for the Winter Olympics. Here are some dead easy tidbits that will make your stay more enjoyable. You may want to bookmark this post for while you're here.

Tip #1: Eating

Typical Vancouver food is Japanese food, mainly sushi. There are many good, cheap Japanese/Sushi places all over the city. If you've never had Japanese food before (and you're nervous), check out the all-you-can-eat places that offer appetizer sizes of anything. Leaving Vancouver without at least trying would be pretty close to a crime.

Once you've had your fill of sushi, use Urban Spoon to find other places to eat (Dine Here is good too - but a little less friendly). If you like greasy-spoon breakfasts, try Joe's Grill or Bon's Off Broadway. I like Templeton (don't let its appearance deceive you!) and Kam's Place (tasty Singaporean food) but they're probably going to be packed while you're here.

Kam's Place has had a two-for-one coupon near the back of the Georgia Straight (weekly free Vancouver newspaper) every week for the past five years. Lastly, there are a number of delicious restaurants in the Commercial Drive area.

Your life will be just as fulfilling if you didn't go to Denny's or White Spot.

Don't forget grocery stores for when you just want to pick up some snacks on the go. Vancouver has a few chains including IGA, Safeway, Save On Foods, Urban Fare.

If you're from outside Canada and want to try "Canadian food", don't ask us. Most of us will have no idea. Look around for nanaimo bars, poutine, bannock, tourtiere, lobster rolls, steamed fiddleheads, and maple sugar pie - all genuinely Canadian. (If you've got a sweet tooth, seek out nanaimo bars). Vancouver also has a number of local breweries, and BC has a number of wineries.

With over 120 Starbucks in Vancouver, we're definitely a coffee-culture city. Other chains include Waves and Blenz (both offering free wifi), as well as the more ubiquitous Tim Horton's (origin of the term double double). If you're a Starbucks fanboy, I encourage you to make your way to the corner of Robson and Thurlow - a famous intersection with two Starbuckseseses.

Tip #2: Getting Around

Don't Drive. I really can't stress this enough. The city has recently extended pay-parking hours to 10pm - seven days a week everywhere in the city. Even without the 2.3 million additional people, driving in Vancouver is a pain. I've read somewhere that all tickets to Olympic events include transportation to/from the event. Please check into this!

Public transportation in Vancouver is run by TransLink. It includes buses, subways (Canada Line), light rail (SkyTrain) and ferries (SeaBus). TransLink fares work on a "zone" system. Your fare corresponds to how many zones you'll be travelling through. I found a pretty good TransLink Map. As a rule, SkyTrain and Canada Line stops have fare machines which accept Interac, credit card and Canadian cash. Buses are exact-fare.

Google Maps has a "Directions by Transit" for getting around the city. I encourage you to use this as a backup during the Olympics as some operating changes just for the Olympics have been made and I'm not sure if Google has been updated. Instead, use the pretty decent
TransLink Trip Planner.

If you don't have prepaid tickets (FareCards), there will be a $5 surcharge (per person) traveling from the airport on TransLink (via the Canada Line). It's still cheaper than taking a taxi. If you see someone at the airport selling FareCards, you may want to take them up on the offer. To ensure it's legit, make sure the magnetic stripe side of the FareCard doesn't already have an expiry time printed on it.

If you use the Trip Planner and you see "get off one SkyTrain, get on another SkyTrain" around Broadway-Commercial, I will confess it is a bit confusing the first time you do so, but you won't be alone in the necessary confusion. It's all well-marked.

Looking for something kinda fun to do? There is a SeaBus that goes from Waterfront to Lonsdale, it's a two-zone ride (cheaper on the weekends). It's a fun little ride. The Lonsdale side has a market with fresh food stuffs too. You'll have a nice view of downtown from the Lonsdale side too. Great on a sunny day!

When walking downtown, streets that run NW/SE are just generally referred to as "North" or "South" by downtowners. "If you can see the mountains, you're facing North." is a rule many locals have. I think (at least downtown) locals are pretty happy to help a lost soul.

Lastly, if you can avoid taking public transportation from 7am-10am and from 3pm-6pm, those of us going to work would greatly appreciate it. :-)

If you are driving, and you're from outside BC, you'll want to know about our BC traffic lights.

Tip #3: Weather

Vancouver is generally a rainy city. The outskirts are cold enough to get snow, but usually not a lot. If you want to see what it looks like right now in Vancouver, the popular and mostly reliable Kat Kam web cam is a great start.

I pretty much swear by Environment Canada's Vancouver Seven Day Forecast.

Dress in layers (guys too), but leave your umbrella at home. Instead, bring a long a hooded rain jacket. You'll navigate the sidewalks a lot more effectively and you're not likely to leave your jacket behind somewhere.

Locals tend to need higher amounts of rain to trigger their "I should put my hood up" feeling.

Vancouver is stunning when it's sunny out. If you can, make your way to the Lookout at Cypress Mountain for pretty much the best view of Vancouver day or night.

Tip #4: Blending In

Here are some random tips you may want to keep in mind:

Just like the Tube, Vancouverites follow "Stand to the side to let people out before getting on" for public transportation. If you do this too, you'll blend in a lot better.

Most people here don't speak French; there are easily more Chinese/Japanese/Korean speakers than French speakers in Vancouver.

Generally speaking in the rain, if you're without a hooded jacket or umbrella, walk nearest to buildings. If you've got an umbrella, walk furthermost from buildings.

On stairs and escalators, we follow the "stand on the right, walk on the left" rule.

The weather is a popular conversation topic and can be used to talk to just about anyone - especially if it's nice out!

That's It

Enjoy your trip! And if you're looking for someone to go for sushi with, drop me an email. :-)

Xen vs KVM

I wanted to chime in on the Xen vs. KVM discussions, and give you some food for thought.

I've been using Xen now for years, having replaced UserModeLinux on my personal server, and it's seen a lot of production use at my current job.

That said, KVM is ultimately the right way to go. Constant maintenance of Xen's hypervisor kernel is what causes Linux distributions grief with Xen. Those that keep doing it, do so because it's performant, tested and well-understood.

So you're stuck wondering which virtualization technology you should pick. Here it is: libvirt (with Xen).

Base your toolset, deployment, configuration, and maintenance on libvirt, and use the Xen hypervisor for now.

libvirt provides an easy upgrade path from Xen to KVM once KVM's performance and stability have been battle hardened. Same virtual machine configuration, same provisioning, same disk images. Just grab a new kernel, change which hypervisor you're using and you'll be in the land of tomorrow.

Provisioning Servers: An Afternoon with Cobbler

I spent some time evaluating Fedora's Cobbler installation service. If you play with many machines - real or virtual - you should definitely take a look. Some handy features:

* Kickstart Setup: For Red Hat breed systems, existing Kickstart configurations (found on installation media) are exposed as profiles in Cobbler. You can add your own Kickstart configurations too (imagine a "rhel5-webserver" profile).

* PXE Management: Tied in with a DHCP server, cobbler will offer profiles up for installation for servers during PXE boot (often called Boot from network), since these profiles are tied with Kickstart configurations, servers can be setup or re-setup with just a few keystrokes. KVM seems to support PXE boots; less hoops for installation your VMs.

* Repository Mirroring: Cobbler can take the URL to a repository and mirror it locally, both yum and apt style repositories (rpm and deb) are supported. Use cobbler reposync to keep local mirrors up to date.

* Distribution Swallowing: Cobbler can copy installation media and copy it locally.

* Boot ISO / Live CD: If your server doesn't support PXE, or if tying into DHCP isn't an option, cobbler can make burnable ISOs (and thus USB thumb images) that can be used to perform network installs.

* MAC Address to Profile Management: Cobbler can associate a particular MAC address with a particular profile.

* Automatic Repository Setup: Cobbler can setup your local yum mirror in your newly created server automatically for you.

* Triggers / Puppet Integration: Cobbler can be configured to register the new machine with puppet, or run other triggers (imagine automatic graphs in Cacti, monitoring in OpsView, etc, etc).

* KOAN: Koan is a command line tool that talks with the Cobbler server. It can be used to set up new VMs (using libvirtd), or with a special --replace-self option, cause an existing installation to reinstall itself. (Only works with Red Hat breed systems.)

* Zeroconf: Avahi is among one of the coolest technologies I've played with recently. Specifying --server=DISCOVER on the koan command line will find the cobbler server on the network. I love stuff like that.

* Configuration Cloning: Move your test environment configuration into production (or mirror for HA, etc)

I really like the architecture of Cobbler too; an XMLRPC server that both the command line tools and web interface use. The web interface is handled by Apache and mod_python, so authentication and authorization are handled by Apache (ie, basically anything you can think of).

Lastly, one nice touch is cobbler check. Run right after a fresh install it checks your environment and lets you know of anything that could potentially hamper Cobbler from working.

I haven't yet tried Cobbler with an existing DHCP server, and I'm not sure it supports ISC DHCP's failover capabilities (even with OMAPI).

Ye Olde Job: Technology That Wasn't

Our system could accept transactions over the phone (via a clerk) or through our website. The clerk on the phone used a semi-web based application to interact with the system.

The headache with this was that the code base for the "semi-web" and "actual web" only became a common code path much, much deeper in the code. So the plan was simple, redo the website code such that it could eventually be leveraged to replace the semi-web code.

We prototyped, mocked up, tested and developed a new part of the application on our website using GWT. It went through load testing, SEO-approval and I believe we even did a couple in-company demos of the technology.

The code was "done" (in the "given to QA to break" sense of the word) when I left, but when I check the website now I see it never made it to production.

I cared about this for the first month or so after I left, so I pinged around to find out what happened. The short version is that it got stuck in developer mythology, and with nobody there to lead it out, I knew it would never see the light of day.

I wonder just how much technology is "done" that has never seen the real world?

Ye Olde Job: First Post - So You're the Boss

It's been seven months since I left my old job, and I feel it's alright to talk about some things.

If you're in a situation where you're finding yourself leading a team of developers without any mentoring from anyone on how the heck to do that, I offer you some food for thought.

As a developer, you measure your own progress by the code you write, the bugs you fix, stored procedures you debug, problems you solve and features that get released. You have days where you go home "having done nothing" because you have a dozen variables in your head and you've been nose deep in code, but whatever it is still isn't working.

Or maybe you wouldn't debug the stored procedures, but instead really wish that you could replace them with something else; a nice cup of Java, easy to unit test and a breeze to step through.

One thing I figured out that really helped me: When leading a team, put the team first.

What the heck does that mean?

It means getting to know the strengths and weaknesses of not only your team, but of the teams you interact with. One developer has a knack at networking, another at databases, another one refactors code from a city dump into a gorgeous mansion. When the work is important - major features on a tight timeline - use peoples strength. When it's less important - minor bugs, features that are 'nice to have' - let them work on their weaknesses. If other teams have weaknesses in areas, work to help them, or fortify against it.

It means being two steps ahead of your team, knowing where you're going next. If nobody else in your company can give you direction two steps ahead, then take matters into your own hands. Don't just start piecemealing things; always have a vision. If you don't know where you're going, you won't know when you've gone off course. Obviously, be flexible with your vision for that idle Wednesday when some manager presents his vision to you so you can embrace it and walk together.

It means giving that improvement away. This was the hardest for me: giving up writing that feature you really wanted or fixing that bug that drives you nuts. You will write less code, but your team will write more code than you ever could. You will want to use a bit of the downtime you'll get to assign yourself some work, but you know you'll never get to it. Use that downtime to better map out your plan, to make sure everyone has got the work they need and to fish around to see if maybe that company plan or vision is hiding somewhere... ..anywhere.. ..maybe?

It means accepting shades of grey. There are many wrong ways to do things and few right ways. But there is definitely more than one right way. The way someone on your team might craft something may not be bytecode-for-bytecode how you would do it, but it may still fall in the realm of right. Accept that it's not how you would do it. That's fine. But is it wrong?

It means keeping it in the team. Just like most code bases that earn a over a billion dollars, the code base I worked on was suboptimal. We all knew it, but we knew we were there not to whine about it, but to make it better. Bit by bit, day by day. Start with the biggest problems, work toward your vision.

It means that stored procedure is going to stay; both the DBA and I know how to tune, monitor and troubleshoot it. We've minimized the row locking in it and fixed a deadlock problem in the logic. Also, next week you'll be fixing it to be resilient against another team's code, because their code does table scans and table locks and it will soon be our busy season and the potential for headaches is not part of my vision.

Updated: While stored procedures vs ORM is part of a future post, I hope that people who are in the "You've been here the longest, you're in charge" situation can benefit a bit from keeping in mind that they need to put their team before themselves. I know it's hard to switch gears.

Seven Reasons I Didn't Pick Your Open Source Project

After a really relaxing summer, I've picked up a job as a Systems Architect at local company suffering from growing pains. After almost a decade, I've put down my debugger and picked up a budget..for now. I've found myself looking at many different options of products some open source some not. I want to apologize to those open source projects I've rejected and wanted to let you know why:

1. Capacity Planning: You lack any realistic guidelines for server sizing. I have no idea how much memory, disk, bandwidth or cpu cycles I need to support 250 users checking their email every morning and you have no tools to simulate load.

2. You're Working On the "Next Gen" Version: You've stopped working on the version people are using in order to rewrite the application using technology that follows your new beliefs. The current version is now suffering from bitrot and your new version is undergoing its third major refactoring.

3. Your Community: Your forums are an abandoned town with tumbleweed rolling through waking up the sleeping armadillos. Questions go unanswered for months at a time, most of the posts are spam or people asking if the project is dead (which it is).

4. Installation: Your twelve-step installation process involving a different version of Perl, MySQL, and PHP plugins I don't have is a headache. This is the world of LiveCDs, embedded servlet containers, virtual machines, Web Start, VNC, hosted-demos or at least RPM/DEB packages.

5. Your Features: They're geared towards other developers and not users. I realize you're "not M$", and it's swell you think "Not Written in Java, so it's Fast!" is important, and it's sure great that you "use AOP extensively", but if those are the features you list instead of "works through NAT","supports 802.3ad bonding" or "integrates with iCal" then you're pretty far removed from your user base.

6. Maintenance Headaches: According to your documentation, your upgrade process is "Backup all data. Install new version. Restore all data." That's fine if "all data" is the dhcpd leases file, but not so great when it's 9TB of files that users access all day.

7. Price: Your project has a "commercially supported" version. That's cool, but when the rates listed are more expensive than the defacto standard you can pretty much guarantee I'm not going to be able to sell it upwards.

I'll let you know what I've picked and how it turns out in the next few months. I'm also looking for a really good Linux sysadmin. :)