[Edit: Since writing this article up back in early March, I've moved on from this job. The folks who are now maintaining it at least know where the pain points are, can run migrations safely, deploy it locally, and to dev servers, and to the main deployment area.  It's a working app, although I never got code coverage above about 45% at least the coverage was decent in the core app areas by the time I left.]

Mike Gunderloy had an interesting article entitled ‘Batting Clean-up‘, which was very timely for me.  I’ve just started maintaining and trying to improve a Rails app developed by an ‘outsourced’ group. The only tests were the ones generated automatically by ‘restful authentication’, and they were never maintained, so they didn’t come close to passing. Swaths of the program are written in terribly complex (and sometimes computed) SQL, migrations didn’t bring up a fresh database (poor use of acts_as_enumerated causes great hurt), and vendor/plugins should have just had one named ‘kitchen_sink’.

It hurts to see Rails abused like that; you want to take the poor application under your arm and say, ‘It’ll be okay…we’ll add some tests and get you right as rain in no time!’, but you know you’d be lying…

I did much of what Mike described (half the gems it used were config.gem’ed, the other half weren’t), vendor’ed rails (it breaks on newer than 2.1.0), and brought the development database kicking and screaming into life. There was no schema.rb, it had been .gitignore’d, and the migrations added data, used models, and everything else you can imagine doing wrong. (Including using a field on a model after adding that column in the previous line…I don’t know what version of Rails that ever worked on…) I didn’t want a production database; who knows what’s been done to that by hand. I want to know what the database is _supposed_ to look like; I can figure out the difference with production later.

Once the clean (only data inserted by migrations) dev database was up, I brought the site up to see if it worked. Surprisingly enough, it did; apparently they used manual QA as their only testing methodology. I appreciate their QA a lot; it means it’s a working application, even if it’s not going to help me refactor it.

I ran flog and flay and looked at the pain points they found to get an idea how bad things might be. I picked an innocuous join table (with some extra data and functionality) to build the first set of tests for, which gave me insight into both sides of the join without having to REALLY dig into the ball of fur on either side. I viciously stripped all the ‘test_truth’ tests. I looked for large files that flog and flay hadn’t picked up to pore over. Check out custom rake tasks, because those often are clear stories and easy to quickly understand in a small context.

Checking out the deployment process tells you a lot also, although it turns out this was stock engine yard capistrano.

Skimming views (sort by size!) will tell you a lot also, especially when you find SQL queries being run in them…

Use the site for a little while, and watch the log in another window. Just let it skim by; if you’ve looked at log files much, things that seem wrong will jump out even if it’s going faster than you can really read.

In my case, the code’s mine now, so it’s my responsibility to make it better before anybody else has to touch it. I’ve got about a week of ‘free fix-it-up time’ before I need to start actually implementing new features and (thankfully) stripping out old ones… At my previous company, I was the guy pushing folks to test, now I’ve inherited a codebase with zero tests. Poetic justice, I suppose…

Good luck!

— Morgan Schweers, CyberFOX!

As with everything else, it’s not that simple…

In the successful startups I’ve worked at, a core of people staying late, working long hours, was a symptom of having an idea that people can believe in.

I have not seen any very successful startups where the developers weren’t at least a little monomaniacal about their work.

On the contrary, I’ve been at two successful startups (defined here as wildly successful IPO’s) where having those fanatic developers was a core reason of why they were successful.

The people who were putting in overwhelming hours at those companies weren’t doing it because they’re workaholics. They were doing it because they were true believers. Both in the company itself and the product they were building.

It’s not about the workaholics making the company successful, it’s about the company being one that the employees can believe in, to the point of _wanting_ to be there, wanting to be making it better.

In those cases, you don’t fire the people who are passionate about building your company. You support them, and accept that they’re going to crash occasionally, and try to nerf the crash some…

In my experience, it’s the fervent employees who are the core of successful startups. This was true at McAfee Associates (went public in 1992), and PayPal (went public in 2002), both successful startups that I was part of.

You also need people who aren’t as fervent, who can see a wider view, so it’s always a balance. So you can’t really ‘fire’ either of them, out of hand.

I’ve been that true believer, focusing everything into a job or project that I deeply care about. I’m a much calmer, more balanced person now, though. We’ll see what happens in 2012…

– Morgan

I want to start by making it clear that I know why testing is good, and that it’s really important, but I think that the TDD proponents are glossing over the most difficult part of a project.

I would very much like someone to address the issue of modifying code that is not new, and not already perfectly tested (or even completely specified!).  That is to say, the vast majority of actual code out there.

TDD is intensely focused on the early development phase (or at least TDD proponents are), and on writing new code, as opposed to what the majority of software developers actually do; maintain and update existing code.

It’s really straightforward (and fun!) to write entirely new code in the TDD fashion.  I’ve done it for about 3 decent sized projects now (one Java and two Rails), and it can be really pleasant, and a great focusing tool.  No arguments there; when you do it from the start, it’s really wonderful.

On the other hand, when you’re making incremental changes here and there throughout a very large, pre-existing, only partially tested codebase, it’s vastly less pleasant to try and do it test-first.

The Ruby Autotest tool is not such a pleasant tool at that point.  You stop wanting to write failing tests, because fixing it means autotest is going to try to do a full retest, which sucks for developer flow…  Even if your tests take ‘only’ 5 minutes to run, breaking a test makes you wince, and writing a failing test and then fixing it is for masochists only (and ones who want to miss the project milestones at that).

The focus of every presentation (this one included) I’ve seen on TDD being on the start of a project makes me wonder why nobody’s talking about later in projects…

Nearly every developer out there is going to face a large codebase with poor testing coverage, and will have to make changes that aren’t entirely new code, to existing code that isn’t entirely tested.  Does TDD have a solution for the ‘large, crufty codebase’, or is it suited only for 1.0 versions, small projects, and projects that were TDD from the start?

This isn’t really a rhetorical question for me.  I really want to get my organization’s culture more oriented towards testing.  I’ve got buy-in from lots of people that when they’re writing new modules and services, they’ll do it test-first (or at least ‘test around the same time as the code’, which is all I can ask for at this point), and that’s great.  But has anybody developed any tools to make TDD better suited to maintenance and improving existing code?

–  Morgan

p.s.  I’m skipping BDD entirely, because BDD is so hardcore in the ‘only for already very well specified solutions’ camp, that it’s meaningless for this question.  I’m also using ‘TDD’ and ‘test-first’ interchangeably, and I probably shouldn’t be.

p.p.s.  The title refers to how in Logans Run, everybody was destroyed at 30, so there weren’t any old people. In the world of TDD (or at least TDD presentations), there are no old projects, every one is fresh and new, so the issues that come with an old code base are never addressed.

This is a hard thing to explain if you haven’t been there. I’ve been there twice, once with McAfee Associates, in full-bore, turbo-charged engineer mode, fighting against the world-wide virus writing epidemic in the very early days of McAfee Associates (from 7 people to more than 120 people). Fewer people would recognize that world as would recognize the world of a modern web site developer, so I’ll focus on the second, PayPal.

I remember working at PayPal during their heyday just before and just after going public, when we were fighting the ‘good fight’ against eBay, and I’d work all hours of the day and night not because I had to, but because it was deeply, personally important to me that we win, and because the work was so deeply enthralling that I lost track of time entirely. That everything be right, and that we be first to market with features, that our code be spectacular, that we be innovative and brilliant and FAST was our world. And until we were bought by eBay finally (demonstrating, imo, that we had the better service), no matter the hour, I never was alone at the office.

When you’re part of a brilliant team, trying to honest to god change the world, it’s not about deadlines. It’s about a form of love. It can be thoroughly, caustically destructive to everything else in your life, but it’s an experience I would be a lesser person if I had missed.

If you’re clocking hours at work, and the passion of what you’re doing isn’t keeping you rooted to your chair at all hours, loving the pure joy of creating, fighting the good fight, and trying to change the world, it doesn’t mean it’s not a good job. It’s just not THAT experience.

I promise, there’s far more call for people who work regular hours, meeting normal deadlines, doing solid, good work, than for those of us who burn so very, very brightly, but for so short a time.

If you’re really in passionate love with the work you’re doing, it’s not about working to 3am to meet a deadline. It’s about finally reaching a temporary point of closure for the days work, and raising up your head to suddenly discover it’s 3am.

And if you really, truly believe in your company, and you believe in your project, and have a fire to ‘win’ in some way (usually against a more powerful competitor) it’s not about accepting an imposed deadline that makes you work hard. It’s about DEMANDING a deadline that makes you work hard, but that you know you can meet. Because you know it’s important, and that every second counts, and you CARE about the company being not just first, but first with a brilliant, innovative, wonderful experience.

After it’s all over, it’s draining. It’s exhausting. It’s mind-numbing. You feel…dead, somehow, once the work is over, and you’ve been brilliant for so long, that you feel like your brain cells have used up all their energy. You go home, don’t show up to work for a week, recharge, find out if you still have any RL friends, do something physical (skydiving, rock climbing, hiking, etc.) to get in touch with your body again. You come back to work eventually, and you work with others to clean up any loose ends, and slowly you get back the energy from your co-workers, and the ambience in the office, and eventually you’re back on track to start another feature that’ll knock the socks off your competitors.

If you’re in a company where you are the dominant force, you don’t work like that. You don’t need to, the hunger isn’t there.

PayPal lost that hunger when eBay bought us. There wasn’t anything to fight for, anymore. We’d won, in a way, and lost in a way. I even asked it, when eBay management had a big meeting with everybody to tell us about their vision for us. I don’t remember if it was the meeting Meg Whitman was at, or not, but I asked something like, “We’ve been fighting eBay for all this time, and now we don’t have to. What will replace that, to keep the drive going?” The answer was a mealy-mouthed mess of future strategy and becoming the dominant payment platform. It wasn’t a battle anymore, we’d become the big company.

I left not terribly long after that, for health reasons. (Remember what I said about caustically destructive? ) But also because I didn’t feel the passion in the hallways anymore.

The woman who I will marry in 36 days stood by me through it despite almost never seeing me, my friends teased that they’d forgotten my name, I ended up needing major surgery for a condition I let go too long… But I was part of one of those winning teams, fighting against terrible odds, doing brilliant work, burning so very, very brightly, and changing the world one line of code at a time.

I think Mark understands that, as Google has Microsoft with an unlimited war-chest bearing down on them. From what I’ve read, I don’t think Scoble completely does get it. He gets that passion is important (Channel 9 certainly shows that), but the fight against overwhelming odds that drives it to fevered peaks, that brings it to a different level…that’s what’s missing.

But it’s okay. Even I’m working a day job these days. I still have the intense passion to program on my own projects, doing it until 4am regularly, but I’m in a larger cycle of recharge, get in touch with life, etc., before maybe doing it again if I find the right company. Or maybe not. I’ll be a married man shortly, settling down in theory. Maybe I can’t fight those fights anymore. Maybe I should work at Microsoft. Just kidding…

– Morgan Schweers, CyberFOX!

Given two computers linked over a slow link (say dialup), each has a 1TB file, how would you determine (1) if the two are different, and (2) what that difference is?

Me being the gadfly that I am, I spoke with my office-mate a little about what the ‘right’ answer should have. He acknowledged that he was looking for a subdivision answer, which determined the files first difference location. It’s a pretty simple answer, conceptually subdivide the file into (say) 4096 separate 256MB blocks. Do the MD5′s on them, pass them over the wire, so each side knows the MD5′s at each 256MB step. If there’s a change, instantly at that point you know it was in the last 256MB’s, and you can subdivide that down by 4096, so you’re MD5′ing 64K blocks. This finds you the 64K block where the first change happened, and you can either subdivide one more time, or just transmit the 64K containing the difference and figure it out right there.

I pointed out that this only finds the first byte where there is a difference, and that if you wanted to know if some portion of the *rest* of the file is the same (or worse, you want to construct a minimum edit difference), you start to need to do more complex things. I went off, and started thinking about algorithms to make the rest of the problem tractable.

This is what I came up with, recorded here for posterity, and in case anybody with a decent sense of algorithmics would like to correct me.

It’s CPU-intensive, firstly. It’s CPU-intensive, because that’s what you’ve got. You can’t use bandwidth, because it’s defined as too low for this, so you have to use what you’ve got, and that’s the CPU power on each side.

So one machine designates itself as the ‘compare to’ server. It then chunks the *remainder* (point from diff on) of the 1TB file into fixed size chunks. (Say 64K chunks, which in the worst case would be 16 million of them.) Each 64K chunk is separately MD5′ed, and the results stored on disk, hashed by their bottom two bytes into a linked-list, along with the block number they’re for. The bottom two bytes are used as an index into a 64KBit bit-vector, and the bit at that location is set. Once the remainder of the file is completely processed, the bit vector is transmitted.

The ‘compare from’ client then creates (this is the computationally sucky part) 64K-1 simultaneous MD5′s, chunking from (diff+1) to (diff+65535). When it’s done chunking 64K of data (all 64K MD5s will complete at the same time), they’re checked, reset, and they all start again (at diff+65536 to…). This is defined as 64K-1 MD5s under the hope that some kind of internal MD5 optimization could be found that would prevent it from behaving as slowly as a single MD5 run on the order of (64K*(EOF-DiffPoint) times.

During the ‘check’, the bottom two bytes are used as a lookup in the bit vector. If any MD5s match a bit, all matching ones are transmitted, and the server checks if any are a recognized block. If any is, it returns the block number, and the next 64K MD5, which the client can quickly check by running a single MD5 on that data. If they match, the client and server can consider the rest of the file as if it were to be checked from the first step again, because there may be more than one point of difference. More precision can be gained by transmitting the difference data from the client. Since we know there is a difference at the first byte (by the first algorithm), we do a backwards compare from the end of the last different block on the server versus the block before we found an MD5 match. We then find the last byte of difference, as well as the first. (Because of the size of the data, you could transmit it to the server, and run a more normal minimum edit difference algorithm against that block, to get more precision.)

As I said before, treat the rest of the file (from server block ‘N’ and client block DIFF+n bytes to the block ‘N’ match) as a new file to compare. Scale down the blocksizes in this new instance, for quicker/greater precision, and do it again.

This should result in a decent minimum edit difference ‘collection’ on the server (or whichever side aggregates this data), without transmitting the entire 1TB of data over the wire. These numbers can be tuned, e.g. the two-byte bit vector can become 3 bytes, and still only takes 2MB up front to transmit the bit-vector, but substantially cuts down the likelihood of having sub-hash matches, and thus fewer packets sent later.

Now, a real problem comes if 1 byte in every 64K is modified on the client. The problem is that it’ll never match another block. The CPU load becomes pretty excessive at this point, effectively doing 64K MD5s of a 1TB(-64K) file.

At every block size, this problem remains, unfortunately, and with a file as large as 1TB, you really want to start with large block sizes. If, for instance, every 64th byte is altered, you’re pretty much doomed as that’s the blocksize for MD5 and you’ll never get a full match, and at that point in order to get a full edit difference between the two files, you will need to transmit one of them to the other.

So this approach is basically restricted to when you have an idea how many changes (or expect that the answer should be zero) have been made.

The original interview question, however, is even more restricted in that it solely finds the first location of a change, and does not identify what kind of change it is, which is in my book a not terribly useful answer.

– Morgan Schweers, CyberFOX!

So today, when I found the toilet not working quite right, I tried to call the operator to find out who I should call, and maybe get transferred to them.

They told me to ‘File a Remedy ticket.’ Now Remedy is an absurdly complex application which tracks trouble tickets, and is generally used to track IT issues. They are, indeed, also using it to track plumbing issues. I cannot imagine who decided to swat this mosquito with this nuclear weapon, but the scariest part to me is that I now am absolutely confident that even the plumbers have SLA’s (Service Level Agreement) they have to meet.

Another sign of the ossification of a company: The focus on ‘process’ as the means of getting things done.

I take this in stride, however, because this was a Very Good Weekend for me. I’ll get to that in a bit.

– Morgan Schweers, CyberFOX!