What alternative do you suggest for using models in migrations? I was in several situations where I had to not only change the underlying db structure but change the contained data, too.

Data changes, especially moving data around, are almost always rake task-worthy in my experience.

The other side of that, populating large amounts of seed data into new databases, is a difficult task no matter the method; seed_fu attempted to deal with it, but it’s not an optimal solution and pretty old. I’m not even sure if it works anymore. It’s worse if you need the seed data to be from a legacy database in tests (e.g. a nutritional database). Reloading lots of data each time a clone_structure_to_test is done makes your tests very slow.

I break down migrations into three kinds; structural (tables, columns, indices, etc.), data (pre-populating tables, etc.) and procedural (moving data around, recalculating counts, etc.). The first is what I strive to limit migrations to. I feel like there should be a good answer for the second and Rails 3.0 has a ‘Simplest Thing That Can Work’ feature in Rails commit #4932f7b. The third, I try to relegate to rake tasks that are usually run once, on deployment of the branch.

The procedural tasks don’t need to be run when building a fresh database, because there isn’t legacy data to correct. That’s why you can usually define the model in the migration to force it to work even if the real model is gone or renamed; there’s no data, so the operations often don’t matter. If they don’t NEED to be run when building a fresh database, I try not to put them in the migrations.

It’s not ‘hard and fast’, because I usually work in startups to small companies, where dogma doesn’t work so well. Imagine, though, a large and thin piece of foam. It’s flexible, and you can make it into all sorts of shapes, and yet it’s simple. Each time you add code that makes reasonable changes in the future painful, it’s like putting a thin glass rod into the foam. It’s still flexible, but there’s some bends you can’t do without breaking. Add too many and you’ve got an inflexible and brittle object, no matter how dynamic the base material is.

The fear of breaking things by changing the code is deeply demotivating for everyone.

I know I waterboarded that analogy, but hopefully it makes sense…

— Morgan Schweers, CyberFOX!

[Edit: Since writing this article up back in early March, I've moved on from this job. The folks who are now maintaining it at least know where the pain points are, can run migrations safely, deploy it locally, and to dev servers, and to the main deployment area.  It's a working app, although I never got code coverage above about 45% at least the coverage was decent in the core app areas by the time I left.]

Mike Gunderloy had an interesting article entitled ‘Batting Clean-up‘, which was very timely for me.  I’ve just started maintaining and trying to improve a Rails app developed by an ‘outsourced’ group. The only tests were the ones generated automatically by ‘restful authentication’, and they were never maintained, so they didn’t come close to passing. Swaths of the program are written in terribly complex (and sometimes computed) SQL, migrations didn’t bring up a fresh database (poor use of acts_as_enumerated causes great hurt), and vendor/plugins should have just had one named ‘kitchen_sink’.

It hurts to see Rails abused like that; you want to take the poor application under your arm and say, ‘It’ll be okay…we’ll add some tests and get you right as rain in no time!’, but you know you’d be lying…

I did much of what Mike described (half the gems it used were config.gem’ed, the other half weren’t), vendor’ed rails (it breaks on newer than 2.1.0), and brought the development database kicking and screaming into life. There was no schema.rb, it had been .gitignore’d, and the migrations added data, used models, and everything else you can imagine doing wrong. (Including using a field on a model after adding that column in the previous line…I don’t know what version of Rails that ever worked on…) I didn’t want a production database; who knows what’s been done to that by hand. I want to know what the database is _supposed_ to look like; I can figure out the difference with production later.

Once the clean (only data inserted by migrations) dev database was up, I brought the site up to see if it worked. Surprisingly enough, it did; apparently they used manual QA as their only testing methodology. I appreciate their QA a lot; it means it’s a working application, even if it’s not going to help me refactor it.

I ran flog and flay and looked at the pain points they found to get an idea how bad things might be. I picked an innocuous join table (with some extra data and functionality) to build the first set of tests for, which gave me insight into both sides of the join without having to REALLY dig into the ball of fur on either side. I viciously stripped all the ‘test_truth’ tests. I looked for large files that flog and flay hadn’t picked up to pore over. Check out custom rake tasks, because those often are clear stories and easy to quickly understand in a small context.

Checking out the deployment process tells you a lot also, although it turns out this was stock engine yard capistrano.

Skimming views (sort by size!) will tell you a lot also, especially when you find SQL queries being run in them…

Use the site for a little while, and watch the log in another window. Just let it skim by; if you’ve looked at log files much, things that seem wrong will jump out even if it’s going faster than you can really read.

In my case, the code’s mine now, so it’s my responsibility to make it better before anybody else has to touch it. I’ve got about a week of ‘free fix-it-up time’ before I need to start actually implementing new features and (thankfully) stripping out old ones… At my previous company, I was the guy pushing folks to test, now I’ve inherited a codebase with zero tests. Poetic justice, I suppose…

Good luck!

— Morgan Schweers, CyberFOX!

As with everything else, it’s not that simple…

In the successful startups I’ve worked at, a core of people staying late, working long hours, was a symptom of having an idea that people can believe in.

I have not seen any very successful startups where the developers weren’t at least a little monomaniacal about their work.

On the contrary, I’ve been at two successful startups (defined here as wildly successful IPO’s) where having those fanatic developers was a core reason of why they were successful.

The people who were putting in overwhelming hours at those companies weren’t doing it because they’re workaholics. They were doing it because they were true believers. Both in the company itself and the product they were building.

It’s not about the workaholics making the company successful, it’s about the company being one that the employees can believe in, to the point of _wanting_ to be there, wanting to be making it better.

In those cases, you don’t fire the people who are passionate about building your company. You support them, and accept that they’re going to crash occasionally, and try to nerf the crash some…

In my experience, it’s the fervent employees who are the core of successful startups. This was true at McAfee Associates (went public in 1992), and PayPal (went public in 2002), both successful startups that I was part of.

You also need people who aren’t as fervent, who can see a wider view, so it’s always a balance. So you can’t really ‘fire’ either of them, out of hand.

I’ve been that true believer, focusing everything into a job or project that I deeply care about. I’m a much calmer, more balanced person now, though. We’ll see what happens in 2012…

– Morgan

I want to start by making it clear that I know why testing is good, and that it’s really important, but I think that the TDD proponents are glossing over the most difficult part of a project.

I would very much like someone to address the issue of modifying code that is not new, and not already perfectly tested (or even completely specified!).  That is to say, the vast majority of actual code out there.

TDD is intensely focused on the early development phase (or at least TDD proponents are), and on writing new code, as opposed to what the majority of software developers actually do; maintain and update existing code.

It’s really straightforward (and fun!) to write entirely new code in the TDD fashion.  I’ve done it for about 3 decent sized projects now (one Java and two Rails), and it can be really pleasant, and a great focusing tool.  No arguments there; when you do it from the start, it’s really wonderful.

On the other hand, when you’re making incremental changes here and there throughout a very large, pre-existing, only partially tested codebase, it’s vastly less pleasant to try and do it test-first.

The Ruby Autotest tool is not such a pleasant tool at that point.  You stop wanting to write failing tests, because fixing it means autotest is going to try to do a full retest, which sucks for developer flow…  Even if your tests take ‘only’ 5 minutes to run, breaking a test makes you wince, and writing a failing test and then fixing it is for masochists only (and ones who want to miss the project milestones at that).

The focus of every presentation (this one included) I’ve seen on TDD being on the start of a project makes me wonder why nobody’s talking about later in projects…

Nearly every developer out there is going to face a large codebase with poor testing coverage, and will have to make changes that aren’t entirely new code, to existing code that isn’t entirely tested.  Does TDD have a solution for the ‘large, crufty codebase’, or is it suited only for 1.0 versions, small projects, and projects that were TDD from the start?

This isn’t really a rhetorical question for me.  I really want to get my organization’s culture more oriented towards testing.  I’ve got buy-in from lots of people that when they’re writing new modules and services, they’ll do it test-first (or at least ‘test around the same time as the code’, which is all I can ask for at this point), and that’s great.  But has anybody developed any tools to make TDD better suited to maintenance and improving existing code?

–  Morgan

p.s.  I’m skipping BDD entirely, because BDD is so hardcore in the ‘only for already very well specified solutions’ camp, that it’s meaningless for this question.  I’m also using ‘TDD’ and ‘test-first’ interchangeably, and I probably shouldn’t be.

p.p.s.  The title refers to how in Logans Run, everybody was destroyed at 30, so there weren’t any old people. In the world of TDD (or at least TDD presentations), there are no old projects, every one is fresh and new, so the issues that come with an old code base are never addressed.

I’ve put up the latest version, 1.0.1 of JBidwatcher.  It’s mainly a bug fix release.  It includes a few new features towards better documentation, error messages, and recognition of eBay states, and a new (still completely optional) approach to the eBay affiliate idea.

One of the important fixes has to do with a wording change; eBay changed ‘You have been outbid’ to ‘You’ve been outbid’, and JBidwatcher didn’t handle the wording change, reporting it as an ‘unknown error’.  Also eBay changed the text on their captcha pages, so JBidwatcher was failing logins for unknown reasons.  Now it should be better at recognizing those situations.

There was a sporadic startup failure when sorting by a currency-based column with unknown values in it, and saving and restarting.  I hope to have addressed that as well.

Very little UI has changed, although the internal FAQ has been enhanced a little, and a whole new FAQ has been put on the web site.  There is also a new community forum area that’s much better than the old SourceForge forums.  In fact, SourceForge is deprecated for everything except bug reports, feature requests, and CVS access to the source.

More information is available at the JBidwatcher 1.0.1 changelog.

• Windows users can download a Windows executable!  (No installation required, it just runs!)
• Mac OS X users can download an app.tar.gz.
  Download (it’ll automatically decompress) and double-click the .tar file to extract the Application!
• You can also download the Java binary for any other platform, including Linux and Solaris.
  Launch with: java -Xmx512m -jar JBidWatcher-1.0.1.jar
• As always, you can download the source and build it for yourself.

Best of luck with your auctions!

–  Morgan Schweers, CyberFOX!

Programming should be fun, it’s what gets us programmers into it in the first place, and it’s what keeps us going at it.

Too many layers have been heaped on programming these days; most IDEs are oppressive, process-oriented beasts.

I, and many other programmers, have been concerned about how the next generation of programmers are going to be introduced to the joy of programming when all the tools out there seem to suck the fun out of it.

Enter why the lucky stiff, and Hackety Hack!. Taking one back to the days when programming was as easy as typing something in, and hitting enter.

Hackety Hack! is about learning to program, and being able to do really cool things in a very few short lines.

I’ve been a fan of _why for a while, with a really wild approach to teaching Ruby, and a lot of really interesting code, and this is even cooler.

If you’re new to programming, or have kids, or just want to play with one of the simplest ways to write programs, get Hackety Hack!

– Morgan Schweers, CyberFOX!

A concerned user recently asked me how I was doing in the aftermath of the issue with eBay sales of JBidwatcher, specifically:

You seemed pretty depressed about it in your post to the website.

I was.

There was a really bad week there, while I was dealing with all of it, back and forth, and just feeling like crap. I got a lot of user feedback, from a LOT of people, that reminded me, as the concerned user put it, not to let the few jerks make me give up.

I took some time to work on other projects, and I’ve been fiddling with the next major rev of JBidwatcher, mostly cleaning up the code, improving the source layout, fixing small things that nobody else will likely ever see, and writing silly features just for the fun of it. (Like making the internal webserver take ‘events’ to be posted to the various subsystems, so you can add an item, do a bid, or even tell it to fire off a sound effect through a REST-ish interface. You could theoretically ’script’ JBidwatcher through that.)

I’ve also mostly moved the code base of JBidwatcher to Java 1.5 (mmmm, tasty generics!), since 1.6 is now out. I’ve also been experimenting with including ‘Derby’, an embeddable (in the ’ship with program’ sense) tiny SQL-based database, so that JBidwatcher’s memory usage doesn’t grow at the same rate as the number of auctions. Also so that it can offload completed auctions, so they’re not kept in memory anymore at all. Yet another thing I’ve been playing with is including a scripting language (something simple) which would get run on certain events, which would allow for making some of the complex rules people have requested as features. I’ve also written up an FAQ I need to publish on the site. (The first question addresses my inability to answer emails consistently, in fact!)

Anyway, all told I (and work!) have been keeping myself busy, albeit quiet. It all helps me get past the issue with the people selling JBidwatcher. Future versions will probably not be open source, however. I may expose the source, or open certain sections, but almost all open source licenses explicitly allow what those folks were doing, and I’ve determined that it’s beyond what I’m comfortable with. One of the things people repeatedly said in private emails was that the open source nature of JBidwatcher was not critical to their appreciation of JBidwatcher. This means I’ll need to extricate myself from Sourceforge in various ways, and cover my own purchase of IntelliJ IDEA, but I think donations will have covered that.

I don’t want to charge for JBidwatcher; I prefer people using it and deciding for themselves what it’s worth to them. Plus, because it’s scraping eBay, I feel bad about asking for money for something that could break the next day. So I expect the program will continue to be no cost. I’m thrilled to get donations, of course, but I don’t build JBidwatcher to make money; I have a day job for that.

At the same time I’m working on building other projects so my morale won’t get torpedoed so badly when someone messes with the sole project I’ve been working on.

One of the other projects I’m working on are a health tracking tool (weight, blood pressure, hours slept, water drank, steps taken, foods eaten with nutrition information, and more stuff like that, with pretty graphs and sparklines (my weight trend: )). The other major one is a comprehensive multi-user outliner tool. Both are entirely web based applications, unlike JBidwatcher, and both are in Ruby on Rails.

I’m sorry that I haven’t been dedicating more time to JBidwatcher, but it’s been fundamentally working okay recently, and I needed to blow off steam by doing cool new stuff. So I’ve been letting it percolate, and rekindling my coding passion by working on other interesting problems.

That’s the status as of now; I hope that this sheds some light on my thought processes, and what I see in the future for JBidwatcher. One important thing to take away is that YES, there is a future for JBidwatcher.

Thank you, every one, who wrote me, donated, or just thought well of me during all this. I appreciate it a great deal more than I can express.

– Morgan Schweers, CyberFOX!

I had a friend recently who asked about what programming languages they should learn. He primarily works as a system and network administrator, and had been bombarded by ‘Learn Ruby!’ from a bunch of evangelists recently. I assured him it wasn’t necessary, and came up with this interesting list.

What language you work in really depends on what you need to do. In my definitely not so humble opinion, here are the best of breed languages/platforms for each of a set of needs, if you just want to Get Stuff Done.

Applications - Java. It’s cross platform with rich libraries, and 1.6 makes it even better.

Glue – Perl and bash are best for gluing things together. On Windows this category doesn’t really exist.

Web - Ruby on Rails is the best tool out there right now. You also need Javascript, CSS and HTML.

Speed – C++ is best when it must be fast, but usually in service to another language, except for…

Video Games – C++, because speed often trumps other considerations.

Database – If you touch a database, you need to know SQL.

Applet – Flash/ActionScript is the best applet language right now.

Embedded – I believe C is still the gold standard, but I haven’t done embedded development in a while.

Mobile – Java ME appears to be the only platform-agnostic answer currently, but again, not my field.

The above is based on my knowledge, as of right now. I can make a strong argument for each of them, but that would require being negative in various ways about other languages for comparison. Instead I prefer to note that other languages or platforms may still be good for each need, but these are the best.

Just like Ruby on Rails exploded onto the scene, this list may not hold true in a week, so it’s more a ’snapshot’ than a ‘bible’. It’s also based on my knowledge of what makes for good software development, balancing (depending on the need) features that make development less error-prone, speed, cross platform capabilities, maturity, match between the development (and mental) model and the need, available tools, etc…

Hope it’s an interesting view into my development mindset.

– Morgan Schweers, CyberFOX!

I’m really hoping this lets me immerse myself in a crowd of very smart people thinking about cool stuff, so that I can kick-start my own brain cells into working a bit better on my personal projects.

If I talk about anything, it’ll probably be my current attempt at working with NLP and specifically extracting information about whether two blogs talk about similar things. Effectively beating at the ‘information overload’ problem.

Beyond that, we’ll see what happens. I’m hoping I can stay awake for a lot of it. I’ve got my toys (PSP, MacBookPro, Camera), and my pillow. I’m nervous, but excited. Time to get going!

– Morgan Schweers, CyberFOX!

This is a hard thing to explain if you haven’t been there. I’ve been there twice, once with McAfee Associates, in full-bore, turbo-charged engineer mode, fighting against the world-wide virus writing epidemic in the very early days of McAfee Associates (from 7 people to more than 120 people). Fewer people would recognize that world as would recognize the world of a modern web site developer, so I’ll focus on the second, PayPal.

I remember working at PayPal during their heyday just before and just after going public, when we were fighting the ‘good fight’ against eBay, and I’d work all hours of the day and night not because I had to, but because it was deeply, personally important to me that we win, and because the work was so deeply enthralling that I lost track of time entirely. That everything be right, and that we be first to market with features, that our code be spectacular, that we be innovative and brilliant and FAST was our world. And until we were bought by eBay finally (demonstrating, imo, that we had the better service), no matter the hour, I never was alone at the office.

When you’re part of a brilliant team, trying to honest to god change the world, it’s not about deadlines. It’s about a form of love. It can be thoroughly, caustically destructive to everything else in your life, but it’s an experience I would be a lesser person if I had missed.

If you’re clocking hours at work, and the passion of what you’re doing isn’t keeping you rooted to your chair at all hours, loving the pure joy of creating, fighting the good fight, and trying to change the world, it doesn’t mean it’s not a good job. It’s just not THAT experience.

I promise, there’s far more call for people who work regular hours, meeting normal deadlines, doing solid, good work, than for those of us who burn so very, very brightly, but for so short a time.

If you’re really in passionate love with the work you’re doing, it’s not about working to 3am to meet a deadline. It’s about finally reaching a temporary point of closure for the days work, and raising up your head to suddenly discover it’s 3am.

And if you really, truly believe in your company, and you believe in your project, and have a fire to ‘win’ in some way (usually against a more powerful competitor) it’s not about accepting an imposed deadline that makes you work hard. It’s about DEMANDING a deadline that makes you work hard, but that you know you can meet. Because you know it’s important, and that every second counts, and you CARE about the company being not just first, but first with a brilliant, innovative, wonderful experience.

After it’s all over, it’s draining. It’s exhausting. It’s mind-numbing. You feel…dead, somehow, once the work is over, and you’ve been brilliant for so long, that you feel like your brain cells have used up all their energy. You go home, don’t show up to work for a week, recharge, find out if you still have any RL friends, do something physical (skydiving, rock climbing, hiking, etc.) to get in touch with your body again. You come back to work eventually, and you work with others to clean up any loose ends, and slowly you get back the energy from your co-workers, and the ambience in the office, and eventually you’re back on track to start another feature that’ll knock the socks off your competitors.

If you’re in a company where you are the dominant force, you don’t work like that. You don’t need to, the hunger isn’t there.

PayPal lost that hunger when eBay bought us. There wasn’t anything to fight for, anymore. We’d won, in a way, and lost in a way. I even asked it, when eBay management had a big meeting with everybody to tell us about their vision for us. I don’t remember if it was the meeting Meg Whitman was at, or not, but I asked something like, “We’ve been fighting eBay for all this time, and now we don’t have to. What will replace that, to keep the drive going?” The answer was a mealy-mouthed mess of future strategy and becoming the dominant payment platform. It wasn’t a battle anymore, we’d become the big company.

I left not terribly long after that, for health reasons. (Remember what I said about caustically destructive? ) But also because I didn’t feel the passion in the hallways anymore.

The woman who I will marry in 36 days stood by me through it despite almost never seeing me, my friends teased that they’d forgotten my name, I ended up needing major surgery for a condition I let go too long… But I was part of one of those winning teams, fighting against terrible odds, doing brilliant work, burning so very, very brightly, and changing the world one line of code at a time.

I think Mark understands that, as Google has Microsoft with an unlimited war-chest bearing down on them. From what I’ve read, I don’t think Scoble completely does get it. He gets that passion is important (Channel 9 certainly shows that), but the fight against overwhelming odds that drives it to fevered peaks, that brings it to a different level…that’s what’s missing.

But it’s okay. Even I’m working a day job these days. I still have the intense passion to program on my own projects, doing it until 4am regularly, but I’m in a larger cycle of recharge, get in touch with life, etc., before maybe doing it again if I find the right company. Or maybe not. I’ll be a married man shortly, settling down in theory. Maybe I can’t fight those fights anymore. Maybe I should work at Microsoft. Just kidding…

– Morgan Schweers, CyberFOX!