Recently I got a casual query from a user who was interested in what else I had going on. I do have a lot of projects that I work on, along with spending time supporting JBidwatcher.  What follows is a lightly edited version of what I told them…

(1) Most obviously there is JBidwatcher which I spend some time supporting every day via email, forum, and the support site, and sadly a little less time developing, but if you’ve poked at my site you’ve noticed that I’ve got several other projects listed.  There is also one or two that are not listed which I’ll talk about.

(2) One of the most obvious other projects to JBidwatcher users is My JBidwatcher.  It has its own configuration tab in JBidwatcher, and it’s pretty far along in terms of features.

Unfortunately I’ve not implemented the ‘Add Auction’ command in My JBidwatcher yet, nor the ability to send snipes back to your desktop JBidwatcher instance.  All the capabilities to do it are in place, I just haven’t seen enough interest from users to implement it.  My JBidwatcher gets a very small number of occasional users, although it’s been invaluable in helping me debug JBidwatcher, especially recently with the ‘null’ priced items problems.  You can link My JBidwatcher to your desktop app just by entering your account information (user name and password) into the JBidwatcher ‘My JBidwatcher’ configuration.

I really want to make it work better, but I’ve not gotten a confident feel that folks want that capability, or what folks would like from a web interface to JBidwatcher, so I’m unsure about spending time working on it as opposed to other projects.

(3, 4) As for Hacker’s Health (and the iPhone companion app Health Hacker)…  Heh!  In 2007 as part of learning Ruby on Rails, I built a health tracking application for myself.  I still use it, and I’d love others to see it, but it needs some cleanup before that happens.  Fast forward to 2010, when I built an iPhone application around communicating with Google Health, and storing stuff in their data format (which is crazy complex, because it had to support health data vaults, essentially).  Then I started working for Google, and put it on hold, because they were talking about doing one themselves and I didn’t want to step on any toes…  Now that I’m not working for Google anymore, and Google Health has sadly gone away, I’ve revived it and updated my web application and have started making the two of them talk to each other.  This project is lots of fun to build, and very useful as every time I actually put focus on my health it gets better.

(5) One of the last of my publicly acknowledged projects, MacBidwatcher is an application I’m actively working on. Right now it can load eBay auctions, trigger updates, shows thumbnails,  track them in folders, and you can drag and drop items between folders. I’ve added the ability to log in, but only on ebay.com so far. (No international support yet.) I’ve got some prototype code that bids, but I need to build the user interface for bidding, then start adding features to support sniping.

MacBidwatcher is a much more Mac-like application than JBidwatcher though, and my plan is (eventually) to offer it on the Mac App Store, and maybe through non-MAS sales as well.  Even folks like me need to eat.

(6) Not really discussed anywhere, is iBidwatcher which is an iPhone version of JBidwatcher. It’s already got a few cool features (including secure over-the-air sync with JBidwatcher if you’re on a wireless network with a running JBidwatcher instance) but it’s severely hampered by the inability of iPhone applications to do anything in the background. This means it can’t keep the price of a listing up to date when not running.  I’ve written code to scrape eBay’s site on the iPhone, and it works great, but it can only run when you are in the app. It also cannot snipe from the phone, because the phone would have to be on and the app running, which is counter to the idea of a sniping application, i.e. you shouldn’t need to think about it. Instead I’m considering partnering with Gixen, and allowing you to place snipes on their service through iBidwatcher if you have a Gixen Mirror subscription. Their price of $6/year seems reasonable, and I have a lot of respect for the person who runs it.

(7) The least interesting to most JBidwatcher users is Outlinr, which has gotten sidelined as I deal with other projects.  It was a blast to build, and was functional for a while, but server upgrades and the relentless progress of browsers has made it no longer work.  I love outlining as a knowledge-capture model, and think it could be done SO much better than anyone is currently doing it, but I have to focus on projects that I know others are more immediately interested in.

I’d be very interested in your thoughts on the various projects I’ve described, and what you’d like to see out of them.  I’m always looking for more insights into what would be helpful and valuable.

– Morgan Schweers, CyberFOX!

Too long to fit in the margins of this book

My journey starts with Matt Aimonetti’s excellent MacRuby: The Definitive Guide, and it’s Core Data chapter. It’s a really good introduction to Core Data, but I needed something a bit deeper and more advanced. My dream, of course, would be to use Core Data with Active Record’s ease of use. Core Data is really complex, though. Books have been written on the subject; and yet my complete exposure to it has been writing a few Objective C iPhone applications for my own use.

My problem

It’s always best to start with a relatively simple action, and figure out how to build on that. My initial problem to solve was allowing the user to select a Category from an outline view on the left (iTunes style) and pull up a list of the items in that Category in a table view in the main part of the UI, like so:

I’m sure I could do this a number of ways, and it’s probably even possible to make it work with the standard NSArrayController like in MacRuby: The Definitive Guide, but there are a number of places where it won’t work as well, such as when I want to find the item ending the soonest from all Categories, or load and modify specific Auction items during updates or when the user chooses to perform actions on individual items. At that point, it’ll become necessary to do the Core Data handling ‘by hand’, and so I felt it’d be best not to start with something I was going to have to bypass anyway.

The Data Model

It may be helpful to sketch a simple data model, described using the data modeler in Xcode; some other models in the project have been elided for clarity:

Because of the reciprocal relationship between Category and Auction, I could simply load a Category with a particular name, and then refer to category.auctions to get the list (in Core Data form) of Auction objects associated with that category. Adding .allObjects returns the actual objects as an array for easy referencing in a TableView. (This is not efficient, but I’m trying to get the code working right now; I’ll manage efficiency once I’ve gotten it in users hands.)

MacRuby vs. Objective C

The Objective C (that I’ve written) to do a similar load takes about 80 lines of code; I’m not embedding it because it’s too long, and it only loads by name. You can go take a look; it’s pretty straightforward. Unlike the Objective C version, the Ruby code can load by any attribute dynamically, and has additional features like limits and offsets.

This is the core of my code when the user clicks a category/folder on the left:

The key code being referenced is this:
@current_category = Category.find_first(context, by_name:category_name)
It’s roughly equivalent to the Objective C code (extracted from the app that uses my ObjC code above):
NSArray *cats = [Category findByName:category fromContext:managedObjectContext];
The main difference is that the Objective C code requires a different method for each findBy* that you want to do, whereas the MacRuby code uses the fact that its parameters are really just hash entries using the ‘:’ syntax for faux named parameters. So, for example, while this code is doing a find_first, if it were to use find_all it could also pass ..., limit:20, offset:7) if you wanted to get 20 items starting at the 7th.

The Magic

The magic, of course, happens in entity.rb which looks like this:

It references inheritable_attrs.rb which is available as a gist of its own

One important part that any entity needs to get is a ‘context’, which is actually a managed object context. Fortunately when you create a Mac Ruby project using the MacRuby Core Data Application template, a context is available from the AppDelegate class. You may notice that in my AuctionTableDatasource there is: attr_writer app_delegate which doesn’t really do anything obvious. That’s an ‘outlet’ to refer to the AppDelegate; using Interface Builder control-click-and-drag from your data source to the App Delegate, and choose the app_delegate outlet to link it up. The default Core Data AppDelegate class instantiates a Managed Object Context which is essentially a link to your Core Data database. It needs to be passed in to code which is going to interact with data from the database, which is why each of the public methods in entity.rb takes a context as their first parameter.

A MacRuby Caveat

I mildly disagree with the default MacRuby Core Data Application template which creates an XML-based Core Data app; specifically it creates a file with the extension of .xml and passes NSXMLStoreType as the persistent store type. I believe it should pass a .sqlite file extension and use NSSQLiteStoreType as the storage type. For small applications, it may not matter, but if you feel you’re likely to be storing enough data that handling Core Data by hand will be necessary, then you’re going to want the SQL-based storage type. The typical recommendation is to start with an XML type and switch to SQL when you’re going to release, nominally because XML is easier to read. I dispute that, though, because there are slight behavioral differences, such as the XML storage type keeping the entire object graph in memory and reports of subtly different handling of (generally incorrectly specified) relations. If you want to use an SQLLite backed database, you’ll want to fix up AppDelegate.rb once you’ve created your project.

Conventions and Features

I know there’s a lot of disagreement over Active Record’s design, but I happen to like it a lot, and many of the guidelines of Rails creep into my code. I especially like the presumption that there’s an easy way to do whatever you’re doing, that the code handles for you automatically, but if you want to fight the convention, you can; it just won’t be as clean. There’s a lot of ‘convention’ involved in the Entity class, especially in the presumptions that you’ll have model classes which are named identically (including case!) to the core data models you’ve created, i.e. generally with the first letter uppercased. You should be able to override that by setting the entity_name in your subclasses using

require 'entity'
Class OddlyNamedModel < Entity
  self.entity_name = 'oddly_named_model'
.
.
.

but it’s not something you should do lightly, or at all if you’re creating a new project, and embarrassingly, I haven’t tested it.

Attributes

Attribute names were something else to deal with; I typically name my attributes in lower case while, as mentioned, my entities have their first letter capitalized. So my Category.find_first(context, by_name:category_name) method will look for an attribute called ‘name‘. I am aware that some people capitalize the first letter of their attributes, so: Category.find_first(context, byName:category_name) would work if you have an attribute called ‘Name‘ and looks very Objective C-ish.

Limits and Offsets

It’s straightforward to index into your results, using Category.find_all(context, by_name:common_name, offset:20, limit:10); this would search for all instances that have common_name in their name attribute, and then starting from the 20th, pull down 10 of them. In SQLLite terms, this translates into something approximately like: SELECT * FROM ZCATEGORY WHERE NAME = ? LIMIT 10 OFFSET 20

Conditions (the escape hatch)

Another useful feature is a conditions key, which provides one entry which is raw predicate logic, essentially. An example for finding all categories that end in ‘nt’ would be: Category.find_all(context, conditions:['name like %@', 'nt']) which translates in SQLLite terms to: SELECT * FROM ZCATEGORY WHERE NSCoreDataLike( ZNAME, ?, 0)

Debugging

One incredibly useful built-in capability (and the source of the above approximations of the commands issued by Core Data) I’ve found for doing debugging of SQLLite-based Core Data code is to pass -com.apple.CoreData.SQLDebug 1 to the application from the command line, in order to view the queries as they’re being done. From the command line, in the root of your app directory, this would look something like this

bash$ build/Release/MyApp.app/Contents/MacOS/MyApp -com.apple.CoreData.SQLDebug 1

Conclusion

I’ve tried to make an extremely simple class that I can subclass for my entities, that allows me to do straightforward find operations in a way that doesn’t feel overly complex, and fits in a Ruby-ish style. It’s working so far for my code, but I can’t promise it’ll work for everybody’s. Especially if you’re facing down historical schemas, or other issues, my defaults are probably not going to be appropriate for you. It’s definitely less code than the Objective C equivalent, and feels…more comfortable for me.

I’m VERY interested in feedback, and improving this code. As I stated at the outset, the best I could be considered is a hobbyist Objective C coder, and I’m only just starting to dig into the power that MacRuby has. Please feel free to provide pointers, comments, ideas, features, or (if necessary) derision. Especially feel free to fork entity.rb and make corrections or additions as you see fit.

– Morgan Schweers, CyberFOX!

Update: A friend and quite excellent Objective C developer David Brown provided his own take on the Entity concept; according to him it’s untested and needs a few tweaks, but the basic concept should work equally well; check it out on his own gist of entity.m. It’s like a Rosetta stone between MacRuby and Objective C!

What alternative do you suggest for using models in migrations? I was in several situations where I had to not only change the underlying db structure but change the contained data, too.

Data changes, especially moving data around, are almost always rake task-worthy in my experience.

The other side of that, populating large amounts of seed data into new databases, is a difficult task no matter the method; seed_fu attempted to deal with it, but it’s not an optimal solution and pretty old. I’m not even sure if it works anymore. It’s worse if you need the seed data to be from a legacy database in tests (e.g. a nutritional database). Reloading lots of data each time a clone_structure_to_test is done makes your tests very slow.

I break down migrations into three kinds; structural (tables, columns, indices, etc.), data (pre-populating tables, etc.) and procedural (moving data around, recalculating counts, etc.). The first is what I strive to limit migrations to. I feel like there should be a good answer for the second and Rails 3.0 has a ‘Simplest Thing That Can Work’ feature in Rails commit #4932f7b. The third, I try to relegate to rake tasks that are usually run once, on deployment of the branch.

The procedural tasks don’t need to be run when building a fresh database, because there isn’t legacy data to correct. That’s why you can usually define the model in the migration to force it to work even if the real model is gone or renamed; there’s no data, so the operations often don’t matter. If they don’t NEED to be run when building a fresh database, I try not to put them in the migrations.

It’s not ‘hard and fast’, because I usually work in startups to small companies, where dogma doesn’t work so well. Imagine, though, a large and thin piece of foam. It’s flexible, and you can make it into all sorts of shapes, and yet it’s simple. Each time you add code that makes reasonable changes in the future painful, it’s like putting a thin glass rod into the foam. It’s still flexible, but there’s some bends you can’t do without breaking. Add too many and you’ve got an inflexible and brittle object, no matter how dynamic the base material is.

The fear of breaking things by changing the code is deeply demotivating for everyone.

I know I waterboarded that analogy, but hopefully it makes sense…

— Morgan Schweers, CyberFOX!

[Edit: Since writing this article up back in early March, I've moved on from this job. The folks who are now maintaining it at least know where the pain points are, can run migrations safely, deploy it locally, and to dev servers, and to the main deployment area.  It's a working app, although I never got code coverage above about 45% at least the coverage was decent in the core app areas by the time I left.]

Mike Gunderloy had an interesting article entitled ‘Batting Clean-up‘, which was very timely for me.  I’ve just started maintaining and trying to improve a Rails app developed by an ‘outsourced’ group. The only tests were the ones generated automatically by ‘restful authentication’, and they were never maintained, so they didn’t come close to passing. Swaths of the program are written in terribly complex (and sometimes computed) SQL, migrations didn’t bring up a fresh database (poor use of acts_as_enumerated causes great hurt), and vendor/plugins should have just had one named ‘kitchen_sink’.

It hurts to see Rails abused like that; you want to take the poor application under your arm and say, ‘It’ll be okay…we’ll add some tests and get you right as rain in no time!’, but you know you’d be lying…

I did much of what Mike described (half the gems it used were config.gem’ed, the other half weren’t), vendor’ed rails (it breaks on newer than 2.1.0), and brought the development database kicking and screaming into life. There was no schema.rb, it had been .gitignore’d, and the migrations added data, used models, and everything else you can imagine doing wrong. (Including using a field on a model after adding that column in the previous line…I don’t know what version of Rails that ever worked on…) I didn’t want a production database; who knows what’s been done to that by hand. I want to know what the database is _supposed_ to look like; I can figure out the difference with production later.

Once the clean (only data inserted by migrations) dev database was up, I brought the site up to see if it worked. Surprisingly enough, it did; apparently they used manual QA as their only testing methodology. I appreciate their QA a lot; it means it’s a working application, even if it’s not going to help me refactor it.

I ran flog and flay and looked at the pain points they found to get an idea how bad things might be. I picked an innocuous join table (with some extra data and functionality) to build the first set of tests for, which gave me insight into both sides of the join without having to REALLY dig into the ball of fur on either side. I viciously stripped all the ‘test_truth’ tests. I looked for large files that flog and flay hadn’t picked up to pore over. Check out custom rake tasks, because those often are clear stories and easy to quickly understand in a small context.

Checking out the deployment process tells you a lot also, although it turns out this was stock engine yard capistrano.

Skimming views (sort by size!) will tell you a lot also, especially when you find SQL queries being run in them…

Use the site for a little while, and watch the log in another window. Just let it skim by; if you’ve looked at log files much, things that seem wrong will jump out even if it’s going faster than you can really read.

In my case, the code’s mine now, so it’s my responsibility to make it better before anybody else has to touch it. I’ve got about a week of ‘free fix-it-up time’ before I need to start actually implementing new features and (thankfully) stripping out old ones… At my previous company, I was the guy pushing folks to test, now I’ve inherited a codebase with zero tests. Poetic justice, I suppose…

Good luck!

— Morgan Schweers, CyberFOX!

Programming should be fun, it’s what gets us programmers into it in the first place, and it’s what keeps us going at it.

Too many layers have been heaped on programming these days; most IDEs are oppressive, process-oriented beasts.

I, and many other programmers, have been concerned about how the next generation of programmers are going to be introduced to the joy of programming when all the tools out there seem to suck the fun out of it.

Enter why the lucky stiff, and Hackety Hack!. Taking one back to the days when programming was as easy as typing something in, and hitting enter.

Hackety Hack! is about learning to program, and being able to do really cool things in a very few short lines.

I’ve been a fan of _why for a while, with a really wild approach to teaching Ruby, and a lot of really interesting code, and this is even cooler.

If you’re new to programming, or have kids, or just want to play with one of the simplest ways to write programs, get Hackety Hack!

– Morgan Schweers, CyberFOX!