The idea behind multiple table inheritance is to combine multiple tables behind a view that is writable and updateable like a normal table, which can allow you to do all sorts of nifty things. The combination of the tables is accomplished via a set of triggers and rules for insertion and deletion. These rules are the heart of the beast. Check out the synopsis for some examples.

mst first spoke about this at the London Perl Workshop in 2008, and he recorded a video about it at nwe.pm in 2009.

The rule-generation is Postgres-specific now, but theoretically, if a database has writable views, MTI should be able to support it. Theoretically.

Try it out!

I’ve been on my new job for 3 weeks now, and I am settling in enough to reflect a bit on the job hunt.

It sure was strange!

Perl experts kept telling me a couple of things over and over:

Interviewer: How do you switch a hash’s keys and values?
Me: reverse %hash…
Interviewer: WRONG! You fail!
Me: Um…
Interviewer How would you print sorted the values of a hash?
Me: print sort values %hash…
Interviewer: WRONG! You fail!
Me: Wait a second, that’s correct…
Interviewer: No it’s not. Next question. Have you ever written your own assembly implementation of partial sort?

The above happened to me twice. Not in so many words, but you get the idea: combine a flagrantly outdated understanding of Perl with the desire to show off hardcore l33tn355, and keep the belly laughs coming!

I wanted to extend on my last post about a way to make sure your multiple-table-inheritance database can be updated properly across whatever views you are running your app off of. But I just got some new code to look at for that, and haven’t quite grokked it yet. So that will have to wait till next time.

Meanwhile, I’d like to present a couple of ideas I have been working on in my Metalabel rewrite, and perhaps solicit some feedback. I’d love to hear suggestions or strategies. The first is a module some people have seen around—it is called Moosifier, and it takes DBIx::Class Schema::Loader result class files, reads them, and adds Moose bits at the top and bottom to allow you to use your auto-generated classes with Reaction or whatever.

Before:

    package Test::Schema::Amazon;

    use strict;
    use warnings;

    use base 'DBIx::Class';

    __PACKAGE__->load_components("TimeStamp", "Core");
    __PACKAGE__->table("amazon");
    __PACKAGE__->add_columns(
      "asin",
      {
        data_type => "text",
        default_value => undef,
        is_nullable => 1,
        size => undef,
      },
    ...
    1;

After:

    package Test::Schema::Amazon;

    use Moose;

    extends 'DBIx::Class';

    has 'asin' => (
                    isa => SimpleStr,
                    is => 'rw',
                    required => 0

    );
    has 'locale' => (
                    isa => SimpleStr,
                    is => 'rw',
                    required => 0

    );
    has 'filename' => (
                    isa => SimpleStr,
                    is => 'rw',
                    required => 0

    );
    has 'refetchdate' => (
                    isa => Int,
                    is => 'rw',
                    required => 0

    );

    use namespace::autoclean;

    __PACKAGE__->load_components("TimeStamp", "Core");
    __PACKAGE__->table("amazon");
    __PACKAGE__->add_columns(
      "asin",
      {
        data_type => "text",
        default_value => undef,
        is_nullable => 1,
        size => undef,
      },
    ...
    __PACKAGE__->meta->make_immutable(inline_constructor => '0');
    1;

I wrote it because I thought I was going to be using Reaction in my rewrite, but phaylon advised against it. It adds a whole other layer to Catalyst’s M, V and C, and I agreed that it’s wisest to make sure the basic rewrite is solid before adding a whole new layer, and, for instance, having to think about chopping my as-yet-unwritten templates into widgets or what have you.

Thus Moosifier. It’s pretty darned simple: it works by reading the result class files and using a bunch of regexes to extract the Moose attributes from the __PACKAGE__->add_columns(...) lines. I was excited to put it up and finish it, but no sooner had I done so than I hit upon another problem. Now that I had my classes written, I had to make my controllers.

Metalabel 2.0 needs at least 76 controllers. There’s no way I’m going to write those by hand. I could use myapp/script/myapp_create.pl helper script to make skeleton controllers, of course. But that’s just a skeleton. What about basic CRUD? There’s no way I’m going to write CRUD for 76 controllers by hand.

I asked on #catalyst whether there are any tools to help generate CRUD automatically in your controllers, and glob recommended Catalyst::Controller::DBIC::API, for which there are RPC and REST implementations. Of course, these just provide a way to write your CRUD. They don’t do it for you. So I had a new automation problem: “How can I automate the creation of basic but smart RPC or REST controllers for my Catalyst app, based on my DBIx::Class::Schema::Loader result classes?”

Of course I asked on irc first. Knowing that I had done something similar with Moosifier, mst suggested that instead of reading the files from disk, I take the table info directly from the database using DBIx::Class::Schema::Loader. Why the hell not? It makes much more sense than stepping through a file with regexes and substitutions and such.

The goal is to create 76+ controllers (in this case, of the RPC subclass) like this sample code from the Catalyst::Controller::DBIC::API documentation:

        
    package MyApp::Controller::API::RPC::Artist;
    use base qw/Catalyst::Controller::DBIC::API::RPC/;

    __PACKAGE__->config ( action => { setup => { PathPart => 'artist', Chained => '/api/rpc/rpc_base' } },
    class => 'MyAppDB::Artist', # DBIC schema class
    create_requires => ['name', 'age'], # columns required to create
    create_allows => ['nickname'], # additional non-required columns that create allows
    update_allows => ['name', 'age', 'nickname'], # columns that update allows
    list_returns => [qw/name age/], # columns that list returns
    list_prefetch => ['cds'], # relationships that are prefetched when no prefetch param is passed
    list_prefetch_allows => [ # every possible prefetch param allowed
        'cds',
        qw/ cds /,
        { cds => 'tracks' },
        { cds => [qw/ tracks /] }
    ],
    list_ordered_by => [qw/age/], # order of generated list
    list_search_exposes => [qw/age nickname/, { cds => [qw/title year/] }],
    );

I figure I can dump out such a controller for every table I need by pulling the whole database into a hash using DBIx::Class::Schema::Loader, and using this hash to populate the __PACKAGE__->config(...) for each table/controller. For instance, the class name is easy. So is create_requires: those are the non-nullable columns in the table, besides any auto-incrementing serials or sequences. create_allows is every column but those in create_required, also excluding serials and sequences. update_allows is the columns in both create_allows and create_requires. I can’t escape some manual labor here: I am going to make list_returns return every column in the table. I’ll go through the controllers by hand to customize those. The same with list_prefetch_allows: the object of every relationship identified by DBIx::Class::Schema::Loader is going into list_prefetch_allows (none are going into the default list_prefetch). I’ll have to customize those manually, as well. list_ordered_by may need to be manually specified later as well, but using date column if it exists might be a good default. list_search_exposes will expose everything by default.

So, I am in the process of writing 2 modules to do this: one for RPC and one for REST. They’ll be on my github soon.

Meanwhile, using DBIx::Class::Schema::Loader to read table definitions directly from the database in these modules has made me realize I need to change Moosifier to do the same. In fact, D::C::S::Loader simply needs to be able to write the Moose declarations itself. Moosifier needs to move into Loader.

That’s my next project.

EDIT:

There’s a branch for it now. I need some tests though.

Well, I am definitely busted back down to paper man now. But I have a real good reason. I was JFDI. I was learning to test (I’d say I am level 1 for the moment). Boy, is testing a strange enterprise for the beginning tester. Like millions before me, I did “testing” the other way round: write, run, tweak, run, tweak, run, etc., etc., ad infinitum. Basically, you have to turn your mind inside out and think about your code in what feels like a pretty much inverted way. But I like the discipline it imposes.

I was working on Catalyst::Helper::Controller::DBIC::API::REST, which I blogged about once before. It’s my first CPAN module, and I’m very proud of that fact! I’m not so proud of the fact that I had to upload the thing about 5 times to get it right. In any case, it’s a good feeling to be a contributing member of the thriving Perl 5 community. Thanks ribasushi, ilmari, mst, Caelum, arcanez, confound, lukes, Fitz Elliott, and everyone else I bothered while putting that thing together.

$ catalyst.pl myapp
$ cd myapp
$ script/myapp_create.pl controller API::REST DBIC::API::REST myapp

Instead of you naming the specific controller you want to create, as usual, you just write API::REST. The module loads your schema from the name of your app, e.g., myapp, and makes all your controllers in lib/myapp/Controller/API/REST/.

This module is incomplete. It remains to basically fill in the template to supply a whole__PACKAGE__->config(...). I also need to write out the lib/myapp/Controller/API.pm and the lib/myapp/Controller/API/REST.pm, in order to supply the base of the chain all the result-class controllers use. I think I’ll release this one to CPAN when I’m done, cause this one is independent, unlike the moosified branch of Loader.

Any and all feedback welcome. Hit me up on IRC (amiri).

One major problem was that Loader runs through its cycle twice, and it dumps the schema twice. The first time there’s no relationship information, because it hasn’t gotten it. The second time, after it gets the info and writes its relationship declarations… you still can’t get it! There’s a huge {_dump_storage} key in Loader’s $self, but if I wanted to parse that language, I would have stayed getting it from the files on disk. I honestly still don’t know what Loader does with all the relationship info after writing it out.

Well, that was dispiriting. So I gave up and just decided to get the info again and build my own relationship hash while iterating through the classes. It works pretty well.

One outstanding issue is how more subtly to handle relationships to other tables. Right now, if the relationship method is might_have Loader says

has 'posted_item' => (
	isa => 'Test::Schema::PostedItem',
	is => 'rw',
);

If the relationship method is has_many, Loader says

has 'discussion_topics' => (
	isa => ArrayRef['Test::Schema::DiscussionTopic'],
	is => 'rw',
);

It’s sort of a one-trick pony.

So, I had no clue what I was doing, really.

My data “model,” or my “domain model,” if it can even be called that, was nowhere near relational. It wasn’t even a data-warehouse model. It was just a bunch of stuff in some tables. There were no foreign key constraints.

I had to manually (in the application, which was no where near MVC) update everything to maintain the merest shred of “referential integrity.” In virtually every Mason component there was a huge <%init> block that not only pulled in data, but did maps, sorts, extractions, validation, etc. I might as well have been grepping through spreadsheets. I remember the day in early May 2007, when someone I considered a hardcore Perl programmer told me, “Oh, you’ll see. My ORM keeps it real simple. You do all that stuff application side.” Ah, hubris.

Here’s how it used to look:

Here’s how it looks now:

Wow. You can’t even really read the newer graph it’s so dense and interrelated.

The thing is, the current model grew organically, and I was just updating my MySQL tables and my class files manually for every additional column and accessor. After 18 months of that, adding new features, expanding this, modifying that, I had grown myself an unmanageably large helping of spaghetti. Everything was application side, remember? No constraints in the DB, no transactions.

I knew I needed some help. I hired shadowcat, and they told me first to send them my current domain model. I did. They pretty much threw it out and we rewrote it from scratch.

Let’s take a couple of objects as an example.

• User
• Artist
• Album
• Metalabel
• Track
• Label
• Scout
• Blog
• Photo
• Friend

I’m only going to talk about a couple of these in detail, but you’ll get the idea. This first thing phaylon noticed was that a user can do a couple of different things. A user can be an artist or a label and a user can create a metalabel. A metalabel does a subset of the functions of a user: it can make friends and it can post blogs and some other stuff. mst said “Smells like a role.” phaylon agreed, and came up with a couple of class declarations:

role Creator {
    has blog_entries_about isa ArrayRef[BlogEntry];
    has blog_entries isa ArrayRef[BlogEntry];
    has bulletins isa ArrayRef[Bulletin];
    has comments_about isa ArrayRef[Comment];
    has comments isa ArrayRef[Comments];
    has discussions isa ArrayRef[DiscussionTopic];
    has events isa ArrayRef[Event];
    has friend_requests isa ArrayRef[Friend] where is_accepted is false;
    has friends isa ArrayRef[Friend] where is_accepted is true;
    has photos isa ArrayRef[Photo];
    might have style isa CustomStyle;
    has creator isa ArtistIdentity |
                    Band |
                    Label |
                    Metalabel |
                    Scout |
                    User;
}

This Creator role will be consumed by all 5 classes that can “create” stuff. Those classes are ArtistIdentity, Band, Label, Metalabel, Scout, and User. Each of those classes have their own definitions. For instance:

class User does Addressable, Artworkable, Bloggable, Commentable, Creator, Discussable, Pollable {
    has artist_identities isa ArrayRef[ArtistIdentity];
    has credit isa CurrencyValue;
    has messages_out isa ArrayRef[Message::Private] where sender is me;
    has messages_in isa ArrayRef[Message::Private] where recipient is me;
    has messages_out isa ArrayRef[Message::Bulletin] where sender is me;
    has messages_out isa ArrayRef[Message::Bulletin::Friend] where sender is me;
    has rsvps isa ArrayRef[Event::RSVP];
    has metalabels isa ArrayRef[Metalabel];
    has labels isa ArrayRef[Label];
    has playlist isa Playlist;
    has address isa Address;
    has favorite isa Genre | Subgenre;
    might have open_id isa OpenID;
    has mail_preferences isa MailPreference;
}

In the User class, we first define the roles it consumes. You can see it consumes multiple roles, among them, Creator. So far, this is fairly clear, I hope. It’s just a sort of MooseX::Declare-like pseudocode now, but the basic idea is that every class does stuff. Some classes do the same stuff. That is a role. Some classes are really just instances of other superclasses. Like the Label, for instance:

class Label isa User, does Addressable, Bloggable, Creator, Commentable, Discussable, Genred, Pollable {
    has address isa Address;
    has artists isa ArrayRef[Label::Artist];
    has billing isa Billing;
    might have paypal_id isa PayPalID
    has company_name isa Str;
    has is_incorporated isa Bool;
    has corp_state isa Str;
    has tax_id isa Str;
}

The real mind-bending stuff for me came when we actually began implementing this stuff in SQL. The most important conceptual thing for me was how everything that consumes a role has to have a unique ID. Because a role is implemented in SQL as a primary-key only table!

create table creator (
    id integer primary key not null
);

Obviously, if 5 classes are going to be consuming this role, you cannot just start each class’s table with “id serial primary key not null,” because then you coulnd’t look up a Creator by its ID. Ther would be multiple Creators with the same ID, and your database would be useless.

So all role consuming classes share a sequence:

create sequence entity_seq start 1;
...

create table artist_identity (
    id integer primary key not null default nextval('entity_seq'),
    date timestamp not null default 'now',
    "user" integer not null,
    name text
);

This doesn’t only go for things like Creators, which are “active” tables, but also for things like Albums, which can be acted upon. My Album in Moosey pseudocode is:

class Album does Artworkable, Bloggable, Commentable, Discussable, Genred, Pollable, Videoable {
    has tracks isa ArrayRef[Track];
    has artist isa MusicProvider;
    has title isa NonEmptySimpleStr;
    has coverart;
    has notes isa Str;
    has is_deleted isa Bool;
    has date isa TimeStamp;
    has creation_date isa Date;
    has album_only isa Bool;
    has free isa Bool;
}

So, the Album object does 7 roles. It can be acted upon in 7 ways. It obviously also needs to share in the database-wide unique sequence, so I can look up any particular, say, Bloggable item by its ID:

create table album (
    id integer primary key not null default nextval('entity_seq'),
    date timestamp not null default 'now',
    provided_by integer not null,
    title text,
    creation_date date not null,
    notes text,
    album_only boolean,
    free boolean,
    deleted boolean
);

Finally, each object that consumes any role needs to have foreign keys to that role. So, the Album object needs a FK to each role it consumes. It needs 7 Fks:

alter table album add foreign key (id) references artworkable (id);
alter table album add foreign key (id) references bloggable (id);
alter table album add foreign key (id) references commentable (id);
alter table album add foreign key (id) references discussable (id);
alter table album add foreign key (id) references genred (id);
alter table album add foreign key (id) references pollable (id);
alter table album add foreign key (id) references videoable (id);

My User object needs 8:

alter table "user" add foreign key (id) references addressable (id);
alter table "user" add foreign key (id) references artworkable (id);
alter table "user" add foreign key (id) references bloggable (id);
alter table "user" add foreign key (id) references commentable (id);
alter table "user" add foreign key (id) references creator (id);
alter table "user" add foreign key (id) references discussable (id);
alter table "user" add foreign key (id) references pollable (id);
alter table "user" add foreign key (id) references videoable (id);

This is to make updating your tables across views easier when records are created or deleted.

I still don’t fully understand that bit, so I’ll try to write about it next time. Meanwhile, if you’re not doing something like the above, and you find yourself doing lots of manual record updates in your code, you’re not using your database effectively!