November 2011 Archives

• Thanks to Gabor for the Jacquard shout out in the latest Perl Weekly. If you’re into Perl, you should subscribe to this newsletter, it’s a good roundup of Perl-related stuff from around the web.
• MongoDB sucks. No, it doesn’t. (Former link from everywhere; latter from the comments over at Flutterby.)
• Charlie Stross on Evil Social Networks. Quoted for Truth: “If you’re not paying for the product, you are the product.”
• Be On Fire - “YOU. MUST. BURN.”

Since our previous installment got us users and authentication, we’re ready to start adding support for various services. One of the features of Jacquard — one of the main points of Jacquard, actually — is interacting with multiple services. To make that possible, we’ve included the accounts attribute in our User class:

# this attribute is going to contain info about all the services
# this user has configured -- i.e., there will be one for Twitter,
# one for Facebook, etc.
has accounts => (
  isa     => 'KiokuDB::Set',
  is      => 'ro',
  lazy    => 1 ,
  default => sub { set() },
);

Before we start actually writing the code for the things that are going to be inside those accounts attributes, it’s worth stepping back and thinking about what the data we want to store looks like, and how we can best structure our classes to help us manipulate that data.

Ideally, what we want — and again, this is a reflection of the underlying raison d’etre of Jacquard — is to be able to interact with all the different services we support in the same way without having to worry how each individual service handles fetching new posts, or writing a new post out. In other words, to be able to do something like this:

# warning: pseudocode
foreach my $account ( $self->accounts->members ) {
  $account->get_new_posts;
}

# or even
my $new_post = get_new_post_from_user();
foreach my $account ( $self->accounts->members ) {
  $account->post( $new_post )
}

Since we’re using Moose, we’re going to have a number of Jacquard::Schema::Account::FOO classes, one for each type of service we’re going to interact with. Each of those classes will consume an API-defining role (called something like Jacquard::Schema::Role::AccountAPI) that will define the required methods for the common interface we want all services to have — that is, making sure they support a get_new_posts method, and a post method, and so forth.

We also need to think about how to structure the other data associated with services. That data will fall into two broad categories: account information (and similar data, like authentication tokens, etc.) and posts on that service. Depending on the exact service, the details of what a ‘post’ is will differ — a tweet is not quite a FaceBook post, and both are very distinct from a blog post entry in an Atom feed — but they share enough commonalities that we can again take a similar approach: a number of Jacquard::Schema::Post::FOO classes, each consuming a Jacquard::Schema::Role::PostAPI role that ensures that they’re implementing a common API.

The benefit of this approach is that it becomes relatively trivial to add support for new services — particularly if there’s already a module on CPAN to handle interacting with that service. As we’ll see in a number of upcoming posts, taking an existing library and wrapping it in a layer of Moose class to implement a particular API is very little work.

I often find it helpful, when I’ve gotten to this point in the process of designing something and I think have a workable solution, to work out how a small set of data would map across instances of these classes. Let’s consider how a single Jacquard user, with a Twitter account configured within Jacquard, would look once a few tweets were stored:

# first, the user
my $user = Jacquard::Schema::User->new( name => 'Bob' );

# then we add the account
$user->add_account( 'Twitter' , %account_details );

# and then we get the most recent posts
$user->get_account( 'Twitter' )->get_new_posts();

And for this design, this is the point where I realize I haven’t really thought at all about how the individual Post objects will be associated with the Account object. The simplest way to do this would be for the Account classes to have a posts attribute, much like the accounts attribute in the User class:

has posts => (
  isa     => 'KiokuDB::Set',
  is      => 'ro',
  lazy    => 1 ,
  default => sub { set() },
);

Assuming that is how we go, after that get_new_posts() method above, we’d end up with the accounts attribute of $user containing a KiokuDB::Set object with a single member — an instance of Jacquard::Schema::Account::Twitter. Inside that object, there would be another KiokuDB::Set object with a bunch of members, each one an instance of Jacquard::Schema::Post::Twitter corresponding to an individual tweet from this user’s timeline (and having attributes like ‘author’, ‘content’, ‘datetime’, ‘id’, and so on).

That all seems to make a reasonable amount of sense — but there’s a looming problem. (If you’ve already spotted it, give yourself a pat on the back.) What happens when we add a second User, also with a Twitter account configured, and that user ends up following some of the same people on Twitter as our first user? We’ll end up with Post objects that are essentially duplicates of each other — but one copy will be inside the Twitter Account object of the first user, and the second will be inside the Twitter Account object of the second. In the long run, that’s going to end up being a big waste of storage space and/or RAM.

How do we solve this problem? We could just maintain a uniform set of Post objects, and include the same object into different Account objects as needed. That would solve the issue with the duplication of information — but it introduces another issue, in that we no longer have a place to store per-user metadata about individual Post objects (e.g., read/unread status).

Instead, we’ll solve this problem the old fashioned way: we’ll introduce another layer of abstraction! Instead of the posts attribute of Account objects containing Jacquard::Schema::Post objects directly, we’ll have a generic Jacquard::Schema::UserPost object that maps between a User account and a particular Post. In return for making our data model slightly more complicated, this approach gives us the best of both worlds, in that we have a place for per-User metadata to live, but the original Post data is only present in our system once, regardless of how many of our users have it present in their Account objects.

This solution also doesn’t impact any of our previous design decisions: the Jacquard::Schema::Role::PostAPI can be consumed by the Jacquard::Schema::UserPost class too, via method delegation to the Jacquard::Schema::Post::FOO object it refers to. (More about that later.)

So, now that we’ve been though a couple of rounds of thinking about the data structures, we can write some code, yes? Well, no, not exactly, not quite yet. Instead we’re going to explore what the API for using this code might look like, and we’re going to do that by writing some test code, or at least outlining some test cases.

First, we’re going to need to be able to add Account objects to User objects, and Post objects to Account objects. The code for that will probably look something like:

## method to associate an Account object with a User
# $model is an instance of Jacquard::Model::KiokuDB, 
# while $user is a Jacquard::Schema::User, 
# and $account->does('Jacquard::Schema::Role::AccountAPI')
$model->add_account_to_user( $account , $user );

## method to add a UserPost object to an Account -- will also save the
## associated Post object if needed 
# $model is an instance of Jacquard::Model::KiokuDB, 
# while $account->does('Jacquard::Schema::Role::AccountAPI')
# and $post->does('Jacquard::Schema::Role::PostAPI')
$model->add_post_to_account( $userpost , $account );

(Aside: I spent more time than I am willing to disclose trying to decide whether the order of arguments in those methods should be ‘more generic object, more specific object’, or if they should instead be the same as in the method name — which is what I finally went with, reasoning that will serve as a mnemonic.)

We’ll also need to be able to remove Account objects from User objects, and modify Post and UserPost objects and save those changes:

## method to remove an Account object from a user
# $model is an instance of Jacquard::Model::KiokuDB, 
# while $account->does('Jacquard::Schema::Role::AccountAPI')
# doesn't need a user object because $account has an 'owner' attribute
$model->remove_account_from_user( $account );

## method to store a modified Post or UserPost object, e.g. after
## changing some of the metadata, or if the underlying post object is
## updated
# $model is an instance of Jacquard::Model::KiokuDB, 
# while $post->does('Jacquard::Schema::Role::PostAPI')
$model->update_post( $post );

Those all look reasonable, at least at first glance. It’s worth noting that none of the methods have an explicit return value — they will just return true on success (and will likely eventually throw some sort of exception object on failure). This approach makes the most sense to me, as it is isolates the model to just marshaling objects to storage. Any modification or manipulation of the objects will happen elsewhere.

At this point, we’ve fleshed out the data model more, we have the beginnings of a plan for implementing the code that will let Jacquard interact with other services, and we have the outline for how we’re going to store and update that data in KiokuDB once we have it. That seems like a good place to stop. Next time, we’ll actually implement the first Account and Post classes!

As always, the code for Jacquard is available on Github. Patches are welcome; share and enjoy.

I started running some time in the late summer of 2010, doing the Couch To 5K program. I’d tried to start running a few other times, and I always had issues with shin splints that would prevent me from establishing any sort of routine. This had been the case for as long as I could remember — I ran track in my senior year of high school, and I had shin splint problems then too.

The thing that fixed those problems for me was this pair of Vibram Five Fingers KSOs. Running in these hurt at first too — but it was a different kind of hurt, a more manageable one, and one that got less the more I ran — the complete opposite of my shin splint problems, in other words.

My last run in these was the Marine Corps Marathon 10K. It was a cold, wet morning, and that hole you can see in the sole of the right shoe didn’t do me any favors — but I finished the race. Seemed like a good time to let these tired soldiers get some rest, so today I picked up a new pair of TrekSports. I’m hoping the slightly thicker sole and treads on the bottom will let me explore some trail running this spring — and who knows, there’s a local half marathon in May…

Those of you that aren’t familiar with the Vibrams, or the idea of minimalist running, may want to check out this NYT article for some background. I can’t say enough good things about these shoes; they’ve made it possible for me to run and enjoy it, and that’s just awesome.

In the last installment, we created Jacquard::Schema::User to describe a Jacquard user, and used Jacquand::Model::KiokuDB (and a helper script) to persist User objects to storage. This time around, we’re going to create our Catalyst app and hook up authentication.

So, first step: create the Catalyst application. The catalyst.pl helper makes this trivial:

$ catalyst.pl Jacquard::Web
created "Jacquard-Web"
created "Jacquard-Web/script"
created "Jacquard-Web/lib"
created "Jacquard-Web/root"
[ couple dozen more lines elided ]

Unfortunately, there’s no easy way to tell it “I’m creating this Cat app as a part of a bigger project” — so after creating it, you have to manually shuffle the files around to get them into the right place. The Catalyst files also come with quite a bit of templated stuff in them that we don’t need (POD skeletons and the like), so cleaning that up and getting the code into line with your personal coding style should be done at this point. Once that’s all done, checkpointing into revision control is a good idea:

$ git ci -m"Create Catalyst application" 
[03-catalyst 3c20771] Create Catalyst application
 11 files changed, 300 insertions(+), 0 deletions(-)
 create mode 100755 jacquard_web.psgi
 create mode 100644 lib/Jacquard/Web.pm
 create mode 100644 lib/Jacquard/Web/Controller/Root.pm
 create mode 100644 root/favicon.ico
 create mode 100644 root/static/images/btn_88x31_built.png
 create mode 100755 script/jacquard_web_cgi.pl
 create mode 100755 script/jacquard_web_create.pl
 create mode 100755 script/jacquard_web_fastcgi.pl
 create mode 100755 script/jacquard_web_server.pl
 create mode 100755 script/jacquard_web_test.pl

Next, we’re going to use CatalystX::SimpleLogin to add authentication to the application, following the outline in the manual. First up, add a number of plugins to the application (in lib/Jacquard/Web.pm):

use Catalyst qw/
    -Debug
    ConfigLoader
    +CatalystX::SimpleLogin
    Authentication
    Session
    Session::Store::File
    Session::State::Cookie
    Static::Simple
/;

Create a default View class by running:

./script/jacquard_web_create.pl view TT TT

Then create the Catalyst Model class that will wrap Jacquard::Model::KiokuDB:

package Jacquard::Web::Model::KiokuDB;
use Moose;

use Jacquard::Model::KiokuDB;

BEGIN { extends qw(Catalyst::Model::KiokuDB) }

has '+model_args'  => ( default => sub { { extra_args => { create => 1 }}});
has '+model_class' => ( default => 'Jacquard::Model::KiokuDB' );

__PACKAGE__->meta->make_immutable;
1;

We also need to add the configuration for these plugins to the application config file:

---
name: Jacquard::Web
Model::KiokuDB:
  dsn: dbi:SQLite:dbname=db/jacquard.db
Plugin::Authentication:
  default:
    credential:
      class: Password
      password_type: self_check
    store:
      class: Model::KiokuDB
      model_name: kiokudb

Finally, we can add a method requiring authentication to the root Controller (which is at lib/Jacquard/Web/Controller/Root.pm):

sub hello_user :Local :Does('NeedsLogin') {
  my( $self , $c ) = @_;

  my $name = $c->user->id;

  $c->response->body( "
Hello, $name!
" );
}

(Don’t forget that you need to change the parent class of the controller as well:

BEGIN { extends 'Catalyst::Controller::ActionRole' }

If things aren’t working, make sure you didn’t forget this.)

With all this in place, you should be able to start up the application, by either running ./script/jacquard_web_server.pl or, if you want to get all Plack-y and modern, plackup jacquard_web.psgi, and then browse to the URL the server reports it’s running on. You should see the Catalyst welcome page at that point. Add ‘/hello_user’ to the end of the URL, and you should get prompted for a username and password. Assuming you created an account using the helper script from the last installment, you should be able to give those same values and see a page that says ‘Hello’ and your test account username.

Now, are we done? Not quite, as we don’t have any tests of the authentication code! The following goes in t/lib/Test/Jacquard/Web/Controller/Root.pm:

package Test::Jacquard::Web::Controller::Root;
use parent 'Test::BASE';

use strict;
use warnings;

use Test::Most;
use Test::WWW::Mechanize::Catalyst;

sub fixtures :Tests(startup) {
  my $test = shift;

  # load test config
  $ENV{CATALYST_CONFIG_LOCAL_SUFFIX} = 'test';

  $test->{mech} = Test::WWW::Mechanize::Catalyst->new( catalyst_app => 'Jacquard::Web' );
}

sub auth_test :Tests() {
  my $test = shift;
  my $mech = $test->{mech};

  $mech->get_ok( '/' , 'basic request works' );
  is( $mech->uri->path , '/' , 'at top page' );

  $mech->get_ok( '/hello_user' , 'request hello_user' );
  is( $mech->uri->path , '/login' , 'redirect to login' );

  $mech->submit_form_ok({
    form_id => 'login_form' ,
    fields  => {
      username => 'test_user' ,
      password => 'test_password' ,
    } ,
    button => 'submit' ,
  } , 'login' );

  is( $mech->uri->path , '/hello_user' , 'redirect to /hello_user' );
  $mech->text_contains( 'Hello, test_user!' , 'see expected greeting' );

  $mech->get_ok( '/' , 'basic request works' );
  is( $mech->uri->path , '/' , 'at home page' );

  $mech->get_ok( '/hello_user' , 'request hello_user' );
  is( $mech->uri->path , '/hello_user' , 'go directly to hello_user' );
  $mech->text_contains( 'Hello, test_user!' , 'still see expected greeting' );

  $mech->get_ok( '/logout' );
  is( $mech->uri->path , '/' , 'back at home page' );

  $mech->get_ok( '/hello_user' , 'request hello_user' );
  is( $mech->uri->path , '/login' , 'redirect to login' );
}


1;

If you’re not familiar with Test::Class-style testing, you should review the links I gave in the post on the Jacquard project infrastructure. Even if you’re not that versed in Test::Class, hopefully the sequence above is fairly self-explanatory: we try to access a restricted URL, verify we get bounced to the login URL, then fill out and submit the login form, and verify we get re-directed to the original request. Following that, we logout and verify that we’re once again unable to get to the restricted resource. Fairly simple.

In order to make this work, we need a distinct application config for testing, one that defines a test user and password for us. Catalyst, of course, provides a way to do this — that’s what that line that sets the CATALYST_CONFIG_LOCAL_SUFFIX environment variable is about. The testing config goes into jacquard_test.yaml, and looks like this:

---
Plugin::Authentication:
  default:
    credential:
      class: Password
      password_field: password
      password_type: clear
    store:
      class: Minimal
      users:
        test_user:
          password: 'test_password'

With that test file and test config in place, we can run the tests:

$ prove -l -I./t/lib ./t/lib/Test/Jacquard/Web/Controller/Root.pm -v
./t/lib/Test/Jacquard/Web/Controller/Root.pm .. # 
# Test::Jacquard::Web::Controller::Root->auth_test

ok 1 - basic request works
ok 2 - at top page
ok 3 - request hello_user
ok 4 - redirect to login
ok 5 - login
ok 6 - redirect to /hello_user
ok 7 - see expected greeting
ok 8 - basic request works
ok 9 - at home page
ok 10 - request hello_user
ok 11 - go directly to hello_user
ok 12 - still see expected greeting
ok 13 - GET /logout
ok 14 - back at home page
ok 15 - request hello_user
ok 16 - redirect to login
1..16
ok
All tests successful.
Files=1, Tests=16,  4 wallclock secs ( 0.04 usr  0.00 sys +  3.67 cusr  0.20 csys =  3.91 CPU)
Result: PASS

and verify they pass. This is also a good point to rerun the whole test suite:

$ prove -l
t/01-run.t .. ok    
All tests successful.
Files=1, Tests=26,  6 wallclock secs ( 0.03 usr  0.01 sys +  4.16 cusr  0.27 csys =  4.47 CPU)
Result: PASS

So, at this point, we have a basic data model (which only models users — but that’s about to change!), a KiokuDB-based persistence framework, and we’ve got all that hooked into a Catalyst-based web application and can use the info in our User object to authenticate to the web app. Sounds like a good place to take a break! In our next installment, we’ll pull back from the coding just a bit and think about the best way to structure the data involved in the services we’re going to connect to, and the posts we’re going to read and write to those connections.

As always, the code for Jacquard is available on Github. Patches are welcome; share and enjoy.

Update: The Jacquard story continues with “Extending The Data Model”.