October 2011 Archives

Okay, we’ve got our tooling support built up so we can easily and quickly run tests — time to do some Real Work!

Before any coding, it’s usually helpful to think about what we’re trying to do, and how best to break down the data involved in the problem. Since the point of Jacquard is to aggregate together multiple social networking sites (and RSS feeds, eventually), we’re going to have a few generic object types in the system: we’ll have User objects representing our users, Service objects representing the different types of social networks, Account objects that map between a Service and a particular User, and Post objects that represent the individual pieces of content on a particular service.

Given that breakdown of the data structures, one plan of attack is to first create the User class, get the basics of that working in the data model layer of the application, then hook it into the web application layer. Generally speaking, I find that getting to the point where I can log in to the web application is a good stopping point.

N.b. : I’m going to be borrowing very heavily from Nothingmuch’s intro to using KiokuDB in Catalyst applications post here — you may want to jump over and read that before continuing.

So, the first thing to create is our user class in our data model. Following the conventions discussed in Nothingmuch’s article, we’ll call this class ‘Jacquard::Schema::User’:

package Jacquard::Schema::User;
# ABSTRACT: Jacquard users
use Moose;
with qw/ KiokuX::User /;  # provides 'id' and 'password' attributes

# we're going to use this to map 'username' to the 'id' attr
use MooseX::Aliases;

# and this is an easy way to verify we're getting a valid email
use MooseX::Types::Email qw/ EmailAddress /;
 # these are a couple of helper classes that make Kioku easier to use
use KiokuDB::Set;
use KiokuDB::Util              qw/ set /;

use namespace::autoclean;

# we want to have our username be our unique ID in Kioku
alias username => 'id';

has email => (
  isa      => EmailAddress ,
  is       => 'ro' ,
  required => 1 ,

# this attribute is going to contain info about all the services
# this user has configured -- i.e., there will be one for Twitter,
# one for Facebook, etc.
has accounts => (
  isa     => 'KiokuDB::Set',
  is      => 'ro',
  lazy    => 1 ,
  default => sub { set() },


To sanity check the basics that we’ve done so far, let’s make a simple test class for Jacquard::Schema::User — something like this:

package Test::Jacquard::Schema::User;
use strict;
use warnings;

use parent 'Test::BASE';

use Test::Most;

use KiokuX::User::Util qw/ crypt_password /;

use Jacquard::Schema::User;

sub test_constructor :Tests(7) {
  my $test = shift;

  my $user = Jacquard::Schema::User->new(
    id       => 'user1' ,
    email    => 'user1@example.com' ,
    password => crypt_password( 'bad_password' ) ,

  isa_ok( $user , 'Jacquard::Schema::User' );

  is( $user->id       , 'user1'             , 'id' );
  is( $user->username , 'user1'             , 'name delegated to id' );
  is( $user->email    , 'user1@example.com' , 'email' );

  ok( $user->check_password( 'bad_password' ) , 'check password' );
  ok( ! $user->check_password( 'wrong password' ), 'check wrong password' );

  is_deeply( [ $user->accounts->members ] , [] , 'no accounts' );


What happens when we run that?

$ prove -lv
t/01-run.t .. # 
# Test::Jacquard::Schema::User->test_constructor

ok 1 - The object isa Jacquard::Schema::User
ok 2 - id
ok 3 - name delegated to id
ok 4 - email
ok 5 - check password
ok 6 - check wrong password
ok 7 - no accounts
All tests successful.
Files=1, Tests=7,  1 wallclock secs ( 0.02 usr  0.01 sys +  0.53 cusr  0.03 csys =  0.59 CPU)
Result: PASS

It’s important to note at this point that we don’t have anything really KiokuDB-specific going on with that User class. Yes, we used a few Kioku-related helper classes, but that’s just because we knew we were headed towards KiokuDB eventually. Those helpers could easily be replaced (or even removed, in a few cases), leaving behind a simple Moose class that describes a User object. In order to really hook this up to KiokuDB, the next step is make a Model class. This will allow our data objects to persist (i.e., save them to disk), and also provides a place to put helper methods to wrap transactions and a way to define a data storage API that will be used by the Catalyst web application layer. (Cooler people might call this a data storage DSL…) Here’s an initial version of this Model class:

package Jacquard::Model::KiokuDB;
# ABSTRACT: KiokuX::Model wrapper for Jacquard
use Moose;
extends qw/ KiokuX::Model /;

sub insert_user {
  my( $self , $user ) = @_;

  my $id = $self->txn_do(sub {
      $self->store( $user )

  return $id;


And of course, we need a test class too — this one uses Kioku’s ability to “store” things to memory, rather than a database, making it easier to set up the tests.

package Test::Jacquard::Model::KiokuDB;
use parent 'Test::BASE';

use strict;
use warnings;

use Test::Most;

use Jacquard::Model::KiokuDB;
use Jacquard::Schema::User;

use KiokuX::User::Util qw/ crypt_password /;

sub fixtures :Tests(startup) {
  my $test = shift;

  $test->{model} = Jacquard::Model::KiokuDB->new( dsn => 'hash' );

sub test_insert_user :Tests(3) {
  my $test = shift;
  my $m    = $test->{model};

    my $s = $m->new_scope;

    my $id = $m->insert_user(
        id       => 'user1' ,
        email    => 'user1@example.com' ,
        password => crypt_password( 'bad_password' ) ,

    ok( $id , 'got id' );

    my $user = $m->lookup( $id );

    isa_ok( $user , 'Jacquard::Schema::User' , 'looked up user' );
    is( $user->username , 'user1' , 'expected name' );


(Note how we’re using the Tests(startup) method attribute to set up our test fixtures.)

Running the test suite shows us that everything is working as we expect:

$ prove -lv
t/01-run.t .. # 
# Test::Jacquard::Model::KiokuDB->test_insert_user

ok 1 - got id
ok 2 - looked up user isa Jacquard::Schema::User
ok 3 - expected name
# Test::Jacquard::Schema::User->test_constructor
ok 4 - The object isa Jacquard::Schema::User
ok 5 - id
ok 6 - name delegated to id
ok 7 - email
ok 8 - check password
ok 9 - check wrong password
ok 10 - no accounts
All tests successful.
Files=1, Tests=10,  2 wallclock secs ( 0.03 usr  0.01 sys +  1.19 cusr  0.06 csys =  1.29 CPU)
Result: PASS

We’re almost ready to create our Catalyst application and hook up the authentication bits to our model/schema code — but first, let’s write a little helper utility to create a user. That will make our interactive testing of the web application a little bit easier. We’ll use SQLite and the KiokuDB/DBIx::Class bridge for the moment; if and when this gets rolled out to more people, we’d want to change that up to something with a bit more power (e.g., by swapping Postgres in for SQLite, or, if we want to hop the NoSQL train, by switching over to CouchDB).

First, a configuration file, so we don’t have to hardcode the database connection details. This goes in jacquard.yaml:

  dsn: dbi:SQLite:dbname=db/jacquard.db

(Down the road, we’ll add the Catalyst application configuration to this file too.)

Next up, the helper script. This just needs to load up the config, prompt for needed info, create a Jacquard::Schema::User object, instantiate a KiokuDB connection with Jacquard::Model::KiokuDB, and use the insert_user method to store the object in the database. This will live in script/create_user:

#! /opt/perl/bin/perl

use strict;
use warnings;
use 5.010;

use FindBin;
use lib "$FindBin::Bin/../lib";

use Jacquard::Model::KiokuDB;
use Jacquard::Schema::User;
use KiokuX::User::Util         qw/ crypt_password /;
use YAML                       qw/ LoadFile       /;

my $dsn = parse_config_file();

my( $username , $email , $password ) = prompt_for_info();

my $user = Jacquard::Schema::User->new(
  username => $username ,
  email    => $email ,
  password => crypt_password( $password ),

my $m = Jacquard::Model::KiokuDB->connect( $dsn , create => 1 );
  my $s = $m->new_scope;

  my $id = $m->insert_user( $user );

  say "STORED USER ID '$id'";

# config file parsing and info prompting details elided; if you're
# really interested, the code is on Github... 

If we wanted to get really fancy, we could make this take command line flags for the needed information — and it should probably not echo the password back to the screen, and should handle the exception that’s going to be thrown if the email address isn’t properly formed, or if the username already exists — but that can all be added down the road. For the moment, we just need to get a User object saved in the database so we have something to authenticate against…

So, we run this to create a test user:

 $ ./script/create-user 
EMAIL? test@example.com
STORED USER ID 'user:test'

At this point, if you’re doing this for real, it’s worth pausing for a few minutes and using the SQLite tool to poke around inside the database that’s been created for you in db/jaquard.db. Since this is already a super long post and one of the overarching points of this series is that KiokuDB means you don’t need to worry about how your stuff lives on disk, we’re gonna skip that.

Next time, we’ll set up our Catalyst app and use this user class to hook up authentication.

Update: See the Catalyst application set up in “Setting Up Authentication”.

Okay, first step is to get some tooling in place to make it easier to deal with developing this thing. Since I'm expecting to work on this for a while, and will probably end releasing the code on CPAN, it makes sense to invest a little up front effort on making testing and releasing the code easier.

Item the first: a Dist::Zilla config. If you're not familiar with Dist::Zilla, it's a set of tools to make it easier to write Perl code with an eye towards distributing it on CPAN. There's more info at the Dist::Zilla site.

Since I've been using dzil for a while, I've taken the step of uploading my own plugin bundle to CPAN -- so my dist.ini file to configure Dist::Zilla is pretty straightforward:

name    = Jacquard
author  = John SJ Anderson <genehack@genehack.org>
license = Perl_5
copyright_holder = John SJ Anderson <genehack@genehack.org>
copyright_year   = 2011


The intro boilerplate is pretty self-explanatory, and all that last line says is "use my standard plugins and settings."

Item the second: Test::Class tooling. I've been using the OO-ish approach to writing tests for a while now and I find it very convenient. I'm not going to try to explain how it works, since Ovid has already written a very nice series of articles over on Modern Perl Books about using this module:

To make it easier to use we're just going to create a test helper to run all our Test::Class tests -- this will go in a file called t/01-run.t:

#! perl
use strict;
use warnings;
use Test::Class::Load qw( t/lib );

All that does is automatically find all our testing libraries under t/lib and automatically run each one in turn. We're also going to create a testing base class at t/lib/Test/BASE.pm that all our test libraries will inherit from:

package Test::BASE;
use parent 'Test::Class';

INIT { Test::Class->runtests }


This is done so that individual test libraries will do the right thing when run with the prove tool -- more about that in a later installment.

Item the third: a top level library file. This isn't strictly needed, and isn't going to contain any code, but I always like to have a module file that matches the name of the distribution. So we'll put this into lib/Jacquard.pm:

package Jacquard;
# ABSTRACT: Jacquard is a social network and RSS feed aggregator.

At some point, we'll come back and revise that abstract line, once we have a better idea of what this thing wants to do.

For the moment, let's make sure things are "working" as expected:

<Jacquard:/> $ dzil test
[DZ] building test distribution under .build/EkNRq3jF7V
[DZ] beginning to build Jacquard
[DZ] guessing dist's main_module is lib/Jacquard.pm
[DZ] extracting distribution abstract from lib/Jacquard.pm
[@GENEHACK/@Basic/ExtraTests] rewriting release test xt/release/pod-coverage.t
[@GENEHACK/@Basic/ExtraTests] rewriting release test xt/release/pod-syntax.t
[@GENEHACK/@Basic/ExtraTests] rewriting release test xt/release/eol.t
[@GENEHACK/@Basic/ExtraTests] rewriting release test xt/release/kwalitee.t
[DZ] Override README from [ReadmeFromPod]
[DZ] writing Jacquard in .build/EkNRq3jF7V
Checking if your kit is complete...
Looks good
Writing Makefile for Jacquard
Writing MYMETA.yml and MYMETA.json
cp lib/Jacquard.pm blib/lib/Jacquard.pm
Manifying blib/man3/Jacquard.3
PERL_DL_NONLAZY=1 /opt/perl-5.14.2/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/li    b', 'blib/arch')" t/*.t
t/00-compile.t ............ ok   
t/01-run.t ................ No subtests run 
t/release-eol.t ........... skipped: these tests are for release candidate testing
t/release-kwalitee.t ...... skipped: these tests are for release candidate testing
t/release-pod-coverage.t .. skipped: these tests are for release candidate testing
t/release-pod-syntax.t .... skipped: these tests are for release candidate testing

Test Summary Report
t/01-run.t              (Wstat: 0 Tests: 0 Failed: 0)
  Parse errors: No plan found in TAP output
Files=6, Tests=1,  0 wallclock secs ( 0.04 usr  0.02 sys +  0.17 cusr  0.03 csys =  0.26 CPU)
Result: FAIL
Failed 1/6 test programs. 0/1 subtests failed.
make: *** [test_dynamic] Error 255
error running make test

And since we haven't actually written any tests yet, that's about what we should expect. That's a nice place to stop and commit what we've done so far:

[master]<Jacquard:/> $ git add dist.ini t/01-run.t t/lib/Test/BASE.pm lib/Jacquard.pm

[master 6cd7634] Set up basic dist.init, Test::Class tooling, and lib/Jacquard.pm
 4 files changed, 132 insertions(+), 0 deletions(-)
 create mode 100644 dist.ini
 create mode 100644 lib/Jacquard.pm
 create mode 100755 t/01-run.t
 create mode 100644 t/lib/Test/BASE.pm

and a good time to stop for the moment. In the next installment, we'll write some code that actually does something, and start figuring out how this thing is actually going to work.

Update: The Jacquard story continues in "Setting Up Users".

Around the beginning of 2010, I started a project that I eventually ended up calling App::StatusSkein. The idea was to provide reading and posting access to a variety of social networks from a single unified location. As these personal projects will, it grew to the point where it was tolerable for my primary use for it, and my desire to develop it further slackened. I had placed some (in retrospect) odd design constraints on it — I was trying to not have a backing database at all, to only run locally, and to only keep a certain minimal amount of state loaded at any given time. After the initial page load, all the interaction between the web browser and the server was AJAX-based, and reloading the page would reset the state to the initial default. Parts of this worked well; parts of it… just worked, but all in all, it was successful at scratching the itch I had at the time.

Now, I’ve got a different itch — same general area, but different design constraints. Now that I’ve got an iPad, it would be nice to have something similar to StatusSkein, but server-based, able to keep track of where in the timeline I’d last left off reading, able to save posts for later, and able to track precisely which posts I’ve read and which I haven’t, regardless of what location I accessed them from — something like a mashup of the UI of Google Reader and the current StatusSkein. Technology-wise, I’d also like a chance to play around with a Catalyst app that uses KiokuDB instead of the usual (for me) DBIx::Class. It would also be nice if I had a better way of maintaining my interest in the development of this app beyond the initial “hey, this itches!” stage. The ideal way to do that is to get users. Failing that, perhaps readers will provide some motivation…

So, welcome to what is hopefully the first in a series of posts outlining bite-sized bits of coding and other noddling around as I work on developing this new trans-social network app that I’m currently calling Jacquard

Update: The Jacquard story begins in “Setting Up Project Infrastructure”.