Just a Theory

Today I’ve finished just about over two and a half weeks of hacking on Bricolage. It has been a couple of years since I gave it much attention, but there was so much good stuff that other people have contributed that, since I had a little time, it seemed worth it to give it some love. So here’s a quick list of all that I’ve done in the last two weeks:

• Fixed all reported issues with Bricolage 1.10. Scott Lanning kindly released 1.10.5 yesterday with all of those fixes.

• I integrated the element occurrence branch that Christian Muise had worked on as his 2006 Google Summer of Code project. Christian’s project added support for maximum and mininum specfications for subelements in Bricolage, which allows administrators to define how many fields and elements can occur in a story or media document. All I had to do was add a few UI tweaks to support the new fields and their specification in the story profile, and all was ready to go. Oh, and I did have to go back and make the SOAP interface work with the feature, but the only reason it never did was lazy hacking of the SOAP interface (way before Christian’s time). Nice work, Christian, and thank you for your contribution!

• I fixed a few bugs with Arsu Andrei’s port of Bricolage to MySQL, which was his 2006 Google Summer of Code project. Arsu did a terrific job with the port, with only a few minor things missed that he likely could not have caught anyway. This work had already been merged into the trunk. Thanks Arsu!

• I fixed a bunch of bugs from Marshall Roch’s AJAXification of Bricolage, carried out during his 2006 Google Summer of Code project. Marshall actually did a lot more stuff than he’d planned, as it all went quite smoothly. I found only a few minor oversights that I was able to eaily address. This work represents the single most visible change to how users user Bricolage since we launched the probject back in 2001. Editing stories, in particular, is now a lot cleaner, with far fewer page loads. Thanks a million, Marshall!

• I completed the work started by Chris Heiland of the University of Washington, Bothell, and Scott Lanning of the World Health Organization to port Bricolage to Apache 2. They really did most of the hard work, and I just spent several days integrating everything, making sure all the features work, and updating the installer to handle differences in configuration. I thought this would take me a day or two, but it actually took the better part of a week! So much has changed, but in truth Bricolage is now better for running on mod_perl 2. Expect to see Apache 2 bet the recommended platform for Bricolage in a release in the near future.

• I integrated a number of patches from Brian Smith of Gossamer Threads to allow the installer to be run as a non-root user. The key here is if the installer has to become the database super user, which is required for ident authentication, and of course whether files are to be installed somewhere on the system requiring super user access. This work is not done, yet, as make upgrade and make uninstall are not quite there yet. But we’re getting there, and it should be all done in time for 2.0, thanks to Brian.

• I added support for a whole slew of environment variables to the installer. Now you can set environment variables to override default settings for installation parameters, such as choice of RDBMS, Apache, location of an SSL cert and key, whether to support SLL, and lots of other stuff, besides. This is all documented in the Quick Installation Instructions section of Bric::Admin/INSTALL.

• I fully tested and fixed a lot of bugs leftover from making the installer database- and Apache-neutral. Now all of these commands should work perfectly:

  • make
  • make cpan
  • make test
  • make install
  • make devtest
  • make clone
  • make uninstall

• I improved the DHTML functionality of the Add More widget, which is used to add contact information to users and contributors, rules to alert types, and extensions to media types. I think it’s pretty slick, now! This was built on Marshall’s AJAX work.

All of these changes have been integrated into the Bricolage trunk and I’ve pushed out a developer release today. Please do check out all the goodness on a test box and send feedback or file bug reports! There are only a couple of other features waiting to go into Bricolage before we start the release candidate process. And, oh yeah, tht title of this blog post? It’s not a lie. The next production release of Bricolage, based on all this work, will be Bricolage 2.0. Enough of the features we’d planned for Bricolage lo these many years ago are in the trunk that the new version number is warranted. I for one will be thrilled to see 2.0 ship in time for OSCON.

And in case it isn’t already clear, many thanks to the Google Summer of Code and participating students for the great contributions! This release would not have been possible without them.

Also in the news today, the Bricolage server has been replaced! The new server, which hosts the Web site, the wiki and the instance of Bricolage used to manage the site itself, is graciously provided by the kind folks at Gossamer Threads. The server is Gossamer Threads’s way of giving back to the Bricolage community as they prepare to launch a hosted Bricolage solution. Thaks GT!

The old Bricolage server was provided by pair Networds for the last five years. I’d just like to thank pair for the generous five-year loan of that box, which helped provided infrastructure for both Bricolage and Kineticode. Thank you, pair!

And with that, I’m going heads-down on some other projects. I’ll pop back up to make sure that Bricolage 2.0 is released in a few months, but otherwise, I’m on to other things again for a while. Watch this space for details!

I’m very pleased to report that the Google Summer of Code Bricolage projects have all been successfully completed. The contributions of the three Summer of Coders, Marshall Roch, Christian Muise, and Andrei Arsu, will be included in the next major release of Bricolage. On behalf of the Bricolage community, like to extend my gratitude to Google for sponsoring these three excellent students to dramatically improve the interface, capabilities, and compatibility of Bricolage.

So what got done? Here’s a rundown:

• Marshall Roch added many slick Ajax features to Bricolage. The story profile now manages the editing of all elements and subelements in a single screen, with no loading of a separate screen for subelements. You can navigate to subelements by clicking on a tree structure right in the story profile. Subelements more than three levels down will be loaded dynamically when you get to them. You can also drag and drop fields and elements to reorder them.

  Other stuff that Marshall Ajaxified:

  • Document and category keyword editing

  • Document category association

  • Document output channel associations

  • Organizations in the source profile

  • The Add More sections of the user, contributor, media type, and alert type profiles

  • Roles in the contributor profile

  • Assets on desks and My Workspace

  Marshall worked hard to integrate more interactive features into this 2000-era application, and I, for one, appreciate his hard work. Great job, Marshall!

• Christian Muise added support for an occurrence specification to element types and field types. That means that when you make an element type a subelement of another element type, you can specify the minimum and/or maximum number of times that it can be a subelement. So when an element of the parent type is created, it will automatically add the minimum number of instances of a subelement specified for that parent type. This will allow an entire element tree to be pre-populated as soon as you create a new story or media document. Leaving the min and max occurrence set to 0 (zero) maintains the old behavior (no required subelements and an unlimited number can be added).

  Christian did the same for field types, too. The old Required and Repeatable attributes are gone; now you just specify a minimum number to require that number of instances of a field, and a maximum number to limit the number of instances. Together with the element type occurrence specification, this functionality allows Bricolage administrators to have a lot more control over the structure of the documents created by editors.

  Christian worked hard to complete this project, despite other huge demands on his time this summer (including a full-time job!). But thanks to his active participation on the developer mail list and his willingness to ask questions of his mentor, Scott Lanning, and myself, he overcame all obstacles to implement these features. He even wrote a number of new tests to ensure that it works properly and will continue to do so for the foreseeable future.

  Excellent work, Christian, and thank you so much for your contribution!

• Andrei Arsu ported Bricolage to MySQL 5. Bricolage has always run on PostgreSQL and used a number of PostgreSQL-specific features to ensure that it ran properly and well. Andrei took these on, converting the existing PostgreSQL DDL files to work on MySQL, figuring out how to convince MySQL to work with some of their idiosyncrasies, and writing compatibility functions in the MySQL driver and upgrade module so that things should largely just work. As a result, for the first time ever, you can now build and run Bricolage on MySQL. Can compatibility with other databases be far behind?

  Andrei picked up Perl very quickly during this project, and was able to understand how such horrible code as the Bricolage installer worked without running screaming from the project. His code was well-written and his approaches to compatibility flexible and scalable. Well done, Andrei!

Future Plans

The next tasks toward getting this code integrated and released are as follows:

• Andrei will merge his MySQL port into subversion trunk. This should actually be fairly straight-forward.

• Marshall will merge his Ajaxification work into trunk. I don’t expect that there will be any conflicts with Andrei’s work, as the two projects were orthogonal.

• Christian will merge his occurrence specification work into trunk. This will require that he work some with Andrei to ensure that his changes to the PostgreSQL DDLs are propagated to the new MySQL DDLS. He will also then need to work with Marshall to make sure that the occurrence specification works properly with the Ajaxified UI.

Once these tasks have been completed, we’ll be ready to release a development version of Bricolage with all three of these major improvements. The development release will allow members of the Bricolage community to start to play with the new features, report bugs, and make further suggestions for improvement. Expect the release sometime in the next six weeks or so.

Again, my thanks to Marshall, Christian, and Andrei for their hard work this summer, and for all that they have contributed to the Bricolage community and project. I hope that each will remain involved in the community, not only to support the features they’ve added, but to work with other members of the community to add new features, help newbies, and generally to spread the word.

Some time ago, I decided that all objects in Bricolage 2 would have Universally Unique Identifiers, also known as UUIDs. A UUID is guaranteed to be universally unique, never to be generated again by the same or any other system now or in the future. As anyone using Bricolage knows, all stories and media already have IDs, so why have UUIDs, as well? How does their purpose differ?

Well, first of all, the existing IDs are not really identifiers. What they are, instead, are primary keys. However, a primary key should ideally be a surrogate key, meaning that it has no other meaning outside of identifying a single database row. Sometimes you can use an intelligent key, meaning an attribute of the object being stored (such as a user login), for the primary key. But the problem with intelligent keys is that, should their values ever be changed (say a user’s name changes and company login name conventions dictate that the login must be changed to represent the new name), all foreign key references will be broken. It is therefore easier, and more agile, to use a surrogate key with no inherent meaning to the object with which it is associated.

Now, once you start using an object ID that is actually a surrogate key for something other than identifying a row in a database, you add new meaning to it. At that point, it is no longer a surrogate. In Bricolage, this comes up when users want to use IDs for story URIs. At that point, the ID is no longer just a primary key identifying a database row, but it is also an object identifier. What happens if that identifier changes? Well, in general, it won’t, so you’d be safe to use it for both purposes. But sometimes it does.

When? Some Bricolage users have decided to upgrade to a newer version of Bricolage by setting up the new version on a different server, exporting their data from the old server, and then importing it into the new. This can work reasonably well, but it has what may be an unintended side-effect for those who use the ID in the URI: all objects will get new primary keys when they’re inserted into the new system.

What? you cry! Yes, that’s right. Because the ID is used solely to identify a row in a database, when you insert an existing object into a new database, it gets stored in a new row. It therefore gets a new ID, and your URIs suddenly start to 404. Ouch.

The solution to this problem is to give Bricolage objects a universally unique identifier that can work anywhere, that means nothing other than this is a unique identifier for this object, and which are guaranteed not to change when you move an object from one system to another. Happily, the UUID standard exists for just this sort of thing. You are free to use a story’s UUID in its URI without having to worry about it ever changing. IDs may change, but you don’t have to worry about those.

For these reason, the forthcoming Bricolage 1.10.0 has added UUIDs to story and media objects, these being the objects most in need of UUIDs, and they are available for use in URIs. Looking to the future, the Kinetic Platform, currently under development and the platform to which Bricolage 2.0 will be ported, never exposes the primary key IDs at all. There is only the UUID for referencing objects externally. I judge this a very good thing.

So, I’m thinking of implementing permissions in the Kinetic Platform differently than they’re implemented in Bricolage. Bricolage has a number of fixed permissions: READ, EDIT, RECALL, CREATE, PUBLISH, and DENY. These permissions are cumulative, so that if you have EDIT permission, it implies READ, and if you have CREATE, it implies RECALL, EDIT, and READ.

This design was based on Windows NT permissions (roughly), and has worked reasonably, well, but is annoying for various reasons. The most serious drawback is that it’s difficult to understand. I always tell people who need to manage Bricolage permissions to read Bric::Security, and then read six more times. But aside from the impenetrability of the current permissions design, it’s also difficult to add new permissions: where should they fit into the hierarchy? This is what happened with RECALL and PUBLISH, which were added in a later version of Bricolage. To this day, it’s a bit confusing to some that, with RECALL permission, you can RECALL a story but not CREATE one.

So I’m looking around for other permissions patterns. Unix is nice, in that READ, WRITE, and EXECUTE permissions are all entirely independent, and apply to three classes of objects (file owner, file group, everyone). But Unix only needs to worry about files; Kinetic applications will have many more objects for which permissions will need to be specified. RT uses discreet permissions with names like AdminQueue, CommentOnTicket, CreateTicket, and StealTicket to be applied to every user or group. This strikes me as somewhat more useful, since the permissions are much more descriptive and can be targeted to particular objects. In fact, the permission names even indicate to what types of objects permissions apply!

So I’m thinking of leaving the cumulative permissions model behind and switching to more descriptive, discreet, and potentially numerous permissions. I’m not, however particularly fond of RT’s approach of storing the permissions as strings in the database. Now, I could keep them as numbers, where each permission has its own unique number. This is similar to how Bricolage permissions work. Only I’d have to always use List::MoreUtil’s any() function to see if a permission was in a list.

For example, say that an object had permissions with the numbers 1, 2, 5, 8, 12, 49, and 50. If these were stored in an array, then every time I had to check permissions, the can_do() function would have to search through those numbers:

sub can_do {
    my ($self, $to_check) = @_;
    my $perms = $self->perms;
    return any { $_ == $to_check } @$perms;
}

This isn’t the most efficient approach, unfortunately. If there are a lot of numbers, and you were always checking one that was towards the end of the list of permissions, it’d always take a long time. So, an alternate approach—one that conveniently works well with discreet permissions—is to use powers of two for the permissions numbers: 1, 2, 4, 8, 16, 32, 64, 128, 256, and so on. Then, for a given object, these can just be stored in a single number that’s created by bit-wise ORing them together: 1 | 2 | 8 | 16. Fortunately, PostgreSQL has a nice aggregate function for this, bit_or(). (I’ll have to create my own for SQLite.) Anyway, then the can_do() function becomes much simpler and more efficient:

sub can_do {
    my ($self, $to_check) = @_;
    return $self->perms & $to_check;
}

If the value of $to_check was in the list bit-ORed into their permissions number, it will be returned; otherwise, 0 will be returned. Not bad, eh?

So anyway, I’m soliciting feedback. Are discreet permissions better than cumulative permissions? And if so, are bit-wise ORed numbers the best way to represent an object ACLs?

% make devtest
PERL_DL_NONLAZY=1 /usr/bin/perl inst/runtests.pl -d
t/Bric/Test/Runner....ok                                                     
        1801/10001 skipped: various reasons
All tests successful, 1801 subtests skipped.
Files=1, Tests=10001, 247 wallclock secs (110.30 cusr +  8.90 csys = 119.20 CPU)

How ‘bout them apples?