Just a Theory

Tonight we had a Portland Perl Mongers Perl 5.10 code sprint. This was the first time I’ve ever worked with the Perl 5 core code, and while things weren’t exactly intuitive, it wasn’t bad to get started, either. I already had a clone of the Perl Git repository, so here’s what I did to get started.

Here’s how to clone the Git repository, and then check out the maint-5.10 branch and build and test Perl:

git clone git://perl5.git.perl.org/perl.git
cd perl
git branch -b maint-5.10 origin/maint-5.10
sh Configure -Ode \
 -DDEBUGGING \
 -Dprefix=/usr/local/perl/blead \
 -Dusedevel \
 -Duseithreads \
 -Dccflags='-I/usr/local/include' -Dldflags='-L/usr/local/lib' \
 -Dlibpth='/usr/local/lib /usr/lib' \
 -Uversiononly \
 -Uinstallusrbinperl $@
make
make test

I had two test failures:

#   at ../lib/ExtUtils/CBuilder/t/00-have-compiler.t line 39.
#          got: '0'
#     expected: '1'
# Looks like you failed 1 test of 4.
FAILED at test 4

And:

#   at pod/pod2usage2.t line 235.
# Got:
# #Usage:
# #    This is a test for CPAN#33020
# #
# #Usage:
# #    And this will be also printed.
# #
# 

So the first thing I want to do is fix those failures. It took a bit of fiddling to figure out how to get it to run a single test. On the advice of Paul Fenwick and Schwern, I tried:

./perl -Ilib lib/ExtUtils/CBuilder/t/00-have-compiler.t

But then the test passed! So something has to be different when running under make test. With help from Duke Leto, I finally figured out the proper incantation:

make test TEST_FILES=../lib/ExtUtils/CBuilder/t/00-have-compiler.t

So the TEST_FILES option tells make test what tests to run, relative to the t/ directory. Shortly thereafter, Schwern told me about:

cd t
./perl TEST ../lib/ExtUtils/CBuilder/t/00-have-compiler.t

Which is nice, because it does a lot less work.

Now I was ready to figure out the ExtUtils::CBuilder bug and fix it. This is good, I thought, because ExtUtils::CBuilder is a Perl module, and while my C is teh suck, my Perl skillz are mad. So I do some poking around, and finally pinned down the failure to the have_compiler() method in ExtUtils::CBuilder::Base. A bit more poking and I had the error printed out:

# error building /tmp/compilet-1529033816.o from '/tmp/compilet-1529033816.c'
at ../../ExtUtils/CBuilder/Base.pm line 110.

Well fuck, that sure looks like a C problem. But then I added some more debugging code to see what command it’s actually running, and from where. So I added this code:

use Cwd;
print STDERR "# CWD: ", getcwd, $/;
print STDERR "# ", join( ' ', @cc, @flags), $/;

And then I got this output:

# CWD: /Users/david/dev/perl/lib/ExtUtils/CBuilder
# ./perl -e1 -- -I/Users/david/dev/perl/perl -c -fno-common -DPERL_DARWIN -no-cpp-precomp -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -O3 -o ./compilet-892348855.o

And of course, there’s no perl in lib/ExtUtils/CBuilder. Why does it think there is? The test file has this code:

$b->{config}{cc} = "$^X -e1 --";
$b->{config}{ld} = "$^X -e1 --";

Well, that cinches it. $^X is the path to the currently-executing Perl, but in core—for me at least—that’s a relative path. The test changes directories away from that path, and so it can’t find it. So I just needed to get it to use the proper Perl. Frankly, this test should fail on every platform that runs it, so I’m not sure how it got committed to the maint-5.10 branch, but here we are.

So I moved up use File::Spec; and assigned File::Spec->rel2abs($^X) to a variable before the chdir, and used that variable for the cc mock. And that was it, the test passed. Yay!

Now it was time to submit a patch to p5p. I committed the fix to my local clone, and then, with help from Jacob, generated a patch with git format-patch -1. The -1 means just a single commit, which is all I’d committed. This generated a file, named 0001-Fixed-failing-test-for-ExtUtils-CBuilder.patch. I edited it to add a comment to the effect that the patch is targeted at the maint-5.10 branch.

Next Jacob helped me get git configured to send mail through my gmail account. I just added these lines to ~/.gitconfig:

[sendemail]
        smtpserver = smtp.gmail.com
        smtpserverport = 587
        smtpuser = justatheory
        smtppass = idontthinkso!
        smtpencryption = tls

I then sent the patch with:

git send-email --to perl5-porters@perl.org 0001-Fixed-failing-test-for-ExtUtils-CBuilder.patch

And that was it! You can see the result here. I started to fiddle with the pod2usage failure, but it was a bit obscure for me. Perhaps next time.

And speaking of next time, I’m forking the Perl repository on GitHub, and will probably start to use my own repository for stuff (if it can be kept up to date; RJBS points out that it’s currently way behind the core repo). I’ll likely also create a topic branch before fixing a bug, and then send the patch from there. That way, when it’s committed to the remote branch in core, I won’t have to rebase my local copy: I can just delete the topic branch. This is a nice way to track individual bug fixes that I’ve worked on, while letting the maint-5.10 branch just track the core repository, with no local changes.

Regular readers will know that I’ve been thinking a lot about testing SQL result sets and how to how to name result testing functions, and various implementation issues. I am very happy to say that I’ve now committed the first three such test functions to the Git repository. They’ve been tested on 8.4 and 8.3. Here’s what I came up with.

I had a pretty good idea how to compare sets and how to compare ordered bags, but ordered sets and unordered bags of results escaped me. During two days of intense hacking and experimentation, I quickly wrote set_eq(), which performs a set comparison of the results of two queries, and obag_eq(), which performs a row-by-row comparison of the results of two queries. I then set to work on bag_eq(), which would do a set comparison but require the same number of duplicate rows between the two queries. set_eq() was easy because I just needed to create temporary tables of the two queries and then execute two EXCEPT queries against them to see where they differ, if at all. bag_eq() was getting kind of hairy, though, so I asked about it on the Freenode #postgresql channel, where depesz looked at my example and pointed out that EXCEPT ALL would do just want I needed.

Hot damn, all it took was the addition a single extra word to the same queries used by set_eq() and I was set. This made me very happy, and such well-thought-out features are the reason I love PostgreSQL. My main man depesz made my day.

But oset_eq(), which was to compare ordered sets of results was proving much harder. The relational operators that operate on sets don’t care about order, so I would have to write the code to care myself. And because dupes needed to be ignored, it got even harder. In fact, it proved just not worth the effort. The main reason I abandoned this test function, though, was not difficulties of implementation (which were significant), but ambiguity of interpretation. After all, if duplicates are allowed but ignored, how does one deal with their effect on order? For example, say that I have two queries that order people based on name. One query might order them like so:

select * from people order by name;
  name  | age 
--------+-----
 Damian |  19
 Larry  |  53
 Tom    |  35
 Tom    |  44
 Tom    |  35

Another run of the same query could give me a different order:

select * from people order by name;
  name  | age 
--------+-----
 Damian |  19
 Larry  |  53
 Tom    |  35
 Tom    |  35
 Tom    |  44

Because I ordered only on “name,” the database was free to sort records with the same name in an undefined way. Meaning that the rows could be in different orders. This is known, if I understand correctly, as a “Partially ordered set,” or “poset.” Which is all well and good, but from my point of view makes it damn near impossible to be able to do a row-by-row comparison and ignore dupes, because they could be in different orders!

So once I gave up on that, I was down to three functions instead of four, and only one depends on ordering. So I also dropped the idea of having the “o” in the function names. Instead, I changed obag_eq() to results_eq(), and now I think I have three much more descriptive names. To summarize, the functions are:

results_eq

Compares two result sets row by row, meaning that they must be in the same order and have the same number of duplicate rows in the same places.

set_eq

Compares result sets to ensure they have the same rows, without regard to order or duplicate rows.

bag_eq

Compares result sets without regard to order, but each must have the same duplicate rows.

I’m very happy with this, because I was able to give up on the stupid function names with the word “order” included or implicit in them. Plus, I have different names for functions that are similar, which is nicely in adherence to the principle of distinction. They all provide nice diagnostics on failure, as well, like this from results_eq():

# Failed test 146
#     Results differ beginning at row 3:
#         have: (1,Anna)
#         want: (22,Betty)

Or this from set_eq() or bag_eq()

# Failed test 146
#     Extra records:
#         (87,Jackson)
#         (1,Jacob)
#     Missing records:
#         (44,Anna)
#         (86,Angelina)

set_eq() and bag_eq() also offer up useful diagnostics when the data types of the rows vary:

# Failed test 147
#     Columns differ between queries:
#         have: (integer,text)
#         want: (text,integer)

results_eq() doesn’t have access to such data, though if I can find some tuits (got any to give me?), I’ll write a quick C function that can return an array of the data types in a record object.

Now, as for the issue of arguments, what I settled on is, like Epic, passing strings of SQL to these functions. However, unlike Epic, if you pass in a simple string with no spaces, or a double-quoted string, pgTAP assumes that it’s the name of a prepared statement. The documentation now recommends prepared statements, which you can use like this:

PREPARE my_test AS SELECT * FROM active_users() WHERE name LIKE ‘A%’;
PREPARE expect AS SELECT * FROM users WHERE active = $1 AND name LIKE $2;
SELECT results_eq('my_test', 'expect');

This allows you to keep your SQL written as SQL, keeping your test, um, SQLish. But in those cases where you have some really simple SQL, you can just use that, too:

SELECT set_eq(
    'SELECT * FROM active_users()',
    'SELECT * FROM users ORDER BY id'
);

This feels like a good compromise to me, allowing the best of both worlds: keeping things in pure SQL to avoid quoting ugliness in SQL strings, while letting users pass in SQL strings if they really want to.

It turns out that I wasn’t able to support cursors for set_eq() or bag_eq(), because they use the statements passed to them to create temporary tables and then compare the records in those temporary tables. But results_eq() uses cursors internally. And it turns out that there’s a data type for cursors, refcursor. So it was easy to add cursor support to results_eq() for those who want to use it:

DECLARE cwant CURSOR FOR SELECT * FROM active_users();
DECLARE chave CURSOR FOR SELECT * FROM users WHERE active ORDER BY name;
SELECT results_eq('cwant'::refcursor, 'chave'::refcursor );

Neat, huh? As I said, I’m very pleased with this approach overall. There are a few caveats, such as less strict comparisons in results_eq() on 8.3 and lower, and less useful diagnostics for data type differences in results_eq(), but overall, I think that the implementation is pretty good, and that these functions will be really useful.

So what do you think? Please clone the Git repository and take the functions for a test drive on 8.3 or 8.4. Let me know what you think!

In the meantime, before releasing a new version, I still plan to add:

• set_includes() - Set includes records in another set.
• set_excludes() - Set excludes records in another set.
• bag_includes() - Bag includes records in another bag.
• bag_excludes() - Bag excludes records in another bag.
• col_eq() - Single column result set equivalent to an array of values.
• row_eq() - Single row form a query equivalent to a record.
• rowtype_is() - The data type of the rows in a query is equivalent to an array of types.

Hopefully I can find some time to work on those next week. The only challenging one is row_eq(), so I may skip that one for now.

I’ve been thinking more about testing SQL result sets and how to name functions that do such testing, and I’ve started to come to some conclusions. Some of the constraints I’m looking at:

• Cursors are required for tests where the order of the results returned is important. It might be best for such functions to create the cursors themselves.

• For comparisons where order isn’t important, the results of each SQL statement must be inserted into a temporary table and then the table used for the comparisons. Otherwise, each statement would be executed twice, as is required to calculate symmetric difference. By executing each once and storing the results in a temporary table, we get around this issue (and indeed, this one of the cases that Epic’s global() function addresses).

• When order is not important, the most efficient way to compare result sets is with symmetric difference. However, said comparison is a set comparison, meaning that duplicate rows are ignored. So if set A has 3 rows and set B has four, but two of those four are identical, then sets A and B can still be equivalent.

I’m starting to think that I would have two basic result set testing functions, set_eq() and bag_eq(). The former would do a set comparison, while the latter would require that duplicate rows be present in both result sets. Unfortunately, that would mean that it would be difficult for set_eq to have a variation that tests ordered sets, as symmetric difference ignores relational ordering. And a simple bag_eq() function would require that the relations be ordered, as it would iterate over each row in each relation in turn and compare row to row. But as I pointed out to commenter “@dave0,” bags are not inherently ordered, so it would be imposing a requirement that’s not necessarily appropriate.

This is starting to drive me a bit nuts.

I think that there are ways to enforce an ordered comparison on a set and an unordered comparison on a bag, but both would be pretty inefficient. Maybe I should do it anyway, include the appropriate caveats in the documentation, and then improve when feasible in the future. In that case, what I’d be looking at is something like this:

set_eq( sql, sql )

Test for set equivalence of two SQL statements.

oset_eq( sql, sql )

Test for ordered set equivalence of two SQL statements.

bag_eq( sql, sql )

Test for row equivalence of two SQL statements.

obag_eq( sql, sql )

Test for ordered row equivalence of two SQL statements.

The preferred tests would be set_eq() and obag_eq(). If a single word is passed to any of these functions, it’s assumed to be a prepared statement. Cursors would be created internally for the functions that require ordered comparison. The non-ordered versions would create temporary tables to hold the values and then use those tables for the comparisons. bag_eq() would also construct cursors on the temporary tables to ensure that rows could be compared in the same order in which they were generated by the SQL statement.

Interface-wise, perhaps a boolean would be preferred to indicate whether or not to compare the rows in an ordered fashion? That would be:

set_eq( sql, sql, bool )

Test for set equivalence of two SQL statements. The sets must be in the same order if the boolean argument is true.

bag_eq( sql, sql, bool )

Test for row equivalence of two SQL statements. The bags must have their rows in the same order if the boolean argument is true.

I like that there are fewer functions this way, but is it harder to remember what the boolean is for? (It would not be required, and would default to false for both functions). Thoughts?

By the way, I would likely throw in a couple of other resultset-comparing functions:

set_includes( sql, sql )

Test that the set returned by the first statement includes the rows returned by the second statement.

set_excludes( sql, sql )

Test that the set returned by the first statement excludes the rows returned by the second statement.

bag_includes( sql, sql )

Test that the bag returned by the first statement includes the rows returned by the second statement, including duplicates.

bag_excludes( sql, sql )

Test that the bag returned by the first statement excludes the rows returned by the second statement, including duplicates.

Seem useful? Please leave a comment with your thoughts.

I’ve been working on a big Bricolage project recently, and one of the requirements is to parse an incoming NewsML feed, turn individual stories into Bricolage SOAP XML, and import them into Bricolage. I’m using the amazing—if hideously documented—XML::LibXML to do the parsing of the incoming NewsML, taking advantage of the power of XPath to pull out the bits I need. But then came the question: what should I use to generate the XML for Bricolage?

Based on feedback from various Tweeps, I tried a few different approaches: XML::LibXML, XML::Genx, and of course the venerable XML::Writer. In truth, they all suck for one reason: interface. As a test, I wanted to generate this dead simple XML:

<?xml version="1.0" encoding="utf-8"?>
<assets xmlns="http://bricolage.sourceforge.net/assets.xsd">
  <story id="1234" type="story">
    <name>Catch as Catch Can</name>
  </story>
</assets>

Just get a load of how you create this XML in XML::LibXML:

use XML::LibXML;

my $dom = XML::LibXML::Document->new( '1.0', 'utf-8' );
my $assets = $dom->createElementNS('http://bricolage.sourceforge.net/assets.xsd', 'assets');
$dom->addChild($assets);

my $story = $dom->createElement('story');
$assets->addChild($story);
$story->addChild( $dom->createAttribute( id => 1234));
$story->addChild( $dom->createAttribute( type => 'story'));
my $name = $dom->createElement('name');
$story->addChild($name);
$name->addChild($dom->createTextNode('Catch as Catch Can'));

say $dom->toString;

Does anyone actually think that this is intuitive? Okay, if you’re used to dealing with the XHTML DOM in JavaScript it’s at least familiar, but that’s hardly an endorsement. XML::Genx isn’t much better:

use XML::Genx::Simple;

my $w = XML::Genx::Simple->new;
$w->StartDocString;

$w->StartElementLiteral( $w->DeclareNamespace( 'http://bricolage.sourceforge.net/assets.xsd', ''), 'assets' );
$w->StartElementLiteral( 'story' );
$w->AddAttributeLiteral( id => 1234);
$w->AddAttributeLiteral( type => 'story');
$w->Element( 'name' => 'Catch as Catch Can' );
$w->EndElement;
$w->EndElement;
$w->EndDocument;

say $w->GetDocString;

It’s not like messing with the DOM, but it’s essentially the same: Use a bunch of camelCase methods to declare each thing one-at-a-time. And you have to count the number of open elements you have yourself, to know how many times to call EndElement() to close elements. Can’t we get the computer to do this for us?

Feeling a bit frustrated, I went back to XML::Writer, which is what Bricolage uses internally to generate the XML exported by its SOAP interface. It looks like this:

use XML::Writer;

my $output = '';
my $writer = XML::Writer->new(
    OUTPUT=> \$output,
    ENCODING => 'utf8',
);

$writer->xmlDecl('UTF-8');
#$writer->startTag(['http://bricolage.sourceforge.net/assets.xsd', 'stories']);
$writer->startTag('assets', xmlns => 'http://bricolage.sourceforge.net/assets.xsd');
$writer->startTag('story', id => 1234, type => 'story');
$writer->dataElement(name => 'Catch as Catch Can');
$writer->endTag('story');

$writer->endTag('assets');

say $output;

That’s a bit better, in that you can specify the attributes and value of an element all in one method call. I still have to count opened elements and figure out where to close them, though. The thing that’s missing, as with the other approaches, is an API that reflects the hierarchical nature of XML itself. I’m outputting a tree-like document; why should the API be so hideously object-oriented and flat?

With this insight, I remembered Jesse Vincent’s Template::Declare. It bills itself as a templating library, but really it provides an interface for declaratively and hierarchically generating XML. After a bit of hacking I came up with this:

package Template::Declare::TagSet::Bricolage;
BEGIN { $INC{'Template/Declare/TagSet/Bricolage.pm'} = __FILE__; }
use base 'Template::Declare::TagSet';

sub get_tag_list {
    return [qw( assets story name )];
}

package My::Template;
use Template::Declare::Tags 'Bricolage';
use base 'Template::Declare';

template bricolage => sub {
    xml_decl { 'xml', version => '1.0', encoding => 'utf-8' };
    assets {
        xmlns is 'http://bricolage.sourceforge.net/assets.xsd';
        story {
            attr { id => 1234, type => 'story' };
            name { 'Catch as Catch Can' }
        };
    };
};

package main;
use Template::Declare;
Template::Declare->init( roots => ['My::Template']);
say Template::Declare->show('bricolage');

Okay, to be fair I had to do a lot more work to set things up. But once I did, the core of the XML generation, there in the bricolage template, is quite simple and straight-forward. Furthermore, thanks to the hierarchical nature of Template::Declare, the tree structure of the resulting XML is apparent in the code. And it’s so concise!

Armed with this information, I whipped up a new module for CPAN: Template::Declare::Bricolage. This module subclasses Template::Declare to provide a dead-simple interface for generating XML for the Bricolage SOAP interface. Using this module to generate the same XML is quite simple:

use Template::Declare::Bricolage;

say bricolage {
    story {
        attr { id => 1234, type => 'story' };
        name { 'Catch as Catch Can' }
    };
};

Yeah. Really. That’s it. Because the Bricolage SOAP interface requires that all XML documents have the top-level <assets> tag, I just had the bricolage function handle that, as well as actually executing the template and returning the XML. More complex XML is just a simple, assuming that you use nice indentation to format your code. Here’s the code to generate XML for a Bricolage workflow object:

use Template::Declare::Bricolage;

say bricolage {
    workflow {
        attr        { id => 1027     };
        name        { 'Blogs'        }
        description { 'Blog Entries' }
        site        { 'Main Site'    }
        type        { 'Story'        }
        active      { 1              }
        desks  {
            desk { attr { start   => 1 }; 'Blog Edit'    }
            desk { attr { publish => 1 }; 'Blog Publish' }
        }
    }
};

Simple, huh? So the next time you need to generate XML, have a look at Template::Declare. It may not be the fastest XML generator around, but if you have a well-defined list of elements you need, it’s certainly the nicest to use.

Oh, and Bricolage users? Just make use of use Template::Declare::Bricolage to deaden the pain.

I’ve been thinking more since I posted about testing SQL result sets, and I think I’ve settled on two sets of functions for pgTAP: one that tests two SQL queries (though you will be encouraged to use a prepared statement), and one to test two cursors. I’m thinking of naming them:

• query_gets()
• cursor_gets()

I had been planning on *_returns() or *_yields(), but they didn’t feel right. “Returns” implies that I would be passing a query and a data structure (to me at least), and while I want to support that, too, it’s not what I was looking for right now. “Yield,” on the other hand, is more related to set-returning functions in my mind (even if PL/pgSQL doesn’t use that term). Anyway, I like the use of “gets” because it’s short and pretty unambiguous.

These function will compare query results as unordered sets, but I want variants that test ordered sets, as well. I’ve been struggling to come up with a decent name for these variants, but not liking any very well. The obvious ones are:

• ordered_query_gets()
• ordered_cursor_gets()

And:

• sorted_query_gets()
• sorted_cursor_gets()

But these are kind of long for functions that will be, I believe, used frequently. I could just add a character to get the same idea, in the spirit of sprintf:

• oquery_gets()
• ocursor_gets()

Or:

• squery_gets()
• scursor_gets()

I think that these are okay, but might be somewhat confusing. I think that the “s” variant probably won’t fly, since for sprintf and friends, the “s” stands for “string.” So I’m leaning towards the “o” variants.

But I’m throwing it out there for the masses to make suggestions: Got any ideas for better function names? Are there some relational terms for ordered sets, for example, that might make more sense? What do you think?

As a side note, I’m also considering:

• col_is() to compare the result of a single column query to an array or other query. This would need an ordered variant, as well.
• row_is(), although I have no idea how I’d be able to support passing a row expression to a function, since PostgreSQL doesn’t allow RECORDs to be passed to functions.

There’s an old saying, “Whoever doesn’t understand X is doomed to reinvent it.”X can stand for any number of things. The other day, I was pointing out that such is the case for ORM developers. Take ActiveRecord, for example. As I demonstrated in a 2007 Presentation, because ActiveRecord doesn’t support simple things like aggregates or querying against functions or changing how objects are identified, you have to fall back on using its find_by_sql() method to actually run the SQL, or using fuck typing to force ActiveRecord to do what you want. There are only two ways to get around this: Abandon the ORM and just use SQL, or keep improving the ORM until it has, in effect, reinvented SQL. Which would you choose?

I was thinking about this as I was hacking on a Drupal installation for a client. The design spec called for the comment form to be styled in a very specific way, with image submit buttons. Drupal has this baroque interface for building forms: essentially an array of arrays. Each element of the array is a form element, unless it’s markup. Or something. I can’t really make heads or tails of it. What’s important is that there are a limited number of form elements you can create, and as of Drupal 5, image isn’t fucking one of them!.

Now, as a software developer, I can understand this. I sometimes overlook a feature when implementing some code. But the trouble is: why have some bizarre data structure to representa subset of HTML when you have something that already works: it’s called HTML. Drupal, it seems, is doomed to reinvent HTML.

So just as I have often had to use find_by_sql() as the fallback to get ActiveRecord to fetch the data I want, as opposed to what it thinks I want, I had to fallback on the Drupal form data structure’s ability to accept embedded HTML like so:

$form['submit_stuff'] = array(
  '#weight' => 20,
  '#type'   => 'markup',
  '#value'  => '<div class="form-submits">’
             . '<label></label><p class="message">(Maximum 3000 characters)</p>'
             . '<div class="btns">'
             . '<input type="image" value="Preview comment" name="op" src="preview.png" />'
             . '<img width="1" height="23" src="divider.png" />'
             . '<input type="image" value="Post comment" name="op" src="post.png" />'
             . '</div></div>',
);

Dear god, why? I understand that you can create images using an array in Drupal 6, but I fail to understand why it was ever a problem. Just give me a templating environment where I can write the fucking HTML myself. Actually, Drupal already has one, it’s called PHP!. Please don’t make me deal with this weird hierarchy of arrrays, it’s just a bad reimplementation of a subset of HTML.

I expect that there actually is some way to get what I want, even in Drupal 5, as I’m doing some templating for comments and pages and whatnot. But that should be the default IMHO. The weird combining of code and markup into this hydra-headed data structure (and don’t even get me started on the need for the #weight key to get things where I want them) is just so unnecessary.

In short, if it ain’t broke, don’t reinvent it!

</rant>

A few days ago, I noticed a bug crawling across the screen of my MacBook Pro as I was working. I tried to shoo it away, but it wouldn’t budge—because it was inside the display! It soon stopped moving, and is right smack in the middle of the screen, where I try to work. So I have an Apple Genius appointment today to see about extracting it. Here’s hoping they have the tools to remove it at the store (an aardvark, perhaps?), else I’ll have to part with it for a week. Grrr.

For old time’s sake, I also dug up and re-posted my cracked PowerBook display entry from way back in 2095. Good times.