Just a Theory

I’m working on a project where I aggregate entries from a slew of feeds into a single feed. The output feed will be a valid Atom feed, and of course I want to make sure that I maintain all the appropriate metadata for each entry I collect. The <source> element seems to be exactly what I need:

If an entry is copied from one feed into another feed, then the source feed’s metadata (all child elements of feed other than the entry elements) should be preserved if the source feed contains any of the child elements author, contributor, rights, or category and those child elements are not present in the source entry.

<source>
  <id>http://example.org/</id>
  <title>Fourty-Two</title>
  <updated>2003-12-13T18:30:02Z</updated>
  <rights>© 2005 Example, Inc.</rights>
</source>

That’s perfect: It allows me to keep the title, link, rights, and icon of the originating blog associated with each entry.

Except, maybe it’s the database expert in me, but I’d like to be able to have it be more normalized. My feed might have 1000 entries in it from 100 sources. Why would I want to dupe that information for every single entry from a given source? Is there now better way to do this, say to have the source data once, and to reference the source ID only for each entry? That would make for a much smaller feed, I expect, and a lot less duplication.

Is there any way to do this in an Atom feed?

If you’re a regular visitor to my blog (and who could blame you?), you likely have noticed a few changes recently. In addition to adding the sociable links a couple days ago, I’ve also been adding bits of embedded JavaScript in the right column displaying my three most recent Tweets and my three most recent Delicious bookmarks. These work reasonably well: I just embed <script> tags with the appropriate stuff, then style the HTML that they deliver.

Tonight I was talking to Skud about embedding like this. It turns out that some folks were getting a big blank area when they viewed a blog entry on her site in RSS readers and the like, they sometimes just saw a big blank area where there was supposed to be a list of books. She was looking for examples of sites that provided HTML snippets that people could cut-n-paste into their blog entries, so that they can avoid this problem, or use it in places that disallow JavaScript embedding, such as LiveJournal. I had no examples for her, but it suddenly occurred to me: Why not embed a link to an HTML URL that serves a snippet of HTML, rather than a bit of JavaScript that uses the document object to write HTML?

A quick Googling and I found a page a great article about the <object> element. It was intended as a general replacement for the <img> and <applet> elements, although tht really hasn’t happened. But what you can do is embed HTML with it. Here’s a quick example:

Hopefully you can see the embedded HTML above. I’ve styled it with a light blue background and dark blue dotted border, so it stands out. That styling is in the <object> tag, BTW, not in the HTML loaded from the snippit. I’m sure I could figure out how to add <param> tags that would tell it to include various styles, too, since it appears that CSS I have in this page has no effect on the content of the object (I have some CSS to make the <code> tag have a green background, but for me at least, it has no effect.

So why isn’t this more common? It seems to work well in a lot of browsers. Would you use it? What are the downsides?

I notice a number of places recently where some blogs had a nice array of small icons to make it easy for readers to add particular entries to their favorite social bookmarking sites. The example I noticed most recently was on Simple Mom. After a bit of digging, I found the Sociable plugin for WordPress. It had just the format I was looking for.

So today I ported it to Blosxom. You can get it yourself here. You’ll also need to download the Sociable WordPress plugin so that you can get all the necessary images and styling. Read the docs for how to use it; it’s pretty simple, and supports a lot of social bookmarking sites, and even good ’ole “Email” and “Print” links.

And of course, also starting today, you can see the links right here on my site. So, yeah, go ahead and link me up!

Lately I’ve been fiddling a bit with Markdown, John Gruber’s minimalist plain text markup syntax. I’ve become more and more attracted to Markdown after I’ve had to spend some time using Trac and, to a lesser degree, Twiki and MediaWiki. The plain-text markup syntax in these projects is…how shall I put this?…gawdawful. Why do I hate these wiki syntaxes? Becaus they’re unnatural. Maybe it’s just because I’m most familiar with it, but Trac’s syntax is just completely random and inconsistent. Trying to get anything other than simple paragraphs formatted just right is just a giant pain in the ass. Just try have multiple paragraphs in a hierarchical bulleted list and you’ll see what I mean. If I wanted to worry about space this much I’d hack Python! I mean, seriously, there’s a reason I write my blog entries in pure HTML. It’s not so user-friendly, but at least I know exactly how something will be formatted when I’m done.

But Markdown is different. It’s syntax is almost exactly like what I’ve been using in lain-text email messages since the mid-1990s. It’s humane in a way that Textile only approaches in its inline markup (as long as you don’t use attributes, of course). There are a few oddities, such as the definition list syntax used by PHP Markdown Extra and MultiMarkdown is a bit unnatural. But overall, it’s quite close to what I type anyway. I’ve been writing the pgTAP documentation in Markdown, using Discount to generate the HTML you see on the Web site (plus my own custom hack to create the table of contents), and it’s just a thrill that it’s so easy to maintain: I can easily read and edit the README file like any other text file, and then generate the HTML for the Web site with a simple make target. It has been such a great experience that I’m tempted to stop writing documentation in POD!

So in my next app, I’ll likely be making use of MultiMarkdown for the end-user management of content. It has nearly everything I want, formatting-wise, and I can likely get used to the few cases where its syntax seems a bit weird to me. Plus, I can then use the generated HTML to output PDFs and other formats from the same document. I expect it to be a dream to work with. (Oh, and thanks to Aristotle Pagaltzis for patiently putting up with my questions about markdown in private email messages; they’ll help keep me from saying anything too embarrassing on the Markdown mail list!)

In the meantime, I’ve modified the comment system on this blog to support Markdown. You can still use HTML in comments, same as always, as Markdown passes HTML through unmolested. But few of you ever did that, and I was always adding HTML tags to the comments. Now maybe I won’t have to: Markdown is so easy and natural to use, that the vast majority of commenters will just leave paragraphs and they’ll look beautiful.

At any rate, you now have one less reason not to leave a comment!

Some of you no doubt noticed that this site was down for several days, starting last Friday and lasting until yesterday. Sorry about that. I had a hard disk failure of some kind on the ca. 1999 OptiPlex I was using. I’ve had a newer box (ca. 2005) to move to for a while, but lacked the tuits. With this change, I was forced to make the switch. Fortunately, a Debian install CD let me login to the OptiPlex and access all my files, so I was able to recover everything. I even managed to keep the file modification times the same, so feeds won’t show everything as unread (which I’ve seen many times when other bloggers I’ve known have switched providers or recovered from some catastrophe). Unless you tried to hit this site over the weekend or on Monday or Tuesday, you should notice no changes at all (except speed, the new box is a lot faster!).

Naturally, I took advantage of this opportunity to get my blog configuration into SVN via my Capistrano deployment system. Hell, none of this stuff was even backed up before (although I did back up all my blog entries about a week before this happened—but not comments, yow!). The new box is now properly backing itself up and backing up the Kineticode server, and I can make changes to Blosxom and configure and reboot the blog from my MBP. Yay! No more remote editing.

I’ve also upgraded my “server room,” moving out the gigantic 17" CRT and putting in the 17" flat panel screen I’ve had floating around. I also plugged a USB keyboard into my KVM, so I no longer have to move keyboards around when I switch between the Linux server and the G3 Mac server. Of course, now that I have large disks and Time Machine running on all the other boxes in the house, we don’t use the G3 anymore. So I think we’ll be donating it soon.

Another change I’ve made was to stop doing my own log analysis (the command-line tools are such a PITA) and switched to Google Analytics and FeedBurner for tracking visitors to the blog and its feeds. I’ve still got the old log files around, so I can see how things have changed since before the switch to outside analytics providers, but I’ll probably just create a report from them and then ignore them from now on. Too much work to track that stuff.

In the future, I’d like to switch from Blosxom to some other tool. Maybe Movable Type, now that it’s open source. It’s pretty well-regarded and written in Perl, so I could hack it pretty easily. What I should do is avoid writing my own Blog engine. Right? Right?!. In the meantime, I have other priorities, so I’ll be sticking to Bloxsom for a while.

For the last several years, I’ve run a Courier-IMAP mail server for all of the mail for this site, Kineticode, Strongrrl and other domains. We mainly used Mail.app on Mac OS X to communicate with the server, and it worked really well. Today, Julie has over 3 GB of mail data, and I have around 1.5 GB, all managed via IMAP.

Recently, I decided it was time to move the mail elsewhere. I’ve been meaning to do it for a while, primarily because the server I was using is now used for the Bricolage project, and because I never set up any spam filtering. Julie was suddenly getting 100s of spam messages in her inbox. (It really didn’t help that she was still using Panther.) So on the advice of a good friend who had been evaluating various mail services—and who for now shall go nameless and therefor blameless—I moved all of our mail to FuseMail.

At first this seamed like a pretty good solution. Our spam rates went way down, I could set up unlimited mail lists, aliases, and forwards, and there was a migration tool that automated moving all of our existing mail from the old IMAP server to the new one. There were some glitches with the migration tool, but in the end all of our mail was moved and in tact.

But that’s when I started to notice the issues. To summarize:

• Mail put into the Sent Items folder by Mail.app was marked as unread. This didn’t happen on the old server, and apparently has something to so with how FuseMail names the sent folder: Sent Items rather than Sent Messages.
• Mail.app is syncing constantly. Even once it had successfully synced the all of our email in all of our IMAP folders (which took days, it is syncing all the time, to the extent that I am sometimes waiting for up to a minute to read a mail when I double-click it, because there are all these other threads doing stuff and taking up all the resources. It can take several minutes for mail I’m sending to be sent (though that might be a delay in Mail.app copying the message to the Sent Items folder rather than the actual sending).
• Deleting mail takes forever! This is probably the same issue as the syncing problem, but when I delete 1000s of messages from my Junk mail folder, it runs forever, and all other activities are delayed eve further. It turns out to be much more efficient to empty the Junk and Deleted Items folders using the webmail interface. And even then, Mail.app can take a while to delete locally-cached items from the folder when it syncs.
• Suddenly, Julie is getting a lot less spam. She went from several hundred messages showing up in her Junk mailbox a few days ago to just five on Friday and two yesterday—one of which was a false positive). As she had been expecting a message from someone that she never got, this naturally made her very suspicious. Where is all the spam? Is she getting all of her mail?
• Since FuseMail uses a mailbox named Sent Items instead of the traditional Sent Messages for all sent mail, I asked if they could move the 1.8 GB of messages from Julie’s Sent Messages to their Sent Items, since Mail.app would just choke on such a task. Though my request was escalated to the FuseMail developers, the answer came back no. Which I guess means that they’re not using Maildir, because in that case it would be a cinch, n’est pas?
• Backups are not really feasible. Of course FuseMail has its own backup regimen, but if I ever want to move elsewhere or deal with some sort of catastrophic failure, I want my own backups. There is no rsync service available for this (remember: no maildir), so I have to use the IMAP interface. I’ve been trying for the past two weeks to get Offline IMAP to back up all of Julie’s and my mail, but it keeps choking. It gets a little further every time I run it; eventually it will get it all. But this only allows me to backup those accounts for which I happen to have a password. I have accounts set up for a few other users, but don’t have access to their passwords, so I can’t back them up. This does not make for very good support for corporate backup and retention policies.
• Mail forwarded by FuseMail has its Return-Path header modified. This made RT break until I hacked it to ignore that header (which is its by-default preferred header for identifying senders.

So I’m pretty fed up. It took me a week to get all of our mail on FuseMail, and now I’m looking at moving it off again (once OfflineIMAP finishes a full sync). Grr. I’m considering finding a virtual host somewhere and setting up my own IMAP server again, but then I have the spam problem again. So then I could use a forwarding service like Pobox, or I can set up my own spam filtering (something I had hoped never to get into managing myself). My old IMAP server required very little maintenance, which was nice, but then the span filtering stuff always seemed daunting. Don’t you have to update things all the time?a

But before I go off and do something else, and unlike before I moved to FuseMail, I wanted to get an idea what other folks are doing? Do you use IMAP? Do you use it to manage a shitload (read: Gigabytes) of mail? Do you get very little spam and still get all of your valid mail? Are IMAP folder maintenance actions fast for you (in Mail.app in particular)? Are you paying a not-unreasonable amount of money for your setup? If you answered yes to all of these questions, please, for the love of all that is good in this world, tell me how you do it. I’m looking for something that I don’t have to work very hard to maintain (hence my original attempt to have some company that specializes in this stuff do it), but I’ll do what I have to to make this thing right. So how do you make it right? And if I have to run my own server, where should I host it that won’t cost me an arm and a leg?

Thanks for your help!

I’ve just added a new feed: teasers only. It makes things a log shorter for those who just want to get a teaser for each blog entry, rather than complete entries, such as Planet Perl and Planet PostgreSQL.

Any questions or problems? Leave a comment. Thanks!