home :: computers :: internet :: mail :: need imap solution

Need Suggestions for IMAP Solution and Migration

For the last several years, I’ve run a Courier-IMAP mail server for all of the mail for this site, Kineticode, Strongrrl and other domains. We mainly used Mail.app on Mac OS X to communicate with the server, and it worked really well. Today, Julie has over 3 GB of mail data, and I have around 1.5 GB, all managed via IMAP.

Recently, I decided it was time to move the mail elsewhere. I’ve been meaning to do it for a while, primarily because the server I was using is now used for the Bricolage project, and because I never set up any spam filtering. Julie was suddenly getting 100s of spam messages in her inbox. (It really didn’t help that she was still using Panther.) So on the advice of a good friend who had been evaluating various mail services—and who for now shall go nameless and therefor blameless—I moved all of our mail to FuseMail.

At first this seamed like a pretty good solution. Our spam rates went way down, I could set up unlimited mail lists, aliases, and forwards, and there was a migration tool that automated moving all of our existing mail from the old IMAP server to the new one. There were some glitches with the migration tool, but in the end all of our mail was moved and in tact.

But that’s when I started to notice the issues. To summarize:

  • Mail put into the Sent Items folder by Mail.app was marked as unread. This didn’t happen on the old server, and apparently has something to so with how FuseMail names the sent folder: Sent Items rather than Sent Messages.
  • Mail.app is syncing constantly. Even once it had successfully synced the all of our email in all of our IMAP folders (which took days, it is syncing all the time, to the extent that I am sometimes waiting for up to a minute to read a mail when I double-click it, because there are all these other threads doing stuff and taking up all the resources. It can take several minutes for mail I’m sending to be sent (though that might be a delay in Mail.app copying the message to the Sent Items folder rather than the actual sending).
  • Deleting mail takes forever! This is probably the same issue as the syncing problem, but when I delete 1000s of messages from my Junk mail folder, it runs forever, and all other activities are delayed eve further. It turns out to be much more efficient to empty the Junk and Deleted Items folders using the webmail interface. And even then, Mail.app can take a while to delete locally-cached items from the folder when it syncs.
  • Suddenly, Julie is getting a lot less spam. She went from several hundred messages showing up in her Junk mailbox a few days ago to just five on Friday and two yesterday—one of which was a false positive). As she had been expecting a message from someone that she never got, this naturally made her very suspicious. Where is all the spam? Is she getting all of her mail?
  • Since FuseMail uses a mailbox named Sent Items instead of the traditional Sent Messages for all sent mail, I asked if they could move the 1.8 GB of messages from Julie’s Sent Messages to their Sent Items, since Mail.app would just choke on such a task. Though my request was escalated to the FuseMail developers, the answer came back no. Which I guess means that they’re not using Maildir, because in that case it would be a cinch, n’est pas?
  • Backups are not really feasible. Of course FuseMail has its own backup regimen, but if I ever want to move elsewhere or deal with some sort of catastrophic failure, I want my own backups. There is no rsync service available for this (remember: no maildir), so I have to use the IMAP interface. I’ve been trying for the past two weeks to get Offline IMAP to back up all of Julie’s and my mail, but it keeps choking. It gets a little further every time I run it; eventually it will get it all. But this only allows me to backup those accounts for which I happen to have a password. I have accounts set up for a few other users, but don’t have access to their passwords, so I can’t back them up. This does not make for very good support for corporate backup and retention policies.
  • Mail forwarded by FuseMail has its Return-Path header modified. This made RT break until I hacked it to ignore that header (which is its by-default preferred header for identifying senders.

So I’m pretty fed up. It took me a week to get all of our mail on FuseMail, and now I’m looking at moving it off again (once OfflineIMAP finishes a full sync). Grr. I’m considering finding a virtual host somewhere and setting up my own IMAP server again, but then I have the spam problem again. So then I could use a forwarding service like Pobox, or I can set up my own spam filtering (something I had hoped never to get into managing myself). My old IMAP server required very little maintenance, which was nice, but then the span filtering stuff always seemed daunting. Don’t you have to update things all the time?a

But before I go off and do something else, and unlike before I moved to FuseMail, I wanted to get an idea what other folks are doing? Do you use IMAP? Do you use it to manage a shitload (read: Gigabytes) of mail? Do you get very little spam and still get all of your valid mail? Are IMAP folder maintenance actions fast for you (in Mail.app in particular)? Are you paying a not-unreasonable amount of money for your setup? If you answered yes to all of these questions, please, for the love of all that is good in this world, tell me how you do it. I’m looking for something that I don’t have to work very hard to maintain (hence my original attempt to have some company that specializes in this stuff do it), but I’ll do what I have to to make this thing right. So how do you make it right? And if I have to run my own server, where should I host it that won’t cost me an arm and a leg?

Thanks for your help!

Tags:

Comments & Trackbacks

Ask Bjørn Hansen wrote:

It sounds like the Fusemail IMAP server doesn't play nicely with Mail.app.

If you want a hosted solution the Fastmail.fm people should be more competent.

For the spam stuff: A few DNSBLs and SpamAssassin are pretty effective. Add SpamBayes or similar and you'll be ~spamfree.

I only use SpamBayes for accounts that get spam after the DNSBL and SpamAssassin filtering.

I have SpamBayes setup to put main into inbox, unsure or spam as appropriate and then I have a train-spam and a train-ham folder for each user that a script makes spambayes train on and clear out once an hour.

- ask

Ricardo Signes wrote:

Pobox FTW

Okay, I'm biased. I work at Pobox. That said, I think it's a really good service and a really unbeatable value. Pobox does extremely effective and highly-customizable spam protection, and you won't lose mail because we can send you reports of messages that were rejected or trapped. I get hourly reports of trapped spam (and ignore bounced spam, because I use the default settings: only the very reliable filters bounce) and it's almost all about wangs and wristwatches. When I do see that I've subscribed to a list that runs on someone's home server with no PTR, I just release the first few messages and whitelist.

Pobox lets you use your own domain, so you don't get locked into us. Who'd want to leave? Well, nobody, but that said, I never sign up for a service that establishes my identity or hosts my data unless I have an exit strategy.

Pobox also offers IMAP (and POP) and webmail (both SquirrelMail and RoundCube) if you don't want to just have your mail forwarded. Currently, we're using Courier, although it would be foolish to believe that that could never change.

I use MyPobox (Pobox hosts my domain's MXes), and I've invited a few people to have addresses there, which is nice an easy. All my mail is forwarded to my own VPS running its own mailtools, but also to a Pobox-stored Mailstore, which Pobox sorts into folders by date. That way, if my experimenting with mail code accidentally blows up all my own mail, Pobox has kept a backup for me.

You can get a 30 day trial with Pobox, so it might be worth your time to just check it out. You can also find a few of Us on #pobox on MAGnet, if you wanted to harass me with questions.

Ask Bjørn Hansen wrote:

Oh - for size: I have ~3-4GB across 7 accounts in such a setup (2GB and hundreds of thousands of mails in the largest). When I start Mail.app after not syncing for a few days it's pretty busy for a while loading a bazillion mailboxes but when it's done it's, uh, done.

Doing a sync all accounts/folders does take a couple of minutes.

Theory wrote:

Re: Pobox FTW

Hey rjbs,

Thanks (as usual) for the comment. I didn't know you worked for Pobox. A couple of questions:

  • If Pobox hosted my mail, would I be able to use rsync or something to back it up?
  • Is there any way to get more than 2GB of storage?
  • Is there some way to migrate an existing IMAP directory structure to the Pobox IMAP server?

Those questions aside, the forwarding service does indeed sound nice. I'd still need somewhere to forward it to, though.

—Theory

James Duncan Davidson wrote:

Fusemail seemed pretty cool and then, well, not so cool. I'm playing with Pobox right now on a trial and so far it seems pretty cool. In my testing so far, it's a nice sane IMAP environment without any bogosities. I like the idea of getting the spam reports via email as well. Syncing around mail in a spam folder has gotten out of hand for me.

I've got a few more questions to work through, such as setting up a group alias with more than 5 people on it, but so far I'm liking it.

Pedro Melo wrote:

Still using Dovecot

Hi,

between me and my wife and her business, we have about 20Gb of email on IMAP, using Dovecot on a small server. For spam, we use two things: postini service (which is great) and if you use Macs, SpamSieve.

I love SpamSieve, and I could not recommend any other solution if you are using Mail.app. The biggest advantage is the integration with Mail.app recent recipients and Address Book. False positives, which are my biggest concern are virtually non-existing.

Best regards,

Ricardo Signes wrote:

Pobox IMAP Answers

As I said, I store my mail for primary consumption on a machine that isn't Pobox -- so I used to rsync it regularly for backup. Now, I don't: I use OfflineIMAP, a python script that it seemed you know about. It's not a filesystem-level backup, but it syncs to basically equivalent Maildirs. (I have a journal entry about the joy of OfflineIMAP, but it's mostly fluff: the point is that you can point mutt against your local replica, do stuff, then resync in both directions without losing mail.)

So, the answer to your question about rsync is no: you can't rsync your mail, because (among other things) we don't want to guarantee an unchanging storage format -- but you won't need to, because OfflineIMAP does it pretty well, with loads of extra features.

You can get extra storage. It comes up so rarely, and is such a non-technical issue, that I don't know the details. It's maybe $5/gb/year? Anyway, it's available.

Finally, migration is easy: OfflineIMAP will sync from one IMAP store to another! I haven't actually *done* this, but I've seen it mentioned as a use case, and it's obvious from the configuration that it should be simple.

Theory wrote:

Ricardo,

Thanks for your reply. The only issue I have with OfflineIMAP (other than that it seems to fail without finishing the sync every time I run it, but maybe it's my DSL that's the problem) is that I can only back up accounts for which I have the username and password. It doesn't work if I have accounts for other users. IOW, there is no comprehensive backup solution for a company that wants to host its mail on FuseMail or Pobox. The same issue applies to migration.

Good to know about the extra disk space, though. I had assumed that that was the case, I just didn't see it on the Pobox site anywhere.

—Theory

Theory wrote:

Ask,

Yes, the performance you get sounds like about what I was getting with my old server. So yeah, FuseMail must not be Mail.app-friendly.

—Theory

blameless wrote:

Need Suggestions for IMAP Solution and Migration

Bookmarked your post over at Blog Bookmarker.com!

Ask Bjørn Hansen wrote:

I didn't realize that pobox does IMAP now. They're clearly in the "clueful" category, too. :-)

-ask

Dan Langille wrote:

DoveCot + imapcopy

David:

I recently moved to IMAP from POP. I used a tool called imapcopy. In short, it connects to both IMAP servers and copies everything from point A to point B. It worked well.

I have a 1.3GB of mail in ~/Maildir.

I primarily use Mail.app and Thunderbird. Sometimes I use Squirrel Mail (web interface).

For spam stuff, I'm using greylisting (via pf and spamd, tools from OpenBSD used on FreeBSD) and amavisd to interface to spamassasin.

Cost: my time. I run the above on my own servers.

Antonio wrote:

Here's a Solution

Have you heard of BlueTie? BlueTie offers a web-based suite of email and collaboration (allowing you to share calendars, contacts, and files).

BlueTie utilizes SpamAssassin and ClamAV for it's virus and spam protection and is closely rated to the level of Postini.

BlueTie offers 10GB email boxes per user, supports POP, IMAP, and SMTP, and also mobile WAP 2.0. You should check them out.

Cost: $5 per month for everything Phone: 1-800-BLUETIE (ext. 1024)

Dan Langille wrote:

Antonio: lemme guess.. you work for Bluetie? :)

Powered by KinoSearch

Syndicate

Technorati

Cork’d Journal