Saturday, January 29, 2011

IMAP server that saves space on identical attachments

Looking for IMAP server that can detect duplicate attachments across mailboxes and would not waste space on them.

So, if 4 local mailboxes receive the same attachment I would like it to be stored only once on the file system.

Open source preferred.

Also, had troubles coming up with concise description for the above, and hence researching online resources, so suggestions for good keywords are also very welcome.

Best

  • Yeah, Dovecots new mailbox format called dbox can do this.

    Single instance attachment storage. If multiple mailboxes/users have the same attachment, it's stored only once in disk.

    Unreason : Great... err wait, the whole quote is 'dbox storage is extensible, so in future there will be other extensions. Some things that are planned: Single instance attachment storage.' Also, according to http://blog.dovecot.org/2010/07/single-instance-attachment-storage.html I think it is only in testing stage (pls correct me if I am wrong). +1 for help with keywords...
    weeheavy : You're right with the 'single instance' point. This is only planned. 'testing stage' is not the right word, it seems more stable than that.
    From weeheavy
  • On which operating system do you want to run it?

    Exchange does exactly that (and not only to attachments, but to whole messages).

    But implementing Exchange only to have an IMAP server would be quite overkill...

    Helvick : FYI Exchange single instancing only does this within mailstores and even that has been dropped for Exchange 2010.
    Massimo : Yes, I know this is applied at the database level (which should anyway be quite obvious when thinking of it); but thanks for pointing it out.
    From Massimo
  • Cyrus IMAP can do this among many other things. But be warned, Cyrus is not as straight-forward to install as Dovecot.

    Unreason : Thanks. What's the terminology for it in Cyrus? +1
    Phil P : singleinstancestore. It's per email, not per attachment. It's turned on by default, but you need to be using LMTP delivery as it only does this for each recipient on a single instance of receiving a message, it does not do correlation to previously-received emails.
  • Bit of a different answer, but what OS? Do you have the option of something like ZFS (which I believe does dedupe)?

    From Hutch

0 comments:

Post a Comment