A drop in the ocean

Evolution with improved IMAP support – IMAPX

with 45 comments

I had been working for a while on improving the imap support in evolution. Took the imapx provider which was written by Michael Zuchhi (aka Notzed) and getting it ready for replacing the existing IMAP provider. To summarize what is present,

* Fetch messages in batches
* Fetch messages with large attachments in multiple pieces
* IDLE support (push mail for imap)
* All operations on messages
* Cancel operations

A small screen cast demo of the same :)

The items which am working on are,

* Store operations (folder delete/create etc.)
* Preference options
* Connection manager to allow concurrent folder access (configurable one)
* Smart background message caching
* Mutliple namespace support

This design follows most of the aspects which are recommended for an IMAP client. Since we now parse all the unsolicited (aka untagged) responses from the server, syncing the changes is just instant when comparing with the existing imap provider even without the IDLE support. Users who live in inbox are going to love this :)

I would be working agressively to provide a wonderful imap experience with evolution for all evo fans :)  With akhils’s help we are setting up a server with multiple namespaces to test shared, public folders. If anyone can provide me an account with servers configured with multiple namespaces, it would be of great help.

If you need anything more get in touch :) Also send ur kudos to Michael Zuchhi!!

About these ads

Written by chenthill

January 11, 2010 at 11:33 pm

Posted in evolution, gnome, suse

45 Responses

Subscribe to comments with RSS.

  1. Wonderful. Add to cart!

    Pete

    January 12, 2010 at 12:27 am

  2. Great to see that Evolution IMAP gets some attention. Please check that the code works correctly on suspend/resume as well. If you close the lid on your laptop so that it suspends for a few hours and then open it, all the tcp-sessions will have timed out on the server but Evolution may still think that they are alive and try using them. This leads to very long timeouts before Evolution works correctly again after a resume.

    Mattias Holmlund

    January 12, 2010 at 12:30 am

  3. Awesome!!
    There is is some visual clutter/flicker in the statusbar, maybe just showing the number of pending actions pr. account instead of each operation?

    That’s the crappy part, we always ask for more ;)

    Frej Soya

    January 12, 2010 at 12:40 am

  4. Great! That’s one of the points that most needed love in Evo IMHO.

    Milan Bouchet-Valat

    January 12, 2010 at 3:09 am

  5. Awesome, really! Keep us updated! :)

    Bart Verwilst

    January 12, 2010 at 3:45 am

  6. Great news! Thank you!

    tuXXX

    January 12, 2010 at 3:45 am

  7. Yet another Evolution/IMAP user saying thank you.

    Leonardo Fontenelle

    January 12, 2010 at 4:47 am

  8. Awesome! IMAPX is a must thing today for Evolution.

    Srini

    January 12, 2010 at 8:03 am

  9. This is awesome! Nice work Chen! we’ve been *needing* this for years .. :)

    Johnny

    January 12, 2010 at 8:38 am

  10. Fantastic update for the world. Waiting eagerly for the connection manager next :D Hungry for more

    Bharath Acharya

    January 12, 2010 at 8:55 am

  11. Chen, don’t forget to include the server-hacks that are present in the current imap connector. Its a good think to work around the broken imap servers. Users gonna need that.

    Srini

    January 12, 2010 at 10:08 am

  12. Big step forward and came in at perfect time when we will be releasing GNOME 3 in a few months :D

    Akhil Laddha

    January 12, 2010 at 10:20 am

  13. and here are the rpms for openSUSE 11.2 for Evolution with IMAPX http://bit.ly/6hry2U Shoot down Chenthill now with bug reports and enhancements :)

    Bharath Acharya

    January 12, 2010 at 10:29 am

  14. What is impact ? A demo for evolution makes use of GMail. This is the impact of GMail ;-)

    Good work nevertheless. Congrats.

    Sankar

    January 12, 2010 at 11:11 am

  15. I appreciate all your comments, thanks everyone :)

    @Frej one is a progress messages and other is status which evo produces while adding the messages which can be hidden.

    @srini yes am frequently looking at the old imap code as well to check for the hacks. But not adding all of them, ignoring ones like hacks for exchange 5.5..

    @sankar thanks and I also love gmail :) evo has some good things to take from there too..

    special thanks to @abharath @jony and @akhil :)

    chenthill

    January 12, 2010 at 11:52 am

  16. Frej/Chen: Its because, the message list regens. The regen of message list should be killed and dynamic update of message list should be done. Have a look at Anjal, it uses a dynamic update of message list and no regen.

    Srini

    January 12, 2010 at 12:02 pm

  17. The number of emails found in “Stable” is not the same with the 2 versions of evolution. Bug or feature? ;)

    Thanks for the work!

    yann

    January 12, 2010 at 12:05 pm

  18. @Srini yup thats correct.

    @yann good observation :) its because as the messages were fetched, i was reading some them, so the unread count decreased in imapx. I never refreshed the folder from old version to get that reflected there..

    chenthill

    January 12, 2010 at 12:14 pm

  19. Great work, the perceived speed increase is immense!

    Jakub Steiner

    January 12, 2010 at 3:51 pm

  20. Excellent work, Chen. Lets keep adding the good pieces together. User’s with huge mailbox (like me) are going to love it.

    varadhan

    January 12, 2010 at 4:53 pm

  21. Another “thank you” here :)!

    Robin

    January 12, 2010 at 5:05 pm

  22. will be ready for 2.30?

    rita

    January 12, 2010 at 6:32 pm

  23. It is great that you tackle IMAP on Evolution, sadly it comes a bit to late as I just moved away from Evolution to the new thunderbird (and IMAP). I was just getting to many crashed with Evolution :-(

    Trust me I will return!

    seb

    January 12, 2010 at 8:56 pm

  24. You can’t know for how long I’ve been waiting for this !
    Thank you, thanks a lot Chen.

    Xav

    January 12, 2010 at 10:02 pm

  25. Nice. Unfortunately for me it hung within one or two minutes of using it.

    FWIW, here is what I did ..
    – Installed
    – Went through the setup wizard, selected IMAP rather than IMAP+. Also selected TLS encryption, Password Authentication

    Would not show any folders. Exit and restart Evolution. Edit->Preferences. Changed type to IMAP+. Got prompted for password. Exit settings, still no folders.
    Back into Preferences, Changed Security under “Receiving Email” to “SSL encryption”. Clicked OK. OK button “depresses” but now all of evolution is hung.

    Bruce

    January 13, 2010 at 4:42 am

  26. Great news,

    I am actually looking for the evolution branch to test it but no way to find it… I know I am probably not gifted ;)

    Thanks

    Thibault Saunier

    January 16, 2010 at 10:03 pm

  27. An excellent article, i’ve bookmarked it so keep it for reference.Thanks for the blog! :)

    AimbotDownload

    January 18, 2010 at 12:31 am

  28. Great Job, Chen.

    Mano

    January 19, 2010 at 7:42 am

  29. Just back after leave. Thanks jakub, varadhan, robin, seb, Xav, thibauly, aimbot and mano :)

    @rita yes it would be ready for 2.30. copy/move messages between folders in offline will not be available though.

    @Bruce, please send me the gdb traces (pchen at gnome dot org)

    @Thibault its being developed in master branch (aka HEAD with svn).

    chenthill

    January 20, 2010 at 11:42 am

  30. Oh boy, when did I write that – 4 years ago? I’m surprised anything is salvageable really.

    Although IIRC the code ended up somewhat simpler and the support for lots of ‘features’ just came as a side effect. It seemed to me to map more to how IMAP was intended to be used in the first place – it isn’t really all that complicated but the original IMAP code turned it into a monster. Camel in general was over-engineered I think – if I were to do it again the main theme would be simplification, and streaming/iterators, so no one operation takes too long and can be easily stopped.

    There was also less of a need for workarounds because it didn’t use features like retrieving messages by part number or the weird s-expression ‘compact header format’ – which were both a real mess to implement to start with, and *very*few* servers ever implemented properly. e.g. instead it retrieves just full message headers and does local parsing which is known to be correct, and retrieves message content by byte range and multiplexes those fetches asynchronously and pipelined to achieve much better interactive response and higher overall throughput as well.

    Good to hear it’s finally seeing the light of day anyway, in some form or another. In hindsight I should’ve just pushed it into the release (as an option) as soon as it did anything remotely useful.

    I was always far too cautious when working with the ‘production’ code base.

    Michael

    January 21, 2010 at 6:19 am

  31. @michael needless to say, you are obsolutely right :) I can see how bad our existing imap provider is.

    Async streaming camel apis are definetly needed to leverage this design. Also to allow operations in the back-ground when needed, especially when the users want to skim through a lot of mails.

    Maybe if imapx had come up before, many backends such as groupwise, mapi etc. would have followed this design rather than a sync one.

    chenthill

    January 22, 2010 at 3:31 pm

  32. [...] Enviado por Leonardo Fontenelle (leonardofΘgnome·org): “O Evolution 2.30 terá um suporte ainda melhor ao IMAP. Essa parte do código está sendo reescrita e já mostrou trazer ganhos notáveis de performance, como vocês verão no vídeo abaixo (recomendo exibir em tela cheia). Além disso, o Evolution ganhou suporte a IDLE, que é análogo ao “object push” do Bluetooth: quando o servidor recebe um e-mail, ele pode enviá-lo ativamente para o Evolution, na hora.” [referência: chenthill.wordpress.com] [...]

  33. It would be nice to have imap IDLE i.e. push email in Evolution 2.30.

    Cool…

    dude111

    February 2, 2010 at 9:10 pm

  34. Is this available in some PPA somewhere? For us Ubuntu Users…

    nucco

    February 19, 2010 at 5:24 pm

  35. i’m running 2.29.91 on Ubuntu 10.04 beta from ppa at https://launchpad.net/~francesco-marella/+archive/unstable-evolution

    till so far it seams to work nicely in my daily usage and i’m testing it to see if i can use it as a Thunderbird alternative.

    i’m using it as an imap client to a zimbra server so the imapidle functionality is what i am looking for.

    imap-idle seems to work, sort off, i don’t have to press the send receive button but i still need to refresh the inbox folder by selecting another folder and then select my inbox again.

    its almost there……

    randall

    March 16, 2010 at 8:40 pm

  36. It’s worth pointing out, especially to Bruce above, that all of your cached message headers will be gone if you switch to this provider. So Evolution will have to fetch headers for all those thousands of messages you have in your Gmail account, again. I thought it hung too, but it does work…eventually. And that’s no fault of this code.

    Andrew

    May 2, 2010 at 3:39 am

  37. Hi,
    I’m testing Evolution 2.30.1 with my own mail server and have quickly implemented RFC 2177, however evolution seems to not issue the “IDLE” command…

    The only place where I see “IDLE” in the communication transcript is when the server replies to CAPABILITY.

    I am a bit puzzled as I was expecting IDLE to work in this version of evolution…

    MagicalTux

    May 27, 2010 at 6:18 am

  38. Replying to myself, I had to explicitly enable IMAP+ protocol for the account. Wasn’t expecting the config to contain two imap options…

    MagicalTux

    May 31, 2010 at 6:02 am

  39. Brilliant. Thankyou…

    Martin

    June 15, 2010 at 3:04 pm

  40. I am not able to have idle working. In fact when I switch the account from imap to imapx. I am not able to see message in Inbox, I can only see subfolders.

    Cedric Pinson

    August 6, 2010 at 2:33 pm

  41. Cedric, there was a bug in 2.30.x where the idle was not enabled by default and one needs to enable from account preferences. Just make sure to enable it. This has been fixed in 2.31.x.

    The option is ‘Enable IDLE if the server supports it’.

    What is the imap server your using?

    chenthill

    August 7, 2010 at 9:27 pm

  42. Still doesn’t show my shared folders from Courier. Too bad; I really want to like Evolution, and I try out about once every year, but have to give up due to abysmal IMAP support. Was hoping this update would at least give me access to my shared folders, but sadly no. So; I guess it’s back to Mutt for another year!

    B. Johannessen

    September 7, 2010 at 6:36 pm

  43. if i want to get mail with in date range ,supposed like: want to get mail between 29-Nov-2011 to 30-Nov-2011 than what to do?how to write search query. using imapx

    Ravindra Pawar

    August 29, 2012 at 2:01 pm

  44. I tried out IMAP+ and it looks good. What currently prevents me from using it permenantly is that it doesn’t support a “real trash folder” like the “old IMAP” does. This is really crucial for me because I also use other mail clients and they tend to clean the INCOMING folder from mails marked as deleted, meaning that mails get lost although I haven’t emptied the trash.

    Michael Lipp

    March 7, 2013 at 2:47 pm

  45. I upgrade my debian to jessie and switch the account from imap to imapx. Now I am not able to see message in Inbox, I can only see subfolders. Evolution version is 3.12.2 and the imap server capability response is: * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2005 Double Precision, Inc.

    Luis Claudio

    August 6, 2014 at 8:12 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 746 other followers

%d bloggers like this: