Thursday
Sep062007
Scaling IMAP and POP3
Thursday, September 6, 2007 at 2:15AM
Another scalability strategy brought to you by Erik Osterman:
Just thought I'd drop a brief suggestion to anyone building a large mail system. Our solution for scaling mail pickup was to develop a sharded architecture whereby accounts are spread across a cluster of servers, each with imap/pop3 capability. Then we use a cluster of reverse proxies (Perdition) speaking to the backend imap/pop3 servers .
The benefit of this approach is you can use simply use round-robin or HA load balancing on the perdition servers that end users connect to (e.g. admins can easily move accounts around on the backend storage servers without affecting end users). Perdition manages routing users to the appropriate backend servers and has MySQL support.
What we also liked about this approach was that it had no dependency on a distributed or networked file system, so less chance of corruption or data consistency issues. When an individual server reaches capacity, we just off load users to a less used server. If any server goes offline, it only affects the fraction of users assigned to that server.
Just thought I'd drop a brief suggestion to anyone building a large mail system. Our solution for scaling mail pickup was to develop a sharded architecture whereby accounts are spread across a cluster of servers, each with imap/pop3 capability. Then we use a cluster of reverse proxies (Perdition) speaking to the backend imap/pop3 servers .
The benefit of this approach is you can use simply use round-robin or HA load balancing on the perdition servers that end users connect to (e.g. admins can easily move accounts around on the backend storage servers without affecting end users). Perdition manages routing users to the appropriate backend servers and has MySQL support.
What we also liked about this approach was that it had no dependency on a distributed or networked file system, so less chance of corruption or data consistency issues. When an individual server reaches capacity, we just off load users to a less used server. If any server goes offline, it only affects the fraction of users assigned to that server.
Reader Comments (4)
Hi!
How many users/messages/servers are you dealing with?
How do you move users between servers? As far as I understand you must take them offline during the migration... correct?
Better to use nginx instead of perdition:
http://blog.fastmail.fm/2007/01/04/webimappop-frontend-proxies-changed-to-nginx/
Thanks for the heads up on the nginx IMAP/POP3 proxy (and for contributing to it's development!) Last I used nginx was as a lighty replacement, due to lighty's shotty fcgi load balancing (imo).
When a user is migrated, their account is temporarily suspended. Then we migrate their mail spool over to the new server. During this time the user is locked out. However, once we reactivate their account, the change is transparent to the user. What I mean by transparent is that the user does not need to reconfigure their mail server IPs or logins; not that the individual user experiences no temporary outages.
How about a POP3/SMTP server pair that store their users, passwords and actual email in Amazon S3? I'm actually in the process of building that, and was actually wondering what the interest in it would be.