Email Architecture

I would like to know email architecture used by large ISPs.. or even used by google.

Can someone point me to some sites??


Thanks..

Comments

[Todd Hoff]

This thread might have some leads you can follow: http://news.ycombinator.com/item?id=92022

[atif.ghaffar]

Hello Anonymous?

Email architecture is quiet a general term.
What exactly are you looking information about?

A medium infrastructure may look something like this.
http://www.worldsoft-postmaster.info/ (note, a year old diagram of our mail infrastructure)

All this can be broken into many different parts and each of them is important and need different handling and care.

Outgoing emails architecture.
Text filtering outgoing emails, using custom IP addresses with reverse DNS and SPF records....

Incoming MX architecture.
Firewalls, spamfilters, quarantines: Servers that accept mails only from other mail servers and do not get in contact with end users.

SMTP Auth architecture
Outgoing architecture for your customers where they login and send emails inside or outside your infrastructure?

Mail delivery infrastructure.
How to store incoming mails to mailboxes on one or more servers.

Mail recieving
POP/IMAP/Webmail proxies and access. How to authenticate, authorize and direct a user to his mailbox.

hope this helps.

[sugiggs]

thanks for all the replies.

Is it possible to elaborate more for each architecture? like softwares used, specific configurations, etc... for example, to store user database, should I use mysql or LDAP?

I'm working at an ISP now (just joined) and we are going to change our current email system which is very very old already.

Thanks

[atif.ghaffar]

Suggigs,

All depends on your requirements and experience.
It also depends on if you want to use only opensource software or proprietary software or you dont care as long as it works.

I would suggest to go with something easy like Axigen. ( http://www.axigen.com/ )
They dont cost you an arm or a leg, are very attractively priced and give great support service. We use it for our corporate email (not for our isp email infrastructure) and are very happy with this product.

If you want to do it yourself. "I would suggest NOT". Its really not worth the trouble.

If you really really still insist, I will pass along all the information (Note: There is too many details in running an email infrastructure)

Another possibility that you may want to look at. Outsource all the Emails to Google apps.
Google Aps also have an ISP version. http://www.google.com/a/partners/

Best regards.

[sugiggs]

Hi atif ghaffar,

Thank you very much for your reply (and the link). It is really useful.

I would like to explore all options we have and presented them to management. If you dont mind, could you please pass me the information? you can reply here or send directly to my email: sugiggs (at) gmail.com

anyway, I love this site very much. I learn a lot of useful things :) Keep it on!

regards,

-sugiggs-

[atif.ghaffar]

Sugiggs,

Before I start writting all the blurb, can you give me some numbers.

How many mailboxes you have and how many to expect to have this year?
How many domains?
What is your incoming Spam/HAM ratio?
What is the frequency of your mailservers ending up in blacklists?
What is the budget? (less than $100K, less than $500K, etc?)

Its not all technical, all really depends on what you want to do and what problems you want to solve.
With more information from you, I might be able to send you in the right direction.

Again, if you core business is not email (I assume it is not), then outsource it.

[sugiggs]

Hi Atif Ghaffar,

Thanks for your willingness to help me :)
I just joined this company, so I ask my senior your questions.

>How many mailboxes you have and how many to expect to have this year?

We currently have around 14k mailboxes. We expected to have 20-25k by the end of this year

>How many domains?

Around 1k

>What is your incoming Spam/HAM ratio?

Sorry, He doesn't know.

>What is the frequency of your mailservers ending up in blacklists?

We seldom blacklisted

>What is the budget? (less than $100K, less than $500K, etc?)

My management never mention about budget. Maybe you can start with Small-Medium ISP budget. Around 0 - $200k maybe.

Thank you for your time ^_^

Regards,

-sugiggs

[Simon Lyall]

14-25k mailboxes is getting towards the low end of economies of scale.

Your big problems is the fact that MAGY [1] give away email for free and have really good systems . Duplicating them will cost your a bit of money and time.

* You will have to do free and good virus and spam filtering, for virus filtering this will cost you money.
* You will need a good webmail system (filters, autoreply, vacation, folders, AJAX etc ).
* You will need storage for a few thousand emails/customer and quotas for at least 100MB and probably more. The storage has to be rock solid (at least RAID5, probably a lot better) and able to handle a high load.
* You will need multiple boxes to handle each of the functions mentioned previously (pop, smtp outgoing, queue servers, webmail, incoming email, virus filteing, spam filtering, provisioning, etc)
* You must ( MUST!!!) overprovision your service at all times and not just for redundancy. About once a year your email load will increase by 50% when the spaming ecosystem changes, you must be able to handle this.

At minimum your ISP should already have:

1. Load balancers to put in front of the servers
2. A provisioning system that can add/delet/update users via LDAP or database pushes to the mail system instantly.
3. A Solid storage backend
4. Ability to casually handle another dozen servers.

If you don't have the above then you'll probably have to build it or you are just wasting your time and there is a good chance that with your size you won't have some ( especially 1 ) and it will cost too much to build.

Seriously building this yourself is a fairly big undertaking and user have high expeciations these days for reliability, filtering and pretty webmail interfaces. All three of those will cost your time and money to get right and a 20k user ISP probably won't have the time and budget to invest in that.

I was going to write a book on this a few years back, it's a little specialised and I don't have the energy though :(

[1] Microsoft, AOL/Amazon, Google, Yahoo

[atif.ghaffar]

sugiggs,

What is your current storage usage?
What is the quota that you allow to each mailbox?

Simon Lyall already mentions most of the basic requirements to setup something reasonable okay.

$200K is not a very large budget but it is just about doable with if you do a lot of D.I.Y. and buy ALL the right stuff.

So if you really want to go the DIY, You are looking here at a

couple of SAN for databases, mailstorage, Look at coraid.com (you can buy a chasis there and fill it in 15 1Tera disk and make a raid 5 on it. This would cost you altogether around $10K for 10TB storage) Take 2 or 3 of them. PS: learn how to use LVM with them.

a few load-balancers (2 should be ok, take one spare) , look at loadbalancer.org. We use them and they are very good. Just dont put them in front of the MX servers.

20 modest boxes to do the proxies and other frontend functions. buy them new but cheapest you can. Dont spend more than $800 on each. Take AMD over Intel. You will get the same power but a little cheaper. Dont care about multi-cores.

At minimum
2 solid boxes to do databases. (disks not important, RAM and is CPU is) (8GB each should be good) should cost you ~$6-8K each. Connect them together with hearbeat and use both of them as active/active but for 2 different databases. When one fails, then the healthy box should run the 2 databases.

2 solid boxes to do mailbox backends. (disks not important, RAM and is CPU is) (4 GB each should be good) Same as before, cluster them and use them as 2 nodes, spread your mailboxes over these and add 2 nodes everything you want to grow.

2 very good boxes to face the harsh world (MX) and do filtering, firewalling (FOR THE MX ONLY)
Depends on what you want to run on them the requirements can easily change.
We run SpamFilter ISP that runs only on Windows server (but a very good product) so we have to put additional firewall in front of them to allow only mail access to the windows boxes and do some additional firewalling to ease the load on the spamfilters.

A couple of boxes to provide helper services to this infrastructure (dns cache, syslog, monitoring, altering, etc)

Use Linux for everything. Ubuntu-server works quiet well.

Around 10 Gigabit switches.

All this can work but I will strongly advice against it again :-)
Go to google for the mails and spend your money and time on writting another Killer-App!

[sugiggs]

Hi Atif Ghaffar,

Thanks for your reply. U did spent your time explaining to me. I really appreciate it. Thank you very much.

-sugiggs