AWS SES MTA

The above subject line is what I searched for this afternoon. I took a moment to be amused as this search term brought up 2.2 million results. What does it even mean!?

AWS = Amazon Web Services (Amazon’s most excellent server hosting services)
SES = Simple Email Service (Amazon’s solution to bulk mailings)
MTA = Mail Transport Agent (sendmail, postfix, etc).

A bit of back story for you. I recently set up a group of servers for a client since their previous single dedicated server wasn’t cutting the mustard. We migrated them over to AWS with a load balancer, small farm of web servers (EC2), NFS server (EC2), a database (RDS), and a CDN for distributing some of their content (S3+CloudFront). It took a bit of doing and learning the “Amazon way” but it all worked out and is quite a nice setup. It allows for rapid scaling up to deal with bursts of traffic and scale down to save costs. I’ll write more about this setup in a future post. I want to talk about email problems today.

Amazon Web Services

When we first set things up we discovered that getting mail sent out form an EC2 server is sort of like sending mail through snail mail…it doesn’t get to its destination. EC2 apparently is blacklisted on most all SPAM blacklists. I can see why with the ease of creating and destroying servers…those nasty spammers must have had a hay-day in the early days of AWS. We were left with the problem of not being able to send out our administrative emails like user account validation, lost password recovery, etc.

Our first solution was to use Google’s awesome corp email service. So I set up the domain with Google and we modified the servers sendmail to relay all our admin type emails through our Google account support account. This worked really well until this morning when I found out that Gmail has a limit of emails that are allowed to go out to different users….again, antispam percautions on their part. Here’s some references:

http://mail.google.com/support/bin/answer.py?answer=22839
http://mail.google.com/support/bin/answer.py?answer=81126

So much for that solution which I was thinking was a nice method. Back to the drawing board.

I ended up finding Amazon’s Simple Email Service (SES) which seemed to do exactly what I needed. I filled out the forms and got an approved account in less than a day. Thank goodness too because Google had cut us off and we were loosing registrations quickly.

Now the trick was to get SES working on the EC2 web servers. We initially set up Sendmail on the server to do our relay…but only because it was already installed on the server and we know how to set up the relay easily enough. So I went through the process of configing sendmail to use SES. The instructions are easy to follow and I did everything they said to do but kept running into problems when I went to actually send the emails. I was getting errors:

SYSERR(root): buildaddr: unknown mailer aws-email
and my emails weren’t getting delivered.

In my php code I know I was sending the emails using a validated account and was successfully able to run the ses-send-email.pl command from commandline without problems.

I gave up on sendmail and installed my trusty MTA postfix. Of course I had to set it up, but again it was an easy config.

It too had errors:

status=bounced (Command died with status 1: "/opt/aws/bin/ses-send-email.pl". Command output: Email address is not verified. )
and
relay=aws-email, delay=0.39, delays=0.01/0/0/0.38, dsn=5.3.0, status=bounced (Command died with status 1: "/opt/aws/bin/ses-send-email.pl". Command output: Missing final '@domain' )

At this point I’m pretty frustrated with the whole thing. After a break I came back it and eventually found this post:

http://benjisimon.blogspot.com/2011/02/gotcha-of-day-ec2-postfix-amazon-simple.html

Ben was right, the email was trying to send from the crazy EC2 hostname.

I did what he said and created a /etc/postfix/sender_canonical file with:
/(.*?)@(.*)/ $1@realdomain.com

And updated the main.cf file:
sender_canonical_maps = regexp:/etc/postfix/sender_canonical

It still didn’t work, BUT it was rewriting the domain portion correctly. I saw that it was apache that was trying to send out the mail so I just tweaked his reg exp to be this:
/(.*?)@(.*)/ validuser@realdomain.com

and there you have it…it worked! I know it’s not exactly right to hard code the valid user email into the postfix config so I’ll fix that up tomorrow, but for now I at least have a working config and can proceed with putting out some other fires.

Thanx Ben for the tip, it really helped me out.

Update (Oct 1, 2013) – changed the method of doing this. Followed these instructions. Just a note to be sure to authorize your sending email addresses or domain otherwise you’ll get a error: “554 Transaction failed: Invalid email address” and “501 Invalid MAIL FROM address provided”.