Send email in bulk using Amazon SES

Question

I’m developing a system that should send bulk emails using the Amazon SES service.

How the shipment is made

To send email, I am using PHP Mailer, with the following code:

$mail = new PHPMailer;

$mail->isSMTP();                                      // Set mailer to use SMTP
$mail->Host = '{HOST DO SES}';  // Specify main and backup server
$mail->SMTPAuth = true;                               // Enable SMTP authentication

$mail->Username = '{USERNAME DO SES}';                            // SMTP username
$mail->Password = '{PASSWORD DO SES}';                           // SMTP password
$mail->SMTPSecure = 'ssl';                            // Enable encryption, 'ssl' also accepted
$mail->SMTPDebug  = 0;
$mail->Port = 465;
$mail->CharSet = 'UTF-8';

$mail->From = '[email protected]';
$mail->FromName = 'Site name';
$mail->addAddress( $email );  // Add a recipient
$mail->addReplyTo('[email protected]', 'Site name');

$mail->WordWrap = 80;                                 // Set word wrap to 50 characters
$mail->isHTML(true);                                  // Set email format to HTML

$mail->Subject = 'Seu amigo(a) ' . $AuthUser->name . ' te indicou';
$mail->Body    = $message;
$mail->AltBody = 'Mensagem alternativo texto puro.';

if(!$mail->send()) {
    throw new FwException( 'Erro ao enviar convite para "' .$email . '" do usuário' );
}
$mail->SmtpClose();

As I enable the user to import a TXT list with multiple emails, I risk having a high number of Bounces, that is, many invalid emails in the imported list. If I have an invalid email return password, my user may be blocked. For this I valid before sending using the one class I found in Google Code.

Some tips on email validation, I found here at Stack Overflow

Using this type of method I can remove most of the invalid emails. This does not work for servers that return code 200 to the command RCPT TO sent to the SMTP server of the tested email. So for all validation send the email I want to test and if it returns me 200, send an email that probably won’t exist, such as "[email protected]".

To send the emails, I recover the valid emails and execute the sending code one by one giving a 1 second Sleep between each one.

Configuration of the SES

My SES provides me with 10,000 emails in 24 hours and 5 emails every second.

I set up the domain and enabled DKIM.

I validated a valid address for the configured domain.

I followed the recommendations to set up SES on Amazon, following the information that can be filled right here on Stack Overflow.

I have two instances of EC2, one for service and the other for the system. Where the service has fixed IP and the system does not. Emails are validated and sent by the system instance.

Doubts

1. Amazon has some easy way to send the mailing list and it manage the mailing queue?

2. If not, the only way is via script as shown above?

3. The email validation code can cause me to log into the Blacklist of some servers due to the high number of requests to the SMTP server. Knew of this risk by specialized services in email validation existing in the market as the Bulk Email Verifier and Mail Chimp.

   Is this risk true? There is a possibility to circumvent this problem?

Answer 1

After studying a lot the various ways I could find on the web for bulk sending control using the Amazon Simple Email Service I arrived in a simple and easy to control way. And I preferred to use a gift solution the Amazon SDK.

To start Amazon does not have in the SES service the means to manage shipping queues, so I chose to use CRON and split the shipments into small batches.

I designed an interface to control the CRON for PHP, so that I can schedule tasks by recording information from a form HTML and processing by PHP. This also allows my table CRON keep only one task that will run a script that will verify the tasks of my system.

For this I created a table in the database (I am using Mysql). Logging tasks in the database can easily enable and disable a task, as well as change it when necessary.

CREATE TABLE `cron_task` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `description` varchar(250) NOT NULL,
  `minute` varchar(10) NOT NULL,
  `hour` varchar(10) NOT NULL,
  `day` varchar(10) NOT NULL,
  `month` varchar(10) NOT NULL,
  `year` varchar(10) NOT NULL,
  `weekday` varchar(10) NOT NULL,
  `type` varchar(20) NOT NULL,
  `active` tinyint(1) NOT NULL,
  `priority` tinyint(3) NOT NULL,
  `first_execution` timestamp NULL DEFAULT NULL,
  `last_execution` timestamp NULL DEFAULT NULL,
  `created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `updated_at` timestamp NULL DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=10 DEFAULT CHARSET=utf8;

I used an interface to standardize my task classes.

interface ICronTask {
    public static function add( ICronTask $task );
    public function activate();
    public function deactivate();
    public function executeTask();
    public function makeLog( $content );
}

Where the method add serves to save a task, activate and deactivate to enable and disable the task, executeTask will contain the context of job execution, be it email sending, url request, backup, or whatever else, and the method makeLog to generate custom logs for the execution of tasks.

I created an abstract class to implement the basic methods for all tasks.

abstract class CronTask implements ICronTask {
    public $id;
    public $description;

    public $minute;
    public $hour;
    public $day;
    public $month;
    public $year;
    public $weekday;

    public $type;
    public $priority;

    public $active;
    public $first_execution;
    public $last_execution;

    public $created_at;
    public $updated_at;

    public $execution_start;
    public $execution_end;

    public function __construct( $id = null ) {
        // caso o $id possua valor, busco informações no banco sobre a tarefa
        // e preencho todos os atributos.
    }

    public final static function add( ICronTask $task ) {
        // Salvo no banco de dados e chamo o método save()
    }

    public final function activate() {
        // Atualizo a tarefa no banco para ativar
    }

    public final function deactivate() {
        // Atualizo a tarefa no banco para desativar
    }

    public final function makeLog( $content ) {
        // Salvo o log da tarefa
    }

    public final function executeTask() {
        // Executo a tarefa
        $this->execution_start = round( microtime(true), 4 );
        $content = $this->execute();
        $this->execution_end = round( microtime(true), 4 );
        $this->makeLog($content);
    }

    public final function isNow() {
        // Faço a verificação da hora de execução, garantindo que deve ser executada
        // no momento em que for chamado.
        return (
            $this->parserTime($this->minute, 'i')   &&
            $this->parserTime($this->hour, 'H')     &&
            $this->parserTime($this->month, 'm')    &&
            $this->parserTime($this->day, 'd')      &&
            $this->parserTime($this->weekday, 'w')
        );
    }

    private function parserTime( $value, $element ) {
        // Obtem o tem atual
    $time = date( $element );

    // Verifica se o valor é igual à "*" representando toda momento.
    if( $value == '*' ) {
        return true;
    }

    // Separa os conjuntos de tempos separados por vírgula
    $groups = explode( ',', $value );
    foreach ( $groups as $part ) {
        // Verifica se é um intervalo composto. Ex: "*/5" ou "20-40/2"
        // Se é um intervalo compost, deverá retornar true se o valor atual
        // estiver dentro do intervalo definido antes da barra, e na frequência
        // definida após a barra.
        if( strpos( $part, '/' ) ) {
            $groupsInterval = explode( '/', $part );
            // Verificando a frequência
            $frequency = $time % $groupsInterval[1] == 0;

            // Verificando o intervalo
            $interval = explode( '-', $groupsInterval[0] );
            $intervalResult = false;
            if( $interval[0] == '*' ) {
                $intervalResult = true;
            } else {
                $intervalResult = ( $time >= $interval[0] && $time <= $interval[1] );
            }
            return $frequency && $intervalResult;
        }

        // Verifica se é um intervalo simples. Ex: "10-50"
        // Se é um intervalo, deverá retornar true se o valor atual estiver
        // dentro desse intervalo.
        if( strpos( $part, '-' ) ) {
            $interval = explode( '-', $part );
            return $time >= $interval[0] && $time <= $interval[1];
        }

        // Se for um número simples verifica se é o tempo certo
        if( $time == $part ) {
            return true;
        }
    }
    return false;
    }

    abstract protected function execute();
    abstract protected function save();
}

And an example of implementing a task class would be

class CronTaskTest extends CronTask {
    public $type = 'Test';
    public $priority = 0;

    protected function execute() {
        return 'Tarefa executada com sucesso';
    }

    protected function save() {
        return true;
    }
}

The main class that will be executed at all times checking the tasks has been implemented as follows:

class Cron {
    public static function execute() {
        $tasks = self::getTasks();
        foreach ( $tasks as $task ) {
            if( $task->isNow() ) {
                $task->executeTask();
            }
        }
    }

    public static function getTasks() {
        try {
            $tasks = // Busco todas as tarefas ativas ordenadas por prioridade DESC;

            $return = array();
            foreach ( $tasks as $record ) {
                $taskName = 'CronTask' . $record['type'];
                require_once __DIR__ . '/tasks/' . $taskName . '.php';
                $return[] = new $taskName( $record['id'] );
            }
        } catch ( PDOException $exception ) {
            die( $exception->getMessage() );
        }
        return $return;
    }
}

I create a PHP file to perform the tasks by calling the method Cron::execute(). And scheduled on CRON

# crontab -e
* * * * * /usr/local/bin/php /var/www/projeto/meu-script.php

So I created a class called CronTaskMailing similar to the example above with the attributes themselves

public $name;
public $subject;
public $body;
public $alternativeBody;
public $startSend;
public $notifyTo;
public $keywords = array();
public $addresses = array();
public $sended = array();

public $startedAt;
public $completedAt;

These attributes I saved in a json. For this I implemented logic in the method save which is called just when the task is added. Also implement another method to load the information from json except for class attributes.

I implemented a method to handle the entire message that will be passed to the Amazon SDK.

private function makeMessage() {
    $msg = array();
    $msg['Source'] = "[email protected]";

    $msg['Message']['Subject']['Data'] = $this->subject;
    $msg['Message']['Subject']['Charset'] = "UTF-8";

    $msg['Message']['Body']['Text']['Data'] = $this->alternativeBody;
    $msg['Message']['Body']['Text']['Charset'] = "UTF-8";

    $msg['Message']['Body']['Html']['Data'] = $this->body;
    $msg['Message']['Body']['Html']['Charset'] = "UTF-8";

    return $msg;
}

I defined a maximum mean of use of SES for mailing, preventing other services that also use SES from being unable to use. This quota is set in percentage to facilitate the accounts.

const MAX_QUOTA_USAGE = 0.8;

And the method of execution of the task was as follows:

protected function execute() {
    $log = '';
    try {
        $this->loadMailingFile( $this->id );
        $ses = SesClient::factory( array(
            'key' => 'ACCESS_KEY',
            'secret' => 'API_SECRET',
            'region' => 'REGION'
        ) );

        if( $this->startedAt ) {
            $this->startedAt = time();
        }

        $msg = $this->makeMessage();

        // Obtenho a minha cota e verifico se está dentro da média máxima definida.
        $quota = $ses->getSendQuota();
        $maxQuota = $quota['Max24HourSend'] * self::MAX_QUOTA_USAGE;

        if( $maxQuota < $quota['SentLast24Hours'] ) {
            $log .= 'Cota máxima para mailing em 24 horas excedida.';
        } else {
            // Calculo o delay entre uma mensagem e outra para não 
            // estourar o número de envios por segundo.
            $rate = $quota['MaxSendRate'];
            $delay = $quota['MaxSendRate'] / 1000000;
            // Calculo o tamanho do lote para que a execução dure no máximo um minuto.
            $maxSendBatch = $rate * 60;

            $count = 0;
            foreach ( $this->addresses as $key => $email ) {
                $msg['Destination']['ToAddresses'] = array( $email );
                $response = $ses->sendEmail( $msg );
                $messageId = $response->get( 'MessageId' );
                $log .= 'Mensagem enviada: ' . $messageId . PHP_EOL;

                $this->sended[] = array(
                    'email' => $email,
                    'message_id' => $messageId,
                    'sended_at' => time()
                );
                unset( $this->addresses[$key] );

                $count++;
                if( $count == $maxSendBatch ) {
                    break;
                }
                usleep($delay);
            }
            $log .= '-----' . PHP_EOL;
            $log .= 'Total de emails enviados: ' . count( $this->sended ) . PHP_EOL;
            $log .= 'Total de emails que faltam: ' . count( $this->addresses ) . PHP_EOL;
            if( count( $this->addresses ) == 0 ) {
                $this->completedAt = time();
                $this->deactivate();
                // Método para notificar por email quando o envio terminar.
                $this->sendNotificationToMailingCompleted();
                $log .= 'Envio de mailing concluído';
            }
        }
        $this->save();
    } catch (Exception $ex) {
        $log .= $ex->getMessage() . PHP_EOL;
    }
    return $log;
}

The quota increases gradually according to the good use of the service, so I elaborated the above code in order to identify this increase and work with the highest possible shipping rate.

In Amazon’s initial settings ( 10,000 emails in 24 hours with 5 emails per second) would be 300 emails per minute, that is, it would take on average 33 minutes to send 10,000 emails. It may take a few more hours using the way I quote here, depending on the limit set in the constant MAX_QUOTA_USAGE.

In addition, Amazon has strict policies for sending email, and having a clean email list, that is, without many invalid emails is essential. To resolve this issue, I implemented another task that runs by checking all emails every 15 days, and storing information indicating whether the email is valid or not.

The Amazon SDK has method that enables this verification (verifyEmailIdentity). However this method should be called with the interval of at least 1 second.