How to add offset in array for newsletter sending?

Question

I have to send a newsletter to all network users. The problem is that the array, containing the result of the query, has about eight thousand records. How can I optimize it so that it does not exceed the memory limit? Add ini_set('memory_limit', '-1'); was not enough.

function enviarNewsletterUsuarios(){
    $usuarios = getUsuarios();


if(!empty($usuarios) OR !isset($usuarios)){


    foreach ($usuarios as $usuario) {


        $objetivo   = getObjetivo($usuario->guid);
        $ideias     = getIdeiaPorObjetivo($objetivo, $usuario->cidade);


        $mail = new PHPMailer;


        $mail->isSMTP();                                      // Set mailer to use SMTP
        $mail->Host = 'host';                 // Specify main and backup SMTP servers
        $mail->SMTPAuth = true;                               // Enable SMTP authentication
        $mail->Username = 'user';              // SMTP username
        $mail->Password = 'pass';           // SMTP password
        $mail->SMTPSecure = 'tls';                            // Enable encryption, 'ssl' also accepted


        $mail->From = 'teste';
        $mail->FromName = 'teste';


        $mail->addAddress($usuario->email);  // Add a recipient


        $mail->WordWrap = 50;                                 // Set word wrap to 50 characters
        $mail->isHTML(true);                                  // Set email format to HTML


        $mail->Subject = 'Newsletter';
        $mail->Body = utf8_decode('tesste'); 


        if(!$mail->send()) {
            echo "Ocorreu um erro ao enviar!";
            //echo 'Mailer Error: ' . $mail->ErrorInfo;
        } else {
            echo 'Newsletter enviado com sucesso!';
        }
    }

    return true;
}

}

Answer 1

When it comes to sending newsletters, 8 Thousand recipients is not a high number, let alone a number that will consume all available memory.

What seems to be missing is an optimization of the objects in use so as not to fill the memory with the same information repeated 8 Thousand times.

Possible optimisation

function enviarNewsletterUsuarios() {

    // recolher os destinatários
    $usuarios = getUsuarios();

    // se temos destinatários
    if ($usuarios) {

        /* Preparar o envio definindo os valores comuns
         */
        $mail = new PHPMailer;

        $mail->isSMTP();                // Set mailer to use SMTP
        $mail->Host = 'host';           // Specify main and backup SMTP servers
        $mail->SMTPAuth = true;         // Enable SMTP authentication
        $mail->Username = 'user';       // SMTP username
        $mail->Password = 'pass';       // SMTP password
        $mail->SMTPSecure = 'tls';      // Enable encryption, 'ssl' also accepted

        $mail->From = 'teste';
        $mail->FromName = 'teste';

        $mail->WordWrap = 50;           // Set word wrap to 50 characters
        $mail->isHTML(true);            // Set email format to HTML

        $mail->Subject = 'Newsletter';
        $mail->Body = utf8_decode('tesste');


        /* Por cada destinatário enviar o email
         */
        foreach ($usuarios as $usuario) {

            $mail->addAddress($usuario->email);  // Add a recipient

            if ($mail->send()) {
                // acção quando correu bem
            } else {
                // correu mal :(
            }

            /* Limpa a lista de destinatários para evitar que o email
             * a enviar vá acumulando destinatários
             */
            $mail->ClearAllRecipients();
        }

        return true;
    }
}

---

In your code I also noticed that you have:

$objetivo = getObjetivo($usuario->guid);
$ideias   = getIdeiaPorObjetivo($objetivo, $usuario->cidade);

In case it is something that will be "injected" in the body of the email or the subject of it, I already recommend a Queue to send in a database because the job of sending the email is exactly that, send the email. Its information must be prepared and ready to use or effectively faces high memory consumption problems.

Example of a table for Queue newsletter:

CREATE TABLE IF NOT EXISTS `newsletter_queue` (
  `id` int(13) NOT NULL AUTO_INCREMENT COMMENT 'Internal ID',
  `newsletter_id` int(13) NOT NULL COMMENT 'ID from the table "newsletter"',
  `entity_id` int(13) NOT NULL COMMENT 'ID from the table "entity".',
  `entity_type` enum('entity','subscriber','admin user','unknown') NOT NULL DEFAULT 'entity' COMMENT 'The entity type for statistics.',
  `send_method` set('mail','smtp') NOT NULL DEFAULT '' COMMENT 'Message is sent using PHP mail() or SMTP.',
  `from_email` varchar(255) NOT NULL COMMENT 'Sets the From email address for the message',
  `from_name` varchar(255) NOT NULL COMMENT 'Sets the From name of the message',
  `smtp_host` varchar(255) NOT NULL COMMENT 'Sets the SMTP hosts. All hosts must be separated by a semicolon. You can also specify a different port for each host by using this format: [hostname:port] (e.g. "smtp1.example.com:25;smtp2.example.com"). Hosts will be tried in order.',
  `smtp_port` int(4) NOT NULL DEFAULT '25' COMMENT 'Sets the default SMTP server port.',
  `smtp_auth` tinyint(1) NOT NULL DEFAULT '1' COMMENT 'Sets SMTP authentication. Utilizes the Username and Password variables.',
  `smtp_secure` enum('','ssl','tls','starttls') NOT NULL DEFAULT '' COMMENT 'Sets connection prefix.',
  `smtp_username` varchar(255) NOT NULL COMMENT 'Sets SMTP username.',
  `smtp_password` varchar(255) NOT NULL COMMENT 'Sets SMTP password.',
  `recipient_mail` varchar(255) NOT NULL,
  `recipient_name` varchar(255) NOT NULL,
  `attachment` varchar(255) NOT NULL,
  `content_type` varchar(50) NOT NULL,
  `priority` enum('1','3','5') NOT NULL DEFAULT '3' COMMENT 'Email priority (1 = High, 3 = Normal, 5 = low)',
  `charset` enum('iso-8859-1','utf-8','windows-1252') NOT NULL DEFAULT 'utf-8' COMMENT 'Sets the CharSet of the message.',
  `send_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`),
  KEY `newsletter_id` (`newsletter_id`),
  KEY `entity_id` (`entity_id`),
  KEY `entity_type` (`entity_type`)
) ENGINE=MyISAM  DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;

With a table like this, you have a function that adds emails to the Queue and your sending function will be limited to sending X emails every X minutes. This way you definitely solve memory issues now or in the future when your CRON runs to send emails.

Answer 2

One thing you should keep in mind is that, in terms of performance, arrays are malignant. You should never have very large arrays.

In the scenario of your problem, the thing is worse because it is the sending of email. You have some possible outputs but overall the idea is basically to work with the same concept of pagination of results.

Lists a "little bit", iterates and sends. Unlike a pagination, which usually rescues 10, 20 or 50 records, you can even go further.

Regarding GUI, you can make a splash screen with a META Refresh X seconds that serves, including so that the operator, if any, knows that the system is still working.

Here, an important factor would be to have a way to control whether a message has already been sent to that recipient. A column in the database, in a table of logs, is suddenly enough.

This is because the action depends on the hardware being operating for the upload to occur. And if there is any power interruption you may have customers/users receiving duplicates.

If it is in batch, which is interesting for 8 thousand records, scheduling with Cronjobs would be well desirable and nothing this would not be necessary.

However, it all depends on the limitations imposed by your server. Just remember that SPAM is a crime! ;)