Installing Sender History Database and Configuring Delay Queue

About this task

To install Sender History Database role and configure Delay Queue:

1. Installation of Sender History Database Role.

a. Installation

1. Use the PMX installer. It will ask for permission to update itself. Answer Yes.

2. Select Custom Installation.

3. Select Sender History Database.

4. Specify the email address for notification messages.

5. Specify the mail server to send notifications to.

6. Specify the directory in which the sender history database file should be stored.

7. Specify the internal host IP address/subnet, if any.

8. Proceed with the installation.

Note: If you have Full installations and want to upgrade to 6.4.0:
  • Upgrade to 6.4.0 (new packages for existings roles will be upgraded).
  • Run the installer, go to Add Additional Components, select Sender History Database Role and install.

b. Configuration

Sender history database is maintained using Redis (https://redis.io/). You need to configure: <pmx_directory>/etc/redis.conf.Explanation of parameters is given below. Check the values of: bind, port, dir and dbfilename.

Basic configuration: Network

Specify the network interface on which Redis should listen for connections. Default is to listen on all the available network interfaces. This configuration is not secure if the system is exposed to Internet. If you choose to listen on all interfaces (we DO NOT recommend this configuration, if the system is exposed to internet), set protected-mode to no. Default port is 6379 .
bind 192.168.56.29
port 6379 
Working directory location, database will be saved here.
 dir /opt/pmx6/var/redis
The name of the database file (this file will be created in the directory specified above).
dbfilename edge.domain-c.vbox_shdb.rdb

Advance configuration (related to performance, fine tune this configuration only if required)

Backup frequency (when to save the database on disk).

save at every 7 minutes, irrespective of keys changed.

save 420 1

save at every 4 minutes if (at least) 500 keys changed.

save 240 500

save at every minute if (at least) 1300 keys changed.

save 60 1000
Maximum number of simultaneous client connections.

Verify if your Linux system (particularly if you are using CentOS) is configured to allow the number of open files grep 'nofile' /etc/security/limits.conf; cat /proc/`pidof redis-server`/limits|grep "Max open files.

maxclients 15000
Do not use AOF method of saving records.
appendonly no 

Note: Verify if the redis service can be reached through the edge server. To ensure this, you may need to adjust firewalling and routing.
Note: Resources system must be initialized (if not, already) on the CSM role for delay queue to work properly. For more information refer:
perldoc pmx-resources-init

2. Optional: Use CSM to share three configuration files: redis_location.conf, delay.conf and policy.siv

These three files are necessary for delay queue. You can either share them with all edge servers using CSM or you can manage them individually on each edge server.
  1. redis_location.conf

    Each edge server needs to know how the redis server can be contacted in order to set and get sender history database entries. The file redis_location.conf should be present on edge server at location: <pmx_directory>/etc/redis_location.conf. Syntax of the file is:

    server_ip = 10.200.193.229
    port = 6379
    Note: Communication between milter and historian DB role will be in plain text. You need to restart the milter service if there is change in this file.
  2. delay.conf
    This file contains parameters that affect the decisions of delaying messages. You need to restart the milter service if there is any change in this file.
    Note: Communication between milter and historian DB role will be in plain text. You need to restart the milter service if there is change in this file.

    This file should be present on each edge server at the location: <pmx_directory>/etc/delay.conf

    delay_status = Collect
    delay_min_minutes = 10
    delay_max_minutes = 60
    delay_max_size = 1024
    send_delayed_mail_to = 127.0.0.1:10025
    delay_status
    Possible values: Collect, On, and Off

    Default: Collect

    Example: delay_status = Collect

    Description:

    Collect: Do not delay mails based on sender history, instead build the sender history database.

    On: Enable snow shoe spam protection

    Off: Disable snow shoe spam protection.

    delay_min_minutes
    Description: minimum time (in minutes) a message can be delayed.

    Default: 2

    Example: delay_min_minutes = 10

    Possible values: The value must not be less than 2 and it must not be greater than or equal to delay_max_minutes.

    delay_max_minutes
    Description: maximum time (in minutes) a message can be delayed.

    Default: 60

    Example: delay_min_minutes = 60

    Possible values: The value must not be greater than 60 and it must not be less than or equal to delay_min_minutes.

    delay_max_size
    Description: Delay queue disk size limit (MB).

    Default: 1024

    Example: delay_min_minutes = 1024

    Possible values: The value must not be less than 256 and it must not be greater than 1024.

    send_delayed_mail_to

    It describes the destination of delayed message when it is time to re-inject the message into the milter.

    For PMX supplied Postfix MTA, refer to the port parameter in the milter section of 'Policy' in the file pmx.conf.

        Example: send_delayed_mail_to = 127.0.0.1:10025

    For PMX supplied Sendmail MTA, use the IP address and port that Sendmail is receiving mails on.

        Example: send_delayed_mail_to = 192.168.1.1:25

    The key here is to use the IP and port so that the mail is reinjected into the same edge server that has processed (and delayed) it before.

    Note: Make sure that relay is accepting the mails (mails that are getting re-injected). If not, mails (that will fill the disk upto the value of delay_max_size) will stack in the delay queue directory until the size of the directory goes beyond the delay_max_size configuration parameter.
  3. policy.siv

    A new test pmx_delayed_mail and a new action pmx_suspect_delay have been added to be used in policy.siv.

    The test pmx_delayed_mail returns true if the message being processed was delayed earlier and is getting reinjected into the milter. Use this test to prevent the actions and test, which are to be applied only to the message when it first arrived, from being applied to the delayed message. For more details, click policy.siv
    Note: It is recommended to use command pmx-policy to edit the policy.siv file to search for syntax error. If you are using any other method to edit the policy (example: a text editor), use command pmx-policy to check the correctness of the policy.

    The action pmx_suspect_delay delays the message for a period specified by the antispam engine. The message will be delayed only if the delay_status is 'On' in <pmx_directory>/etc/delay.conf file. This action should be used in conjuction with the test pmx_spam_hit like following:

    if pmx_spam_hit :comparator "i;ascii-casemap" :matches ["DQ_SUSP_?"]
        {
            pmx_suspect_delay;
        }
    Note: Do not use stop action. It is implicit, if the message is delayed, message processing will stop internally.

    A sample policy script is given that will help learning how this new action and test should be used.

    3. Upgrading the edge servers

    Upgrade the edge servers.