Postfix OpenLDAP LMDB Howto


Introduction

Warning: LMDB databases have a show-stopper bug: they do not grow beyond a specified limit, and introduce a "database full" hard error condition that does not exist with any other Postfix database type. This is a problem for programs whose database grows with system load. Examples are postscreen(8), greylisting, verify(8) and tlsmgr(8).

You have been warned.

Postfix uses databases of various kinds to store and look up information. Postfix databases are specified as "type:name". OpenLDAP LMDB implements the Postfix database type "lmdb". The name of a Postfix OpenLDAP LMDB database is the name of the database file without the ".lmdb" suffix. OpenLDAP LMDB databases are maintained with the postmap(1) and postalias(1) commands.

This document describes:

  1. How to build Postfix with OpenLDAP LMDB support.

  2. How to configure LMDB settings.

  3. Missing pthread library trouble.

  4. Unexpected failure modes that don't exist with other Postfix databases.

Building Postfix with OpenLDAP LMDB support

Postfix normally does not enable OpenLDAP LMDB support. To build Postfix after you installed OpenLDAP LMDB from source code, use something like:

% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
    AUXLIBS="-L/usr/local/lib -llmdb"
% make

Solaris needs this:

% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
    AUXLIBS="-R/usr/local/lib -L/usr/local/lib -llmdb"
% make

The exact pathnames depend on how OpenLDAP LMDB was installed.

Configure LMDB settings

Postfix provides a configuration parameter that controls how large an OpenLDAP LMDB database may grow.

Missing pthread library trouble

When building Postfix fails with:

undefined reference to `pthread_mutexattr_destroy'
undefined reference to `pthread_mutexattr_init'
undefined reference to `pthread_mutex_lock'

Add the "-lpthread" library to the "make makefiles" command.

% make makefiles .... AUXLIBS="... -lpthread"

Source code for OpenLDAP LMDB is available at http://www.openldap.org. More information is available at http://highlandsun.com/hyc/mdb/.

Unexpected failure modes of Postfix LMDB databases.

As documented below, conversion to LMDB introduces a number of failure modes that don't exist with other Postfix databases.

Unexpected postmap(1)/postalias(1) "database full" errors.

Problem:

The "postmap lmdb:filename" command fails with an MDB_MAP_FULL error. This problem does not exist with other Postfix databases.

Background:

LMDB databases have a hard size limit (configured with the lmdb_map_size configuration parameter).

When executing "postmap lmdb:filename", the Postfix LMDB database client stores the new data in a transaction which takes up space in addition to the existing data, and commits the transaction when it closes the database. Only then can the space for old data be reused.

Impact:

This failure does not affect Postfix availability, because the old data still exists in the database.

Mitigation:

When the postmap(1) or postalias(1) command fails with an MDB_MAP_FULL error, it expands the database file size to the current LMDB map size limit before terminating.

When the postmap(1) or postalias(1) command opens an LMDB file larger than lmdb_map_size/3, it logs a warning and uses a larger size limit instead:

warning: filename.lmdb: file size 15024128 ≥ (lmdb map size limit 16777216)/3 -- using a larger map size limit

The two steps above can be used to automate recovery and avoid the need for human intervention. Just repeat "postmap lmdb:filename" (up to some limit). After each failure it will use a 3x larger size limit, and eventually the "database full" error will disappear.

Prevention:

Monitor your LMDB files and make sure that lmdb_map_size > 3x the largest LMDB file size.

Unexpected Postfix daemon "database full" errors.

Problem:

Postfix daemon programs fail with "database full" errors, such as postscreen(8), tlsmgr(8) or verify(8). This problem does not exist with other Postfix databases.

Impact:

This failure temporarily affects Postfix availability. The daemon restarts automatically and tries to open the database again as described next.

Mitigation:

When a Postfix daemon opens an LMDB file larger than lmdb_map_size/3, it logs a warning and uses a larger size limit instead:

warning: filename.lmdb: file size 15024128 ≥ (lmdb map size limit 16777216)/3 -- using a larger map size limit

This can be used to automate recovery and avoid the need for human intervention. Each time the daemon runs into a "database full" error, it restarts and uses a 3x larger size limit. The "database full" error will disappear, at least for a while.

Prevention:

Monitor your LMDB files and make sure that lmdb_map_size > 3x the largest LMDB file size.

Non-obvious recovery with postmap(1)/postalias(1)/tlsmgr(8) from a corrupted database.

Problem:

You cannot rebuild a corrupted LMDB database simply by running postmap(1) or postalias(1), or by waiting until the tlsmgr(8) daemon restarts automatically. This problem does not exist with other Postfix databases.

Background:

The Postfix LMDB database client does not truncate the database file. Instead it attempts to create a transaction for a "drop" request plus subsequent "store" requests. That is obviously not possible with a corrupted database file.

Impact:

Postfix does not process mail until someone fixes the problem.

Recovery:

First delete the ".lmdb" file by hand, then rebuild the file with the postmap(1) or postalias(1) command, or wait until the tlsmgr(8) daemon restarts automatically.

Prevention:

Arrange your file systems such that they never run out of free space.

Use ECC memory to detect and correct silent corruption of in-memory file system data and metadata.

Use a file system such as ZFS to detect and correct silent corruption of on-disk file system data and metadata.