Backup and Restore¶

389 Directory Server¶

Backup¶

A backup procedure for 389 Directory Server consists of three separate steps:

The exection of a pre-script to export the databases.
The inclusion of the exported databases into the backup, along with the directory server configuration directory /etc/dirsrv/.
Clean-up using a post-script.

A sample combined pre- and post-script that could be used:

#!/bin/bash

if [ "$1" == "--before" ]; then
    for dir in `find /etc/dirsrv/ -mindepth 1 -maxdepth 1 -type d \
            -name "slapd-*" | xargs -n 1 basename`; do

        for nsdb in `find /var/lib/dirsrv/${dir}/db/ -mindepth 1 \
                -maxdepth 1 -type d | xargs -n 1 basename`; do

            ns-slapd db2ldif -D /etc/dirsrv/${dir} -n ${nsdb} \
                -a /tmp/$(hostname)-$(echo ${dir} | sed -e 's/slapd-//g')-${nsdb}.ldif \
                >/dev/null 2>&1

        done
    done

elif [ "$1" == "--after" ]; then
    rm -rf /tmp/*.ldif
fi

This will result in a number of files in /tmp/, one per database to be exact, that you will want to include in the backup.

Database Restore¶

For the following procedure to be executed, you must shut down the directory server service:

# service dirsrv stop

After the exported databases have been restored from backup, import the LDIFs back on to the database:

# ns-slapd ldif2db \
    -D /etc/dirsrv/slapd-*$instance_name* \
    -n *$database_name* \
    -i /path/to/exported/database.ldif

where:

$instance_name

is the name of an instance configured on this LDAP server.

$database_name

is the name for the target database the restore should occur to.

Note

The database for the restore should already exist. In the example situation of a migrating an LDAP tree from one LDAP server to another, first add the domain on the target server, so that the database is created, then stop the service, then restore to the created database.

For example, restoring the previously exported root suffix “dc=example,dc=org” to a database named example_org on a system kolab.example.org, run:

# ns-slapd ldif2db \
    -D /etc/dirsrv/slapd-kolab \
    -n example_org \
    -i /tmp/kolab.example.org-kolab-example_org.ldif

Note

The location of the LDIF file to restore from as well as the LDIF file to restore from itself must be accessible for the user account the directory server is configured to run under (usually the unprivileged nobody account).

Cyrus IMAP¶

If you stuck with the defaults, and you are not doing a migration or disaster recovery, you only need to restore from backup what was deleted more than 69 days ago.

Kolab Groupware specifies these defaults to enable you to make sure that whatever ends up in the IMAP spool at any given point in time can be guaranteed to be included in at least 1 full backup, even if you create a full backup only quarterly.

You may have altered this magic number to fit your backup strategy. The default is 3 months – two potentially 31 days, one 30 days – plus 1 week – the margin of error when using “first saturday night of the month”.

Assuming all full backups succeed, even if you cycle full backups monthly, you would need to keep 1 volume around per quarter to comply with regulations your environment may be subject to – noted that you need to keep around the last full backup of any given quarter, and not the first of quarter #1 and the last of quarter #2.

Keep around full backups until you know they are eligible for purging – you never know ahead of time whether the next full backup is going to be successful.

Your exact backup needs, regulatory and/or legal requirements, and an efficient and cost-effective strategy are considered consultancy that `Kolab Systems AG`_ and Certified Partners are more than happy to provide you with.

Backup¶

If you have read the Partitioning section of the Installation Guide, you can consider using LVM snapshots to backup the Cyrus IMAP config and spool directories.

Files and directories to include (or not include) in the backup of Cyrus IMAP include:

/var/spool/imap/

You may want to exclude files named:
- cyrus.squat
- cyrus.cache.NEW
- cyrus.expunge.NEW
- cyrus.index.NEW
as well as directories named stage./ and sync./
/var/lib/imap/

You will want to exclude:
- /var/lib/imap/socket/
- *.lock*

Restore¶

Restore Single Messages (unexpunge)¶

To undelete a message it helps to have some of the following informations available under your fingertips. This will help you to speed up the search and ensure you’re undeleting the correct message.

Sender (mail address)
Date Sent/Receive/Deleted
Folder where the email was stored!
Message Subject

With those information you can easily identify the message.

You must know the exact folder name in which the message was located before it got deleted.

You can find get a listing of the folders belonging the a user using the kolab cli tools.
```
# kolab lm user/john.doe*@example.org
user/john.doe@example.org
user/john.doe/Archive@example.org
user/john.doe/Calendar@example.org
[...]
```

Get a listing of all the deleted/expunged messages in a specific folder, search for the right message and write down the message UID.

# su - -s /bin/bash cyrus -c "/usr/lib/cyrus-imapd/unexpunge -l user/john.doe/Archive@example.org"
UID: 1
        Size: 634
        Sent: Wed Mar  4 12:00:00 2015
        Recv: Wed Mar  4 00:17:33 2015
        Expg: Wed Mar  4 00:22:47 2015
        From: Doe, Jane <jane.doe@example.org>
        To  : Doe, John <john.doe@example.org>
        Cc  :
        Bcc :
        Subj: "Important notes for the next trip!"

Restore the message (single message)

# su - -s /bin/bash cyrus -c "/usr/lib/cyrus-imapd/unexpunge -u -d -v user/john.doe/Archive@example.org 1"
restoring expunged messages in mailbox 'example/org!user/john.doe/Archive'
Unexpunged example/org!user/john.doe/Archive: 1 => 2
restored 1 expunged messages

(Alternative) Restore all message from this folder (double check before!)

# su - -s /bin/bash cyrus -c "/usr/lib/cyrus-imapd/unexpunge -a -d -v user/john.doe/Archive@example.org"
restoring all expunged messages in mailbox 'example/org!user/john.doe/Archive'
Unexpunged example/org!user/john.doe/Archive: 2 => 3
restored 1 expunged messages

When working and the storage layer it sometimes can be useful to reset the caches for a specific user. Kolab componants like iRony are heavy taking use of cache.
```
# cd /usr/share/roundcubemail/
# plugins/libkolab/bin/modcache.sh clear -u john.doe@example.org -h localhost
4 records deleted from 'kolab_folders'
```

Restore a Deleted Folder¶

To restore a complete folder that got deleted you can make use of the kolab cli tools.

Identify the folder that needs to get restored:

# kolab list-deleted-mailboxes
DELETED/user/john.doe/Calendar/54EBC651@example.org (Deleted at 2015-02-24 01:31:13)

Note

the 54EBC651 part is an hex-encoded unixtimestamp

# perl -ke 'print scalar(localtime(hex("54EBC651")));
Thu Feb 24 01:31:13 2015

Undelete the whole folder

# kolab undelete mailbox DELETED/user/john.doe/Calendar/54EBC651@example.org

If you encounter problems think about clearing the cache (see above).

MySQL¶

Databases and Tables You Care About¶

There’s little purpose to backing up caches, so consider either:

Backing them up under a significantly more volatile strategy, or
Not including them in the backup at all.

Backup¶

The backup of MySQL too consists of three steps:

Making the data to backup available in a form the backup program understands,
Including the data in the actual backup,
Cleanup.

For the execution of a pre-script, and therefore what data to include in the backup, as well as the cleanup, a choice should be made between Using LVM Snapshots and Using mysqldump.

Using mysqldump¶

Using LVM Snapshots¶

The larger the MySQL database(s), the less likely you are going to backup MySQL using mysqldump. With larger databases, for one, the contents of the database may change while you’re exporting the database. Luckily, there’s a --single-transaction option to mysqldump, but again in the case of larger MySQL databases, this directly impacts production.

Do NOT use this method if you can afford to run one or more read-only slaves. Use whichever method taking the read-only slave offline temporarily and then back up the data from it.

An alternative to backup MySQL with mysqldump is to use LVM snapshots on a logical volume underneath MySQL:

Flush everything MySQL may have cached to disk,
Lock MySQL,
Create an LVM snapshot,
Unlock MySQL,
Sync the contents of the snapshot anywhere you like.

The trick is in between step 2 and 4, as InnoDB automatically removes any standing locks at the end of the session. You MUST therefore hold on to the session while you create the LVM snapshot:

(
    date > /var/log/backup-mysql.log && \
    echo "FLUSH TABLES WITH READ LOCK;" && \
    sleep $x && \
    lvcreate --size 10G --snapshot \
        --name lv_mysql_snap /dev/vg_db01/lv_mysql >> \
        /var/log/backup-mysql.log 2>&1 && \
    echo "SHOW MASTER STATUS;" && \
    echo "UNLOCK TABLES;" && \
    date >> /var/log/backup-mysql.log && \
    echo "quit" \
) | mysql >> /var/log/backup-mysql.log 2>&1

Creating a subshell with the output of that subshell piped through MySQL keeps the session open while creating a snapshot of the logical volume, however:

The flushing of tables establishing a read lock is a command that returns immediately. As it requests a global read lock, that waits for all other read locks to expire before it actually goes about flushing tables and read locking them, this must not be used for database servers that have long-running queries (which a Kolab Groupware server has not).
Use the $x in “sleep $x” to establish the margin in between issuing the flush command to MySQL, and the snapshot being created.

An increased number for $x would indicate slower storage and/or larger queries.

Warning

While this is actually tested, and it works, please note that you should still test for yourself – using a restore.

On decent database infrastructure with just Kolab Groupware making use of the environment, outside of office and/or peak hours, flushing, locking and snapshot creation can take as little as about 3ms.

Backup and Restore¶

389 Directory Server¶

Backup¶

Database Restore¶

Cyrus IMAP¶

Backup¶

Restore¶

Restore Single Messages (unexpunge)¶

Restore a Deleted Folder¶

MySQL¶

Databases and Tables You Care About¶

Backup¶

Using mysqldump¶

Using LVM Snapshots¶

Table of Contents

Previous topic

Next topic

This Page