obnam man page

obnam — make, restore, and manipulate backups

Synopsis

obnam [--always-restore-setuid] [--no-always-restore-setuid] [--checkpoint=SIZE] [--checksum-algorithm=CHECKSUM] [--chunk-bag-size=SIZE] [--chunk-cache-size=SIZE] [--chunk-size=SIZE] [--chunkids-per-group=NUM] [--client-name=CLIENT-NAME] [--compress-with=PROGRAM] [--config=FILE] [--dump-config] [--dump-setting-names] [--generate-manpage=TEMPLATE] [-h] [--help] [--help-all] [--list-config-files] [--version] [--no-default-configs] [--crash-limit=COUNTER] [--critical-age=AGE] [--deduplicate=MODE] [--dir-bag-size=SIZE] [--dir-cache-size=SIZE] [--dump-memory-profile=METHOD] [--dump-repo-file-metadata] [--no-dump-repo-file-metadata] [--encrypt-with=ENCRYPT-WITH] [--exclude=EXCLUDE] [--exclude-caches] [--no-exclude-caches] [--exclude-from=FILE] [--fsck-fix] [--no-fsck-fix] [--fsck-ignore-chunks] [--no-fsck-ignore-chunks] [--fsck-ignore-client=NAME] [--fsck-last-generation-only] [--no-fsck-last-generation-only] [--fsck-rm-unused] [--no-fsck-rm-unused] [--fsck-skip-checksums] [--no-fsck-skip-checksums] [--fsck-skip-dirs] [--no-fsck-skip-dirs] [--fsck-skip-files] [--no-fsck-skip-files] [--fsck-skip-generations] [--no-fsck-skip-generations] [--fsck-skip-per-client-b-trees] [--no-fsck-skip-per-client-b-trees] [--fsck-skip-shared-b-trees] [--no-fsck-skip-shared-b-trees] [--fuse-opt=FUSE] [--generation=GENERATION] [--gnupghome=HOMEDIR] [--idpath-bits=IDPATH-BITS] [--idpath-depth=IDPATH-DEPTH] [--idpath-skip=IDPATH-SKIP] [--include=INCLUDE] [--keep=KEEP] [--key-details] [--no-key-details] [--keyid=KEYID] [--leave-checkpoints] [--no-leave-checkpoints] [--lock-timeout=TIMEOUT] [--log=FILE] [--log-keep=N] [--log-level=LEVEL] [--log-max=SIZE] [--log-mode=MODE] [--lru-size=SIZE] [--memory-dump-interval=SECONDS] [--node-size=SIZE] [--one-file-system] [--no-one-file-system] [--output=FILE] [--pretend] [--dry-run] [--no-act] [--no-pretend] [--no-dry-run] [--no-no-act] [--pretend-time=TIMESTAMP] [--pure-paramiko] [--no-pure-paramiko] [--quiet] [--silent] [--no-quiet] [--no-silent] [-rURL] [--repository=URL] [--repository-format=FORMAT] [--root=URL] [--sftp-delay=SFTP-DELAY] [--small-files-in-btree] [--no-small-files-in-btree] [--ssh-command=EXECUTABLE] [--ssh-host-keys-check=VALUE] [--ssh-key=FILENAME] [--ssh-known-hosts=FILENAME] [--strict-ssh-host-keys] [--no-strict-ssh-host-keys] [--symmetric-key-bits=BITS] [--testing-fail-matching=REGEXP] [--to=TO] [--trace=TRACE] [--upload-queue-size=SIZE] [--verbose] [--no-verbose] [--verify-randomly=N] [--warn-age=AGE] [--weak-random] [--no-weak-random]

obnam [options] _lock
obnam [options] add-key [CLIENT-NAME]...
obnam [options] backup [DIRECTORY|URL]...
obnam [options] client-keys
obnam [options] clients
obnam [options] diff [GENERATION1]GENERATION2
obnam [options] dump-repo
obnam [options] force-lock
obnam [options] forget [GENERATION]...
obnam [options] fsck
obnam [options] generations
obnam [options] genids
obnam [options] help
obnam [options] help-all
obnam [options] kdirstat [FILE]...
obnam [options] list-errors
obnam [options] list-formats
obnam [options] list-keys
obnam [options] list-toplevels
obnam [options] ls [FILE]...
obnam [options] mount [ROOT]
obnam [options] nagios-last-backup-age
obnam [options] remove-client [CLIENT-NAME]...
obnam [options] remove-key [CLIENT-NAME]...
obnam [options] restore [DIRECTORY]...
obnam [options] verify [DIRECTORY]...

Description

obnam makes, restores, manipulates, and otherwise deals with backups. It can store backups on a local disk or to a server via sftp. Every backup generation looks like a fresh snapshot, but is really incremental: the user does not need to worry whether it's a full backup or not. Only changed data is backed up, and if a chunk of data is already backed up in another file, that data is re-used.

The place where backed up data is placed is called the backup repository. A repository may be, for example, a directory on an sftp server, or a directory on a USB hard disk. A single repository may contain backups from several clients. Their data will intermingle as if they were using separate repositories, but if one client backs up a file, the others may re-use the data.

obnam command line syntax consists of a command possibly followed by arguments. The commands are list below.

Making backups

When you run a backup, obnam uploads data into the backup repository. The data is divided into chunks, and if a chunk already exists in the backup repository, it is not uploaded again. This allows obnam to deal with files that have been changed or renamed since the previous backup run. It also allows several backup clients to avoid uploading the same data. If, for example, everyone in the office has a copy of the same sales brochures, only one copy needs to be stored in the backup repository.

Every backup run is a generation. In addition, obnam will make checkpoint generations every now and then. These are exactly like normal generations, but are not guaranteed to be a complete snapshot of the live data. If the backup run needs to be aborted in the middle, the next backup run can continue from the latest checkpoint, avoiding the need to start over from scratch.

If one backup run drops a backup root directory, the older generations will still keep it: nothing changes in the old generations just because there is a new one. If the root was dropped by mistake, it can be added back and the next backup run will re-use the existing data in the backup repository, and will only back up the file metadata (filenames, permissions, etc).

Verifying backups

What good is a backup system you cannot rely on? How can you rely on something you cannot test? The obnam verify command checks that data in the backup repository matches actual user data. It retrieves one or more files from the repository and compares them to the user data. This is essentially the same as doing a restore, then comparing restored files with the original files using cmp(1), but easier to use.

By default, verification happens on all files. You can also specify the files to be verified by listing them on the command line. You should specify the full paths to the files, not relative to the current directory.

The output lists files that fail verification for some reason. If you verify everything, it is likely that some files (e.g., parent directories of backup root) may have changed without it being a problem. Note that you will need to specify the whole path to the files or directories to be verified, not relative to the backup root. You still need to specify at least one of the backup roots on the command line or via the --root option so that obnam will find the filesystem, in case it is a remote one.

URL syntax

Whenever obnam accepts a URL, it can be either a local pathname, or an sftp URL. An sftp URL has the following form:

sftp://[user@]domain[:port]/path

where domain is a normal Internet domain name for a server, user is your username on that server, port is an optional port number, and path is a pathname on the server side. Like bzr(1), but unlike the sftp URL standard, the pathname is absolute, unless it starts with /~/ in which case it is relative to the user's home directory on the server.

See the Example section for examples of URLs.

You can use sftp URLs for the repository, or the live data (root), but note that due to limitations in the protocol, and its implementation in the paramiko library, some things will not work very well for accessing live data over sftp. Most importantly, the handling of of hardlinks is rather suboptimal. For live data access, you should not end the URL with /~/ and should append a dot at the end in this special case.

Generation specifications

When not using the latest generation, you will need to specify which one you need. This will be done with the --generation option, which takes a generation specification as its argument. The specification is either the word latest, meaning the latest generation (also the default), or a number. See the generations command to see what generations are available, and what their numbers are.

Policy for keeping and removing backup generations

The forget command can follow a policy to automatically keep some and remove other backup generations. The policy is set with the --keep=POLICY option.

POLICY is comma-separated list of rules. Each rule consists of a count and a time period. The time periods are h, d, w, m, and y, for hour, day, week, month, and year.

A policy of 30d means to keep the latest backup for each day when a backup was made, and keep the last 30 such backups. Any backup matched by any policy rule is kept, and any backups in between will be removed, as will any backups older than the oldest kept backup.

As an example, assume backups are taken every hour, on the hour: at 00:00, 01:00, 02:00, and so on, until 23:00. If the forget command is run at 23:15, with the above policy, it will keep the backup taken at 23:00 on each day, and remove every other backup that day. It will also remove backups older than 30 days.

If backups are made every other day, at noon, forget would keep the 30 last backups, or 60 days worth of backups, with the above policy.

Note that obnam will only inspect timestamps in the backup repository, and does not care what the actual current time is. This means that if you stop making new backups, the existing ones won't be removed automatically. In essence, obnam pretends the current time is just after the latest backup when forget is run.

The rules can be given in any order, but will be sorted to ascending order of time period before applied. (It is an error to give two rules for the same period.) A backup generation is kept if it matches any rule.

For example, assume the same backup frequency as above, but a policy of 30d,52w. This will keep the newest daily backup for each day for thirty days, and the newest weekly backup for 52 weeks. Because the hourly backups will be removed daily, before they have a chance to get saved by a weekly rule, the effect is that the 23:00 o'clock backup for each day is saved for a month, and the 23:00 backup on Sundays is saved for a year.

If, instead, you use a policy of 72h,30d,52w, obnam would keep the last 72 hourly backups, and the last backup of each calendar day for 30 days, and the last backup of each calendar week for 52 weeks. If the backup frequency was once per day, obnam would keep the backup of each calendar hour for which a backup was made, for 72 such backups. In other words, it would effectively keep the last 72 daily backups.

Sound confusing? Just think how confused the developer was when writing the code.

If no policy is given, forget will keep everything.

A typical policy might be 72h,7d,5w,12m, which means: keep the last 72 hourly backups, the last 7 daily backups, the last 5 weekly backups and the last 12 monthly backups. If the backups are systematically run on an hourly basis, this will mean keeping hourly backups for three days, daily backups for a week, weekly backups for a month, and monthly backups for a year.

The way the policy works is a bit complicated. Run forget with the --pretend option to make sure you're removing the right ones.

Using encryption

obnam can encrypt all the data it writes to the backup repository. It uses gpg(1) to do the encryption. You need to create a key pair using gpg --gen-key (or use an existing one), and then tell obnam about it using the --encrypt-with option. You may optionally use a separate home directory using the --gnupghome option. By default, the default directory for gpg(1) will be used.

Configuration files

obnam will look for configuration files in a number of locations. See the Files section for a list. All these files together are treated as one big file with the contents of all files concatenated.

The files are in INI format, and only the [config] section is used (any other sections are ignored).

The long names of options are used as keys for configuration variables. Any setting that can be set from the command line can be set in a configuration file, in the [config] section.

For example, the options in the following command line:

obnam --repository=/backup --exclude='.wav$' backup

could be replaced with the following configuration file:

[config]
repository: /backup
exclude: .wav$

(You can use either foo=value or foo: value syntax in the files.)

The only unusual thing about the files is the way options that can be used many times are expressed. All values are put in a single logical line, separated by commas (and optionally spaces as well). For example:

[config]
exclude = foo, bar, \.mp3$

The above has three values for the exclude option: any files that contain the words foo or bar anywhere in the fully qualified pathname, or files with names ending with a period and mp3 (because the exclusions are regular expressions).

A long logical line can be broken into several physical ones, by starting a new line at white space, and indenting the continuation lines:

[config]
exclude = foo,
    bar,
    \.mp3$

The above example adds three exclusion patterns.

Multiple clients and locking

obnam supports sharing a repository between multiple clients. The clients can share the file contents (chunks), so that if client A backs up a large file, and client B has the same file, then B does not need to upload the large file to the repository a second time. For this to work without confusion, the clients use a simple locking protocol that allows only one client at a time to modify the shared data structures. Locks do not prevent read-only access at the same time: this allows you to restore while someone else is backing up.

Sometimes a read-only operation happens to access a data structure at the same time as it is being modified. This can result in a crash. It will not result in corrupt data, or incorrect restores. However, you may need to restart the read-only operation after a crash.

Options

--always-restore-setuid

restore setuid/setgid bits in restored files, even if not root or backed up file had different owner than user running restore

--no-always-restore-setuid

opposite of --always-restore-setuid

--checksum-algorithm=CHECKSUM

use CHECKSUM for checksum algorithm (not for repository format 6); one of: sha512, sha224, sha256, sha384

--client-name=CLIENT-NAME

name of client (defaults to hostname)

--compress-with=PROGRAM

use PROGRAM to compress repository with (one of none, deflate)

--critical-age=AGE

for nagios-last-backup-age: maximum age (by default in hours) for the most recent backup before statis is critical. Accepts one char unit specifier (s,m,h,d for seconds, minutes, hours, and days.

--dump-repo-file-metadata

dump metadata about files?

--no-dump-repo-file-metadata

opposite of --dump-repo-file-metadata

--generate-manpage=TEMPLATE

fill in manual page TEMPLATE

--generation=GENERATION

which generation to restore

-h, --help

show this help message and exit

--keep=KEEP

policy for what generations to keep when forgetting

--lock-timeout=TIMEOUT

when locking in the backup repository, wait TIMEOUT seconds for an existing lock to go away before giving up

--output=FILE

write output to FILE, instead of standard output

--pretend, --dry-run, --no-act

do not actually change anything (works with backup, forget and restore only, and may only simulate approximately real behavior)

--no-pretend, --no-dry-run, --no-no-act

opposite of --pretend

--quiet, --silent

be silent: show only error messages, no progress updates

--no-quiet, --no-silent

opposite of --quiet

-r, --repository=URL

name of backup repository (can be pathname or supported URL)

--repository-format=FORMAT

use FORMAT for new repositories; one of "6", "green-albatross-20160813"

--to=TO

where to restore or FUSE mount; for restores, must be empty or must not exist

--verbose

be verbose: tell the user more of what is going on and generally make sure the user is aware of what is happening or at least that something is happening and also make sure their screen is getting updates frequently and that there is changes happening all the time so they do not get bored and that they in fact get frustrated by getting distracted by so many updates that they will move into the Gobi desert to live under a rock

--no-verbose

opposite of --verbose

--verify-randomly=N

verify N files randomly from the backup (default is zero, meaning everything)

--version

show program's version number and exit

--warn-age=AGE

for nagios-last-backup-age: maximum age (by default in hours) for the most recent backup before status is warning. Accepts one char unit specifier (s,m,h,d for seconds, minutes, hours, and days.

Backing up

--checkpoint=SIZE

make a checkpoint after a given SIZE

--deduplicate=MODE

find duplicate data in backed up data and store it only once; three modes are available: never de-duplicate, verify that no hash collisions happen, or (the default) fatalistically accept the risk of collisions

--exclude=EXCLUDE

regular expression for pathnames to exclude from backup (can be used multiple times)

--exclude-caches

exclude directories (and their subdirs) that contain a CACHEDIR.TAG file (see http://www.brynosaurus.com/cachedir/spec.html for what it needs to contain, and http://liw.fi/cachedir/ for a helper tool)

--no-exclude-caches

opposite of --exclude-caches

--exclude-from=FILE

read exclude patterns from FILE

--include=INCLUDE

regular expression for pathnames to include from backup even if it matches an exclude rule (can be used multiple times)

--leave-checkpoints

leave checkpoint generations at the end of a successful backup run

--no-leave-checkpoints

opposite of --leave-checkpoints

--one-file-system

exclude directories (and their subdirs) that are in a different filesystem

--no-one-file-system

opposite of --one-file-system

--root=URL

what to backup

--small-files-in-btree

this is available only for backwards compatibility; do not use it, and remove it from your configuration

--no-small-files-in-btree

opposite of --small-files-in-btree

Configuration files and settings

--config=FILE

add FILE to config files

--dump-config

write out the entire current configuration

--dump-setting-names

write out all names of settings and quit

--help-all

show all options

--list-config-files

list all possible config files

--no-default-configs

clear list of configuration files to read

Development of Obnam itself

--crash-limit=COUNTER

artificially crash the program after COUNTER files written to the repository; this is useful for crash testing the application, and should not be enabled for real use; set to 0 to disable (disabled by default)

--pretend-time=TIMESTAMP

pretend it is TIMESTAMP (YYYY-MM-DD HH:MM:SS); this is only useful for testing purposes

--sftp-delay=SFTP-DELAY

add an artificial delay (in milliseconds) to all SFTP transfers

--testing-fail-matching=REGEXP

development testing helper: simulate failures during backup for files that match the given regular expressions

--trace=TRACE

add to filename patters for which trace debugging logging happens

Encryption

--encrypt-with=ENCRYPT-WITH

PGP key with which to encrypt data in the backup repository

--gnupghome=HOMEDIR

home directory for GPG

--key-details

show additional user IDs for all keys

--no-key-details

opposite of --key-details

--keyid=KEYID

PGP key id to add to/remove from the backup repository

--symmetric-key-bits=BITS

size of symmetric key, in bits

--weak-random

use /dev/urandom instead of /dev/random to generate symmetric keys

--no-weak-random

opposite of --weak-random

Integrity checking (fsck)

--fsck-fix

should fsck try to fix problems? Implies --fsck-rm-unused

--no-fsck-fix

opposite of --fsck-fix

--fsck-ignore-chunks

ignore chunks when checking repository integrity (assume all chunks exist and are correct)

--no-fsck-ignore-chunks

opposite of --fsck-ignore-chunks

--fsck-ignore-client=NAME

do not check repository data for client NAME

--fsck-last-generation-only

check only the last generation for each client

--no-fsck-last-generation-only

opposite of --fsck-last-generation-only

--fsck-rm-unused

should fsck remove unused chunks?

--no-fsck-rm-unused

opposite of --fsck-rm-unused

--fsck-skip-checksums

do not check checksums of files

--no-fsck-skip-checksums

opposite of --fsck-skip-checksums

--fsck-skip-dirs

do not check anything about directories and their files

--no-fsck-skip-dirs

opposite of --fsck-skip-dirs

--fsck-skip-files

do not check anything about files

--no-fsck-skip-files

opposite of --fsck-skip-files

--fsck-skip-generations

do not check any generations

--no-fsck-skip-generations

opposite of --fsck-skip-generations

--fsck-skip-per-client-b-trees

do not check per-client B-trees

--no-fsck-skip-per-client-b-trees

opposite of --fsck-skip-per-client-b-trees

--fsck-skip-shared-b-trees

do not check shared B-trees

--no-fsck-skip-shared-b-trees

opposite of --fsck-skip-shared-b-trees

Logging

--log=FILE

write log entries to FILE (default is to not write log files at all); use "syslog" to log to system log, "stderr" to log to the standard error output, or "none" to disable logging

--log-keep=N

keep last N logs (10)

--log-level=LEVEL

log at LEVEL, one of debug, info, warning, error, critical, fatal (default: info)

--log-max=SIZE

rotate logs larger than SIZE, zero for never (default: 0)

--log-mode=MODE

set permissions of new log files to MODE (octal; default 0600)

Mounting with FUSE

--fuse-opt=FUSE

options to pass directly to Fuse

Peformance

--dump-memory-profile=METHOD

make memory profiling dumps using METHOD, which is one of: none, simple, or meliae (default: simple)

--memory-dump-interval=SECONDS

make memory profiling dumps at least SECONDS apart

Performance tweaking

--chunk-size=SIZE

size of chunks of file data backed up

--chunkids-per-group=NUM

encode NUM chunk ids per group

--idpath-bits=IDPATH-BITS

chunk id level size

--idpath-depth=IDPATH-DEPTH

depth of chunk id mapping

--idpath-skip=IDPATH-SKIP

chunk id mapping lowest bits skip

--lru-size=SIZE

size of LRU cache for B-tree nodes

--node-size=SIZE

size of B-tree nodes on disk; only affects new B-trees so you may need to delete a client or repository to change this for existing repositories

--upload-queue-size=SIZE

length of upload queue for B-tree nodes

Repository format green-albatross

--chunk-bag-size=SIZE

approximate maximum size of bag combining many chunk objects

--chunk-cache-size=SIZE

size of in-memory cache for file data chunk objects

--dir-bag-size=SIZE

approximate maximum size of bags combining many DIR objects

--dir-cache-size=SIZE

size of in-memory cache for DIR objects

SSH/Sftp

--pure-paramiko

do not use openssh even if available, use paramiko only instead

--no-pure-paramiko

opposite of --pure-paramiko

--ssh-command=EXECUTABLE

alternative executable to be used instead of "ssh" (full path is allowed, no arguments may be added)

--ssh-host-keys-check=VALUE

If "yes", require that the ssh host key must be known and correct to be accepted. If "no", do not require that. If "ask", the user is interactively asked to accept new hosts. The default ("ssh-config") is to rely on the settings of the underlying SSH client

--ssh-key=FILENAME

use FILENAME as the ssh RSA private key for sftp access (default is using keys known to ssh-agent)

--ssh-known-hosts=FILENAME

filename of the user's known hosts file

--strict-ssh-host-keys

DEPRECATED, use --ssh-host-keys-check instead

--no-strict-ssh-host-keys

opposite of --strict-ssh-host-keys

Option values

The SIZE value in options mentioned above specifies a size in bytes, with optional suffixes to indicate kilobytes (k), kibibytes (Ki), megabytes (M), mebibyts (Mi), gigabytes (G), gibibytes (Gi), terabytes (T), tibibytes (Ti). The suffixes are case-insensitive.

Exit Status

obnam will exit with zero if everything went well, and non-zero otherwise.

Environment

obnam will pass on the environment it gets from its parent, without modification. It does not obey any unusual environment variables, but it does obey the usual ones when running external programs, creating temporary files, etc.

Files

/etc/obnam.conf
/etc/obnam/*.conf
~/.obnam.conf
~/.config/obnam/*.conf

Configuration files for obnam. It is not an error for any or all of the files to not exist.

Example

To back up your home directory to a server:

obnam backup --repository sftp://your.server/~/backups $HOME

To restore your latest backup from the server:
        obnam restore --repository sftp://your.server/~/backups \
    --to /var/tmp/my.home.dir

To restore just one file or directory:
        obnam restore --repository sftp://your.server/~/backups \
    --to /var/tmp/my.home.dir $HOME/myfile.txt

Alternatively, mount the backup repository using the FUSE filesystem (note that the --to option is necessary):

mkdir my-repo
obnam mount --repository sftp://your.server/~/backups \
--to my-repo
cp my-repo/latest/$HOME/myfile.txt
fusermount -u my-repo

To check that the backup worked:
        obnam verify --repository sftp://your.server/~/backups \
    /path/to/file

To remove old backups, keeping the newest backup for each day for
ten years:
        obnam forget --repository sftp://your.server/~/backups \
    --keep 3650d

To verify that the backup repository is OK:
        obnam fsck --repository sftp://your.server/~/backups

To view the backed up files in the backup repository using FUSE:

obnam mount --to my-fuse
ls -lh my-fuse
fusermount -u my-fuse

See Also

obnam comes with a manual in HTML and PDF forms. See /usr/share/doc/obnam if you have Obnam installed system-wide, or in the subdirectory manual in the source tree.

cliapp(5)

Referenced By

obnam-viewprof(1), seivot(1), seivots-summary(1).