vdirsyncer man page

vdirsyncer — vdirsyncer Documentation

·
Documentation
·
Source code

Vdirsyncer synchronizes your calendars and addressbooks between two storages. The most popular purpose is to synchronize a CalDAV/CardDAV server with a local folder or file. The local data can then be accessed via a variety of programs, none of which have to know or worry about syncing to a server.

It aims to be for CalDAV and CardDAV what OfflineIMAP is for IMAP.

When Do I Need Vdirsyncer?

Why not Dropbox + todo.txt?

Projects like todo.txt criticize the complexity of modern productivity apps, and that rightfully. So they set out to create a new, super-simple, human-readable format, such that vim suffices for viewing the raw data. However, when they're faced with the question how to synchronize that data across multiple devices, they seemed to have reached the dead end with their novel idea: "Let's just use Dropbox".

What does file sync software do if both files have changed since the last sync? The answer is to ignore the question, just sync as often as possible, and hope for the best. Because if it comes to a sync conflict, most sync services are not daring to merge files, and create two copies on each computer instead. Merging the two task lists is left to the user.

A better idea would've been to use git to synchronize the todo.txt file, which is at least able to resolve some basic conflicts.

Why not file sync (Dropbox, git, ...) + vdir?

Since vdirs are just a bunch of files, it is obvious to try file synchronization for synchronizing your data between multiple computers, such as:

·
Syncthing
·
Dropbox or one of the gajillion services like it
·
unison
·
Just git with a sshd.

The disadvantages of those solutions largely depend on the exact file sync program chosen:

·
Like with todo.txt, Dropbox and friends are obviously agnostic/unaware of the files' contents. If a file has changed on both sides, Dropbox just copies both versions to both sides.

This is a good idea if the user is directly interfacing with the file system and is able to resolve conflicts themselves. Here it might lead to errorneous behavior with e.g. khal, since there are now two events with the same UID.

This point doesn't apply to git: It has very good merging capabilities, better than what vdirsyncer currently has.
·
Such a setup doesn't work at all with smartphones. Vdirsyncer, on the other hand, synchronizes with CardDAV/CalDAV servers, which can be accessed with e.g. DAVDroid or the apps by dmfs.

Tutorial

Before starting, consider if you actually need vdirsyncer. There are better alternatives available for particular usecases.

Installation

Unless you want to contribute to vdirsyncer, you should use the packages from your distribution:

·
ArchLinux (AUR)
·
pkgsrc
·
Fedora
·
nixpkg
·
GNU Guix
·
homebrew
·
Gentoo
·
Debian and Ubuntu don't have packages, but make a manual installation especially hard. See debian-urllib3.

If there is no package for your distribution, you'll need to install vdirsyncer manually. There is an easy command to copy-and-paste for this as well, but you should be aware of its consequences.

Configuration

NOTE:

·
The config.example from the repository contains a very terse version of this.
·
In this example we set up contacts synchronization, but calendar sync works almost the same. Just swap type = carddav for type = caldav and fileext = .vcf for fileext = .ics.
·
Take a look at the problems page if anything doesn't work like planned.

By default, vdirsyncer looks for its configuration file in the following locations:

·
The file pointed to by the VDIRSYNCER_CONFIG environment variable.
·
~/.vdirsyncer/config.
·
$XDG_CONFIG_HOME/vdirsyncer/config, which is normally ~/.config/vdirsyncer/config. See the XDG-Basedir specification.

The config file should start with a general section, where the only required parameter is status_path. The following is a minimal example:

[general]
status_path = ~/.vdirsyncer/status/

After the general section, an arbitrary amount of pair and storage sections might come.

In vdirsyncer, synchronization is always done between two storages. Such storages are defined in storage sections, and which pairs of storages should actually be synchronized is defined in pair section. This format is copied from OfflineIMAP, where storages are called repositories and pairs are called accounts.

The following example synchronizes ownCloud's default addressbook to ~/.contacts/:

[pair my_contacts]
a = my_contacts_local
b = my_contacts_remote
collections = null

[storage my_contacts_local]
type = filesystem
path = ~/.contacts/
fileext = .vcf

[storage my_contacts_remote]
type = carddav
url = https://owncloud.example.com/remote.php/carddav/addressbooks/bob/default/
username = bob
password = asdf

NOTE:

Configuration for other servers can be found at supported-servers.

After running vdirsyncer discover and vdirsyncer sync, ~/.contacts/ will contain a bunch of .vcf files which all contain a contact in VCARD format each. You can modify their content, add new ones and delete some [1], and your changes will be synchronized to the CalDAV server after you run vdirsyncer sync again. For further reference, it uses the storages filesystem and carddav.

[1]
You'll want to use a helper program for this.

More Configuration

Conflict resolution

What if the same item is changed on both sides? What should vdirsyncer do? By default, it will show an ugly error message, which is surely a way to avoid the problem. Another way to solve that ambiguity is to add another line to the pair section:

[pair my_contacts]
...
conflict_resolution = b wins

Earlier we wrote that b = my_contacts_remote, so when vdirsyncer encounters the situation where an item changed on both sides, it will simply overwrite the local item with the one from the server. Of course a wins is also a valid value.

Collection discovery

The above configuration only syncs a single addressbook. This is denoted by collections = null (collection = addressbook/calendar). We can change this line to let vdirsyncer automatically sync all addressbooks it can find:

[pair my_contacts]
a = my_contacts_local
b = my_contacts_remote
collections = ["from a", "from b"]  # changed from `null`

[storage my_contacts_local]
type = filesystem
path = ~/.contacts/
fileext = .vcf

[storage my_contacts_remote]
type = carddav

# We can simplify this URL here as well. In theory it shouldn't matter.
url = https://owncloud.example.com/remote.php/carddav/
username = bob
password = asdf

With the above configuration, vdirsyncer discover will fetch all available collections from the server, and create subdirectories for each of them in ~/.contacts/ after confirmation. For example, ownCloud's default addressbook "default" would be synchronized to the location ~/.contacts/default/.

After that, vdirsyncer sync will synchronize all your addressbooks as expected. However, if new collections are created on the server, it will not automatically start synchronizing those [2]. You need to run vdirsyncer discover again to re-fetch this list instead.

[2]
Because collections are added rarely, and checking for this case before every synchronization isn't worth the overhead.

Metadata synchronization

Besides items, vdirsyncer can also synchronize metadata like the addressbook's or calendar's "human-friendly" name (internally called "displayname") or the color associated with a calendar. For the purpose of explaining this feature, let's switch to a different base example. This time we'll synchronize calendars:

[pair my_calendars]
a = my_calendars_local
b = my_calendars_remote
collections = ["from a", "from b"]
metadata = ["color"]

[storage my_calendars_local]
type = filesystem
path = ~/.calendars/
fileext = .ics

[storage my_calendars_remote]
type = caldav

url = https://owncloud.example.com/remote.php/caldav/
username = bob
password = asdf

Run vdirsyncer discover for discovery. Then you can use vdirsyncer metasync to synchronize the color property between your local calendars in ~/.calendars/ and your ownCloud. Locally the color is just represented as a file called color within the calendar folder.

SSL and Certificate Validation

All SSL configuration is done per-storage.

Pinning by fingerprint

To pin the certificate by fingerprint:

[storage foo]
type = caldav
...
verify_fingerprint = "94:FD:7A:CB:50:75:A4:69:82:0A:F8:23:DF:07:FC:69:3E:CD:90:CA"
#verify = false  # Optional: Disable CA validation, useful for self-signed certs

SHA1-, SHA256- or MD5-Fingerprints can be used. They're detected by their length.

You can use the following command for obtaining a SHA-1 fingerprint:

echo -n | openssl s_client -connect unterwaditzer.net:443 | openssl x509 -noout -fingerprint

Custom root CAs

To point vdirsyncer to a custom set of root CAs:

[storage foo]
type = caldav
...
verify = "/path/to/cert.pem"

Vdirsyncer uses the requests library, which, by default, uses its own set of trusted CAs.

However, the actual behavior depends on how you have installed it. Some Linux distributions, such as Debian, patch their python-requests package to use the system certificate CAs. Normally these two stores are similar enough for you to not care.

But there are cases where certificate validation fails even though you can access the server fine through e.g. your browser. This usually indicates that your installation of the requests library is somehow broken. In such cases, it makes sense to explicitly set verify or verify_fingerprint as shown above.

Client Certificates

Client certificates may be specified with the auth_cert parameter. If the key and certificate are stored in the same file, it may be a string:

[storage foo]
type = caldav
...
auth_cert = "/path/to/certificate.pem"

If the key and certificate are separate, a list may be used:

[storage foo]
type = caldav
...
auth_cert = ["/path/to/certificate.crt", "/path/to/key.key"]

Storing Passwords

Changed in version 0.7.0: Password configuration got completely overhauled.

Vdirsyncer can fetch passwords from several sources other than the config file.

Command

Say you have the following configuration:

[storage foo]
type = caldav
url = ...
username = foo
password = bar

But it bugs you that the password is stored in cleartext in the config file. You can do this:

[storage foo]
type = caldav
url = ...
username = foo
password.fetch = ["command", "~/get-password.sh", "more", "args"]

You can fetch the username as well:

[storage foo]
type = caldav
url = ...
username.fetch = ["command", "~/get-username.sh"]
password.fetch = ["command", "~/get-password.sh"]

Or really any kind of parameter in a storage section.

Accessing the system keyring

As shown above, you can use the command strategy to fetch your credentials from arbitrary sources. A very common usecase is to fetch your password from the system keyring.

The keyring Python package contains a command-line utility for fetching passwords from the OS's password store. Installation:

pip install keyring

Basic usage:

password.fetch = ["command", "keyring", "get", "example.com", "foouser"]

Password Prompt

You can also simply prompt for the password:

[storage foo]
type = caldav
username = myusername
password.fetch = ["prompt", "Password for CalDAV"]

Full Configuration Manual

Vdirsyncer uses an ini-like format for storing its configuration. All values are JSON, invalid JSON will get interpreted as string:

x = "foo"  # String
x = foo  # Shorthand for same string

x = 42  # Integer

x = ["a", "b", "c"]  # List of strings

x = true  # Boolean
x = false

x = null  # Also known as None

General Section

[general]
status_path = ...
·
status_path: A directory where vdirsyncer will store some additional data for the next sync.

The data is needed to determine whether a new item means it has been added on one side or deleted on the other. Relative paths will be interpreted as relative to the configuration file's directory.

See A simple synchronization algorithm for what exactly is in there.

Pair Section

[pair pair_name]
a = ...
b = ...
#collections = null
#conflict_resolution = null
·
Pair names can consist of any alphanumeric characters and the underscore.
·
a and b reference the storages to sync by their names.
·

collections: A list of collections to synchronize when vdirsyncer sync is executed.

The special values "from a" and "from b", tell vdirsyncer to try autodiscovery on a specific storage.

If the collection you want to sync doesn't have the same name on each side, you may also use a value of the form ["config_name", "name_a", "name_b"]. This will synchronize the collection name_a on side A with the collection name_b on side B. The config_name will be used for representation in CLI arguments and logging.

Examples:

·
collections = ["from b", "foo", "bar"] makes vdirsyncer synchronize the collections from side B, and also the collections named "foo" and "bar".
·
collections = ["from b", "from a"] makes vdirsyncer synchronize all existing collections on either side.
·
collections = [["bar", "bar_a", "bar_b"], "foo"] makes vdirsyncer synchronize bar_a from side A with bar_b from side B, and also synchronize foo on both sides with each other.
·

conflict_resolution: Optional, define how conflicts should be handled. A conflict occurs when one item (event, task) changed on both sides since the last sync.

Valid values are:

·
"a wins" and "b wins", where the whole item is taken from one side. Vdirsyncer will not attempt to merge the two items.
·
null, the default, where an error is shown and no changes are done.
·

metadata: Metadata keys that should be synchronized when vdirsyncer metasync is executed. Example:

metadata = ["color", "displayname"]

This synchronizes the color and the displayname properties. The conflict_resolution parameter applies here as well.

Storage Section

[storage storage_name]
type = ...
·
Storage names can consist of any alphanumeric characters and the underscore.
·
type defines which kind of storage is defined. See Supported Storages.
·
read_only defines whether the storage should be regarded as a read-only storage. The value true means synchronization will discard any changes made to the other side. The value false implies normal 2-way synchronization.
·
Any further parameters are passed on to the storage class.

Supported Storages

CalDAV and CardDAV

caldav

CalDAV.

[storage example_for_caldav]
type = caldav
#start_date = null
#end_date = null
#item_types = []
url = "..."
#username = ""
#password = ""
#verify = true
#auth = null
#useragent = "vdirsyncer"
#verify_fingerprint = null
#auth_cert = null

You can set a timerange to synchronize with the parameters start_date and end_date. Inside those parameters, you can use any Python expression to return a valid datetime.datetime object. For example, the following would synchronize the timerange from one year in the past to one year in the future:

start_date = datetime.now() - timedelta(days=365)
end_date = datetime.now() + timedelta(days=365)

Either both or none have to be specified. The default is to synchronize everything.

You can set item_types to restrict the kind of items you want to synchronize. For example, if you want to only synchronize events (but don't download any tasks from the server), set item_types = ["VEVENT"]. If you want to synchronize events and tasks, but have some VJOURNAL items on the server you don't want to synchronize, use item_types = ["VEVENT", "VTODO"].

Parameters
·
start_date -- Start date of timerange to show, default -inf.
·
end_date -- End date of timerange to show, default +inf.
·
item_types -- Kind of items to show. The default, the empty list, is to show all. This depends on particular features on the server, the results are not validated.
·
url -- Base URL or an URL to a collection.
·
username -- Username for authentication.
·
password -- Password for authentication.
·
verify -- Verify SSL certificate, default True. This can also be a local path to a self-signed SSL certificate. See ssl-tutorial for more information.
·
verify_fingerprint -- Optional. SHA1 or MD5 fingerprint of the expected server certificate. See ssl-tutorial for more information.
·
auth -- Optional. Either basic, digest or guess. The default is preemptive Basic auth, sending credentials even if server didn't request them. This saves from an additional roundtrip per request. Consider setting guess if this causes issues with your server.
·
auth_cert -- Optional. Either a path to a certificate with a client certificate and the key or a list of paths to the files with them.
·
useragent -- Default vdirsyncer.

NOTE:

Please also see supported-servers, as some servers may not work well.

carddav

CardDAV.

[storage example_for_carddav]
type = carddav
url = "..."
#username = ""
#password = ""
#verify = true
#auth = null
#useragent = "vdirsyncer"
#verify_fingerprint = null
#auth_cert = null
Parameters
·
url -- Base URL or an URL to a collection.
·
username -- Username for authentication.
·
password -- Password for authentication.
·
verify -- Verify SSL certificate, default True. This can also be a local path to a self-signed SSL certificate. See ssl-tutorial for more information.
·
verify_fingerprint -- Optional. SHA1 or MD5 fingerprint of the expected server certificate. See ssl-tutorial for more information.
·
auth -- Optional. Either basic, digest or guess. The default is preemptive Basic auth, sending credentials even if server didn't request them. This saves from an additional roundtrip per request. Consider setting guess if this causes issues with your server.
·
auth_cert -- Optional. Either a path to a certificate with a client certificate and the key or a list of paths to the files with them.
·
useragent -- Default vdirsyncer.

NOTE:

Please also see supported-servers, as some servers may not work well.

Google

At first run you will be asked to authorize application for google account access.

To use this storage type, you need to install some additional dependencies:

pip install vdirsyncer[google]

Furthermore you need to register vdirsyncer as an application yourself to obtain client_id and client_secret, as it is against Google's Terms of Service to hardcode those into opensource software:

1.
Go to the Google API Manager and create a new project under any name.
2.
Within that project, enable the "CalDAV" and "CardDAV" APIs (not the Calendar and Contacts APIs, those are different and won't work). There should be a searchbox where you can just enter those terms.
3.
In the sidebar, select "Credentials" and create a new "OAuth Client ID". The application type is "Other".

You'll be prompted to create a OAuth consent screen first. Fill out that form however you like.
4.
Finally you should have a Client ID and a Client secret. Provide these in your storage config.

You can select which calendars to sync on CalDav settings page.

google_calendar

Google calendar.

[storage example_for_google_calendar]
type = google_calendar
token_file = "..."
client_id = "..."
client_secret = "..."
#start_date = null
#end_date = null
#item_types = []

Please refer to caldav regarding the item_types and timerange parameters.

Parameters
·
token_file -- A filepath where access tokens are stored.
·
client_id/client_secret -- OAuth credentials, obtained from the Google API Manager.
google_contacts

Google contacts.

[storage example_for_google_contacts]
type = google_contacts
token_file = "..."
client_id = "..."
client_secret = "..."

NOTE:

Google's CardDAV implementation is allegedly a disaster in terms of data safety. See this blog post for the details. Always back up your data.

Parameters
·
token_file -- A filepath where access tokens are stored.
·
client_id/client_secret -- OAuth credentials, obtained from the Google API Manager.

remoteStorage

remoteStorage is an open per-user data storage protocol. Vdirsyncer contains highly experimental support for it.

NOTE:

Do not use this storage if you're not prepared for data-loss and breakage.

To use them, you need to install some optional dependencies with:

pip install vdirsyncer[remotestorage]
remotestorage_contacts

remoteStorage contacts. Uses the vdir_contacts scope.

[storage example_for_remotestorage_contacts]
type = remotestorage_contacts
account = "..."
#verify = true
#verify_fingerprint = null
#auth_cert = null
#access_token = null
Parameters
·
account -- remoteStorage account, "user@example.com".
·
username -- Username for authentication.
·
password -- Password for authentication.
·
verify -- Verify SSL certificate, default True. This can also be a local path to a self-signed SSL certificate. See ssl-tutorial for more information.
·
verify_fingerprint -- Optional. SHA1 or MD5 fingerprint of the expected server certificate. See ssl-tutorial for more information.
·
auth -- Optional. Either basic, digest or guess. The default is preemptive Basic auth, sending credentials even if server didn't request them. This saves from an additional roundtrip per request. Consider setting guess if this causes issues with your server.
·
auth_cert -- Optional. Either a path to a certificate with a client certificate and the key or a list of paths to the files with them.
·
useragent -- Default vdirsyncer.
remotestorage_calendars

remoteStorage calendars. Uses the vdir_calendars scope.

[storage example_for_remotestorage_calendars]
type = remotestorage_calendars
account = "..."
#verify = true
#verify_fingerprint = null
#auth_cert = null
#access_token = null
Parameters
·
account -- remoteStorage account, "user@example.com".
·
username -- Username for authentication.
·
password -- Password for authentication.
·
verify -- Verify SSL certificate, default True. This can also be a local path to a self-signed SSL certificate. See ssl-tutorial for more information.
·
verify_fingerprint -- Optional. SHA1 or MD5 fingerprint of the expected server certificate. See ssl-tutorial for more information.
·
auth -- Optional. Either basic, digest or guess. The default is preemptive Basic auth, sending credentials even if server didn't request them. This saves from an additional roundtrip per request. Consider setting guess if this causes issues with your server.
·
auth_cert -- Optional. Either a path to a certificate with a client certificate and the key or a list of paths to the files with them.
·
useragent -- Default vdirsyncer.

Local

filesystem

Saves each item in its own file, given a directory.

[storage example_for_filesystem]
type = filesystem
path = "..."
fileext = "..."
#encoding = "utf-8"
#post_hook = null

Can be used with khal. See vdir for a more formal description of the format.

Parameters
·
path -- Absolute path to a vdir/collection. If this is used in combination with the collections parameter in a pair-section, this should point to a directory of vdirs instead.
·
fileext -- The file extension to use (e.g. .txt). Contained in the href, so if you change the file extension after a sync, this will trigger a re-download of everything (but should not cause data-loss of any kind).
·
encoding -- File encoding for items, both content and filename.
·
post_hook -- A command to call for each item creation and modification. The command will be called with the path of the new/updated file.
singlefile

Save data in single local .vcf or .ics file.

[storage example_for_singlefile]
type = singlefile
path = "..."
#encoding = "utf-8"

The storage basically guesses how items should be joined in the file.

New in version 0.1.6.

NOTE:

This storage is very slow, and that is unlikely to change. You should consider using filesystem if it fits your usecase.

Parameters
·
path -- The filepath to the file to be written to. If collections are used, this should contain %s as a placeholder for the collection name.
·
encoding -- Which encoding the file should use. Defaults to UTF-8.

Example for syncing with caldav:

[pair my_calendar]
a = my_calendar_local
b = my_calendar_remote
collections = ["from a", "from b"]

[storage my_calendar_local]
type = singlefile
path = ~/.calendars/%s.ics

[storage my_calendar_remote]
type = caldav
url = https://caldav.example.org/
#username =
#password =

Example for syncing with caldav using a null collection:

[pair my_calendar] a = my_calendar_local b = my_calendar_remote

[storage my_calendar_local] type = singlefile path = ~/my_calendar.ics

[storage my_calendar_remote] type = caldav url = https://caldav.example.org/username/my_calendar/ #username = #password =

Read-only storages

These storages don't support writing of their items, consequently read_only is set to true by default. Changing read_only to false on them leads to an error.

http

Use a simple .ics file (or similar) from the web.

[storage example_for_http]
type = http
url = "..."
#username = ""
#password = ""
#verify = true
#auth = null
#useragent = "vdirsyncer"
#verify_fingerprint = null
#auth_cert = null
Parameters
·
url -- URL to the .ics file.
·
username -- Username for authentication.
·
password -- Password for authentication.
·
verify -- Verify SSL certificate, default True. This can also be a local path to a self-signed SSL certificate. See ssl-tutorial for more information.
·
verify_fingerprint -- Optional. SHA1 or MD5 fingerprint of the expected server certificate. See ssl-tutorial for more information.
·
auth -- Optional. Either basic, digest or guess. The default is preemptive Basic auth, sending credentials even if server didn't request them. This saves from an additional roundtrip per request. Consider setting guess if this causes issues with your server.
·
auth_cert -- Optional. Either a path to a certificate with a client certificate and the key or a list of paths to the files with them.
·
useragent -- Default vdirsyncer.

A simple example:

[pair holidays]
a = holidays_local
b = holidays_remote

[storage holidays_local]
type = filesystem
path = ~/.config/vdir/calendars/holidays/
fileext = .ics

[storage holidays_remote]
type = http
url = https://example.com/holidays_from_hicksville.ics

Supported Software

Client applications

The following software has been reported to work well with vdirsyncer, however, none of it is regularly tested.

Calendars

·
khal, a CLI calendar application supporting vdir. You can use filesystem with it.
·
Many graphical calendar apps such as dayplanner, Orage or rainlendar save a calendar in a single .ics file. You can use singlefile with those.

Task/Todo managers

The iCalendar format also supports saving tasks in form of VTODO-entries, with the same file extension as normal events: .ics. All CalDAV servers support synchronizing tasks, vdirsyncer does too.

·
todoman, a CLI task manager supporting vdir. You can use filesystem with it.

Its interface is similar to the ones of Taskwarrior or the todo.txt CLI app and should be intuitively usable.

Contacts

·
khard, a commandline addressbook supporting vdir. You can use filesystem with it.
·
contactquery.c, a small program explicitly written for querying vdirs from mutt.
·
mates, a commandline addressbook supporting vdir.
·
vdirel, access vdir contacts from Emacs.

Supported servers

CalDAV and CardDAV servers not listed here may work anyway.

Radicale

Radicale is a very lightweight server, however, it intentionally doesn't implement the CalDAV and CardDAV standards completely, which might lead to issues even with very well-written clients. Apart from its non-conformity with standards, there are multiple other problems with its code quality and the way it is maintained.

That said, vdirsyncer is continuously tested against the git version and the latest PyPI release of Radicale.

·
Vdirsyncer can't create collections on Radicale.
·
Radicale doesn't support time ranges in the calendar-query of CalDAV, so setting start_date and end_date for caldav will have no or unpredicted consequences.
·
Versions of Radicale older than 0.9b1 choke on RFC-conform queries for all items of a collection.

You have to set item_types = ["VTODO", "VEVENT"] in caldav for vdirsyncer to work with those versions.

ownCloud

Vdirsyncer is continuously tested against the latest version of ownCloud:

[storage cal]
type = caldav
url = https://example.com/owncloud/remote.php/caldav/
username = ...
password = ...

[storage card]
type = carddav
url = https://example.com/owncloud/remote.php/carddav/
username = ...
password = ...
·
Versions older than 7.0.0: ownCloud uses SabreDAV, which had problems detecting collisions and race-conditions. The problems were reported and are fixed in SabreDAV's repo, and the corresponding fix is also in ownCloud since 7.0.0. See issue #16 for more information.

FastMail

Vdirsyncer is irregularly tested against FastMail. There are no known issues with it. FastMail's support pages provide the settings to use:

[storage cal]
type = caldav
url = https://caldav.messagingengine.com/
username = ...
password = ...

[storage card]
type = carddav
url = https://carddav.messagingengine.com/
username = ...
password = ...

iCloud

Vdirsyncer is irregularly tested against iCloud.

[storage cal]
type = caldav
url = https://caldav.icloud.com/
username = ...
password = ...

[storage card]
type = carddav
url = https://contacts.icloud.com/
username = ...
password = ...

Problems:

·
Vdirsyncer can't do two-factor auth with iCloud (there doesn't seem to be a way to do two-factor auth over the DAV APIs) You'll need to use app-specific passwords instead.
·
Vdirsyncer can't create collections on iCloud.

DavMail (Exchange, Outlook)

Using vdirsyncer with DavMail is possible, but you might get confronted with weird errors coming from old Exchange servers, malformed calendar entries with special characters and/or using an old DavMail version.

Make absolutely sure you use the latest DavMail:

[storage outlook]
type = caldav
url = http://localhost:1080/
username = user@example.com
password = ...
·
Older versions of DavMail handle URLs case-insensitively. See issue #144.
·
DavMail is handling old broken events very poorly. In such cases the Calendar Checking Tool for Outlook might help.

If you encounter any problems, do report them to me, but it's probably not vdirsyncer's fault.

Baikal

Vdirsyncer is continuously tested against the latest version of Baikal.

·
Baikal up to 0.2.7 also uses an old version of SabreDAV, with the same issue as ownCloud, see issue #160. This issue is fixed in later versions.

Google

Using vdirsyncer with Google Calendar is possible as of 0.10, but it is not tested frequently. You can use google_contacts and google_calendar.

For more information see issue #202 and issue #8.

Known Problems

For any unanswered questions or problems, see contact.

Manual installation

If your distribution doesn't provide a package for vdirsyncer, you still can use Python's package manager "pip". First, you'll have to check that the following things are installed:

·
A compatible version of Python (2.7+ or 3.3+) and the corresponding pip package
·
libxml and libxslt
·
zlib

On Linux systems, using the distro's package manager is the best way to do this, for example, using Ubuntu:

sudo apt-get install libxml2 libxslt1.1 zlib1g python

Then you have several options:

The dirty, easy way

The easiest way to install vdirsyncer at this point would be to run:

pip install --user vdirsyncer

This method has a major flaw though: Pip doesn't keep track of the files it installs. Vdirsyncer's files would be located somewhere in ~/.local/lib/python*, but you can't possibly know which packages were installed as dependencies of vdirsyncer and which ones were not, should you decide to uninstall it. In other words, using pip that way would pollute your home directory.

The clean but hard way

There is a way to install Python software without scattering stuff across your filesystem: virtualenv. There are a lot of resources on how to use it, the simplest possible way would look something like:

virtualenv ~/vdirsyncer_env
~/vdirsyncer_env/bin/pip install vdirsyncer
alias vdirsyncer="~/vdirsyncer_env/bin/vdirsyncer

You'll have to put the last line into your .bashrc or .bash_profile.

This method has two advantages:

·
It separately installs all Python packages into ~/vdirsyncer_env/, without relying on the system packages. This works around OS- or distro-specific issues.
·
You can delete ~/vdirsyncer_env/ to uninstall vdirsyncer entirely.

The new, perfect way

pipsi is a new package manager for Python-based software that automatically sets up a virtualenv for each program you install. Assuming you have it installed on your operating system, you can do:

pipsi install --python python3 vdirsyncer

and .local/bin/vdirsyncer will be your new vdirsyncer installation. To update vdirsyncer to the latest version:

pipsi upgrade vdirsyncer

If you're done with vdirsyncer, you can do:

pipsi uninstall vdirsyncer

and vdirsyncer will be uninstalled, including its dependencies.

Contributing to This Project

Important: Please read contact for questions and support requests.

The issue tracker

We use GitHub issues for organizing bug reports and feature requests.

The following labels are of interest:

·
"Planning" is for issues that are still undecided, but where at least some discussion exists.
·
"Blocked" is for issues that can't be worked on at the moment because some other unsolved problem exists. This problem may be a bug in some software dependency, for instance.
·
"Ready" contains issues that are ready to work on.

All of those labels are also available as a kanban board on waffles.io. It is really just an alternative overview over all issues, but might be easier to comprehend.

Feel free to contact me or comment on the relevant issues for further information.

Reporting bugs

·
Make sure your problem isn't already listed in problems.
·
Make sure you have the latest version by executing pip install --user --upgrade vdirsyncer.
·
Use --verbosity=DEBUG when including output from vdirsyncer.

Suggesting features

If you're suggesting a feature, keep in mind that vdirsyncer tries not to be a full calendar or contacts client, but rather just the piece of software that synchronizes all the data. Take a look at the documentation for software working with vdirsyncer.

Submitting patches, pull requests

·
Discuss everything in the issue tracker first (or contact me somehow else) before implementing it.
·
Make sure the tests pass. See below for running them.
·
But not because you wrote too few tests.
·
Add yourself to AUTHORS.rst, and add a note to Changelog.rst too.

Running tests, how to set up your development environment

For many patches, it might suffice to just let Travis run the tests. However, Travis is slow, so you might want to run them locally too. For this, set up a virtualenv and run this inside of it:

make install-dev  # install vdirsyncer from the repo into the virtualenv
make install-test  # install test dependencies
make install-style  # install dependencies for stylechecking
make install-docs  # install dependencies for building documentation

Then you can run:

make test  # The normal testsuite
make style  # Stylechecker
make docs  # Build the HTML docs, output is at docs/_build/html/

If you have any questions, feel free to open issues about it.

The Vdir Storage Format

This document describes a standard for storing calendars and contacts on a filesystem, with the main goal of being easy to implement.

Vdirsyncer synchronizes to vdirs via filesystem. Each vdir (basically just a directory with some files in it) represents a calendar or addressbook.

Basic Structure

The main folder (root) contains an arbitrary number of subfolders (collections), which contain only files (items). Synonyms for "collection" may be "addressbook" or "calendar".

An item is:

·
A vCard file, in which case the file extension must be .vcf, or
·
An iCalendar file, in which case the file extension must be .ics.

An item should contain a UID property as described by the vCard and iCalendar standards. If it contains more than one UID property, the values of those must not differ.

The file must contain exactly one event, task or contact. In most cases this also implies only one VEVENT/VTODO/VCARD component per file, but e.g. recurrence exceptions would require multiple VEVENT components per event.

The filename should consist of the ident, followed by the file extension. The ident is either the UID, if the item has one, else a string with similar properties as the UID. However, several restrictions of the underlying filesystem might make an implementation of this naming scheme for items' filenames impossible. The approach to deal with such cases is left to the client, which are free to choose a different scheme for filenames instead.

Metadata

Any of the below metadata files may be absent. None of the files listed below have any file extensions.

·
A file called color inside the vdir indicates the vdir's color, a property that is only relevant in UI design.

Its content is an ASCII-encoded hex-RGB value of the form #RRGGBB. For example, a file content of #FF0000 indicates that the vdir has a red (user-visible) color. No short forms or informal values such as red (as known from CSS, for example) are allowed. The prefixing # must be present.
·
A file called displayname contains a UTF-8 encoded label that may be used to represent the vdir in UIs.

Writing to vdirs

Creating and modifying items or metadata files should happen atomically.

Writing to a temporary file on the same physical device, and then moving it to the appropriate location is usually a very effective solution. For this purpose, files with the extension .tmp may be created inside collections.

When changing an item, the original filename must be used.

Reading from vdirs

·
Any file ending with the .tmp or no file extension must not be treated as an item.
·
The ident part of the filename should not be parsed to improve the speed of item lookup.

Considerations

The primary reason this format was chosen is due to its compatibility with the CardDAV and CalDAV standards.

Performance

Currently, vdirs suffer from a rather major performance problem, one which current implementations try to mitigate by building up indices of the collections for faster search and lookup.

The reason items' filenames don't contain any extra information is simple: The solutions presented induced duplication of data, where one duplicate might become out of date because of bad implementations. As it stands right now, a index format could be formalized separately though.

vdirsyncer doesn't really have to bother about efficient item lookup, because its synchronization algorithm needs to fetch the whole list of items anyway. Detecting changes is easily implemented by checking the files' modification time.

Packaging Guidelines

Thank you very much for packaging vdirsyncer! The following guidelines should help you to avoid some common pitfalls.

While they are called guidelines and therefore theoretically not mandatory, if you consider going a different direction, please first open an issue or contact me otherwise instead of just going ahead. These guidelines exist for my own convenience too.

Obtaining the source code

The main distribution channel is PyPI, and source tarballs can be obtained there. Do not use the ones from GitHub: Their tarballs contain useless junk and are more of a distraction than anything else.

I give each release a tag in the git repo. If you want to get notified of new releases, GitHub's feed is a good way.

Dependency versions

It is strongly discouraged to package vdirsyncer as a Python 2 application. Future releases will only work on Python 3.3 and newer versions.

As with most Python packages, setup.py denotes the dependencies of vdirsyncer. It also contains lower-bound versions of each dependency. Older versions will be rejected by the testsuite.

Testing

Everything testing-related goes through the Makefile in the root of the repository or PyPI package. Trying to e.g. run py.test directly will require a lot of environment variables to be set (for configuration) and you probably don't want to deal with that.

You can install the testing dependencies with:

make install-test

You probably don't want this since it will use pip to download the dependencies. Alternatively you can find the testing dependencies in test-requirements.txt, again with lower-bound version requirements.

You also have to have vdirsyncer fully installed at this point. Merely cd-ing into the tarball will not be sufficient.

Running the tests happens with:

make test

Hypothesis will randomly generate test input. If you care about deterministic tests, set the DETERMINISTIC_TESTS variable to "true":

make DETERMINISTIC_TESTS=true test

Documentation

Using Sphinx you can generate the documentation you're reading right now in a variety of formats, such as HTML, PDF, or even as a manpage. That said, I only take care of the HTML docs' formatting.

You can find a list of dependencies in docs-requirements.txt. Again, you can install those using pip with:

make install-docs

Then change into the docs/ directory and build whatever format you want using the Makefile in there (run make for the formats you can build).

Support and Contact

·
The #pimutils IRC channel on Freenode might be active, depending on your timezone. Use it for support and general (including off-topic) discussion.
·
Open a GitHub issue for concrete bug reports and feature requests.
·
Lastly, you can also contact the author directly.

Changelog

This changelog only contains information that might be useful to end users and package maintainers. For further info, see the git commit log.

Package maintainers and users who have to manually update their installation may want to subscribe to GitHub's tag feed.

Version 0.12.1

released on 20 August 2016

·
Fix a crash for Google and DAV storages. See pull request #492.
·
Fix an URL-encoding problem with DavMail. See issue #491.

Version 0.12

released on 19 August 2016

·
singlefile now supports collections. See pull request #488.

Version 0.11.3

released on 29 July 2016

·
Default value of auth parameter was changed from guess to basic to resolve issues with the Apple Calendar Server (issue #457) and improve performance. See issue #461.
·
Packagers: The click-threading requirement is now >=0.2. It was incorrect before. See issue #478.
·
Fix a bug in the DAV XML parsing code that would make vdirsyncer crash on certain input. See issue #480.
·
Redirect chains should now be properly handled when resolving well-known URLs. See pull request #481.

Version 0.11.2

released on 15 June 2016

·
Fix typo that would break tests.

Version 0.11.1

released on 15 June 2016

·
Fix a bug in collection validation.
·
Fix a cosmetic bug in debug output.
·
Various documentation improvements.

Version 0.11.0

released on 19 May 2016

·
Discovery is no longer automatically done when running vdirsyncer sync. vdirsyncer discover now has to be explicitly called.
·
Add a .plist example for Mac OS X.
·
Usage under Python 2 now requires a special config parameter to be set.
·

Various deprecated configuration parameters do no longer have specialized errormessages. The generic error message for unknown parameters is shown.

·
Vdirsyncer no longer warns that the passwordeval parameter has been renamed to password_command.
·
The keyring fetching strategy has been dropped some versions ago, but the specialized error message has been dropped.
·
A old status format from version 0.4 is no longer supported. If you're experiencing problems, just delete your status folder.

Version 0.10.0

released on 23 April 2016

·
New storage types google_calendar and google_contacts have been added.
·
New global command line option --config, to specify an alternative config file. See issue #409.
·
The collections parameter can now be used to synchronize differently-named collections with each other.
·
Packagers: The lxml dependency has been dropped.
·
XML parsing is now a lot stricter. Malfunctioning servers that used to work with vdirsyncer may stop working.

Version 0.9.3

released on 22 March 2016

·
singlefile and http now handle recurring events properly.
·
Fix a typo in the packaging guidelines.
·
Moved to pimutils organization on GitHub. Old links should redirect, but be aware of client software that doesn't properly handle redirects.

Version 0.9.2

released on 13 March 2016

·
Fixed testsuite for environments that don't have any web browser installed. See pull request #384.

Version 0.9.1

released on 13 March 2016

·
Removed leftover debug print statement in vdirsyncer discover, see commit 3d856749f37639821b148238ef35f1acba82db36.
·
metasync will now strip whitespace from the start and the end of the values. See issue #358.
·
New Packaging Guidelines have been added to the documentation.

Version 0.9.0

released on 15 February 2016

·
The collections parameter is now required in pair configurations. Vdirsyncer will tell you what to do in its error message. See issue #328.

Version 0.8.1

released on 30 January 2016

·
Fix error messages when invalid parameter fetching strategy is used. This is important because users would receive awkward errors for using deprecated keyring fetching.

Version 0.8.0

released on 27 January 2016

·
Keyring support has been removed, which means that password.fetch = ["keyring", "example.com", "myuser"] doesn't work anymore.

For existing setups: Use password.fetch = ["command", "keyring", "get", "example.com", "myuser"] instead, which is more generic. See the documentation for details.
·
Now emitting a warning when running under Python 2. See issue #219.

Version 0.7.5

released on 23 December 2015

·
Fixed a bug in remotestorage that would try to open a CLI browser for OAuth.
·
Fix a packaging bug that would prevent vdirsyncer from working with newer lxml versions.

Version 0.7.4

released on 22 December 2015

·
Improved error messages instead of faulty server behavior, see issue #290 and issue #300.
·
Safer shutdown of threadpool, avoid exceptions, see issue #291.
·
Fix a sync bug for read-only storages see commmit ed22764921b2e5bf6a934cf14aa9c5fede804d8e.
·
Etag changes are no longer sufficient to trigger sync operations. An actual content change is also necessary. See issue #257.
·
remotestorage now automatically opens authentication dialogs in your configured GUI browser.
·
Packagers: lxml>=3.1 is now required (newer lower-bound version).

Version 0.7.3

released on 05 November 2015

·
Make remotestorage-dependencies actually optional.

Version 0.7.2

released on 05 November 2015

·
Un-break testsuite.

Version 0.7.1

released on 05 November 2015

·
Packagers: The setuptools extras keyring and remotestorage have been added. They're basically optional dependencies. See setup.py for more details.
·
Highly experimental remoteStorage support has been added. It may be completely overhauled or even removed in any version.
·
Removed mentions of old password_command in documentation.

Version 0.7.0

released on 27 October 2015

·
Packagers: New dependencies are click_threading, click_log and click>=5.0.
·
password_command is gone. Keyring support got completely overhauled. See keyring.

Version 0.6.0

released on 06 August 2015

·
password_command invocations with non-zero exit code are now fatal (and will abort synchronization) instead of just producing a warning.
·
Vdirsyncer is now able to synchronize metadata of collections. Set metadata = ["displayname"] and run vdirsyncer metasync.
·
Packagers: Don't use the GitHub tarballs, but the PyPI ones.
·
Packagers: build.sh is gone, and Makefile is included in tarballs. See the content of Makefile on how to run tests post-packaging.
·
verify_fingerprint doesn't automatically disable verify anymore.

Version 0.5.2

released on 15 June 2015

·
Vdirsyncer now checks and corrects the permissions of status files.
·
Vdirsyncer is now more robust towards changing UIDs inside items.
·
Vdirsyncer is now handling unicode hrefs and UIDs correctly. Software that produces non-ASCII UIDs is broken, but apparently it exists.

Version 0.5.1

released on 29 May 2015

·
N.b.: The PyPI upload of 0.5.0 is completely broken.
·
Raise version of required requests-toolbelt to 0.4.0.
·
Command line should be a lot faster when no work is done, e.g. for help output.
·
Fix compatibility with iCloud again.
·
Use only one worker if debug mode is activated.
·
verify=false is now disallowed in vdirsyncer, please use verify_fingerprint instead.
·
Fixed a bug where vdirsyncer's DAV storage was not using the configured useragent for collection discovery.

Version 0.4.4

released on 12 March 2015

·
Support for client certificates via the new auth_cert parameter, see issue #182 and pull request #183.
·
The icalendar package is no longer required.
·
Several bugfixes related to collection creation.

Version 0.4.3

released on 20 February 2015

·
More performance improvements to singlefile-storage.
·
Add post_hook param to filesystem-storage.
·
Collection creation now also works with SabreDAV-based servers, such as Baikal or ownCloud.
·
Removed some workarounds for Radicale. Upgrading to the latest Radicale will fix the issues.
·
Fixed issues with iCloud discovery.
·
Vdirsyncer now includes a simple repair command that seeks to fix some broken items.

Version 0.4.2

released on 30 January 2015

·
Vdirsyncer now respects redirects when uploading and updating items. This might fix issues with Zimbra.
·
Relative status_path values are now interpreted as relative to the configuration file's directory.
·
Fixed compatibility with custom SabreDAV servers. See issue #166.
·
Catch harmless threading exceptions that occur when shutting down vdirsyncer. See issue #167.
·
Vdirsyncer now depends on atomicwrites.
·
Massive performance improvements to singlefile-storage.
·
Items with extremely long UIDs should now be saved properly in filesystem-storage. See issue #173.

Version 0.4.1

released on 05 January 2015

·
All create arguments from all storages are gone. Vdirsyncer now asks if it should try to create collections.
·
The old config values True, False, on, off and None are now invalid.
·
UID conflicts are now properly handled instead of ignoring one item. Card- and CalDAV servers are already supposed to take care of those though.
·
Official Baikal support added.

Version 0.4.0

released on 31 December 2014

·
The passwordeval parameter has been renamed to password_command.
·
The old way of writing certain config values such as lists is now gone.
·
Collection discovery has been rewritten. Old configuration files should be compatible with it, but vdirsyncer now caches the results of the collection discovery. You have to run vdirsyncer discover if collections were added or removed on one side.
·
Pair and storage names are now restricted to certain characters. Vdirsyncer will issue a clear error message if your configuration file is invalid in that regard.
·
Vdirsyncer now supports the XDG-Basedir specification. If the VDIRSYNCER_CONFIG environment variable isn't set and the ~/.vdirsyncer/config file doesn't exist, it will look for the configuration file at $XDG_CONFIG_HOME/vdirsyncer/config.
·
Some improvements to CardDAV and CalDAV discovery, based on problems found with FastMail. Support for .well-known-URIs has been added.

Version 0.3.4

released on 8 December 2014

·
Some more bugfixes to config handling.

Version 0.3.3

released on 8 December 2014

·
Vdirsyncer now also works with iCloud. Particularly collection discovery and etag handling were fixed.
·
Vdirsyncer now encodes Cal- and CardDAV requests differently. This hasn't been well-tested with servers like Zimbra or SoGo, but isn't expected to cause any problems.
·
Vdirsyncer is now more robust regarding invalid responses from CalDAV servers. This should help with future compatibility with Davmail/Outlook.
·
Fix a bug when specifying item_types of caldav in the deprecated config format.
·
Fix a bug where vdirsyncer would ignore all but one character specified in unsafe_href_chars of caldav and carddav.

Version 0.3.2

released on 3 December 2014

·
The current config format has been deprecated, and support for it will be removed in version 0.4.0. Vdirsyncer warns about this now.

Version 0.3.1

released on 24 November 2014

·
Fixed a bug where vdirsyncer would delete items if they're deleted on side A but modified on side B. Instead vdirsyncer will now upload the new items to side A. See issue #128.
·
Synchronization continues with the remaining pairs if one pair crashes, see issue #121.
·
The processes config key is gone. There is now a --max-workers option on the CLI which has a similar purpose. See pull request #126.
·
The Read The Docs-theme is no longer required for building the docs. If it is not installed, the default theme will be used. See issue #134.

Version 0.3.0

released on 20 September 2014

·
Add verify_fingerprint parameter to http, caldav and carddav, see issue #99 and pull request #106.
·
Add passwordeval parameter to general_config, see issue #108 and pull request #117.
·
Emit warnings (instead of exceptions) about certain invalid responses from the server, see issue #113. This is apparently required for compatibility with Davmail.

Version 0.2.5

released on 27 August 2014

·
Don't ask for the password of one server more than once and fix multiple concurrency issues, see issue #101.
·
Better validation of DAV endpoints.

Version 0.2.4

released on 18 August 2014

·
Include workaround for collection discovery with latest version of Radicale.
·
Include metadata files such as the changelog or license in source distribution, see issue #97 and issue #98.

Version 0.2.3

released on 11 August 2014

·
Vdirsyncer now has a --version flag, see issue #92.
·
Fix a lot of bugs related to special characters in URLs, see issue #49.

Version 0.2.2

released on 04 August 2014

·
Remove a security check that caused problems with special characters in DAV URLs and certain servers. On top of that, the security check was nonsensical. See issue #87 and issue #91.
·
Change some errors to warnings, see issue #88.
·
Improve collection autodiscovery for servers without full support.

Version 0.2.1

released on 05 July 2014

·
Fix bug where vdirsyncer shows empty addressbooks when using CardDAV with Zimbra.
·
Fix infinite loop when password doesn't exist in system keyring.
·
Colorized errors, warnings and debug messages.
·
vdirsyncer now depends on the click package instead of argvard.

Version 0.2.0

released on 12 June 2014

·
vdirsyncer now depends on the icalendar package from PyPI, to get rid of its own broken parser.
·
vdirsyncer now also depends on requests_toolbelt. This makes it possible to guess the authentication type instead of blankly assuming basic.
·
Fix a semi-bug in caldav and carddav storages where a tuple (href, etag) instead of the proper etag would have been returned from the upload method. vdirsyncer might do unnecessary copying when upgrading to this version.
·
Add the storage singlefile. See issue #48.
·
The collections parameter for pair sections now accepts the special values from a and from b for automatically discovering collections. See pair_config.
·
The read_only parameter was added to storage sections. See storage_config.

Version 0.1.5

released on 14 May 2014

·
Introduced changelogs
·
Many bugfixes
·
Many doc fixes
·
vdirsyncer now doesn't necessarily need UIDs anymore for synchronization.
·
vdirsyncer now aborts if one collection got completely emptied between synchronizations. See issue #42.

Credits and License

Contributors

In alphabetical order:

·
Ben Boeckel
·
Christian Geier
·
Clément Mondon
·
Hugo Osvaldo Barrera
·
Julian Mehne
·
Marek Marczykowski-Górecki
·
Markus Unterwaditzer
·
Michael Adler
·
Thomas Weißschuh

License

Copyright (c) 2014-2016 Markus Unterwaditzer & contributors

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR Copyright HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Author

Markus Unterwaditzer

Referenced By

khal(1).

Sep 04, 2016 0.12 vdirsyncer