nbdkit-vddk-plugin - Man Page
nbdkit VMware VDDK plugin
Synopsis
nbdkit vddk [file=]FILENAME [compression=none|zlib|fastlz|skipz] [config=FILENAME] [cookie=COOKIE] [libdir=LIBRARY] [nfchostport=PORT] [single-link=true] [password=PASSWORD | password=- | password=+FILENAME | password=-FD] [port=PORT] [server=HOSTNAME] [snapshot=MOREF] [thumbprint=THUMBPRINT] [transports=MODE:MODE:...] [unbuffered=true] [user=USERNAME] [vm=moref=ID] nbdkit vddk --dump-plugin
Description
nbdkit-vddk-plugin
is an nbdkit(1) plugin that serves disks from local VMware VMDK files, VMware ESXi servers, VMware VCenter servers, and other sources.
It requires VMware's proprietary VDDK library that you must download yourself (see “Library Location” below).
Examples
Open a local VMDK file
nbdkit vddk /absolute/path/to/file.vmdk
When opening local files the filename must be an absolute path.
Because VDDK needs to lock the file, the file must be on a writable filesystem. You can avoid this by opening the file read-only (the -r option):
nbdkit -r vddk /absolute/path/to/file.vmdk
Open a file on a remote VMware ESXi hypervisor
Connect directly to a VMware ESXi hypervisor and export a particular file:
nbdkit vddk user=root password=+/tmp/rootpw \ server=esxi.example.com thumbprint=xx:xx:xx:... \ vm=moref=2 \ "[datastore1] Fedora/Fedora.vmdk"
user
and password
must be specified. Use password=+FILENAME
to provide the password securely in a file.
server
is the hostname of the ESXi server.
thumbprint
is the thumb print for validating the SSL certificate. How to find the thumb print of a server is described in “Thumbprints” below.
vm
is the Managed Object Reference (“moref”) of the virtual machine. See “Managed Object Reference” below.
The file parameter is the file you want to open, usually in the form "[datastore] vmname/vmname.vmdk"
. See “File Parameter” below.
Open a file on a remote VMware vCenter server
Connect via VMware vCenter and export a particular file:
nbdkit vddk user=root password=vmware \ server=vcenter.example.com thumbprint=xx:xx:xx:... \ vm=moref=vm-16 \ "[datastore1] Fedora/Fedora.vmdk"
user
and password
must be specified. Use password=+FILENAME
to provide the password securely in a file.
server
is the hostname of the vCenter server.
thumbprint
is the thumb print for validating the SSL certificate. How to find the thumb print of a server is described in “Thumbprints” below.
vm
is the Managed Object Reference (“moref”) of the virtual machine. See “Managed Object Reference” below.
The file parameter is the file you want to open, usually in the form "[datastore] vmname/vmname.vmdk"
. See “File Parameter” below.
Parameters
For opening a local VMDK file, the file
parameter is required and must be an absolute path.
For opening a remote connection, file
, server
, thumbprint
, user
, password
and vm
are required.
All other parameters are optional.
- compression=none
- compression=zlib
- compression=fastlz
- compression=skipz
- compression=zlib
(nbdkit ≥ 1.24, vSphere ≥ 6.5)
Select the compression type used over the network between VDDK and the VMware server. The default is
none
. See VMware document “Best Practices for NBD Transport”.- config=FILENAME
The name of the VDDK configuration file.
The config file controls rarely adjusted VDDK settings like log level, caching and timeouts. See the VDDK documentation for a full list of settings.
VDDK itself looks in a few default locations for the configuration file, usually including /etc/vmware/config and
$HOME
/.vmware/config. Usingconfig
overrides these defaults.- cookie=COOKIE
(vSphere ≥ 6.7)
Cookie from existing authenticated session on the host.
This changes the authentication type from
VIXDISKLIB_CRED_UID
toVIXDISKLIB_CRED_SESSIONID
which can improve performance. The cookie can be found by connecting to a VCenter Server over HTTPS and retrieving thevmware_soap_session
cookie.- [file=]FILENAME
- [file=][datastore] vmname/vmname.vmdk
Set the name of the VMDK file to serve.
For local files you must supply an absolute path. For remote files see “File Parameter” section below.
If a VM has multiple disks, nbdkit can only serve one at a time. To serve more than one you must run multiple copies of nbdkit. (See “Notes” below).
file=
is a magic config key and may be omitted in most cases. See “Magic parameters” in nbdkit(1).- libdir=PATHNAME
This sets the path of the VMware VDDK distribution.
VDDK uses this to load its own plugins, if this path is unspecified or wrong then VDDK will work with reduced functionality. See “Library Location” below.
- nfchostport=PORT
Port used to establish an NFC connection to ESXi. Defaults to 902.
- password=PASSWORD
Set the password to use when connecting to the remote server.
Note that passing this on the command line is not secure on shared machines.
- password=-
Ask for the password (interactively) when nbdkit starts up.
- password=+FILENAME
Read the password from the named file. This is a secure method to supply a password, as long as you set the permissions on the file appropriately.
- password=-FD
Read the password from file descriptor number
FD
, inherited from the parent process when nbdkit starts up. This is also a secure method to supply a password.- port=PORT
The port on the VCenter/ESXi host. Defaults to 443.
- server=HOSTNAME
The hostname or IP address of VCenter or ESXi host.
- single-link=true
(nbdkit ≥ 1.12)
Open the current link, not the entire chain. This corresponds to the
VIXDISKLIB_FLAG_OPEN_SINGLE_LINK
flag.- snapshot=MOREF
The Managed Object Reference of the snapshot. See “Managed Object Reference” below.
- thumbprint=THUMBPRINT
The SSL (SHA1) thumbprint for validating the SSL certificate.
The format is
xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
(20 hex digit pairs).See “Thumbprints” below for how to get this.
- transports=MODE:MODE:...
List of one or more transport modes to use. Possible values include ‘nbd’, ‘nbdssl’, ‘san’, ‘hotadd’, ‘file’ (there may be others). If not given, VDDK will try to choose the best transport mode.
- unbuffered=true
(nbdkit ≥ 1.12)
Disable host caching. This corresponds to the
VIXDISKLIB_FLAG_OPEN_UNBUFFERED
flag.- user=USERNAME
The username to connect to the remote server as.
- vm=moref=ID
The Managed Object Reference (“moref”) of the virtual machine. See “Managed Object Reference” below.
- vimapiver=APIVER
This parameter is ignored for backwards compatibility.
Library Location
The VDDK library should not be placed on a system library path such as /usr/lib. The reason is that the VDDK library ships with recompiled libraries like libcrypto.so and libstdc++.so that conflict with system libraries.
You have two choices:
Place VDDK in the default libdir which is compiled into this plugin, for example:
$ nbdkit vddk --dump-plugin | grep ^vddk_default_libdir vddk_default_libdir=/usr/lib64/vmware-vix-disklib
But the best advice is to unpack the VDDK tarball anywhere you like and set the
libdir=/path/to/vmware-vix-disklib-distrib
. For example:nbdkit vddk \ libdir=/opt/vmware-vix-disklib-distrib \ /path/to/file.vmdk
No need to set LD_LIBRARY_PATH
In nbdkit ≤ 1.16 you had to set the environment variable LD_LIBRARY_PATH
when using this plugin. In nbdkit ≥ 1.18 this is not recommended.
File Parameter
The For remote files you can find the path using virsh(1). For ESXi: For vCenter the command is the same but the URI starts with See also: https://libvirt.org/drvesx.html The file
parameter can either be a local file, in which case it must be the absolute path. Or it can refer to a remote file on the VMware server in the format "[datastore] vmname/vmname.vmdk"
.Finding the remote filename
$ virsh -c 'esx://esxi.example.com?no_verify=1' dumpxml guestname
...
<disk type='file' device='disk'>
<source file='[datastore] vmname/vmname.vmdk'/>
...
vpx://
: $ virsh -c 'vpx://vcenter.example.com/Datacenter/esxi.example.com?no_verify=1' \
dumpxml guestname
Optional file= prefix
file=
part is optional, so these commands are equivalent: nbdkit vddk file=/path/to/file.vmdk
nbdkit vddk /path/to/file.vmdk
Thumbprints
The thumbprint is a 20 byte string containing the SSL (SHA1) fingerprint of the remote VMware server and it is required when making a remote connection. There are several ways to obtain this. Visit The following command will print the thumbprint of a VMware server called Log in to the ESXi hypervisor shell and run this command: For VMware vCenter servers the thumbprint is printed on the text console of the server or is available by logging in to the server and using this command: Another way to get the thumbprint of a server is to connect to the server using a bogus thumbprint with debugging enabled: The nbdkit process will try to connect (and fail because the thumbprint is wrong). However in the debug output will be a message such as this: This gives you the server’s real thumbprint. Of course this method is not secure since it allows a Man-in-the-Middle (MITM) attack.Using a web browser
https://SERVER-NAME/folder
and log in. Click the lock icon next to the URL bar and navigate to the SHA-1 fingerprint of the site’s certificate. This 20 hex digit pair string can be directly copied to the thumbprint=
parameter.Using openssl remotely
SERVER-NAME
: $ openssl s_client -connect SERVER-NAME:443 </dev/null |
openssl x509 -in /dev/stdin -fingerprint -sha1 -noout
By logging in to the ESXi or vCenter server
# openssl x509 -in /etc/vmware/ssl/rui.crt -fingerprint -sha1 -noout
# openssl x509 -in /etc/vmware-vpx/ssl/rui.crt -fingerprint -sha1 -noout
Trick: Get VDDK to tell you the thumbprint
nbdkit -U - -fv vddk server=esxi.example.com [...] thumbprint=12 \
--run 'qemu-img info "$uri"'
nbdkit: debug: VixDiskLibVim: Failed to verify SSL certificate: actual thumbprint=B2:31:BD:DE:9F:DB:9D:E0:78:EF:30:42:8A:41:B0:28:92:93:C8:DD expected=12
Managed Object Reference
Some use cases require you to pass in the Managed Object Reference (“moref”) of an object on the VMware server.
For VMware ESXi hypervisors, the vm
moref is a number (eg. vm=moref=2
). For VMware VCenter it is a string beginning with "vm-"
) (eg. vm=moref=vm-16
). Across ESXi and vCenter the numbers are different even for the same virtual machine.
If you have libvirt ≥ 3.7, the moref is available in the virsh(1) dumpxml
output:
$ virsh -c 'esx://esxi.example.com?no_verify=1' dumpxml guestname ... <vmware:moref>2</vmware:moref> ...
or:
$ virsh -c 'vpx://vcenter.example.com/Datacenter/esxi.example.com?no_verify=1' \ dumpxml guestname ... <vmware:moref>vm-16</vmware:moref> ...
An alternative way to find the moref of a VM is using the moRefFinder.pl
script written by William Lam (http://www.virtuallyghetto.com/2011/11/vsphere-moref-managed-object-reference.html https://blogs.vmware.com/vsphere/2012/02/uniquely-identifying-virtual-machines-in-vsphere-and-vcloud-part-2-technical.html).
Dump-Plugin Output
To query more information about the plugin (and whether it is working), use:
nbdkit vddk --dump-plugin
If the plugin is not present, not working or the library path is wrong you will get an error.
If it works the output will include:
- vddk_default_libdir=...
The compiled-in library path. Use
libdir=PATHNAME
to override this at runtime.- vddk_has_nfchostport=1
If this is printed then the
nfchostport=PORT
parameter is supported by this build.- vddk_dll=...
Prints the full path to the VDDK shared library. Since this requires a glibc extension it may not be available in all builds of the plugin.
Notes
Sector size limitation
The VDDK plugin can only answer read/write requests on whole 512 byte sector boundaries. This is because the VDDK Read and Write APIs only take sector numbers. If your client needs finer granularity, you can use nbdkit-blocksize-filter(1) with the setting minblock=512
.
Threads
Handling threads in the VDDK API is complex and does not map well to any of the thread models offered by nbdkit (see “THREADS” in nbdkit-plugin(3)). The plugin uses the nbdkit SERIALIZE_ALL_REQUESTS
model, but technically even this is not completely safe. This is a subject of future work.
Out of memory errors
In the verbose log you may see errors like:
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 2097176 bytes
This seems especially common when there are multiple parallel connections open to the VMware server.
These can be caused by resource limits set on the VMware server. You can increase the limit for the NFC service by editing /etc/vmware/hostd/config.xml and adjusting the <maxMemory>
setting:
<nfcsvc> <path>libnfcsvc.so</path> <enabled>true</enabled> <maxMemory>50331648</maxMemory> <maxStreamMemory>10485760</maxStreamMemory> </nfcsvc>
and restarting the hostd
service:
# /etc/init.d/hostd restart
For more information see https://bugzilla.redhat.com/1614276.
Supported Versions of VDDK
This plugin requires VDDK ≥ 5.5.5, which in turn means that it is only supported on x64-64 platforms.
It has been tested with all versions up to 7.0.0 (but should work with future versions).
VDDK ≥ 6.0 should be used if possible. This is the first version which added Flush support which is crucial for data integrity when writing.
Debug Flags
Debugging messages can be very helpful if you have problems connecting to VMware servers, or to find the list of available transport modes, or to diagnose SAN problems:
nbdkit -f -v vddk file=FILENAME [...]
Additional debug flags are available:
- -D vddk.diskinfo=1
Debug disk information returned by
GetInfo
.- -D vddk.extents=1
Debug extents returned by
QueryAllocatedBlocks
.- -D vddk.datapath=0
Suppress debugging of datapath calls (
Read
andWrite
).
Files
- $plugindir/nbdkit-vddk-plugin.so
The plugin.
Use
nbdkit --dump-config
to find the location of$plugindir
.
Version
nbdkit-vddk-plugin
first appeared in nbdkit 1.2.
See Also
nbdkit(1), nbdkit-plugin(3), nbdkit-blocksize-filter(1), nbdkit-readahead-filter(1), nbdkit-retry-filter(1), virsh(1), https://libvirt.org/drvesx.html, https://www.vmware.com/support/developer/vddk/, VMware document “Best Practices for NBD Transport”.
Authors
Richard W.M. Jones
Copyright
Copyright (C) 2013-2020 Red Hat Inc.
License
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
- Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
- Neither the name of Red Hat nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Referenced By
nbdkit(1), nbdkit-blocksize-filter(1), nbdkit-cacheextents-filter(1), nbdkit-plugin(3), nbdkit-readahead-filter(1), nbdkit-release-notes-1.12(1), nbdkit-release-notes-1.14(1), nbdkit-release-notes-1.16(1), nbdkit-release-notes-1.18(1), nbdkit-release-notes-1.20(1), nbdkit-release-notes-1.22(1), nbdkit-release-notes-1.24(1), nbdkit-release-notes-1.6(1), nbdkit-release-notes-1.8(1), virt-v2v(1), virt-v2v-input-vmware(1).