nvmetcli nvmetcli clear nvmetcli restore [filename.json]
nvmetcli is a program used for viewing, editing, saving, and starting a Linux kernel NVMe Target, used for an NVMe-over-Fabrics network configuration. It allows an administrator to export a storage resource (such as NVMe devices, files, and volumes) to a local block device and expose them to remote systems based on the NVMe-over-Fabrics specification from http://www.nvmexpress.org.
nvmetcli is run as root and has two modes:
- An interactive configuration shell
- Command-line mode which uses an argument
The term NQN used throughout this man page is the NVMe Qualified Name format which an NVMe endpoint (device, subsystem, etc) must follow to guarantee a unique name under the NVMe standard. Any name in a network system setup can be used, but if it does not follow the NQN format, it may not be unique on an NVMe-over-Fabrics network.
Note that some of the fields set for an NVMe Target port under interactive mode are defined in the "Discovery Log Page" section of NVMe-over-Fabrics specification. Each NVMe Target has a discovery controller mechanism that an NVMe Host can use to determine the NVM subsystems it can access. nvmetcli can be used to add a new record to the discovery controller upon each new subsystem entry and port entry that the newly created subsystem entry binds to (see Options and Examples sections). Each NVMe Host only gets to see the discovery entries defined in /subsystems/[NQN NAME]/allowed_hosts and the IP port it is connected to the NVMe Target. An NVMe Host can retrieve these discovery logs via the nvme-cli tool (https://github.com/linux-nvme/nvme-cli).
Interactive Configuration Shell
To start the interactive configuration shell, type nvmetcli on the command-line. nvmetcli interacts with the Linux kernel NVMe Target configfs subsystem starting at base nvmetcli directories /port, /subsystem, and /host. Configuration changes entered by the administrator are made immediately to the kernel target configuration. The following commands can be used while in the interactive configuration shell mode:
|cd||Allows to move around the tree.|
|ls||Lists contents of current tree node.|
|create [NQN name]/[#]||Create a new object using the specified name or number. If a [NQN name]/[#] is not specified, a random entry will be used.|
|delete [NQN name]/[#]||Delete an object with the specified name or number.|
|set attr allow_any_host=[0/1]||Used under /subsystems/[NQN name] to specify if any NVMe Host can connect to the subsystem.|
|set device path=[device path]||Used under /subsystems/[NQN name]/namespaces to set the (storage) device to be used.|
|set device nguid=[string]||Used under /subsystems/[NQN name]/namespaces to set the unique id of the device to the defined namespace.|
|enable/disable||Used under /subsystems/[NQN name]/namespaces to enable and disable the namespace.|
|set addr [discovery log page field]=[string]||Used under /ports/[#] to create a port which access is allowed. See Examples for more information.|
|saveconfig [filename.json]||Save the NVMe Target configuration in .json format. Without specifying the filename this will save as /etc/nvmet/config.json. This file is in JSON format and can be edited directly using a prefered file editor.|
|exit||Quits interactive configuration shell mode.|
Command Line Mode
Typing nvmetcli [cmd] on the command-line will execute a command and not enter the interactive configuration shell.
|restore [filename.json]||Loads a saved NVMe Target configuration. Without specifying the filename this will use /etc/nvmet/config.json.|
|clear||Clears a current NVMe Target configuration.|
|ls||Dumps the current NVMe Target configuration.|
Make sure to run nvmetcli as root, the nvmet module is loaded, your devices and all dependent modules are loaded, and configfs is mounted on /sys/kernel/config using:
mount -t configfs none /sys/kernel/config
The following section walks through a configuration example.
- To get started with the interactive mode and the nvmetcli command prompt, type (in root):
# ./nvmetcli ...>
- Create a subsystem. If you do not specify a name a NQN will be generated, which is probably the best choice. We don’t do it here as the name would be random:
> cd /subsystems ...> create testnqn
- Add access for a specific NVMe Host by it’s NQN:
...> cd /hosts ...> create hostnqn ...> cd /subsystems/testnqn ...> set attr allow_any_host=0 ...> cd /subsystems/testnqn/allowed_hosts/ ...> create hostnqn
- Remove access of a subsystem by deleting the Host NQN:
...> cd /subsystems/testnqn/allowed_hosts/ ...> delete hostnqn
- Alternatively this allows any Host to connect to the subsystsem. Only use this in tightly controlled environments:
...> cd /subsystems/testnqn/ ...> set attr allow_any_host=1
- Create a new namespace. If you do not specify a namespace ID the fist unused one will be used:
...> cd /subsystems/testnqn/namespaces ...> create 1 ...> cd 1 ...> set device path=/dev/nvme0n1 ...> enable
Note that in the above setup the device_nguid attribute does not have to be set for correct NVMe Target functionality (but to correctly match a namespace to the exact device upon clear and restore operations, it is advised to set the device_nguid parameter).
- Create a loopback port that can be used with nvme-loop module on the same physical machine...
...> cd /ports/ ...> create 1 ...> cd 1/ ...> set addr trtype=loop ...> cd subsystems/ ...> create testnqn
- or create an RDMA (IB, RoCE, iWarp) port using IPv4 addressing. 4420 is the IANA assigned default port for NVMe over Fabrics using RDMA:
...> cd /ports/ ...> create 2 ...> cd 2/ ...> set addr trtype=rdma ...> set addr adrfam=ipv4 ...> set addr traddr=192.168.6.68 ...> set addr trsvcid=4420 ...> cd subsystems/ ...> create testnqn
- or create an FC port. traddr is the WWNN/WWPN of the FC port.
...> cd /ports/ ...> create 3 ...> cd 3/ ...> set addr trtype=fc ...> set addr adrfam=fc ...> set addr traddr=nn-0x1000000044001123:pn-0x2000000055001123 ...> set addr trsvcid=none ...> cd subsystems/ ...> create testnqn
- Saving the NVMe Target configuration:
./nvmetcli ...> saveconfig test.json
- Loading an NVMe Target configuration:
./nvmetcli restore test.json
- Clearing a current NVMe Target configuration:
nvmetcli has the ability to start and stop the NVMe Target configuration on boot and shutdown through the systemctl Linux utility via a .service file. nvmetcli package comes with nvmet.service which when installed, it can automatically restore the default, saved NVMe Target configuration from /etc/nvmet/config.json. nvmet.service can be installed in directories such as /lib/systemd/system.
To explicitly enable the service, type:
systemctl enable nvmet
To explicitly disable the service, type:
systemctl disable nvmet
See also systemctl(1).
This man page was written by Jay Freyensee. nvmetcli was originally written by Christoph Hellwig.
Reporting Bugs & Development
Please send patches and bug reports to firstname.lastname@example.org for review and acceptance.
nvmetcli is licensed under the Apache License, Version 2.0. Software distributed under this license is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either expressed or implied.
- Jay Freyensee
- Christoph Hellwig