ublk - Man Page

ublk <command> [<command options>] [<type specific options>]

ublk is a utility that allows you to create, recover, view or delete user-space block devices.

ublk by default comes with several different types of devices, such as iscsi, nbd, nfs, null and loop.

The following commands are supported:

Command to add a ublk device.

add {-t, --type} TYPE [{-n, --number} DEV_ID] [{-q, --queues} NR_HW_QUEUES] [{-d, --depth} QUEUE_DEPTH] [{-u, --uring_comp} URING_COMP] [{-g, --need-get-data} NEED_GET_DATA] [{-r, --user_recovery} {0|1}] [{-i, --user_recovery_reissue} {0|1}] [{-e, --user_recovery_fail_io} {0|1}] [--debug_mask=0x{DBG_MASK}] [--unprivileged] [--usercopy] [--max_io_buf_bytes={BYTES}] [{-z, --zerocopy}] [<type specific options>]

-t,  --type

Specifies the type of device to create. The five types of supported devices are iscsi, nbd, nfs, null and loop.

-n,  --number

Create a device with this id. The device node will be /dev/ublkb_n_.

-q,  --queue

Number of queues to create. Each queue is services by a dedicated child process. Default is 1.

-d,  --depth

Maximum queue-depthfor each queue. Default is 4096.

-u,  --uring_comp

Force to complete io cmd via io_uring_cmd_complete_in_task so that performance comparison is done easily with using task_work_add.

-g,  --need_get_data

User should issue io cmd again for write requests to set io buffer address and copy data from bio vectors to the userspace io buffer.

-r,  --user_recovey

Block devices are recoverable if ublk server exits and restarts. Outstanding I/O when ublk server exits is met with errors. I/O issued while there is no ublk server queues.

-i,  --user_recovey_reissue

Block devices are recoverable if ublk server exits and restarts Outstanding I/O when ublk server exits is reissued I/O issued while there is no ublk server queues

-e,  --user_recovey_fail_io

Block devices are recoverable if ublk server exits and restarts Outstanding I/O when ublk server exits is met with errors I/O issued while there is no ublk server is met with errors

--debug_mask

Bitmask specifying which debug features to enable.

--max_io_buf_bytes

Maximum I/O size. Default is 512kb.

NOTE: Memory buffers are pre-allocated with one buffer for each entry in queue_depth.

--unprivileged

Unprivileged user can create /dev/ublkcN and /dev/ublkbN.

/dev/ublk-control needs to be available for unprivileged user, and it can be done via udev rule to make all control commands available to unprivileged user. Except for the command of UBLK_CMD_ADD_DEV, all other commands are only allowed for the owner of the specified device.

When userspace sends UBLK_CMD_ADD_DEV, the device pair's owner_uid and owner_gid are stored to ublksrv_ctrl_dev_info by kernel, so far only the current user's uid/gid is stored, that said owner of the created device is always the current user.

We still need udev rule to apply OWNER/GROUP with the stored owner_uid and owner_gid.

Then ublk server can be run as unprivileged user, and /dev/ublkbN can be accessed and managed by its owner represented by owner_uid/owner_gid.

--user_copy

Copy between request and user buffer by pread()/pwrite()

-z,  --zerocopy

Zero-copy is based on io-uring uring_cmd of REGISTER_IO_BUF & UNREGISTER_IO_BUF, which avoids data copy between ublk frontend request buffer and ublk server buffer, so memory bandwidth is saved, and throughput & latency improvement can be often observed on large I/O size

This requires Linux kernel 6.15 or later.

    # ublk add -t loop -n 0 -f 10M.raw

    # ublk add -t nfs -n 0 --nfs nfs://10.0.0.1/export/10M.raw

    # ublk add -t iscsi -n 0 --iscsi iscsi://iscsi-stgt/iqn.2001-04.com.ronnie.sr0/1 --initiator-name iqn.ronnie.test

Command to delete a ublk device.

del {-n, --number} DEV_ID [-a, --all] [--async]

-n, --number

Delete the device with this id.

-a, --all

Delete all devices.

Example: Deleting a loop block device

    # ublk del -n 0

Command to change queue affinity.

set_affinity {-n, --number} DEV_ID [-q, --queue] QID --cpuset="[SET]"

-n, --number

Change the affinity on this device.

-q, --queue

Which queue to change the affinity for.

--cpuset="[SET]"

The new cpuset for this device/queue. Format is a comma-separated list of CPUs within squre brackets.

Example: Set affinity to core 7 for device 0, queue 1

    # ublk set_affinity -n 0 -q 1 --cpuset="[7]"

List one or all devices and show their configutaion.

list {-n, --number} DEV_ID [-v, --verbose]

-n, --number

List the device with this id. If omitted all devices will be listed

-v, --verbose

Verbose listing. Include the JSON device arguments in the output.

Recover a failed ublk device.

recover {-n, --number} DEV_ID

-n, --number

Device to recover.

Show supported features for the ublk driver.

features

Show generic ot type specific help.

help [{-t, --type} TYPE]

-t, --type

Show help page. It -t is specified, show help page for the specific device type.

Show help page..

{-v, --version}

There are three arguments that control how ublk will behave in case of a failure, such as crashing. The default behavior is no recovery and the device will fail and be removed once the target exists.

To enable recovery mode set "--recovery 1" on the command line. Then instead of removing the device upon failure it will instead become inactive in a quiesced state.

dev id 0: nr_hw_queues 1 queue_depth 128 block size 4096 dev_capacity 20480
    max rq size 524288 daemon pid 1239110 state QUIESCED
    flags 0x4a [ URING_CMD_COMP_IN_TASK RECOVERY CMD_IOCTL_ENCODE ]
    ublkc: 511:0 ublkb: 259:4 owner: 0:0
    queue 0: tid 1239112 affinity(0 1 2 3 4 5 6 7 )
    target {"backing_file":"10M","dev_size":10485760,"direct_io":1,"name":"loop","offset":0,"type":0}

In this state the block device still exists but no I/O can be performed.

To recover a QUIESCED device you can use the recover command: ublk recover -n DEV_ID

There are two additional flags that control how ublk will handle I/O that were in flight when a device is recovered.

http://github.com/ublk-org/ublksrv