clpbar man page

clpbar — show information about a data transfer

Synopsis

clpbar
[ I/O-options ] [ display-options ] [ color-options ]
[ input-file ] [ output-file ]
[ -h | --help ] [ -v | --version ]

Description

clpbar is a simple tool to process a stream of data and print a display for the user on stderr showing (a) the amount of data passed, (b) the throughput of the data transfer, and, if the total size of the data stream is known, (c) estimated time remaining, percent complete, and a progress bar.

clpbar was originally written for the purpose of estimating the amount of time needed to transfer large amounts (many, many gigabytes) of data across a network. (Usually in an SSH/tar pipe.)

I/O Command Line Options

-if input-file
--in-file input-file

Read input from input-file. Default: stdin

-of output-file
--out-file output-file

Write output to output-file. If the output file is a directory, then clpbar will attempt to create a file in the output directory with the same name as the input file, and attempt to copy the input file mode as well as it's data. Default: stdout

Please notice that if no -if, --in-file, -of, or --out-file options are specified on the command line, and an unknown command line option is encountered, then clpbar will assume that the first unknown command line option is a path to an input file, and the second (if found) is a path to an output file.

-s size
--size size

Expect an input stream of size bytes.

When reading a regular file or a link to a regular file, clpbar will extract the file size on it's own. However, this flag is useful for reading from a character- or block-special device file, or from a pipe. size may be followed by 'k', 'm', 'g', 't', 'p', or 'e' for kilobytes, megabytes, gigabytes, terabytes, petabytes, or exabytes, respectively (see also the -k option below). Alternatively, size may also be specified in terms of 'b' for blocks (see the -bl option below). See examples below.

-c size
--completed size

Instruct clpbar that size bytes of the data stream have already been copied, and that this is a continuation of a previous data stream. Note that use of this option will throw off throughput and ETA calculations at first, but they should settle down as the transfer continues.

-bs buffer-size
--buffer-size buffer-size

Allocate an I/O buffer of buffer-size bytes. The same modifiers may apply here ('k', 'm', 'g', 't', 'p', 'e' and 'b') as for the -s flag above. Changing the buffer size can improve throughput, depending on your application of clpbar. For fast I/O operations, say from a ramdisk for instance, it might be worth your while to experiment with a large buffer (circa 1MB for instance). But for slow I/O operations, like from a tape drive, you could merely be wasting your memory. Default: 52488 (512KB)

-th rate
--throttle rate

Restrict I/O throughput to rate bytes per second. The same modifiers apply here ('k', 'm', 'g', 't', 'p', 'e' and 'b') as for the -s flag above.

-i seconds
--interval seconds

Update the display every seconds seconds. Default: 1 second

-t microseconds
--timeout microseconds

The number of microseconds to wait for a change in I/O state before select() times out. Default: 250000 (1/4 second)

-k 1000|1024
--kilo 1000|1024

Use either 1000 or 1024 as the definition of a kilobyte. Default: 1024

-bl size
--block-size size

When reading sizes from the command line that are specified in terms of blocks, assume a single block is size bytes. Size may be followed by 'k', 'm', 'g', 't', 'p', or 'e' for kilobytes, megabytes, gigabytes, terabytes, petabytes, or exabytes, respectively. Block size must be set before specifying any sizes in terms of blocks or the default value will be used instead. Specifying size in terms of 'b' for blocks is not allowed for this option. Default: 512

Display Command Line Options

-sw width
--screen-width width

Assume a screen width of width characters.

clpbar will attempt to retrieve the width of the terminal it is running on, and will adjust that width if the terminal is resized. If clpbar cannot determine the terminal width, then clpbar will assume a default width of 79 characters. Use the --screen-width command line option to override this behavior and specify a fixed width for clpbar to use. (When this option is used, clpbar will ignore terminal resized signals and continue to use the value provided by the user.)

-sw-1 | --screen-width-minus-one
-sw-0 | --screen-width-minus-zero

Instruct clpbar to use either the entire column width reported by termio, or one less than reported by termio. I.e. If termio reports that you are running clpbar in a terminal that's 80 characters wide, using the command line option --screen-width-minus-one instructs clpbar to only use 79 characters to print the display. If you're using a terminal or shell that wraps the line whenever clpbar prints the last character then this should alleviate that problem. Default is to use the full terminal's width.

-ti string | --title string

Set the title to string.

-dti | -nti
--display-title | --no-title

Turn on/off the title display. Even if on, if no title string is set then no title will be displayed. Default is on.

-dtw | --display-twiddle
-ntw | --no-twiddle

Turn on/off the twiddle in the display.

-dc | --display-count
-nc | --no-count

Turn on/off the data count in the display. Default is on.

-dcb | -ncb
--display-count-bits | --no-count-bits

Display the data count at bits instead of as bytes. Default is off.

By default clpbar will display the data count as bytes using the notation of "B". Using this option, clpbar will display the throughput as bits using the notation of "b".

-dth | --display-throughput
-nth | --no-throughput

Turn on/off the data throughput in the display. Default is on.

-dthb | -nthb
--display-throughput-bits | --no-throughput-bits

Display throughput as bits/second instead of as bytes/second. Default is off.

By default clpbar will display the throughput as bytes/second using the notation of "B/s". Using this option, clpbar will display the throughput as bits/second using the notation of "b/s".

-dt | --display-time
-nt | --no-time

Turn on/off the time elapsed or eta in the display. Default is on.

-de | --display-elapsed-only
-ne | --no-elapsed-only

Force clpbar to display the elapsed time instead of the eta. Default is off.

-dp | --display-percent
-np | --no-percent

Turn on/off percent complete in the display. Default is on.

-db | --display-bar
-nb | --no-bar

Turn on/off the progress bar in the display. Default is on.

-ds | --display-summary
-ns | --no-summary

Turn on/off the summary information displayed when the operation is complete. Default is on.

-da | --display-all
-dn | --display-none

Turn on/off all displays. -dn is equivalent to -ntw -nc -nth -nt -np -nb. (Using -dn followed by -db would be equivalent to -ntw -nc -nth -nt -np.) -da is equivalent to -dtw -dc -dth -dt -dp -db.

Color Command Line Options

For the following color-specific command line options, the following keywords are recognized as valid color names: normal, black, red, green, yellow, blue, magenta, cyan, and white

-dan | --display-ansi
-nan | --no-ansi

Turn on/off the use of ansi color codes in the display.

-spbg color | --space-background color

Use color as the background color for spacing between display objects. Default: normal

-twfg color | --twiddle-foreground color
-twbg color | --twiddle-background color

Use color as the twiddle color in the display. Default: normal

-twb | --twiddle-bold
-twn | --twiddle-normal

Turn on/off the use of bold font when displaying the twiddle. Default off

-tifg color | --title-foreground color
-tibg color | --title-background color

Use color as the title color in the display. Default: normal

-tib | --title-bold
-tin | --title-normal

Turn on/off the use of bold font when displaying the title. Default off

-cfg color | --count-foreground color
-cbg color | --count-background color

Use color as the data count color in the display. Default: normal

-cb | --count-bold
-cn | --count-normal

Turn on/off the use of bold font when displaying the data count. Default off

-thlfg color | --throughput-label-foreground color
-thlbg color | --throughput-label-background color

Use color as the throughput label color in the display. Default: normal

-thlb | --throughput-label-bold
-thln | --throughput-label-normal

Turn on/off the use of bold font when displaying the throughput label. Default off

-thfg color | --throughput-foreground color
-thbg color | --throughput-background color

Use color as the throughput color in the display. Default: normal

-thb | --throughput-bold
-thn | --throughput-normal

Turn on/off the use of bold font when displaying the throughput. Default off

-tlfg color | --time-label-foreground color
-tlbg color | --time-label-background color

Use color as the time label color in the display. Default: normal

-tlb | --time-label-bold
-tln | --time-label-normal

Turn on/off the use of bold font when displaying the time label. Default off

-tfg color | --time-foreground color
-tbg color | --time-background color

Use color as the time color in the display. Default: normal

-tb | --time-bold
-tn | --time-normal

Turn on/off the use of bold font when displaying the time. Default off

-pfg color | --percent-foreground color
-pbg color | --percent-background color

Use color as the percent color in the display. Default: normal

-pb | --percent-bold
-pn | --percent-normal

Turn on/off the use of bold font when displaying the percent. Default off

-bbfg color | --bar-brace-foreground color
-bbbg color | --bar-brace-background color

Use color as the brace color around the progress bar in the display. Default: normal

-bbb | --bar-brace-bold
-bbn | --bar-brace-normal

Turn on/off the use of bold font when displaying the bar braces. Default off

-bfg color | --bar-foreground color
-bbg color | --bar-background color

Use color as the color of the progress bar in the display. Default: normal

-bb | --bar-bold
-bn | --bar-normal

Turn on/off the use of bold font when displaying the progress bar. Default off

-h | --help

Display this text and exit.

-v | --version

Display the program version and exit.

Resource File Options

Some command line options may be specified in a resource file. clpbar will search for a resource file by the name of ~/.clpbarrc and, if found, clpbar will use the values within by default. In addition, clpbar will also search for a file in the directory in which it is run, ./.clpbarrc. If this file exists, it's values will override the values found in ~/.clpbarrc. Values in both files may be overridden by command line flags. Lines that begin with a # are ignored.

For resource options requiring a boolean value, the following values are recognized: on and off, yes and no, (and the single-character abbreviations y and n), true and false, (and the single-character abbreviations t and f), 0 and 1.

For resource options requiring a color value, the same keywords are recognized as for the color-specific command line options above: normal, black, red, green, yellow, blue, magenta, cyan, and white

buffer-size: buffer-size

Allocate an I/O buffer of buffer-size bytes. See the --buffer-size command line option above.

throttle: rate

Restrict I/O throughput to rate bytes per second. See the --throttle command line option above.

interval: seconds

Update the display every seconds seconds. See the --interval command line option above.

timeout: microseconds

The number of microseconds to wait for a change in I/O state before select() times out. See the --timeout command line option above.

kilobyte: 1000|1024

Use either 1000 or 1024 as the definition of a kilobyte. See the --kilo command line option above.

block-size: size

When parsing sizes specified in terms of blocks, assume a single block is size bytes. See the --block-size command line option above.

screen-width: width

Override termio and assume that the screen is width characters wide. See the --screen-width command line option above.

screen-width-minus-one: boolean

Instruct clpbar to restrict the number of columns reported by termio by one. See the --screen-width-minus-one command line option above.

display-twiddle: boolean

Instruct clpbar to turn on/off the twirling twiddle character in the display. See the --display-twiddle command line option above.

display-title: boolean

Instruct clpbar to turn on/off the title in the display. See the --display-title command line option above.

display-count: boolean

Instruct clpbar to turn on/off the data count in the display. See the --display-count command line option above.

display-count-bits: boolean

Display the data count as bits instead of as bytes. See the --display-count-bits command line option above.

display-throughput: boolean

Instruct clpbar to turn on/off the data throughput in the display. See the --display-throughput command line option above.

display-throughput-bits: boolean

Display throughput as bits/sec instead of as bytes/sec. See the --display-throughput-bits command line option above.

display-time: boolean

Instruct clpbar to turn on/off the time in the display. See the --display-time command line option above.

display-elapsed-only: boolean

Force clpbar to display the elapsed time instead of the eta. See the --display-elapsed-only command line option above.

display-percent: boolean

Instruct clpbar to turn on/off the percent complete in the display. See the --display-percent command line option above.

display-bar: boolean

Instruct clpbar to turn on/off the progress bar in the display. See the --display-bar command line option above.

display-summary: boolean

Instruct clpbar to turn on/off the summary information displayed when operation is complete. See the --display-summary command line option above.

display-ansi: boolean

Instruct clpbar to turn on/off the use of ansi color codes in the display. See the --display-ansi command line option above.

space-background: color

Use color as the background color for spacing between display objects. See the --space-background command line option above.

twiddle-foreground: color
twiddle-background: color
twiddle-bold: boolean

Use the specified colors for the foreground and background of the twiddle, and use a bold font. See the --twiddle-foreground, --twiddle-background, and --twiddle-bold command line options above.

title: string

Set the title string for the display. See the --title command line option above.

title-foreground: color
title-background: color
title-bold: boolean

Use the specified colors for the foreground and background of the title, and use a bold font. See the --title-foreground, --title-background, and --title-bold command line options above.

count-foreground: color
count-background: color
count-bold: boolean

Use the specified colors for the foreground and background of the data count, and use a bold font. See the --count-foreground, --count-background, and --count-bold command line options above.

throughput-label-foreground: color
throughput-label-background: color
throughput-label-bold: boolean

Use the specified colors for the foreground and background of the throughput label, and use a bold font. See the --throughput-label-foreground, --throughput-label-background, and --throughput-label-bold command line options above.

throughput-foreground: color
throughput-background: color
throughput-bold: boolean

Use the specified colors for the foreground and background of the throughput, and use a bold font. See the --throughput-foreground, --throughput-background, and --throughput-bold command line options above.

time-label-foreground: color
time-label-background: color
time-label-bold: boolean

Use the specified colors for the foreground and background of the time label, and use a bold font. See the --time-label-foreground, --time-label-background, and --time-label-bold command line options above.

time-foreground: color
time-background: color
time-bold: boolean

Use the specified colors for the foreground and background of the time, and use a bold font. See the --time-foreground, --time-background, and --time-bold command line options above.

percent-foreground: color
percent-background: color
percent-bold: boolean

Use the specified colors for the foreground and background of the percent, and use a bold font. See the --percent-foreground, --percent-background, and --percent-bold command line options above.

bar-brace-foreground: color
bar-brace-background: color
bar-brace-bold: boolean

Use the specified colors for the foreground and background of the brace surrounding the progress bar, and use a bold font. See the --bar-brace-foreground, --bar-brace-background, and --bar-brace-bold command line options above.

bar-foreground: color
bar-background: color
bar-bold: boolean

Use the specified colors for the foreground and background of the progress bar, and use a bold font. See the --bar-foreground, --bar-background, and --bar-bold command line options above.

Examples

Example 1: Using clpbar to copy a 2.4gb file from a device (in this case a tape drive) to a file, using a 64k buffer.

prompt% clpbar --in-file /dev/rmt/1cbn --out-file \
tape-restore.tar --size 2.4g --buffer-size 64k

Example 2: Using clpbar to copy a 37tb file across the network using SSH.

prompt% ssh remote 'dd if=file' | clpbar --size 37t > file

Example 3: Using clpbar inside a tar-pipe command:

Normal tar-pipe command might be:

prompt% (cd /some/dir/somewhere && tar -cf - *) \
| (cd /some/other/dir && tar -xBpf -)

3a: Using clpbar within the tar-pipe:

prompt% (cd /some/dir/somewhere && tar -cf - *) \
| clpbar \
| (cd /some/other/dir && tar -xBpf -)

3b: Using clpbar with the --size option in a tar-pipe:

prompt% du -sk /some/dir/somewhere
6281954 /some/dir/somewhere

prompt% (cd /some/dir/somewhere && tar -cf - *) \
| clpbar --size 6281954k \
| (cd /some/other/dir && tar -xBpf -)

Example 4: Using clpbar on a regular file. (Note that the --size option is not needed here, as clpbar will retrieve the file size itself.)

prompt% clpbar --in-file ./file | ssh remote 'cd /some/dir && dd of=file'

Example 5: Generating a 512k file of random data.

prompt% dd if=/dev/random bs=1024 count=512 \
| clpbar -s 512k -of ./random

Example 6: An example .clpbarrc file.

#
# This is an example of what a ~/.clpbarrc file
# might look like. Note that lines beginning
# with a # are ignored.
#
display-twiddle: no
display-ansi: yes
# space-background: black
twiddle-foreground: green
# twiddle-background: normal
# twiddle-bold: no
count-foreground: green
# count-background: magenta
count-bold: yes
throughput-label-foreground: normal
# throughput-label-background: red
throughput-label-bold: no
throughput-foreground: green
# throughput-background: black
throughput-bold: yes
time-label-foreground: normal
# time-label-background: red
time-label-bold: no
time-foreground: green
# time-background: black
time-bold: yes
percent-foreground: green
# percent-background: green
percent-bold: yes
bar-brace-foreground: red
# bar-brace-background: blue
bar-brace-bold: no
bar-foreground: yellow
# bar-background: blue
bar-bold: yes

Notes

-
The --size option is only used by clpbar in calculating information about the data transfer. clpbar will not cease copying data once it has reached the number of bytes specified with the --size option, but instead clpbar will continue to copy data until and end of input is reached. If this behavior is undesirable then clpbar may be used in conjunction with dd, where the count option is used with dd to specify when to cut off the input stream. (See examples above.)
-
When using other commands such as du -k to calculate the expected size of a data transfer stream, the value returned may not be exactly the number of bytes counted by clpbar in the actual data transfer. Common causes for this discrepancy could be attributed to round-off error or the use of 1000 bytes as a kilobyte rather than 1024. (If the later is the case, then using the -k 1000 option to clpbar will help.) When such discrepancies occur, clpbar may report that the data stream contained only 98% or as much as 101% of it's expected size. (If you have doubts, you should definitely verify your data using md5sum, diff, or cmp.)
-
When the value of a calculation exceeds the size alloted for the display, the value +99... will be substituted in it's place. The complete value will be displayed in a summary statement after clpbar has reached the end of input.
-

clpbar assumes a linear relationship between the speed of the data transfer and the amount of time remaining. Specifically the calculation is based on the following:

elapsed time / eta = bytes written / total size

However, it has been the author's experience that the throughput speed will change, particularly at the beginning of the transfer, and this will affect the estimated time remaining. The author does not believe this is a bug, but a side-effect of this method of calculation.

-
clpbar assumes that there are 8 bits in both a byte and a char.

Bugs

-

clpbar uses the open() and fstat() functions to open and retrieve the size of regular files when using either the --in-file or --out-file command line options. Some OS's do not support Large Files (file sizes up to (2**63)-1 bytes) natively. Some OS's support Large Files but require _FILE_OFFSET_BITS or _LARGE_FILES to be defined properly at compile time. Other OS's support neither, but still allow programs to open files in excess of (2**32)-1 through an O_LARGEFILE option that can be passed to the open() function.

When trying to open files greater than 2gb on an OS without Large File support, clpbar will exit with the message: "File too large". When trying to write more than 2gb of data to a file, clpbar will write 2**32-1 bytes and then the OS may terminate clpbar with a message similar to: "File size limit exceeded".

When trying to open files greater than 2gb on an OS without Large File support, but with the O_LARGEFILE option that can be passed to open(), clpbar will receive an error when trying to retrieve the file's size, but clpbar will be able to open the file anyway. Under these circumstances, clpbar will print a "File too large" error message, but will then proceed to transfer the data. Since clpbar will not be able to retrieve the file's size on it's own, the --size command line option must be used after the --in-file option to tell clpbar the file size manually. On such OS's, clpbar should be able to write more than 2gb of data to a file without any problems.

For OS's that support files greater than 2gb, either natively or through the Large File extension definitions mentioned above, clpbar should work as expected.

-
The author has noticed that when running clpbar over an SSH connection, sometimes window resize events are not captured until after the display has gone through one or two more updates, which can cause the line to wrap.
-
The author has noticed that on some systems the use of aligned memory allocation, through either memalign() or posix_memalign(), causes clpbar to commit a segmentation fault the first time read() or readv() is called and passed a pointer to the aligned memory as it's input buffer. Attempts were made to try to isolate systems in which this bug bites through tests in configure, but all tests devised passed with flying colors. Therefore aligned memory allocation is turned off by default, and may only be enabled by passing --enable-use-memalign to configure when building the executable.

Report all bugs to the author.

clpbar was developed on a Sun workstation running Solaris 8. To the best of the author's knowledge clpbar should compile and run on other platforms without much trouble. Should other OS's require modifications to the code, the author welcomes all patch submissions, but requests that you include the file config.log and the output of gcc -dumpspecs (or a listing of predefined variables, if not using gcc).

Distribution

The latest version of clpbar can always be found at:

Author

clpbar was written by Michael Peek. See Distribution above for contact information.

Occasionally, the author fancies that he knows what he's doing. It is at these times more than ever that his coworkers should cower in fear...

Info

4 November 2003