hsx - Man Page

FILE

Path to input task file (default: <stdin>).

--ssh HOST...

Launch directly with SSH host(s). This can be a single host, a comma-separated list of hosts, or an expandable pattern, e.g., "cluster-a[00-04].xyz".

See also --ssh-group and --ssh-args.

--mpi

Same as --launcher=mpirun.

--launcher ARGS...

Use specific launch interface. This can be any program that handles process management on a distributed system. For example, on a SLURM cluster one might want to use srun. In this case you would specify --launcher=srun; however, the ARGS are not merely the executable but the full listing, e.g., --launcher='srun --mpi=pmi2'.

-N,  --num-tasks NUM

Number of task executors per client (default: 1).

For example, -N4 would create four workers, but -N4 --ssh 'cluster-a[00-01].xyz' creates two clients and a total of eight workers.

-t,  --template CMD

Command-line template pattern (default: "{}").

This is expanded by the client just before execution. With the default "{}" the input command-line will be run verbatim. Specifying a template pattern allows for simple input arguments (e.g., file paths) to be transformed into some common form; such as -t './some_command.py {} >outputs/{/-}.out'.

See section on templates.

-p,  --port NUM

Port number (default: 50001).

This is an arbitrary choice and simply must be an available port. The default option chosen here is typically available on most platforms and is not expected by any known major software.

-b,  --bundlesize SIZE

Size of task bundle (default: 1).

The default value allows for greater concurrency and responsiveness on small scales. This is used by the submit thread to accumulate bundles for either database commits and/or publishing to the queue. If a database is in use, the scheduler thread selects tasks from the database in batches of this size.

Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

See also --num-tasks and --bundlewait.

-w,  --bundlewait SEC

Seconds to wait before flushing tasks (default: 5).

This is used by both the submit thread and forwarded to each client. The client collector thread that accumulates finished task bundles to return to the server will push out a bundle after this period of time regardless of whether it has reached the preferred bundle size.

See also --bundlesize.

-r,  --max-retries NUM

Auto-retry failed tasks (default: 0).

If a database is in use, then there is an opportunity to automatically retry failed tasks. A task is considered to have failed if it has a non-zero exit status. Setting this value greater than zero defines the number of attempts for the task. The original is not over-written, a new task is submitted and later scheduled.

See also --eager.

--eager

Schedule failed tasks before new tasks. If --max-retries is greater than one, this option defines the appetite for re-submitting failed tasks. By default, failed tasks will only be scheduled when there are no more remaining novel tasks.

--no-db

Disable database (submit directly to clients).

By default, a scheduler thread selects tasks from a database that were previously submitted. With --no-db enabled, there is no scheduler and instead the submit thread publishes bundles directly to the queue.

--initdb

Auto-initialize database.

If a database is configured for use with the workflow (e.g., Postgres), auto-initialize tables if they don't already exist. This is a short-hand for pre-creating tables with the hs initdb command. This happens by default with SQLite databases.

Mutually exclusive to --no-db. See hs initdb command.

--no-confirm

Disable client confirmation of task bundle received.

To achieve even higher throughput at large scales, optionally disable confirmation payloads from clients. Consider using this option when also using --no-db.

--forever

Schedule forever.

Typically, the cluster will process some finite set of submitted tasks. When there are no more tasks left to schedule, the cluster will begin its shutdown procedure. With --forever enabled, the scheduler will continue to wait for new tasks indefinitely.

Conflicts with --no-db and mutually exclusive to --restart.

--restart

Start scheduling from last completed task.

Instead of pulling a new list of tasks from some input FILE, with --restart enabled the cluster will restart scheduling tasks where it left off. Any task in the database that was previously scheduled but not completed will be reverted.

For very large workflows, an effective strategy is to first use the submit workflow to populate the database, and then to use --restart so that if the cluster is interrupted, it can easily continue where it left off, halting if nothing to be done.

Conflicts with --no-db and mutually exclusive to --forever.

--ssh-args ARGS...

Command-line arguments for SSH. For example, --ssh-args '-i ~/.ssh/my_key'.

--ssh-group NAME

SSH nodelist group in config.

In your configuration under [ssh.nodelist] can be one or more named lists. These lists should contain host names to associate with the group name.

See configuration section.

-E,  --env

Send environment variables. Only for --ssh mode, all HYPERSHELL_ prefixed environment variables can be exported to the remote clients.

--remote-exe PATH

Path to remote executable on the client side.

By default the executable path used to launch clients is the same as that for the cluster. This option is necessary for clients with a different install path on client hosts.

-d,  --delay-start SEC

Delay time in seconds for launching clients (default: 0).

At larger scales it can be advantageous to uniformly delay the client launch sequence. Hundreds or thousands of clients connecting to the server all at once is a challenge. Even if the server could handle the load, your task throughput would be unbalanced, coming in waves.

Use --delay-start with a negative number to impose a uniform random delay up to the magnitude specified (e.g., --delay-start=-600 would delay the client up to ten minutes). This also has the effect of staggering the workload. If your tasks take on the order of 30 minutes and you have 1000 nodes, choose --delay-start=-1800.

-c,  --capture

Capture individual task <stdout> and <stderr>.

By default, the stdout and stderr streams of all tasks are fused with that of the client thread, and in turn the cluster. If tasks are producing output that needs to be isolated, the tasks need to manage their own output, you can specify a redirect as part of a --template, or use --capture to capture these as .out and .err files.

These are stored local to the client. Task outputs can be automatically retrieved via SFTP, see task usage.

-o,  --output PATH

File path for task outputs (default: <stdout>).

If local only (not --ssh, --mpi or --launcher), then the client can redirect all stdout from tasks to some file PATH together.

-e,  --errors PATH

File path for task errors (default: <stderr>).

If local only (not --ssh, --mpi or --launcher), then the client can redirect all stderr from tasks to some file PATH together.

-f,  --failures PATH

File path to write failed task args (default: <none>).

The server acts like a sieve, reading task args from stdin and redirecting those original args to stdout if the task had a non-zero exit status. The cluster will run the server for you and if --failures is enabled these task args will be sent to a local file PATH.

-T,  --timeout SEC

Timeout in seconds for clients. Automatically shutdown if no tasks received (default: never).

This option is only valid for an --autoscaling cluster. This feature allows for gracefully scaling down a cluster when task throughput subsides.

-W,  --task-timeout SEC

Task-level walltime limit (default: none).

Executors will send a progression of SIGINT, SIGTERM, and SIGKILL. If the process still persists the executor itself will shutdown.

-S,  --signalwait SEC

Task-level signal escalation wait period in seconds (default: 10).

When tasks fail to halt following an initial SIGINT, the program waits this interval in seconds before escalating to the next level of interrupt.

See also --task-timeout.

-A,  --autoscaling [MODE]

Enable autoscaling (default: disabled). Used with --launcher.

Specifying this option on its own triggers the use of the autoscaler, with the default policy or the configured policy. The policy can be specified directly here as either fixed or dynamic (e.g., --autoscaling=dynamic). The default is fixed.

The specified --launcher is used to bring up each individual instance of the client as a discrete scaling unit. This is different than using --launcher on its own where it specifies a single invocation that should launch all clients (e.g., like an mpirun). Without this option, clients will simply be run locally.

A fixed policy will seek to maintain a definite size and allows for recovery in the event that clients halt for some reason (e.g., due to expected faults or timeouts).

A dynamic policy maintains a --min-size (default: 0) and grows up to some --max-size depending on the observed task pressure given the specified scaling --factor.

See also --factor, --period, --init-size, --min-size, and --max-size.

-F,  --factor VALUE

Scaling factor (default: 1).

A configurable, dimensionless quantity used by the --autoscaling=dynamic policy. This value expresses some multiple of the average task duration in seconds.

The autoscaler periodically checks toc / (factor x avg_duration), where toc is the estimated time of completion for all remaining tasks given current throughput of active clients. This ratio is referred to as task pressure, and if it exceeds 1, the pressure is considered high and we will add another client if we are not already at the given --max-size of the cluster.

For example, if the average task length is 30 minutes, and we set --factor=2, then if the estimated time of completion of remaining tasks given currently connected executors exceeds 1 hour, we will scale up by one unit.

See also --period.  Only valid with --autoscaling.

-P,  --period SEC

Scaling period in seconds (default: 60).

The autoscaler waits for this period of time in between checks and scaling events. A shorter period makes the scaling behavior more responsive but can effect database performance if checks happen too rapidly.

Only valid with --autoscaling.

-I,  --init-size SIZE

Initial size of cluster (default: 1).

When the cluster starts, this number of clients will be launched. For a fixed policy cluster, this should be given with a --min-size, and likely the same value.

Only valid with --autoscaling.

-X,  --min-size SIZE

Minimum size of cluster (default: 0).

Regardless of autoscaling policy, if the number of launched clients drops below this value we will scale up by one. Allowing --min-size=0 is an important feature for efficient use of computing resources in the absence of tasks.

Only valid with --autoscaling.

-Y,  --max-size SIZE

Maximum size of cluster (default: 2).

For a dynamic autoscaling policy, this sets an upper limit on the number of launched clients. When this number is reached, scaling stops regardless of task pressure.

Only valid with --autoscaling.

FILE

Path to input task file (default: <stdin>).

-H, --bind ADDR

Bind address (default: localhost).

When running locally, the default is recommended. To allow remote clients to connect over the network, bind the server to 0.0.0.0.

-p, --port NUM

Port number (default: 50001).

This is an arbitrary choice and simply must be an available port. The default option chosen here is typically available on most platforms and is not expected by any known major software.

-k, --auth KEY

Cryptographic authentication key to connect with server (default: <not secure>).

The default KEY used by the server and client is not secure and only a place holder. It is expected that the user choose a secure KEY. The cluster automatically generates a secure one-time KEY.

-b, --bundlesize SIZE

Size of task bundle (default: 1).

The default value allows for greater concurrency and responsiveness on small scales. This is used by the submit thread to accumulate bundles for either database commits and/or publishing to the queue. If a database is in use, the scheduler thread selects tasks from the database in batches of this size.

Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

See also --num-tasks and --bundlewait.

-w, --bundlewait SEC

Seconds to wait before flushing tasks (default: 5).

This is used by both the submit thread and forwarded to each client. The client collector thread that accumulates finished task bundles to return to the server will push out a bundle after this period of time regardless of whether it has reached the preferred bundle size.

See also --bundlesize.

-r, --max-retries NUM

Auto-retry failed tasks (default: 0).

If a database is in use, then there is an opportunity to automatically retry failed tasks. A task is considered to have failed if it has a non-zero exit status. Setting this value greater than zero defines the number of attempts for the task. The original is not over-written, a new task is submitted and later scheduled.

See also --eager.

--eager

Schedule failed tasks before new tasks. If --max-retries is greater than one, this option defines the appetite for re-submitting failed tasks. By default, failed tasks will only be scheduled when there are no more remaining novel tasks.

--no-db

Disable database (submit directly to clients).

By default, a scheduler thread selects tasks from a database that were previously submitted. With --no-db enabled, there is no scheduler and instead the submit thread publishes bundles directly to the queue.

--initdb

Auto-initialize database.

If a database is configured for use with the workflow (e.g., Postgres), auto-initialize tables if they don't already exist. This is a short-hand for pre-creating tables with the hs initdb command. This happens by default with SQLite databases.

Mutually exclusive to --no-db. See hs initdb command.

--no-confirm

Disable client confirmation of task bundle received.

To achieve even higher throughput at large scales, optionally disable confirmation payloads from clients. Consider using this option when also using --no-db.

--forever

Schedule forever.

Typically, the cluster will process some finite set of submitted tasks. When there are no more tasks left to schedule, the cluster will begin its shutdown procedure. With --forever enabled, the scheduler will continue to wait for new tasks indefinitely.

Conflicts with --no-db and mutually exclusive to --restart.

--restart

Start scheduling from last completed task.

Instead of pulling a new list of tasks from some input FILE, with --restart enabled the cluster will restart scheduling tasks where it left off. Any task in the database that was previously scheduled but not completed will be reverted.

For very large workflows, an effective strategy is to first use the submit workflow to populate the database, and then to use --restart so that if the cluster is interrupted, it can easily continue where it left off, halting if nothing to be done.

Conflicts with --no-db and mutually exclusive to --forever.

--print

Print failed task args to <stdout>.

Mutually exclusive to -f/--failures.

-f, --failures PATH

File path to write failed task args (default: <none>).

The server acts like a sieve, reading task args from some input source. Tasks with a non-zero exit status can have their original command-line args printed to an output stream. With -f/--failures, specify a local file PATH.

Mutually exclusive to --print.

-N, --num-tasks NUM

Number of task executors (default: 1).

-t, --template CMD

Command-line template pattern (default: "{}").

This is expanded by the client just before execution. With the default "{}" the input command-line will be run verbatim. Specifying a template pattern allows for simple input arguments (e.g., file paths) to be transformed into some common form; such as -t './some_command.py {} >outputs/{/-}.out'.

See section on templates.

-b, --bundlesize SIZE

Size of task bundle (default: 1).

Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

See also --num-tasks and --bundlewait.

-w, --bundlewait SEC

Seconds to wait before flushing tasks (default: 5).

The client collector thread that accumulates finished task bundles to return to the server will push out a bundle after this period of time regardless of whether it has reached the preferred bundle size.

See also --bundlesize.

-H, --host ADDR

Hostname or IP address to connect with server (default: localhost).

-p, --port NUM

Port number to connect with server (default: 50001).

-k, --auth KEY

Cryptographic authentication key to connect with server (default: <not secure>).

The default KEY used by the server and client is not secure and only a place holder. It is expected that the user choose a secure KEY. The cluster automatically generates a secure one-time KEY.

-d, --delay-start SEC

Delay time in seconds before connecting to server (default: 0).

At larger scales it can be advantageous to uniformly delay the client launch sequence. Hundreds or thousands of clients connecting to the server all at once is a challenge. Even if the server could handle the load, your task throughput would be unbalanced, coming in waves.

Use --delay-start with a negative number to impose a uniform random delay up to the magnitude specified (e.g., --delay-start=-600 would delay the client up to ten minutes). This also has the effect of staggering the workload. If your tasks take on the order of 30 minutes and you have 1000 nodes, choose --delay-start=-1800.

--no-confirm

Disable client confirmation of task bundle received.

To achieve even higher throughput at large scales, optionally disable confirmation payloads from clients. Consider using this option when also using --no-db.

-o, --output PATH

File path for task outputs (default: <stdout>).

-e, --errors PATH

File path for task errors (default: <stderr>).

-c, --capture

Capture individual task <stdout> and <stderr>.

By default, the stdout and stderr streams of all tasks are fused with that of the client thread, and in turn the cluster. If tasks are producing output that needs to be isolated, the tasks need to manage their own output, you can specify a redirect as part of a --template, or use --capture to capture these as .out and .err files.

These are stored local to the client under <prefix>/lib/task/<uuid>.[out,err]. Task outputs can be automatically retrieved via SFTP, see task usage.

Mutually exclusive with both --output and --errors.

--timeout SEC

Timeout in seconds for client. Automatically shutdown if no tasks received (default: never).

This feature allows for gracefully scaling down a cluster when task throughput subsides.

--task-timeout SEC

Task-level walltime limit (default: none).

Executors will send a progression of SIGINT, SIGTERM, and SIGKILL. If the process still persists the executor itself will shutdown.

-S, --signalwait SEC

Task-level signal escalation wait period in seconds (default: 10).

When tasks fail to halt following an initial SIGINT, the program waits this interval in seconds before escalating to the next level of interrupt.

See also --task-timeout.

ARGS...

Command-line task arguments (for single task submission).

-f, --task-file FILE

Input file containing one task per line.

--template CMD

Command-line template pattern (default: "{}").

This is expanded at submit-time before sending to the database. With the default "{}" the input command-line will be run verbatim. Specifying a template pattern allows for simple input arguments (e.g., file paths) to be transformed into some common form; such as --template './some_command.py {} >outputs/{/-}.out'.

See section on templates.

-b, --bundlesize SIZE

Size of task bundle (default: 1).

The default value allows for greater concurrency and responsiveness on small scales. Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

See also --bundlewait.

-w, --bundlewait SEC

Seconds to wait before flushing tasks (default: 5).

If this period of time expires since the previous bundle was pushed to the database, The current bundle will be pushed regardless of how many tasks have been accumulated.

See also --bundlesize.

--initdb

Auto-initialize database.

If a database is configured for use with the workflow (e.g., Postgres), auto-initialize tables if they don't already exist. This is a short-hand for pre-creating tables with the hs initdb command. This happens by default with SQLite databases.

See hs initdb command.

-t, --tag TAG...

Assign one or more tags.

Tags allow for user-defined tracking of information related to individual tasks or large groups of tasks. They are defined with both a key and value (e.g., --tag file:a). The default value for tags is blank. When searching with tags, not specifying a value will return any task with that key defined regardless of value (including blank).

--vacuum

Vacuum an existing database.

--backup PATH

Vacuum into backup file (SQLite only).

--rotate

Rotate completed tasks to new database (SQLite only).

-t, --truncate

Truncate database (task metadata will be lost).

-y,  --yes

Auto-confirm action (default will prompt).

ID

Unique task UUID.

-f, --format FORMAT

Format task info ([normal], json, yaml).

--json

Format metadata output as JSON.

--yaml

Format metadata output as YAML.

-x, --extract FIELD

Print this field only (e.g., -x submit_time).

--stdout

Print <stdout> of task if captured, fetch from client if necessary.

--stderr

Print <stderr> of task if captured, fetch from client if necessary.

ID

Unique task UUID.

-n, --interval SEC

Time in seconds to wait between polling (default: 5).

-i, --info

Print task info after completion.

See task info command.

-f, --format FORMAT

Format task info ([normal], json, yaml).

--json

Format metadata output as JSON.

--yaml

Format metadata output as YAML.

-s, --status

Print task exit status only.

-r, --return

Use exit status from task.

ARGS...

Command-line arguments.

The full command-line for some shell task. To use options, preface with leading --.

-n, --interval SEC

Time in seconds to wait between polling (default: 5).

-t, --tag TAG...

Assign one or more tags.

Tags allow for user-defined tracking of information related to individual tasks or large groups of tasks. They are defined with both a key and value (e.g., --tag file:a). The default value for tags is blank. When searching with tags, not specifying a value will return any task with that key defined regardless of value (including blank).

FIELD

Select specific named fields to include in output. Default is to include all fields.

-w, --where COND...

List of conditional statements to filter results (e.g., -w 'duration >= 600').

Operators include ==, !=, >=, <=, >, <, ~. The ~ operator applies a regular expression.

-t, --with-tag TAG...

Filter on one or more tags. (e.g., -t special or -t file:a).

Leaving the value unspecified will return any task for which the key exists. Specifying a full key:value pair will match on both.

-s, --order-by FIELD [--desc]

Order results by field. Optionally, in descending order.

-F, --failed

Alias for -w 'exit_status != 0'.

-S, --succeeded

Alias for -w 'exit_status == 0'.

-C, --completed

Alias for -w 'exit_status != null'.

For backwards compatibility, --finished is also valid.

-R, --remaining

Alias for -w 'exit_status == null'.

-f, --format FORMAT

Specify output format (either normal, plain, table, csv, json).

Default is normal for whole-task output. If any FIELD names are given, output is formatted in simple plain text; use csv for compliant output. The pretty-printed table formatting is good for presentation on wide screens.

See --csv, --json, and --delimiter.

--csv

Format output as CSV. (Shorthand for --format=csv).

--json

Format output as JSON. (Shorthand for --format=json).

-d, --delimiter CHAR

Field seperator for plain/csv formats.

-l, --limit NUM

Limit the number of results.

-c, --count

Show count of results.

-i, --ignore-partitions

Suppress auto-union feature (SQLite only).

When using the sqlite provider, all databases matching the numbering pattern applied by hs initdb --rotate will automatically be attached as partitions and used within a temporary view, making full task history searchable.

ARGS...

Assignment pairs for update.

--cancel

Cancel specified tasks.

Cancelling a task means it will no longer be scheduled. This is done by setting the schedule_time to now and the exit_status to -1. A task cannot be cancelled after it is sent to remote clients.

--revert

Revert specified tasks.

A reverted task retains its ID and submit info along with any tags. It will be as if it were new and never scheduled.

--delete

Delete specified tasks.

Deleting a task fully drops the record from the database. All task information will be lost and not recoverable.

--remove-tag TAG...

Strip the specified tag from task records.

-w, --where COND...

List of conditional statements to filter results (e.g., -w 'duration >= 600').

Operators include ==, !=, >=, <=, >, <, ~. The ~ operator applies a regular expression.

-t, --with-tag TAG...

Filter on one or more tags. (e.g., -t special or -t file:a).

Leaving the value unspecified will return any task for which the key exists. Specifying a full key:value pair will match on both.

-s, --order-by FIELD [--desc]

Order results by field. Optionally, in descending order.

When used in an update command, a --limit is required. For example, to delete the most recently submitted task, --order-by submit_time --desc --limit 1.

-l, --limit NUM

Limit the number of results.

-F, --failed

Alias for -w 'exit_status != 0'.

-S, --succeeded

Alias for -w 'exit_status == 0'.

-C, --completed

Alias for -w 'exit_status != null'.

-R, --remaining

Alias for -w 'exit_status == null'.

-f, --no-confirm

Do not ask for confirmation.

The program first checks the number of affected tasks. The user must confirm the update interactively unless provided with --no-confirm.

In previous releases of the software the update command had the following signature. Executing the command with these three positional arguments is still valid.

ID

Unique UUID.

FIELD

Task field name (e.g., "args").

VALUE

New value.

Use key:value notation for tags. Updating tag will add or update any pre-existing tag with that key.

SECTION[...].VAR

Path to variable (default is entire configuration).

--system

Load from system configuration.

--user

Load from user configuration.

--local

Load from local configuration.

--default

Load from default configuration.

-x, --expand

Expand variable.

If the special _env or _eval variant of the option is present in the configuration, it will expand the environment variable or shell command, respectively.

-r, --raw

Disable formatting on single value output.

SECTION[...].VAR

Path to variable.

VALUE

Value to be set.

--system

Apply to system configuration.

--user

Apply to user configuration. (default)

--local

Apply to local configuration.

--system

Edit system configuration.

--user

Edit user configuration (default).

--local

Edit local configuration.

SECTION[...].VAR

Path to variable.

--site

Output originating site name only.

Shell commands often operate on filepaths. In such cases, it may be useful to manipulate these paths. Instead of using a shell interpolation (see below), use one of the available shorthand notations listed here.

{.}

Expand to immediate parent directory of given file. E.g., /some/path/to/file.h5 translates to /some/path/to.

{..}

Expand to second parent directory of given file. E.g., /some/path/to/file.h5 translates to /some/path.

{/}

The basename of the given file. E.g., /some/path/to/file.h5 translates to file.h5.

{/-}

The basename of the given file without its file type extension. E.g., /some/path/to/file.h5 translates to file.

{-}

The full path of the given file without the extension. This is useful for targeting adjacent files with a different extension. E.g., /some/path/to/file.h5 translates to /some/path/to/file.

{+}

The file type extension for the given file. E.g., /some/path/to/file.h5 translates to .h5.

{++}

The file type extension for the given file without the leading dot. E.g., /some/path/to/file.h5 translates to h5.

Command-line inputs are understood as individual arguments delimited by whitespace. Slice into the argument vector using the {[]} notation. Arguments follow zero-based indexing. Negative index values are counting backwards from the end.

Select with a singular value.

{[0]}

The first argument.

{[1]}

The second argument.

{[-1]}

The last argument.

Select a range with a start and stop value, non-inclusive of the stop value. Including a leading or trailing colon implies the default value (inclusive).

{[1:3]}

The second and third argument.

{[:4]}

The first four arguments.

{[-2:]}

The last two arguments.

Include a third value as a step (or sometimes referred to as stride). Leaving out the start and step value implies starting from the first element (inclusive).

{[::2]}

Every second argument starting from the first.

{[1::2]}

Every odd argument.

General purpose shell commands can be expanded with the {% %} notation. The incoming command-line args can be substituted with an @.

{% basename @ %}

Equivalent to {/}.

{% mktemp -d %}

Create temporary directory and insert its path.

Arbitrary Python expressions can be expanded with the {= =} notation. The input argument can be used within the expression with the variable x.

Exposed standard library modules include os, os.path as path, math and datetime as dt.

Incoming arguments are intelligently coerced into the expected type. E.g., 2 will be an integer, 4.67 a float, null and none will be a Python None, and true/false will be the appropriate boolean value.

{= x * math.pi =}

Multiply the incoming argument (expected to be a float) by Pi.

{= dt.datetime.fromtimestamp(x) =}

Convert incoming POSIX timestamp to ISO format.

[logging]

Logging configuration. See also logging section.

.level

One of TRACE, DEBUG, INFO, WARNING, ERROR, or CRITICAL (default: WARNING)

INFO level messages are reserved for clients when tasks begin running. There are numerous WARNING events (e.g., non-zero exit status of a task). DEBUG level messages signal component thread start/stop and individual task level behavior. TRACE contains detailed information on all other behavior, particular iterative messages while components are waiting for something.

ERROR messages track when things fail but the application can continue; e.g., when command template expansion fails on an individual task.

CRITICAL messages are emitted when the application will halt or crash. Some of these are expected (such as incorrect command-line arguments) but in the event of an uncaught exception within the application a full traceback is written to a file and logged.

.datefmt

Date/time format, standard codes apply (default: '%Y-%m-%d %H:%M:%S'`)

.format

Log message format.

Default set by the "default" logging.style. See the available attributes defined by the underlying Python logging interface.

Additional attributes provided beyond the standard include app_id, hostname, hostname_short, relative_name, time formats in elapsed, elapsed_ms, elapsed_delta, and elapsed_hms, as well as all ANSI colors and formats as ansi_x where x is one of reset, bold, faint, italic, underline, black, red, green, yellow, blue, magenta, cyan, white, and ansi_level contains the standard color for the current message severity level.

.style

Presets for logging.format which can be difficult to define correctly. Options are default, detailed, detailed-compact, and system.

[database]

Database configuration and connection details. See also database section.

.provider

Database provider (default: 'sqlite'). Supported alternatives include 'postgres' (or compatible). Support for other providers may be considered in the future.

.file

Only applicable for SQLite provider. SQLite does not understand any other connection detail.

.database

Name for database. Not applicable for SQLite.

.schema

Not applicable for all RDMS providers. For Postgres the default schema is public. Specifying the schema may be useful for having multiple instances within the same database.

.host

Hostname or address of database server (default: localhost).

.port

Port number to connect with database server. The default value depends on the provider, e.g., 5432 for Postgres.

.user

Username for databaser server account. If provided a password must also be provided. Default is the local account.

.password

Password for database server account. If provided a user must also be provided. Default is the local account.

See also note on _env and _eval.

.echo

Special parameter enables verbose logging of all database transactions.

[connection_args]

Specify additional connection details for the underlying SQL dialect provider, e.g., sqlite3 or psycopg2.

[pragmas]

Specify one or more pragmas to apply to database connections (SQLite only). For example pragmas = { journal_mode = "wal" } to enable WAL-mode.

*

Any additional arguments are forwarded to the provider, e.g., encoding = 'utf-8'.

[server]

Section for server workflow parameters.

.bind

Bind address (default: localhost).

When running locally, the default is recommended. To allow remote clients to connect over the network, bind the server to 0.0.0.0.

.port

Port number (default: 50001).

This is an arbitrary choice and simply must be an available port. The default option chosen here is typically available on most platforms and is not expected by any known major software.

.auth

Cryptographic authentication key to connect with server (default: <not secure>).

The default KEY used by the server and client is not secure and only a place holder. It is expected that the user choose a secure KEY. The cluster automatically generates a secure one-time KEY.

.queuesize

Maximum number of task bundles on the shared queue (default: 1).

This blocks the next bundle from being published by the scheduler until a client has taken the current prepared bundle. On smaller scales this is probably best and is only of modest performance impact, limiting the scheduler from getting so far ahead of the currently running tasks.

On large scale workflows with many clients (e.g., 100) it may be advantageous to allow the scheduler to work ahead in selecting new tasks.

.bundlesize

Size of task bundle (default: 1).

The default value allows for greater concurrency and responsiveness on small scales. This is used by the submit thread to accumulate bundles for either database commits and/or publishing to the queue. If a database is in use, the scheduler thread selects tasks from the database in batches of this size.

Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

See also -b/--bundlesize command-line option.

.attempts

Attempts for auto-retry on failed tasks (default: 1).

If a database is in use, then there is an opportunity to automatically retry failed tasks. A task is considered to have failed if it has a non-zero exit status. The original is not over-written, a new task is submitted and later scheduled.

Counterpart to the -r/--max-retries command-line option. Setting --max-retries 1 is equivalent to setting .attempts to 2.

See also .eager.

.eager

Schedule failed tasks before new tasks (default: false).

If .attempts is greater than one, this option defines the appetite for re-submitting failed tasks. By default, failed tasks will only be scheduled when there are no more remaining novel tasks.

.wait

Polling interval in seconds for database queries during scheduling (default: 5). This waiting only occurs when no tasks are returned by the query.

.evict

Eviction period in seconds for clients (default: 600).

If a client fails to register a heartbeat after this period of time it is considered defunct and is evicted. When there are no more tasks to schedule the server sends a disconnect request to all registered clients, and waits until a confirmation is returned for each. If a client is defunct, this will hang the shutdown process.

[client]

Section for client workflow parameters.

.bundlesize

Size of task bundle (default: 1).

The default value allows for greater concurrency and responsiveness on small scales.

Using larger bundles is a good idea for larger distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client. It is also a good idea to coordinate bundle size between the client and server so that the client returns the same sized bundles that it receives.

See also -b/--bundlesize command-line option.

.bundlewait

Seconds to wait before flushing task bundle (default: 5).

If this period of time expires since the previous bundle was returned to the server, the current group of finished tasks will be pushed regardless of bundlesize.

For larger distributed workflows it is a good idea to make this waiting period sufficiently long so that most bundles are returned whole.

See also -w/--bundlewait command-line option.

.heartrate

Interval in seconds between heartbeats sent to server (default 10).

Even on the largest scales the default interval should be fine.

.timeout

Timeout in seconds for client. Automatically shutdown if no tasks received (default: never).

This feature allows for gracefully scaling down a cluster when task throughput subsides.

[submit]

Section for submit workflow parameters.

.bundlesize

Size of task bundle (default: 1).

The default value allows for greater concurrency and responsiveness on small scales. Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

See also -b/--bundlesize command-line option.

.bundlewait

Seconds to wait before flushing tasks (default: 5).

If this period of time expires since the previous bundle was pushed to the database, the current bundle will be pushed regardless of how many tasks have been accumulated.

See also -w/--bundlewait command-line option.

[task]

Section for task runtime settings.

.cwd

Explicitly set the working directory for all tasks.

.timeout

Task-level walltime limit (default: none).

Executors will send a progression of SIGINT, SIGTERM, and SIGKILL. If the process still persists the executor itself will shutdown.

.signalwait

Wait period in seconds between signal escalation on task cancellation (default: 10).

See also -S, --signalwait command-line option.

[ssh]

SSH configuration section.

.args

SSH connection arguments; e.g., -i ~/.ssh/some.key. It is preferable to configure SSH directly however, in ~/.ssh/config.

[nodelist]

This can be a single list of hostnames or a section when multiple named lists. Reference named groups from the command-line with --ssh-group.

Such as,

.mycluster = ['mycluster-01', 'mycluster-02', 'mycluster-03']

[autoscale]

Define an autoscaling policy and parameters.

.policy

Either fixed or dynamic.

A fixed policy will seek to maintain a definite size and allows for recovery in the event that clients halt for some reason (e.g., due to expected faults or timeouts).

A dynamic policy maintains a minimum size and grows up to some maximum size depending on the observed task pressure given the specified scaling factor.

See also .factor, .period, .size.init, .size.min, and .size.max.

.factor

Scaling factor (default: 1).

A dimensionless quantity used by the dynamic policy. This value expresses some multiple of the average task duration in seconds.

The autoscaler periodically checks toc / (factor x avg_duration), where toc is the estimated time of completion for all remaining tasks given current throughput of active clients. This ratio is referred to as task pressure, and if it exceeds 1, the pressure is considered high and we will add another client if we are not already at the maximum size of the cluster.

For example, if the average task length is 30 minutes, and we set factor = 2, then if the estimated time of completion of remaining tasks given currently connected executors exceeds 1 hour, we will scale up by one unit.

See also .period.

.period

Scaling period in seconds (default: 60).

The autoscaler waits for this period of time in between checks and scaling events. A shorter period makes the scaling behavior more responsive but can effect database performance if checks happen too rapidly.

[size]

.init

Initial size of cluster (default: 1).

When the the cluster starts, this number of clients will be launched. For a fixed policy cluster, this should be given with a .min size, and likely the same value.

.min

Minimum size of cluster (default: 0).

Regardless of autoscaling policy, if the number of launched clients drops below this value we will scale up by one. Allowing min = 0 is an important feature for efficient use of computing resources in the absence of tasks.

.max

Maximum size of cluster (default: 2).

For a dynamic autoscaling policy, this sets an upper limit on the number of launched clients. When this number is reached, scaling stops regardless of task pressure.

[console]

Rich text display and output parameters.

.theme

Color scheme to use by default in output (such as with task info and task search).

This option is passed to the rich library.

[export]

Any variable defined here is injected as an environment variable for tasks.

Example,

foo = 1

The environment variable FOO=1 would be defined for all tasks.