condor_annex man page

condor_annexAdd cloud resources to the pool.



condor_annex[-aws-region <region>] -setup [/full/path/to/access/key/file [/full/path/to/secret/key/file]]

condor_annex[-aws-on-demand] -annex-name <name of the annex>-count <integer number of instances>[-aws-on-demand-*] [common options]

condor_annex[-aws-spot-fleet] -annex-name <name of the annex>-slots <integer weight>[-aws-spot-fleet-*] [common options]

condor_annex-annex-name <name of the annex>-duration hours

condor_annex[-annex-name <name of the annex>] -status[-classad]


condor_annex<condor_annex options>status<condor_status options>


condor_annexadds clouds resources to the pool. (&ldquo;The pool&rdquo; is determined in the usual manner for HTCondor daemons and tools.) This version supports only Amazon Web Services (`AWS'). To add &ldquo;on-demand&rdquo; instances, use the third form listed above; to add &ldquo;spot&rdquo; instances, use the fourth. For an explanation of terms, consult either the HTCondor manual (chapter ) or the AWS documentation.

Using condor_annexwith AWS requires a one-time setup procedure performed by invoking condor_annexwith the -setupflag (the second form listed above). You may check if this procedure has been performed with the -check-setupflag (the seventh form listed above).

To reset the lease on an existing annex, invoke condor_annexwith only the -annex-nameoption and -durationflag (the fifth form listed above).

To determine which of the instances previously requested for a particular annex are not currently in the pool, invoke condor_annexwith the -statusflag and the -annex-nameoption (the sixth form listed above). The output of this command is intended to be human-readable; specifying the -classadflag will produce the same information in ClassAd format. If you omit -annex-name, information for all annexes will be returned.

Starting in 8.7.3, you may instead invoke condor_annexwith statusas a command argument (the eighth form listed above). This will cause condor_annexto use condor_statusto present annex instance data. Arguments and options on the command line after statuswill be passed unmodified to condor_status, but not all arguments and options will behave as expected. (See below.) condor_annexwill construct an ad for each annex instance and pass that information to condor_status; condor_statuswill (unless you specify otherwise using its command line) query the collector for more information about the instances. Information from the collector will be presented as usual; instances which did not have ads in the collector will be presented last, in their own table. These instances can not be presented in the usual way because the annex instance ads generated by condor_annexdo not (and can not) have the same information in them as ads generated by a condor_startdrunning in the instance. See the condor_statusdocumentation (section ) for details about the &ldquo;merge&rdquo; mode of condor_statusused by this command argument. Note that both condor_annexand condor_statushave -annex-nameoptions; if you're interested in a particular annex, put this flag on the command line beforethe statuscommand argument to avoid confusing results.

Common options are listed first, followed by options specific to AWS, followed by options specific to AWS' on-demand instances, followed by options specific to AWS' spot instances, followed by options intended for use by experts.



Print a usage reminder.

-setup [/full/path/to/access/key/file /full/path/to/secret/key/file]

Do the first-time setup.

-duration hours

Set the maximum lease duration in decimal hours. After this amount of time, all instances will terminated, regardless of their idleness. Defaults to 50 minutes.

-idle hours

Set the maximum idle duration in decimal hours. An instance idle for longer than this duration will terminate itself. Defaults to 15 minutes.

-config-dir /full/path/to/directory

Copy the contents of /full/path/to/directoryto each instance's configuration directory.

-owner owner[, owner]*

Configure the annex so that only ownermay start jobs there. By default, configure the annex so that only the user running condor_annexmay start jobs there.


Configure the annex so that anyone in the pool may use the annex.

-aws-region region

Specify the region in which to create the annex.

-aws-user-data user-data

Set the instance user data to user-data.

-aws-user-data-file /full/path/to/file

Set the instance user data to the contents of the file /full/path/to/file.

-aws-default-user-data user-data

Set the instance user data to user-data, if it's not already set. Only applies to spot fleet requests.

-aws-default-user-data-file /full/path/to/file

Set the instance user data to the contents of the file /full/path/to/file, if it's not already set. Only applies to spot fleet requests.

-aws-on-demand-instance-type instance-type

This annex will requests instances of type instance-type. The default for v8.7.1 is `m4.large'.

-aws-on-demand-ami-id ami-id

This annex will start instances of the AMI ami-id. The default for v8.7.1 is `ami-35b13223', a GPU-compatible Amazon Linux image with HTCondor pre-installed.

-aws-on-demand-security-group-ids group-id[,group-id]

This annex will start instances with the listed security group IDs. The default is the security group created by -setup.

-aws-on-demand-key-name key-name

This annex will start instances with the key pair named key-name. The default is the key pair created by -setup.

-aws-spot-fleet-config-file /full/path/to/file

Use the JSON blob in /full/path/to/filefor the spot fleet request.

-aws-access-key-file /full/path/to/access-key-file

Experts only.

-aws-secret-key-file /full/path/to/secret-key-file

Experts only.

-aws-ec2-url https://ec2.<region>

Experts only.

-aws-events-url https://events.<region>

Experts only.

-aws-lambda-url https://lambda.<region>

Experts only.

-aws-s3-url https://s3.<region>

Experts only.

-aws-spot-fleet-lease-function-arn sfr-lease-function-arn

Developers only.

-aws-on-demand-lease-function-arn odi-lease-function-arn

Developers only.

-aws-on-demand-instance-profile-arn instance-profile-arn

Developers only.

General Remarks

Currently, only AWS is supported. The AMI configured by setup runs HTCondor v8.6.10 on Amazon Linux 2016.09, and the default instance type is &ldquo;m4.large&rdquo;. The default AMI has the appropriate drivers for AWS' GPU instance types.


To start an on-demand annex named `MyFirstAnnex' with one core, using the default AMI and instance type, run

 condor_annex -count 1 -annex-name MyFirstAnnex

You will be asked to confirm that the defaults are what you want.

As of 2017-04-17, the following example will cost a minimum of $90.

To start an on-demand annex with 100 GPUs that job owners `big' and `little' may use (be sure to include yourself!), run

 condor_annex -count 100 -annex-name MySecondAnnex \
   -aws-on-demand-instance-type p2.xlarge -owner "big, little"

Exit Status

condor_annexwill exit with a status value of 0 (zero) on success.


Center for High Throughput Computing, University of Wisconsin&ndash;Madison