python-swiftclient man page

python-swiftclient — python-swiftclient 3.4.0

Introduction

Introduction

Where to Start?

The python-swiftclient project comprises a command line tool and two separate APIs for accessing swift programmatically. Choosing the most appropriate method for a given use case is the first problem a user needs to solve.

Use Cases

Alongside the command line tool, the python-swiftclient includes two levels of API:

  • A low level client API that provides simple Python wrappers around the various authentication mechanisms and the individual HTTP requests.
  • A high level service API that provides methods for performing common operations in parallel on a thread pool.

Example use cases:

  • Uploading and retrieving data

    Use the command line tool if you are simply uploading and downloading files and directories to and from your filesystem. The command line tool can be integrated into a shell script to automate tasks.

  • Integrating into an automated Python workflow

    Use the SwiftService API to perform operations offered by the CLI if your use case requires integration with a Python-based workflow. This method offers greater control and flexibility over individual object operations, such as the metadata set on each object. The SwiftService class provides methods to perform multiple sets of operations against a swift object store using a configurable shared thread pool. A single instance of the SwiftService class can be shared between multiple threads in your own code.

  • Developing an application in Python to access a swift object store

    Use the SwiftService API to develop Python applications that use swift to store and retrieve objects. A SwiftService instance provides a configurable thread pool for performing all operations supported by the CLI.

  • Fine-grained control over threading or the requests being performed

    Use the Connection API if your use case requires fine grained control over advanced features or you wish to use your own existing threading model. Examples of advanced features requiring the use of the Connection API include creating an SLO manifest that references already existing objects, or fine grained control over the query strings supplied with each HTTP request.

Important considerations

This section covers some important considerations, helpful hints, and things to avoid when integrating an object store into your workflow.

An object store is not a filesystem

It cannot be stressed enough that your usage of the object store should reflect the proper use case, and not treat the storage like a traditional filesystem. There are two main restrictions to bear in mind when designing an application that uses an object store:

  • You cannot rename objects. Due to fact that the name of an object is one of the factors that determines where the object and its replicas are stored, renaming would require multiple copies of the data to be moved between physical storage devices. If you want to rename an object you must upload to the new location, or make a server side copy request to the new location, and then delete the original.
  • You cannot modify objects. Objects are stored in multiple locations and are checked for integrity based on the MD5 sum calculated during upload. In order to modify the contents of an object, the entire desired contents must be re-uploaded. In certain special cases it is possible to work around this restriction using large objects, but no general file-like access is available to modify a stored object.

Objects cannot be locked

There is no mechanism to perform a combination of reading the data/metadata from an object and writing an update to that data/metadata in an atomic way. Any user with access to a container could update the contents or metadata associated with an object at any time.

Workflows that assume that no updates have been made since the last read of an object should be discouraged. Enabling a workflow of this type requires an external object locking mechanism and/or cooperation between all clients accessing the data.

Developer Documentation

CLI

The swift tool is a command line utility for communicating with an OpenStack Object Storage (swift) environment. It allows one to perform several types of operations.

For help on a specific swift command, enter:

$ swift COMMAND --help

swift usage

Usage: swift [--version] [--help] [--os-help] [--snet] [--verbose]
             [--debug] [--info] [--quiet] [--auth <auth_url>]
             [--auth-version <auth_version> |
                 --os-identity-api-version <auth_version> ]
             [--user <username>]
             [--key <api_key>] [--retries <num_retries>]
             [--os-username <auth-user-name>] [--os-password <auth-password>]
             [--os-user-id <auth-user-id>]
             [--os-user-domain-id <auth-user-domain-id>]
             [--os-user-domain-name <auth-user-domain-name>]
             [--os-tenant-id <auth-tenant-id>]
             [--os-tenant-name <auth-tenant-name>]
             [--os-project-id <auth-project-id>]
             [--os-project-name <auth-project-name>]
             [--os-project-domain-id <auth-project-domain-id>]
             [--os-project-domain-name <auth-project-domain-name>]
             [--os-auth-url <auth-url>] [--os-auth-token <auth-token>]
             [--os-storage-url <storage-url>] [--os-region-name <region-name>]
             [--os-service-type <service-type>]
             [--os-endpoint-type <endpoint-type>]
             [--os-cacert <ca-certificate>] [--insecure]
             [--os-cert <client-certificate-file>]
             [--os-key <client-certificate-key-file>]
             [--no-ssl-compression]
             <subcommand> [--help] [<subcommand options>]

Subcommands:

delete

Delete a container or objects within a container.

download

Download objects from containers.

list

Lists the containers for the account or the objects for a container.

post

Updates meta information for the account, container, or object; creates containers if not present.

copy

Copies object, optionally adds meta

stat

Displays information for the account, container, or object.

upload

Uploads files or directories to the given container.

capabilities

List cluster capabilities.

tempurl

Create a temporary URL.

auth

Display auth related environment variables.

swift optional arguments

--version

show program’s version number and exit

-h, --help

show this help message and exit

--os-help

Show OpenStack authentication options.

-s, --snet

Use SERVICENET internal network.

-v, --verbose

Print more info.

--debug

Show the curl commands and results of all http queries regardless of result status.

--info

Show the curl commands and results of all http queries which return an error.

-q, --quiet

Suppress status output.

-A AUTH, --auth=AUTH

URL for obtaining an auth token.

-V AUTH_VERSION, --auth-version=AUTH_VERSION, --os-identity-api-version=AUTH_VERSION

Specify a version for authentication. Defaults to env[ST_AUTH_VERSION], env[OS_AUTH_VERSION], env[OS_IDENTITY_API_VERSION] or 1.0.

-U USER, --user=USER

User name for obtaining an auth token.

-K KEY, --key=KEY

Key for obtaining an auth token.

-R RETRIES, --retries=RETRIES

The number of times to retry a failed connection.

--insecure

Allow swiftclient to access servers without having to verify the SSL certificate. Defaults to env[SWIFTCLIENT_INSECURE] (set to ‘true’ to enable).

--no-ssl-compression

This option is deprecated and not used anymore. SSL compression should be disabled by default by the system SSL library.

Authentication

This section covers the options for authenticating with a swift object store. The combinations of options required for each authentication version are detailed below, but are just a subset of those that can be used to successfully authenticate. These are the most common and recommended combinations.

You should obtain the details of your authentication version and credentials from your storage provider. These details should make it clearer which of the authentication sections below are most likely to allow you to connect to your storage account.

Keystone v3

swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \
      --os-project-name project1 --os-project-domain-name domain1 \
      --os-username user --os-user-domain-name domain1 \
      --os-password password list

swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \
      --os-project-id 0123456789abcdef0123456789abcdef \
      --os-user-id abcdef0123456789abcdef0123456789 \
      --os-password password list

Manually specifying the options above on the command line can be avoided by setting the following combinations of environment variables:

ST_AUTH_VERSION=3
OS_USERNAME=user
OS_USER_DOMAIN_NAME=domain1
OS_PASSWORD=password
OS_PROJECT_NAME=project1
OS_PROJECT_DOMAIN_NAME=domain1
OS_AUTH_URL=https://api.example.com:5000/v3

ST_AUTH_VERSION=3
OS_USER_ID=abcdef0123456789abcdef0123456789
OS_PASSWORD=password
OS_PROJECT_ID=0123456789abcdef0123456789abcdef
OS_AUTH_URL=https://api.example.com:5000/v3

Keystone v2

swift --os-auth-url https://api.example.com:5000/v2.0 \
      --os-tenant-name tenant \
      --os-username user --os-password password list

Manually specifying the options above on the command line can be avoided by setting the following environment variables:

ST_AUTH_VERSION=2.0
OS_USERNAME=user
OS_PASSWORD=password
OS_TENANT_NAME=tenant
OS_AUTH_URL=https://api.example.com:5000/v2.0

Legacy auth systems

You can configure swift to work with any number of other authentication systems that we will not cover in this document. If your storage provider is not using Keystone to provide access tokens, please contact them for instructions on the required options. It is likely that the options will need to be specified as below:

swift -A https://api.example.com/v1.0 -U user -K api_key list

Specifying the options above manually on the command line can be avoided by setting the following environment variables:

ST_AUTH_VERSION=1.0
ST_AUTH=https://api.example.com/v1.0
ST_USER=user
ST_KEY=key

It is also possible that you need to use a completely separate auth system, in which case swiftclient cannot request a token for you. In this case you should make the authentication request separately and access your storage using the token and storage URL options shown below:

swift --os-auth-token 6ee5eb33efad4e45ab46806eac010566 \
      --os-storage-url https://10.1.5.2:8080/v1/AUTH_ced809b6a4baea7aeab61a \
      list
NOTE:

Leftover environment variables are a common source of confusion when authorization fails.

CLI commands

Auth

Usage: swift auth

Display authentication variables in shell friendly format. Command to run to export storage URL and auth token into OS_STORAGE_URL and OS_AUTH_TOKEN: swift auth. Command to append to a runcom file (e.g. ~/.bashrc, /etc/profile) for automatic authentication: swift auth -v -U test:tester -K testing.

swift stat

Usage: swift stat [--lh] [--header <header:value>]
                  [<container> [<object>]]

Displays information for the account, container, or object depending on the arguments given (if any). In verbose mode, the storage URL and the authentication token are displayed as well.

Positional arguments:

[container]

Name of container to stat from.

[object]

Name of object to stat.

Optional arguments:

--lh

Report sizes in human readable format similar to ls -lh.

-H, --header <header:value>

Adds a custom request header to use for stat.

swift list

Usage: swift list [--long] [--lh] [--totals] [--prefix <prefix>]
                  [--delimiter <delimiter>] [--header <header:value>]
                  [<container>]

Lists the containers for the account or the objects for a container. The -p <prefix> or --prefix <prefix> is an option that will only list items beginning with that prefix. The -d <delimiter> or --delimiter <delimiter> is an option (for container listings only) that will roll up items with the given delimiter (see OpenStack Swift general documentation <http://docs.openstack.org/swift/latest/> for what this means).

The -l and --lh options provide more detail, similar to ls -l and ls -lh, the latter providing sizes in human readable format (For example: 3K, 12M, etc). The latter two switches use more overhead to retrieve the displayed details, which is directly proportional to the number of container or objects listed.

Positional arguments:

[container]

Name of container to list object in.

Optional arguments:

-l, --long

Long listing format, similar to ls -l.

--lh

Report sizes in human readable format similar to ls -lh.

-t, --totals

Used with -l or –lh, only report totals.

-p <prefix>, --prefix <prefix>

Only list items beginning with the prefix.

-d <delim>, --delimiter <delim>

Roll up items with the given delimiter. For containers only. See OpenStack Swift API documentation for what this means.

-H, --header <header:value>

Adds a custom request header to use for listing.

swift upload

Usage: swift upload [--changed] [--skip-identical] [--segment-size <size>]
                    [--segment-container <container>] [--leave-segments]
                    [--object-threads <thread>] [--segment-threads <threads>]
                    [--header <header>] [--use-slo] [--ignore-checksum]
                    [--object-name <object-name>]
                    <container> <file_or_directory> [<file_or_directory>] [...]

Uploads the files and directories specified by the remaining arguments to the given container. The -c or --changed is an option that will only upload files that have changed since the last upload. The --object-name <object-name> is an option that will upload a file and name object to <object-name> or upload a directory and use <object-name> as object prefix. If the file name is “-“, client reads content from standard input. In this case --object-name is required to set the name of the object and no other files may be given.  The -S <size> or --segment-size <size> and --leave-segments are options as well (see --help for more).

Positional arguments:

<container>

Name of container to upload to.

<file_or_directory>

Name of file or directory to upload. Specify multiple times for multiple uploads.

Optional arguments:

-c, --changed

Only upload files that have changed since the last upload.

--skip-identical

Skip uploading files that are identical on both sides.

-S, --segment-size <size>

Upload files in segments no larger than <size> (in Bytes) and then create a “manifest” file that will download all the segments as if it were the original file.

--segment-container <container>

Upload the segments into the specified container. If not specified, the segments will be uploaded to a <container>_segments container to not pollute the main <container> listings.

--leave-segments

Indicates that you want the older segments of manifest objects left alone (in the case of overwrites).

--object-threads <threads>

Number of threads to use for uploading full objects. Default is 10.

--segment-threads <threads>

Number of threads to use for uploading object segments. Default is 10.

-H, --header <header:value>

Adds a customized request header. This option may be repeated. Example: -H “content-type:text/plain” -H “Content-Length: 4000”.

--use-slo

When used in conjunction with –segment-size it will create a Static Large Object instead of the default Dynamic Large Object.

--object-name <object-name>

Upload file and name object to <object-name> or upload dir and use <object-name> as object prefix instead of folder name.

--ignore-checksum

Turn off checksum validation for uploads.

swift post

Usage: swift post [--read-acl <acl>] [--write-acl <acl>] [--sync-to]
                  [--sync-key <sync-key>] [--meta <name:value>]
                  [--header <header>]
                  [<container> [<object>]]

Updates meta information for the account, container, or object depending on the arguments given. If the container is not found, the swiftclient will create it automatically, but this is not true for accounts and objects. Containers also allow the -r <read-acl> (or --read-acl <read-acl>) and -w <write-acl> (or --write-acl <write-acl>) options. The -m or --meta option is allowed on accounts, containers and objects, and is used to define the user metadata items to set in the form Name:Value. You can repeat this option. For example: post -m Color:Blue -m Size:Large

For more information about ACL formats see the documentation: ACLs.

Positional arguments:

[container]

Name of container to post to.

[object]

Name of object to post.

Optional arguments:

-r, --read-acl <acl>

Read ACL for containers. Quick summary of ACL syntax: .r:*, .r:-.example.com, .r:www.example.com, account1 (v1.0 identity API only), account1:*, account2:user2 (v2.0+ identity API).

-w, --write-acl <acl>

Write ACL for containers. Quick summary of ACL syntax: account1 (v1.0 identity API only), account1:*, account2:user2 (v2.0+ identity API).

-t, --sync-to <sync-to>

Sync To for containers, for multi-cluster replication.

-k, --sync-key <sync-key>

Sync Key for containers, for multi-cluster replication.

-m, --meta <name:value>

Sets a meta data item. This option may be repeated.

Example: -m Color:Blue -m Size:Large

-H, --header <header:value>

Adds a customized request header. This option may be repeated.

Example: -H “content-type:text/plain” -H “Content-Length: 4000”

swift download

Usage: swift download [--all] [--marker <marker>] [--prefix <prefix>]
                      [--output <out_file>] [--output-dir <out_directory>]
                      [--object-threads <threads>] [--ignore-checksum]
                      [--container-threads <threads>] [--no-download]
                      [--skip-identical] [--remove-prefix]
                      [--header <header:value>] [--no-shuffle]
                      [<container> [<object>] [...]]

Downloads everything in the account (with --all), or everything in a container, or a list of objects depending on the arguments given. For a single object download, you may use the -o <filename> or --output <filename> option to redirect the output to a specific file or - to redirect to stdout. The --ignore-checksum is an option that turn off checksum validation. You can specify optional headers with the repeatable cURL-like option -H [--header <name:value>]. --ignore-mtime ignores the x-object-meta-mtime metadata entry on the object (if present) and instead creates the downloaded files with fresh atime and mtime values.

Positional arguments:

<container>

Name of container to download from. To download a whole account, omit this and specify –all.

<object>

Name of object to download. Specify multiple times for multiple objects. Omit this to download all objects from the container.

Optional arguments:

-a, --all

Indicates that you really want to download everything in the account.

-m, --marker <marker>

Marker to use when starting a container or account download.

-p, --prefix <prefix>

Only download items beginning with <prefix>

-r, --remove-prefix

An optional flag for –prefix <prefix>, use this option to download items without <prefix>

-o, --output <out_file>

For a single file download, stream the output to <out_file>. Specifying “-” as <out_file> will redirect to stdout.

-D, --output-dir <out_directory>

An optional directory to which to store objects. By default, all objects are recreated in the current directory.

--object-threads <threads>

Number of threads to use for downloading objects. Default is 10.

--container-threads <threads>

Number of threads to use for downloading containers. Default is 10.

--no-download

Perform download(s), but don’t actually write anything to disk.

-H, --header <header:value>

Adds a customized request header to the query, like “Range” or “If-Match”. This option may be repeated.

Example: –header “content-type:text/plain”

--skip-identical

Skip downloading files that are identical on both sides.

--ignore-checksum

Turn off checksum validation for downloads.

--no-shuffle

By default, when downloading a complete account or container, download order is randomised in order to reduce the load on individual drives when multiple clients are executed simultaneously to download the same set of objects (e.g. a nightly automated download script to multiple servers). Enable this option to submit download jobs to the thread pool in the order they are listed in the object store.

swift delete

Usage: swift delete [--all] [--leave-segments]
                    [--object-threads <threads>]
                    [--container-threads <threads>]
                    [--header <header:value>]
                    [<container> [<object>] [...]]

Deletes everything in the account (with --all), or everything in a container, or a list of objects depending on the arguments given. Segments of manifest objects will be deleted as well, unless you specify the --leave-segments option.

Positional arguments:

[<container>]

Name of container to delete from.

[<object>]

Name of object to delete. Specify multiple times for multiple objects.

Optional arguments:

-a, --all

Delete all containers and objects.

--leave-segments

Do not delete segments of manifest objects.

-H, --header <header:value>

Adds a custom request header to use for deleting objects or an entire container.

--object-threads <threads>

Number of threads to use for deleting objects. Default is 10.

--container-threads <threads>

Number of threads to use for deleting containers. Default is 10.

swift copy

Usage: swift copy [--destination </container/object>] [--fresh-metadata]
                  [--meta <name:value>] [--header <header>] <container>
                  <object> [<object>] [...]

Copies an object to a new destination or adds user metadata to an object. Depending on the options supplied, you can preserve existing metadata in contrast to the post command. The --destination option sets the copy target destination in the form /container/object. If not set, the object will be copied onto itself which is useful for adding metadata. You can use the -M or --fresh-metadata option to copy an object without existing user meta data, and the -m or --meta option to define user meta data items to set in the form Name:Value. You can repeat this option. For example: copy -m Color:Blue -m Size:Large.

Positional arguments:

<container>

Name of container to copy from.

<object>

Name of object to copy. Specify multiple times for multiple objects

Optional arguments:

-d, --destination </container[/object]>

The container and name of the destination object. Name of destination object can be omitted, then will be same as name of source object. Supplying multiple objects and destination with object name is invalid.

-M, --fresh-metadata

Copy the object without any existing metadata, If not set, metadata will be preserved or appended

-m, --meta <name:value>

Sets a meta data item. This option may be repeated.

Example: -m Color:Blue -m Size:Large

-H, --header <header:value>

Adds a customized request header. This option may be repeated.

Example: -H “content-type:text/plain” -H “Content-Length: 4000”

swift capabilities

Usage: swift capabilities [--json] [<proxy_url>]

Displays cluster capabilities. The output includes the list of the activated Swift middlewares as well as relevant options for each ones. Additionally the command displays relevant options for the Swift core. If the proxy-url option is not provided, the storage URL retrieved after authentication is used as proxy-url.

Optional positional arguments:

<proxy_url>

Proxy URL of the cluster to retrieve capabilities.

--json

Print the cluster capabilities in JSON format.

swift tempurl

Usage: swift tempurl [--absolute] [--prefix-based]
                     <method> <seconds> <path> <key>

Generates a temporary URL for a Swift object. method option sets an HTTP method to allow for this temporary URL that is usually GET or PUT. time option sets the amount of time the temporary URL will be valid for. time can be specified as an integer, denoting the number of seconds from now on until the URL shall be valid; or, if --absolute is passed, the Unix timestamp when the temporary URL will expire. But beyond that, time can also be specified as an ISO 8601 timestamp in one of following formats:

i.

Complete date: YYYY-MM-DD (eg 1997-07-16)

ii.

Complete date plus hours, minutes and seconds: YYYY-MM-DDThh:mm:ss (eg 1997-07-16T19:20:30)

iii.

Complete date plus hours, minutes and seconds with UTC designator:

YYYY-MM-DDThh:mm:ssZ (eg 1997-07-16T19:20:30Z)

Please be aware that if you don’t provide the UTC designator (i.e., Z) the timestamp is generated using your local timezone. If only a date is specified, the time part used will equal to 00:00:00.

path option sets the full path to the Swift object. Example: /v1/AUTH_account/c/o. key option is the secret temporary URL key set on the Swift cluster. To set a key, run swift post -m "Temp-URL-Key: <your secret key>". To generate a prefix-based temporary URL use the --prefix-based option. This URL will contain the path to the prefix. Do not forget to append the desired objectname at the end of the path portion (and before the query portion) before sharing the URL. It is possible to use ISO 8601 UTC timestamps within the URL by using the --iso8601 option.

Positional arguments:

<method>

An HTTP method to allow for this temporary URL. Usually ‘GET’ or ‘PUT’.

<seconds>

The amount of time in seconds the temporary URL will be valid for; or, if –absolute is passed, the Unix timestamp when the temporary URL will expire.

<path>

The full path to the Swift object.

Example: /v1/AUTH_account/c/o or: http://saio:8080/v1/AUTH_account/c/o

<key>

The secret temporary URL key set on the Swift cluster. To set a key, run ‘swift post -m “Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4”’

Optional arguments:

--absolute

Interpret the <seconds> positional argument as a Unix timestamp rather than a number of seconds in the future.

--prefix-based

If present, a prefix-based tempURL will be generated.

Examples

In this section we present some example usage of the swift CLI. To keep the examples as short as possible, these examples assume that the relevant authentication options have been set using environment variables. You can obtain the full list of commands and options available in the swift CLI by executing the following:

> swift --help
> swift <command> --help

Simple examples

List the existing swift containers:

> swift list

container_1

Create a new container:

> swift post TestContainer

Upload an object into a container:

> swift upload TestContainer testSwift.txt

testSwift.txt

List the contents of a container:

> swift list TestContainer

testSwift.txt

Copy an object to new destination:

> swift copy -d /DestContainer/testSwift.txt SourceContainer testSwift.txt

SourceContainer/testSwift.txt copied to /DestContainer/testSwift.txt

Delete an object from a container:

> swift delete TestContainer testSwift.txt

testSwift.txt

Delete a container:

> swift delete TestContainer

TestContainer

Display auth related authentication variables in shell friendly format:

> swift auth

export OS_STORAGE_URL=http://127.0.0.1:8080/v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7
export OS_AUTH_TOKEN=c597015ae19943a18438b52ef3762e79

Download an object from a container:

> swift download TestContainer testSwift.txt

testSwift.txt [auth 0.028s, headers 0.045s, total 0.045s, 0.002 MB/s]
NOTE:

To upload an object to a container, your current working directory must be where the file is located or you must provide the complete path to the file. In other words, the –object-name <object-name> is an option that will upload file and name object to <object-name> or upload directory and use <object-name> as object prefix. In the case that you provide the complete path of the file, that complete path will be the name of the uploaded object.

For example:

> swift upload TestContainer /home/swift/testSwift/testSwift.txt

home/swift/testSwift/testSwift.txt

> swift list TestContainer

home/swift/testSwift/testSwift.txt

More complex examples

Swift has a single object size limit of 5GiB. In order to upload files larger than this, we must create a large object that consists of smaller segments. The example below shows how to upload a large video file as a static large object in 1GiB segments:

> swift upload videos --use-slo --segment-size 1G myvideo.mp4

myvideo.mp4 segment 8
myvideo.mp4 segment 4
myvideo.mp4 segment 2
myvideo.mp4 segment 7
myvideo.mp4 segment 0
myvideo.mp4 segment 1
myvideo.mp4 segment 3
myvideo.mp4 segment 6
myvideo.mp4 segment 5
myvideo.mp4

This command will upload segments to a container named videos_segments, and create a manifest file describing the entire object in the videos container. For more information on large objects, see the documentation here.

> swift list videos

myvideo.mp4

> swift list videos_segments

myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000000
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000001
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000002
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000003
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000004
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000005
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000006
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000007
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000008

Firstly, the key should be set, then generate a temporary URL for a Swift object:

> swift post -m "Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4"

> swift tempurl GET 6000 /v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7\
  /firstcontainer/clean.sh b3968d0207b54ece87cccc06515a89d4

/v1/AUTH_/firstcontainer/clean.sh?temp_url_sig=\
9218fc288cc09e5edd857b6a3d43cf2122b906dc&temp_url_expires=1472203614

The swiftclient.SwiftService API

A higher-level API aimed at allowing developers an easy way to perform multiple operations asynchronously using a configurable thread pool. Documentation for each service method call can be found here: swiftclient.service.

Authentication

This section covers the various options for authenticating with a swift object store. The combinations of options required for each authentication version are detailed below. Once again, these are just a subset of those that can be used to successfully authenticate, but they are the most common and recommended.

The relevant authentication options are presented as python dictionaries that should be added to any other options you are supplying to your SwiftService instance. As indicated in the python code, you can also set these options as environment variables that will be loaded automatically if the relevant option is not specified.

The SwiftService authentication attempts to automatically select the auth version based on the combination of options specified, but supplying options from multiple different auth versions can cause unexpected behaviour.

NOTE:

Leftover environment variables are a common source of confusion when authorization fails.

Keystone V3

{
    ...
    "auth_version": environ.get('ST_AUTH_VERSION'),  # Should be '3'
    "os_username": environ.get('OS_USERNAME'),
    "os_password": environ.get('OS_PASSWORD'),
    "os_project_name": environ.get('OS_PROJECT_NAME'),
    "os_project_domain_name": environ.get('OS_PROJECT_DOMAIN_NAME'),
    "os_auth_url": environ.get('OS_AUTH_URL'),
    ...
}
{
    ...
    "auth_version": environ.get('ST_AUTH_VERSION'),  # Should be '3'
    "os_username": environ.get('OS_USERNAME'),
    "os_password": environ.get('OS_PASSWORD'),
    "os_project_id": environ.get('OS_PROJECT_ID'),
    "os_project_domain_id": environ.get('OS_PROJECT_DOMAIN_ID'),
    "os_auth_url": environ.get('OS_AUTH_URL'),
    ...
}

Keystone V2

{
    ...
    "auth_version": environ.get('ST_AUTH_VERSION'),  # Should be '2.0'
    "os_username": environ.get('OS_USERNAME'),
    "os_password": environ.get('OS_PASSWORD'),
    "os_tenant_name": environ.get('OS_TENANT_NAME'),
    "os_auth_url": environ.get('OS_AUTH_URL'),
    ...
}

Legacy Auth

{
    ...
    "auth_version": environ.get('ST_AUTH_VERSION'),  # Should be '1.0'
    "auth": environ.get('ST_AUTH'),
    "user": environ.get('ST_USER'),
    "key": environ.get('ST_KEY'),
    ...
}

Configuration

When you create an instance of a SwiftService, you can override a collection of default options to suit your use case. Typically, the defaults are sensible to get us started, but depending on your needs you might want to tweak them to improve performance (options affecting large objects and thread counts can significantly alter performance in the right situation).

Service level defaults and some extra options can also be overridden on a per-operation (or even in some cases per-object) basis, and you will call out which options affect which operations later in the document.

The configuration of the service API is performed using an options dictionary passed to the SwiftService during initialisation. The options available in this dictionary are described below, along with their defaults:

Options

retries: 5

The number of times that the library should attempt to retry HTTP actions before giving up and reporting a failure.

container_threads: 10

object_dd_threads: 10

object_uu_threads: 10

segment_threads: 10

The above options determine the size of the available thread pools for performing swift operations. Container operations (such as listing a container) operate in the container threads, and a similar pattern applies to object and segment threads.

NOTE:

Object threads are separated into two separate thread pools: uu and dd. This stands for “upload/update” and “download/delete”, and the corresponding actions will be run on separate threads pools.

segment_size: None

If specified, this option enables uploading of large objects. Should the object being uploaded be larger than 5G in size, this option is mandatory otherwise the upload will fail. This option should be specified as a size in bytes.

use_slo: False

Used in combination with the above option, use_slo will upload large objects as static rather than dynamic. Only static large objects provide error checking for the downloaded object, so we recommend this option.

segment_container: None

Allows the user to select the container into which large object segments will be uploaded. We do not recommend changing this value as it could make locating orphaned segments more difficult in the case of errors.

leave_segments: False

Setting this option to true means that when deleting or overwriting a large object, its segments will be left in the object store and must be cleaned up manually. This option can be useful when sharing large object segments between multiple objects in more advanced scenarios, but must be treated with care, as it could lead to ever increasing storage usage.

changed: None

This option affects uploads and simply means that those objects which already exist in the object store will not be overwritten if the mtime and size of the source is the same as the existing object.

skip_identical: False

A slightly more thorough case of the above, but rather than mtime and size uses an object’s MD5 sum.

yes_all: False

This options affects only download and delete, and in each case must be specified in order to download/delete the entire contents of an account. This option has no effect on any other calls.

no_download: False

This option only affects download and means that all operations proceed as normal with the exception that no data is written to disk.

header: []

Used with upload and post operations to set headers on objects. Headers are specified as colon separated strings, e.g. “content-type:text/plain”.

meta: []

Used to set metadata on an object similarly to headers.

NOTE:

Setting metadata is a destructive operation, so when updating one of many metadata values all desired metadata for an object must be re-applied.

long: False

Affects only list operations, and results in more metrics being made available in the results at the expense of lower performance.

fail_fast: False

Applies to delete and upload operations, and attempts to abort queued tasks in the event of errors.

prefix: None

Affects list operations; only objects with the given prefix will be returned/affected. It is not advisable to set at the service level, as those operations that call list to discover objects on which they should operate will also be affected.

delimiter: None

Affects list operations, and means that listings only contain results up to the first instance of the delimiter in the object name. This is useful for working with objects containing ‘/’ in their names to simulate folder structures.

dir_marker: False

Affects uploads, and allows empty ‘pseudofolder’ objects to be created when the source of an upload is None.

checksum: True

Affects uploads and downloads. If set check md5 sum for the transfer.

shuffle: False

When downloading objects, the default behaviour of the CLI is to shuffle lists of objects in order to spread the load on storage drives when multiple clients are downloading the same files to multiple locations (e.g. in the event of distributing an update). When using the SwiftService directly, object downloads are scheduled in the same order as they appear in the container listing. When combined with a single download thread this means that objects are downloaded in lexically-sorted order. Setting this option to True gives the same shuffling behaviour as the CLI.

destination: None

When copying objects, this specifies the destination where the object will be copied to.  The default of None means copy will be the same as source.

fresh_metadata: None

When copying objects, this specifies that the object metadata on the source will not be applied to the destination object - the destination object will have a new fresh set of metadata that includes only the metadata specified in the meta option if any at all.

Other available options can be found in swiftclient/service.py in the source code for python-swiftclient. Each SwiftService method also allows for an optional dictionary to override those specified at init time, and the appropriate docstrings show which options modify each method’s behaviour.

Available Operations

Each operation provided by the service API may raise a SwiftError or ClientException for any call that fails completely (or a call which performs only one operation at an account or container level). In the case of a successful call an operation returns one of the following:

  • A dictionary detailing the results of a single operation.
  • An iterator that produces result dictionaries (for calls that perform multiple sub-operations).

A result dictionary can indicate either the success or failure of an individual operation (detailed in the success key), and will either contain the successful result, or an error key detailing the error encountered (usually an instance of Exception).

An example result dictionary is given below:

result = {
    'action': 'download_object',
    'success': True,
    'container': container,
    'object': obj,
    'path': path,
    'start_time': start_time,
    'finish_time': finish_time,
    'headers_receipt': headers_receipt,
    'auth_end_time': conn.auth_end_time,
    'read_length': bytes_read,
    'attempts': conn.attempts
}

All the possible action values are detailed below:

[
    'stat_account',
    'stat_container',
    'stat_object',
    'post_account',
    'post_container',
    'post_object',
    'list_part',          # list yields zero or more 'list_part' results
    'download_object',
    'create_container',   # from upload
    'create_dir_marker',  # from upload
    'upload_object',
    'upload_segment',
    'delete_container',
    'delete_object',
    'delete_segment',     # from delete_object operations
    'capabilities',
]

Stat

Stat can be called against an account, a container, or a list of objects to get account stats, container stats or information about the given objects. In the first two cases a dictionary is returned containing the results of the operation, and in the case of a list of object names being supplied, an iterator over the results generated for each object is returned.

Information returned includes the amount of data used by the given object/container/account and any headers or metadata set (this includes user set data as well as content-type and modification times).

See swiftclient.service.SwiftService.stat for docs generated from the method docstring.

Valid calls for this method are as follows:

  • stat([options]): Returns stats for the configured account.
  • stat(<container>, [options]): Returns stats for the given container.
  • stat(<container>, <object_list>, [options]): Returns stats for each of the given objects in the given container (through the returned iterator).

Results from stat are dictionaries indicating the success or failure of each operation. In the case of a successful stat against an account or container, the method returns immediately with one of the following results:

{
    'action': 'stat_account',
    'success': True,
    'items': items,
    'headers': headers
}
{
    'action': 'stat_container',
    'container': <container>,
    'success': True,
    'items': items,
    'headers': headers
}

In the case of stat called against a list of objects, the method returns a generator that returns the results of individual object stat operations as they are performed on the thread pool:

{
    'action': 'stat_object',
    'object': <object_name>,
    'container': <container>,
    'success': True,
    'items': items,
    'headers': headers
}

In the case of a failure the dictionary returned will indicate that the operation was not successful, and will include the keys below:

{
    'action': <'stat_object'|'stat_container'|'stat_account'>,
    'object': <'object_name'>,      # Only for stat with objects list
    'container': <container>,       # Only for stat with objects list or container
    'success': False,
    'error': <error>,
    'traceback': <trace>,
    'error_timestamp': <timestamp>
}

Example

The code below demonstrates the use of stat to retrieve the headers for a given list of objects in a container using 20 threads. The code creates a mapping from object name to headers which is then pretty printed to the log.

import logging
import pprint

from swiftclient.service import SwiftService
from sys import argv

logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

_opts = {'object_dd_threads': 20}
with SwiftService(options=_opts) as swift:
    container = argv[1]
    objects = argv[2:]
    header_data = {}
    stats_it = swift.stat(container=container, objects=objects)
    for stat_res in stats_it:
        if stat_res['success']:
            header_data[stat_res['object']] = stat_res['headers']
        else:
            logger.error(
                'Failed to retrieve stats for %s' % stat_res['object']
            )
    pprint.pprint(header_data)

List

List can be called against an account or a container to retrieve the containers or objects contained within them. Each call returns an iterator that returns pages of results (by default, up to 10000 results in each page).

See swiftclient.service.SwiftService.list for docs generated from the method docstring.

If the given container or account does not exist, the list method will raise a SwiftError, but for all other success/failures a dictionary is returned. Each successfully listed page returns a dictionary as described below:

{
    'action': <'list_account_part'|'list_container_part'>,
    'container': <container>,      # Only for listing a container
    'prefix': <prefix>,            # The prefix of returned objects/containers
    'success': True,
    'listing': [Item],             # A list of results
                                   # (only in the event of success)
    'marker': <marker>             # The last item name in the list
                                   # (only in the event of success)
}

Where an item contains the following keys:

{
    'name': <name>,
    'bytes': 10485760,
    'last_modified': '2014-12-11T12:02:38.774540',
    'hash': 'fb938269cbeabe4c234e1127bbd3b74a',
    'content_type': 'application/octet-stream',
    'meta': <metadata>    # Full metadata listing from stat'ing each object
                          # this key only exists if 'long' is specified in options
}

Any failure listing an account or container that exists will return a failure dictionary as described below:

{
    'action': <'list_account_part'|'list_container_part'>,,
    'container': container,         # Only for listing a container
    'prefix': options['prefix'],
    'success': success,
    'marker': marker,
    'error': error,
    'traceback': <trace>,
    'error_timestamp': <timestamp>
}

Example

The code below demonstrates the use of list to list all items in a container that are over 10MiB in size:

import logging

from swiftclient.service import SwiftService, SwiftError
from sys import argv

logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

container = argv[1]
minimum_size = 10*1024**2
with SwiftService() as swift:
    try:
        list_parts_gen = swift.list(container=container)
        for page in list_parts_gen:
            if page["success"]:
                for item in page["listing"]:

                    i_size = int(item["bytes"])
                    if i_size > minimum_size:
                        i_name = item["name"]
                        i_etag = item["hash"]
                        print(
                            "%s [size: %s] [etag: %s]" %
                            (i_name, i_size, i_etag)
                        )
            else:
                raise page["error"]

    except SwiftError as e:
        logger.error(e.value)

Post

Post can be called against an account, container or list of objects in order to update the metadata attached to the given items. In the first two cases a single dictionary is returned containing the results of the operation, and in the case of a list of objects being supplied, an iterator over the results generated for each object post is returned.

Each element of the object list may be a plain string of the object name, or a SwiftPostObject that allows finer control over the options and metadata applied to each of the individual post operations. When a string is given for the object name, the options and metadata applied are a combination of those supplied to the call to post() and the defaults of the SwiftService object.

If the given container or account does not exist, the post method will raise a SwiftError. Successful metadata update results are dictionaries as described below:

{
    'action': <'post_account'|'post_container'|'post_object'>,
    'success': True,
    'container': <container>,
    'object': <object>,
    'headers': {},
    'response_dict': <HTTP response details>
}
NOTE:

Updating user metadata keys will not only add any specified keys, but will also remove user metadata that has previously been set. This means that each time user metadata is updated, the complete set of desired key-value pairs must be specified.

Example

The code below demonstrates the use of post to set an archive folder in a given container to expire after a 24 hour delay:

import logging

from swiftclient.service import SwiftService, SwiftError
from sys import argv

logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

container = argv[1]
with SwiftService() as swift:
    try:
        list_options = {"prefix": "archive_2016-01-01/"}
        list_parts_gen = swift.list(container=container)
        for page in list_parts_gen:
            if page["success"]:
                objects = [obj["name"] for obj in page["listing"]]
                post_options = {"header": "X-Delete-After:86400"}
                for post_res in swift.post(
                        container=container,
                        objects=objects,
                        options=post_options):
                    if post_res['success']:
                        print("Object '%s' POST success" % post_res['object'])
                    else:
                        print("Object '%s' POST failed" % post_res['object'])
            else:
                raise page["error"]
    except SwiftError as e:
        logger.error(e.value)

Download

Download can be called against an entire account, a single container, or a list of objects in a given container. Each element of the object list is a string detailing the full name of an object to download.

In order to download the full contents of an entire account, you must set the value of yes_all to True in the options dictionary supplied to either the SwiftService instance or the call to download.

If the given container or account does not exist, the download method will raise a SwiftError, otherwise an iterator over the results generated for each object download is returned.

See swiftclient.service.SwiftService.download for docs generated from the method docstring.

For each successfully downloaded object, the results returned by the iterator will be a dictionary as described below (results are not returned for completed container or object segment downloads):

{
    'action': 'download_object',
    'container': <container>,
    'object': <object name>,
    'success': True,
    'path': <local path to downloaded object>,
    'pseudodir': <if true, the download created an empty directory>,
    'start_time': <time download started>,
    'end_time': <time download completed>,
    'headers_receipt': <time the headers from the object were retrieved>,
    'auth_end_time': <time authentication completed>,
    'read_length': <bytes_read>,
    'attempts': <attempt count>,
    'response_dict': <HTTP response details>
}

Any failure uploading an object will return a failure dictionary as described below:

{
    'action': 'download_object',
    'container': <container>,
    'object': <object name>,
    'success': False,
    'path': <local path of the failed download>,
    'pseudodir': <if true, the failed download was an empty directory>,
    'attempts': <attempt count>,
    'error': <error>,
    'traceback': <trace>,
    'error_timestamp': <timestamp>,
    'response_dict': <HTTP response details>
}

Example

The code below demonstrates the use of download to download all PNG images from a dated archive folder in a given container:

import logging

from swiftclient.service import SwiftService, SwiftError
from sys import argv

logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

def is_png(obj):
    return (
        obj["name"].lower().endswith('.png') or
        obj["content_type"] == 'image/png'
    )

container = argv[1]
with SwiftService() as swift:
    try:
        list_options = {"prefix": "archive_2016-01-01/"}
        list_parts_gen = swift.list(container=container)
        for page in list_parts_gen:
            if page["success"]:
                objects = [
                    obj["name"] for obj in page["listing"] if is_png(obj)
                ]
                for down_res in swift.download(
                        container=container,
                        objects=objects):
                    if down_res['success']:
                        print("'%s' downloaded" % down_res['object'])
                    else:
                        print("'%s' download failed" % down_res['object'])
            else:
                raise page["error"]
    except SwiftError as e:
        logger.error(e.value)

Upload

Upload is always called against an account and container and with a list of objects to upload. Each element of the object list may be a plain string detailing the path of the object to upload, or a SwiftUploadObject that allows finer control over some aspects of the individual operations.

When a simple string is supplied to specify a file to upload, the name of the object uploaded is the full path of the specified file and the options used for the upload are those supplied to the call to upload.

Constructing a SwiftUploadObject allows the user to supply an object name for the uploaded file, and modify the options used by upload at the granularity of individual files.

If the given container or account does not exist, the upload method will raise a SwiftError, otherwise an iterator over the results generated for each object upload is returned.

See swiftclient.service.SwiftService.upload for docs generated from the method docstring.

For each successfully uploaded object (or object segment), the results returned by the iterator will be a dictionary as described below:

{
    'action': 'upload_object',
    'container': <container>,
    'object': <object name>,
    'success': True,
    'status': <'uploaded'|'skipped-identical'|'skipped-changed'>,
    'attempts': <attempt count>,
    'response_dict': <HTTP response details>
}

{
    'action': 'upload_segment',
    'for_container': <container>,
    'for_object': <object name>,
    'segment_index': <segment_index>,
    'segment_size': <segment_size>,
    'segment_location': <segment_path>
    'segment_etag': <etag>,
    'log_line': <object segment n>
    'success': True,
    'response_dict': <HTTP response details>,
    'attempts': <attempt count>
}

Any failure uploading an object will return a failure dictionary as described below:

{
    'action': 'upload_object',
    'container': <container>,
    'object': <object name>,
    'success': False,
    'attempts': <attempt count>,
    'error': <error>,
    'traceback': <trace>,
    'error_timestamp': <timestamp>,
    'response_dict': <HTTP response details>
}

{
    'action': 'upload_segment',
    'for_container': <container>,
    'for_object': <object name>,
    'segment_index': <segment_index>,
    'segment_size': <segment_size>,
    'segment_location': <segment_path>,
    'log_line': <object segment n>,
    'success': False,
    'error': <error>,
    'traceback': <trace>,
    'error_timestamp': <timestamp>,
    'response_dict': <HTTP response details>,
    'attempts': <attempt count>
}

Example

The code below demonstrates the use of upload to upload all files and folders in a given directory, and rename each object by replacing the root directory name with ‘my-<d>-objects’, where <d> is the name of the uploaded directory:

import logging

from os import walk
from os.path import join
from swiftclient.multithreading import OutputManager
from swiftclient.service import SwiftError, SwiftService, SwiftUploadObject
from sys import argv

logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

_opts = {'object_uu_threads': 20}
dir = argv[1]
container = argv[2]
with SwiftService(options=_opts) as swift, OutputManager() as out_manager:
    try:
        # Collect all the files and folders in the given directory
        objs = []
        dir_markers = []
        for (_dir, _ds, _fs) in walk(dir):
            if not (_ds + _fs):
                dir_markers.append(_dir)
            else:
                objs.extend([join(_dir, _f) for _f in _fs])

        # Now that we've collected all the required files and dir markers
        # build the ``SwiftUploadObject``s for the call to upload
        objs = [
            SwiftUploadObject(
                o, object_name=o.replace(
                    dir, 'my-%s-objects' % dir, 1
                )
            ) for o in objs
        ]
        dir_markers = [
            SwiftUploadObject(
                None, object_name=d.replace(
                    dir, 'my-%s-objects' % dir, 1
                ), options={'dir_marker': True}
            ) for d in dir_markers
        ]

        # Schedule uploads on the SwiftService thread pool and iterate
        # over the results
        for r in swift.upload(container, objs + dir_markers):
            if r['success']:
                if 'object' in r:
                    print(r['object'])
                elif 'for_object' in r:
                    print(
                        '%s segment %s' % (r['for_object'],
                                           r['segment_index'])
                        )
            else:
                error = r['error']
                if r['action'] == "create_container":
                    logger.warning(
                        'Warning: failed to create container '
                        "'%s'%s", container, error
                    )
                elif r['action'] == "upload_object":
                    logger.error(
                        "Failed to upload object %s to container %s: %s" %
                        (container, r['object'], error)
                    )
                else:
                    logger.error("%s" % error)

    except SwiftError as e:
        logger.error(e.value)

Delete

Delete can be called against an account or a container to remove the containers or objects contained within them. Each call to delete returns an iterator over results of each resulting sub-request.

If the number of requested delete operations is large and the target swift cluster is running the bulk middleware, the call to SwiftService.delete will make use of bulk operations and the returned result iterator will return bulk_delete results rather than individual delete_object, delete_container or delete_segment results.

See swiftclient.service.SwiftService.delete for docs generated from the method docstring.

For each successfully deleted container, object or segment, the results returned by the iterator will be a dictionary as described below:

{
    'action': <'delete_object'|'delete_segment'>,
    'container': <container>,
    'object': <object name>,
    'success': True,
    'attempts': <attempt count>,
    'response_dict': <HTTP response details>
}

{
    'action': 'delete_container',
    'container': <container>,
    'success': True,
    'response_dict': <HTTP response details>,
    'attempts': <attempt count>
}

{
    'action': 'bulk_delete',
    'container': <container>,
    'objects': <[objects]>,
    'success': True,
    'attempts': <attempt count>,
    'response_dict': <HTTP response details>
}

Any failure in a delete operation will return a failure dictionary as described below:

{
    'action': ('delete_object'|'delete_segment'),
    'container': <container>,
    'object': <object name>,
    'success': False,
    'attempts': <attempt count>,
    'error': <error>,
    'traceback': <trace>,
    'error_timestamp': <timestamp>,
    'response_dict': <HTTP response details>
}

{
    'action': 'delete_container',
    'container': <container>,
    'success': False,
    'error': <error>,
    'traceback': <trace>,
    'error_timestamp': <timestamp>,
    'response_dict': <HTTP response details>,
    'attempts': <attempt count>
}

{
    'action': 'bulk_delete',
    'container': <container>,
    'objects': <[objects]>,
    'success': False,
    'attempts': <attempt count>,
    'error': <error>,
    'traceback': <trace>,
    'error_timestamp': <timestamp>,
    'response_dict': <HTTP response details>
}

Example

The code below demonstrates the use of delete to remove a given list of objects from a specified container. As the objects are deleted the transaction ID of the relevant request is printed along with the object name and number of attempts required. By printing the transaction ID, the printed operations can be easily linked to events in the swift server logs:

import logging

from swiftclient.service import SwiftService
from sys import argv


logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

_opts = {'object_dd_threads': 20}
container = argv[1]
objects = argv[2:]
with SwiftService(options=_opts) as swift:
    del_iter = swift.delete(container=container, objects=objects)
    for del_res in del_iter:
        c = del_res.get('container', '')
        o = del_res.get('object', '')
        a = del_res.get('attempts')
        if del_res['success'] and not del_res['action'] == 'bulk_delete':
            rd = del_res.get('response_dict')
            if rd is not None:
                t = dict(rd.get('headers', {}))
                if t:
                    print(
                        'Successfully deleted {0}/{1} in {2} attempts '
                        '(transaction id: {3})'.format(c, o, a, t)
                    )
                else:
                    print(
                        'Successfully deleted {0}/{1} in {2} '
                        'attempts'.format(c, o, a)
                    )

Copy

Copy can be called to copy an object or update the metadata on the given items.

Each element of the object list may be a plain string of the object name, or a SwiftCopyObject that allows finer control over the options applied to each of the individual copy operations (destination, fresh_metadata, options).

Destination should be in format /container/object; if not set, the object will be copied onto itself. Fresh_metadata sets mode of operation on metadata. If not set, current object user metadata will be copied/preserved; if set, all current user metadata will be removed.

Returns an iterator over the results generated for each object copy (and may also include the results of creating destination containers).

When a string is given for the object name, destination and fresh metadata will default to None and None, which result in adding metadata to existing objects.

Successful copy results are dictionaries as described below:

{
    'action': 'copy_object',
    'success': True,
    'container': <container>,
    'object': <object>,
    'destination': <destination>,
    'headers': {},
    'fresh_metadata': <boolean>,
    'response_dict': <HTTP response details>
}

Any failure in a copy operation will return a failure dictionary as described below:

{
    'action': 'copy_object',
    'success': False,
    'container': <container>,
    'object': <object>,
    'destination': <destination>,
    'headers': {},
    'fresh_metadata': <boolean>,
    'response_dict': <HTTP response details>,
    'error': <error>,
    'traceback': <traceback>,
    'error_timestamp': <timestamp>
}

Example

The code below demonstrates the use of copy to add new user metadata for objects a and b, and to copy object c to d (with added metadata).

import logging

from swiftclient.service import SwiftService, SwiftCopyObject, SwiftError

logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

with SwiftService() as swift:
    try:
        obj = SwiftCopyObject("c", {"Destination": "/cont/d"})
        for i in swift.copy(
                "cont", ["a", "b", obj],
                {"meta": ["foo:bar"], "Destination": "/cc"}):
            if i["success"]:
                if i["action"] == "copy_object":
                    print(
                        "object %s copied from /%s/%s" %
                        (i["destination"], i["container"], i["object"])
                    )
                if i["action"] == "create_container":
                    print(
                        "container %s created" % i["container"]
                    )
            else:
                if "error" in i and isinstance(i["error"], Exception):
                    raise i["error"]
    except SwiftError as e:
        logger.error(e.value)

Capabilities

Capabilities can be called against an account or a particular proxy URL in order to determine the capabilities of the swift cluster. These capabilities include details about configuration options and the middlewares that are installed in the proxy pipeline.

See swiftclient.service.SwiftService.capabilities for docs generated from the method docstring.

For each successful call to list capabilities, a result dictionary will be returned with the contents described below:

{
    'action': 'capabilities',
    'timestamp': <time of the call>,
    'success': True,
    'capabilities': <dictionary containing capability details>
}

The contents of the capabilities dictionary contain the core swift capabilities under the key swift; all other keys show the configuration options for additional middlewares deployed in the proxy pipeline. An example capabilities dictionary is given below:

{
    'account_quotas': {},
    'bulk_delete': {
        'max_deletes_per_request': 10000,
        'max_failed_deletes': 1000
    },
    'bulk_upload': {
        'max_containers_per_extraction': 10000,
        'max_failed_extractions': 1000
    },
    'container_quotas': {},
    'container_sync': {'realms': {}},
    'formpost': {},
    'keystoneauth': {},
    'slo': {
        'max_manifest_segments': 1000,
        'max_manifest_size': 2097152,
        'min_segment_size': 1048576
    },
    'swift': {
        'account_autocreate': True,
        'account_listing_limit': 10000,
        'allow_account_management': True,
        'container_listing_limit': 10000,
        'extra_header_count': 0,
        'max_account_name_length': 256,
        'max_container_name_length': 256,
        'max_file_size': 5368709122,
        'max_header_size': 8192,
        'max_meta_count': 90,
        'max_meta_name_length': 128,
        'max_meta_overall_size': 4096,
        'max_meta_value_length': 256,
        'max_object_name_length': 1024,
        'policies': [
            {'default': True, 'name': 'Policy-0'}
        ],
        'strict_cors_mode': False,
        'version': '2.2.2'
    },
    'tempurl': {
        'methods': ['GET', 'HEAD', 'PUT']
    }
}

Example

The code below demonstrates the use of capabilities to determine if the Swift cluster supports static large objects, and if so, the maximum number of segments that can be described in a single manifest file, along with the size restrictions on those objects:

import logging

from swiftclient.exceptions import ClientException
from swiftclient.service import SwiftService

logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

with SwiftService() as swift:
    try:
        capabilities_result = swift.capabilities()
        capabilities = capabilities_result['capabilities']
        if 'slo' in capabilities:
            print('SLO is supported')
        else:
            print('SLO is not supported')
    except ClientException as e:
        logger.error(e.value)

The swiftclient.Connection API

A low level API that provides methods for authentication and methods that correspond to the individual REST API calls described in the swift documentation.

For usage details see the client docs: swiftclient.client.

Authentication

This section covers the various combinations of kwargs required when creating an instance of the Connection object for communicating with a swift object store. The combinations of options required for each authentication version are detailed below, but are just a subset of those that can be used to successfully authenticate. These are the most common and recommended combinations.

Keystone Session

from keystoneauth1 import session
from keystoneauth1.identity import v3

# Create a password auth plugin
auth = v3.Password(auth_url='http://127.0.0.1:5000/v3/',
                   username='tester',
                   password='testing',
                   user_domain_name='Default',
                   project_name='Default',
                   project_domain_name='Default')

# Create session
keystone_session = session.Session(auth=auth)

# Create swiftclient Connection
swift_conn = Connection(session=keystone_session)

Keystone v3

_authurl = 'http://127.0.0.1:5000/v3/'
_auth_version = '3'
_user = 'tester'
_key = 'testing'
_os_options = {
    'user_domain_name': 'Default',
    'project_domain_name': 'Default',
    'project_name': 'Default'
}

conn = Connection(
    authurl=_authurl,
    user=_user,
    key=_key,
    os_options=_os_options,
    auth_version=_auth_version
)

Keystone v2

_authurl = 'http://127.0.0.1:5000/v2.0/'
_auth_version = '2'
_user = 'tester'
_key = 'testing'
_tenant_name = 'test'

conn = Connection(
    authurl=_authurl,
    user=_user,
    key=_key,
    tenant_name=_tenant_name,
    auth_version=_auth_version
)

Legacy Auth

_authurl = 'http://127.0.0.1:8080/'
_auth_version = '1'
_user = 'tester'
_key = 'testing'
_tenant_name = 'test'

conn = Connection(
    authurl=_authurl,
    user=_user,
    key=_key,
    tenant_name=_tenant_name,
    auth_version=_auth_version
)

Examples

In this section we present some simple code examples that demonstrate the usage of the Connection API. You can find full details of the options and methods available to the Connection API in the docstring generated documentation: swiftclient.client.

List the available containers:

resp_headers, containers = conn.get_account()
print("Response headers: %s" % resp_headers)
for container in containers:
    print(container)

Create a new container:

container = 'new-container'
conn.put_container(container)
resp_headers, containers = conn.get_account()
if container in containers:
    print("The container was created")

Create a new object with the contents of a local text file:

container = 'new-container'
with open('local.txt', 'r') as local:
    conn.put_object(
        container,
        'local_object.txt',
        contents=local,
        content_type='text/plain'
    )

Confirm presence of the object:

obj = 'local_object.txt'
container = 'new-container'
try:
    resp_headers = conn.head_object(container, obj)
    print('The object was successfully created')
except ClientException as e:
    if e.http_status = '404':
        print('The object was not found')
    else:
        print('An error occurred checking for the existence of the object')

Download the created object:

obj = 'local_object.txt'
container = 'new-container'
resp_headers, obj_contents = conn.get_object(container, obj)
with open('local_copy.txt', 'w') as local:
    local.write(obj_contents)

Delete the created object:

obj = 'local_object.txt'
container = 'new-container'
try:
    conn.delete_object(container, obj)
    print("Successfully deleted the object")
except ClientException as e:
    print("Failed to delete the object with error: %s" % e)

Code-Generated Documentation

swiftclient

OpenStack Swift Python client binding.

swiftclient.authv1

swiftclient.client

OpenStack Swift client library used internally

class swiftclient.client.Connection(authurl=None, user=None, key=None, retries=5, preauthurl=None, preauthtoken=None, snet=False, starting_backoff=1, max_backoff=64, tenant_name=None, os_options=None, auth_version='1', cacert=None, insecure=False, cert=None, cert_key=None, ssl_compression=True, retry_on_ratelimit=False, timeout=None, session=None)

Bases: object

Convenience class to make requests that will also retry the request

Requests will have an X-Auth-Token header whose value is either the preauthtoken or a token obtained from the auth service using the user credentials provided as args to the constructor. If os_options includes a service_username then requests will also have an X-Service-Token header whose value is a token obtained from the auth service using the service credentials. In this case the request url will be set to the storage_url obtained from the auth service for the service user, unless this is overridden by a preauthurl.

Parameters
  • authurl – authentication URL
  • user – user name to authenticate as
  • key – key/password to authenticate with
  • retries – Number of times to retry the request before failing
  • preauthurl – storage URL (if you have already authenticated)
  • preauthtoken – authentication token (if you have already authenticated) note authurl/user/key/tenant_name are not required when specifying preauthtoken
  • snet – use SERVICENET internal network default is False
  • starting_backoff – initial delay between retries (seconds)
  • max_backoff – maximum delay between retries (seconds)
  • auth_version – OpenStack auth version, default is 1.0
  • tenant_name – The tenant/account name, required when connecting to an auth 2.0 system.
  • os_options – The OpenStack options which can have tenant_id, auth_token, service_type, endpoint_type, tenant_name, object_storage_url, region_name, service_username, service_project_name, service_key
  • insecure – Allow to access servers without checking SSL certs. The server’s certificate will not be verified.
  • cert – Client certificate file to connect on SSL server requiring SSL client certificate.
  • cert_key – Client certificate private key file.
  • ssl_compression – Whether to enable compression at the SSL layer. If set to ‘False’ and the pyOpenSSL library is present an attempt to disable SSL compression will be made. This may provide a performance increase for https upload/download operations.
  • retry_on_ratelimit – by default, a ratelimited connection will raise an exception to the caller. Setting this parameter to True will cause a retry after a backoff.
  • timeout – The connect timeout for the HTTP connection.
  • session – A keystoneauth session object.

close()

copy_object(container, obj, destination=None, headers=None, fresh_metadata=None, response_dict=None)

Wrapper for copy_object()

delete_container(container, response_dict=None, query_string=None, headers={})

Wrapper for delete_container()

delete_object(container, obj, query_string=None, response_dict=None, headers=None)

Wrapper for delete_object()

get_account(marker=None, limit=None, prefix=None, end_marker=None, full_listing=False, headers=None)

Wrapper for get_account()

get_auth()

get_capabilities(url=None)

get_container(container, marker=None, limit=None, prefix=None, delimiter=None, end_marker=None, path=None, full_listing=False, headers=None, query_string=None)

Wrapper for get_container()

get_object(container, obj, resp_chunk_size=None, query_string=None, response_dict=None, headers=None)

Wrapper for get_object()

get_service_auth()

head_account(headers=None)

Wrapper for head_account()

head_container(container, headers=None)

Wrapper for head_container()

head_object(container, obj, headers=None)

Wrapper for head_object()

http_connection(url=None)

post_account(headers, response_dict=None, query_string=None, data=None)

Wrapper for post_account()

post_container(container, headers, response_dict=None)

Wrapper for post_container()

post_object(container, obj, headers, response_dict=None)

Wrapper for post_object()

put_container(container, headers=None, response_dict=None, query_string=None)

Wrapper for put_container()

put_object(container, obj, contents, content_length=None, etag=None, chunk_size=None, content_type=None, headers=None, query_string=None, response_dict=None)

Wrapper for put_object()

class swiftclient.client.HTTPConnection(url, proxy=None, cacert=None, insecure=False, cert=None, cert_key=None, ssl_compression=False, default_user_agent=None, timeout=None)

Bases: object

Make an HTTPConnection or HTTPSConnection

Parameters
  • url – url to connect to
  • proxy – proxy to connect through, if any; None by default; str of the format ‘http://127.0.0.1:8888’ to set one
  • cacert – A CA bundle file to use in verifying a TLS server certificate.
  • insecure – Allow to access servers without checking SSL certs. The server’s certificate will not be verified.
  • cert – Client certificate file to connect on SSL server requiring SSL client certificate.
  • cert_key – Client certificate private key file.
  • ssl_compression – SSL compression should be disabled by default and this setting is not usable as of now. The parameter is kept for backward compatibility.
  • default_user_agent – Set the User-Agent header on every request. If set to None (default), the user agent will be “python-swiftclient-<version>”. This may be overridden on a per-request basis by explicitly setting the user-agent header on a call to request().
  • timeout – socket read timeout value, passed directly to the requests library.
Raises

ClientException – Unable to handle protocol scheme

getresponse()

Adapt requests response to httplib interface

putrequest(full_path, data=None, headers=None, files=None)

Use python-requests files upload

Parameters
  • data – Use data generator for chunked-transfer
  • files – Use files for default transfer
request(method, full_path, data=None, headers=None, files=None)

Encode url and header, then call requests.request

swiftclient.client.LOGGER_SENSITIVE_HEADERS = ['x-auth-token', 'x-auth-key', 'x-service-token', 'x-storage-token', 'x-account-meta-temp-url-key', 'x-account-meta-temp-url-key-2', 'x-container-meta-temp-url-key', 'x-container-meta-temp-url-key-2', 'set-cookie']

A list of sensitive headers to redact in logs. Note that when extending this list, the header names must be added in all lower case.

swiftclient.client.copy_object(url, token, container, name, destination=None, headers=None, fresh_metadata=None, http_conn=None, response_dict=None, service_token=None)

Copy object

Parameters
  • url – storage URL
  • token – auth token; if None, no token will be sent
  • container – container name that the source object is in
  • name – source object name
  • destination – The container and object name of the destination object in the form of /container/object; if None, the copy will use the source as the destination.
  • headers – additional headers to include in the request
  • fresh_metadata – Enables object creation that omits existing user metadata, default None
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
  • service_token – service auth token
Raises

ClientException – HTTP Copy request failed

swiftclient.client.delete_container(url, token, container, http_conn=None, response_dict=None, service_token=None, query_string=None, headers=None)

Delete a container

Parameters
  • url – storage URL
  • token – auth token
  • container – container name to delete
  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
  • service_token – service auth token
  • query_string – if set will be appended with ‘?’ to generated path
  • headers – additional headers to include in the request
Raises

ClientException – HTTP Delete request failed

swiftclient.client.delete_object(url, token=None, container=None, name=None, http_conn=None, headers=None, proxy=None, query_string=None, response_dict=None, service_token=None)

Delete object

Parameters
  • url – storage URL
  • token – auth token; if None, no token will be sent
  • container – container name that the object is in; if None, the container name is expected to be part of the url
  • name – object name to delete; if None, the object name is expected to be part of the url
  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)
  • headers – additional headers to include in the request
  • proxy – proxy to connect through, if any; None by default; str of the format ‘http://127.0.0.1:8888’ to set one
  • query_string – if set will be appended with ‘?’ to generated path
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
  • service_token – service auth token
Raises

ClientException – HTTP Delete request failed

swiftclient.client.encode_meta_headers(headers)

Only encode metadata headers keys

swiftclient.client.encode_utf8(value)

swiftclient.client.get_account(url, token, marker=None, limit=None, prefix=None, end_marker=None, http_conn=None, full_listing=False, service_token=None, headers=None)

Get a listing of containers for the account.

Parameters
  • url – storage URL
  • token – auth token
  • marker – marker query
  • limit – limit query
  • prefix – prefix query
  • end_marker – end_marker query
  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)
  • full_listing – if True, return a full listing, else returns a max of 10000 listings
  • service_token – service auth token
  • headers – additional headers to include in the request
Returns

a tuple of (response headers, a list of containers) The response headers will be a dict and all header names will be lowercase.

Raises

ClientException – HTTP GET request failed

swiftclient.client.get_auth(auth_url, user, key, **kwargs)

Get authentication/authorization credentials.

Parameters
  • auth_version – the api version of the supplied auth params
  • os_options – a dict, the openstack identity service options
Returns

a tuple, (storage_url, token)

N.B. if the optional os_options parameter includes a non-empty ‘object_storage_url’ key it will override the default storage url returned by the auth service.

The snet parameter is used for Rackspace’s ServiceNet internal network implementation. In this function, it simply adds snet- to the beginning of the host name for the returned storage URL. With Rackspace Cloud Files, use of this network path causes no bandwidth charges but requires the client to be running on Rackspace’s ServiceNet network.

swiftclient.client.get_auth_1_0(url, user, key, snet, **kwargs)

swiftclient.client.get_auth_keystone(auth_url, user, key, os_options, **kwargs)

Authenticate against a keystone server.

We are using the keystoneclient library for authentication.

swiftclient.client.get_capabilities(http_conn)

Get cluster capability infos.

Parameters

http_conn – a tuple of (parsed url, HTTPConnection object)

Returns

a dict containing the cluster capabilities

Raises

ClientException – HTTP Capabilities GET failed

swiftclient.client.get_container(url, token, container, marker=None, limit=None, prefix=None, delimiter=None, end_marker=None, path=None, http_conn=None, full_listing=False, service_token=None, headers=None, query_string=None)

Get a listing of objects for the container.

Parameters
  • url – storage URL
  • token – auth token
  • container – container name to get a listing for
  • marker – marker query
  • limit – limit query
  • prefix – prefix query
  • delimiter – string to delimit the queries on
  • end_marker – marker query
  • path – path query (equivalent: “delimiter=/” and “prefix=path/”)
  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)
  • full_listing – if True, return a full listing, else returns a max of 10000 listings
  • service_token – service auth token
  • headers – additional headers to include in the request
  • query_string – if set will be appended with ‘?’ to generated path
Returns

a tuple of (response headers, a list of objects) The response headers will be a dict and all header names will be lowercase.

Raises

ClientException – HTTP GET request failed

swiftclient.client.get_keystoneclient_2_0(auth_url, user, key, os_options, **kwargs)

swiftclient.client.get_object(url, token, container, name, http_conn=None, resp_chunk_size=None, query_string=None, response_dict=None, headers=None, service_token=None)

Get an object

Parameters
  • url – storage URL
  • token – auth token
  • container – container name that the object is in
  • name – object name to get
  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)
  • resp_chunk_size – if defined, chunk size of data to read. NOTE: If you specify a resp_chunk_size you must fully read the object’s contents before making another request.
  • query_string – if set will be appended with ‘?’ to generated path
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
  • headers – an optional dictionary with additional headers to include in the request
  • service_token – service auth token
Returns

a tuple of (response headers, the object’s contents) The response headers will be a dict and all header names will be lowercase.

Raises

ClientException – HTTP GET request failed

swiftclient.client.head_account(url, token, http_conn=None, headers=None, service_token=None)

Get account stats.

Parameters
  • url – storage URL
  • token – auth token
  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)
  • headers – additional headers to include in the request
  • service_token – service auth token
Returns

a dict containing the response’s headers (all header names will be lowercase)

Raises

ClientException – HTTP HEAD request failed

swiftclient.client.head_container(url, token, container, http_conn=None, headers=None, service_token=None)

Get container stats.

Parameters
  • url – storage URL
  • token – auth token
  • container – container name to get stats for
  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)
  • headers – additional headers to include in the request
  • service_token – service auth token
Returns

a dict containing the response’s headers (all header names will be lowercase)

Raises

ClientException – HTTP HEAD request failed

swiftclient.client.head_object(url, token, container, name, http_conn=None, service_token=None, headers=None)

Get object info

Parameters
  • url – storage URL
  • token – auth token
  • container – container name that the object is in
  • name – object name to get info for
  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)
  • service_token – service auth token
  • headers – additional headers to include in the request
Returns

a dict containing the response’s headers (all header names will be lowercase)

Raises

ClientException – HTTP HEAD request failed

swiftclient.client.http_connection(*arg, **kwarg)
Returns

tuple of (parsed url, connection object)

swiftclient.client.http_log(args, kwargs, resp, body)

swiftclient.client.logger_settings = {'redact_sensitive_headers': True, 'reveal_sensitive_prefix': 16}

Default behaviour is to redact header values known to contain secrets, such as X-Auth-Key and X-Auth-Token. Up to the first 16 chars may be revealed.

To disable, set the value of redact_sensitive_headers to False.

When header redaction is enabled, reveal_sensitive_prefix configures the maximum length of any sensitive header data sent to the logs. If the header is less than twice this length, only int(len(value)/2) chars will be logged; if it is less than 15 chars long, even less will be logged.

swiftclient.client.parse_header_string(data)

swiftclient.client.post_account(url, token, headers, http_conn=None, response_dict=None, service_token=None, query_string=None, data=None)

Update an account’s metadata.

Parameters
  • url – storage URL
  • token – auth token
  • headers – additional headers to include in the request
  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
  • service_token – service auth token
  • query_string – if set will be appended with ‘?’ to generated path
  • data – an optional message body for the request
Raises

ClientException – HTTP Post request failed

Returns

resp_headers, body

swiftclient.client.post_container(url, token, container, headers, http_conn=None, response_dict=None, service_token=None)

Update a container’s metadata.

Parameters
  • url – storage URL
  • token – auth token
  • container – container name to update
  • headers – additional headers to include in the request
  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
  • service_token – service auth token
Raises

ClientException – HTTP Post request failed

swiftclient.client.post_object(url, token, container, name, headers, http_conn=None, response_dict=None, service_token=None)

Update object metadata

Parameters
  • url – storage URL
  • token – auth token
  • container – container name that the object is in
  • name – name of the object to update
  • headers – additional headers to include in the request
  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
  • service_token – service auth token
Raises

ClientException – HTTP Post request failed

swiftclient.client.put_container(url, token, container, headers=None, http_conn=None, response_dict=None, service_token=None, query_string=None)

Create a container

Parameters
  • url – storage URL
  • token – auth token
  • container – container name to create
  • headers – additional headers to include in the request
  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
  • service_token – service auth token
  • query_string – if set will be appended with ‘?’ to generated path
Raises

ClientException – HTTP PUT request failed

swiftclient.client.put_object(url, token=None, container=None, name=None, contents=None, content_length=None, etag=None, chunk_size=None, content_type=None, headers=None, http_conn=None, proxy=None, query_string=None, response_dict=None, service_token=None)

Put an object

Parameters
  • url – storage URL
  • token – auth token; if None, no token will be sent
  • container – container name that the object is in; if None, the container name is expected to be part of the url
  • name – object name to put; if None, the object name is expected to be part of the url
  • contents – a string, a file-like object or an iterable to read object data from; if None, a zero-byte put will be done
  • content_length – value to send as content-length header; also limits the amount read from contents; if None, it will be computed via the contents or chunked transfer encoding will be used
  • etag – etag of contents; if None, no etag will be sent
  • chunk_size – chunk size of data to write; it defaults to 65536; used only if the contents object has a ‘read’ method, e.g. file-like objects, ignored otherwise
  • content_type – value to send as content-type header, overriding any value included in the headers param; if None and no value is found in the headers param, an empty string value will be sent
  • headers – additional headers to include in the request, if any
  • http_conn – a tuple of (parsed url, HTTPConnection object), (If None, it will create the conn object)
  • proxy – proxy to connect through, if any; None by default; str of the format ‘http://127.0.0.1:8888’ to set one
  • query_string – if set will be appended with ‘?’ to generated path
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
  • service_token – service auth token
Returns

etag

Raises

ClientException – HTTP PUT request failed

swiftclient.client.quote(value, safe='/')

Patched version of urllib.quote that encodes utf8 strings before quoting. On Python 3, call directly urllib.parse.quote().

swiftclient.client.resp_header_dict(resp)

swiftclient.client.safe_value(name, value)

Only show up to logger_settings[‘reveal_sensitive_prefix’] characters from a sensitive header.

Parameters
  • name – Header name
  • value – Header value
Returns

Safe header value

swiftclient.client.scrub_headers(headers)

Redact header values that can contain sensitive information that should not be logged.

Parameters

headers – Either a dict or an iterable of two-element tuples

Returns

Safe dictionary of headers with sensitive information removed

swiftclient.client.store_response(resp, response_dict)

store information about an operation into a dict

Parameters
  • resp – an http response object containing the response headers
  • response_dict – a dict into which are placed the status, reason and a dict of lower-cased headers

swiftclient.service

class swiftclient.service.ResultsIterator(futures)

Bases: six.Iterator

class swiftclient.service.SwiftCopyObject(object_name, options=None)

Bases: object

Class for specifying an object copy, allowing the destination/headers/metadata/fresh_metadata to be specified separately for each individual object. destination and fresh_metadata should be set in options

exception swiftclient.service.SwiftError(value, container=None, obj=None, segment=None, exc=None)

Bases: exceptions.Exception

class swiftclient.service.SwiftPostObject(object_name, options=None)

Bases: object

Class for specifying an object post, allowing the headers/metadata to be specified separately for each individual object.

class swiftclient.service.SwiftService(options=None)

Bases: object

Service for performing swift operations

capabilities(url=None, refresh_cache=False)

List the cluster capabilities.

Parameters

url – Proxy URL of the cluster to retrieve capabilities.

Returns

A dictionary containing the capabilities of the cluster.

Raises

ClientException

copy(container, objects, options=None)

Copy operations on a list of objects in a container. Destination containers will be created.

Parameters
  • container – The container from which to copy the objects.
  • objects

    A list of object names (strings) or SwiftCopyObject instances containing an object name and an options dict (can be None) to override the options for that individual copy operation:

    [
        'object_name',
        SwiftCopyObject(
            'object_name',
             options={
                'destination': '/container/object',
                'fresh_metadata': False,
                ...
                }),
        ...
    ]

    The options dict is described below.

  • options

    A dictionary containing options to override the global options specified during the service object creation. These options are applied to all copy operations performed by this call, unless overridden on a per object basis. The options “destination” and “fresh_metadata” do not need to be set, in this case objects will be copied onto themselves and metadata will not be refreshed. The option “destination” can also be specified in the format ‘/container’, in which case objects without an explicit destination will be copied to the destination /container/original_object_name. Combinations of multiple objects and a destination in the format ‘/container/object’ is invalid. Possible options are given below:

    {
        'meta': [],
        'header': [],
        'destination': '/container/object',
        'fresh_metadata': False,
    }
Returns

A generator returning the results of copying the given list of objects.

Raises

SwiftError

delete(container=None, objects=None, options=None)

Delete operations on an account, optional container and optional list of objects.

Parameters
  • container – The container to delete or delete from.
  • objects – The list of objects to delete.
  • options

    A dictionary containing options to override the global options specified during the service object creation:

    {
        'yes_all': False,
        'leave_segments': False,
        'prefix': None,
        'header': [],
    }
Returns

A generator for returning the results of the delete operations. Each result yielded from the generator is either a ‘delete_container’, ‘delete_object’, ‘delete_segment’, or ‘bulk_delete’ dictionary containing the results of an individual delete operation.

Raises
  • ClientException
  • SwiftError
download(container=None, objects=None, options=None)

Download operations on an account, optional container and optional list of objects.

Parameters
  • container – The container to download from.
  • objects – A list of object names to download (a list of strings).
  • options

    A dictionary containing options to override the global options specified during the service object creation:

    {
        'yes_all': False,
        'marker': '',
        'prefix': None,
        'no_download': False,
        'header': [],
        'skip_identical': False,
        'out_directory': None,
        'checksum': True,
        'out_file': None,
        'remove_prefix': False,
        'shuffle' : False
    }
Returns

A generator for returning the results of the download operations. Each result yielded from the generator is a ‘download_object’ dictionary containing the results of an individual file download.

Raises
  • ClientException
  • SwiftError
list(container=None, options=None)

List operations on an account, container.

Parameters
  • container – The container to make the list operation against.
  • options

    A dictionary containing options to override the global options specified during the service object creation:

    {
        'long': False,
        'prefix': None,
        'delimiter': None,
        'header': []
    }
Returns

A generator for returning the results of the list operation on an account or container. Each result yielded from the generator is either a ‘list_account_part’ or ‘list_container_part’, containing part of the listing.

post(container=None, objects=None, options=None)

Post operations on an account, container or list of objects

Parameters
  • container – The container to make the post operation against.
  • objects

    A list of object names (strings) or SwiftPostObject instances containing an object name, and an options dict (can be None) to override the options for that individual post operation:

    [
        'object_name',
        SwiftPostObject('object_name', options={...}),
        ...
    ]

    The options dict is described below.

  • options

    A dictionary containing options to override the global options specified during the service object creation. These options are applied to all post operations performed by this call, unless overridden on a per object basis. Possible options are given below:

    {
        'meta': [],
        'header': [],
        'read_acl': None,   # For containers only
        'write_acl': None,  # For containers only
        'sync_to': None,    # For containers only
        'sync_key': None    # For containers only
    }
Returns

Either a single result dictionary in the case of a post to a container/account, or an iterator for returning the results of posts to a list of objects.

Raises

SwiftError

stat(container=None, objects=None, options=None)

Get account stats, container stats or information about a list of objects in a container.

Parameters
  • container – The container to query.
  • objects – A list of object paths about which to return information (a list of strings).
  • options

    A dictionary containing options to override the global options specified during the service object creation. These options are applied to all stat operations performed by this call:

    {
        'human': False,
        'header': []
    }
Returns

Either a single dictionary containing stats about an account or container, or an iterator for returning the results of the stat operations on a list of objects.

Raises

SwiftError

upload(container, objects, options=None)

Upload a list of objects to a given container.

Parameters
  • container – The container (or pseudo-folder path) to put the uploads into.
  • objects

    A list of file/directory names (strings) or SwiftUploadObject instances containing a source for the created object, an object name, and an options dict (can be None) to override the options for that individual upload operation:

    [
        '/path/to/file',
        SwiftUploadObject('/path', object_name='obj1'),
        ...
    ]

    The options dict is as described below.

    The SwiftUploadObject source may be one of:

    • A file-like object (with a read method)
    • A string containing the path to a local file or directory
    • None, to indicate that we want an empty object
  • options

    A dictionary containing options to override the global options specified during the service object creation. These options are applied to all upload operations performed by this call, unless overridden on a per object basis. Possible options are given below:

    {
        'meta': [],
        'header': [],
        'segment_size': None,
        'use_slo': False,
        'segment_container': None,
        'leave_segments': False,
        'changed': None,
        'skip_identical': False,
        'fail_fast': False,
        'dir_marker': False  # Only for None sources
    }
Returns

A generator for returning the results of the uploads.

Raises
  • SwiftError
  • ClientException
class swiftclient.service.SwiftUploadObject(source, object_name=None, options=None)

Bases: object

Class for specifying an object upload, allowing the object source, name and options to be specified separately for each individual object.

swiftclient.service.get_conn(options)

Return a connection building it from the options.

swiftclient.service.get_from_queue(q, timeout=864000)

swiftclient.service.get_future_result(f, timeout=86400)

swiftclient.service.interruptable_as_completed(fs, timeout=86400)

swiftclient.service.mkdirs(path)

swiftclient.service.process_options(options)

swiftclient.service.split_headers(options, prefix=u'')

Splits ‘Key: Value’ strings and returns them as a dictionary.

Parameters
  • options – Must be one of: * an iterable of ‘Key: Value’ strings * an iterable of (‘Key’, ‘Value’) pairs * a dict of {‘Key’: ‘Value’} pairs
  • prefix – String to prepend to all of the keys in the dictionary. reporting.

swiftclient.exceptions

exception swiftclient.exceptions.ClientException(msg, http_scheme='', http_host='', http_port='', http_path='', http_query='', http_status=None, http_reason='', http_device='', http_response_content='', http_response_headers=None)

Bases: exceptions.Exception

classmethod from_response(resp, msg=None, body=None)

swiftclient.multithreading

class swiftclient.multithreading.ConnectionThreadPoolExecutor(create_connection, max_workers)

Bases: concurrent.futures.thread.ThreadPoolExecutor

A wrapper class to maintain a pool of connections alongside the thread pool. We start by creating a priority queue of connections, and each job submitted takes one of those connections (initialising if necessary) and passes it as the first arg to the executed function.

At the end of execution that connection is returned to the queue.

By using a PriorityQueue we avoid creating more connections than required. We will only create as many connections as are required concurrently.

submit(fn, *args, **kwargs)

class swiftclient.multithreading.MultiThreadingManager(create_connection, segment_threads=10, object_dd_threads=10, object_uu_threads=10, container_threads=10)

Bases: object

One object to manage context for multi-threading.  This should make bin/swift less error-prone and allow us to test this code.

Parameters
  • segment_threads – The number of threads allocated to segment uploads
  • object_dd_threads – The number of threads allocated to object download/delete jobs
  • object_uu_threads – The number of threads allocated to object upload/update based jobs
  • container_threads – The number of threads allocated to container/account level jobs
class swiftclient.multithreading.OutputManager(print_stream=None, error_stream=None)

Bases: object

One object to manage and provide helper functions for output.

This object is a context manager and returns itself into the context.  When entering the context, two printing threads are created (see below) and they are waited on and cleaned up when exiting the context.

Also, thread-safe printing to two streams is provided.  The print_msg() method will print to the supplied print_stream (defaults to sys.stdout) and the error() method will print to the supplied error_stream (defaults to sys.stderr).  Both of these printing methods will format the given string with any supplied *args (a la printf). On Python 2, Unicode messages are encoded to utf8.

The attribute self.error_count is incremented once per error message printed, so an application can tell if any worker threads encountered exceptions or otherwise called error() on this instance. The swift command-line tool uses this to exit non-zero if any error strings were printed.

Parameters
  • print_stream – The stream to which print_msg() sends formatted messages.
  • error_stream – The stream to which error() sends formatted messages.

On Python 2, Unicode messages are encoded to utf8.

DEFAULT_OFFSET = 14

error(msg, *fmt_args)

get_error_count()

print_items(items, offset=14, skip_missing=False)

print_msg(msg, *fmt_args)

print_raw(data)

warning(msg, *fmt_args)

swiftclient.utils

Miscellaneous utility functions for use with Swift.

class swiftclient.utils.LengthWrapper(readable, length, md5=False)

Bases: object

Wrap a filelike object with a maximum length.

Fix for https://github.com/kennethreitz/requests/issues/1648. It is recommended to use this class only on files opened in binary mode.

Parameters
  • readable – The filelike object to read from.
  • length – The maximum amount of content that can be read from the filelike object before it is simulated to be empty.
  • md5 – Flag to enable calculating the MD5 of the content as it is read.

get_md5sum()

read(size=-1)

reset

class swiftclient.utils.NoopMD5(*a, **kw)

Bases: object

hexdigest(*a, **kw)

update(*a, **kw)

class swiftclient.utils.ReadableToIterable(content, chunk_size=65536, md5=False)

Bases: object

Wrap a filelike object and act as an iterator.

It is recommended to use this class only on files opened in binary mode. Due to the Unicode changes in Python 3, files are now opened using an encoding not suitable for use with the md5 class and because of this hit the exception on every call to next. This could cause problems, especially with large files and small chunk sizes.

Parameters
  • content – The filelike object that is yielded from.
  • chunk_size – The max size of each yielded item.
  • md5 – Flag to enable calculating the MD5 of the content as it is yielded.

get_md5sum()

next()

swiftclient.utils.config_true_value(value)

Returns True if the value is either True or a string in TRUE_VALUES. Returns False otherwise. This function comes from swift.common.utils.config_true_value()

swiftclient.utils.generate_temp_url(path, seconds, key, method, absolute=False, prefix=False, iso8601=False)

Generates a temporary URL that gives unauthenticated access to the Swift object.

Parameters
·

path – .INDENT 2.0

The full path to the Swift object or prefix if

a prefix-based temporary URL should be generated. Example:

/v1/AUTH_account/c/o or /v1/AUTH_account/c/prefix.

·

seconds – time in seconds or ISO 8601 timestamp. If absolute is False and this is the string representation of an integer, then this specifies the amount of time in seconds for which the temporary URL will be valid. If absolute is True then this specifies an absolute time at which the temporary URL will expire.

·

key – The secret temporary URL key set on the Swift cluster. To set a key, run ‘swift post -m “Temp-URL-Key: <substitute tempurl key here>”’

·

method – A HTTP method, typically either GET or PUT, to allow for this temporary URL.

·

absolute – if True then the seconds parameter is interpreted as a Unix timestamp, if seconds represents an integer.

·

prefix – if True then a prefix-based temporary URL will be generated.

·

iso8601 – if True, a URL containing an ISO 8601 UTC timestamp instead of a UNIX timestamp will be created.

Raises

ValueError – if timestamp or path is not in valid format.

Returns

the path portion of a temporary URL

swiftclient.utils.get_body(headers, body)

swiftclient.utils.iter_wrapper(iterable)

swiftclient.utils.n_at_a_time(seq, n)

swiftclient.utils.n_groups(seq, n)

swiftclient.utils.parse_api_response(headers, body)

swiftclient.utils.prt_bytes(num_bytes, human_flag)

convert a number > 1024 to printable format, either in 4 char -h format as with ls -lh or return as 12 char right justified string

swiftclient.utils.report_traceback()

Reports a timestamp and full traceback for a given exception.

Returns

Full traceback and timestamp.

swiftclient.utils.split_request_headers(options, prefix='')

Indices and Tables

License

Copyright 2013 OpenStack, LLC.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Info

Oct 11, 2017 3.4.0 python-swiftclient