bpkg-repository-types - Man Page

This help topic describes the repository types recognized by bpkg, their structure, and the format of their URLs. Currently three types of repositories are supported: archive-based pkg, directory-based dir, and version control-based git.

The repository location may specify the repository type as part of the URL scheme component in the type+protocol form. For example:

git+https://example.com/foo
dir+file:///tmp/repo

Note that the explicit specification is only needed when the correct type cannot be guessed from the URL. See bpkg-rep-add(1) for details.

A pkg repository is archive-based. That is, it contains a collection of various packages/versions as archive files. For more information on the structure of pkg repositories refer to The build2 Package Manager Manual. The pkg repository location can be a local directory path or an http(s):// URL.

A dir repository is directory-based. That is, it contains a collection of various packages as directories but only a single version per package can be present in such a repository. The dir repository location can be a local directory path or a file:// URL.

A dir repository is expected to contain either the manifest or packages.manifest file in the root directory of the repository. If it only contains manifest, then it is assumed to be a simple, single-package repository with the manifest file being its package manifest. Otherwise, the packages.manifest file should list the locations of available packages as described in Package List Manifest for dir Repositories (#manifest-package-list-dir).

A dir repository may also contain the repositories.manifest file in the root directory of the repository. This file can be used to describe the repository itself as well as specify its prerequisite and complement repositories. See Repository List Manifest (#manifest-repository-list) for details on the format and semantics of this file.

A git repository is version control-based. That is, it normally contains multiple versions of the same package (but can also contain several, usually related, packages in the same repository).

A git repository has the same structure and manifest files as the dir repository. See Package List Manifest for dir Repositories (#manifest-package-list-dir) and Repository List Manifest (#manifest-repository-list) for details on their format and semantics.

Theoretically, a git repository may contain as many package versions as there are commits. Practically, however, we are normally only interested in a small subset of them while fetching and processing the necessary information for all of them could be prohibitively expensive.  As a result, by default, only advertised tags in the refs/tags/v* form where the part after v is also a valid standard version (#module-version) are considered to be sources of useful package versions. These commits normally correspond to released versions and are called the default set. Note that only the latest revision of each such version is considered.

Instead of the default set, it is possible to provide a custom set of available versions by specifying one or more commit ids and/or references and/or reference patterns in the repository URL fragment (see git-ls-remote(1) for details on advertised references). For example:

https://example.com/foo.git#v1.2.3
https://example.com/foo.git#master
https://example.com/foo.git#af234f56
https://example.com/foo.git#tags/releases/*
https://example.com/foo.git#HEAD,tags/v1.*.*,heads/feature-*

Furthermore, it is possible to expand (or narrow down) the default set using the special ## fragment notation. For example:

https://example.com/foo.git##HEAD     - default set plus HEAD
https://example.com/foo.git##heads/*  - default set plus branches
https://example.com/foo.git##-v1.*    - default set minus v1.*

A git repository URL fragment is a comma-separated list of reference filters in the following form:

[refname][@commit]

Either refname, commit, or both must be specified. If both are specified then refname is only used to minimize the amount of data fetched and commit is expected to belong to its history. For example:

.../foo.git#master@48fba3625d65941bb85a39061bcf795d4949c778

The refname part can be an abbreviated commit id or an advertised reference or reference pattern under refs/. While commit must be the complete, 40-characters SHA1 that need not be advertised. For convenience, a 40-characters filter that consists of only hexadecimal digits is assumed to be commit even if not prefixed with @. In an unlikely event this produces an incorrect result, the @-form with omitted commit can be used. For example:

.../foo.git#48fba3625d65941bb85a39061bcf795d4949c778   (commit id)
.../foo.git#deadbeefdeadbeefdeadbeefdeadbeefdeadbeef@  (reference)

The refname part can use the * and ? wildcard pattern characters with the standard semantics as well as the ** character sequence which matches in subdirectories, recursively. For example:

.../foo.git#tags/v*    - tags/v1.2.3 but not tags/old/v0.1.0
.../foo.git#tags/v**   - tags/v1.2.3 and tags/old/v0.1.0

A relative refname is searched for in refs/, refs/tags/, and refs/heads/ as well as among symbolic references like HEAD. To anchor it to refs/ make it absolute, for example:

.../foo.git#tags/v*   - refs/tags/v1.2.3 but also refs/heads/tags/voo
.../foo.git#/tags/v*  - refs/tags/v1.2.3 only

While a refname pattern is allowed not match any references, a non-pattern that doesn't resolve to a reference is invalid.

If a refname starts with minus (-) then it is treated as an exclusion filter – any references that it matches are excluded from the set included by the preceding filters (or the default set). For example:

.../foo.git#v*,-v1.*  - exclude v1.* from v*
.../foo.git##-v1.*    - exclude v1.* from default set

To support specifying literal leading minus, a refname that starts with plus (+) is treated as an inclusion filter. For example:

.../foo.git#+x   - include  x
.../foo.git#+-x  - include -x
.../foo.git#++x  - include +x

Currently supported git protocols are git://, ssh:// (but not scp pseudo-URL syntax), http://, and https:// for remote repositories and file:// for local repositories. While bpkg tries to minimize the amount of information (history) fetched, it is not always possible for some protocols and/or server configurations, as discussed next.

A git repository accessible via http(s):// can use either dumb or smart protocol (refer to the git documentation for details). The dumb protocol provides only limited support for fetch minimization and if this protocol is used, then bpkg has no choice but to download a substantial amount of history.

The smart protocol allows fetching of minimal history for tags and branches. Whether this is also possible for (all) commit ids depends on whether the server is configured to allow fetching unadvertised commits. For details, refer to the uploadpack.allowReachableSHA1InWant and uploadpack.allowAnySHA1InWant git configuration values.

The git:// and ssh:// protocols are similar to smart http:// in that they support fetching minimal history for tags and branches and may or may not support this for commit ids depending on the server configuration. Note, however, that unlike for http(s)://, for these protocols bpkg does not try to sense if fetching unadvertised commits is allowed and always assumes that it is not. Also note that the sensed or assumed protocol capabilities can be overridden for a git repository URL prefix using the --git-capabilities option (bpkg-common-options(1)).

Based on this information, to achieve optimal results the recommended protocol for remote repositories is smart https://. Additionally, if you are planning to refer to unadvertised commit ids, then also consider configuring the server to allow fetching unadvertised commits.

The file:// protocol has the same fetch minimization support as git:// and is therefore treated the same.

Send bug reports to the users@build2.org mailing list.

Copyright (c) 2014-2023 the build2 authors.

Permission is granted to copy, distribute and/or modify this document under the terms of the MIT License.

bdep-ci(1), bpkg(1), bpkg-common-options(1), bpkg-rep-add(1).