conda-index - Man Page

To track whether a package is indexed in the cache or not, conda-index uses a table named stat, with a compound primary key (stage, path). Think of packages moving from "upstream" to "downstream" by being duplicated in the stat table for each stage.

The main stages are 'fs' which is called the upstream stage, and 'indexed'. 'fs' means that the artifact is on the filesystem. 'indexed' means that the entry already exists in the database (same filename, same timestamp, same hash), and its package metadata has been extracted to the index_json etc. tables. Paths in 'fs' but not in 'indexed' need to be unpacked to have their metadata added to the database. Paths in 'indexed' but not in 'fs' will be ignored and left out of repodata.json.

First, conda-index adds all files in a subdir to the 'fs' upstream stage. Each package has an entry ('fs', path, mtime, size, ...). This involves a listdir() and stat() for each file in the index.

Next, conda-index looks for all changed_packages(): paths in the upstream (fs) stage that are either missing from or have a different size, mtime than those in the indexed stage.

The changed_packages() are examined one by one, and their metadata is stored as json in various tables in conda-index's database.

Finally, a join between the upstream stage, usually 'fs', and the index_json table yields repodata_from_packages.json without any repodata patches.

SELECT path, index_json 
FROM stat JOIN index_json
USING (path) 
WHERE stat.stage = :upstream_stage

The steps to create repodata.json, including any repodata patches, and to create current_repodata.json with only the latest versions of each package, are similar to pre-sqlite3 conda-index. The raw repodata_from_packages.json is loaded, each record is sent through a patch function (if provided) than can modify or exclude that record, and the result is serialized as repodata.json.

The other cached metadata tables are used to create channeldata.json, an optional file that aggregates packages from every subdir into a channel listing.

Other techniques are possible but generally require using the conda-index API and are not available from the command line interface.

Sometimes it is useful to create an index without unpacking real packages from the local filesystem; for example, when translating .whl package metadata to conda repodata. As of version 0.12.0, conda-index adds a md or metadata stage to support this mode. The md stage doesn't participate in changed_packages() or conda-index's package extraction pipeline. Instead, the user inserts stat table entries and metadata into conda-indexs database either directly or by using conda-index APIs. Then, the output query is changed to

SELECT path, index_json 
FROM stat JOIN index_json 
USING (path) 
WHERE stat.stage in ('fs', 'md')

When it's time to output repodata, packages that are in the fs or md stage, and also have a row in index_json, are included.

It is possible to index without calling stat() on each package, or without even having all packages stored on the indexing machine. This can be done by subclassing CondexIndexCache() and replacing the save_fs_state() and changed_packages() methods.

Advanced users can use the CLI or the API to run conda_index on a partial local package repository. It is possible to add a few local packages to a much larger index instead of keeping every package on the machine running conda-index.

For example, by running python -m conda_index --db postgresql --update-only [DIR], conda-index will add or update packages in [DIR] to repodata, while keeping already-indexed packages in the output repodata.json. The output repodata can then be copied to a server that has every package.

If --update-only is used, the stat table must be altered to remove packages from repodata.json, e.g. DELETE FROM stat WHERE path = '<prefix>/<subdir>/package.conda' AND stage = 'fs'.

When using this option, care must be taken to never run conda-index without --update-only or all the "missing" packages will be dropped from the index.

Standalone conda-index uses a per-subdir sqlite database to track package metadata, unlike the older version which used millions of tiny .json files. The new strategy is much faster because we don't have to pay for many individual stat() or open() calls.

The whole schema looks like this:

<subdir>/.cache % sqlite3 cache.db
SQLite version 3.41.2 2023-03-22 11:56:21
Enter ".help" for usage hints.
sqlite> .schema
CREATE TABLE about (path TEXT PRIMARY KEY, about BLOB);
CREATE TABLE index_json (path TEXT PRIMARY KEY, index_json BLOB);
CREATE TABLE recipe (path TEXT PRIMARY KEY, recipe BLOB);
CREATE TABLE recipe_log (path TEXT PRIMARY KEY, recipe_log BLOB);
CREATE TABLE run_exports (path TEXT PRIMARY KEY, run_exports BLOB);
CREATE TABLE post_install (path TEXT PRIMARY KEY, post_install BLOB);
CREATE TABLE icon (path TEXT PRIMARY KEY, icon_png BLOB);
CREATE TABLE stat (
                stage TEXT NOT NULL DEFAULT 'indexed',
                path TEXT NOT NULL,
                mtime NUMBER,
                size INTEGER,
                sha256 TEXT,
                md5 TEXT,
                last_modified TEXT,
                etag TEXT
            );
CREATE UNIQUE INDEX idx_stat ON stat (path, stage);
CREATE INDEX idx_stat_stage ON stat (stage, path);

sqlite> select stage, path from stat where path like 'libcurl%';
fs|libcurl-7.84.0-hc6d1d07_0.conda
fs|libcurl-7.86.0-h0f1d93c_0.conda
fs|libcurl-7.87.0-h0f1d93c_0.conda
fs|libcurl-7.88.1-h0f1d93c_0.conda
fs|libcurl-7.88.1-h9049daf_0.conda
indexed|libcurl-7.84.0-hc6d1d07_0.conda
indexed|libcurl-7.86.0-h0f1d93c_0.conda
indexed|libcurl-7.87.0-h0f1d93c_0.conda
indexed|libcurl-7.88.1-h0f1d93c_0.conda
indexed|libcurl-7.88.1-h9049daf_0.conda

Most of these tables store json-format metadata extracted from each package.

select * from index_json where path = 'libcurl-7.88.1-h9049daf_0.conda';
'libcurl-7.88.1-h9049daf_0.conda'
'{"build":"h9049daf_0",...,"sha256":"37b8d58c05386ac55d1d8e196c90b92b0a63f3f1fe2fa916bf5ed3e1656d8e14","size":321706}'

Megabytes added per day:

select
  date(mtime, 'unixepoch') as d,
  printf('%0.2f', sum(size) / 1e6) as MB
from
  stat
group by
  date(mtime, 'unixepoch')
order by
  mtime desc

As of conda-index 0.7.0, conda-index can use a PostgreSQL database. conda-index uses a database to store package metadata. Once all the metadata is stored, it creates repodata from a query. By default it uses a sqlite3 database stored alongside the package files, but it can optionally use PostgreSQL.

The database backend is controlled by the --db <backend> and --db-url command line arguments, or the CONDA_INDEX_DBURL environment variable replaces --db-url. For example, python -m conda_index --db postgresql chooses PostgreSQL with the default postgresql:///conda_index database URL.

To use a PostgreSQL database with conda-index, install conda-index's PostgreSQL-specific dependencies into its environment:

conda install sqlalchemy psycopg2

Then, one way to get a PostgreSQL server is to install it with conda:

# Create a local PostgreSQL installation and conda_index database
conda install postgresql
initdb -D conda-index-db
pg_ctl -D conda-index-db -l logfile start
createdb conda_index

Finally, run the following command:

python -m conda_index --db postgresql --db-url postgresql:///conda_index [DIR]

conda_index stores package metadata in the PostgreSQL database given by a SQLAlchemy database URL <https://docs.sqlalchemy.org/en/20/core/engines.html#database-urls>.

The schema <https://github.com/conda/conda-index/blob/main/conda_index/postgres/model.py> is similar to the one used for sqlite3 <>, except that while sqlite3 uses a database file per subdirectory, in PostgreSQL all subdirectories are stored in the same database. conda_index creates a random prefix in [DIR]/.cache/cache.json to differentiate this channel from any others that may be stored in the same PostgreSQL database. Each package name is stored with the format <prefix>/<subdir>/<package>.conda in a single database.

This module provides the main entry point to create indexes from collections of conda packages.

This module provides a filesystem abstraction for sourcing packages.

• Add a metadata only stage to repodata generation for adding packages to repodata without requiring the package artifacts. (#284)

• Explicitly close sqlite3.Connection objects opened by CondaIndexCache inside ChannelIndex instead of relying on the garbage collector. This avoids ResourceWarning: unclosed database on Python 3.13+ and releases cache database files promptly so API consumers (e.g. conda-build) can open them immediately after update_index() returns. (#236)
• Fix missing f-string prefix in error message. (#291)

• Drop click dependency, use argparse. (#288)

• @dholth
• @jezdez
• @soapy1
• @conda-bot
• @jaimergp

• Support CEP 21 "run_exports in shards" (#232)
• Remove locking. conda-index will no longer try to prevent concurrent conda-index processes from indexing the same channel. (#256)
• Support experimental "repodata v3" (package data in ["v3"][<package format>] dict under the top level) to support conda-pypi wheel repodata generation. A new --repodata-next command line flag places package data under v3 key. (#262)

• Test md5=None in index_json when calling cache.store() (e.g. PyPI metadata); add type hints to assist callers. (#269)

• BaseCondaIndexCache.indexed_packages() returns dataclass instead of tuple. (#262)
• Remove experimental python -m conda_index.json2jlap script to run after indexing; sharded repodata is the new strategy for incremental repodata transfer. (#273)
• cache.indexed_shards() yields instances of a dataclass instead of (name, {}) tuple. (#280)

• Require conda-package-streaming >=0.12.0 (#260)
• Many type hints added, updated. (#262)
• BaseCondaIndexCache's list of valid package extensions can be overridden. (#262)
• Separate sharded, monolithic queries for PostgreSQL (#277)
• Remove redundant extension check when fetching shards (#281)

• @danyeaw
• @dholth
• @ryanskeith

• Add a flag --update-only that adds new or changed packages to the index, but keeps already-indexed packages in repodata even if they are missing from the filesystem. (#239)

• Improve run_exports support for PostgreSQL cache. (#239)
• Use correct "version": 1 field in repodata_shards instead of "repodata_version": 2. Remove json-only "repodata_version" key from shards index. (#231)
• Add .info.created_at field to shards index. (#237)
• Fix bug where postgresql cache would not filter packages by channel when writing repodata. (#240)
• Fix PostgreSQL cache load_all_from_cache() to use both conditions in the query. (#253)

• Default to --no-current-repodata in the CLI. This file is slow to generate for large channels and can only be used by conda's "classic" solver. Pass --current-repodata if you still need current_repodata.json. API users are not affected by this change. (#246)

• Require Python >= 3.10 (#250)

• @dholth
• @jezdez
• @kodegard
• @pavelzw

• Add postgresql as a supported database backend in addition to sqlite. (#199)
• Show error when --no-write-monolithic is combined with --current-repodata, --run-exports, or --channeldata. (#224)
• Add html title popup with dependencies for each build to index.html. (#205)
• "--html-dependencies/--no-html-dependencies" flag toggles popups. (#218)

• Include narrative documentation for python -m conda_index --db postgresql ... in Sphinx (https://conda.github.io/conda-index/). (#219)

• Update conda index command plugin to avoid re-exported type. (#227)

• @dholth
• @jtroe
• @ryanskeith

• Added support for Python 3.13 in the CI test matrix and updated related configurations. (#203)

• In sharded repodata, set base_url and shards_base_url to "" instead of leaving them undefined, for pixi compatibility. (#209)

• Add database-independent base class for (sqlite specific) CondaIndexCache. Return parsed data instead of str in run_exports(). (#206)
• Update sqlite3 create_function() arguments for "positional-only in Python 3.15" warning. (#211)

• Add --channeldata/--no-channeldata flag to toggle generating channeldata.
• Add sharded repodata (repodata split into separate files per package name).

• Remove WAL mode <https://www.sqlite.org/wal.html> from database create script, in case conda-index is used on a network file system. Note WAL mode is persistent, PRAGMA journal_mode=DELETE can be used to convert a WAL database back to a rollback journal mode. (#177)
• Separate current_repodata generation into own file, raising possibility of "doesn't depend on conda" mode.
• Update tests to account for conda-build removals. (#180)
• Publish new conda-index releases on PyPI automatically. (#195)

See also https://github.com/conda/conda-index/releases/tag/0.6.0

• Add experimental python -m conda_index.json2jlap script to run after indexing, to create repodata.jlap patch sets for incremental repodata downloads. (#125)
• Add --current-repodata/--no-current-repodata flags to control whether current_repodata.json is generated. (#139)
• Add support for CEP-15 base_url to host packages separate from repodata. (#150)
• Support fsspec (in the API only) to index any fsspec-supported remote filesystem. Also enables the input packages folder to be separate from the cache and output folders. (#143)

• Move run_exports.json query into cache, instead of directly using SQL in ChannelIndex. (#163)
• Create parents when creating <subdir>/.cache (#166)

• Approach 100% code coverage in test suite; reformat with ruff. (#145)
• Update CI configuration to test on more platforms (#142)
• Drop support for Python 3.7; support Python 3.8+ only. (#130)

• @dholth
• @jezdez
• @conda-bot

• Add --compact-json/--no-compact-json option, default to compact. (#120)
• Add an index subcommand using conda's new subcommand plugin hook, allowing conda index instead of python -m conda_index. Note the CLI has changed <https://conda.github.io/conda-index/cli.html> compared to old conda-index. When conda-build < 24.1.0 is installed, the older conda-index code will still be used instead of this plugin. (#81 via #131)

• Check size in addition to mtime when deciding which packages to index. (#108)
• Update cached index.json, not just stat values, for changed packages that are already indexed. (#108)

• Improve test coverage (#123)
• Apply ruff --fix; reformat code; syntax cleanup (#128)

• Add --run-exports to generate CEP-12 compliant run_exports.json documents for each subdir. (#102 via #110)
• Don't pretty-print repodata.json by default, saving time and space. (#111)

• Improve documentation.

• Require conda >= 4.14 (or any of the >= 22.x.y calver releases)
• Index <>
• Module Index <>
• Search Page <>