Adding a test to the dataset#

This page walks through porting one upstream conda solver test into the YAML dataset in pytest_conda_solvers/conda-solver-tests/. See Conda solver tests for what the dataset is, and the test schema for the full field reference — this page is a practical, step-by-step companion to that reference.

Note

Every test case must trace back to a real test in the conda/conda repository via its provenance block (see Provenance below). There’s currently no supported way to add a case that isn’t anchored to an existing upstream conda test function — if you want to exercise a scenario upstream doesn’t already cover, find the closest related upstream test and cite it, explaining the deviation in description (see strengthening a check).

1. Find the upstream test#

Pick a test from conda’s own suite — typically tests/core/test_solve.py, conda/testing/solver_helpers.py, or tests/test_solvers.py at the commit currently pinned across the dataset (check provenance.commit in an existing entry, e.g. basic.yaml). Note:

  • Its exact pytest node ID (e.g. tests/core/test_solve.py::test_solve_1, or, for a class-based test, conda/testing/solver_helpers.py::SolverTests.test_iopro_mkl).

  • If the upstream test performs several solves in one function body (common in test_solve.py), you’ll split it into one YAML entry per solve, with a ::N suffix appended to the node ID for each stage (::1, ::2, …).

2. Choose the kind and expected outcome#

Pick the kind that matches what the upstream test actually asserts:

kind

Use when upstream asserts…

solve

the full final environment state after a solve

solve_for_diff

only the packages unlinked/linked, not the whole state

determine_constricting_specs

which installed packages block an install/upgrade

unsatisfiable

the solver raises an error

For unsatisfiable, also pick the error type (UnsatisfiableError, ResolvePackageNotFound, PackagesNotFoundError, or SpecsConfigurationConflictError) matching the exception upstream expects — see the “Errors” section of the test schema for the shape of each. Classic and libmamba sometimes raise different exceptions or messages for the same scenario; when that happens, add two entries with the same node ID, restrict each with solvers: classic / solvers: libmamba, and give the second a b-suffixed id (see Choosing an id) and a name suffixed with _libmamba (e.g. test_nonexistent_1 / id B056, and test_nonexistent_1_libmamba / id B056b, in basic.yaml).

3. Pick channels, or extend the fixtures#

Most tests reuse one of the bundled channels (channel-1 … channel-14, channel-freeze, channel-empty, conda_format_repo) — see the TestChannel enumeration in the test schema for the full list, and browse pytest_conda_solvers/conda-solver-tests/*.yaml for tests already using the packages you need. Prefer an existing channel before adding a new one.

If the upstream test relies on packages that genuinely aren’t covered by any bundled channel, its repodata needs to be converted and added to pytest_conda_solvers/data/. This is a heavier, code-level change (not just a new YAML file):

  1. Add <channel-name>_noarch.json and/or <channel-name>_non-noarch.json under pytest_conda_solvers/data/, derived from upstream’s index.json/repodata.json fixtures — see the jq recipe at the top of pytest_conda_solvers/data/__init__.py for the expected split.

  2. Add the new channel name to the TestChannel enum in pytest_conda_solvers/models.py.

  3. If a new platform subdir is involved, also check TestSubdir and SUBDIR_MAP in pytest_conda_solvers/data/__init__.py.

Consider raising this as a separate PR/discussion first, since new fixture data affects every test that might reuse it.

4. Fill in the required fields#

Every test spec needs:

  • name — a human-readable, globally unique name (checked by tests/test_provenance.py::TestProvenanceUniqueness). Match the upstream function name where practical, adding a numeric or solver-name suffix for split/variant stages (solve_1_1, solve_1_2, test_nonexistent_1_libmamba).

  • id — a stable, globally unique short ID (also uniqueness-checked). See Choosing an id.

  • provenance — see below.

  • input — see Encode inputs and outputs.

  • output (or error for unsatisfiable).

Optional but encouraged: a description explaining anything not obvious from the YAML alone — flaky upstream markers, deliberate strengthenings of upstream’s assertions, or why a particular solver is skipped/xfailed. Link to the exact upstream lines when useful.

Provenance#

provenance:
  node_id: tests/core/test_solve.py::test_solve_1::1
  commit: 03329e0f4a627c9b9aa92ef34f7f93b9aa83e438
  url: https://github.com/conda/conda/blob/03329e0f4a627c9b9aa92ef34f7f93b9aa83e438/tests/core/test_solve.py#L58-L124

  • commit is the full 40-character SHA of the conda/conda commit the test was ported from — normally the same commit already pinned by other tests in the dataset.

  • url must be a GitHub blob permalink at that same commit, with an #L<start>-L<end> range that matches the exact start/end lines of the test function, as parsed by AST. tests/test_provenance.py enforces all of this over the network (commit-in-URL, path-matches-node-id, valid line range, and that the file exists and the range matches the function boundary at that commit) — get this wrong and CI will fail, not just a local lint. The easiest way to get an exact match: open the file on GitHub at that commit, select the function’s lines, and use GitHub’s “Copy permalink” action.

  • If you split one upstream function into multiple YAML entries, reuse the same node_id/commit/url for each, appending ::1, ::2, … to node_id per stage (this suffix is stripped again by tooling like tools/collect_ported_node_ids.py).

If the dataset’s pinned commit is ever bumped, tools/update_provenance.py re-derives commit/url (and line ranges) for every existing entry in bulk by AST-diffing the old and new commits — you shouldn’t need it when adding a single new test, only when the whole dataset moves to a newer conda revision.

Choosing an id and a file#

Each bundled YAML file uses its own id prefix:

File

Prefix

Covers

basic.yaml

B

Core solve/install scenarios

integration.yaml

I

Broader multi-step / integration scenarios

cuda.yaml

C

CUDA/glibc virtual-package overrides

constricting_specs.yaml

S

determine_constricting_specs cases

Add your test to the file matching its kind/topic, using the next unused number for that prefix (check the highest existing id in the file). Use a trailing letter suffix (B056b) for a solver-specific variant of the same upstream test, as described above. Only start a new file if you’re adding a new topical grouping that doesn’t fit any existing one — pick a new, unused prefix letter if so, and register it in this table.

5. Encode inputs and outputs#

input mirrors the fields of TestInput (see the “Input” section of the test schema for the complete list) — most fields are optional and can be omitted. A minimal solve input usually only needs channels and specs_to_add:

input:
  channels: channel-1
  specs_to_add: numpy

history_specs takes MatchSpec strings such as numpy=1.7.1. solution_records takes PrefixRecord mappings, as documented in the Input section of the test schema.

Distribution strings (used in prefix, output.final_state, output.unlink_precs, and output.link_precs) follow the form:

<channel>/${{ arch }}::<name>-<version>-<build>

Always use the literal ${{ arch }} placeholder rather than hardcoding linux-64 — the harness substitutes the actual test arch at run time. For example:

output:
  final_state:
    - channel-1/${{ arch }}::python-3.3.2-0
    - channel-1/${{ arch }}::numpy-1.7.1-py33_0

For unsatisfiable tests, error.entries takes match-spec strings (or lists of them for multi-hop conflict chains) rather than dist strings — see the “Errors” examples in the test schema for each exception type.

6. A minimal annotated example#

Here’s B001 from basic.yaml — the first stage of a two-stage upstream test — with every field annotated. It’s a real entry already in the dataset, not a template to copy verbatim; use it as a reference for the shape and pick your own name/id/provenance as described above.

tests:
  - name: solve_1_1          # human-readable, globally unique
    id: B001                 # short, globally unique id (next free number for the file's prefix)
    provenance:
      node_id: tests/core/test_solve.py::test_solve_1::1   # ::1 = first stage of a multi-solve upstream test
      commit: 03329e0f4a627c9b9aa92ef34f7f93b9aa83e438      # pinned conda/conda commit
      url: https://github.com/conda/conda/blob/03329e0f4a627c9b9aa92ef34f7f93b9aa83e438/tests/core/test_solve.py#L58-L124
    kind: solve               # asserts a full final environment state
    description: |
      Optional context: why this test exists, any deliberate strengthening
      of upstream's (weaker) assertions, or solver quirks worth flagging.
    input:
      channels: channel-1      # one of the bundled TestChannel values
      specs_to_add: numpy      # string or list of match-spec strings
    output:
      final_state:              # exact, ordered expected solve result
        - channel-1/${{ arch }}::openssl-1.0.1c-0
        - channel-1/${{ arch }}::readline-6.2-0
        - channel-1/${{ arch }}::sqlite-3.7.13-0
        - channel-1/${{ arch }}::system-5.8-1
        - channel-1/${{ arch }}::tk-8.5.13-0
        - channel-1/${{ arch }}::zlib-1.2.7-0
        - channel-1/${{ arch }}::python-3.3.2-0
        - channel-1/${{ arch }}::numpy-1.7.1-py33_0

7. Validate and run locally#

The plugin decodes every YAML file with msgspec as part of normal test collection (forbid_unknown_fields=True on every model, so a typo’d field name fails immediately with a decode error) — there’s no separate schema step to run for a plain new test case. From a repository checkout with pixi:

# run everything (bundled suite + the provenance meta-tests), per solver:
pixi run test-classic-solver
pixi run test-libmamba-solver

# or target just your new test by its `name` while iterating (spaces become
# underscores in the parametrized test id, e.g. -k solve_1_1 for "solve_1_1"):
pixi run pytest --conda-solver=classic -k "<your test's name>"
pixi run pytest --conda-solver=libmamba -k "<your test's name>"

Run against both solvers before opening a PR, unless you’ve deliberately restricted the test with solvers: to just one of them.

If you changed pytest_conda_solvers/models.py (new fields, not just a new test case), also regenerate the committed schema:

pixi run generate-schema   # or: pixi run check-schema, to only verify it's up to date

8. PR expectations#

  • name and id are unique across the whole dataset (test_provenance.py checks this for you locally and in CI).

  • provenance resolves correctly against GitHub at the exact pinned commit (network-checked in CI).

  • The test passes against classic and/or libmamba, matching whatever solvers/xfail_solvers you declared.

  • If your change touches pytest_conda_solvers/models.py, the committed docs/cst_schema.json is regenerated (pixi run check-schema is run in CI and will fail on a stale schema).

  • If you’re bumping the dataset’s pinned conda commit (as opposed to adding one test), that’s a separate, larger change driven by tools/update_provenance.py — call it out explicitly and expect it to touch many existing entries, not just yours.

Strengthening a check#

It’s common and encouraged to assert more than the upstream test does — for example, pinning the full ordered final_state where upstream only checks package count or membership. When you do this, say so explicitly in description, with a link to the weaker upstream assertion, so reviewers and future maintainers can tell an intentional strengthening apart from an overly strict, hand-copied assumption. Several existing entries in basic.yaml and constricting_specs.yaml follow this pattern, and unsatisfiable tests strengthen error-message checks across solvers by convention — see Error assertion semantics.

On this page
Edit on GitHub
Show Source