Skip to content

developer.md

Latest commit

 

History

History
203 lines (139 loc) · 7.99 KB

developer.md

File metadata and controls

203 lines (139 loc) · 7.99 KB
Raw

Developer setup

Setting up Python Virtual Environment (VE)

Using uv

Once you have installed uv, in the root of your source tree you can:

Install the required Python version and create the venv (one-off task)

uv python install 3.13
uv venv --python 3.13

Enter the venv (for the lifetime of the shell)

source .venv/bin/activate

Docker requirements

Most modules require docker and docker-compose to be installed to run tests.

Installation of PIXL modules

You can install all PIXL Python modules by running the following command from the PIXL/ directory:

uv sync

See each service's README for instructions for individual developing and testing instructions.

Non-editable installation

By default, uv will install pixl and its workspace members in editable mode. To install in non-editable mode:

uv sync --no-editable

You will need to set the PIXL_ROOT, HOST_EXPORT_ROOT_DIR, and HOST_EXPORT_ROOT_DIR_MOUNT environment variables if you install PIXL in non-editable mode. See the cli docs for more info.

Testing

Module-level testing

Once you have installed each module, you can run the tests for a module using the pytest command, e.g.

cd pixl_core/
pytest

Alternatively, you can run most of the module-level tests from the root of the repo with:

pytest #to test all tests `testpaths` pytest.ini

The pytest.ini file in the root of the repo contains the configuration for running most of the module-level tests at once. However, pixl_dcmd and hasher have conftests.py files that clash, so only pixl_dcmd is included as a testpath in the top-level pytest.ini. You will therefore need to run tests for hasher from the hasher directory.

Enabling default Docker socket for testing pixl_core

We have tests in pixl_core for uploading DICOM to XNAT as an endpoint. These tests use xnat4tests to spin up a docker container running XNAT.

xnat4tests requires you to allow the Docker daemon to listen for Docker Engine API requests via the default socket. This is because xnat4tests set up the XNAT Container Service for launching other containers that run analysis pipelines.

If you are using Docker Desktop, you will need to enable Docker to listen on the default socket by going to Settings > Advanced and checking the box Allow the default Docker socket to be used.

If your are running Docker Engine on Linux, listening on this socket should be enabled by default.

Integration tests

There are also integration tests in PIXL/test/ directory that can be run using the PIXL/test/run-system-test.sh. See the integration test docs for more info.

Workflow

Before raising a PR, make sure to run all tests for each PIXL module and not just the component you have been working on as this will help us catch unintentional regressions without spending GH actions minutes.

Linting

For Python development we use ruff and mypy alongside pytest. There is support (sometimes through plugins) for these tools in most IDEs & editors.

We run pre-commit as part of the GitHub Actions CI.

To run it locally as a one-off:

pre-commit run --all-files

To install the git pre-commit hook locally so it runs every time you make a commit:

pre-commit install

The pre-commit configuration can be found in .pre-commit-config.yml.

Environment variables

Running the pixl pipeline and the tests requires a set of environment variables to be set. The test/ directory contains a complete .env file that can be used to run the pipeline and tests locally. Either run any pixl commands from the test/ directory, or copy the test/.env file to the root of the repository.

Secrets

PIXL uses an Azure Keyvault to store authentication details for external services. We have a development keyvault for testing. Access to this keyvault is provided by a set of environment variables specified in test/.secrets.env.sample. To run the pipeline locally, you will need to copy this file to test/.secrets.env and fill out the necessary values, which can be found in the pixl-dev-secrets.env shared LastPass note.

Observability

PIXL can export structured logs to an OpenTelemetry (OTel) Collector. Observability is opt-in; all services run as normal without it.

Enabling OTel in PIXL

PIXL exports telemetry via the OpenTelemetry Protocol (OTLP) and works with any OTel-compatible observability backend.

To enable observability, set OTEL_SDK_DISABLED to false and define an OTEL_EXPORTER_OTLP_ENDPOINT in the .env. The endpoint be for the gRPC endpoint of an OTel collector, e.g. localhost:4317 (4317 is the standard OTLP gRPC port).

After starting the PIXL services, logs should start to appear in your collector's UI.

Disabling OTel

Set OTEL_SDK_DISABLED to true to disable all telemetry. No other configuration is needed.

Adding context to logs

To make logs filterable and to link related logs together (e.g. logs related given DICOM study), we attach structured context fields to them. Any field bound with loguru's logger.contextualize() or logger.bind() is exported as a top-level, queryable attribute on the OTel log record.

When adding context:

  • Bind each field as soon as it is known, e.g. bind pseudo_study_uid as soon as a study has been anonymised.
  • Bind fields at the start of a unit of work using logger.contextualize(...). For example, bind study_uid per study when iterating over studies, or orthanc_resource_id when iterating over resources.
  • For a single log call, logger.bind(...).info(...) is preferable to the context manager.
  • Use the same field names across services so logs can be joined up
  • If a field has been pseudonymised, bind a new field prefixed with pseudo_, e.g. study_uid becomes pseudo_study_uid after pseudonymisation

Adding metrics

Custom metrics are defined centrally in core.metrics. To add a new metric:

  1. Add a field for it on the PixlMetrics dataclass, and create the instrument (e.g. a counter) in initialise_metrics(). Metric names use dots as separators, e.g. pixl.studies.exported.
  2. Add a record_* helper that records a value on the instrument. Guard against the instrument being None (it is unset when telemetry is disabled) and return early if so.
  3. Call the record_* helper from the relevant service(s).

You can pass attributes to the metric that can later be used for filtering and aggregation, e.g. project_name. It's highly recommended to keep attribute values low-cardinality - each distinct combination of attribute values creates a separate time series, so avoid unbounded values like raw IDs or full tracebacks.

RabbitMQ queue metrics

The RabbitMQ Docker image includes a Prometheus endpoint that exposes queue backlog depth metrics. We use this to scrape queue depth metrics for each queue in the PIXL RabbitMQ broker, rather than manually defining a metric within PIXL. This does, however, require defining a scrape job in the Prometheus configuration of the OTel Collector, although the configuration is fairly minimal.

Because metrics are scraped from the Prometheus endpoint, they are independent of OTEL_SDK_DISABLED. This means queue metrics will always be collected whenever something is scraping the endpoint.

Note, queue depth is per-queue only, and cannot be broken down by project.