Feature request: new service for "constants"

[wernersun]

@vkuznet: we would like to store time-dependent detector configurations and calibrations in a new FOXDEN service, similar to CLEO3 constants or CMS calibrations, with well-defined intervals of validity.

[vkuznet]

This is mostly notes for myself.

For time-dependent detector configs/calibrations with intervals of validity (IOV), this is the same problem many HEP experiments solved using different RDBMS backends, e.g. Oracle. We can use PostgreSQL open-source backend, for the following reasons:

• Native range types + exclusion constraints. It can guarantee that two active IOVs for the same (tag, channel) can never overlap.
• ACID + strong consistency.
• JSONB for the payload. You still get schema flexibility for the actual calibration blob (different detectors need different constant shapes) while keeping the IOV bookkeeping (tag, channel, since/till, revision) fully relational and indexable.

[vkuznet]

I created a first prototype of new CalibrationService

Here is an example of API usage

echo
echo "insert new caliration record ..."
echo
curl -H "Authorization: bearer $t" -X POST -d'{"tag": "tracker-alignment", "channel_id": 12, "since": 100000, "till":  105000, "data": {"dx": 0.012, "dy": -0.004, "dz": 0.0}, "inserted_by": "alice", "comment": "initial alignment from cosmic run"}' \
  http://localhost:8399

echo
echo "Resolve the calibration valid at a given run/time..."
echo
curl -H "Authorization: bearer $t" "http://localhost:8399/tracker-alignment/valid?at=102000&channel_id=12"

echo
echo "List active IOVs for a tag..."
echo
curl -H "Authorization: bearer $t" "http://localhost:8399/tracker-alignment?channel_id=12"

echo
echo "Resolve the calibration valid at a given run/time..."
echo
curl -H "Authorization: bearer $t" "http://localhost:8399/tracker-alignment/valid?at=102000&channel_id=12"

echo
echo "List active IOVs for a tag..."
echo
curl -H "Authorization: bearer $t" "http://localhost:8399/tracker-alignment?channel_id=12"

echo "Full revision history (including superseded IOVs)..."
curl -H "Authorization: bearer $t" "http://localhost:8399/tracker-alignment/history?channel_id=12"

echo
echo "Correct/replace a range..."
echo
curl -H "Authorization: bearer $t" -X PUT -d'{"since": 100000, "till": 105000, "data": {"dx": 0.011, "dy": -0.0038, "dz": 0.0}, "inserted_by": "bob", "comment": "reprocessed with better cosmic sample"}' \
  "http://localhost:8399/tracker-alignment/correct?channel_id=12"

echo
echo "Retract a single IOV"
echo
curl -H "Authorization: bearer $t" -X DELETE http://localhost:8399/iov/2

And, here is an example of outcome from Calibration service

# New record
{"id":1,"tag_id":1,"tag_name":"tracker-alignment","channel_id":12,"payload_id":1,"since":100000,"till":105000,"revision":1,"is
_active":true,"inserted_at":"2026-07-29T10:09:22.056895-04:00","inserted_by":"alice","comment":"initial alignment from cosmic run"}

List active IOVs for a tag
[{"id":1,"tag_id":1,"tag_name":"tracker-alignment","channel_id":12,"payload_id":1,"since":100000,"till":105000,"revision":1,"i
s_active":true,"inserted_at":"2026-07-29T10:09:22.056895-04:00","inserted_by":"alice","comment":"initial alignment from cosmic
 run"}]

Next step would be to discuss examples of calibration constants for different CHESS beamlines.

[wernersun]

Example constants: https://chesscomputing.github.io/ChessAnalysisPipeline/workflows/EDD.html#recommended-xps23-detector-configuration

Providing both JSON and YAML views would be helpful.

For the API, I think the inputs should be

• timestamp (same convention/units as for Metadata Service)
• name/label for constants stream (perhaps the name needs to be a hierarchical path)

[vkuznet]

@keara-soloway , @rolfverberg, @wernersun please have a look at example of new calibration service and let me know if this correspond to your vision:

https://github.com/CHESSComputing/CalibrationService#example-creating-a-calibration-from-calibyaml

I already deployed new service on foxden-dev but didn't have time to test it as it requires PostgresDB setup (will do that next week). Here is a shell script example I used for testing on my laptop:

#!/bin/bash

# obtain foxden token with read+write+delete scopes
# foxden token create read+write+delete --expires=22000
t=eyJhbG....

echo 
echo "insert new caliration record ..."
echo

curl -X POST http://localhost:8399 \
  -H "Authorization: bearer $t" \
  -H "Content-Type: application/x-yaml" \
  --data-binary @CalibrationService/request.yaml

echo
echo "Resolve the calibration valid at a given run/time..."
echo
curl -H "Authorization: bearer $t" \
  "http://localhost:8399/valid/3b/btr123/cycle123/sampleName?at=102000&format=yaml"


echo
echo "Get results in json format..."
echo
curl -H "Authorization: bearer $t" \
  "http://localhost:8399/valid/3b/btr123/cycle123/sampleName?at=102000"

echo
echo "delete calibration record by label..."
echo
curl -X DELETE \
  -H "Authorization: bearer $t" \
  "http://localhost:8399/label/3b/btr123/cycle123/sampleName?channel_id=0"

And, here is an example of calibration yaml file I used in my tests: https://github.com/CHESSComputing/CalibrationService/blob/main/request.yaml

[vkuznet]

Now, Calibration services works on foxden-dev:

Here is test script

cat test_calib.sh
#!/bin/bash

set -euo pipefail
host=localhost
host=foxden-dev.classe.cornell.edu

t=eyJhbGciOi....

echo
echo "insert new caliration record ..."
echo

curl --fail-with-body --silent --show-error \
        -X POST https://$host:8399 \
        -H "Authorization: bearer $t" \
        -H "Content-Type: application/x-yaml" \
        --data-binary @CalibrationService/request.yaml

echo
echo "Resolve the calibration valid at a given run/time..."
echo
curl --fail-with-body --silent --show-error \
        -H "Authorization: bearer $t" \
        "https://$host:8399/valid/3b/btr123/cycle123/sampleName?at=102000&format=yaml"


echo
echo "Get results in json format..."
echo
curl --fail-with-body --silent --show-error \
        -H "Authorization: bearer $t" \
        "https://$host:8399/valid/3b/btr123/cycle123/sampleName?at=102000"

echo
echo "delete calibration record by label..."
echo
curl --fail-with-body --silent --show-error \
        -X DELETE \
        -H "Authorization: bearer $t" \
        "https://$host:8399/label/3b/btr123/cycle123/sampleName?channel_id=0"

And, here are results from running test script

./test_calib.sh

insert new calibration record ...

channel_id: 0
comment: initial eta mapping
id: 1
inserted_at: "2026-08-01T12:21:49.988032-04:00"
inserted_by: vk
is_active: true
label: /3b/btr123/cycle123/sampleName
label_id: 1
payload_id: 1
revision: 1
since: 100000
till: 105000

Resolve the calibration valid at a given run/time...
iov:
    channel_id: 0
    comment: initial eta mapping
    id: 1
    inserted_at: "2026-08-01T12:21:49.988032-04:00"
    inserted_by: vk
    is_active: true
    label: /3b/btr123/cycle123/sampleName
    label_id: 1
    payload_id: 1
    revision: 1
    since: 100000
    till: 105000
payload:
    detectors:
        - attrs:
            eta: -180
          id: 0
        - attrs:
            eta: -171.8181818181818
          id: 1
...

Get results in json format...

{"iov":{"id":1,"label_id":1,"label":"/3b/btr123/cycle123/sampleName","channel_id":0,"payload_id":1,"since":100000,"till":105000,"revision":1,"is_active":true,"inserted_at":"2026-08-01T12:21:49.988032-04:00","inserted_by":"vk","comment":"initial eta mapping"},"payload":{"detectors":[{"id":0,"attrs":{"eta":-180}},{"id":1,"attrs":{"eta":-171.8181818181818}},{"id":2,"attrs":{"eta":-163.63636363636363}},{"id":3,"attrs":{"eta":-155.45454545454544}},{"id":4,"attrs":{"eta":-147.27272727272728}},{"id":5,"attrs":{"eta":-139.0909090909091}},{"id":6,"attrs":{"eta":-130.9090909090909}},{"id":7,"attrs":{"eta":-122.72727272727272}},{"id":8,"attrs":{"eta":-114.54545454545455}},{"id":9,"attrs":{"eta":-106.36363636363636}},{"id":11,"attrs":{"eta":-98.18181818181819}},{"id":10,"attrs":{"eta":-90}},{"id":12,"attrs":{"eta":-81.81818181818181}},{"id":13,"attrs":{"eta":-73.63636363636364}},{"id":14,"attrs":{"eta":-65.45454545454545}},{"id":15,"attrs":{"eta":-57.27272727272728}},{"id":16,"attrs":{"eta":-49.09090909090909}},{"id":17,"attrs":{"eta":-40.90909090909091}},{"id":18,"attrs":{"eta":-32.72727272727272}},{"id":19,"attrs":{"eta":-24.54545454545456}},{"id":21,"attrs":{"eta":-8.181818181818187}},{"id":22,"attrs":{"eta":0}}]}}

delete calibration record by label...

[wernersun]

Looks great! We had discussed that the IOV schema does not need "till" because that's determined by "since" of the next record in the time sequence. Otherwise, when a user creates a new calibration with an unknown "till", they have to assign an artificial infinite "till", and the service has to close out the previous IOV.

On the flip side, without a "till", records are not self-contained, and retrieval becomes more complicated.

[vkuznet]

I dropped the "till" in schema and APIs, and updated README. Please review and try it out.

[vkuznet]

@keara-soloway , feel free to test and provide your observations in this ticket.