Workflow to run ingest.py

[briantoby]

This is not yet working (see https://github.com/briantoby/Query-GSAS/actions/runs/28973420246/job/85974517585) but shows what I'd like to get working as a workflow and opening a PR gives you access.

I see that there are nearly duplicate files between the main directory and the gsas_query directory and it seems that files are being used from both places in ingest.py. Probably best to clean this up.

[pawantr]

Thanks Brian — your diagnosis was exactly right. I've pushed a fix to main that addresses both issues:

Duplicate files — ingest.py, rag.py, gui.py, and sources.py in the root are now thin stubs that delegate to gsas_query.*. All the logic lives in one place.

CI workflow — I've added .github/workflows/ingest.yml to main. The exit 143 was the runner being OOM-killed by the CUDA PyTorch wheel (~2.5 GB). The fix is installing CPU-only torch first:

pip install torch --index-url https://download.pytorch.org/whl/cpu

The workflow also sets TOKENIZERS_PARALLELISM=false / OMP_NUM_THREADS=1, uploads the built chroma_db as a 90-day artifact, and only runs when sources.py or ingest.py change.

You can close this PR — the workflow is already in main. Feel free to pull and let me know if the local python ingest.py kill is also fixed (the same env vars are now set at the very top of the stub).

[briantoby]

Thanks for what you have done. Alas, I was not able to run the ingest.py process either from a GH action or from a terminal window. Please let me know if I should be doing something different.

[pawantr]

On the terminal kill: The root ingest.py is now a thin stub that delegates to the gsas_query package, so you need the package installed in your conda env before running it. From the repo root:

pip install -e .          # install package in editable mode (once)
python ingest.py          # or: gsas-query --setup --html-only

If you prefer not to install from source, the package is now on PyPI:

pip install gsas-query
gsas-query --setup --html-only

The Python 3.13 SIGKILL fix (setting TOKENIZERS_PARALLELISM=false before any imports) is in place, but you need to pull the latest main first.

On the CI runner shutdowns: The runner is being killed by OOM — even with CPU-only torch (~400 MB), loading PyTorch + sentence-transformers + ChromaDB + processing 129 HTML pages uses more than the 7 GB available on ubuntu-latest. I'm looking at switching to ChromaDB's built-in ONNX embedding function which skips PyTorch entirely (~200 MB vs ~2 GB). That will fix CI. Will push the change shortly.

[pawantr]

Update (v0.2.1): I've dropped PyTorch and sentence-transformers entirely. Embeddings now use ChromaDB's built-in ONNX runtime (~80 MB model download, no loky, no multiprocessing issues). This should fix the terminal kill.

Steps to try now:

# Pull latest and install from PyPI (or -e . from the repo)
git pull
pip install --upgrade gsas-query   # installs v0.2.1

# Build the index (first run downloads the 80 MB ONNX model)
gsas-query --setup --html-only

# Verify it worked
gsas-query --stats

If you still see a kill, please share the full terminal output — it may be a plain OOM (model + chromadb loading ~1 GB total). In that case we can try ulimit -v to check available virtual memory.

On the CI ingest workflow: The auto-trigger has been disabled — chromadb pulls in grpcio, kubernetes, onnxruntime and the combined footprint OOMs the 7 GB ubuntu-latest runner. The workflow now only runs on manual dispatch. For building a distributable index, a self-hosted runner (or a local machine) is the right place.

[pawantr]

Update (v0.2.2) — root cause of the terminal kill found and fixed

The process was hanging indefinitely (not crashing) due to a bug in chunk_text: after processing the last chunk, start = end - overlap backed up into already-processed text, creating an infinite loop. On macOS the process would eventually be killed by the OS after several minutes. This happened only for sections longer than 1200 characters, which is why short test pages appeared to work.

A second bug caused a DuplicateIDError crash if any source had two sections with the same heading — fixed by switching to content-hash IDs.

Verified end-to-end on Python 3.13: full ingest completes in ~3 minutes, 1747 chunks indexed from 84 HTML sources.

pip install --upgrade gsas-query   # installs v0.2.2
gsas-query --setup --reset --html-only
gsas-query --stats

The --reset flag drops any partial index from previous attempts. The first run downloads the ~80 MB ONNX embedding model automatically.