fix: report the real cluster error instead of "check LOG"

[tabedzki]

🤖 Human guided, AI assisted PR (using this skill). AI text below. 🤖

Problem

When a slurm job failed, both the recording-process pipeline and the pupillometry pipeline set the error message to An error occured in processing (check LOG). The slack notification therefore said nothing about what went wrong, and the log it referred to was never named — finding the failure meant manually locating the right job_id_<n>.log.

What changed

The error log is now parsed and the actual error is reported:

error message : Job failed - MemoryError: Unable to allocate 30.0 GiB for an array (LOG: /logs/job_id_1234.log)

• summarize_error_log() picks the most informative line of the log: the last line matching an error pattern (*Error/*Exception, failed, no such file, out of memory, time limit, …), falling back to the last line. Python/MATLAB tracebacks end with the exception; Kilosort writes plain messages to stderr and keeps logging afterwards, so the last line alone is not enough.
• build_error_message() assembles <slurm status> - <error from log> (LOG: <path>), sized to fit the 255-char DB column with the path intact.
• Both live in u19_pipeline/utils/file_utils.py (they only need re), so they are importable and testable without the pipeline extra.
• The log-path logic inside transfer_log_file() is factored into get_log_file_local_path() / get_log_file_cluster_path(), so the reported path is always the same file that was transferred. No behavior change there.
• If the log could not be copied from the cluster or is empty, the message says so and points at the log on the cluster (user@host:path) rather than a local path that does not exist. error_exception gets that note too, instead of being blank.
• check_processed_pupillometry_sessions() reported no log at all (only the slurm state message). It now copies ErrorLog/job_id_<video stem>.log from spock and reports it the same way.

Tests

tests/automatic_job/test_error_log_reporting.py covers the summary (traceback, trailing noise after the error line, no error keyword, empty log, over-long line, the space-joined form get_error_log_str produces) and the message building (path always present, 255-char cap, over-long path cropped from the left keeping the job id).

uv run pytest tests/automatic_job -q

10 passed. The pipeline paths themselves (scp of the log, slurm status) were not exercised — that needs DB and cluster access.

Assisted-by: ClaudeCode:claude-opus-5

🤖 Generated with Claude Code