LLM log analyzer: the documented provider contract does not match the code in three places 🤖🤖🤖

[kev365]

Summary

docs/developers/log-analyzer-agent.md invites you to "experiment with your own log
reasoning agent", and the API contract it documents diverges from what
log_analyzer.py actually accepts in three ways. Each fails silently or with a confusing
error rather than telling you what is wrong.

All three were reproduced on a live instance with a minimal stub provider.

#	Documented / implied	Actually required	Symptom when wrong
1	summaries wrapper key (docstrings, test name)	findings	0 findings, status success
2	priority "can be low, medium, or high"	notice or critical	No label set, no error
3	—	SUPPORTS_STREAMING must be declared	AttributeError

Filed together because they are all on the same "write your own provider" path. Happy to
split if preferred.

---

1. The response wrapper key is findings, not summaries

log_analyzer.py:170:

findings_list = response_data.get("findings", [])

But summaries appears in the places a provider author reads:

• class docstring, line 46: "expected to be an object with a summaries key"
• execute docstring, lines 89 and 92
• the unit test is named test_execute_with_summaries_format and its docstring says
  "with a 'summaries' key" — while its payload correctly uses findings
• docs/developers/log-analyzer-agent.md documents the per-finding schema but never names
  the wrapper key, and describes a **JSON Summary of Findings** markdown marker that the
  code does not look for

Observed: emitting summaries produces

LogAnalyzer: JSON is valid, but the findings list is empty.

and returns status: "success" with "The AI provider returned a valid response but did not
identify any specific findings." No questions are created. Changing only that one key to
findings creates the question.

The reference provider secgemini_log_analyzer_agent.py emits findings, so code, test
payload and reference implementation agree — only the prose disagrees.

---

2. priority accepts notice/critical, not low/medium/high

The docs table states priority "Can be low, medium, or high".

log_analyzer.py:369-372:

LLM_PRIORITY_TO_TS_PRIORITY = {
    "notice": "low",
    "critical": "high",
}

Observed, two runs differing only in this value:

emitted priority	labels on the resulting question
"critical"	['__ts_priority_high']
"high"	[]

"high" is silently ignored — no label, no warning, no error. medium is unreachable from
provider output entirely.

---

3. SUPPORTS_STREAMING is not defined on the base provider class

log_analyzer.py:117:

if not llm_provider.SUPPORTS_STREAMING:
    raise ValueError(f'LLM provider "{llm_provider.NAME}" does not support streaming operations!')

The intent is clearly a readable error for a non-streaming provider, but
SUPPORTS_STREAMING is not defined on LLMProvider
(providers/interface.py) —
only secgemini_log_analyzer_agent declares it. Any provider that omits it, including the
shipped ollama, vertexai, aistudio and azureai, hits an attribute error instead.

Observed:

AttributeError: 'ReproNoStreamingFlag' object has no attribute 'SUPPORTS_STREAMING'

---

Reproduction

One stub provider covers all three; behaviour is driven from config so nothing but
timesketch.conf changes between cases. Save as
timesketch/lib/llms/providers/repro_provider.py and import it in
providers/__init__.py:

import json
from timesketch.lib.llms.providers import interface, manager


class ReproLogAnalyzer(interface.LLMProvider):
    NAME = "repro_log_analyzer"
    SUPPORTS_STREAMING = True

    def generate_stream_from_logs(self, log_events_generator, prompt=None, sketch=None):
        for _ in log_events_generator:
            pass
        yield json.dumps({
            self.config.get("wrapper_key", "findings"): [{
                "log_records": [{"record_id": self.config["record_id"]}],
                "annotations": [{
                    "investigative_question": "Repro " + self.config.get("wrapper_key", ""),
                    "conclusions": ["Repro conclusion"],
                    "priority": self.config.get("priority", "critical"),
                }],
            }],
            "report_summary": "repro",
        })

    def generate(self, prompt, response_schema=None):
        raise NotImplementedError()


class ReproNoStreamingFlag(interface.LLMProvider):
    """Deliberately omits SUPPORTS_STREAMING."""

    NAME = "repro_no_streaming_flag"

    def generate_stream_from_logs(self, log_events_generator, prompt=None, sketch=None):
        yield json.dumps({"findings": []})

    def generate(self, prompt, response_schema=None):
        raise NotImplementedError()


manager.LLMManager.register_provider(ReproLogAnalyzer)
manager.LLMManager.register_provider(ReproNoStreamingFlag)

Set one case in timesketch.conf, restart the Celery worker so it picks the config up, then
run the llm_log_analyzer analyzer:

# case 1  -> 0 questions, status success
{"repro_log_analyzer": {"wrapper_key": "summaries", "priority": "critical", "record_id": "<_id>"}}
# case 1b -> 1 question created
{"repro_log_analyzer": {"wrapper_key": "findings",  "priority": "critical", "record_id": "<_id>"}}
# case 2  -> question created with no priority label
{"repro_log_analyzer": {"wrapper_key": "findings",  "priority": "high",     "record_id": "<_id>"}}
# case 3  -> AttributeError
{"repro_no_streaming_flag": {}}

Two things that cost me time and may cost a reviewer the same:

• the Celery worker reads timesketch.conf at startup, so it must be restarted between
  cases or the previous provider is silently reused;
• tasks.py:620-636 skips an analyzer that already completed on the timeline with the same
  kwargs hash, so pass a distinct prompt kwarg per run or the second and later runs do not
  execute at all.

Note that every one of these runs also hits the unrelated session_id UnboundLocalError
in #3890, which reports the analyzer as ERROR regardless of outcome — check the
created questions rather than the analyzer status when reproducing.

Suggested adjustments

1. Correct the three docstrings and the test name to findings, and name the wrapper key in
   the developer docs' API contract.
2. Either correct the docs' priority values to notice/critical, or widen
   LLM_PRIORITY_TO_TS_PRIORITY to accept the documented ones. Accepting both seems kindest
   to provider authors.
3. Add SUPPORTS_STREAMING = False to the base LLMProvider so the existing ValueError
   is reachable.

All three are small and I am happy to send PRs for whichever are wanted.

Environment

Timesketch master @ 0c45e475, docker/dev stack, OpenSearch 3.7.0, Python 3.14.4,
Ubuntu 26.04.

Note

Prepared with AI assistance, hence the 🤖🤖🤖 per CONTRIBUTING.md.