Result for semantic search with query "[[HasLabel::BMD_StandardCU_V3]]" delivers random results

[Masltov]

semantic search performed on query "[[HasLabel::BMD_StandardCU_V3]]" either finds correct Item (Item:OSWb1d840f8f9df4e74b9a0e2f7d2977269) or results in "Query '[[HasLabel::BMD_StandardCU_V3]]|limit=1000' returned no results"
No specific order of results but seems rather random.

Maybe linked to default limit of 1000 (results? entries considered in OSLab?)?

[simontaurus]

Reference API request (now) returns stable result:

https://healthbatt.projects01.open-semantic-lab.org/w/api.php?action=ask&query=[[HasLabel::BMD_StandardCU_V3]]|limit=1000&format=json

Probably a data rebuild was running, leading to inconsistencies.

[LukasGold]

The "random results" reproduce on healthbatt.projects01.open-semantic-lab.org and are not a data rebuild. Two causes, both measured.

1. exists flickers server-side

semantic_search only keeps results whose exists field equals "1" (https://github.com/OpenSemanticLab/osw-python/blob/main/src/osw/wiki_tools.py#L263). The field only ever takes '1' or '', but it is not stable. Eight identical [[Category:Item]]|limit=1000 calls:

call 0: raw=1000  exists!=1 -> 6
call 1: raw=1000  exists!=1 -> 2
call 2: raw=1000  exists!=1 -> 6
call 3: raw=1000  exists!=1 -> 6
call 4: raw=1000  exists!=1 -> 6
call 5: raw=1000  exists!=1 -> 2
call 6: raw=1000  exists!=1 -> 2
call 7: raw=1000  exists!=1 -> 6

distinct `exists` values seen: {"'1'": 7964, "''": 36}
pages reported non-existing in SOME call:  8
pages reported non-existing in EVERY call: 0

No page is consistently '', and the flips are perfectly correlated across all 8 pages (calls 0,2,3,4,7 agree; calls 1,5,6 agree). That bimodal pattern points at inconsistent state between backends or an SMW query-result cache, not at real deletions. Consecutive identical calls to semantic_search returned 998 or 994 titles.

This needs fixing on the OSL instance side. The library can only make it visible.

2. Silent truncation at the default limit

The guess in the original report about the 1000 limit was right. SearchParam.limit defaults to 1000 and there is no pagination, so a larger match set is silently reduced to an arbitrary subset:

[[Category:Item]] via |offset= pagination -> 2637 unique pages
[[Category:Item]] via semantic_search     -> 1000 (no warning)

Not a cause: sort order

I checked whether adding |sort=|order=asc stabilises the truncated subset. It does not - the result hash still varies across identical calls, identically to the unsorted control. Appending a sort clause would also silently override a caller's own sort, since SMW honours the last one. So that is not a viable workaround.

What changed

#150 adds a warning when a query hits its limit and a warning naming how many results were dropped as non-existing, so neither failure mode is silent any more. It deliberately does not change the filter semantics or the default limit.

Reopening is probably warranted for cause 1, since it is still live.