CSV/JSONL import: field-mapping limit is under-counted, and a timeline can report "ready" with zero events 🤖🤖🤖

[kev365]

Summary

run_csv_jsonl sizes the OpenSearch field-mapping limit as len(event.keys()) * 2 * 1.1
and refuses imports above OPENSEARCH_MAPPING_UPPER_LIMIT. For two independent reasons that
estimate is low — by about 1.4x on flat data and 1.9x in the nested example below —
and the limit it computes is never applied at all. The practical result is a window of field
counts where the import is accepted, the timeline reports status ready, and no events are
indexed.

Measured empirically: a 330-field JSONL file imports; a 340-field file produces a ready
timeline containing 0 of 5 events.

The arithmetic was correct when it was added in #3257; it drifted when #3825 added a second
sub-field to every string. Details below.

This came up while importing wide cloud audit logs. It is easy to hit with any source that
has a few hundred fields or makes heavy use of nested objects — everything below reproduces
with synthetic data.

Environment

• Timesketch master @ 0c45e475
• docker/dev stack, Python 3.14.4, Ubuntu 26.04
• Default config: OPENSEARCH_MAPPING_UPPER_LIMIT = 1000, OPENSEARCH_MAPPING_BUFFER = 0.1

The OpenSearch-side behaviour was checked on both the 3.7.0 used by docker/dev and the
2.19.5 used by the e2e workflow, with identical results:

	OpenSearch 2.19.5	OpenSearch 3.7.0
default index.mapping.total_fields.limit	1000	1000
"type" declarations in _mapping for 100 string fields under generic.mappings	304	304
100 string fields into an index with total_fields.limit: 100	rejected — Limit of total fields [100] has been exceeded	rejected — same
control: same document with total_fields.limit: 1000	accepted	accepted

So multi-fields count toward the limit on both versions.

Reproduction

# 5 rows, N distinct string fields plus message/datetime/timestamp_desc/data_type
import json
N = 340
with open(f"/tmp/wide_{N}.jsonl", "w") as fh:
    for row in range(5):
        rec = {"message": f"row {row}", "datetime": "2026-01-15T12:00:00+00:00",
               "timestamp_desc": "Recorded Time", "data_type": "test:wide:entry"}
        for i in range(N - 4):
            rec[f"pad_field_{i:04d}"] = f"value_{i:04d}_{row}"
        fh.write(json.dumps(rec) + "\n")

timesketch_importer --host http://127.0.0.1:5000 --username dev --password dev \
    --sketch_name w340 --timeline_name wide-340 /tmp/wide_340.jsonl

Observed, per field count:

unique keys	new_limit computed	import accepted	events indexed	timeline status
250	550	yes	5 / 5	ready
330	726	yes	5 / 5	ready
340	748	yes	0 / 5	ready
400	880	yes	0 / 5	ready
602	1324	no — refused by the pre-check	—	fail

The two failure modes report very differently. The 602-field file is refused up front and
says exactly what to do:

timeline refuse-600  status=fail
  Error: Indexing timeline [refuse-600] into [...] exceeds the upper field mapping
  limit of 1000. New calculated mapping limit: 1324. Review your import data or
  adjust OPENSEARCH_MAPPING_UPPER_LIMIT.

The 340- and 400-field files are accepted, report ready, and index nothing:

timeline wide-340  status=ready  docs=0
  0 out of 5 events imported. Most common error type is "illegal_argument_exception"

The second message names neither the field limit nor OPENSEARCH_MAPPING_UPPER_LIMIT, and
the timeline status gives no indication anything went wrong.

Why the estimate is low

1. Each string field costs three mapping entries, not two — this changed under the code.

The * 2 was correct when it was written. In #3257 (Jan 2025)
data/generic.mappings carried no dynamic template, so strings took OpenSearch's default
mapping of text plus .keyword — two entries.

#3825 (Jun 2026, "Native Wildcard Field Search Support") added a dynamic template giving
every string two sub-fields, keyword and wildcard. That makes it three entries per
string field, and the arithmetic in tasks.py was not updated to match. The comment at
tasks.py:1243-1245
still reads "Each unique key is counted twice due to the keyword type", which describes
the pre-#3825 mapping.

So this is drift between two changes rather than a fault in either one. OpenSearch counts
multi-fields toward index.mapping.total_fields.limit (verified on 2.19.5 and 3.7.0, above).

Measured directly from _mapping:

source keys	properties	multi-field sub-fields	total entries	entries per key
250	253	496	749	3.0
400	402	796	1198	3.0

2. Nested objects are not counted.

tasks.py:1242
uses unique_keys.update(event.keys()), which sees only top-level keys. Dynamic mapping
then expands nested objects into additional leaf paths that were never budgeted.

A single document with 10 top-level keys, one of them an object with 5 leaves:

doc = {"message": "nested demo", "datetime": "2026-01-15T12:00:00+00:00",
       "timestamp_desc": "Recorded Time", "data_type": "test:nested"}
for i in range(5):
    doc[f"flat_{i}"] = "v"
doc["permissions"] = {f"leaf_{i}": "read" for i in range(5)}

len(event.keys()) as tasks.py counts	10
properties OpenSearch mapped	15
multi-field sub-fields	26
total mapping entries	41
budget int(10 * 2 * 1.1)	22

So the budget is under by 1.9x on this shape. Sources that nest permission sets,
configuration blocks or similar will diverge much faster than the flat case above.

The computed limit is never applied

On a fresh index there is no explicit total_fields setting, so
tasks.py:1234-1235
falls to except KeyError: current_limit = 1000. The raise is then guarded by:

if new_limit > current_limit and current_limit < upper_mapping_limit:

With current_limit == upper_mapping_limit == 1000, the second clause is always false, and
the first requires new_limit > 1000 — which the check two lines above has already
rejected. So put_settings is never reached.

Confirmed by measurement: index.mapping was null on every index created this way
(the 250-, 330-, 340- and 400-field imports above, and a normal ~55-field import). The
effective ceiling is always OpenSearch's own default of 1000 entries, so the dynamic increase
added in #3257 does not engage in a default configuration.

Net effect

• True ceiling ≈ 333 string fields (333 × 3 = 999)
• Pre-check only objects at 455 fields
• ≈334–454 fields: accepted, ready, zero events indexed

On the reporting of that failure: _set_datasource_status is called with fail only on the
pre-check path. When OpenSearch rejects the documents instead, the timeline's own status is
ready and it appears in the sketch's timeline list alongside successful ones; the error
text is reachable only via the data source's error_message. Confirmed via the API:

timeline wide-340  status=ready  docs=0
  data source error_message: 0 out of 5 events imported.
      Most common error type is "illegal_argument_exception"

The error_message also does not carry the underlying OpenSearch reason
(Limit of total fields [1000] has been exceeded), which is what identifies the cause.

Raising the existing config does not resolve it

The obvious first response is "raise OPENSEARCH_MAPPING_UPPER_LIMIT". I tested that —
set to 5000, worker restarted, same two files re-imported:

file	unique keys	new_limit	limit actually set on index	events indexed
wide-400	400	880	null — never applied	0 / 5
wide-600	602	1324	1324 — raise did fire	0 / 5

Two distinct failures:

• Below 1000, the raise is unreachable regardless of OPENSEARCH_MAPPING_UPPER_LIMIT,
  because it is gated on new_limit > current_limit and current_limit is OpenSearch's
  1000 default. Raising the config is a no-op here — this is the entire ~334–454 window.
• Above 1000, the raise fires, but sets a value derived from the same undercounting
  arithmetic. 602 keys need ~1800 entries; 1324 was applied; the import still indexed
  nothing.

There is also no supported way to raise OpenSearch's own total_fields.limit from
Timesketch config:
create_index
calls indices.create(index=index_name, body={"mappings": _document_mapping}) — only
mappings is passed, so data/generic.mappings and data/plaso.mappings cannot carry a
settings block. Short of applying a cluster-side index template out of band, an operator
has no knob that fixes this today.

That is why I think the arithmetic itself is worth adjusting rather than the ceiling.

Scope

This affects run_csv_jsonl only. run_plaso has no equivalent field-count check — it
calls create_index with plaso.mappings and hands off to psort, so it is subject to the
same OpenSearch limit but performs no counting and has no pre-check to be wrong. I have not
tested the plaso path and am not making a claim about it here.

Worth noting that data/plaso.mappings also gained the wildcard sub-field in the same
change: 20 of its 22 explicitly declared properties now carry both keyword and wildcard,
so its baseline cost is 62 mapping entries before any dynamic field is seen.

What I am asking for

The first two go together — (1) on its own would make things stricter, not better. (3) is
one possible way to satisfy (2) and is offered as an option rather than a separate ask; (4)
and (5) are minor.

1. Count what OpenSearch will actually map. Walk nested objects when collecting
unique_keys, and derive the per-field multiplier from the mappings file in use rather than
the constant 2.

Note this alone reduces what can be imported: the cutoff moves from 455 fields down to
about 303. That is a real improvement — a wide file would fail immediately with the existing
readable OPENSEARCH_MAPPING_UPPER_LIMIT message instead of silently indexing nothing — but
it does not make wide data importable.

2. Make the limit actually raisable, so correctly-counted wide data can be ingested. As
shown above, the current raise cannot fire below 1000, and create_index passes no
settings, so there is no path to a higher total_fields.limit today. Something like:
setting the index's total_fields.limit at creation from the computed estimate, and/or
changing the raise condition so it can move the limit up to upper_mapping_limit rather
than only above current_limit.

3. One option for (2): a per-import override. Offered as a suggestion, not a separate
request — if there is a better way to satisfy (2) I would rather follow that.

OPENSEARCH_MAPPING_UPPER_LIMIT is global config applied to every upload on the server, and
there is currently no way to raise it for a single known-wide dataset: run_csv_jsonl takes
no such parameter, and neither timesketch_importer, the import client, nor the upload API
resource expose one.

A per-import ceiling — a --max-fields style flag on the importer, plumbed through the
upload form to the task — would let an operator opt one dataset into a higher limit without
raising it for everyone. That keeps the conservative default for routine uploads, which is
where mapping-explosion risk actually lives. It also avoids having to pick a new global
default that is right for every deployment.

I appreciate the ceiling exists to protect against mapping explosion, and I am not asking
for it to be removed or for the default to be raised blindly. The ask is that the estimate be
accurate, and that the guard be adjustable — ideally per import.

Two smaller points, take or leave:

4. Surface the failure as a failure. If 0 events were indexed, the timeline arguably
should not be ready, and the data source error_message could carry the underlying
OpenSearch reason — the pre-check path already produces a message of exactly the quality
needed here.

5. Document the practical field ceiling for wide sources.

Happy to put a PR together for whichever of these is wanted — raising the measurements first
per CONTRIBUTING.md rather than arriving with a patch.

Note

The generator above is self-contained and reproduces the behaviour from scratch — no
external data is needed.

This issue was prepared with AI assistance, hence the 🤖🤖🤖 in the title per
CONTRIBUTING.md.