A 1.4 MB ods grows past 1.2 GB in Html::translate and the OS kills the app

[andiwand]

odr-public/odr-private corpus file odr-private/ods/efficiency-big-1.ods (1,401,603 bytes) takes the whole process down while translating. Found sweeping the corpus for the OpenDocument.droid 6.11.0 bump; it is not a 6.11.0 regression — 6.10.1 behaves the same.

What happens

Resident memory of the app process, sampled every 0.3 s from /proc/<pid>/status on a 2 GB device with 819 MB available at the start:

RSS	device MemAvailable
99 MB	819 MB
218 MB	791 MB
439 MB	579 MB
863 MB	155 MB
1071 MB	0
1188 MB	12 MB
killed by lmkd

Roughly 850x the file size, still climbing when it died, so 1.19 GB is a floor rather than the requirement. lowmemorykiller takes it — there is no OutOfMemoryError, and the JVM heap growth limit on this device is 192 MB, so this is native allocation.

It dies after the file type is known and before any view exists, i.e. inside Html::translate.

The html limits do not bound it

HtmlConfig::spreadsheet_limit and spreadsheet_cell_limit bound what is written, not what is read to get there, so no setting of them helps:

rows / cells	result
100000 / 500000 (6.11.0 defaults)	killed
10000 / unset (the 6.10.x shape)	killed
100 / 2000	killed

The last row is the interesting one: a 100-row, 2000-cell budget still costs over a gigabyte, which says the cost is in parsing the sheet rather than emitting it.

Reproducing

Any host will do; this was an Android app calling Html.translate through the JNI bindings.

Odr.open("efficiency-big-1.ods") -> ok
Html.translate(file, cache, config)  // config limits irrelevant, see above

Versions

• 6.11.0 (3fa5603c) and 6.10.1, identical behaviour
• Android 12 emulator, x86_64, 2 GB RAM

Note

A second corpus file, odr-private/ods/efficiency+style-big-1.ods (33,038 bytes), survives on its own but was killed when opened after other large documents in the same sweep — consistent with the same allocation pattern at a smaller scale.

[andiwand]

Reproduced, and profiled

cli/translate on odr-private/ods/efficiency-big-1.ods, RelWithDebInfo, macOS (arm64):

4.11 real   peak memory footprint 1,544,291,840   (1.54 GB)

Same shape as on the device: all of it is spent before a single byte of HTML is written.

Why the file is a bomb

efficiency-big-1.ods  1,401,603 bytes zipped
  └─ content.xml     297,442,866 bytes  ← 212:1

10 sheets; one of them has 1,048,576 rows. In total 1,049,713 <table:table-row>,
3,169,072 <table:table-cell>, 1,053,010 <text:p>. Every row is literally the same
three cells:

<table:table-row table:style-name="ro1" table:visibility="filter">
  <table:table-cell table:number-columns-repeated="13"/>
  <table:table-cell office:value-type="string"><text:p>Genius</text:p></table:table-cell>
  <table:table-cell table:number-columns-repeated="1010"/>
</table:table-row>

The rows are not number-rows-repeated-compressed, so this really is a million row
elements. That makes the 850x-of-file-size figure misleading: the real ratio is
~5x the uncompressed XML, and the XML is the bomb.

Where the memory goes

Measured with phys_footprint (the metric jetsam/lmkd actually uses — plain RSS lies on
macOS because of the memory compressor) at each phase of the odf::Document constructor,
plus differential builds that switch off individual pieces of parse_sheet. The numbers
repeat to 0.1 MB across runs.

stage	delta	cumulative
inflate content.xml into the malloc buffer (xml::parse(const File &))	285 MB	285 MB
pugixml DOM over it — kept alive for the document's lifetime	593 MB	878 MB
table_rows() — materialises a vector<xml_node> of all 1.05M rows	18 MB	896 MB
Sheet::rows / Row::cells — the std::map interval index	275 MB	1171 MB
element registry: 1.05M sheet cells + 1.05M paragraphs + 1.05M texts	490 MB	~1.6 GB

The registry ends up holding 3,161,846 elements. HtmlConfig's limits never enter this:
they are applied in sheet_rendered_extent() (internal/html/document_element.cpp:82),
long after odf::Document's constructor has built the whole thing — which is exactly the
effect the issue describes.

What can be avoided

1. pugixml compact mode — 593 MB → 115 MB. Measured directly on the extracted
content.xml, same buffer, same parse call:

normal    buffer 284.8 MB → after parse 877.6 MB     0.30s
compact   buffer 284.8 MB → after parse 399.5 MB     0.19s

xml_node_struct goes 64 → ~20 bytes, xml_attribute_struct 40 → ~12. Not slower to
parse; traversal pays a small pointer-decode cost. Catch: PUGIXML_COMPACT is
ABI-affecting and the conan recipe for pugixml/1.15 exposes no such option, so it needs
header_only=True plus a global define, or a patched recipe. Biggest single win by far,
and it helps every format that goes through util::xml.

2. std::map → sorted vector for the sheet index — ~275 MB → ~120 MB. Per cell today:
a 32 B rb-tree node plus 24 B payload, rounded up to a 64 B malloc chunk, for 12 bytes of
data. util::map::lookup_greater_than is just upper_bound, so a sorted
vector<pair<uint32_t, Cell>> is a drop-in. One wrinkle: the keys are monotone except
that the row loop registers the whole run and then re-registers each repeat individually
(odf_parser.cpp:175 vs :207); the first call is redundant whenever the per-repeat path
runs, so dropping it makes the sequence purely append-only. Self-contained in
odf_element_registry.*.

3. ElementIdentifier is std::uint64_t (definitions.hpp:7).
ElementRegistry::Element is 56 bytes, 40 of which are five ids. As uint32_t it drops to
32 bytes → −75 MB on m_elements alone here, plus smaller keys in every side map. It is
public API, so the bindings need a look first.

4. m_elements growth slack. At 3.16M elements the vector's capacity is 4.19M
(235 MB holding 177 MB of data), and the abandoned 117 MB block is still sitting in
malloc's cache. Chunked storage removes both the slack and the realloc spike — and is
safer anyway, since create_element hands out an Element & over a vector that
reallocates.

5. The structural one: an element per cell. Each non-empty cell costs three registry
elements (cell + paragraph + text) even though the renderer will emit at most
spreadsheet_cell_limit of them. Either bound the parse — which means the limits have to
reach decode time, not just HtmlConfig — or parse cell content lazily: Sheet::Cell
already holds the pugi::xml_node, so the children can be built on first access. −490 MB
here.

6. Minor. sheet_content() (odf_document.cpp:488) re-walks the entire sheet and
allocates a fresh 8 MB table_rows() vector on every call, and sheet_rendered_extent() +
sheet_cut() call it twice per sheet. Time rather than peak, but it is a third full pass
over a million rows.

The 285 MB source buffer is irreducible while the DOM points into it in place
(load_buffer_inplace_own); only a streaming parse would remove that.

Expected outcome

• 1 + 2 alone, no architectural change: ~1.54 GB → ~870 MB
• 1 + 2 + 5: 285 (buffer) + 115 (compact DOM) + ~120 (vector index) ≈ 520 MB, which
  survives the 2 GB device.

[andiwand]

#776 and #780 take the measured ladder: 1667 MB peak on efficiency-big-1.ods to 914 MB to 626 MB, rendered output unchanged throughout.

What is left is filed separately, being different work: #781 (a non-empty cell still builds three elements before any html limit applies, ~85 MB) and #782 (the 284 MB source buffer, which needs a streaming parser).