Skip to content

faircode profile data.json silently misparses split-orient JSON that omits the index key, instead of raising or falling back #317

Description

@yakew7

Where: faircode/loaders_extra.py:36-51.

README.md (around line 928) documents that faircode profile data.json supports "JSON records (or split-orient)". The split-orient fallback is:

try:
    return pd.read_json(path)
except ValueError:
    return pd.read_json(path, orient="split")

The bug: for a minimal, natural split-format file that omits the optional index key — e.g. {"columns": ["sex","age"], "data": [["M",30],["F",25]]} (exactly the shape assets/profiler-engine.js's parseJSON() documents and accepts in its own comment: "2. Split format: { columns: [...], data: [...] }") — pd.read_json(path) (the default-orientation attempt) does not raise a ValueError, so the orient="split" fallback never runs.

Repro:

import pandas as pd, json, tempfile
data = {"columns": ["sex","age"], "data": [["M",30],["F",25]]}
# write to a .json temp file, then:
pd.read_json(path)
#   columns     data
# 0     sex  [M, 30]
# 1     age  [F, 25]

This returns a 2-row DataFrame with columns literally named ["columns", "data"]" and list-valued cells — silently wrong, no error, no warning — while the JS engine correctly parses the identical input into {columns:["sex","age"], rows:[{sex:"M",age:30},{sex:"F",age:25}]}`.

tests/test_loaders.py::test_read_table_json_orientations only covers DataFrame.to_json(orient="split") output, which happens to include an extra `"index"" key that does trigger the ValueError fallback — so this specific, documented-as-supported shape is untested.

Suggested fix: detect split-orient by checking for {"columns", "data"} (optionally "index") keys in the parsed JSON up front, rather than relying on pd.read_json's default-orientation guess to fail.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions