Support Big Numbers

[brimworks]

Since all numbers are represented as double, larger numbers (like longs) get truncated. For example, the output of this code...

cjson.encode(cjson.decode("9007199254740992"))```

...is not the same as the input. This is needed since I need to sometimes parse the JSON output of APIs that I do not control and that use numbers that do not fit into doubles.

A proposed fix is in this pull request:

https://github.com/mpx/lua-cjson/pull/36

[mpx]

..continuing the discussion from #36 here..

@brimworks, thanks for providing the motivating example of an API that returns larger numbers as an opaque ID field. Unfortunately it shows one of the limitations of JSON for serialising data. The RFC (https://tools.ietf.org/html/rfc7159#section-6) notes it's best to stick within 64-bit floats for interoperability. However, upstream APIs exist that break this..

I'm still in favour of leaving the number field a number to avoid confusion and placing an undue burden on all developers to check for this occurrence.

However, we could optionally add a metatable to the number like: { __jsonvalue = "12345678901234567890.1234567890" }

This has several advantages:

• The decoded number is always available as a usable number within Lua.
• We can quickly check the parsed number length in bytes to determine if there is a risk of precision loss. Eg, more digits than guaranteed to fit within a 64-bit float.
• Applications can choose to use the exact value when and where it is needed.

In future this __jsonvalue field could be extended to allow arbitrary JSON to be substituted during encoding for a value.

As mentioned on the pull request, if there is a need for arbitrary precision in future it should be relatively straight forward to link in support and replace the existing number encode/decode support. However, the dependency should only be added if someone has a use case that requires it.

[mpx]

Erk.. wasn't thinking - numbers don't have individual metatables..

Back to:

1. Runtime option to return JSON numbers as strings for application to handle.

• Pro: Numeric methods still work due to automatic type conversion
• Pro: Simple
• Con: Doesn't round-trip

2. Runtime option to return a table with the original string value for every number. Use a metatable to describe the special type and provide methods.

• Pro: Numeric methods still work due to the metatable.
• Pro: Can round trip using array/object metatable solution for encoding
• Con: Complex: table per number, requires metatable methods, slower

3. Compile time option to support arbitrary precision library

• Pro: Numbers can be decoded/used/encoded without loss of precision
• Con: Requires external dependency

Despite the complexity, option 2 is probably best since the lack of reliable round-trip support is typically painful (eg, arrays vs objects).

[brimworks]

What about option (4):

Add a new option that allows numbers to be transformed from string into whatever type is desired? This means consumers of CJSON can choose option (1), (2), or (3) simply by defining a transform function as they see fit.

Thanks,
-Brian

[smartynov]

Upvoting this issue, as I suffer from data distortion too. For example, Viber API uses big numbers: "message_token":5124645880214536549.

It seems that initially proposed approach (just decode as strings) is the best. Anyway, one can manipulate this string in any fashion. Hope it will be resolved.

[imolein]

While learning programming I tried to solve a coding challenge, where I had to send a big number to an API and ran into this problem, because lua-requests depends on lua-cjson. I hope it will be fixed in the future :)

[frank0r]

same issue, found another solution

https://forums.solar2d.com/t/precision-lost-when-do-json-decode-a-very-large-integer-number/345152/5

http://regex.info/blog/lua/json

treat big number as table