Commit b09cac05 authored by Philip Gladstone's avatar Philip Gladstone Committed by Johny Mattsson
Browse files

Add support for streaming JSON encoder/decoder (#1755)

Replaces the problematic cjson module.
parent b6ef1ffe
# CJSON Module # CJSON Module
| Since | Origin / Contributor | Maintainer | Source |
| :----- | :-------------------- | :---------- | :------ |
| 2015-03-16 | [Mark Pulford](http://kyne.com.au/~mark/software/lua-cjson.php), [Zeroday](https://github.com/funshine) | [Zeroday](https://github.com/funshine) | [cjson](../../../app/modules/cjson.c) |
The JSON support module. Allows encoding and decoding to/from JSON. This module has been replaced by [sjson](sjson.md). It provides a superset of functionality. All references to `cjson` can be replaced by `sjson`.
Please note that nested tables can require a lot of memory to encode. To catch out-of-memory errors, use `pcall()`.
## cjson.encode()
Encode a Lua table to a JSON string. For details see the [documentation of the original Lua library](http://kyne.com.au/~mark/software/lua-cjson-manual.html#encode).
####Syntax
`cjson.encode(table)`
####Parameters
`table` data to encode
While it also is possible to encode plain strings and numbers rather than a table, it is not particularly useful to do so.
####Returns
JSON string
####Example
```lua
ok, json = pcall(cjson.encode, {key="value"})
if ok then
print(json)
else
print("failed to encode!")
end
```
## cjson.decode()
Decode a JSON string to a Lua table. For details see the [documentation of the original Lua library](http://kyne.com.au/~mark/software/lua-cjson-manual.html#_decode).
####Syntax
`cjson.decode(str)`
####Parameters
`str` JSON string to decode
####Returns
Lua table representation of the JSON data
####Example
```lua
t = cjson.decode('{"key":"value"}')
for k,v in pairs(t) do print(k,v) end
```
# SJSON Module
| Since | Origin / Contributor | Maintainer | Source |
| :----- | :-------------------- | :---------- | :------ |
| 2017-02-01 | [Philip Gladstone](https://github.com/pjsg) | [Philip Gladstone](https://github.com/pjsg) | [sjson](../../../app/modules/sjson.c) |
The JSON support module. Allows encoding and decoding to/from JSON.
Please note that nested tables can require a lot of memory to encode. To catch out-of-memory errors, use `pcall()`.
This code using the streaming json library [jsonsl](https://github.com/mnunberg/jsonsl) to do the parsing of the string.
This module can be used in two ways. The simpler way is to use it as a direct drop-in for cjson (you can just do `_G.cjson = sjson`).
The more advanced approach is to use the streaming interface. This allows encoding and decoding of significantly larger objects.
The handling of json null is as follows:
- By default, the decoder represents null as sjson.NULL (which is a userdata object). This is the behavior of cjson.
- The encoder always converts any userdata object into null.
- Optionally, a single string can be specified in both the encoder and decoder. This string will be used in encoding/decoding to represent json null values. This string should not be used
anywhere else in your data structures. A suitable value might be `"\0"`.
When encoding a lua object, if a function is found, then it is invoked (with no arguments) and the (single) returned value is encoded in the place of the function.
## sjson.encoder()
This creates an encoder object that can convert a LUA object into a JSON encoded string.
####Syntax
`sjson.encoder(table [, opts])`
####Parameters
- `table` data to encode
- `opts` an optional table of options. The possible entries are:
- `depth` the maximum encoding depth needed to encode the table. The default is 20 which should be enough for nearly all situations.
- `null` the string value to treat as null.
####Returns
A `sjson.encoder` object.
## sjson.encoder:read
This gets a chunk of JSON encoded data.
####Syntax
`encoder:read([size])`
####Parameters
- `size` an optional value for the number of bytes to return. The default is 1024.
####Returns
A string of up to `size` bytes, or `nil` if the encoding is complete and all data has been returned.
#### Example
The following example prints out (in 64 byte chunks) a JSON encoded string containing the first 4k of every file in the file system. The total string
can be bigger than the total amount of memory on the NodeMCU.
```
function files()
result = {}
for k,v in pairs(file.list()) do
result[k] = function() return file.open(k):read(4096) end
end
return result
end
local encoder = sjson.encoder(files())
while true do
data = encoder:read(64)
if not data then
break
end
print(data)
end
```
## sjson.encode()
Encode a Lua table to a JSON string. This is a convenience method provided for backwards compatibility with `cjson`.
####Syntax
`sjson.encode(table [, opts])`
####Parameters
- `table` data to encode
- `opts` an optional table of options. The possible entries are:
- `depth` the maximum encoding depth needed to encode the table. The default is 20 which should be enough for nearly all situations.
- `null` the string value to treat as null.
####Returns
JSON string
####Example
```lua
ok, json = pcall(sjson.encode, {key="value"})
if ok then
print(json)
else
print("failed to encode!")
end
```
## sjson.decoder()
This makes a decoder object that can parse a JSON encoded string into a lua object. A metatable can be specified for all the newly created lua tables. This allows
you to handle each value as it is inserted into each table (by implementing the `__newindex` method).
####Syntax
`sjson.decoder([opts])`
#### Parameters
- `opts` an optional table of options. The possible entries are:
- `depth` the maximum encoding depth needed to encode the table. The default is 20 which should be enough for nearly all situations.
- `null` the string value to treat as null.
- `metatable` a table to use as the metatable for all the new tables in the returned object.
#### Returns
A `sjson.decoder` object
####Metatable
There are two principal methods that are invoked in the metatable (if it is present).
- `__newindex` this is the standard method invoked whenever a new table element is created.
- `checkpath` this is invoked (if defined) whenever a new table is created. It is invoked with two arguments:
- `table` this is the newly created table
- `path` this is a list of the keys from the root.
It must return `true` if this object is wanted in the result, or `false` otherwise.
For example, when decoding `{ "foo": [1, 2, []] }` the checkpath will be invoked as follows:
- `checkpath({}, {})` the `table` argument is the object that will correspond with the value of the JSON object.
- `checkpath({}, {"foo"})` the `table` argument is the object that will correspond with the value of the outer JSON array.
- `checkpath({}, {"foo", 3})` the `table` argument is the object that will correspond to the empty inner JSON array.
When the `checkpath` method is called, the metatable has already be associated with the new table. Thus the `checkpath` method can replace it
if desired. For example, if you are decoding `{ "foo": { "bar": [1,2,3,4], "cat": [5] } }` and, for some reason, you did not want to capture the
value of the `"bar"` key, then there are various ways to do this:
* In the `__newindex` metamethod, just check for the value of the key and skip the `rawset` if the key is `"bar"`. This only works if you want to skip all the
`"bar"` keys.
* In the `checkpath` method, if the path is `["foo"]`, then return `false`.
* Use the following `checkpath`: `checkpath=function(tab, path) tab['__json_path'] = path return true end` This will save the path in each constructed object. Now the `__newindex` method can perform more sophisticated filtering.
The reason for being able to filter is that it enables processing of very large JSON responses on a memory constrained platform. Many APIs return lots of information
which would exceed the memory budget of the platform. For example, `https://api.github.com/repos/nodemcu/nodemcu-firmware/contents` is over 13kB, and yet, if
you only need the `download_url` keys, then the total size is around 600B. This can be handled with a simple `__newindex` method.
## sjson.decoder:write
This provides more data to be parsed into the lua object.
####Syntax
`decoder:write(string)`
####Parameters
- `string` the next piece of JSON encoded data
####Returns
The constructed lua object or `nil` if the decode is not yet complete.
####Errors
If a parse error occurrs during this decode, then an error is thrown and the parse is aborted. The object cannot be used again.
## sjson.decoder:result
This gets the decoded lua object, or raises an error if the decode is not yet complete. This can be called multiple times and will return the
same object each time.
####Syntax
`decoder:result()`
####Errors
If the decode is not complete, then an error is thrown.
####Example
```
local decoder = sjson.decoder()
decoder:write("[10, 1")
decoder:write("1")
decoder:write(", \"foo\"]")
for k,v in pairs(decoder:result()) do
print (k, v)
end
```
The next example demonstrates the use of the metatable argument. In this case it just prints out the operations, but it could suppress the assignment
altogether if desired.
```
local decoder = sjson.decoder({metatable=
{__newindex=function(t,k,v) print("Setting '" .. k .. "' = '" .. tostring(v) .."'")
rawset(t,k,v) end}})
decoder:write('[1, 2, {"foo":"bar"}]')
```
## sjson.decode()
Decode a JSON string to a Lua table. This is a convenience method provided for backwards compatibility with `cjson`.
####Syntax
`sjson.decode(str[, opts])`
####Parameters
- `str` JSON string to decode
- `opts` an optional table of options. The possible entries are:
- `depth` the maximum encoding depth needed to encode the table. The default is 20 which should be enough for nearly all situations.
- `null` the string value to treat as null.
- `metatable` a table to use as the metatable for all the new tables in the returned object. See the metatable section in the description of `sjson.decoder()` above.
####Returns
Lua table representation of the JSON data
####Errors
If the string is not valid JSON, then an error is thrown.
####Example
```lua
t = sjson.decode('{"key":"value"}')
for k,v in pairs(t) do print(k,v) end
```
##Constants
There is one constant -- `sjson.NULL` -- which is used in lua structures to represent the presence of a JSON null.
...@@ -5,7 +5,7 @@ MEMORY ...@@ -5,7 +5,7 @@ MEMORY
dport0_0_seg : org = 0x3FF00000, len = 0x10 dport0_0_seg : org = 0x3FF00000, len = 0x10
dram0_0_seg : org = 0x3FFE8000, len = 0x14000 dram0_0_seg : org = 0x3FFE8000, len = 0x14000
iram1_0_seg : org = 0x40100000, len = 0x8000 iram1_0_seg : org = 0x40100000, len = 0x8000
irom0_0_seg : org = 0x40210000, len = 0xD0000 irom0_0_seg : org = 0x40210000, len = 0xE0000
} }
PHDRS PHDRS
......
...@@ -18,7 +18,7 @@ end ...@@ -18,7 +18,7 @@ end
-- payload(json): {"cmd":xxx,"content":xxx} -- payload(json): {"cmd":xxx,"content":xxx}
function topic1func(m,pl) function topic1func(m,pl)
print("get1: "..pl) print("get1: "..pl)
local pack = cjson.decode(pl) local pack = sjson.decode(pl)
if pack.content then if pack.content then
if pack.cmd == "open" then file.open(pack.content,"w+") if pack.cmd == "open" then file.open(pack.content,"w+")
elseif pack.cmd == "write" then file.write(pack.content) elseif pack.cmd == "write" then file.write(pack.content)
......
-- Somfy module example (beside somfy module requires also CJSON module) -- Somfy module example (beside somfy module requires also SJSON module)
-- The rolling code number is stored in the file somfy.cfg. A cached write of the somfy.cfg file is implemented in order to reduce the number of write to the EEPROM memory. Together with the logic of the file module it should allow long lasting operation. -- The rolling code number is stored in the file somfy.cfg. A cached write of the somfy.cfg file is implemented in order to reduce the number of write to the EEPROM memory. Together with the logic of the file module it should allow long lasting operation.
config_file = "somfy." config_file = "somfy."
...@@ -43,7 +43,7 @@ function readconfig() ...@@ -43,7 +43,7 @@ function readconfig()
end end
if not ln then ln = "{}" end if not ln then ln = "{}" end
print("Configuration: "..ln) print("Configuration: "..ln)
config = cjson.decode(ln) config = sjson.decode(ln)
config_saved = deepcopy(config) config_saved = deepcopy(config)
end end
...@@ -52,7 +52,7 @@ function writeconfighard() ...@@ -52,7 +52,7 @@ function writeconfighard()
file.remove(config_file.."bak") file.remove(config_file.."bak")
file.rename(config_file.."cfg", config_file.."bak") file.rename(config_file.."cfg", config_file.."bak")
file.open(config_file.."cfg", "w+") file.open(config_file.."cfg", "w+")
local ok, cfg = pcall(cjson.encode, config) local ok, cfg = pcall(sjson.encode, config)
if ok then if ok then
file.writeline(cfg) file.writeline(cfg)
else else
...@@ -68,8 +68,8 @@ function writeconfig() ...@@ -68,8 +68,8 @@ function writeconfig()
local savenow = false local savenow = false
local savelater = false local savelater = false
--print("Config: "..cjson.encode(config)) --print("Config: "..sjson.encode(config))
--print("Config saved: "..cjson.encode(config)) --print("Config saved: "..sjson.encode(config))
local count = 0 local count = 0
for _ in pairs(config_saved) do count = count + 1 end for _ in pairs(config_saved) do count = count + 1 end
...@@ -134,7 +134,7 @@ end ...@@ -134,7 +134,7 @@ end
--======================================================================================================-- --======================================================================================================--
if not config then readconfig() end if not config then readconfig() end
if #config == 0 then -- somfy.cfg does not exist if #config == 0 then -- somfy.cfg does not exist
config = cjson.decode([[{"window1":{"rc":1,"address":123},"window2":{"rc":1,"address":124}}]]) config = sjson.decode([[{"window1":{"rc":1,"address":123},"window2":{"rc":1,"address":124}}]])
config_saved = deepcopy(config) config_saved = deepcopy(config)
end end
down('window1', down('window1',
......
...@@ -72,6 +72,7 @@ pages: ...@@ -72,6 +72,7 @@ pages:
- 'rtcmem': 'en/modules/rtcmem.md' - 'rtcmem': 'en/modules/rtcmem.md'
- 'rtctime': 'en/modules/rtctime.md' - 'rtctime': 'en/modules/rtctime.md'
- 'sigma delta': 'en/modules/sigma-delta.md' - 'sigma delta': 'en/modules/sigma-delta.md'
- 'sjson': 'en/modules/sjson.md'
- 'sntp': 'en/modules/sntp.md' - 'sntp': 'en/modules/sntp.md'
- 'somfy': 'en/modules/somfy.md' - 'somfy': 'en/modules/somfy.md'
- 'spi': 'en/modules/spi.md' - 'spi': 'en/modules/spi.md'
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment