• sundb's avatar
    Avoid mostly harmless integer overflow in cjson (#12456) · da9c2804
    sundb authored
    This PR mainly fixes a possible integer overflow in `json_append_string()`.
    When we use `cjson.encoding()` to encode a string larger than 2GB, at specific
    compilation flags, an integer overflow may occur leading to truncation, resulting
    in the part of the string larger than 2GB not being encoded.
    On the other hand, this overflow doesn't cause any read or write out-of-range or segment fault.
    
    1) using -O0 for lua_cjson (`make LUA_DEBUG=yes`)
        In this case, `i` will overflow and leads to truncation.
        When `i` reaches `INT_MAX+1` and overflows to INT_MIN, when compared to
        len, `i` (1000000..00) is expanded to 64 bits signed integer (1111111.....000000) .
        At this point i will be greater than len and jump out of the loop, so `for (i = 0; i < len; i++)`
        will loop up to 2^31 times, and the part of larger than 2GB will be truncated.
    
    ```asm
    `i` => -0x24(%rbp)
    <+253>:   addl   $0x1,-0x24(%rbp)       ; overflow if i large than 2^31
    <+257>:   mov    -0x24(%rbp),%eax
    <+260>:   movslq %eax,%rdx	            ; move a 32-bit value with sign extension into a 64-bit signed
    <+263>:   mov    -0x20(%rbp),%rax
    <+267>:   cmp    %rax,%rdx              ; check `i < len`
    <+270>:   jb     0x212600 <json_append_string+148>
    ```
       
    2) using -O2/-O3 for lua_cjson (`make LUA_DEBUG=no`, **the default**)
        In this case, because singed integer overflow is an undefined behavior, `i` will not overflow.
       `i` will be optimized by the compiler and use 64-bit registers for all subsequent instructions.
    
    ```asm
    <+180>:   add    $0x1,%rbx           ; Using 64-bit register `rbx` for i++
    <+184>:   lea    0x1(%rdx),%rsi
    <+188>:   mov    %rsi,0x10(%rbp)
    <+192>:   mov    %al,(%rcx,%rdx,1)
    <+195>:   cmp    %rbx,(%rsp)         ; check `i < len`
    <+199>:   ja     0x20b63a <json_append_string+154>
    ```
    
    3) using 32bit
        Because `strbuf_ensure_empty_length()` preallocates memory of length (len * 6 + 2),
        in 32-bit `cjson.encode()` can only handle strings smaller than ((2 ^ 32) - 3 ) / 6.
        So 32bit is not affected.
    
    Also change `i` in `strbuf_append_string()` to `size_t`.
    Since its second argument `str` is taken from the `char2escape` string array which is never
    larger than 6, so `strbuf_append_string()` is not at risk of overflow (the bug was unreachable).
    da9c2804
strbuf.c 4.4 KB