• sundb's avatar
    Change lzf to handle values larger than UINT32_MAX (#9776) · 985430b4
    sundb authored
    Redis supports inserting data over 4GB into string (and recently for lists too, see #9357),
    But LZF compression used in RDB files (see `rdbcompression` config), and in quicklist
    (see `list-compress-depth` config) does not support compress/decompress data over
    UINT32_MAX, which will result in corrupting the rdb after compression.
    
    Internal changes:
    1. Modify the `unsigned int` parameter of `lzf_compress/lzf_decompress` to `size_t`.
    2. Modify the variable types in `lzf_compress` involving offsets and lengths to `size_t`.
    3. Set LZF_USE_OFFSETS to 0.
        When LZF_USE_OFFSETS is 1, lzf store offset into `LZF_HSLOT`(32bit). 
        Even in 64-bit, `LZF_USE_OFFSETS` defaults to 1, because lzf assumes that it only
        compresses and decompresses data smaller than UINT32_MAX.
        But now we need to make lzf support 64-bit, turning on `LZF_USE_OFFSETS` will make
        it impossible to store 64-bit offsets or pointers.
        BTW, disable LZF_USE_OFFSETS also brings a few performance improvements.
    
    Tests:
    1. Add test for compress/decompress string large than UINT32_MAX.
    2. Add unittest for compress/decompress quicklistNode.
    985430b4
lzf_d.c 6.29 KB