• debing.sun's avatar
    Determine the large limit of the quicklist node based on fill (#12659) · 165afc5f
    debing.sun authored
    Following #12568
    
    In issue #9357, when inserting an element larger than 1GB, we currently
    store it in a plain node instead of a listpack.
    Presently, when we insert an element that exceeds the maximum size of a
    packed node, it cannot be accommodated in any other nodes, thus ending
    up isolated like a large element.
    I.e. it's a node with only one element, but it's listpack encoded rather
    than a plain buffer.
    
    This PR lowers the threshold for considering an element as 'large' from
    1GB to the maximum size of a node.
    While this change doesn't completely resolve the bug mentioned in the
    previous PR, it does mitigate its potential impact.
    
    As a result of this change, we can now only use LSET to replace an
    element with another element that falls below the maximum size
    threshold.
    In the worst-case scenario, with a fill of -5, the largest packed node
    we can create is 2GB (32k * 64k):
    * 32k: The smallest element in a listpack is 2 bytes, which allows us to
    store up to 32k elements.
    * 64k: This is the maximum size for a single quicklist node.
    
    ## Others
    To fully fix #9357, we need more work, as discussed in #12568, when we
    insert an element into a quicklistNode, it may be created in a new node,
    put into another node, or merged, and we can't correctly delete the node
    that was supposed to be deleted.
    I'm not sure it's worth it, since it involves a lot of modifications.
    165afc5f
debug.c 102 KB