• Wang Yuan's avatar
    Limit the main db and expires dictionaries to expand (#7954) · 75f9dec6
    Wang Yuan authored
    As we know, redis may reject user's requests or evict some keys if
    used memory is over maxmemory. Dictionaries expanding may make
    things worse, some big dictionaries, such as main db and expires dict,
    may eat huge memory at once for allocating a new big hash table and be
    far more than maxmemory after expanding.
    There are related issues: #4213 #4583
    
    More details, when expand dict in redis, we will allocate a new big
    ht[1] that generally is double of ht[0], The size of ht[1] will be
    very big if ht[0] already is big. For db dict, if we have more than
    64 million keys, we need to cost 1GB for ht[1] when dict expands.
    
    If the sum of used memory and new hash table of dict needed exceeds
    maxmemory, we shouldn't allow the dict to expand. Because, if we
    enable keys eviction, we still couldn't add much more keys after
    eviction and rehashing, what's worse, redis will keep less keys when
    redis only remains a little memory for storing new hash table instead
    of users' data. Moreover users can't write data in redis if disable
    keys eviction.
    
    What this commit changed ?
    
    Add a new member function expandAllowed for dict type, it provide a way
    for caller to allow expand or not. We expose two parameters for this
    function: more memory needed for expanding and dict current load factor,
    users can implement a function to make a decision by them.
    For main db dict and expires dict type, these dictionaries may be very
    big and cost huge memory for expanding, so we implement a judgement
    function: we can stop dict to expand provisionally if used memory will
    be over maxmemory after dict expands, but to guarantee the performance
    of redis, we still allow dict to expand if dict load factor exceeds the
    safe load factor.
    Add test cases to verify we don't allow main db to expand when left
    memory is not enough, so that avoid keys eviction.
    
    Other changes:
    
    For new hash table size when expand. Before this commit, the size is
    that double used of dict and later _dictNextPower. Actually we aim to
    control a dict load factor between 0.5 and 1.0. Now we replace *2 with
    +1, since the first check is that used >= size, the outcome of before
    will usually be the same as _dictNextPower(used+1). The only case where
    it'll differ is when dict_can_resize is false during fork, so that later
    the _dictNextPower(used*2) will cause the dict to jump to *4 (i.e.
    _dictNextPower(1025*2) will return 4096).
    Fix rehash test cases due to changing algorithm of new hash table size
    when expand.
    75f9dec6
latency.c 27.2 KB