1. 12 Jan, 2021 24 commits
    • Yang Bodong's avatar
      Swapdb should make transaction fail if there is any client watching keys (#8239) · f464cf23
      Yang Bodong authored
      
      
      This PR not only fixes the problem that swapdb does not make the
      transaction fail, but also optimizes the FLUSHALL and FLUSHDB command to
      set the CLIENT_DIRTY_CAS flag to avoid unnecessary traversal of clients.
      
      FLUSHDB was changed to first iterate on all watched keys, and then on the
      clients watching each key.
      Instead of iterating though all clients, and for each iterate on watched keys.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 10f94b0a)
      f464cf23
    • Itamar Haber's avatar
      Adds 'use-memory' to GEORADIUS[BYMEMBER] (#8107) · 02c395a1
      Itamar Haber authored
      Partial resolution for #6860, item 7
      
      (cherry picked from commit feba7cbf)
      02c395a1
    • Oran Agra's avatar
      fix valgrind warning created by recent pidfile fix (#8235) · acdccab0
      Oran Agra authored
      This isn't a leak, just an warning due to unreachable
      allocation on the fork child.
      Problem created by 92a483bc
      
      (cherry picked from commit 2426aaa0)
      acdccab0
    • Meir Shpilraien (Spielrein)'s avatar
      Fix issue where fork process deletes the parent pidfile (#8231) · e9f674e5
      Meir Shpilraien (Spielrein) authored
      Turns out that when the fork child crashes, the crash log was deleting
      the pidfile from the disk (although the parent is still running.
      
      Now we set the pidfile of the fork process to NULL so the fork process
      will never deletes it.
      
      (cherry picked from commit 92a483bc)
      e9f674e5
    • Wen Hui's avatar
      0f8d47c9
    • Wen Hui's avatar
      fix memory leak in processInlineBuffer error handling code (#8295) · f1d56c98
      Wen Hui authored
      This code path is normally executed only when v6.0 and above replicates from v2.4
      
      (cherry picked from commit cfcd0fa6)
      f1d56c98
    • huangzhw's avatar
      sdscatfmt call sdsMakeRoomFor, asked for more space than intended (#8286) · ea595784
      huangzhw authored
      instead of asking for the extra new space it wanted, it asked to grow the
      string by the size it already has too.
      i.e. a string of 1000 bytes, needing to grow by 10 bytes, would have been
      asking for an **additional** 1010 bytes.
      
      (cherry picked from commit 595ecd5f)
      ea595784
    • huangzhw's avatar
      sort Command lookupKeyRead and lookupKeyWrite are used on the opposite (#8283) · 549fe1f3
      huangzhw authored
      This is a recent problem, introduced by 74717438 (redis 6.0)
      
      The implications are:
      The sole difference between LookupKeyRead and LookupKeyWrite is for command
      executed on a replica, which are not received from its master client. (for the master,
      and for the master client on the replica, these two functions behave the same)!
      
      Since SORT is a write command, this bug only implicates a writable-replica.
      And these are its implications:
      
      - SORT STORE will behave as it did before the above mentioned commit (like before
        redis 6.0). on a writable-replica an already logically expired the key would have
        appeared missing. (store dest key would be deleted, instead of being populated
        with the data from the already logically expired key)
      - SORT (the non store variant, which in theory could have been executed on
        read-only-replica if it weren't for the write flag), will (in redis 6.0) have a new bug
        and return the data from the already logically expired key instead of empty response.
      
      (cherry picked from commit 08ad6abd)
      549fe1f3
    • Oran Agra's avatar
      Fix rdb checksum / crc64 on bigendian (#8270) · cf6b0f9c
      Oran Agra authored
      Turns out the RDB checksum in Redis 6.0 on bigendian is broken.
      It always returned 0, so the RDB files are generated as if checksum is
      disabled, and will be loaded ok on littleendian, and on bigendian.
      But it'll not be able to load RDB files generated on littleendian or older versions.
      
      Similarly DUMP and RESTORE will work on the same version (0==0),
      but will be unable to exchange dump payloads with littleendian or old versions.
      
      (cherry picked from commit 324070c8)
      cf6b0f9c
    • Oran Agra's avatar
      fix crash in redis-cli after making cluster backup (#8267) · 02e93062
      Oran Agra authored
      getRDB is "designed" to work in two modes: one for redis-cli --rdb and
      one for redis-cli --cluster backup.
      in the later case it uses the hiredis connection from the cluster nodes
      and it used to free it without nullifying the context, so a later
      attempt to free the context would crash.
      
      I suppose the reason it seems to want to free the hiredis context ASAP
      is that it wants to disconnect the replica link, so that replication
      buffers will not be accumulated.
      
      (cherry picked from commit 41b2ed2b)
      02e93062
    • Oran Agra's avatar
      Fix wrong order of key/value in Lua map response (#8266) · ec56906b
      Oran Agra authored
      When a Lua script returns a map to redis (a feature which was added in
      redis 6 together with RESP3), it would have returned the value first and
      the key second.
      
      If the client was using RESP2, it was getting them out of order, and if
      the client was in RESP3, it was getting a map of value => key.
      This was happening regardless of the Lua script using redis.setresp(3)
      or not.
      
      This also affects a case where the script was returning a map which it got
      from from redis by doing something like: redis.setresp(3); return redis.call()
      
      This fix is a breaking change for redis 6.0 users who happened to rely
      on the wrong order (either ones that used redis.setresp(3), or ones that
      returned a map explicitly).
      
      This commit also includes other two changes in the tests:
      1. The test suite now handles RESP3 maps as dicts rather than nested
         lists
      2. Remove some redundant (duplicate) tests from tracking.tcl
      
      (cherry picked from commit 2017407b)
      ec56906b
    • Oran Agra's avatar
      Crash log would crash half way on commands with no arguments (#8260) · 23d29923
      Oran Agra authored
      The crash log attempts to print the current client info, and when it
      does that it attempts to check if the first argument happens to be a key
      but it did so for commands with no arguments too, which caused the crash
      log to crash half way and not reach its end.
      
      (cherry picked from commit 152b5d46)
      23d29923
    • Yossi Gottlieb's avatar
      Fix crashes with io-threads-do-reads enabled. (#8230) · f84c181e
      Yossi Gottlieb authored
      Normally IO threads should simply read data from the socket into the
      buffer and attempt to parse it.
      
      If a protocol error is detected, a reply is generated which may result
      with installing a write handler which is not thread safe. This fix
      delays that until the client is processed back in the main thread.
      
      Fixes #8220
      
      (cherry picked from commit e7047ec2)
      f84c181e
    • George Prekas's avatar
      Add check for the MADV_FREE/fork arm64 Linux kernel bug (#8224) · 8ff02263
      George Prekas authored
      
      
      Older arm64 Linux kernels have a bug that could lead to data corruption during
      background save under the following scenario:
      
      1) jemalloc uses MADV_FREE on a page,
      2) jemalloc reuses and writes the page,
      3) Redis forks the background save process, and
      4) Linux performs page reclamation.
      
      Under these conditions, Linux will reclaim the page wrongfully and the
      background save process will read zeros when it tries to read the page.
      
      The bug has been fixed in Linux with commit:
      ff1712f953e27f0b0718762ec17d0adb15c9fd0b ("arm64: pgtable: Ensure dirty bit is
      preserved across pte_wrprotect()")
      
      This Commit adds an ignore-warnings config, when not found, redis will
      print a warning and exit on startup (default behavior).
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit b02780c4)
      
      in 6.0 this warning is ignored by default in order to avoid adding
      regression, specifically for deployments that don't need persistence or
      replication
      8ff02263
    • Qu Chen's avatar
      Not over-allocate client query buffer when reading large objects. (#5954) · d756b11f
      Qu Chen authored
      In response to large client query buffer optimization introduced in 1898e6ce. The calculation of the amount of
      remaining bytes we need to write to the query buffer was calculated wrong, as a result we are unnecessarily
      growing the client query buffer by sdslen(c->querybuf) always. This fix corrects that behavior.
      
      Please note the previous behavior prior to the before-mentioned change was correctly calculating the remaining
      additional bytes, and this change makes that calculate to be consistent.
      
      Useful context, the argument of size `ll` starts at qb_pos (which is now the beginning of the sds), but much of it
      may have already been read from the socket, so we only need to grow the sds for the remainder of it.
      
      (cherry picked from commit 11b3325e)
      d756b11f
    • Oran Agra's avatar
      Handle output buffer limits for Module blocked clients (#8141) · ec74ae7e
      Oran Agra authored
      Module blocked clients cache the response in a temporary client,
      the reply list in this client would be affected by the recent fix
      in #7202, but when the reply is later copied into the real client,
      it would have bypassed all the checks for output buffer limit, which
      would have resulted in both: responding with a partial response to
      the client, and also not disconnecting it at all.
      
      (cherry picked from commit 48efc25f)
      ec74ae7e
    • Yossi Gottlieb's avatar
      Fix setproctitle related crashes. (#8150) · 785851a7
      Yossi Gottlieb authored
      Makes spt_init more careful with assumptions about what memory regions
      may be overwritten. It will now only consider a contiguous block of argv
      and envp elements and mind any gaps.
      
      (cherry picked from commit ec02c761)
      785851a7
    • Yossi Gottlieb's avatar
      Fix use-after-free issue in spt_copyenv. (#8088) · 2a2d5a7a
      Yossi Gottlieb authored
      Seems to have gone unnoticed for a long time, because at least with
      glibc it will only be triggered if setenv() was called before spt_init,
      which Redis doesn't.
      
      Fixes #8064.
      
      (cherry picked from commit 7e5a6313)
      2a2d5a7a
    • Wang Yuan's avatar
      Backup keys to slots map and restore when fail to sync if diskless-load type... · a60ed4a5
      Wang Yuan authored
      
      Backup keys to slots map and restore when fail to sync if diskless-load type is swapdb in cluster mode (#8108)
      
      When replica diskless-load type is swapdb in cluster mode, we didn't backup
      keys to slots map, so we will lose keys to slots map if fail to sync.
      Now we backup keys to slots map at first, and restore it properly when fail.
      
      This commit includes a refactory/cleanup of the backups mechanism (moving it to db.c and re-structuring it a bit).
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit b55a827e)
      a60ed4a5
    • Wang Yuan's avatar
      Reset average ttl when empty databases (#8106) · 5ed66621
      Wang Yuan authored
      On FLUSHDB or full sync, reset old average TTL stat.
      This Stat is incrementally collected by the master over time when it searches for expired keys. 
      
      (cherry picked from commit c85bf235)
      5ed66621
    • Oran Agra's avatar
      Fix bug with module GIL being released prematurely (#8061) · f885e364
      Oran Agra authored
      This is hopefully usually harmles.
      The server.ready_keys will usually be empty so the code after releasing
      the GIL will soon be done.
      The only case where it'll actually process things is when a module
      releases a client (or module) blocked on a key, by triggering this NOT
      from within a command (e.g. a timer event).
      
      This bug was introduced in redis 6.0.9, see #7903
      
      (cherry picked from commit e6fa4738)
      f885e364
    • Oran Agra's avatar
      Fix oom-score-adj-values range, abs options, and bug when used in config file (#8046) · 3ac666b3
      Oran Agra authored
      Fix: When oom-score-adj-values is provided in the config file after
      oom-score-adj yes, it'll take an immediate action, before
      readOOMScoreAdj was acquired, resulting in an error (out of range score
      due to uninitialized value. delay the reaction the real call is made by
      main().
      
      Since the values are clamped to -1000..1000, and they're
      applied as an offset from the value at startup (which may be -1000), we
      need to allow the offsets to reach to +2000 so that a value of +1000 is
      achievable in case the value at startup was -1000.
      
      Adding an option for absolute values rather than relative ones.
      
      (cherry picked from commit 61954951)
      3ac666b3
    • guybe7's avatar
      EXISTS should not alter LRU, OBJECT should not reveal expired keys on replica (#8016) · 4b37eb13
      guybe7 authored
      The bug was introduced by #5021 which only attempted avoid EXIST on an
      already expired key from returning 1 on a replica.
      
      Before that commit, dbExists was used instead of
      lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU)
      
      Other than that, this commit fixes OBJECT to also come empty handed on
      expired keys in replica.
      
      And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from
      the key regardless of it's expired state)
      
      (cherry picked from commit f8ae9917)
      4b37eb13
    • Wang Yuan's avatar
      Disable rehash when redis has child process (#8007) · 3a13c654
      Wang Yuan authored
      In redisFork(), we don't set child pid, so updateDictResizePolicy()
      doesn't take effect, that isn't friendly for copy-on-write.
      
      The bug was introduced this in redis 6.0: 56258c6b
      
      (cherry picked from commit 89c78a98)
      3a13c654
  2. 27 Oct, 2020 16 commits
    • Oran Agra's avatar
      Redis 6.0.9. · 25214bd7
      Oran Agra authored
      25214bd7
    • Yossi Gottlieb's avatar
      Fix wrong zmalloc_size() assumption. (#7963) · 3cc673bd
      Yossi Gottlieb authored
      When using a system with no malloc_usable_size(), zmalloc_size() assumed
      that the heap allocator always returns blocks that are long-padded.
      
      This may not always be the case, and will result with zmalloc_size()
      returning a size that is bigger than allocated. At least in one case
      this leads to out of bound write, process crash and a potential security
      vulnerability.
      
      Effectively this does not affect the vast majority of users, who use
      jemalloc or glibc.
      
      This problem along with a (different) fix was reported by Drew DeVault.
      
      (cherry picked from commit 9824fe3e)
      3cc673bd
    • David CARLIER's avatar
      cpu affinity: DragonFlyBSD support (#7956) · 5aa0fef2
      David CARLIER authored
      
      (cherry picked from commit 27f4c212)
      5aa0fef2
    • WuYunlong's avatar
      Update rdb_last_bgsave_time_sec in INFO on diskless replication (#7917) · 6f16e52a
      WuYunlong authored
      `info Persistence` will include correct (updated) rdb_last_bgsave_time_sec
      For diskless bgsave (sockets) too (like a few other persistence info fields).
      
      Refactor code to reduce duplicate code.
      
      (cherry picked from commit e05a7df7)
      6f16e52a
    • Wen Hui's avatar
      do not add save parameter during config rewrite in sentinel mode (#7945) · c6c2de95
      Wen Hui authored
      Previous code would have added default redis save parameters
      to the config file on rewrite, which would have been silently ignored
      when the config file is loaded.
      
      The new code avoids adding this, and also actively removes these lines
      If added by a previous config rewrite. 
      
      (cherry picked from commit 0f370f9b)
      c6c2de95
    • Qu Chen's avatar
      WATCH no longer ignores keys which have expired for MULTI/EXEC. (#7920) · cebc1f26
      Qu Chen authored
      This wrong behavior was backed by a test, and also documentation, and dates back to 2010.
      But it makes no sense to anyone involved so it was decided to change that.
      
      Note that 20eeddfb (invalidate watch on expire on access) was released in 6.0 RC2
      and 2d1968f8 released in in 6.0.0 GA (invalidate watch when key is evicted).
      both of which do similar changes.
      
      (cherry picked from commit 556acefe)
      cebc1f26
    • filipe oliveira's avatar
      Fixed bug concerning redis-benchmark non clustered benchmark forcing always... · 71c1427a
      filipe oliveira authored
      
      Fixed bug concerning redis-benchmark non clustered benchmark forcing always the same hash tag {tag} (#7931)
      
      Adding the ":{tag}" only if --cluster is used, so that when used against
      a proxy it generates traffic to all shards.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 6cf23d66)
      71c1427a
    • Oran Agra's avatar
      fix 32bit build warnings (#7926) · 30bb0dd5
      Oran Agra authored
      
      (cherry picked from commit a425e1d2)
      30bb0dd5
    • Wen Hui's avatar
      fix double fclose in aofrewrite (#7919) · 99f39d32
      Wen Hui authored
      minor fix for a bug which happen on error handling code
      and doesn't look like it could have caused any real harm
      (fd number wouldn't have been reused yet)
      
      (cherry picked from commit 04a0af90)
      99f39d32
    • Oran Agra's avatar
      INFO report peak memory before eviction (#7894) · 851ec530
      Oran Agra authored
      In some cases one command added a very big bulk of memory, and this
      would be "resolved" by the eviction before the next command.
      
      Seeing an unexplained mass eviction we would wish to
      know the highest momentary usage too.
      
      Tracking it in call() and beforeSleep() adds some hooks in AOF and RDB
      loading.
      
      The fix in clientsCronTrackExpansiveClients is related to #7874
      
      (cherry picked from commit 457b7073)
      851ec530
    • Oran Agra's avatar
      Allow requirepass config to clear the password (#7899) · b20af834
      Oran Agra authored
      This is a compatibility issue with redis 5.0 that was introduced by ACL.
      Before this commit, setting requirepass to an empty string will result
      in a server that needs an empty AUTH, unlike redis 5.0 which would
      accept connections without an AUTH.
      
      (cherry picked from commit 19418b6b)
      b20af834
    • Wang Yuan's avatar
      Remove temporary aof and rdb files in a background thread (#7905) · 14532772
      Wang Yuan authored
      If we fail or stop to rewrite aof, we need to remove temporary aof.
      We also remove temporary rdb when replicas abort to receive rdb.
      But currently we delete them in main thread, to avoid blocking,
      we should use bg_unlink to remove them in a background thread.
      
      Btw, we have already used this way to removed child process temporary rdb.
      
      (cherry picked from commit aaacb8c9)
      14532772
    • guybe7's avatar
      Minor improvements to module blocked on keys (#7903) · e4e24460
      guybe7 authored
      - Clarify some documentation comments
      - Make sure blocked-on-keys client privdata is accessible
        from withing the timeout callback
      - Handle blocked clients in beforeSleep - In case a key
        becomes "ready" outside of processCommand
      
      See #7879 #7880
      
      (cherry picked from commit addf47dc)
      e4e24460
    • Andreas Lind's avatar
      Support redis-cli -u rediss://... (#7900) · db9cffe7
      Andreas Lind authored
      
      (cherry picked from commit 8b497881)
      db9cffe7
    • Yossi Gottlieb's avatar
      Modules: fix RM_GetCommandKeys API. (#7901) · 9c475272
      Yossi Gottlieb authored
      This cleans up and simplifies the API by passing the command name as the
      first argument. Previously the command name was specified explicitly,
      but was still included in the argv.
      
      (cherry picked from commit 056a43e1)
      9c475272
    • Meir Shpilraien (Spielrein)'s avatar
      Add Module API for version and compatibility checks (#7865) · 6f2c894d
      Meir Shpilraien (Spielrein) authored
      
      
      * Introduce a new API's: RM_GetContextFlagsAll, and
      RM_GetKeyspaceNotificationFlagsAll that will return the
      full flags mask of each feature. The module writer can
      check base on this value if the Flags he needs are
      supported or not.
      
      * For each flag, introduce a new value on redismodule.h,
      this value represents the LAST value and should be there
      as a reminder to update it when a new value is added,
      also it will be used in the code to calculate the full
      flags mask (assuming flags are incrementally increasing).
      In addition, stated that the module writer should not use
      the LAST flag directly and he should use the GetFlagAll API's.
      
      * Introduce a new API: RM_IsSubEventSupported, that returns for a given
      event and subevent, whether or not the subevent supported.
      
      * Introduce a new macro RMAPI_FUNC_SUPPORTED(func) that returns whether
      or not a function API is supported by comparing it to NULL.
      
      * Introduce a new API: int RM_GetServerVersion();, that will return the
      current Redis version in the format 0x00MMmmpp; e.g. 0x00060008;
      
      * Changed unstable version from 999.999.999 to 255.255.255
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      (cherry picked from commit adc3183c)
      6f2c894d