1. 28 Feb, 2023 2 commits
    • Madelyn Olson's avatar
      Prevent Redis from crashing from key tracking invalidations (#11814) · 683a4ce4
      Madelyn Olson authored
      (cherry picked from commit f7150c45bc5d6f03c8ba86a9a9296d024c6848dc)
      683a4ce4
    • uriyage's avatar
      Optimization: sdsRemoveFreeSpace to avoid realloc on noop (#11766) · f084778c
      uriyage authored
      
      
      In #7875 (Redis 6.2), we changed the sds alloc to be the usable allocation
      size in order to:
      
      > reduce the need for realloc calls by making the sds implicitly take over
      the internal fragmentation
      
      This change was done most sds functions, excluding `sdsRemoveFreeSpace` and
      `sdsResize`, the reason is that in some places (e.g. clientsCronResizeQueryBuffer)
      we call sdsRemoveFreeSpace when we see excessive free space and want to trim it.
      so if we don't trim it exactly to size, the caller may still see excessive free space and
      call it again and again.
      
      However, this resulted in some excessive calls to realloc, even when there's no need
      and it's gonna be a no-op (e.g. when reducing 15 bytes allocation to 13).
      
      It turns out that a call for realloc with jemalloc can be expensive even if it ends up
      doing nothing, so this PR adds a check using `je_nallocx`, which is cheap to avoid
      the call for realloc.
      
      in addition to that this PR unifies sdsResize and sdsRemoveFreeSpace into common
      code. the difference between them was that sdsResize would avoid using SDS_TYPE_5,
      since it want to keep the string ready to be resized again, while sdsRemoveFreeSpace
      would permit using SDS_TYPE_5 and get an optimal memory consumption.
      now both methods take a `would_regrow` argument that makes it more explicit.
      
      the only actual impact of that is that in clientsCronResizeQueryBuffer we call both sdsResize
      and sdsRemoveFreeSpace for in different cases, and we now prevent the use of SDS_TYPE_5 in both.
      
      The new test that was added to cover this concern used to pass before this PR as well,
      this PR is just a performance optimization and cleanup.
      
      Benchmark:
      `redis-benchmark -c 100 -t set  -d 512 -P 10  -n  100000000`
      on i7-9850H with jemalloc, shows improvement from 1021k ops/sec to 1067k (average of 3 runs).
      some 4.5% improvement.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 46393f98)
      (cherry picked from commit b12eeccddd9318a5d97a5aee2dad88999dfad53f)
      f084778c
  2. 17 Jan, 2023 1 commit
    • Oran Agra's avatar
      Revert change to KEYS command from (#11676) · e12aacf3
      Oran Agra authored
      in Redis 7.0 this fix covers KEYS as well, but in 6.2 and 6.0 it doesn't,
      this is because in 7.0 there's a mechanism to avoid sending partial replies
      to the client, and in older releases there isn't, and without it there's a
      risk that the client would be able to read what looks like a complete KEYS
      command.
      e12aacf3
  3. 16 Jan, 2023 3 commits
    • Oran Agra's avatar
      Fix range issues in ZRANDMEMBER and HRANDFIELD (CVE-2023-22458) · 37b3b2a7
      Oran Agra authored
      missing range check in ZRANDMEMBER and HRANDIFLD leading to panic due
      to protocol limitations
      37b3b2a7
    • Oran Agra's avatar
      Avoid integer overflows in SETRANGE and SORT (CVE-2022-35977) · 54538998
      Oran Agra authored
      Authenticated users issuing specially crafted SETRANGE and SORT(_RO)
      commands can trigger an integer overflow, resulting with Redis attempting
      to allocate impossible amounts of memory and abort with an OOM panic.
      54538998
    • Oran Agra's avatar
      Obuf limit, exit during loop in *RAND* commands and KEYS · 3148f3e8
      Oran Agra authored
      Related to the hang reported in #11671
      Currently, redis can disconnect a client due to reaching output buffer limit,
      it'll also avoid feeding that output buffer with more data, but it will keep
      running the loop in the command (despite the client already being marked for
      disconnection)
      
      This PR is an attempt to mitigate the problem, specifically for commands that
      are easy to abuse, specifically: KEYS, HRANDFIELD, SRANDMEMBER, ZRANDMEMBER.
      The RAND family of commands can take a negative COUNT argument (which is not
      bound to the number of elements in the key), so it's enough to create a key
      with one field, and then these commands can be used to hang redis.
      For KEYS the caller can use the existing keyspace in redis (if big enough).
      3148f3e8
  4. 12 Dec, 2022 13 commits
    • Yossi Gottlieb's avatar
      Fix TLS tests on newer tcl-tls/OpenSSL. (#10910) · de4d78d7
      Yossi Gottlieb authored
      Before this commit, TLS tests on Ubuntu 22.04 would fail as dropped
      connections result with an ECONNABORTED error thrown instead of an empty
      read.
      
      (cherry picked from commit 69d55768)
      de4d78d7
    • Yossi Gottlieb's avatar
      Use 'gcc' instead of 'ld' to link test modules. (#9710) · e5e642aa
      Yossi Gottlieb authored
      This solves several problems in a more elegant way:
      
      * No need to explicitly use `-lc` on x86_64 when building with `-m32`.
      * Avoids issues with undefined floating point emulation funcs on ARM.
      
      (cherry picked from commit f26e90be)
      e5e642aa
    • Yossi Gottlieb's avatar
      Fix daily failures due to macos-latest change. (#9637) · 52cccfbe
      Yossi Gottlieb authored
      * Fix test modules linking on macOS 11.x.
      * Use macOS 10.x for FreeBSD VM as VirtualBox is not yet supported on
        11.
      
      (cherry picked from commit 6d5a9117)
      52cccfbe
    • Oran Agra's avatar
      Try to fix a race in psync2 test (#11553) · 0a3ae0f0
      Oran Agra authored
      This test sets the master ping interval to 1 hour, in order to avoid
      pings in the replicatoin stream incrementing the replication offset,
      however, it didn't increase the repl-timeout so on slow machines
      where the test took more than 60 seconds, the replicas would drop
      and reconnect.
      
      ```
      *** [err]: PSYNC2: Partial resync after restart using RDB aux fields in tests/integration/psync2.tcl
      Replica didn't partial sync
      ```
      
      The test would detect 4 additional partial syncs where it expects
      only one.
      
      (cherry picked from commit b0250b45)
      0a3ae0f0
    • uriyage's avatar
      Module CLIENT_CHANGE, Fix crash on free blocked client with DB!=0 (#11500) · 39e5a6fa
      uriyage authored
      
      
      In moduleFireServerEvent we change the real client DB to 0 on freeClient in case the event is REDISMODULE_EVENT_CLIENT_CHANGE.
      It results in a crash if the client is blocked on a key on other than DB 0.
      
      The DB change is not necessary even for module-client, as we set its DB to 0 on either createClient or moduleReleaseTempClient.
      Co-authored-by: default avatarMadelyn Olson <34459052+madolson@users.noreply.github.com>
      Co-authored-by: default avatarBinbin <binloveplay1314@qq.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit e4eb18b3)
      39e5a6fa
    • Oran Agra's avatar
      fixes for fork child exit and test: #11463 (#11499) · 4ac3d79b
      Oran Agra authored
      
      
      Fix a few issues with the recent #11463
      * use exitFromChild instead of exit
      * test should ignore defunct process since that's what we expect to
        happen for thees child processes when the parent dies.
      * fix typo
      Co-authored-by: default avatarBinbin <binloveplay1314@qq.com>
      (cherry picked from commit 4c54528f)
      4ac3d79b
    • Oran Agra's avatar
      diskless master, avoid bgsave child hung when fork parent crashes (#11463) · 51fa40ff
      Oran Agra authored
      During a diskless sync, if the master main process crashes, the child would
      have hung in `write`. This fix closes the read fd on the child side, so that if the
      parent crashes, the child will get a write error and exit.
      
      This change also fixes disk-based replication, BGSAVE and AOFRW.
      In that case the child wouldn't have been hang, it would have just kept
      running until done which may be pointless.
      
      There is a certain degree of risk here. in case there's a BGSAVE child that could
      maybe succeed and the parent dies for some reason, the old code would have let
      the child keep running and maybe succeed and avoid data loss.
      On the other hand, if the parent is restarted, it would have loaded an old rdb file
      (or none), and then the child could reach the end and rename the rdb file (data
      conflicting with what the parent has), or also have a race with another BGSAVE
      child that the new parent started.
      
      Note that i removed a comment saying a write error will be ignored in the child
      and handled by the parent (this comment was very old and i don't think relevant).
      
      (cherry picked from commit ccaef5c9)
      51fa40ff
    • DarrenJiang13's avatar
      fix the client type in trackingInvalidateKey() (#11052) · ec5564a4
      DarrenJiang13 authored
      Fix bug with scripts ignoring client tracking NOLOOP and
      send an invalidation message anyway.
      
      (cherry picked from commit 44859a41)
      ec5564a4
    • Huang Zhw's avatar
      tracking pending invalidation message of flushdb sent by (#11068) · 2cd5f6f3
      Huang Zhw authored
      trackingHandlePendingKeyInvalidations should use proto.
      
      (cherry picked from commit 61451b02)
      2cd5f6f3
    • Huang Zhw's avatar
      When client tracking is on, invalidation message of flushdb in a (#11038) · fbcd591b
      Huang Zhw authored
      When FLUSHDB / FLUSHALL / SWAPDB is inside MULTI / EXEC, the
      client side tracking invalidation message was interleaved with transaction response.
      
      (cherry picked from commit 6f0a27e3)
      fbcd591b
    • Meir Shpilraien (Spielrein)'s avatar
      Fix #11030, use lua_rawget to avoid triggering metatables and crash. (#11032) · e1b17f12
      Meir Shpilraien (Spielrein) authored
      Fix #11030, use lua_rawget to avoid triggering metatables.
      
      #11030 shows how return `_G` from the Lua script (either function or eval), cause the
      Lua interpreter to Panic and the Redis processes to exit with error code 1.
      Though return `_G` only panic on Redis 7 and 6.2.7, the underline issue exists on older
      versions as well (6.0 and 6.2). The underline issue is returning a table with a metatable
      such that the metatable raises an error.
      
      The following example demonstrate the issue:
      ```
      127.0.0.1:6379> eval "local a = {}; setmetatable(a,{__index=function() foo() end}) return a" 0
      Error: Server closed the connection
      ```
      ```
      PANIC: unprotected error in call to Lua API (user_script:1: Script attempted to access nonexistent global variable 'foo')
      ```
      
      The Lua panic happened because when returning the result to the client, Redis needs to
      introspect the returning table and transform the table into a resp. In order to scan the table,
      Redis uses `lua_gettable` api which might trigger the metatable (if exists) and might raise an error.
      This code is not running inside `pcall` (Lua protected call), so raising an error causes the
      Lua to panic and exit. Notice that this is not a crash, its a Lua panic that exit with error code 1.
      
      Returning `_G` panics on Redis 7 and 6.2.7 because on those versions `_G` has a metatable
      that raises error when trying to fetch a none existing key.
      
      ### Solution
      
      Instead of using `lua_gettable` that might raise error and cause the issue, use `lua_rawget`
      that simply return the value from the table without triggering any metatable logic.
      This is promised not to raise and error.
      
      The downside of this solution is that it might be considered as breaking change, if someone
      rely on metatable in the returned value. An alternative solution is to wrap this entire logic
      with `pcall` (Lua protected call), this alternative require a much bigger refactoring.
      
      ### Back Porting
      
      The same fix will work on older versions as well (6.2, 6.0). Notice that on those version,
      the issue can cause Redis to crash if inside the metatable logic there is an attempt to accesses
      Redis (`redis.call`). On 7.0, there is not crash and the `redis.call` is executed as if it was done
      from inside the script itself.
      
      ### Tests
      
      Tests was added the verify the fix
      
      (cherry picked from commit 020e046b)
      e1b17f12
    • Oran Agra's avatar
      optimize zset conversion on large ZRANGESTORE (#10789) · 70aaf500
      Oran Agra authored
      when we know the size of the zset we're gonna store in advance,
      we can check if it's greater than the listpack encoding threshold,
      in which case we can create a skiplist from the get go, and avoid
      converting the listpack to skiplist later after it was already populated.
      
      (cherry picked from commit 21891003)
      70aaf500
    • Vitaly's avatar
      Fix ZRANGESTORE crash when zset_max_listpack_entries is 0 (#10767) · fd7bde57
      Vitaly authored
      When `zrangestore` is called container destination object is created.
      Before this PR we used to create a listpack based object even if `zset-max-ziplist-entries`
      or equivalent`zset-max-listpack-entries` was set to 0.
      This triggered immediate conversion of the listpack into a skiplist in `zrangestore`, which hits
      an assertion resulting in an engine crash.
      
      Added a TCL test that reproduces this issue.
      
      (cherry picked from commit 6461f09f)
      fd7bde57
  5. 27 Apr, 2022 21 commits
    • Yossi Gottlieb's avatar
      Fix test modules build issue on OS X 11. (#9658) · 30533370
      Yossi Gottlieb authored
      
      (cherry picked from commit 8bf4c2e3)
      30533370
    • Oran Agra's avatar
      Skip cluster test unit in TLS mode. · 215c5eac
      Oran Agra authored
      see 7d6744c7
      215c5eac
    • Oran Agra's avatar
    • Yossi Gottlieb's avatar
      Clean unused var compiler warning in module test. (#9289) · ba2feb30
      Yossi Gottlieb authored
      
      (cherry picked from commit 8bf433dc)
      ba2feb30
    • sundb's avatar
      Fix memory leak due to missing freeCallback in blockonbackground moduleapi test (#9499) · aa0606db
      sundb authored
      Before #9497, before redis-server was shut down, we did not manually shut down all the clients,
      which would have prevented valgrind from detecting a memory leak in the client's argc.
      
      (cherry picked from commit 1376d833)
      aa0606db
    • Oran Agra's avatar
    • qetu3790's avatar
      Fix geo search bounding box check causing missing results (#10018) · 159981e7
      qetu3790 authored
      
      
      Consider the following example:
      1. geoadd k1 -0.15307903289794921875 85 n1 0.3515625 85.00019260486917005437 n2.
      2. geodist k1 n1 n2 returns  "4891.9380"
      3. but GEORADIUSBYMEMBER k1 n1 4891.94 m only returns n1.
      n2 is in the  boundingbox but out of search areas.So we let  search areas contain boundingbox to get n2.
      Co-authored-by: default avatarBinbin <binloveplay1314@qq.com>
      (cherry picked from commit b2d393b9)
      159981e7
    • Oran Agra's avatar
      Fix and improve module error reply statistics (#10278) · d6a8e64e
      Oran Agra authored
      This PR handles several aspects
      1. Calls to RM_ReplyWithError from thread safe contexts don't violate thread safety.
      2. Errors returning from RM_Call to the module aren't counted in the statistics (they
        might be handled silently by the module)
      3. When a module propagates a reply it got from RM_Call to it's client, then the error
        statistics are counted.
      
      This is done by:
      1. When appending an error reply to the output buffer, we avoid updating the global
        error statistics, instead we cache that error in a deferred list in the client struct.
      2. When creating a RedisModuleCallReply object, the deferred error list is moved from
        the client into that object.
      3. when a module calls RM_ReplyWithCallReply we copy the deferred replies to the dest
        client (if that's a real client, then that's when the error statistics are updated to the server)
      
      Note about RM_ReplyWithCallReply: if the original reply had an array with errors, and the module
      replied with just a portion of the original reply, and not the entire reply, the errors are currently not
      propagated and the errors stats will not get propagated.
      
      Fix #10180
      
      (cherry picked from commit b099889a)
      d6a8e64e
    • Binbin's avatar
      LPOP/RPOP with count against non existing list return null array (#10095) · e24b947c
      Binbin authored
      It used to return `$-1` in RESP2, now we will return `*-1`.
      This is a bug in redis 6.2 when COUNT was added, the `COUNT`
      option was introduced in #8179. Fix #10089.
      
      the documentation of [LPOP](https://redis.io/commands/lpop) says
      ```
      When called without the count argument:
      Bulk string reply: the value of the first element, or nil when key does not exist.
      
      When called with the count argument:
      Array reply: list of popped elements, or nil when key does not exist.
      ```
      
      (cherry picked from commit 39feee8e)
      e24b947c
    • guybe7's avatar
      lpGetInteger returns int64_t, avoid overflow (#10068) · c5a753c8
      guybe7 authored
      Fix #9410
      
      Crucial for the ms and sequence deltas, but I changed all
      calls, just in case (e.g. "flags")
      
      Before this commit:
      `ms_delta` and `seq_delta` could have overflown, causing `currid` to be wrong,
      which in turn would cause `streamTrim` to trim the entire rax node (see new test)
      
      (cherry picked from commit 7cd6a64d)
      c5a753c8
    • Madelyn Olson's avatar
      Redact ACL SETUSER arguments if the user has spaces (#9935) · 066b6832
      Madelyn Olson authored
      
      (cherry picked from commit c40d23b8)
      066b6832
    • Meir Shpilraien (Spielrein)'s avatar
      Clean Lua stack before parsing call reply to avoid crash on a call with many arguments (#9809) · 93c1d31d
      Meir Shpilraien (Spielrein) authored
      This commit 0f8b634c (CVE-2021-32626 released in 6.2.6, 6.0.16, 5.0.14)
      fixes an invalid memory write issue by using `lua_checkstack` API to make
      sure the Lua stack is not overflow. This fix was added on 3 places:
      1. `luaReplyToRedisReply`
      2. `ldbRedis`
      3. `redisProtocolToLuaType`
      
      On the first 2 functions, `lua_checkstack` is handled gracefully while the
      last is handled with an assert and a statement that this situation can
      not happened (only with misbehave module):
      
      > the Redis reply might be deep enough to explode the LUA stack (notice
      that currently there is no such command in Redis that returns such a nested
      reply, but modules might do it)
      
      The issue that was discovered is that user arguments is also considered part
      of the stack, and so the following script (for example) make the assertion reachable:
      ```
      local a = {}
      for i=1,7999 do
          a[i] = 1
      end
      return redis.call("lpush", "l", unpack(a))
      ```
      
      This is a regression because such a script would have worked before and now
      its crashing Redis. The solution is to clear the function arguments from the Lua
      stack which makes the original assumption true and the assertion unreachable.
      
      (cherry picked from commit 6b0b04f1)
      93c1d31d
    • Binbin's avatar
      Add tests to cover EXPIRE overflow fix (#9839) · 8fca090e
      Binbin authored
      In #8287, some overflow checks have been added. But when
      `when *= 1000` overflows, it will become a positive number.
      And the check not able to catch it. The key will be added with
      a short expiration time and will deleted a few seconds later.
      
      In #9601, will check the overflow after `*=` and return an
      error first, and avoiding this situation.
      
      In this commit, added some tests to cover those code paths.
      Found it in #9825, and close it.
      
      (cherry picked from commit 9273d09d)
      8fca090e
    • Oran Agra's avatar
      fix invalid read on corrupt ziplist (#9831) · e38d0b5a
      Oran Agra authored
      If the last bytes in ziplist are corrupt and we decode from tail to head,
      we may reach slightly outside the ziplist.
      
      (cherry picked from commit a3a01429)
      e38d0b5a
    • Itamar Haber's avatar
      Fixes LPOP/RPOP wrong replies when count is 0 (#9692) · 968cd2b9
      Itamar Haber authored
      Introduced in #8179, this fixes the command's replies in the 0 count edge case.
      [BREAKING] changes the reply type when count is 0 to an empty array (instead of nil)
      Moves LPOP ... 0 fast exit path after type check to reply with WRONGTYPE
      
      (cherry picked from commit 06dd202a)
      968cd2b9
    • Oran Agra's avatar
      fix new cluster tests issues (#9657) · 3ba7c6ac
      Oran Agra authored
      Following #9483 the daily CI exposed a few problems.
      
      * The cluster creation code (uses redis-cli) is complicated to test with TLS enabled.
        for now i'm just skipping them since the tests we run there don't really need that kind of coverage
      * cluster port binding failures
        note that `find_available_port` already looks for a free cluster port
        but the code in `wait_server_started` couldn't detect the failure of binding
        (the text it greps for wasn't found in the log)
      
      (cherry picked from commit 7d6744c7)
      3ba7c6ac
    • qetu3790's avatar
      Release clients blocked on module commands in cluster resharding and down state (#9483) · 936ee017
      qetu3790 authored
      
      
      Prevent clients from being blocked forever in cluster when they block with their own module command
      and the hash slot is migrated to another master at the same time.
      These will get a redirection message when unblocked.
      Also, release clients blocked on module commands when cluster is down (same as other blocked clients)
      
      This commit adds basic tests for the main (non-cluster) redis test infra that test the cluster.
      This was done because the cluster test infra can't handle some common test features,
      but most importantly we only build the test modules with the non-cluster test suite.
      
      note that rather than really supporting cluster operations by the test infra, it was added (as dup code)
      in two files, one for module tests and one for non-modules tests, maybe in the future we'll refactor that.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 4962c552)
      936ee017
    • Huang Zhw's avatar
      Make tracking invalidation messages always after command's reply (#9422) · 0fb96d55
      Huang Zhw authored
      Tracking invalidation messages were sometimes sent in inconsistent order,
      before the command's reply rather than after.
      In addition to that, they were sometimes embedded inside other commands
      responses, like MULTI-EXEC and MGET.
      
      (cherry picked from commit fd135f3e)
      0fb96d55
    • Binbin's avatar
      Add missing pause tcl test to test_helper.tcl (#9158) · f5d8a369
      Binbin authored
      * Add keyname tags to avoid CROSSSLOT errors in external server CI
      * Use new wait_for_blocked_clients_count in pause.tcl
      
      (cherry picked from commit 5dddf496)
      f5d8a369
    • meir's avatar
      Protect any table which is reachable from globals and added globals allow list. · 11b602fb
      meir authored
      The allow list is done by setting a metatable on the global table before initializing
      any library. The metatable set the `__newindex` field to a function that check
      the allow list before adding the field to the table. Fields which is not on the
      allow list are simply ignored.
      
      After initialization phase is done we protect the global table and each table
      that might be reachable from the global table. For each table we also protect
      the table metatable if exists.
      11b602fb
    • meir's avatar
      Protect globals of evals scripts. · b2ce3719
      meir authored
      Use the new `lua_enablereadonlytable` Lua API to protect the global tables of
      evals scripts. The implemetation is easy, we simply call `lua_enablereadonlytable`
      on the global table to turn it into a readonly table.
      b2ce3719