1. 18 Oct, 2021 2 commits
    • Oran Agra's avatar
      Merge pull request #9623 from yoav-steinberg/upgrade_jemalloc_5.2.1 · c4b4b6c0
      Oran Agra authored
      Upgraded to jemalloc 5.2.1 from 5.1.0.
      Cherry picked all relevant fixes (by diffing our 5.1.0 to upstream 5.10 and finding relevant commits).
      Details of what was done:
      
      [cherry-picked] fd7d51c3 2021-05-03 Resolve nonsense static analysis warnings (Oran Agra)
      [cherry-picked] 448c435b 2020-09-29 Fix compilation warnings in Lua and jemalloc dependencies (#7785) (YoongHM)
      [skipped - already in upstream] 9216b96b 2020-09-21 Fix compilation warning in jemalloc's malloc_vsnprintf (#7789) (YoongHM)
      [cherry-picked] 88d71f47 2020-05-20 fix a rare active defrag edge case bug leading to stagnation (Oran Agra)
      [skipped - already in upstream] 2fec7d9c 2019-05-30 Jemalloc: Avoid blocking on background thread lock for stats.
      [cherry-picked] 920158ec 2018-07-11 Active defrag fixes for 32bit builds (again) (Oran Agra)
      [cherry-picked] e8099cab 2018-06-26 add defrag hint support into jemalloc 5 (Oran Agra)
      [re-done] 4e729fcd 2018-05-24 Generate configure for Jemalloc. (antirez)
      
      Additionally had to do this:
      7727cc2 2021-10-10 Fix defrag to support sharded bins in arena (added in v5.2.1) (Yoav Steinberg)
      
      When reviewing please look at all except the first commit which is just replacing 5.1.0 with 5.2.1 sources.
      Also I think we should merge this without squashing to preserve the changes we did to to jemalloc.
      c4b4b6c0
    • Oran Agra's avatar
      Attempt to fix a valgrind test failure due to timing (#9643) · 276b460e
      Oran Agra authored
      in the past few days i've seen two failures in the valgrind daily test.
      
      *** [err]: slave fails full sync and diskless load swapdb recovers it in tests/integration/replication.tcl
      Replica didn't get into loading mode
      
      can't reproduce it, but i'm hoping it's just too slow (to start loading within 5 seconds)
      276b460e
  2. 17 Oct, 2021 4 commits
  3. 16 Oct, 2021 1 commit
  4. 15 Oct, 2021 1 commit
  5. 14 Oct, 2021 1 commit
  6. 13 Oct, 2021 2 commits
  7. 12 Oct, 2021 6 commits
  8. 11 Oct, 2021 2 commits
  9. 10 Oct, 2021 8 commits
  10. 08 Oct, 2021 2 commits
  11. 07 Oct, 2021 3 commits
  12. 06 Oct, 2021 5 commits
    • Andy Pan's avatar
      Implement anetPipe() to combine creating pipe and setting flags (#9511) · 2391aefd
      Andy Pan authored
      
      
      Implement createPipe() to combine creating pipe and setting flags, also reduce
      system calls by prioritizing pipe2() over pipe().
      
      Without createPipe(), we have to call pipe() to create a pipe and then call some
      functions (like anetCloexec() and anetNonBlock()) of anet.c to set flags respectively,
      which leads to some extra system calls, now we can leverage pipe2() to combine
      them and make the process of creating pipe more convergent in createPipe().
      Co-authored-by: default avatarViktor Söderqvist <viktor.soderqvist@est.tech>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      2391aefd
    • yoav-steinberg's avatar
      Test fails when flushdb triggers a bgsave (#9535) · 123cc1a1
      yoav-steinberg authored
      Flush db and *then* wait for the bgsave to complete.
      123cc1a1
    • yoav-steinberg's avatar
      Avoid argv memcpy when queuing a multi command. (#9602) · 5725088f
      yoav-steinberg authored
      When queuing a multi command we duplicated the argv (meaning an alloc
      and a memcpy). This isn't needed since we can use the previously allocated
      argv and just reset the client objects argv to NULL. This should saves some
      memory and is a minor optimization in heavy MULTI/EXEC traffic, especially
      if there are lots of arguments.
      5725088f
    • Meir Shpilraien (Spielrein)'s avatar
      Added module-acquire-GIL latency stats (#9608) · 4fb39b67
      Meir Shpilraien (Spielrein) authored
      The new value indicates how long Redis wait to
      acquire the GIL after sleep. This can help identify
      problems where a module perform some background
      operation for a long time (with the GIL held) and
      blocks the Redis main thread.
      4fb39b67
    • tzongw's avatar
      improve latency when a client is unblocked by module timer (#9593) · f5160ed0
      tzongw authored
      Scenario:
      1. client block on command `XREAD BLOCK 0 STREAMS mystream  $`
      2. in a module, calling `XADD mystream * field value` via lua from a timer callback
      3. client will receive response after some latency up to 100ms
      
      Reason:
      When `XADD` signal the key `mystream` as ready, `beforeSleep` in next eventloop will call
      `handleClientsBlockedOnKeys` to unblock the client and add pending data to write but not
      actually install a write handler, so next redis will block in `aeApiPoll` up to 100ms given `hz`
      config as default 10, pending data will be sent in another next eventloop by
      `handleClientsWithPendingWritesUsingThreads`.
      
      Calling `handleClientsBlockedOnKeys` before `handleClientsWithPendingWritesUsingThreads`
      in `beforeSleep` solves the problem.
      f5160ed0
  13. 05 Oct, 2021 2 commits
  14. 04 Oct, 2021 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Fix invalid memory write on lua stack overflow (CVE-2021-32626) (#9591) · 0f8b634c
      Meir Shpilraien (Spielrein) authored
      When LUA call our C code, by default, the LUA stack has room for 10
      elements. In most cases, this is more than enough but sometimes it's not
      and the caller must verify the LUA stack size before he pushes elements.
      
      On 3 places in the code, there was no verification of the LUA stack size.
      On specific inputs this missing verification could have lead to invalid
      memory write:
      1. On 'luaReplyToRedisReply', one might return a nested reply that will
         explode the LUA stack.
      2. On 'redisProtocolToLuaType', the Redis reply might be deep enough
         to explode the LUA stack (notice that currently there is no such
         command in Redis that returns such a nested reply, but modules might
         do it)
      3. On 'ldbRedis', one might give a command with enough arguments to
         explode the LUA stack (all the arguments will be pushed to the LUA
         stack)
      
      This commit is solving all those 3 issues by calling 'lua_checkstack' and
      verify that there is enough room in the LUA stack to push elements. In
      case 'lua_checkstack' returns an error (there is not enough room in the
      LUA stack and it's not possible to increase the stack), we will do the
      following:
      1. On 'luaReplyToRedisReply', we will return an error to the user.
      2. On 'redisProtocolToLuaType' we will exit with panic (we assume this
         scenario is rare because it can only happen with a module).
      3. On 'ldbRedis', we return an error.
      0f8b634c