1. 08 Mar, 2023 14 commits
  2. 07 Mar, 2023 3 commits
    • Madelyn Olson's avatar
      Always compact nodes in stream listpacks after creating new nodes (#11885) · 2bb29e4a
      Madelyn Olson authored
      This change attempts to alleviate a minor memory usage degradation for Redis 6.2 and onwards when using rather large objects (~2k) in streams. Introduced in #6281, we pre-allocate the head nodes of a stream to be 4kb, to limit the amount of unnecessary initial reallocations that are done. However, if we only ever allocate one object because 2 objects exceeds the max_stream_entry_size, we never actually shrink it to fit the single item. This can lead to a lot of excessive memory usage. For smaller item sizes this becomes less of an issue, as the overhead decreases as the items become smaller in size.
      
      This commit also changes the MEMORY USAGE of streams, since it was reporting the lpBytes instead of the allocated size. This introduced an observability issue when diagnosing the memory issue, since Redis reported the same amount of used bytes pre and post change, even though the new implementation allocated more memory.
      2bb29e4a
    • Binbin's avatar
      Solve race in CLIENT NO-TOUCH lru test (#11883) · 9958ab8b
      Binbin authored
      I've seen it fail here (test-centos7-tls-module-no-tls and test-freebsd):
      ```
      *** [err]: Operations in no-touch mode do not alter the last access time of a key in tests/unit/introspection-2.tcl
      Expected '244296' to be more than '244296' (context: type eval line 12 cmd {assert_morethan $newlru $oldlru} proc ::test)
      ```
      
      Our LRU_CLOCK_RESOLUTION value is 1000ms, and default hz is 10, so if the
      test is really fast, or the timing is just right, newlru will be the same
      as oldlru. We fixed this by changing `after 1000` to `after 1100`.
      9958ab8b
    • sundb's avatar
      Skip test for sdsRemoveFreeSpace when mem_allocator is not jemalloc (#11878) · 3fba3ccd
      sundb authored
      Test `trim on SET with big value` (introduced from #11817) fails under mac m1 with libc mem_allocator.
      The reason is that malloc(33000) will allocate 65536 bytes(>42000).
      This test still passes under ubuntu with libc mem_allocator.
      
      ```
      *** [err]: trim on SET with big value in tests/unit/type/string.tcl
      Expected [r memory usage key] < 42000 (context: type source line 471 file /Users/iospack/data/redis_fork/tests/unit/type/string.tcl cmd {assert {[r memory usage key] < 42000}} proc ::test)
      ```
      
      simple test under mac m1 with libc mem_allocator:
      ```c
      void *p = zmalloc(33000);
      printf("malloc size: %zu\n", zmalloc_size(p));
      
      # output
      malloc size: 65536
      ```
      3fba3ccd
  3. 05 Mar, 2023 1 commit
  4. 04 Mar, 2023 1 commit
    • Binbin's avatar
      Increase the threshold of the AOF loading defrag test (#11871) · bfe50a30
      Binbin authored
      This test is very sensitive and fragile. It often fails in Daily,
      in most cases, it failed in test-ubuntu-32bit (the AOF loading one),
      with the range in (31, 40):
      ```
      [err]: Active defrag in tests/unit/memefficiency.tcl
      Expected 38 <= 30 (context: type eval line 113 cmd {assert {$max_latency <= 30}} proc ::test)
      ```
      
      The AOF loading part isn't tightly fixed to the cron hz. It calls
      processEventsWhileBlocked once in every 1024 command calls.
      ```
              /* Serve the clients from time to time */
              if (!(loops++ % 1024)) {
                  off_t progress_delta = ftello(fp) - last_progress_report_size;
                  loadingIncrProgress(progress_delta);
                  last_progress_report_size += progress_delta;
                  processEventsWhileBlocked();
                  processModuleLoadingProgressEvent(1);
              }
      ```
      
      In this case, we can either decrease the 1024 or increase the
      threshold of just the AOF part of that test. Considering the test
      machines are sometimes slow, and all sort of quirks could happen
      (which do not indicate a bug), and we've already set to 30, we suppose
      we can set it a little bit higher, set it to 40. We can have this instead of
      adding another testing config (we can add it when we really need it).
      
      Fixes #11868
      bfe50a30
  5. 03 Mar, 2023 1 commit
  6. 28 Feb, 2023 4 commits
  7. 26 Feb, 2023 1 commit
    • ranshid's avatar
      assert in case resize output buffer will attempt to shrink too much (#11839) · 4972760b
      ranshid authored
      Currently there is no BUG. However during some internal code changes
      I found that it can happen (for example in case new code will not update
      the buf_peak) which can currently lead to memory overrun which is much
      harder to detect and root cause.
      
      Why did I please the assert here? The reason is to be able to have the
      buf_peak value without the risk of it being overriden by the peak_reset
      4972760b
  8. 23 Feb, 2023 3 commits
  9. 21 Feb, 2023 3 commits
    • Binbin's avatar
      Speed up test: client evicted due to client tracking prefixes (#11823) · cd58af4d
      Binbin authored
      We noticed that `client evicted due to client tracking prefixes`
      takes over 200 seconds with valgrind.
      
      We combine three prefixes in each command, this will probably
      save us half the testing time.
      
      Before: normal: 3508ms, valgrind: 289503ms -> 290s
      With three prefixes, normal: 1500ms, valgrind: 135742ms -> 136s
      
      Since we did not actually count the memory usage of all prefixes, see
      getClientMemoryUsage, so we can not use larger prefixes to speed up the
      test here. Also this PR cleaned up some spaces (IDE jobs) and typos.
      cd58af4d
    • Madelyn Olson's avatar
      Prevent Redis from crashing from key tracking invalidations (#11814) · dca5927a
      Madelyn Olson authored
      There is a built in limit to client side tracking keys, which when exceeded will invalidate keys. This occurs in two places, one in the server cron and other before executing a command. If it happens in the second scenario, the invalidations will be queued for later since current client is set. This queue is never drained if a command is not executed (through call) such as a multi-exec command getting queued. This results in a later server assert crashing.
      dca5927a
    • M Sazzadul Hoque's avatar
      Fix HELLO error message command syntax suggestion (#11809) · 4cc2b0dc
      M Sazzadul Hoque authored
      A simple HELLO command to a password protected Redis server replies
      with an error with another command suggestion. This omits protocol version
      from HELLO command arguments which causes another error.
      This PR adds the protocol version in the command suggestion.
      4cc2b0dc
  10. 20 Feb, 2023 1 commit
  11. 19 Feb, 2023 2 commits
  12. 16 Feb, 2023 3 commits
    • Oran Agra's avatar
      skip new page cache reclame unit test when running in valgrind (#11808) · 5b61b0dc
      Oran Agra authored
      the new test is incompatible with valgrind.
      added a new `--valgrind` argument to `redis-server tests` mode,
      which will cause that test to be skipped..
      5b61b0dc
    • Oran Agra's avatar
      Cleanup around script_caller, fix tracking of scripts and ACL logging for RM_Call (#11770) · 233abbbe
      Oran Agra authored
      * Make it clear that current_client is the root client that was called by
        external connection
      * add executing_client which is the client that runs the current command
        (can be a module or a script)
      * Remove script_caller that was used for commands that have CLIENT_SCRIPT
        to get the client that called the script. in most cases, that's the current_client,
        and in others (when being called from a module), it could be an intermediate
        client when we actually want the original one used by the external connection.
      
      bugfixes:
      * RM_Call with C flag should log ACL errors with the requested user rather than
        the one used by the original client, this also solves a crash when RM_Call is used
        with C flag from a detached thread safe context.
      * addACLLogEntry would have logged info about the script_caller, but in case the
        script was issued by a module command we actually want the current_client. the
        exception is when RM_Call is called from a timer event, in which case we don't
        have a current_client.
      
      behavior changes:
      * client side tracking for scripts now tracks the keys that are read by the script
        instead of the keys that are declared by the caller for EVAL
      
      other changes:
      * Log both current_client and executing_client in the crash log.
      * remove prepareLuaClient and resetLuaClient, being dead code that was forgotten.
      * remove scriptTimeSnapshot and snapshot_time and instead add cmd_time_snapshot
        that serves all commands and is reset only when execution nesting starts.
      * remove code to propagate CLIENT_FORCE_REPL from the executed command
        to the script caller since scripts aren't propagated anyway these days and anyway
        this flag wouldn't have had an effect since CLIENT_PREVENT_PROP is added by scriptResetRun.
      * fix a module GIL violation issue in afterSleep that was introduced in #10300 (unreleased)
      233abbbe
    • zhaozhao.zz's avatar
      a35e0837
  13. 15 Feb, 2023 1 commit
    • Binbin's avatar
      Remove wrong code in list pot timeout test (#11805) · 7d5382c0
      Binbin authored
      In #9373, actually need to replace `$rd $pop blist1{t} blist2{t} 1`
      with `bpop_command_two_key $rd $pop blist1{t} blist2{t} 1` but forgot
      to delete the latter.
      
      This doesn't affect the test, because the later assert_error "WRONGTYPE"
      is expected (and right). And if we read $rd again, it will get the
      wrong result, like 'ERR unknown command 'BLMPOP_LEFT' | 'BLMPOP_RIGHT'
      7d5382c0
  14. 14 Feb, 2023 2 commits
    • Wen Hui's avatar
      Update codes (#11804) · a7051845
      Wen Hui authored
      In this PR, we use function pointer *isPresent replace the variable "present" in auxFieldHandler, so that in the future, when we have more aux fields, we could decide if the aux field is displayed or not.
      a7051845
    • guybe7's avatar
      Minor changes around the blockonkeys test module (#11803) · 9483ab0b
      guybe7 authored
      All of the POP commands must not decr length below 0.
      So, get_fsl will delete the key if the length is 0 (unless
      the caller wished to create if doesn't exist)
      
      Other:
      1. Use REDISMODULE_WRITE where needed (POP commands)
      2. Use wait_for_blokced_clients in test
      
      Unrelated:
      Use quotes instead of curly braces in zset.tcl, for variable expansion
      9483ab0b