1. 29 Jun, 2021 2 commits
  2. 28 Jun, 2021 1 commit
    • Wang Yuan's avatar
      Remove unnecessary replication backlog memory copy (#9157) · 4fa3e230
      Wang Yuan authored
      in the past, the reply list was a list of sds objects, so this didn't have any overhead,
      but now addReplySds just copies the data from the sds and frees it, so there's no
      need to make a copy of the buffer before copying again.
      this reduces an excessive allocation and free and a memcpy.
      4fa3e230
  3. 27 Jun, 2021 1 commit
  4. 26 Jun, 2021 1 commit
  5. 24 Jun, 2021 5 commits
    • Yossi Gottlieb's avatar
      Add bind-source-addr configuration argument. (#9142) · f233c4c5
      Yossi Gottlieb authored
      In the past, the first bind address that was explicitly specified was
      also used to bind outgoing connections. This could result with some
      problems. For example: on some systems using `bind 127.0.0.1` would
      result with outgoing connections also binding to `127.0.0.1` and failing
      to connect to remote addresses.
      
      With the recent change to the way `bind` is handled, this presented
      other issues:
      
      * The default first bind address is '*' which is not a valid address.
      * We make no distinction between user-supplied config that is identical
      to the default, and the default config.
      
      This commit addresses both these issues by introducing an explicit
      configuration parameter to control the bind address on outgoing
      connections.
      f233c4c5
    • Huang Zhw's avatar
      Clean redis-benchmark Throughput output. (#9139) · d1a21e02
      Huang Zhw authored
      some leftovers from print are visible when the new line is printed.
      d1a21e02
    • ZhaolongLi's avatar
      change streamAppendItem to use raxEOF instead of raxNext (#9138) · 89ae3537
      ZhaolongLi authored
      The call to raxNext didn't really progress in the rax, since we were already on the last item.
      instead, all it does is check that it is indeed a valid item, so the new code clearer.
      89ae3537
    • Oran Agra's avatar
      Re-fix daily CI (#9141) · a5bc54f0
      Oran Agra authored
      The freebsd and macos jobs were still broken.
      also add a few more skip jobs options.
      a5bc54f0
    • Oran Agra's avatar
      5ffdbae1
  6. 23 Jun, 2021 2 commits
  7. 22 Jun, 2021 10 commits
    • Oran Agra's avatar
      Add manual triggers for Daily CI (#9119) · b466471a
      Oran Agra authored
      Hopefully now we'll be able to manually trigger extensive tests to PR branches without modifying them.
      b466471a
    • Oran Agra's avatar
      Adjustments to recent RM_StringTruncate fix (#3718) (#9125) · ae418eca
      Oran Agra authored
      - Introduce a new sdssubstr api as a building block for sdsrange.
        The API of sdsrange is many times hard to work with and also has
        corner case that cause bugs. sdsrange is easy to work with and also
        simplifies the implementation of sdsrange.
      - Revert the fix to RM_StringTruncate and just use sdssubstr instead of
        sdsrange.
      - Solve valgrind warnings from the new tests introduced by the previous
        PR.
      ae418eca
    • Yossi Gottlieb's avatar
      a49b7668
    • Yossi Gottlieb's avatar
      Fix typo in test. (#9128) · 8284544a
      Yossi Gottlieb authored
      8284544a
    • Yossi Gottlieb's avatar
      Improve bind and protected-mode config handling. (#9034) · 07b0d144
      Yossi Gottlieb authored
      * Specifying an empty `bind ""` configuration prevents Redis from listening on any TCP port. Before this commit, such configuration was not accepted.
      * Using `CONFIG GET bind` will always return an explicit configuration value. Before this commit, if a bind address was not specified the returned value was empty (which was an anomaly).
      
      Another behavior change is that modifying the `bind` configuration to a non-default value will NO LONGER DISABLE protected-mode implicitly.
      07b0d144
    • Evan's avatar
      modules: Add newlen == 0 handling to RM_StringTruncate (#3717) (#3718) · 1ccf2ca2
      Evan authored
      Previously, passing 0 for newlen would not truncate the string at all.
      This adds handling of this case, freeing the old string and creating a new empty string.
      
      Other changes:
      - Move `src/modules/testmodule.c` to `tests/modules/basics.c`
      - Introduce that basic test into the test suite
      - Add tests to cover StringTruncate
      - Add `test-modules` build target for the main makefile
      - Extend `distclean` build target to clean modules too
      1ccf2ca2
    • Oran Agra's avatar
      solve test timing issues in replication tests (#9121) · d0819d61
      Oran Agra authored
      # replication-3.tcl
      had a test timeout failure with valgrind on daily CI:
      ```
      *** [err]: SLAVE can reload "lua" AUX RDB fields of duplicated scripts in tests/integration/replication-3.tcl
      Replication not started.
      ```
      replication took more than 70 seconds.
      https://github.com/redis/redis/runs/2854037905?check_suite_focus=true
      
      on my machine it takes only about 30, but i can see how 50 seconds isn't enough.
      
      # replication.tcl
      loading was over too quickly in freebsd daily CI:
      ```
      *** [err]: slave fails full sync and diskless load swapdb recovers it in tests/integration/replication.tcl
      Expected '0' to be equal to '1' (context: type eval line 44 cmd {assert_equal [s -1 loading] 1} proc ::start_server)
      ```
      
      # rdb.tcl
      loading was over too quickly.
      increase the time loading takes, and decrease the amount of work we try to achieve in that time.
      d0819d61
    • Binbin's avatar
      Remove extra semicolon (#9117) · bf92000e
      Binbin authored
      Remove extra semicolon.
      bf92000e
    • Wen Hui's avatar
      add tilt mode since in sentinel info (#9000) · 81d5f05b
      Wen Hui authored
      Sentinel shows tilt mode boolean in info, now it'll show an indication of how long ago it started.
      81d5f05b
    • Oran Agra's avatar
      Fix race in client side tracking (#9116) · 9b564b52
      Oran Agra authored
      The `Tracking gets notification of expired keys` test in tracking.tcl
      used to hung in valgrind CI quite a lot.
      
      It turns out the reason is that with valgrind and a busy machine, the
      server cron active expire cycle could easily run in the same event loop
      as the command that created `mykey`, so that when they key got expired,
      there were two change events to broadcast, one that set the key and one
      that expired it, but since we used raxTryInsert, the client that was
      associated with the "last" change was the one that created the key, so
      the NOLOOP filtered that event.
      
      This commit adds a test that reproduces the problem by using lazy expire
      in a multi-exec which makes sure the key expires in the same event loop
      as the one that added it.
      9b564b52
  8. 21 Jun, 2021 2 commits
  9. 20 Jun, 2021 4 commits
  10. 17 Jun, 2021 4 commits
    • gourav's avatar
    • sundb's avatar
      Make readQueryFromClient more aggressive when reading big arg again (Followup for #9003) (#9100) · 1a22445d
      sundb authored
      Due to the change in #9003, a long-standing bug was raised under `valgrind`.
      This bug can cause the master-slave sync to take a very long time, causing the `pendingquerybuf.tcl` test to fail.
      This problem does not only occur in master-slave sync, it is triggered when the big arg is greater than 32k.
      step:
      ```sh
      dd if=/dev/zero of=bigfile bs=1M count=32
      ./src/redis-cli -x hset a a < bigfile
      ```
      
      1) Make room for querybuf in processMultibulkBuffer, now the alloc of querybuf will be more than 32k.
      2) If this happens to trigger the `clientsCronResizeQueryBuffer`, querybuf will be resized to 0.
      3) Finally, in readQueryFromClient, we expand the querybuf non-greedily, from 0 to 32k.
          Old code, make room for querybuf is greedy, so it only needs 11 times to expand to 32M(16k*(2^11)),
          but now we need 2048(32*1024/16) times to reach it, due to the slow allocation under valgrind that exposed the problem.
      
      The fix for the excessive shrinking of the query buf to 0, will be handled in #5013 (that other change on it's own can fix failing test too), but the fix in this PR will also fix the failing test.
      
      The fix in this PR will makes the reading in `readQueryFromClient` more aggressive when working on a big arg (so that it is in par with the same code in `processMultibulkBuffer` (i.e. the two calls to `sdsMakeRoomForNonGreedy` should both use the bulk size).
      In the code before this fix the one in readQueryFromClient always has `readlen = PROTO_IOBUF_LEN`
      1a22445d
    • sundb's avatar
      Fix leak and double free issues in datatype2 module test (#9102) · eae0983d
      sundb authored
      * Add missing call for RedisModule_DictDel in datatype2 test
      * Fix memory leak in datatype2 test
      eae0983d
    • DarrenJiang13's avatar
      Improve objectComputeSize() with allocator fragmentaiton. (#9095) · ada60d80
      DarrenJiang13 authored
      This commit improve MEMORY USAGE command to include internal fragmentation overheads of:
      1. EMBSTR encoded strings
      2. ziplist encoded zsets and hashes
      3. List type nodes
      ada60d80
  11. 16 Jun, 2021 6 commits
    • sundb's avatar
      Fix querybuf test failure (#9091) · b586d5b5
      sundb authored
      Fix test failure which introduced by #9003.
      The following case will occur when querybuf expansion will allocate memory equal to (16*1024)k.
      1) make use ```CFLAGS=-DNO_MALLOC_USABLE_SIZE```.
      2) ```malloc``` will not allocate more under ```alpine```.
      b586d5b5
    • Sam Bortman's avatar
      Support glob pattern matching for config include files (#8980) · c2b93ff8
      Sam Bortman authored
      This will allow distros to use an "include conf.d/*.conf" statement in the default configuration file
      which will facilitate customization across upgrades/downgrades.
      
      The change itself is trivial: instead of opening an individual file, the glob call creates a vector of files to open, and each file is opened in turn, and its content is added to the configuration.
      c2b93ff8
    • yoav-steinberg's avatar
      Remove gopher protocol support. (#9057) · 362786c5
      yoav-steinberg authored
      Gopher support was added mainly because it was simple (trivial to add).
      But apparently even something that was trivial at the time, does cause complications
      down the line when adding more features.
      We recently ran into a few issues with io-threads conflicting with the gopher support.
      We had to either complicate the code further in order to solve them, or drop gopher.
      AFAIK it's completely unused, so we wanna chuck it, rather than keep supporting it.
      362786c5
    • chenyang8094's avatar
      Enhance mem_usage/free_effort/unlink/copy callbacks and add GetDbFromIO api. (#8999) · e0cd3ad0
      chenyang8094 authored
      Create new module type enhanced callbacks: mem_usage2, free_effort2, unlink2, copy2.
      These will be given a context point from which the module can obtain the key name and database id.
      In addition the digest and defrag context can now be used to obtain the key name and database id.
      e0cd3ad0
    • Jason Elbaum's avatar
      Change return value type for ZPOPMAX/MIN in RESP3 (#8981) · 7f342020
      Jason Elbaum authored
      When using RESP3, ZPOPMAX/ZPOPMIN should return nested arrays for consistency
      with other commands (e.g. ZRANGE).
      
      We do that only when COUNT argument is present (similarly to how LPOP behaves).
      for reasoning see https://github.com/redis/redis/issues/8824#issuecomment-855427955
      
      This is a breaking change only when RESP3 is used, and COUNT argument is present!
      7f342020
    • Uri Shachar's avatar
      Cleaning up the cluster interface by moving almost all related declar… (#9080) · c7e502a0
      Uri Shachar authored
      * Cleaning up the cluster interface by moving almost all related declarations into cluster.h
      (no logic change -- just moving declarations/definitions around)
      
      This initial effort leaves two items out of scope - the configuration parsing into the server
      struct and the internals exposed by the clusterNode struct.
      
      * Remove unneeded declarations of dictSds*
      Ideally all the dictSds functionality would move from server.c into a dedicated module
      so we can avoid the duplication in redis-benchmark/cli
      
      * Move crc16 back into server.h, will be moved out once we create a seperate header file for
      hashing functions
      c7e502a0
  12. 15 Jun, 2021 2 commits
    • sundb's avatar
      Fix the wrong reisze of querybuf (#9003) · e5d8a5eb
      sundb authored
      The initialize memory of `querybuf` is `PROTO_IOBUF_LEN(1024*16) * 2` (due to sdsMakeRoomFor being greedy), under `jemalloc`, the allocated memory will be 40k.
      This will most likely result in the `querybuf` being resized when call `clientsCronResizeQueryBuffer` unless the client requests it fast enough.
      
      Note that this bug existed even before #7875, since the condition for resizing includes the sds headers (32k+6).
      
      ## Changes
      1. Use non-greedy sdsMakeRoomFor when allocating the initial query buffer (of 16k).
      1. Also use non-greedy allocation when working with BIG_ARG (we won't use that extra space anyway)
      2. in case we did use a greedy allocation, read as much as we can into the buffer we got (including internal frag), to reduce system calls.
      3. introduce a dedicated constant for the shrinking (same value as before)
      3. Add test for querybuf.
      4. improve a maxmemory test by ignoring the effect of replica query buffers (can accumulate many ACKs on slow env)
      5. improve a maxmemory by disabling slowlog (it will cause slight memory growth on slow env).
      e5d8a5eb
    • Binbin's avatar
      Check `CLIENT_DIRTY_CAS` flag before process transaction. (#9086) · eb15c456
      Binbin authored
      Do not queue command in an already aborted MULTI state.
      We can detect an error (watched key).
      So in queueMultiCommand, we also can return early.
      Like we deal with `CLIENT_DIRTY_EXEC`.
      eb15c456