1. 20 Oct, 2021 3 commits
    • Yossi Gottlieb's avatar
      8bf4c2e3
    • Oran Agra's avatar
      fix new cluster tests issues (#9657) · 7d6744c7
      Oran Agra authored
      Following #9483 the daily CI exposed a few problems.
      
      * The cluster creation code (uses redis-cli) is complicated to test with TLS enabled.
        for now i'm just skipping them since the tests we run there don't really need that kind of coverage
      * cluster port binding failures
        note that `find_available_port` already looks for a free cluster port
        but the code in `wait_server_started` couldn't detect the failure of binding
        (the text it greps for wasn't found in the log)
      7d6744c7
    • guybe7's avatar
      Treat subcommands as commands (#9504) · 43e736f7
      guybe7 authored
      ## Intro
      
      The purpose is to allow having different flags/ACL categories for
      subcommands (Example: CONFIG GET is ok-loading but CONFIG SET isn't)
      
      We create a small command table for every command that has subcommands
      and each subcommand has its own flags, etc. (same as a "regular" command)
      
      This commit also unites the Redis and the Sentinel command tables
      
      ## Affected commands
      
      CONFIG
      Used to have "admin ok-loading ok-stale no-script"
      Changes:
      1. Dropped "ok-loading" in all except GET (this doesn't change behavior since
      there were checks in the code doing that)
      
      XINFO
      Used to have "read-only random"
      Changes:
      1. Dropped "random" in all except CONSUMERS
      
      XGROUP
      Used to have "write use-memory"
      Changes:
      1. Dropped "use-memory" in all except CREATE and CREATECONSUMER
      
      COMMAND
      No changes.
      
      MEMORY
      Used to have "random read-only"
      Changes:
      1. Dropped "random" in PURGE and USAGE
      
      ACL
      Used to have "admin no-script ok-loading ok-stale"
      Changes:
      1. Dropped "admin" in WHOAMI, GENPASS, and CAT
      
      LATENCY
      No changes.
      
      MODULE
      No changes.
      
      SLOWLOG
      Used to have "admin random ok-loading ok-stale"
      Changes:
      1. Dropped "random" in RESET
      
      OBJECT
      Used to have "read-only random"
      Changes:
      1. Dropped "random" in ENCODING and REFCOUNT
      
      SCRIPT
      Used to have "may-replicate no-script"
      Changes:
      1. Dropped "may-replicate" in all except FLUSH and LOAD
      
      CLIENT
      Used to have "admin no-script random ok-loading ok-stale"
      Changes:
      1. Dropped "random" in all except INFO and LIST
      2. Dropped "admin" in ID, TRACKING, CACHING, GETREDIR, INFO, SETNAME, GETNAME, and REPLY
      
      STRALGO
      No changes.
      
      PUBSUB
      No changes.
      
      CLUSTER
      Changes:
      1. Dropped "admin in countkeysinslots, getkeysinslot, info, nodes, keyslot, myid, and slots
      
      SENTINEL
      No changes.
      
      (note that DEBUG also fits, but we decided not to convert it since it's for
      debugging and anyway undocumented)
      
      ## New sub-command
      This commit adds another element to the per-command output of COMMAND,
      describing the list of subcommands, if any (in the same structure as "regular" commands)
      Also, it adds a new subcommand:
      ```
      COMMAND LIST [FILTERBY (MODULE <module-name>|ACLCAT <cat>|PATTERN <pattern>)]
      ```
      which returns a set of all commands (unless filters), but excluding subcommands.
      
      ## Module API
      A new module API, RM_CreateSubcommand, was added, in order to allow
      module writer to define subcommands
      
      ## ACL changes:
      1. Now, that each subcommand is actually a command, each has its own ACL id.
      2. The old mechanism of allowed_subcommands is redundant
      (blocking/allowing a subcommand is the same as blocking/allowing a regular command),
      but we had to keep it, to support the widespread usage of allowed_subcommands
      to block commands with certain args, that aren't subcommands (e.g. "-select +select|0").
      3. I have renamed allowed_subcommands to allowed_firstargs to emphasize the difference.
      4. Because subcommands are commands in ACL too, you can now use "-" to block subcommands
      (e.g. "+client -client|kill"), which wasn't possible in the past.
      5. It is also possible to use the allowed_firstargs mechanism with subcommand.
      For example: `+config -config|set +config|set|loglevel` will block all CONFIG SET except
      for setting the log level.
      6. All of the ACL changes above required some amount of refactoring.
      
      ## Misc
      1. There are two approaches: Either each subcommand has its own function or all
         subcommands use the same function, determining what to do according to argv[0].
         For now, I took the former approaches only with CONFIG and COMMAND,
         while other commands use the latter approach (for smaller blamelog diff).
      2. Deleted memoryGetKeys: It is no longer needed because MEMORY USAGE now uses the "range" key spec.
      4. Bugfix: GETNAME was missing from CLIENT's help message.
      5. Sentinel and Redis now use the same table, with the same function pointer.
         Some commands have a different implementation in Sentinel, so we redirect
         them (these are ROLE, PUBLISH, and INFO).
      6. Command stats now show the stats per subcommand (e.g. instead of stats just
         for "config" you will have stats for "config|set", "config|get", etc.)
      7. It is now possible to use COMMAND directly on subcommands:
         COMMAND INFO CONFIG|GET (The pipeline syntax was inspired from ACL, and
         can be used in functions lookupCommandBySds and lookupCommandByCString)
      8. STRALGO is now a container command (has "help")
      
      ## Breaking changes:
      1. Command stats now show the stats per subcommand (see (5) above)
      43e736f7
  2. 19 Oct, 2021 3 commits
    • qetu3790's avatar
      Release clients blocked on module commands in cluster resharding and down state (#9483) · 4962c552
      qetu3790 authored
      
      
      Prevent clients from being blocked forever in cluster when they block with their own module command
      and the hash slot is migrated to another master at the same time.
      These will get a redirection message when unblocked.
      Also, release clients blocked on module commands when cluster is down (same as other blocked clients)
      
      This commit adds basic tests for the main (non-cluster) redis test infra that test the cluster.
      This was done because the cluster test infra can't handle some common test features,
      but most importantly we only build the test modules with the non-cluster test suite.
      
      note that rather than really supporting cluster operations by the test infra, it was added (as dup code)
      in two files, one for module tests and one for non-modules tests, maybe in the future we'll refactor that.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      4962c552
    • Bjorn Svensson's avatar
      Move config `unixsocketperm` to generic configs (#9607) · c9fabc2e
      Bjorn Svensson authored
      Since the size of mode_t is platform dependant we handle the
      `unixsocketperm` configuration as a generic int type.
      mode_t is either an unsigned int or unsigned short (macOS) and
      the range-limits allows for a simple cast to a mode_t.
      c9fabc2e
    • Wen Hui's avatar
      Make Cluster-bus port configurable with new cluster-port config (#9389) · 1c2b5f53
      Wen Hui authored
      Make Cluster-bus port configurable with new cluster-port config
      1c2b5f53
  3. 18 Oct, 2021 5 commits
    • Viktor Söderqvist's avatar
      Add RedisModule_KeyExists (#9600) · b7f2a1a2
      Viktor Söderqvist authored
      The LRU of the key is not touched. Locically expired keys are
      logically not existing, so they're treated as such.
      b7f2a1a2
    • DarrenJiang13's avatar
      add missed error counting (#9646) · aa6deff0
      DarrenJiang13 authored
      * add: add missed error counting in sentinel.c and cluster.c
      aa6deff0
    • yoav-steinberg's avatar
      Skip Active-defrag edge case test until we fix it. (#9645) · 81095b1b
      yoav-steinberg authored
      Test started failing consistently in 32bit builds after upgrading to jemalloc 5.2.1 (#9623).
      81095b1b
    • Oran Agra's avatar
      Merge pull request #9623 from yoav-steinberg/upgrade_jemalloc_5.2.1 · c4b4b6c0
      Oran Agra authored
      Upgraded to jemalloc 5.2.1 from 5.1.0.
      Cherry picked all relevant fixes (by diffing our 5.1.0 to upstream 5.10 and finding relevant commits).
      Details of what was done:
      
      [cherry-picked] fd7d51c3 2021-05-03 Resolve nonsense static analysis warnings (Oran Agra)
      [cherry-picked] 448c435b 2020-09-29 Fix compilation warnings in Lua and jemalloc dependencies (#7785) (YoongHM)
      [skipped - already in upstream] 9216b96b 2020-09-21 Fix compilation warning in jemalloc's malloc_vsnprintf (#7789) (YoongHM)
      [cherry-picked] 88d71f47 2020-05-20 fix a rare active defrag edge case bug leading to stagnation (Oran Agra)
      [skipped - already in upstream] 2fec7d9c 2019-05-30 Jemalloc: Avoid blocking on background thread lock for stats.
      [cherry-picked] 920158ec 2018-07-11 Active defrag fixes for 32bit builds (again) (Oran Agra)
      [cherry-picked] e8099cab 2018-06-26 add defrag hint support into jemalloc 5 (Oran Agra)
      [re-done] 4e729fcd 2018-05-24 Generate configure for Jemalloc. (antirez)
      
      Additionally had to do this:
      7727cc2 2021-10-10 Fix defrag to support sharded bins in arena (added in v5.2.1) (Yoav Steinberg)
      
      When reviewing please look at all except the first commit which is just replacing 5.1.0 with 5.2.1 sources.
      Also I think we should merge this without squashing to preserve the changes we did to to jemalloc.
      c4b4b6c0
    • Oran Agra's avatar
      Attempt to fix a valgrind test failure due to timing (#9643) · 276b460e
      Oran Agra authored
      in the past few days i've seen two failures in the valgrind daily test.
      
      *** [err]: slave fails full sync and diskless load swapdb recovers it in tests/integration/replication.tcl
      Replica didn't get into loading mode
      
      can't reproduce it, but i'm hoping it's just too slow (to start loading within 5 seconds)
      276b460e
  4. 17 Oct, 2021 4 commits
  5. 16 Oct, 2021 1 commit
  6. 15 Oct, 2021 1 commit
  7. 14 Oct, 2021 1 commit
  8. 13 Oct, 2021 2 commits
  9. 12 Oct, 2021 6 commits
  10. 11 Oct, 2021 2 commits
  11. 10 Oct, 2021 8 commits
  12. 08 Oct, 2021 2 commits
  13. 07 Oct, 2021 2 commits
    • yoav-steinberg's avatar
      obuf based eviction tests run until eviction occurs (#9611) · 834e8843
      yoav-steinberg authored
      obuf based eviction tests run until eviction occurs instead of assuming a certain
      amount of writes will fill the obuf enough for eviction to occur.
      This handles the kernel buffering written data and emptying the obuf even though
      no one actualy reads from it.
      
      The tests have a new timeout of 20sec: if the test doesn't pass after 20 sec it'll fail.
      Hopefully this enough for our slow CI targets.
      
      This also eliminates the need to skip some tests in TLS.
      834e8843
    • Huang Zhw's avatar
      Make tracking invalidation messages always after command's reply (#9422) · fd135f3e
      Huang Zhw authored
      Tracking invalidation messages were sometimes sent in inconsistent order,
      before the command's reply rather than after.
      In addition to that, they were sometimes embedded inside other commands
      responses, like MULTI-EXEC and MGET.
      fd135f3e