1. 29 Sep, 2022 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Avoid crash on crash report when a bad function pointer was called (#11298) · 0bf90d94
      Meir Shpilraien (Spielrein) authored
      If Redis crashes due to calling an invalid function pointer,
      the `backtrace` function will try to dereference this invalid pointer
      which will cause a crash inside the crash report and will kill
      the processes without having all the crash report information.
      
      Example:
      
      ```
      === REDIS BUG REPORT START: Cut & paste starting from here ===
      198672:M 19 Sep 2022 18:06:12.936 # Redis 255.255.255 crashed by signal: 11, si_code: 1
      198672:M 19 Sep 2022 18:06:12.936 # Accessing address: 0x1
      198672:M 19 Sep 2022 18:06:12.936 # Crashed running the instruction at: 0x1
      // here the processes is crashing
      ```
      
      This PR tries to fix this crash be:
      1. Identify the issue when it happened.
      2. Replace the invalid pointer with a pointer to some dummy function
         so that `backtrace` will not crash.
      
      I identification is done by comparing `eip` to `info->si_addr`, if they
      are the same we know that the crash happened on the same address it tries to
      accesses and we can conclude that it tries to call and invalid function pointer.
      
      To replace the invalid pointer we introduce a new function, `setMcontextEip`,
      which is very similar to `getMcontextEip` and it knows to set the Eip for the
      different supported OS's. After printing the trace we retrieve the old `Eip` value.
      0bf90d94
  2. 28 Sep, 2022 4 commits
    • sundb's avatar
      Fix the missing server.dirty increment and redundant signalModifiedKey in... · f106beeb
      sundb authored
      Fix the missing server.dirty increment and redundant signalModifiedKey in serveClientBlockedOnList (#11326)
      
      Mainly fix two minor bug
      1. When handle BL*POP/BLMOVE commands with blocked clients, we should increment server.dirty.
      2.  `listPopRangeAndReplyWithKey()` in `serveClientBlockedOnList()` should not repeat calling
         `signalModifiedKey()` which has been called when an element was pushed on the list.
         (was skipped in all bpop commands, other than blmpop) 
      
      Other optimization
      add `signal` param for `listElementsRemoved` to skip `signalModifiedKey()` to unify all pop operation.
      
      Unifying all pop operations also prepares us for #11303, so that we can avoid having to deal with the
      conversion from quicklist to listpack() in various places when the list shrinks.
      f106beeb
    • guybe7's avatar
      RM_CreateCommand should not set CMD_KEY_VARIABLE_FLAGS automatically (#11320) · 3330ea18
      guybe7 authored
      The original idea behind auto-setting the default (first,last,step) spec was to use
      the most "open" flags when the user didn't provide any key-spec flags information.
      
      While the above idea is a good approach, it really makes no sense to set
      CMD_KEY_VARIABLE_FLAGS if the user didn't provide the getkeys-api flag:
      in this case there's not way to retrieve these variable flags, so what's the point?
      
      Internally in redis there was code to ignore this already, so this fix doesn't change
      redis's behavior, it only affects the output of COMMAND command.
      3330ea18
    • Phoeniwx's avatar
      fix: redis-cli --memkeys-samples add check lastarg (#11269) · c0725abf
      Phoeniwx authored
      doing redis-cli --memkeys-samples without any additional arguments
      would have lead to a crash of the cli.
      c0725abf
    • guybe7's avatar
      Remove redundant arity checks in XINFO (#11331) · bd40d315
      guybe7 authored
      The arity in the JSON files of the subcommands reneder this
      code unreachable
      bd40d315
  3. 27 Sep, 2022 1 commit
    • Steffen Moser's avatar
      Fixing compilation by removing flock() when compiling on Solaris (#11327) · 6aab4cb7
      Steffen Moser authored
      SunOS/Solaris and its relatives don't support the flock() function.
      While "redis" has been excluding setting up the lock using flock() on the cluster
      configuration file when compiling under Solaris, it was still using flock() in the
      unlock call while shutting down. 
      
      This pull request eliminates the flock() call also in the unlocking stage
      for Oracle Solaris and its relatives.
      
      Fix compilation regression from #10912
      6aab4cb7
  4. 26 Sep, 2022 1 commit
    • Ozan Tezcan's avatar
      Ignore RM_Call deny-oom flag if maxmemory is zero (#11319) · 18920813
      Ozan Tezcan authored
      If a command gets an OOM response and then if we set maxmemory to zero
      to disable the limit, server.pre_command_oom_state never gets updated
      and it stays true. As RM_Call() calls with "respect deny-oom" flag checks
      server.pre_command_oom_state, all calls will fail with OOM.
      
      Added server.maxmemory check in RM_Call() to process deny-oom flag
      only if maxmemory is configured.
      18920813
  5. 22 Sep, 2022 5 commits
    • Binbin's avatar
      Fix CLUSTER SHARDS showing empty hostname (#11297) · 1de675b3
      Binbin authored
      * Fix CLUSTER SHARDS showing empty hostname
      
      In #10290, we changed clusterNode hostname from `char*`
      to `sds`, and the old `node->hostname` was changed to
      `sdslen(node->hostname)!=0`.
      
      But in `addNodeDetailsToShardReply` it is missing.
      It results in the return of an empty string hostname
      in CLUSTER SHARDS command if it unavailable.
      
      Like this (note that we listed it as optional in the doc):
      ```
       9) "hostname"
      10) ""
      ```
      1de675b3
    • Shaya Potter's avatar
      Add RM_SetContextUser to support acl validation in RM_Call (and scripts) (#10966) · 6e993a5d
      Shaya Potter authored
      Adds a number of user management/ACL validaiton/command execution functions to improve a
      Redis module's ability to enforce ACLs correctly and easily.
      
      * RM_SetContextUser - sets a RedisModuleUser on the context, which RM_Call will use to both
        validate ACLs (if requested and set) as well as assign to the client so that scripts executed via
        RM_Call will have proper ACL validation.
      * RM_SetModuleUserACLString - Enables one to pass an entire ACL string, not just a single OP
        and have it applied to the user
      * RM_GetModuleUserACLString - returns a stringified version of the user's ACL (same format as dump
        and list).  Contains an optimization to cache the stringified version until the underlying ACL is modified.
      * Slightly re-purpose the "C" flag to RM_Call from just being about ACL check before calling the
        command, to actually running the command with the right user, so that it also affects commands
        inside EVAL scripts. see #11231
      6e993a5d
    • Oran Agra's avatar
      Fix heap overflow vulnerability in XAUTOCLAIM (CVE-2022-35951) (#11301) · 6d215601
      Oran Agra authored
      Executing an XAUTOCLAIM command on a stream key in a specific state, with a
      specially crafted COUNT argument may cause an integer overflow, a subsequent
      heap overflow, and potentially lead to remote code execution.
      The problem affects Redis versions 7.0.0 or newer.
      6d215601
    • Valentino Geron's avatar
      Replica that asks for rdb only should be closed right after the rdb part (#11296) · e53bf652
      Valentino Geron authored
      The bug is that the the server keeps on sending newlines to the client.
      As a result, the receiver might not find the EOF marker since it searches
      for it only on the end of each payload it reads from the socket.
      The but only affects `CLIENT_REPL_RDBONLY`.
      This affects `redis-cli --rdb` (depending on timing)
      
      The fixed consist of two steps:
      1. The `CLIENT_REPL_RDBONLY` should be closed ASAP (we cannot
         always call to `freeClient` so we use `freeClientAsync`)
      2. Add new replication state `SLAVE_STATE_RDB_TRANSMITTED`
      e53bf652
    • Binbin's avatar
      ACL default newly created user set USER_FLAG_SANITIZE_PAYLOAD flag (#11279) · bb6513cb
      Binbin authored
      Starting from 6.2, after ACL SETUSER user reset, the user
      will carry the sanitize-payload flag. It was added in #7807,
      and then ACL SETUSER reset is inconsistent with default
      newly created user which missing sanitize-payload flag.
      
      Same as `off` and `on` these two bits are mutually exclusive,
      the default created user needs to have sanitize-payload flag.
      Adds USER_FLAG_SANITIZE_PAYLOAD flag to ACLCreateUser.
      
      Note that the bug don't have any real implications,
      since the code in rdb.c (rdbLoadObject) checks for
      `USER_FLAG_SANITIZE_PAYLOAD_SKIP`, so the fact that
      `USER_FLAG_SANITIZE_PAYLOAD` is missing doesn't really matters.
      
      Added tests to make sure it won't be broken in the future,
      and updated the comment in ACLSetUser and redis.conf
      bb6513cb
  6. 21 Sep, 2022 1 commit
  7. 19 Sep, 2022 2 commits
    • Binbin's avatar
      Fix Invalid node address specified in redis-cli --cluster create/add-node (#11151) · c2b0c13d
      Binbin authored
      This bug was introduced in #10344 (7.0.3), and it breaks the
      redis-cli --cluster create usage in #10436 (7.0 RC3).
      
      At the same time, the cluster-port support introduced in #10344
      cannot use the DNS lookup brought by #10436.
      c2b0c13d
    • sundb's avatar
      Fix crash due to delete entry from compress quicklistNode and wrongly split quicklistNode (#11242) · 13d25dd9
      sundb authored
      This PR mainly deals with 2 crashes introduced in #9357,
      and fix the QUICKLIST-PACKED-THRESHOLD mess in external test mode.
      
      1. Fix crash due to deleting an entry from a compress quicklistNode
         When inserting a large element, we need to create a new quicklistNode first,
         and then delete its previous element, if the node where the deleted element is
         located is compressed, it will cause a crash.
         Now add `dont_compress` to quicklistNode, if we want to use a quicklistNode
         after some operation, we can use this flag like following:
      
          ```c
          node->dont_compress = 1; /* Prevent to be compressed */
          some_operation(node); /* This operation might try to compress this node */
          some_other_operation(node); /* We can use this node without decompress it */
          node->dont_compress = 0; /* Re-able compression */
          quicklistCompressNode(node);
          ```
      
         Perhaps in the future, we could just disable the current entry from being
         compressed during the iterator loop, but that would require more work.
      
      2. Fix crash due to wrongly split quicklist
         before #9357, the offset param of _quicklistSplitNode() will not negative.
         For now, when offset is negative, the split extent will be wrong.
         following example:
          ```c
          int orig_start = after ? offset + 1 : 0;
          int orig_extent = after ? -1 : offset;
          int new_start = after ? 0 : offset;
          int new_extent = after ? offset + 1 : -1;
          # offset: -2, after: 1, node->count: 2
          # current wrong range: [-1,-1] [0,-1]
          # correct range: [1,-1] [0, 1]
          ```
      
         Because only `_quicklistInsert()` splits the quicklistNode and only
         `quicklistInsertAfter()`, `quicklistInsertBefore()` call _quicklistInsert(), 
         so `quicklistReplaceEntry()` and `listTypeInsert()` might occur this crash.
         But the iterator of `listTypeInsert()` is alway from head to tail(iter->offset is
         always positive), so it is not affected.
         The final conclusion is this crash only occur when we insert a large element
         with negative index into a list, that affects `LSET` command and `RM_ListSet`
         module api.
           
      3. In external test mode, we need to restore quicklist packed threshold after
         when the end of test.
      4. Show `node->count` in quicklistRepr().
      5. Add new tcl proc `config_get_set` to support restoring config in tests.
      13d25dd9
  8. 18 Sep, 2022 1 commit
    • zhaozhao.zz's avatar
      fix infinite sleep in performEvictions when have lazyfree jobs (#11237) · 464aa041
      zhaozhao.zz authored
      This bug is introduced in #7653. (Redis 6.2.0)
      
      When `server.maxmemory_eviction_tenacity` is 100, `eviction_time_limit_us` is
      `ULONG_MAX`, and if we cannot find the best key to delete (e.g. maxmemory-policy
      is `volatile-lru` and all keys with ttl have been evicted), in `cant_free` redis will sleep
      forever if some items are being freed in the lazyfree thread.
      464aa041
  9. 15 Sep, 2022 1 commit
  10. 14 Sep, 2022 1 commit
  11. 13 Sep, 2022 1 commit
  12. 08 Sep, 2022 4 commits
  13. 06 Sep, 2022 2 commits
  14. 05 Sep, 2022 3 commits
  15. 28 Aug, 2022 4 commits
  16. 26 Aug, 2022 2 commits
  17. 24 Aug, 2022 4 commits
    • Oran Agra's avatar
      Fix assertion when a key is lazy expired during cluster key migration (#11176) · c789fb0a
      Oran Agra authored
      Redis 7.0 has #9890 which added an assertion when the propagation queue
      was not flushed and we got to beforeSleep.
      But it turns out that when processCommands calls getNodeByQuery and
      decides to reject the command, it can lead to a key that was lazy
      expired and is deleted without later flushing the propagation queue.
      
      This change prevents lazy expiry from deleting the key at this stage
      (not as part of a command being processed in `call`)
      c789fb0a
    • Binbin's avatar
      Bump codespell from 2.1.0 to 2.2.1 in /.codespell (#11184) · 78259826
      Binbin authored
      add a few terms to the white list, and fix a few newly detected typos
      78259826
    • Brad Dunbar's avatar
      Cleanup in GETDEL: Strings are never freed lazily (#11175) · c0721237
      Brad Dunbar authored
      The GETDEL command only operates on strings, and strings are never freed
      lazily, so there's no need to use `dbAsyncDelete` or `shared.unlink`.
      c0721237
    • Meir Shpilraien (Spielrein)'s avatar
      Reverts most of the changes of #10969 (#11178) · c1bd61a4
      Meir Shpilraien (Spielrein) authored
      The PR reverts the changes made on #10969.
      The reason for revert was trigger because of occasional test failure
      that started after the PR was merged.
      
      The issue is that if there is a lazy expire during the command invocation,
      the `del` command is added to the replication stream after the command
      placeholder. So the logical order on the primary is:
      
      * Delete the key (lazy expiration)
      * Command invocation
      
      But the replication stream gets it the other way around:
      
      * Command invocation (because the command is written into the placeholder)
      * Delete the key (lazy expiration)
      
      So if the command write to the key that was just lazy expired we will get
      inconsistency between primary and replica.
      
      One solution we considered is to add another lazy expire replication stream
      and write all the lazy expire there. Then when replicating, we will replicate the
      lazy expire replication stream first. This will solve this specific test failure but
      we realize that the issues does not ends here and the more we dig the more
      problems we find.One of the example we thought about (that can actually
      crashes Redis) is as follow:
      
      * User perform SINTERSTORE
      * When Redis tries to fetch the second input key it triggers lazy expire
      * The lazy expire trigger a module logic that deletes the first input key
      * Now Redis hold the robj of the first input key that was actually freed
      
      We believe we took the wrong approach and we will come up with another
      PR that solve the problem differently, for now we revert the changes so we
      will not have the tests failure.
      
      Notice that not the entire code was revert, some parts of the PR are changes
      that we would like to keep. The changes that **was** reverted are:
      
      * Saving a placeholder for replication at the beginning of the command (`call` function)
      * Order of the replication stream on active expire and eviction (we will decide how
        to handle it correctly on follow up PR)
      * `Spop` changes are no longer needed (because we reverted the placeholder code)
      
      Changes that **was not** reverted:
      
      * On expire/eviction, wrap the `del` and the notification effect in a multi exec.
      * `PropagateNow` function can still accept a special dbid, -1, indicating not to replicate select.
      * Keep optimisation for reusing the `alsoPropagate` array instead of allocating it each time.
      
      Tests:
      
      * All tests was kept and only few tests was modify to work correctly with the changes
      * Test was added to verify that the revert fixes the issues.
      c1bd61a4
  18. 23 Aug, 2022 2 commits
    • Oran Agra's avatar
      Build TLS as a loadable module · 4faddf18
      Oran Agra authored
      * Support BUILD_TLS=module to be loaded as a module via config file or
        command line. e.g. redis-server --loadmodule redis-tls.so
      * Updates to redismodule.h to allow it to be used side by side with
        server.h by defining REDISMODULE_CORE_MODULE
      * Changes to server.h, redismodule.h and module.c to avoid repeated
        type declarations (gcc 4.8 doesn't like these)
      * Add a mechanism for non-ABI neutral modules (ones who include
        server.h) to refuse loading if they detect not being built together with
        redis (release.c)
      * Fix wrong signature of RedisModuleDefragFunc, this could break
        compilation of a module, but not the ABI
      * Move initialization of listeners in server.c to be after loading
        the modules
      * Config TLS after initialization of listeners
      * Init cluster after initialization of listeners
      * Add TLS module to CI
      * Fix a test suite race conditions:
        Now that the listeners are initialized later, it's not sufficient to
        wait for the PID message in the log, we need to wa...
      4faddf18
    • Ariel Shtul's avatar
      [PERF] use snprintf once in addReplyDouble (#11093) · 90223759
      Ariel Shtul authored
      The previous implementation calls `snprintf` twice, the second time used to
      'memcpy' the output of the first, which could be a very large string.
      The new implementation reserves space for the protocol header ahead
      of the formatted double, and then prepends the string length ahead of it.
      
      Measured improvement of simple ZADD of some 25%.
      90223759