1. 21 Sep, 2022 8 commits
    • Meir Shpilraien (Spielrein)'s avatar
      Fix #11030, use lua_rawget to avoid triggering metatables and crash. (#11032) · ddf1bcf7
      Meir Shpilraien (Spielrein) authored
      Fix #11030, use lua_rawget to avoid triggering metatables.
      
      #11030 shows how return `_G` from the Lua script (either function or eval), cause the
      Lua interpreter to Panic and the Redis processes to exit with error code 1.
      Though return `_G` only panic on Redis 7 and 6.2.7, the underline issue exists on older
      versions as well (6.0 and 6.2). The underline issue is returning a table with a metatable
      such that the metatable raises an error.
      
      The following example demonstrate the issue:
      ```
      127.0.0.1:6379> eval "local a = {}; setmetatable(a,{__index=function() foo() end}) return a" 0
      Error: Server closed the connection
      ```
      ```
      PANIC: unprotected error in call to Lua API (user_script:1: Script attempted to access nonexistent global variable 'foo')
      ```
      
      The Lua panic happened because when returning the result to the client, Redis needs to
      introspect the returning table and transform the table into a resp. In order to scan the table,
      Redis uses `lua_gettable` api which might trigger the metatable (if exists) and might raise an error.
      This code is not running inside `pcall` (Lua protected call), so raising an error causes the
      Lua to panic and exit. Notice that this is not a crash, its a Lua panic that exit with error code 1.
      
      Returning `_G` panics on Redis 7 and 6.2.7 because on those versions `_G` has a metatable
      that raises error when trying to fetch a none existing key.
      
      ### Solution
      
      Instead of using `lua_gettable` that might raise error and cause the issue, use `lua_rawget`
      that simply return the value from the table without triggering any metatable logic.
      This is promised not to raise and error.
      
      The downside of this solution is that it might be considered as breaking change, if someone
      rely on metatable in the returned value. An alternative solution is to wrap this entire logic
      with `pcall` (Lua protected call), this alternative require a much bigger refactoring.
      
      ### Back Porting
      
      The same fix will work on older versions as well (6.2, 6.0). Notice that on those version,
      the issue can cause Redis to crash if inside the metatable logic there is an attempt to accesses
      Redis (`redis.call`). On 7.0, there is not crash and the `redis.call` is executed as if it was done
      from inside the script itself.
      
      ### Tests
      
      Tests was added the verify the fix
      
      (cherry picked from commit 020e046b)
      ddf1bcf7
    • Binbin's avatar
      fsync the old aof file when open a new INCR AOF (#11004) · 5da7fdb7
      Binbin authored
      
      
      In rewriteAppendOnlyFileBackground, after flushAppendOnlyFile(1),
      and before openNewIncrAofForAppend, we should call redis_fsync
      to fsync the aof file.
      
      Because we may open a new INCR AOF in openNewIncrAofForAppend,
      in the case of using everysec policy, the old AOF file may not
      be fsynced in time (or even at all).
      
      When using everysec, we don't want to pay the disk latency from
      the main thread, so we will do a background fsync.
      
      Adding a argument for bioCreateCloseJob, a `need_fsync` flag to
      indicate that a fsync is required before the file is closed. So we will
      fsync the old AOF file before we close it.
      
      A cleanup, we make union become a union, since the free_* args and
      the fd / fsync args are never used together.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 03fff10a)
      5da7fdb7
    • chenyang8094's avatar
      Register abs-expire apis (#11025) · c0394286
      chenyang8094 authored
      RM_SetAbsExpire and RM_GetAbsExpire were not actually operational since
      they were introduced, due to omission in API registration.
      
      (cherry picked from commit 39d216a3)
      c0394286
    • Pavel Krush's avatar
      fixed complexity of bzmpop and zmpop commands (#11026) · 61a59b53
      Pavel Krush authored
      
      
      Swap M and N in the complexity formula of [B]ZMPOP
      Co-authored-by: default avatarPavel Krush <neon@pushwoosh.com>
      (cherry picked from commit 5879e490)
      61a59b53
    • Tian's avatar
      Don't update node ip when peer fd is closed (#10696) · babe90c9
      Tian authored
      
      (cherry picked from commit d00b8af8)
      babe90c9
    • guybe7's avatar
      Adds LASTID to XCLAIM docs (#11017) · 9e3dfa96
      guybe7 authored
      It seems it was overlooked when we first created the json files
      
      (cherry picked from commit 6d6e932f)
      9e3dfa96
    • Wen Hui's avatar
      Fix EVALSHA_RO and EVAL_RO command json file (#11015) · d28314cb
      Wen Hui authored
      
      
      these are missing from the RO_ commands, present in the other ones.
      Co-authored-by: default avatarUbuntu <lucas.guang.yang1@huawei.com>
      (cherry picked from commit 56828bab)
      d28314cb
    • Wen Hui's avatar
      Add optional for FCALL and FCALL_RO command json file (#10988) · a8b6efea
      Wen Hui authored
      
      
      According to the Redis functions documentation, FCALL command format could be
      FCALL function_name numberOfKeys [key1, key2, key3.....] [arg1, arg2, arg3.....]
      
      So in the json file of fcall and fcall_ro, we should add optional for key and arg part.
      Just like EVAL...
      Co-authored-by: default avatarBinbin <binloveplay1314@qq.com>
      (cherry picked from commit 2d3240f3)
      a8b6efea
  2. 18 Jul, 2022 3 commits
  3. 11 Jul, 2022 8 commits
  4. 10 Jul, 2022 1 commit
  5. 07 Jul, 2022 1 commit
  6. 06 Jul, 2022 2 commits
  7. 05 Jul, 2022 1 commit
  8. 04 Jul, 2022 5 commits
    • Harkrishn Patro's avatar
      9bcdd153
    • Wen Hui's avatar
      Add tests for error messages during slot migrations (#10449) · f620e6ac
      Wen Hui authored
      
      
      * Add tests for error messages during slot migrations
      Co-authored-by: default avatarUbuntu <lucas.guang.yang1@huawei.com>
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      f620e6ac
    • Harkrishn Patro's avatar
      Optimize number of realloc syscall during multi/exec flow (#10921) · a3704d4e
      Harkrishn Patro authored
      ## Issue
      During the MULTI/EXEC flow, each command gets queued until the `EXEC`
      command is received and during this phase on every command queue, a
      `realloc` is being invoked. This could be expensive based on the realloc
      behavior (if copy to a new memory location). 
      
      
      ## Solution
      In order to reduce the no. of syscall, couple of optimization I've used.
      
      1. By default, reserve memory for atleast two commands. `MULTI/EXEC` for a
      single command doesn't have any significance. Hence, I believe customer wouldn't use it.
      2. For further reservation, increase the memory allocation in exponent growth (power of 2).
      This reduces the no. of `realloc` call from `N` to `log(N)` times.
      
      ## Other changes:
      
      * Include multi exec queued command array in client memory consumption calculation
      (affects client eviction too)
      a3704d4e
    • Qu Chen's avatar
      Unlock cluster config file upon server shutdown. (#10912) · 33b7ff38
      Qu Chen authored
      Currently in cluster mode, Redis process locks the cluster config file when
      starting up and holds the lock for the entire lifetime of the process.
      When the server shuts down, it doesn't explicitly release the lock on the
      cluster config file. We noticed a problem with restart testing that if you shut down
      a very large redis-server process (i.e. with several hundred GB of data stored),
      it takes the OS a while to free the resources and unlock the cluster config file.
      So if we immediately try to restart the redis server process, it might fail to acquire
      the lock on the cluster config file and fail to come up.
      
      This fix explicitly releases the lock on the cluster config file upon a shutdown rather
      than relying on the OS to release the lock, which is a cleaner and safer approach to
      free up resources acquired. 
      33b7ff38
    • Harkrishn Patro's avatar
      Account sharded pubsub channels memory consumption (#10925) · 0ab885a6
      Harkrishn Patro authored
      Account sharded pubsub channels memory consumption in client memory usage
      computation to accurately evict client based on the set threshold for `maxmemory-clients`.
      0ab885a6
  9. 03 Jul, 2022 4 commits
    • Binbin's avatar
      Set aof rewrite status in some backgroundRewriteDoneHandler errors (#10923) · 679344a2
      Binbin authored
      We should also set aof_lastbgrewrite_status to C_ERR on these
      errors. Because aof rewrite did fail, and we did not finish the
      manifest update. Also maintain the stat_aofrw_consecutive_failures.
      679344a2
    • Yossi Gottlieb's avatar
      Fix TLS issues with large replies (#10909) · 0b645d63
      Yossi Gottlieb authored
      This problem was introduced by 496375fc and seems to more easily reproduce on macOS since OpenSSL writes more frequently return with EAGAIN.
      0b645d63
    • Yossi Gottlieb's avatar
      Fix TLS tests on newer tcl-tls/OpenSSL. (#10910) · 69d55768
      Yossi Gottlieb authored
      Before this commit, TLS tests on Ubuntu 22.04 would fail as dropped
      connections result with an ECONNABORTED error thrown instead of an empty
      read.
      69d55768
    • Binbin's avatar
      Always set server.aof_last_write_errno in aof write error (#10917) · 2ab67677
      Binbin authored
      The `can_log` variable prevents us from outputting too
      many error logs. But it should not include the modification
      of server.aof_last_write_errno.
      
      We are doing this because:
      1. In the short write case, we always set aof_last_write_errno
      to ENOSPC, we don't care the `can_log` flag.
      
      2. And we always set aof_last_write_status to C_ERR in aof write
      error (except for FSYNC_ALWAYS, we exit). So there may be a chance
      that `aof_last_write_errno` is not right.
      
      An innocent bug or just a code cleanup.
      2ab67677
  10. 30 Jun, 2022 1 commit
    • Binbin's avatar
      Add SENTINEL command flag to CLIENT/COMMANDS subcommands (#10904) · 35e836c2
      Binbin authored
      This was harmless because we marked the parent command
      with SENTINEL flag. So the populateCommandTable was ok.
      And we also don't show the flag (SENTINEL and ONLY-SENTNEL)
      in COMMAND INFO.
      
      In this PR, we also add the same CMD_SENTINEL and CMD_ONLY_SENTINEL
      flags check when populating the sub-commands.
      so that in the future it'll be possible to add some sub-commands to sentinel or sentinel-only but not others.
      35e836c2
  11. 29 Jun, 2022 1 commit
  12. 28 Jun, 2022 2 commits
  13. 27 Jun, 2022 1 commit
    • Viktor Söderqvist's avatar
      Add missing REDISMODULE_CLIENTINFO_INITIALIZER (#10885) · 6af02100
      Viktor Söderqvist authored
      The module API docs mentions this macro, but it was not defined (so no one could have used it).
      
      Instead of adding it as is, we decided to add a _V1 macro, so that if / when we some day extend this struct,
      modules that use this API and don't need the extra fields, will still use the old version
      and still be compatible with older redis version (despite being compiled with newer redismodule.h)
      6af02100
  14. 26 Jun, 2022 2 commits
    • RinChanNOW!'s avatar
      Support conversion between `RedisModuleString` and `unsigned long long` (#10889) · 28546373
      RinChanNOW! authored
      
      
      Since the ranges of `unsigned long long` and `long long` are different, we cannot read an
      `unsigned long long` integer from a `RedisModuleString` by `RedisModule_StringToLongLong` . 
      
      So I added two new Redis Module APIs to support the conversion between these two types:
      * `RedisModule_StringToULongLong`
      * `RedisModule_CreateStringFromULongLong`
      Signed-off-by: default avatarRinChanNOWWW <hzy427@gmail.com>
      28546373
    • Binbin's avatar
      redis-server command line arguments allow passing config name and value in the same arg (#10866) · d443e312
      Binbin authored
      This commit has two topics.
      
      ## Passing config name and value in the same arg
      In #10660 (Redis 7.0.1), when we supported the config values that can start with `--` prefix (one of the two topics of that PR),
      we broke another pattern: `redis-server redis.config "name value"`, passing both config name
      and it's value in the same arg, see #10865
      
      This wasn't a intended change (i.e we didn't realize this pattern used to work).
      Although this is a wrong usage, we still like to fix it.
      
      Now we support something like:
      ```
      src/redis-server redis.conf "--maxmemory '700mb'" "--maxmemory-policy volatile-lru" --proc-title-template --my--title--template --loglevel verbose
      ```
      
      ## Changes around --save
      Also in this PR, we undo the breaking change we made in #10660 on purpose.
      1. `redis-server redis.conf --save --loglevel verbose` (missing `save` argument before anotehr argument).
          In 7.0.1, it was throwing an wrong arg error.
          Now it will work and reset the save, similar to how it used to be in 7.0.0 and 6.2.x.
      3. `redis-server redis.conf --loglevel verbose --save` (missing `save` argument as last argument).
          In 6.2, it did not reset the save, which was a bug (inconsistent with the previous bullet).
          Now we will make it work and reset the save as well (a bug fix).
      d443e312