1. 10 May, 2022 3 commits
    • Mariya Markova's avatar
      Replace float zero comparison to FP_ZERO comparison (#10675) · c2d8d4e6
      Mariya Markova authored
      I suggest to use "[fpclassify](https://en.cppreference.com/w/cpp/numeric/math/fpclassify)" for float
      comparison with zero, because of expression "value == 0" with value very close to zero can be
      considered as true with some performance compiler optimizations.
      
      Note: this code was introduced by 9d520a7f to accept zset scores that get ERANGE in conversion
      due to precision loss near 0.
      But with Intel compilers, ICC and ICX, where optimizations for 0 check are more aggressive, "==0" is
      true for mentioned functions, however should not be. Behavior is seen starting from O2.
      This leads to a failure in the ZSCAN test in scan.tcl
      c2d8d4e6
    • Binbin's avatar
      CLUSTER SHARDS should returns slots as integers, not strings (#10683) · 2a1ea8c7
      Binbin authored
      It used to returns slots as strings, like:
      ```
      redis> cluster shards
      1) 1) "slots"
         2) 1) "10923"
            2) "16383"
      ```
      
      CLUSTER SHARDS docs and the top comment of #10293 says that it returns integers.
      Note other commands like CLUSTER SLOTS, it returns slots as integers.
      Use addReplyLongLong instead of addReplyBulkLongLong, now it returns slots as integers:
      ```
      redis> cluster shards
      1) 1) "slots"
         2) 1) (integer) 10923
            2) (integer) 16383
      ```
      
      This is a small breaking change, introduced in 7.0.0 (7.0 RC3, #10293)
      
      Fixes #10680
      2a1ea8c7
    • Meir Shpilraien (Spielrein)'s avatar
      Fix #10705, avoid relinking the same library twice. (#10706) · 442e73ea
      Meir Shpilraien (Spielrein) authored
      Set `old_li` to NULL to avoid linking it again on error.
      Before the fix, loading an already existing library will cause the existing library to be added again. This cause not harm other then wrong statistics. The statistics that are effected  by the issue are:
      * `libraries_count` and `functions_count` returned by `function stats` command
      * `used_memory_functions` returned on `info memory` command
      * `functions.caches` returned on `memory stats` command
      442e73ea
  2. 09 May, 2022 5 commits
  3. 08 May, 2022 2 commits
  4. 03 May, 2022 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Expose Lua error in case of string error. (#10677) · f44c3432
      Meir Shpilraien (Spielrein) authored
      In general, our error handler make sure the error
      object is always a table. In some rare cases (such
      as OOM error), the error handler will not be called
      and the error object will be a string. The PR expose
      the error even if its a string and not a table.
      
      Currently there is no way to test it but if it'll ever happen,
      it is better to propagate this string upwards than just
      generate a generic error without any specific info.
      f44c3432
  5. 02 May, 2022 3 commits
  6. 01 May, 2022 1 commit
  7. 28 Apr, 2022 2 commits
  8. 27 Apr, 2022 1 commit
  9. 26 Apr, 2022 9 commits
    • meir's avatar
      Protect any table which is reachable from globals and added globals white list. · efa162bc
      meir authored
      The white list is done by setting a metatable on the global table before initializing
      any library. The metatable set the `__newindex` field to a function that check
      the white list before adding the field to the table. Fields which is not on the
      white list are simply ignored.
      
      After initialization phase is done we protect the global table and each table
      that might be reachable from the global table. For each table we also protect
      the table metatable if exists.
      efa162bc
    • meir's avatar
      Protect globals of both evals scripts and functions. · 3731580b
      meir authored
      Use the new `lua_enablereadonlytable` Lua API to protect the global tables of
      both evals scripts and functions. For eval scripts, the implemetation is easy,
      We simply call `lua_enablereadonlytable` on the global table to turn it into
      a readonly table.
      
      On functions its more complecated, we want to be able to switch globals between
      load run and function run. To achieve this, we create a new empty table that
      acts as the globals table for function, we control the actual globals using metatable
      manipulation. Notice that even if the user gets a pointer to the original tables, all
      the tables are set to be readonly (using `lua_enablereadonlytable` Lua API) so he can
      not change them. The following inlustration better explain the solution:
      
      ```
      Global table {} <- global table metatable {.__index = __real_globals__}
      ```
      
      The `__real_globals__` is set depends on the run context (function load or function call).
      
      Why this solution is needed and its not enough to simply switch globals?
      When we run in the context of function load and create our functions, our function gets
      the current globals that was set when they were created. Replacing the globals after
      the creation will not effect them. This is why this trick it mandatory.
      3731580b
    • meir's avatar
      Move user eval function to be located on Lua registry. · 992f9e23
      meir authored
      Today, Redis wrap the user Lua code with a Lua function.
      For example, assuming the user code is:
      
      ```
      return redis.call('ping')
      ```
      
      The actual code that would have sent to the Lua interpreter was:
      
      ```
      f_b3a02c833904802db9c34a3cf1292eee3246df3c() return redis.call('ping') end
      ```
      
      The wraped code would have been saved on the global dictionary with the
      following name: `f_<script sha>` (in our example `f_b3a02c833904802db9c34a3cf1292eee3246df3c`).
      
      This approach allows one user to easily override the implementation a another user code, example:
      
      ```
      f_b3a02c833904802db9c34a3cf1292eee3246df3c = function() return 'hacked' end
      ```
      
      Running the above code will cause `evalsha b3a02c833904802db9c34a3cf1292eee3246df3c 0` to return
      hacked although it should have returned `pong`.
      
      Another disadventage is that Redis basically runs code on the loading (compiling) phase without been
      aware of it. User can do code injection like this:
      
      ```
      return 1 end <run code on compling phase> function() return 1
      ```
      
      The wraped code will look like this and the entire `<run code on compling phase>` block will run outside
      of eval or evalsha context:
      
      ```
      f_<sha>() return 1 end <run code on compling phase> function() return 1 end
      ```
      992f9e23
    • Oran Agra's avatar
      Add module API flag for using enum configs as bit flags (#10643) · 81926254
      Oran Agra authored
      Enables registration of an enum config that'll let the user pass multiple keywords that
      will be combined with `|` as flags into the integer config value.
      
      ```
          const char *enum_vals[] = {"none", "one", "two", "three"};
          const int int_vals[] = {0, 1, 2, 4};
      
          if (RedisModule_RegisterEnumConfig(ctx, "flags", 3, REDISMODULE_CONFIG_DEFAULT | REDISMODULE_CONFIG_BITFLAGS, enum_vals, int_vals, 4, getFlagsConfigCommand, setFlagsConfigCommand, NULL, NULL) == REDISMODULE_ERR) {
              return REDISMODULE_ERR;
          }
      ```
      doing:
      `config set moduleconfigs.flags "two three"` will result in 6 being passed to`setFlagsConfigCommand`.
      81926254
    • chenyang8094's avatar
      Fix bug when AOF enabled after startup. put the new incr file in the manifest... · 46ec6ad9
      chenyang8094 authored
      Fix bug when AOF enabled after startup. put the new incr file in the manifest only when AOFRW is done. (#10616)
      
      Changes:
      
      - When AOF is enabled **after** startup, the data accumulated during `AOF_WAIT_REWRITE`
        will only be stored in a temp INCR AOF file. Only after the first AOFRW is successful, we will
        add it to manifest file.
        Before this fix, the manifest referred to the temp file which could cause a restart during that
        time to load it without it's base.
      - Add `aof_rewrites_consecutive_failures` info field for  aofrw limiting implementation.
      
      Now we can guarantee that these behaviors of MP-AOF are the same as before (past redis releases):
      - When AOF is enabled after startup, the data accumulated during `AOF_WAIT_REWRITE` will only
        be stored in a visible place. Only after the first AOFRW is successful, we will add it to manifest file.
      - When disable AOF, we did not delete the AOF file in the past so there's no need to change that
        behavior now (yet).
      - When toggling AOF off and then on (could be as part of a full-sync), a crash or restart before the
        first rewrite is completed, would result with the previous version being loaded (might not be right thing,
        but that's what we always had).
      46ec6ad9
    • Eduardo Semprebon's avatar
      Allow configuring signaled shutdown flags (#10594) · 3a1d1425
      Eduardo Semprebon authored
      
      
      The SHUTDOWN command has various flags to change it's default behavior,
      but in some cases establishing a connection to redis is complicated and it's easier
      for the management software to use signals. however, so far the signals could only
      trigger the default shutdown behavior.
      Here we introduce the option to control shutdown arguments for SIGTERM and SIGINT.
      
      New config options:
      `shutdown-on-sigint [nosave | save] [now] [force]` 
      `shutdown-on-sigterm [nosave | save] [now] [force]`
      
      Implementation:
      Support MULTI_ARG_CONFIG on createEnumConfig to support multiple enums to be applied as bit flags.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      3a1d1425
    • Binbin's avatar
      Fix syntax error in replicationErrorBehavior enum (#10642) · 156836bf
      Binbin authored
      Missing a typeof, we will get errors like this:
      - multiple definition of `replicationErrorBehavior'
      - ld: error: duplicate symbol: replicationErrorBehavior
      
      Introduced in #10504
      156836bf
    • Madelyn Olson's avatar
      Set replicas to panic on disk errors, and optionally panic on replication errors (#10504) · 6fa8e4f7
      Madelyn Olson authored
      * Till now, replicas that were unable to persist, would still execute the commands
        they got from the master, now they'll panic by default, and we add a new
        `replica-ignore-disk-errors` config to change that.
      * Till now, when a command failed on a replica or AOF-loading, it only logged a
        warning and a stat, we add a new `propagation-error-behavior` config to allow
        panicking in that state (may become the default one day)
      
      Note that commands that fail on the replica can either indicate a bug that could
      cause data inconsistency between the replica and the master, or they could be
      in some cases (specifically in previous versions), a result of a command (e.g. EVAL)
      that failed on the master, but still had to be propagated to fail on the replica as well.
      6fa8e4f7
    • Madelyn Olson's avatar
      By default prevent cross slot operations in functions and scripts with # (#10615) · efcd1bf3
      Madelyn Olson authored
      Adds the `allow-cross-slot-keys` flag to Eval scripts and Functions to allow
      scripts to access keys from multiple slots.
      The default behavior is now that they are not allowed to do that (unlike before).
      This is a breaking change for 7.0 release candidates (to be part of 7.0.0), but
      not for previous redis releases since EVAL without shebang isn't doing this check.
      
      Note that the check is done on both the keys declared by the EVAL / FCALL command
      arguments, and also the ones used by the script when making a `redis.call`.
      
      A note about the implementation, there seems to have been some confusion
      about allowing access to non local keys. I thought I missed something in our
      wider conversation, but Redis scripts do block access to non-local keys.
      So the issue was just about cross slots being accessed.
      efcd1bf3
  10. 25 Apr, 2022 4 commits
  11. 24 Apr, 2022 3 commits
  12. 20 Apr, 2022 5 commits
    • filipe oliveira's avatar
      Optimization: Use either monotonic or wall-clock to measure command execution... · 3cd8baf6
      filipe oliveira authored
      
      Optimization: Use either monotonic or wall-clock to measure command execution time, to regain up to 4% execution time (#10502)
      
      In #7491 (part of redis 6.2), we started using the monotonic timer instead of mstime to measure
      command execution time for stats, apparently this meant sampling the clock 3 times per command
      rather than two (wince we also need the wall-clock time).
      In some cases this causes a significant overhead.
      
      This PR fixes that by avoiding the use of monotonic timer, except for the cases were we know it
      should be extremely fast.
      This PR also adds a new INFO field called `monotonic_clock` that shows which clock redis is using.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      3cd8baf6
    • Oran Agra's avatar
      Fixes around clients that must be obeyed. Replica report disk errors in PING. (#10603) · ee220599
      Oran Agra authored
      This PR unifies all the places that test if the current client is the
      master client or AOF client, and uses a method to test that on all of
      these.
      
      Other than some refactoring, these are the actual implications:
      - Replicas **don't** ignore disk error when processing commands not
        coming from their master.
        **This is important for PING to be used for health check of replicas**
      - SETRANGE, APPEND, SETBIT, BITFIELD don't do proto_max_bulk_len check for AOF
      - RM_Call in SCRIPT_MODE ignores disk error when coming from master /
        AOF
      - RM_Call in cluster mode ignores slot check when processing AOF
      - Scripts ignore disk error when processing AOF
      - Scripts **don't** ignore disk error on a replica, if the command comes
        from clients other than the master
      - SCRIPT KILL won't kill script coming from AOF
      - Scripts **don't** skip OOM check on replica if the command comes from
        clients other than the master
      
      Note that Script, AOF, and module clients don't reach processCommand,
      which is why some of the changes don't actually have any implications.
      
      Note, reverting the change done to processCommand in 2f4240b9
      should be dead code due to the above mentioned fact.
      ee220599
    • menwen's avatar
      Stop RDB child before flushing and parsing the RDB in Diskless replication too (#10602) · 046654a7
      menwen authored
      
      
      We should stop RDB child in advance before flushing to reduce COW in diskless replication too.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      046654a7
    • yoav-steinberg's avatar
      Optimized `hdr_value_at_percentile` (#10606) · 5075e743
      yoav-steinberg authored
      `hdr_value_at_percentile()` is part of the Hdr_Histogram library
      used when generating `latencystats` report. 
      
      There's a pending optimization for this function which greatly
      affects the performance of `info latencystats`.
      https://github.com/HdrHistogram/HdrHistogram_c/pull/107
      
      
      
      This PR:
      1. Upgrades the sources in _deps/hdr_histogram_ to the latest Hdr_Histogram
        version 0.11.5
      2. Applies the referenced optimization.
      3. Adds minor documentation about the hdr_histogram dependency which was
        missing under _deps/README.md_.
      
      benchmark on my machine:
      running: `redis-benchmark -n 100000 info latencystats` on a clean build with no data.
      
      | benchmark | RPS |
      | ---- | ---- |
      | before upgrade to v0.11.05  | 7,681 |
      | before optimization | 12,474 |
      | after optimization | 52,606 |
      Co-authored-by: default avatarfilipe oliveira <filipecosta.90@gmail.com>
      5075e743
    • David CARLIER's avatar
      Add socket-mark-id support for marking sockets. (#10349) · aba2865c
      David CARLIER authored
      Add a configuration option to attach an operating system-specific identifier to Redis sockets, supporting advanced network configurations using iptables (Linux) or ipfw (FreeBSD).
      aba2865c
  13. 19 Apr, 2022 1 commit