1. 30 Sep, 2021 1 commit
  2. 09 Sep, 2021 1 commit
  3. 08 Aug, 2021 1 commit
  4. 04 Aug, 2021 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Unified Lua and modules reply parsing and added RESP3 support to RM_Call (#9202) · 2237131e
      Meir Shpilraien (Spielrein) authored
      
      
      ## Current state
      1. Lua has its own parser that handles parsing `reds.call` replies and translates them
        to Lua objects that can be used by the user Lua code. The parser partially handles
        resp3 (missing big number, verbatim, attribute, ...)
      2. Modules have their own parser that handles parsing `RM_Call` replies and translates
        them to RedisModuleCallReply objects. The parser does not support resp3.
      
      In addition, in the future, we want to add Redis Function (#8693) that will probably
      support more languages. At some point maintaining so many parsers will stop
      scaling (bug fixes and protocol changes will need to be applied on all of them).
      We will probably end up with different parsers that support different parts of the
      resp protocol (like we already have today with Lua and modules)
      
      ## PR Changes
      This PR attempt to unified the reply parsing of Lua and modules (and in the future
      Redis Function) by introducing a new parser unit (`resp_parser.c`). The new parser
      handles parsing the reply and calls different callbacks to allow the users (another
      unit that uses the parser, i.e, Lua, modules, or Redis Function) to analyze the reply.
      
      ### Lua API Additions
      The code that handles reply parsing on `scripting.c` was removed. Instead, it uses
      the resp_parser to parse and create a Lua object out of the reply. As mentioned
      above the Lua parser did not handle parsing big numbers, verbatim, and attribute.
      The new parser can handle those and so Lua also gets it for free.
      Those are translated to Lua objects in the following way:
      1. Big Number - Lua table `{'big_number':'<str representation for big number>'}`
      2. Verbatim - Lua table `{'verbatim_string':{'format':'<verbatim format>', 'string':'<verbatim string value>'}}`
      3. Attribute - currently ignored and not expose to the Lua parser, another issue will be open to decide how to expose it.
      
      Tests were added to check resp3 reply parsing on Lua
      
      ### Modules API Additions
      The reply parsing code on `module.c` was also removed and the new resp_parser is used instead.
      In addition, the RedisModuleCallReply was also extracted to a separate unit located on `call_reply.c`
      (in the future, this unit will also be used by Redis Function). A nice side effect of unified parsing is
      that modules now also support resp3. Resp3 can be enabled by giving `3` as a parameter to the
      fmt argument of `RM_Call`. It is also possible to give `0`, which will indicate an auto mode. i.e, Redis
      will automatically chose the reply protocol base on the current client set on the RedisModuleCtx
      (this mode will mostly be used when the module want to pass the reply to the client as is).
      In addition, the following RedisModuleAPI were added to allow analyzing resp3 replies:
      
      * New RedisModuleCallReply types:
         * `REDISMODULE_REPLY_MAP`
         * `REDISMODULE_REPLY_SET`
         * `REDISMODULE_REPLY_BOOL`
         * `REDISMODULE_REPLY_DOUBLE`
         * `REDISMODULE_REPLY_BIG_NUMBER`
         * `REDISMODULE_REPLY_VERBATIM_STRING`
         * `REDISMODULE_REPLY_ATTRIBUTE`
      
      * New RedisModuleAPI:
         * `RedisModule_CallReplyDouble` - getting double value from resp3 double reply
         * `RedisModule_CallReplyBool` - getting boolean value from resp3 boolean reply
         * `RedisModule_CallReplyBigNumber` - getting big number value from resp3 big number reply
         * `RedisModule_CallReplyVerbatim` - getting format and value from resp3 verbatim reply
         * `RedisModule_CallReplySetElement` - getting element from resp3 set reply
         * `RedisModule_CallReplyMapElement` - getting key and value from resp3 map reply
         * `RedisModule_CallReplyAttribute` - getting a reply attribute
         * `RedisModule_CallReplyAttributeElement` - getting key and value from resp3 attribute reply
         
      * New context flags:
         * `REDISMODULE_CTX_FLAGS_RESP3` - indicate that the client is using resp3
      
      Tests were added to check the new RedisModuleAPI
      
      ### Modules API Changes
      * RM_ReplyWithCallReply might return REDISMODULE_ERR if the given CallReply is in resp3
        but the client expects resp2. This is not a breaking change because in order to get a resp3
        CallReply one needs to specifically specify `3` as a parameter to the fmt argument of
        `RM_Call` (as mentioned above).
      
      Tests were added to check this change
      
      ### More small Additions
      * Added `debug set-disable-deny-scripts` that allows to turn on and off the commands no-script
      flag protection. This is used by the Lua resp3 tests so it will be possible to run `debug protocol`
      and check the resp3 parsing code.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      2237131e
  5. 22 Jun, 2021 1 commit
    • Evan's avatar
      modules: Add newlen == 0 handling to RM_StringTruncate (#3717) (#3718) · 1ccf2ca2
      Evan authored
      Previously, passing 0 for newlen would not truncate the string at all.
      This adds handling of this case, freeing the old string and creating a new empty string.
      
      Other changes:
      - Move `src/modules/testmodule.c` to `tests/modules/basics.c`
      - Introduce that basic test into the test suite
      - Add tests to cover StringTruncate
      - Add `test-modules` build target for the main makefile
      - Extend `distclean` build target to clean modules too
      1ccf2ca2
  6. 16 Jun, 2021 1 commit
    • yoav-steinberg's avatar
      Remove gopher protocol support. (#9057) · 362786c5
      yoav-steinberg authored
      Gopher support was added mainly because it was simple (trivial to add).
      But apparently even something that was trivial at the time, does cause complications
      down the line when adding more features.
      We recently ran into a few issues with io-threads conflicting with the gopher support.
      We had to either complicate the code further in order to solve them, or drop gopher.
      AFAIK it's completely unused, so we wanna chuck it, rather than keep supporting it.
      362786c5
  7. 26 Apr, 2021 2 commits
  8. 10 Apr, 2021 1 commit
  9. 10 Mar, 2021 1 commit
    • sundb's avatar
      Add run all test support with define REDIS_TEST (#8570) · 95d6297d
      sundb authored
      1. Add `redis-server test all` support to run all tests.
      2. Add redis test to daily ci.
      3. Add `--accurate` option to run slow tests for more iterations (so that
         by default we run less cycles (shorter time, and less prints).
      4. Move dict benchmark to REDIS_TEST.
      5. fix some leaks in tests
      6. make quicklist tests run on a specific fill set of options rather than huge ranges
      7. move some prints in quicklist test outside their loops to reduce prints
      8. removing sds.h from dict.c since it is now used in both redis-server and
         redis-cli (uses hiredis sds)
      95d6297d
  10. 24 Feb, 2021 1 commit
  11. 23 Dec, 2020 2 commits
    • Greg Femec's avatar
      Fix random element selection for large hash tables. (#8133) · 266949c7
      Greg Femec authored
      
      
      When a database on a 64 bit build grows past 2^31 keys, the underlying hash table expands to 2^32 buckets. After this point, the algorithms for selecting random elements only return elements from half of the available buckets because they use random() which has a range of 0 to 2^31 - 1. This causes problems for eviction policies which use dictGetSomeKeys or dictGetRandomKey. Over time they cause the hash table to become unbalanced because, while new keys are spread out evenly across all buckets, evictions come from only half of the available buckets. Eventually this half of the table starts to run out of keys and it takes longer and longer to find candidates for eviction. This continues until no more evictions can happen.
      
      This solution addresses this by using a 64 bit PRNG instead of libc random().
      Co-authored-by: default avatarGreg Femec <gfemec@google.com>
      266949c7
    • Felix Bünemann's avatar
      Fix TLS build on macOS arm64 systems (#8197) · b51f5da3
      Felix Bünemann authored
      Homebrew for darwin-arm64 uses /opt/homebrew instead of /usr/local as
      the prefix, so that it can coexist with darwin-x86_64 using Rosetta 2.
      b51f5da3
  12. 17 Dec, 2020 1 commit
  13. 13 Dec, 2020 1 commit
    • Yossi Gottlieb's avatar
      Several (mostly Solaris-related) cleanups (#8171) · 86e3395c
      Yossi Gottlieb authored
      * Allow runtest-moduleapi use a different 'make', for systems where GNU Make is 'gmake'.
      * Fix issue with builds on Solaris re-building everything from scratch due to CFLAGS/LDFLAGS not stored.
      * Fix compile failure on Solaris due to atomicvar and a bunch of warnings.
      * Fix garbled log timestamps on Solaris.
      86e3395c
  14. 22 Nov, 2020 1 commit
    • Yossi Gottlieb's avatar
      Clean up building with USE_SYSTEMD. (#8073) · 08d3e929
      Yossi Gottlieb authored
      When USE_SYSTEMD=yes is specified, try to use pkg-config to determine
      libsystemd linker flags. If not found, silently fall back to simply
      using "-lsystemd".
      
      We now use a LIBSYSTEMD_LIBS variable so users can explicitly override
      it and specify their own library.
      
      If USE_SYSTEMD is unspecified the old behavior of auto-enabling it if
      both pkg-config and libsystemd are available is retained.
      08d3e929
  15. 28 Oct, 2020 1 commit
  16. 26 Oct, 2020 1 commit
  17. 01 Oct, 2020 1 commit
  18. 29 Sep, 2020 1 commit
  19. 28 Sep, 2020 1 commit
  20. 23 Sep, 2020 1 commit
  21. 21 Sep, 2020 1 commit
    • Oran Agra's avatar
      Fix C11 detection in the makefile (#7822) · f11a0c8f
      Oran Agra authored
      445a4b66 introudced a makefile script that detects if the toolchain
      supports c11, and it looked that it was passing on MacOS and fails on
      Ubuntu, looks like Ubuntu's Dash was spawning a background process,
      deleted foo.c before gcc tried to compile it.
      f11a0c8f
  22. 17 Sep, 2020 1 commit
    • Wang Yuan's avatar
      Implement redisAtomic to replace _Atomic C11 builtin (#7707) · 445a4b66
      Wang Yuan authored
      Redis 6.0 introduces I/O threads, it is so cool and efficient, we use C11
      _Atomic to establish inter-thread synchronization without mutex. But the
      compiler that must supports C11 _Atomic can compile redis code, that brings a
      lot of inconvenience since some common platforms can't support by default such
      as CentOS7, so we want to implement redis atomic type to make it more portable.
      
      We have implemented our atomic variable for redis that only has 'relaxed'
      operations in src/atomicvar.h, so we implement some operations with
      'sequentially-consistent', just like the default behavior of C11 _Atomic that
      can establish inter-thread synchronization. And we replace all uses of C11
      _Atomic with redis atomic variable.
      
      Our implementation of redis atomic variable uses C11 _Atomic, __atomic or
      __sync macros if available, it supports most common platforms, and we will
      detect automatically which feature we use. In Makefile we use a dummy file to
      detect if the compiler supports C11 _Atomic. Now for gcc, we can compile redis
      code theoretically if your gcc version is not less than 4.1.2(starts to support
      __sync_xxx operations). Otherwise, we remove use mutex fallback to implement
      redis atomic variable for performance and test. You will get compiling errors
      if your compiler doesn't support all features of above.
      
      For cover redis atomic variable tests, we add other CI jobs that build redis on
      CentOS6 and CentOS7 and workflow daily jobs that run the tests on them.
      For them, we just install gcc by default in order to cover different compiler
      versions, gcc is 4.4.7 by default installation on CentOS6 and 4.8.5 on CentOS7.
      
      We restore the feature that we can test redis with Helgrind to find data race
      errors. But you need install Valgrind in the default path configuration firstly
      before running your tests, since we use macros in helgrind.h to tell Helgrind
      inter-thread happens-before relationship explicitly for avoiding false positives.
      Please open an issue on github if you find data race errors relate to this commit.
      
      Unrelated:
      - Fix redefinition of typedef 'RedisModuleUserChangedFunc'
        For some old version compilers, they will report errors or warnings, if we
        re-define function type.
      445a4b66
  23. 01 Sep, 2020 1 commit
  24. 28 Aug, 2020 1 commit
    • Jim Brunner's avatar
      Use H/W Monotonic clock and updates to AE (#7644) · c01e94a4
      Jim Brunner authored
      Update adds a general source for retrieving a monotonic time.
      In addition, AE has been updated to utilize the new monotonic
      clock for timer processing.
      
      This performance improvement is **not** enabled in a default build due to various H/W compatibility
      concerns, see README.md for details. It does however change the default use of gettimeofday with
      clock_gettime and somewhat improves performance.
      
      This update provides the following
      1. An interface for retrieving a monotonic clock. getMonotonicUs returns a uint64_t (aka monotime)
         with the number of micro-seconds from an arbitrary point. No more messing with tv_sec/tv_usec.
         Simple routines are provided for measuring elapsed milli-seconds or elapsed micro-seconds (the
         most common use case for a monotonic timer). No worries about time moving backwards.
      2. High-speed assembler implementation for x86 and ARM. The standard method for retrieving the
         monotonic clock is POSIX.1b (1993): clock_gettime(CLOCK_MONOTONIC, timespec*). However, most
         modern processors provide a constant speed instruction clock which can be retrieved in a fraction
         of the time that it takes to call clock_gettime. For x86, this is provided by the RDTSC
         instruction. For ARM, this is provided by the CNTVCT_EL0 instruction. As a compile-time option,
         these high-speed timers can be chosen. (Default is POSIX clock_gettime.)
      3. Refactor of event loop timers. The timer processing in ae.c has been refactored to use the new
         monotonic clock interface. This results in simpler/cleaner logic and improved performance.
      c01e94a4
  25. 25 Aug, 2020 1 commit
  26. 11 Aug, 2020 1 commit
  27. 10 Jul, 2020 1 commit
  28. 12 May, 2020 1 commit
  29. 02 May, 2020 1 commit
    • zhenwei pi's avatar
      Support setcpuaffinity on linux/bsd · 1a0deab2
      zhenwei pi authored
      Currently, there are several types of threads/child processes of a
      redis server. Sometimes we need deeply optimise the performance of
      redis, so we would like to isolate threads/processes.
      
      There were some discussion about cpu affinity cases in the issue:
      https://github.com/antirez/redis/issues/2863
      
      
      
      So implement cpu affinity setting by redis.conf in this patch, then
      we can config server_cpulist/bio_cpulist/aof_rewrite_cpulist/
      bgsave_cpulist by cpu list.
      
      Examples of cpulist in redis.conf:
      server_cpulist 0-7:2      means cpu affinity 0,2,4,6
      bio_cpulist 1,3           means cpu affinity 1,3
      aof_rewrite_cpulist 8-11  means cpu affinity 8,9,10,11
      bgsave_cpulist 1,10-11    means cpu affinity 1,10,11
      
      Test on linux/freebsd, both work fine.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      1a0deab2
  30. 25 Apr, 2020 1 commit
  31. 27 Mar, 2020 1 commit
  32. 12 Mar, 2020 1 commit
  33. 01 Jan, 2020 1 commit
  34. 29 Nov, 2019 1 commit
  35. 19 Nov, 2019 1 commit
    • Johannes Truschnigg's avatar
      Auto-detect and link libsystemd at compile-time · 129d14e1
      Johannes Truschnigg authored
      This adds Makefile/build-system support for USE_SYSTEMD=(yes|no|*). This
      variable's value determines whether or not libsystemd will be linked at
      build-time.
      
      If USE_SYSTEMD is set to "yes", make will use PKG_CONFIG to check for
      libsystemd's presence, and fail the build early if it isn't
      installed/detected properly.
      
      If USE_SYSTEM is set to "no", libsystemd will *not* be linked, even if
      support for it is available on the system redis is being built on.
      
      For any other value that USE_SYSTEM might assume (e.g. "auto"),
      PKG_CONFIG will try to determine libsystemd's presence, and set up the
      build process to link against it, if it was indicated as being
      installed/available.
      
      This approach has a number of repercussions of its own, most importantly
      the following: If you build redis on a system that actually has systemd
      support, but no libsystemd-dev package(s) installed, you'll end up
      *without* support for systemd notification/status reporting support in
      redis-server. This changes established runtime behaviour.
      
      I'm not sure if the build system and/or the server binary should
      indicate this. I'm also wondering if not actually having
      systemd-notify-support, but requesting it via the server's config,
      should result in a fatal error now.
      129d14e1
  36. 16 Oct, 2019 1 commit
  37. 07 Oct, 2019 2 commits
    • Yossi Gottlieb's avatar
      TLS: Configuration options. · 61733ded
      Yossi Gottlieb authored
      Add configuration options for TLS protocol versions, ciphers/cipher
      suites selection, etc.
      61733ded
    • Yossi Gottlieb's avatar
      TLS: Connections refactoring and TLS support. · b087dd1d
      Yossi Gottlieb authored
      * Introduce a connection abstraction layer for all socket operations and
      integrate it across the code base.
      * Provide an optional TLS connections implementation based on OpenSSL.
      * Pull a newer version of hiredis with TLS support.
      * Tests, redis-cli updates for TLS support.
      b087dd1d