1. 15 Sep, 2024 1 commit
  2. 21 Aug, 2024 1 commit
  3. 16 Jul, 2024 1 commit
    • Oran Agra's avatar
      Test infra adjustments for external CI runs (#13421) · fa46aa4d
      Oran Agra authored
      - when uploading server logs, make sure they don't overwrite each other.
      - sort the test units to get consistent order between them (following
      #13220)
      - backup and restore the entire server configuration, to protect one
      unit from config changes another unit performs
      fa46aa4d
  4. 08 Jul, 2024 1 commit
  5. 02 Jul, 2024 1 commit
  6. 15 May, 2024 1 commit
  7. 17 Mar, 2024 1 commit
  8. 12 Feb, 2024 1 commit
    • Binbin's avatar
      Fix CLIENAT KILL MAXAGE test timing issue (#13047) · 8eeece4a
      Binbin authored
      This test fails occasionally:
      ```
      *** [err]: CLIENT KILL maxAGE will kill old clients in tests/unit/introspection.tcl
      Expected 2 == 1 (context: type eval line 14 cmd {assert {$res == 1}} proc ::test)
      ```
      
      This test is very likely to do a false positive if the execute time
      takes longer than the max age, for example, if the execution time
      between sleep and kill exceeds 1s, rd2 will also be killed due to
      the max age.
      
      The test can adjust the order of execution statements to increase
      the probability of passing, but this is still will be a timing issue
      in some slow machines, so decided give it a few more chances.
      
      The test was introduced in #12299.
      8eeece4a
  9. 23 Jan, 2024 1 commit
  10. 04 Jan, 2024 1 commit
  11. 28 Dec, 2023 1 commit
  12. 06 Dec, 2023 2 commits
  13. 30 Nov, 2023 1 commit
  14. 13 Nov, 2023 1 commit
  15. 12 Nov, 2023 1 commit
    • Roshan Khatri's avatar
      Add DEBUG_ASSERTIONS option to custom assert (#12667) · 88e83e51
      Roshan Khatri authored
      This PR introduces a new macro, serverAssertWithInfoDebug, to do complex assertions only for debugging. The main intention is to allow running complex operations during tests without impacting runtime performance. This assertion is enabled when setting DEBUG_ASSERTIONS.
      
      The DEBUG_ASSERTIONS flag is set for the daily and CI variants of `test-sanitizer-address`.
      88e83e51
  16. 06 Nov, 2023 1 commit
    • Yossi Gottlieb's avatar
      Use cross-platform-actions for FreeBSD support. (#12732) · 6223355c
      Yossi Gottlieb authored
      This change overcomes many stability issues experienced with the
      vmactions action.
      
      We need to limit VMs to 8GB for better stability, as the 13GB default
      seems to hang them occasionally.
      
      Shell code has been simplified since this action seem to use `bash -e`
      which will abort on non-zero exit codes anyway.
      6223355c
  17. 02 Nov, 2023 1 commit
  18. 18 Oct, 2023 1 commit
  19. 13 Jul, 2023 1 commit
    • Chayim I. Kirshen's avatar
      Squashed 'deps/hiredis/' changes from b6a052fe0..60e5075d4 · 67ed3183
      Chayim I. Kirshen authored
      60e5075d4 Version 1.2.0 (#1202)
      adef139a7 Remove support in deprecated TLS versions 1.0 and 1.1 (#1205)
      d543baba6 Install major version symlink of shared objects.
      052f99ab2 Ensure functionality without _MSC_VER definition
      ded32c7d1 Add a test for the TCP_USER_TIMEOUT option. (#1192)
      5cbd1f296 Add -Werror as a default. (#1193)
      af1445638 CI: Update homebrew Redis version. (#1191)
      6de326e87 Fix a typo in b6a052f.
      
      git-subtree-dir: deps/hiredis
      git-subtree-split: 60e5075d4ac77424809f855ba3e398df7aacefe8
      67ed3183
  20. 27 Jun, 2023 1 commit
  21. 30 May, 2023 1 commit
    • Yossi Gottlieb's avatar
      Squashed 'deps/hiredis/' changes from f8de9a4bd..b6a052fe0 · fad6c713
      Yossi Gottlieb authored
      b6a052fe0 Helper for setting TCP_USER_TIMEOUT socket option (#1188)
      3fa9b6944 Add RedisModule adapter (#1182)
      d13c091e9 Fix wincrypt symbols conflict
      5d84c8cfd Add a test ensuring we don't clobber connection error.
      3f95fcdae Don't attempt to set a timeout if we are in an error state.
      aacb84b8d Fix typo in makefile.
      563b062e3 Accept -nan per the RESP3 spec recommendation.
      04c1b5b02 Fix colliding option values
      4ca8e73f6 Rework searching for openssl
      cd208812f Attempt to find the correct path for openssl.
      011f7093c Allow specifying the keepalive interval
      e9243d4f7 Cmake static or shared (#1160)
      1cbd5bc76 Write a version file for the CMake package (#1165)
      6f5bae8c6 fix typo
      acd09461d CMakeLists.txt: respect BUILD_SHARED_LIBS
      97fcf0fd1 Add sdevent adapter
      ccff093bc Bump dev version for the next release cycle.
      c14775b4e Prepare for v1.1.0 GA
      f0bdf8405 Add support for nan in RESP3 double (#1133)
      991b0b0b3 Add an example that calls redisCommandArgv (#1140)
      a36686f84 CI updates (#1139)
      8ad4985e9 fix flag reference
      7583ebb1b Make freeing a NULL redisAsyncContext a no op.
      2c53dea7f Update version in dev branch.
      f063370ed Prepare for v1.1.0-rc1
      2b069573a CI fixes in preparation of release
      e1e9eb40d Add author information to release-drafter template.
      afc29ee1a Update for mingw cross compile
      ceb8a8815 fixed cpp build error with adapters/libhv.h
      3b15a04b5 Fixup of PR734: Coverage of hiredis.c (#1124)
      c245df9fb CMake corrections for building on Windows (#1122)
      9c338a598 Fix PUSH handler tests for Redis >= 7.0.5
      6d5c3ee74 Install on windows fixes (#1117)
      68b29e1ad Add timeout support to libhv adapter. (#1109)
      722e3409c Additional include directory given by pkg-config (#1118)
      bd9ccb8c4 Use __attribute__ when building with clang on windows
      5392adc26 set default SSL certificate directory
      560e66486 Minor refactor
      d756f68a5 Add libhv example to our standard Makefile
      a66916719 Add adapters/libhv
      855b48a81 Fix pkgconfig for hiredis_ssl
      79ae5ffc6 Fix protocol error (#1106)
      61b5b299f Use a windows specific keepalive function. (#1104)
      fce8abc1c Introduce .close method for redisContextFuncs
      cfb6ca881 Add REDIS_OPT_PREFER_UNSPEC (#1101)
      cc7c35ce6 Update documentation to explain redisConnectWithOptions.
      bc8d837b7 fix heap-buffer-overflow (#957)
      ca4a0e850 uvadapter: reduce number of uv_poll_start calls
      35d398c90 Fix cmake config path on Linux. CMake config files were installed to `/usr/local/share/hiredis`, which is not recognizable by `find_package()`. I'm not sure why it was set that way. Given the commit introducing it is for Windows, I keep that behavior consistent there, but fix the rest.
      10c78c6e1 Add possibility to prefer IPv6, IPv4 or unspecified
      1abe0c828 fuzzer: No alloc in redisFormatCommand() when fail
      329eaf9ba Fix heap-buffer-overflow issue in redisvFormatCommad
      eaae7321c Polling adapter requires sockcompat.h
      0a5fa3dde Regression test for off-by-one parsing error
      9e174e8f7 Add do while(0) protection for macros
      4ad99c69a Rework asSleep to be a generic millisleep function.
      75cb6c1ea Do store command timeout in the context for redisSetTimeout (#593)
      c57cad658 CMake: remove dict.c form hiredis_sources
      8491a65a9 Add Github Actions CI workflow for hiredis: Arm, Arm64, 386, windows. (#943)
      77e4f09ea Merge pull request #964 from afcidk/fix-createDoubleObject
      9219f7e7c Merge pull request #901 from devnexen/illumos_test_fix
      810cc6104 Merge pull request #905 from sundb/master
      df8b74d69 Merge pull request #1091 from redis/ssl-error-ub-fix
      0ed6cdec3 Fix some undefined behaviour
      507a6dcaa Merge pull request #1090 from Nordix/subscribe-oom-error
      b044eaa6a Copy error to redisAsyncContext when finding subscribe cb
      e0200b797 Merge pull request #1087 from redis/const-and-non-const-callback
      6a3e96ad2 Maintain backward compatibiliy withour onConnect callback.
      e7afd998f Merge pull request #1079 from SukkaW/drop-macos-10.15-runner
      17c8fe079 Merge pull request #931 from kristjanvalur/pr2
      b808c0c20 Merge pull request #1083 from chayim/ck-drafter
      367a82bf0 Merge pull request #1085 from stanhu/ssl-improve-options-setting
      71119a71d Make it possible to set SSL verify mode
      dd7979ac1 Merge pull request #1084 from stanhu/sh-improve-ssl-docs
      c71116178 Improve example for SSL initialization in README.md
      5c9b6b571 Release drafter
      a606ccf2a CI: use recommended `vmactions/freebsd-vm@v0`
      0865c115b Merge pull request #1080 from Nordix/readme-corrections
      f6cee7142 Fix README typos
      06be7ff31 Merge pull request #1050 from smmir-cent/fix-cmake-version
      7dd833d54 CI: bump macos runner version
      f69fac769 Drop `const` on redisAsyncContext in redisConnectCallback Since the callback is now re-entrant, it can call apis such as redisAsyncDisconnect()
      005d7edeb Support calling redisAsyncDisconnect from the onConnected callback, by deferring context deletion
      6ed060920 Add async regression test for issue #931
      eaa2a7ee7 Merge pull request #932 from kristjanvalur/pr3
      2ccef30f3 Add regression test for issue #945
      4b901d44a Initial async tests
      31c91408e Polling adapter and example
      8a15f4d65 Merge pull request #1057 from orgads/static-name
      902dd047f Merge pull request #1054 from kristjanvalur/pr08
      c78d0926b Merge pull request #1074 from michael-grunder/kristjanvalur-pr4
      2b115d56c Whitespace
      1343988ce Fix typos
      47b57aa24 Add some documentation on connect/disconnect callbacks and command callbacks
      a890d9ce2 Merge pull request #1073 from michael-grunder/kristjanvalur-pr1
      f246ee433 Whitespace, style
      94c1985bd Use correct type for getsockopt()
      5e002bc21 Support failed async connects on windows.
      5d68ad2f4 Merge pull request #1072 from michael-grunder/fix-redis7-unit-tests
      f4b6ed289 Fix tests so they work for Redis 7.0
      95a0c1283 Merge pull request #1058 from orgads/win64
      eedb37a65 Fix warnings on Win64
      47c3ecefc Merge pull request #1062 from yossigo/fix-push-notification-order
      e23d91c97 Merge pull request #1061 from yossigo/update-redis-apt
      34211ad54 Merge pull request #1063 from redis/fix-windows-tests
      9957af7e3 Whitelist hiredis repo path in cygwin
      b455b3381 Handle push notifications before or after reply.
      aed9ce446 Use official repository for redis package.
      d7683f35a Merge pull request #1047 from Nordix/unsubscribe-handling
      7c44a9d7e Merge pull request #1045 from Nordix/sds-updates
      dd4bf9783 Use the same name for static and shared libraries
      ff57c18b9 Embed debug information in windows static lib, rather than create a .pdb file
      8310ad4f5 fix cmake version
      7123b87f6 Handle any pipelined unsubscribe in async
      b6fb548fc Ignore pubsub replies without a channel/pattern
      00b82683b Handle overflows as errors instead of asserting
      64062a1d4 Catch size_t overflows in sds.c
      066c6de79 Use size_t/long to avoid truncation
      c6657ef65 Merge branch 'redis:master' into master
      50cdcab49 Fix potential fault at createDoubleObject
      fd033e983 Remove semicolon after do-while in _EL_CLEANUP
      664c415e7 Illumos test fixes, error message difference fot bad hostname test.
      
      git-subtree-dir: deps/hiredis
      git-subtree-split: b6a052fe0959dae69e16b9d74449faeb1b70dbe1
      fad6c713
  22. 24 May, 2023 1 commit
  23. 25 Apr, 2023 1 commit
  24. 18 Apr, 2023 1 commit
    • sundb's avatar
      Fix some compile warnings and errors when building with gcc-12 or clang (#12035) · 42c8c618
      sundb authored
      This PR is to fix the compilation warnings and errors generated by the latest
      complier toolchain, and to add a new runner of the latest toolchain for daily CI.
      
      ## Fix various compilation warnings and errors
      
      1) jemalloc.c
      
      COMPILER: clang-14 with FORTIFY_SOURCE
      
      WARNING:
      ```
      src/jemalloc.c:1028:7: warning: suspicious concatenation of string literals in an array initialization; did you mean to separate the elements with a comma? [-Wstring-concatenation]
                          "/etc/malloc.conf",
                          ^
      src/jemalloc.c:1027:3: note: place parentheses around the string literal to silence warning
                      "\"name\" of the file referenced by the symbolic link named "
                      ^
      ```
      
      REASON:  the compiler to alert developers to potential issues with string concatenation
      that may miss a comma,
      just like #9534 which misses a comma.
      
      SOLUTION: use `()` to tell the compiler that these two line strings are continuous.
      
      2) config.h
      
      COMPILER: clang-14 with FORTIFY_SOURCE
      
      WARNING:
      ```
      In file included from quicklist.c:36:
      ./config.h:319:76: warning: attribute declaration must precede definition [-Wignored-attributes]
      char *strcat(char *restrict dest, const char *restrict src) __attribute__((deprecated("please avoid use of unsafe C functions. prefer use of redis_strlcat instead")));
      ```
      
      REASON: Enabling _FORTIFY_SOURCE will cause the compiler to use `strcpy()` with check,
      it results in a deprecated attribute declaration after including <features.h>.
      
      SOLUTION: move the deprecated attribute declaration from config.h to fmacro.h before "#include <features.h>".
      
      3) networking.c
      
      COMPILER: GCC-12
      
      WARNING: 
      ```
      networking.c: In function ‘addReplyDouble.part.0’:
      networking.c:876:21: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
        876 |         dbuf[start] = '$';
            |                     ^
      networking.c:868:14: note: at offset -5 into destination object ‘dbuf’ of size 5152
        868 |         char dbuf[MAX_LONG_DOUBLE_CHARS+32];
            |              ^
      networking.c:876:21: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
        876 |         dbuf[start] = '$';
            |                     ^
      networking.c:868:14: note: at offset -6 into destination object ‘dbuf’ of size 5152
        868 |         char dbuf[MAX_LONG_DOUBLE_CHARS+32];
      ```
      
      REASON: GCC-12 predicts that digits10() may return 9 or 10 through `return 9 + (v >= 1000000000UL)`.
      
      SOLUTION: add an assert to let the compiler know the possible length;
      
      4) redis-cli.c & redis-benchmark.c
      
      COMPILER: clang-14 with FORTIFY_SOURCE
      
      WARNING:
      ```
      redis-benchmark.c:1621:2: warning: embedding a directive within macro arguments has undefined behavior [-Wembedded-directive] #ifdef USE_OPENSSL
      redis-cli.c:3015:2: warning: embedding a directive within macro arguments has undefined behavior [-Wembedded-directive] #ifdef USE_OPENSSL
      ```
      
      REASON: when _FORTIFY_SOURCE is enabled, the compiler will use the print() with
      check, which is a macro. this may result in the use of directives within the macro, which
      is undefined behavior.
      
      SOLUTION: move the directives-related code out of `print()`.
      
      5) server.c
      
      COMPILER: gcc-13 with FORTIFY_SOURCE
      
      WARNING:
      ```
      In function 'lookupCommandLogic',
          inlined from 'lookupCommandBySdsLogic' at server.c:3139:32:
      server.c:3102:66: error: '*(robj **)argv' may be used uninitialized [-Werror=maybe-uninitialized]
       3102 |     struct redisCommand *base_cmd = dictFetchValue(commands, argv[0]->ptr);
            |                                                              ~~~~^~~
      ```
      
      REASON: The compiler thinks that the `argc` returned by `sdssplitlen()` could be 0,
      resulting in an empty array of size 0 being passed to lookupCommandLogic.
      this should be a false positive, `argc` can't be 0 when strings are not NULL.
      
      SOLUTION: add an assert to let the compiler know that `argc` is positive.
      
      6) sha1.c
      
      COMPILER: gcc-12
      
      WARNING:
      ```
      In function ‘SHA1Update’,
          inlined from ‘SHA1Final’ at sha1.c:195:5:
      sha1.c:152:13: warning: ‘SHA1Transform’ reading 64 bytes from a region of size 0 [-Wstringop-overread]
        152 |             SHA1Transform(context->state, &data[i]);
            |             ^
      sha1.c:152:13: note: referencing argument 2 of type ‘const unsigned char[64]’
      sha1.c: In function ‘SHA1Final’:
      sha1.c:56:6: note: in a call to function ‘SHA1Transform’
         56 | void SHA1Transform(uint32_t state[5], const unsigned char buffer[64])
            |      ^
      In function ‘SHA1Update’,
          inlined from ‘SHA1Final’ at sha1.c:198:9:
      sha1.c:152:13: warning: ‘SHA1Transform’ reading 64 bytes from a region of size 0 [-Wstringop-overread]
        152 |             SHA1Transform(context->state, &data[i]);
            |             ^
      sha1.c:152:13: note: referencing argument 2 of type ‘const unsigned char[64]’
      sha1.c: In function ‘SHA1Final’:
      sha1.c:56:6: note: in a call to function ‘SHA1Transform’
         56 | void SHA1Transform(uint32_t state[5], const unsigned char buffer[64])
      ```
      
      REASON: due to the bug[https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80922], when
      enable LTO, gcc-12 will not see `diagnostic ignored "-Wstringop-overread"`, resulting in a warning.
      
      SOLUTION: temporarily set SHA1Update to noinline to avoid compiler warnings due
      to LTO being enabled until the above gcc bug is fixed.
      
      7) zmalloc.h
      
      COMPILER: GCC-12
      
      WARNING: 
      ```
      In function ‘memset’,
          inlined from ‘moduleCreateContext’ at module.c:877:5,
          inlined from ‘RM_GetDetachedThreadSafeContext’ at module.c:8410:5:
      /usr/include/x86_64-linux-gnu/bits/string_fortified.h:59:10: warning: ‘__builtin_memset’ writing 104 bytes into a region of size 0 overflows the destination [-Wstringop-overflow=]
         59 |   return __builtin___memset_chk (__dest, __ch, __len,
      ```
      
      REASON: due to the GCC-12 bug [https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96503],
      GCC-12 cannot see alloc_size, which causes GCC to think that the actual size of memory
      is 0 when checking with __glibc_objsize0().
      
      SOLUTION: temporarily set malloc-related interfaces to `noinline` to avoid compiler warnings
      due to LTO being enabled until the above gcc bug is fixed.
      
      ## Other changes
      1) Fixed `ps -p [pid]`  doesn't output `<defunct>` when using procps 4.x causing `replication
        child dies when parent is killed - diskless` test to fail.
      2) Add a new fortify CI with GCC-13 and ubuntu-lunar docker image.
      42c8c618
  25. 12 Apr, 2023 2 commits
    • Binbin's avatar
      Don't pass --fail-commands-not-all-hit to validator if we don't run the full testsuite (#12023) · 810ea67b
      Binbin authored
      In daily.yml, if the input suggests we don't run the full testsuite,
      do not pass --fail-commands-not-all-hit to the validator.
      
      This fixes the first point in #11954. Credit goes to the comment
      on the open issue for GH actions: actions/runner#409
      
      Also improve prints to show the dispatch arguments in every job.
      810ea67b
    • Oran Agra's avatar
      Attempt to solve MacOS CI issues in GH Actions (#12013) · 997fa41e
      Oran Agra authored
      The MacOS CI in github actions often hangs without any logs. GH argues that
      it's due to resource utilization, either running out of disk space, memory, or CPU
      starvation, and thus the runner is terminated.
      
      This PR contains multiple attempts to resolve this:
      1. introducing pause_process instead of SIGSTOP, which waits for the process
        to stop before resuming the test, possibly resolving race conditions in some tests,
        this was a suspect since there was one test that could result in an infinite loop in that
       case, in practice this didn't help, but still a good idea to keep.
      2. disable the `save` config in many tests that don't need it, specifically ones that use
        heavy writes and could create large files.
      3. change the `populate` proc to use short pipeline rather than an infinite one.
      4. use `--clients 1` in the macos CI so that we don't risk running multiple resource
        demanding tests in parallel.
      5. enable `--verbose` to be repeated to elevate verbosity and print more info to stdout
        when a test or a server starts.
      997fa41e
  26. 05 Apr, 2023 1 commit
    • Oran Agra's avatar
      Increase threshold for flaky cache reclaim test (#12004) · f263b6da
      Oran Agra authored
      This test produces 1GB of data and moves it around, and was expecting less
      than 500kb to be present in the system page cache.
      It sometimes fails with up to some 6mb in the page cache (0 in the actual RDB files),
      increasing the threshold. It looks like some background tasks in the container are
      occupying the page cache.
      
      It is safe to ignore the above since we also explicitly check the pages of our dump.rdb
      are not cached (matching `vmtouch -v` to `0%`).
      An additional fix is to match ` 0%` (add space), so that we don't successfully match `10%`.
      
      details in https://github.com/redis/redis/pull/11818
      f263b6da
  27. 23 Mar, 2023 1 commit
  28. 11 Mar, 2023 1 commit
    • guybe7's avatar
      Add reply_schema to command json files (internal for now) (#10273) · 4ba47d2d
      guybe7 authored
      Work in progress towards implementing a reply schema as part of COMMAND DOCS, see #9845
      Since ironing the details of the reply schema of each and every command can take a long time, we
      would like to merge this PR when the infrastructure is ready, and let this mature in the unstable branch.
      Meanwhile the changes of this PR are internal, they are part of the repo, but do not affect the produced build.
      
      ### Background
      In #9656 we add a lot of information about Redis commands, but we are missing information about the replies
      
      ### Motivation
      1. Documentation. This is the primary goal.
      2. It should be possible, based on the output of COMMAND, to be able to generate client code in typed
        languages. In order to do that, we need Redis to tell us, in detail, what each reply looks like.
      3. We would like to build a fuzzer that verifies the reply structure (for now we use the existing
        testsuite, see the "Testing" section)
      
      ### Schema
      The idea is to supply some sort of schema for the various replies of each command.
      The schema will describe the conceptual structure of the reply (for generated clients), as defined in RESP3.
      Note that the reply structure itself may change, depending on the arguments (e.g. `XINFO STREAM`, with
      and without the `FULL` modifier)
      We decided to use the standard json-schema (see https://json-schema.org/) as the reply-schema.
      
      Example for `BZPOPMIN`:
      ```
      "reply_schema": {
          "oneOf": [
              {
                  "description": "Timeout reached and no elements were popped.",
                  "type": "null"
              },
              {
                  "description": "The keyname, popped member, and its score.",
                  "type": "array",
                  "minItems": 3,
                  "maxItems": 3,
                  "items": [
                      {
                          "description": "Keyname",
                          "type": "string"
                      },
                      {
                          "description": "Member",
                          "type": "string"
                      },
                      {
                          "description": "Score",
                          "type": "number"
                      }
                  ]
              }
          ]
      }
      ```
      
      #### Notes
      1.  It is ok that some commands' reply structure depends on the arguments and it's the caller's responsibility
        to know which is the relevant one. this comes after looking at other request-reply systems like OpenAPI,
        where the reply schema can also be oneOf and the caller is responsible to know which schema is the relevant one.
      2. The reply schemas will describe RESP3 replies only. even though RESP3 is structured, we want to use reply
        schema for documentation (and possibly to create a fuzzer that validates the replies)
      3. For documentation, the description field will include an explanation of the scenario in which the reply is sent,
        including any relation to arguments. for example, for `ZRANGE`'s two schemas we will need to state that one
        is with `WITHSCORES` and the other is without.
      4. For documentation, there will be another optional field "notes" in which we will add a short description of
        the representation in RESP2, in case it's not trivial (RESP3's `ZRANGE`'s nested array vs. RESP2's flat
        array, for example)
      
      Given the above:
      1. We can generate the "return" section of all commands in [redis-doc](https://redis.io/commands/)
        (given that "description" and "notes" are comprehensive enough)
      2. We can generate a client in a strongly typed language (but the return type could be a conceptual
        `union` and the caller needs to know which schema is relevant). see the section below for RESP2 support.
      3. We can create a fuzzer for RESP3.
      
      ### Limitations (because we are using the standard json-schema)
      The problem is that Redis' replies are more diverse than what the json format allows. This means that,
      when we convert the reply to a json (in order to validate the schema against it), we lose information (see
      the "Testing" section below).
      The other option would have been to extend the standard json-schema (and json format) to include stuff
      like sets, bulk-strings, error-string, etc. but that would mean also extending the schema-validator - and that
      seemed like too much work, so we decided to compromise.
      
      Examples:
      1. We cannot tell the difference between an "array" and a "set"
      2. We cannot tell the difference between simple-string and bulk-string
      3. we cannot verify true uniqueness of items in commands like ZRANGE: json-schema doesn't cover the
        case of two identical members with different scores (e.g. `[["m1",6],["m1",7]]`) because `uniqueItems`
        compares (member,score) tuples and not just the member name. 
      
      ### Testing
      This commit includes some changes inside Redis in order to verify the schemas (existing and future ones)
      are indeed correct (i.e. describe the actual response of Redis).
      To do that, we added a debugging feature to Redis that causes it to produce a log of all the commands
      it executed and their replies.
      For that, Redis needs to be compiled with `-DLOG_REQ_RES` and run with
      `--reg-res-logfile <file> --client-default-resp 3` (the testsuite already does that if you run it with
      `--log-req-res --force-resp3`)
      You should run the testsuite with the above args (and `--dont-clean`) in order to make Redis generate
      `.reqres` files (same dir as the `stdout` files) which contain request-response pairs.
      These files are later on processed by `./utils/req-res-log-validator.py` which does:
      1. Goes over req-res files, generated by redis-servers, spawned by the testsuite (see logreqres.c)
      2. For each request-response pair, it validates the response against the request's reply_schema
        (obtained from the extended COMMAND DOCS)
      5. In order to get good coverage of the Redis commands, and all their different replies, we chose to use
        the existing redis test suite, rather than attempt to write a fuzzer.
      
      #### Notes about RESP2
      1. We will not be able to use the testing tool to verify RESP2 replies (we are ok with that, it's time to
        accept RESP3 as the future RESP)
      2. Since the majority of the test suite is using RESP2, and we want the server to reply with RESP3
        so that we can validate it, we will need to know how to convert the actual reply to the one expected.
         - number and boolean are always strings in RESP2 so the conversion is easy
         - objects (maps) are always a flat array in RESP2
         - others (nested array in RESP3's `ZRANGE` and others) will need some special per-command
           handling (so the client will not be totally auto-generated)
      
      Example for ZRANGE:
      ```
      "reply_schema": {
          "anyOf": [
              {
                  "description": "A list of member elements",
                  "type": "array",
                  "uniqueItems": true,
                  "items": {
                      "type": "string"
                  }
              },
              {
                  "description": "Members and their scores. Returned in case `WITHSCORES` was used.",
                  "notes": "In RESP2 this is returned as a flat array",
                  "type": "array",
                  "uniqueItems": true,
                  "items": {
                      "type": "array",
                      "minItems": 2,
                      "maxItems": 2,
                      "items": [
                          {
                              "description": "Member",
                              "type": "string"
                          },
                          {
                              "description": "Score",
                              "type": "number"
                          }
                      ]
                  }
              }
          ]
      }
      ```
      
      ### Other changes
      1. Some tests that behave differently depending on the RESP are now being tested for both RESP,
        regardless of the special log-req-res mode ("Pub/Sub PING" for example)
      2. Update the history field of CLIENT LIST
      3. Added basic tests for commands that were not covered at all by the testsuite
      
      ### TODO
      
      - [x] (maybe a different PR) add a "condition" field to anyOf/oneOf schemas that refers to args. e.g.
        when `SET` return NULL, the condition is `arguments.get||arguments.condition`, for `OK` the condition
        is `!arguments.get`, and for `string` the condition is `arguments.get` - https://github.com/redis/redis/issues/11896
      - [x] (maybe a different PR) also run `runtest-cluster` in the req-res logging mode
      - [x] add the new tests to GH actions (i.e. compile with `-DLOG_REQ_RES`, run the tests, and run the validator)
      - [x] (maybe a different PR) figure out a way to warn about (sub)schemas that are uncovered by the output
        of the tests - https://github.com/redis/redis/issues/11897
      - [x] (probably a separate PR) add all missing schemas
      - [x] check why "SDOWN is triggered by misconfigured instance replying with errors" fails with --log-req-res
      - [x] move the response transformers to their own file (run both regular, cluster, and sentinel tests - need to
        fight with the tcl including mechanism a bit)
      - [x] issue: module API - https://github.com/redis/redis/issues/11898
      - [x] (probably a separate PR): improve schemas: add `required` to `object`s - https://github.com/redis/redis/issues/11899
      
      Co-authored-by: default avatarOzan Tezcan <ozantezcan@gmail.com>
      Co-authored-by: default avatarHanna Fadida <hanna.fadida@redislabs.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarShaya Potter <shaya@redislabs.com>
      4ba47d2d
  29. 19 Feb, 2023 1 commit
    • Oran Agra's avatar
      Stablize page reclaim CI test (#11818) · 3ac83577
      Oran Agra authored
      stabilize the test introduced in #11248
      * remove random aspect of the test by using DEBUG POPULATE instead of redis-benchmark
      * disable rdbcompression, so that the rdb file is always about 1GB.
      
      when fadvise was disabled, i get about 1GB in the page cace
      when enabled i get less than 200KB
      so for now, i'll keep the 500kb threshold.
      3ac83577
  30. 16 Feb, 2023 1 commit
  31. 12 Feb, 2023 1 commit
    • Tian's avatar
      Reclaim page cache of RDB file (#11248) · 7dae142a
      Tian authored
      # Background
      The RDB file is usually generated and used once and seldom used again, but the content would reside in page cache until OS evicts it. A potential problem is that once the free memory exhausts, the OS have to reclaim some memory from page cache or swap anonymous page out, which may result in a jitters to the Redis service.
      
      Supposing an exact scenario, a high-capacity machine hosts many redis instances, and we're upgrading the Redis together. The page cache in host machine increases as RDBs are generated. Once the free memory drop into low watermark(which is more likely to happen in older Linux kernel like 3.10, before [watermark_scale_factor](https://lore.kernel.org/lkml/1455813719-2395-1-git-send-email-hannes@cmpxchg.org/) is introduced, the `low watermark` is linear to `min watermark`, and there'is not too much buffer space for `kswapd` to be wake up to reclaim memory), a `direct reclaim` happens, which means the process would stall to wait for memory allocation.
      
      # What the PR does
      The PR introduces a capability to reclaim the cache when the RDB is operated. Generally there're two cases, read and write the RDB. For read it's a little messy to address the incremental reclaim, so the reclaim is done in one go in background after the load is finished to avoid blocking the work thread. For write, incremental reclaim amortizes the work of reclaim so no need to put it into background, and the peak watermark of cache can be reduced in this way.
      
      Two cases are addresses specially, replication and restart, for both of which the cache is leveraged to speed up the processing, so the reclaim is postponed to a right time. To do this, a flag is added to`rdbSave` and `rdbLoad` to control whether the cache need to be kept, with the default value false.
      
      # Something deserve noting
      1. Though `posix_fadvise` is the POSIX standard, but only few platform support it, e.g. Linux, FreeBSD 10.0.
      2. In Linux `posix_fadvise` only take effect on writeback-ed pages, so a `sync`(or `fsync`, `fdatasync`) is needed to flush the dirty page before `posix_fadvise` if we reclaim write cache.
      
      # About test
      A unit test is added to verify the effect of `posix_fadvise`.
      In integration test overall cache increase is checked, as well as the cache backed by RDB as a specific TCL test is executed in isolated Github action job.
      7dae142a
  32. 21 Nov, 2022 1 commit
    • Binbin's avatar
      Fix sentinel update loglevel tls test (#11528) · 0f857131
      Binbin authored
      Apparently we used to set `loglevel debug` for tls in spawn_instance.
      I.e. cluster and sentinel tests used to run with debug logging, only when tls mode was enabled.
      this was probably a leftover from when creating the tls mode tests.
      it cause a new test created for #11214 to fail in tls mode.
      
      At the same time, in order to better distinguish the tests, change the
      name of `test-centos7-tls` to `test-centos7-tls-module`, change the name
      of `test-centos7-tls-no-tls` to `test-centos7-tls-module-no-tls`.
      
      Note that in `test-centos7-tls-module`, we did not pass `--tls-module`
      in sentinel test because it is not supported, see 4faddf18
      
      , added in #9320.
      So only `test-ubuntu-tls` fails in daily CI.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      0f857131
  33. 04 Nov, 2022 1 commit
    • Binbin's avatar
      Bump vmactions/freebsd-vm to 0.3.0 to fix FreeBSD daily (#11476) · 5246bf45
      Binbin authored
      Our FreeBSD daily has been failing recently:
      ```
        Config file: freebsd-13.1.conf
        cd: /Users/runner/work/redis/redis: No such file or directory
        gmake: *** No targets specified and no makefile found.  Stop.
      ```
      
      Upgrade vmactions/freebsd-vm to the latest version (0.3.0) can work.
      I've tested it, but don't know why, but first let's fix it.
      5246bf45
  34. 31 Aug, 2022 1 commit
  35. 23 Aug, 2022 1 commit
    • Oran Agra's avatar
      Build TLS as a loadable module · 4faddf18
      Oran Agra authored
      
      
      * Support BUILD_TLS=module to be loaded as a module via config file or
        command line. e.g. redis-server --loadmodule redis-tls.so
      * Updates to redismodule.h to allow it to be used side by side with
        server.h by defining REDISMODULE_CORE_MODULE
      * Changes to server.h, redismodule.h and module.c to avoid repeated
        type declarations (gcc 4.8 doesn't like these)
      * Add a mechanism for non-ABI neutral modules (ones who include
        server.h) to refuse loading if they detect not being built together with
        redis (release.c)
      * Fix wrong signature of RedisModuleDefragFunc, this could break
        compilation of a module, but not the ABI
      * Move initialization of listeners in server.c to be after loading
        the modules
      * Config TLS after initialization of listeners
      * Init cluster after initialization of listeners
      * Add TLS module to CI
      * Fix a test suite race conditions:
        Now that the listeners are initialized later, it's not sufficient to
        wait for the PID message in the log, we need to wait for the "Server
        Initialized" message.
      * Fix issues with moduleconfigs test as a result from start_server
        waiting for "Server Initialized"
      * Fix issues with modules/infra test as a result of an additional module
        present
      
      Notes about Sentinel:
      Sentinel can't really rely on the tls module, since it uses hiredis to
      initiate connections and depends on OpenSSL (won't be able to use any
      other connection modules for that), so it was decided that when TLS is
      built as a module, sentinel does not support TLS at all.
      This means that it keeps using redis_tls_ctx and redis_tls_client_ctx directly.
      
      Example code of config in redis-tls.so(may be use in the future):
      RedisModuleString *tls_cfg = NULL;
      
      void tlsInfo(RedisModuleInfoCtx *ctx, int for_crash_report) {
          UNUSED(for_crash_report);
          RedisModule_InfoAddSection(ctx, "");
          RedisModule_InfoAddFieldLongLong(ctx, "var", 42);
      }
      
      int tlsCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
      {
          if (argc != 2) return RedisModule_WrongArity(ctx);
          return RedisModule_ReplyWithString(ctx, argv[1]);
      }
      
      RedisModuleString *getStringConfigCommand(const char *name, void *privdata) {
          REDISMODULE_NOT_USED(name);
          REDISMODULE_NOT_USED(privdata);
          return tls_cfg;
      }
      
      int setStringConfigCommand(const char *name, RedisModuleString *new, void *privdata, RedisModuleString **err) {
          REDISMODULE_NOT_USED(name);
          REDISMODULE_NOT_USED(err);
          REDISMODULE_NOT_USED(privdata);
          if (tls_cfg) RedisModule_FreeString(NULL, tls_cfg);
          RedisModule_RetainString(NULL, new);
          tls_cfg = new;
          return REDISMODULE_OK;
      }
      
      int RedisModule_OnLoad(void *ctx, RedisModuleString **argv, int argc)
      {
          ....
          if (RedisModule_CreateCommand(ctx,"tls",tlsCommand,"",0,0,0) == REDISMODULE_ERR)
              return REDISMODULE_ERR;
      
          if (RedisModule_RegisterStringConfig(ctx, "cfg", "", REDISMODULE_CONFIG_DEFAULT, getStringConfigCommand, setStringConfigCommand, NULL, NULL) == REDISMODULE_ERR)
              return REDISMODULE_ERR;
      
          if (RedisModule_LoadConfigs(ctx) == REDISMODULE_ERR) {
              if (tls_cfg) {
                  RedisModule_FreeString(ctx, tls_cfg);
                  tls_cfg = NULL;
              }
              return REDISMODULE_ERR;
          }
          ...
      }
      Co-authored-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      4faddf18
  36. 02 Aug, 2022 1 commit
  37. 19 Jul, 2022 1 commit
  38. 06 Jun, 2022 1 commit