1. 23 Aug, 2022 1 commit
    • Oran Agra's avatar
      Build TLS as a loadable module · 4faddf18
      Oran Agra authored
      
      
      * Support BUILD_TLS=module to be loaded as a module via config file or
        command line. e.g. redis-server --loadmodule redis-tls.so
      * Updates to redismodule.h to allow it to be used side by side with
        server.h by defining REDISMODULE_CORE_MODULE
      * Changes to server.h, redismodule.h and module.c to avoid repeated
        type declarations (gcc 4.8 doesn't like these)
      * Add a mechanism for non-ABI neutral modules (ones who include
        server.h) to refuse loading if they detect not being built together with
        redis (release.c)
      * Fix wrong signature of RedisModuleDefragFunc, this could break
        compilation of a module, but not the ABI
      * Move initialization of listeners in server.c to be after loading
        the modules
      * Config TLS after initialization of listeners
      * Init cluster after initialization of listeners
      * Add TLS module to CI
      * Fix a test suite race conditions:
        Now that the listeners are initialized later, it's not sufficient to
        wait for the PID message in the log, we need to wait for the "Server
        Initialized" message.
      * Fix issues with moduleconfigs test as a result from start_server
        waiting for "Server Initialized"
      * Fix issues with modules/infra test as a result of an additional module
        present
      
      Notes about Sentinel:
      Sentinel can't really rely on the tls module, since it uses hiredis to
      initiate connections and depends on OpenSSL (won't be able to use any
      other connection modules for that), so it was decided that when TLS is
      built as a module, sentinel does not support TLS at all.
      This means that it keeps using redis_tls_ctx and redis_tls_client_ctx directly.
      
      Example code of config in redis-tls.so(may be use in the future):
      RedisModuleString *tls_cfg = NULL;
      
      void tlsInfo(RedisModuleInfoCtx *ctx, int for_crash_report) {
          UNUSED(for_crash_report);
          RedisModule_InfoAddSection(ctx, "");
          RedisModule_InfoAddFieldLongLong(ctx, "var", 42);
      }
      
      int tlsCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
      {
          if (argc != 2) return RedisModule_WrongArity(ctx);
          return RedisModule_ReplyWithString(ctx, argv[1]);
      }
      
      RedisModuleString *getStringConfigCommand(const char *name, void *privdata) {
          REDISMODULE_NOT_USED(name);
          REDISMODULE_NOT_USED(privdata);
          return tls_cfg;
      }
      
      int setStringConfigCommand(const char *name, RedisModuleString *new, void *privdata, RedisModuleString **err) {
          REDISMODULE_NOT_USED(name);
          REDISMODULE_NOT_USED(err);
          REDISMODULE_NOT_USED(privdata);
          if (tls_cfg) RedisModule_FreeString(NULL, tls_cfg);
          RedisModule_RetainString(NULL, new);
          tls_cfg = new;
          return REDISMODULE_OK;
      }
      
      int RedisModule_OnLoad(void *ctx, RedisModuleString **argv, int argc)
      {
          ....
          if (RedisModule_CreateCommand(ctx,"tls",tlsCommand,"",0,0,0) == REDISMODULE_ERR)
              return REDISMODULE_ERR;
      
          if (RedisModule_RegisterStringConfig(ctx, "cfg", "", REDISMODULE_CONFIG_DEFAULT, getStringConfigCommand, setStringConfigCommand, NULL, NULL) == REDISMODULE_ERR)
              return REDISMODULE_ERR;
      
          if (RedisModule_LoadConfigs(ctx) == REDISMODULE_ERR) {
              if (tls_cfg) {
                  RedisModule_FreeString(ctx, tls_cfg);
                  tls_cfg = NULL;
              }
              return REDISMODULE_ERR;
          }
          ...
      }
      Co-authored-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      4faddf18
  2. 22 Aug, 2022 10 commits
    • zhenwei pi's avatar
      Introduce .listen into connection type · 0b27cfe3
      zhenwei pi authored
      
      
      Introduce listen method into connection type, this allows no hard code
      of listen logic. Originally, we initialize server during startup like
      this:
          if (server.port)
              listenToPort(server.port,&server.ipfd);
          if (server.tls_port)
              listenToPort(server.port,&server.tlsfd);
          if (server.unixsocket)
              anetUnixServer(...server.unixsocket...);
      
          ...
          if (createSocketAcceptHandler(&server.ipfd, acceptTcpHandler) != C_OK)
          if (createSocketAcceptHandler(&server.tlsfd, acceptTcpHandler) != C_OK)
          if (createSocketAcceptHandler(&server.sofd, acceptTcpHandler) != C_OK)
          ...
      
      If a new connection type gets supported, we have to add more hard code
      to setup listener.
      
      Introduce .listen and refactor listener, and Unix socket supports this.
      this allows to setup listener arguments and create listener in a loop.
      
      What's more, '.listen' is defined in connection.h, so we should include
      server.h to import 'struct socketFds', but server.h has already include
      'connection.h'. To avoid including loop(also to make code reasonable),
      define 'struct connListener' in connection.h instead of 'struct socketFds'
      in server.h. This leads this commit to get more changes.
      
      There are more fields in 'struct connListener', hence it's possible to
      simplify changeBindAddr & applyTLSPort() & updatePort() into a single
      logic: update the listener config from the server.xxx, and re-create
      the listener.
      
      Because of the new field 'priv' in struct connListener, we expect to pass
      this to the accept handler(even it's not used currently), this may be used
      in the future.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      0b27cfe3
    • zhenwei pi's avatar
      Use connection name of string · 45617385
      zhenwei pi authored
      
      
      Suggested by Oran, use an array to store all the connection types
      instead of a linked list, and use connection name of string. The index
      of a connection is dynamically allocated.
      
      Currently we support max 8 connection types, include:
      - tcp
      - unix socket
      - tls
      
      and RDMA is in the plan, then we have another 4 types to support, it
      should be enough in a long time.
      
      Introduce 3 functions to get connection type by a fast path:
      - connectionTypeTcp()
      - connectionTypeTls()
      - connectionTypeUnix()
      
      Note that connectionByType() is designed to use only in unlikely code path.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      45617385
    • zhenwei pi's avatar
      Abstract accept handler · 0ae02ce9
      zhenwei pi authored
      
      
      Abstract accept handler for socket&TLS, and add helper function
      'connAcceptHandler' to get accept handler by specified type.
      
      Also move acceptTcpHandler into socket.c, and move
      acceptTLSHandler into tls.c.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      0ae02ce9
    • zhenwei pi's avatar
      Fully abstract connection type · 1234e3a5
      zhenwei pi authored
      
      
      Abstract common interface of connection type, so Redis can hide the
      implementation and uplayer only calls connection API without macro.
      
                     uplayer
                        |
                 connection layer
                   /          \
                socket        TLS
      
      Currently, for both socket and TLS, all the methods of connection type
      are declared as static functions.
      
      It's possible to build TLS(even socket) as a shared library, and Redis
      loads it dynamically in the next step.
      
      Also add helper function connTypeOfCluster() and
      connTypeOfReplication() to simplify the code:
      link->conn = server.tls_cluster ? connCreateTLS() : connCreateSocket();
      -> link->conn = connCreate(connTypeOfCluster());
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      1234e3a5
    • zhenwei pi's avatar
      Introduce TLS specified APIs · c4c02f80
      zhenwei pi authored
      
      
      Introduce .get_peer_cert, .get_ctx and .get_client_ctx for TLS, also
      hide redis_tls_ctx & redis_tls_client_ctx.
      
      Then outside could access the variables by connection API only:
      - redis_tls_ctx -> connTypeGetCtx(CONN_TYPE_TLS)
      - redis_tls_client_ctx -> connTypeGetClientCtx(CONN_TYPE_TLS)
      
      Also remove connTLSGetPeerCert(), use connGetPeerCert() instead.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      c4c02f80
    • zhenwei pi's avatar
      Introduce pending data for connection type · 709b55b0
      zhenwei pi authored
      
      
      Introduce .has_pending_data and .process_pending_data for connection
      type, and hide tlsHasPendingData() and tlsProcessPendingData(). Also
      set .has_pending_data and .process_pending_data as NULL explicitly in
      socket.c.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      709b55b0
    • zhenwei pi's avatar
      Introduce connection layer framework · 8234a512
      zhenwei pi authored
      
      
      Use connTypeRegister() to register a connection type into redis, and
      query connection by connectionByType() via type.
      
      With this change, we can hide TLS specified methods into connection
      type:
      - void tlsInit(void);
      - void tlsCleanup(void);
      - int tlsConfigure(redisTLSContextConfig *ctx_config);
      - int isTlsConfigured(void);
      
      Merge isTlsConfigured & tlsConfigure, use an argument *reconfigure*
      to distinguish:
         tlsConfigure(&server.tls_ctx_config)
      -> onnTypeConfigure(CONN_TYPE_TLS, &server.tls_ctx_config, 1)
      
         isTlsConfigured() && tlsConfigure(&server.tls_ctx_config)
      -> connTypeConfigure(CONN_TYPE_TLS, &server.tls_ctx_config, 0)
      
      Finally, we can remove USE_OPENSSL from config.c. If redis is built
      without TLS, and still run redis with TLS, then redis reports:
       # Missing implement of connection type 1
       # Failed to configure TLS. Check logs for more info.
      
      The log can be optimised, let's leave it in the future. Maybe we can
      use connection type as a string.
      
      Although uninitialized fields of a static struct are zero, we still
      set them as NULL explicitly in socket.c, let them clear to read & maintain:
          .init = NULL,
          .cleanup = NULL,
          .configure = NULL,
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      8234a512
    • zhenwei pi's avatar
      Introduce connAddr · bff7ecc7
      zhenwei pi authored
      
      
      Originally, connPeerToString is designed to get the address info from
      socket only(for both TCP & TLS), and the API 'connPeerToString' is
      oriented to operate a FD like:
      int connPeerToString(connection *conn, char *ip, size_t ip_len, int *port) {
          return anetFdToString(conn ? conn->fd : -1, ip, ip_len, port, FD_TO_PEER_NAME);
      }
      
      Introduce connAddr and implement .addr method for socket and TLS,
      thus the API 'connAddr' and 'connFormatAddr' become oriented to a
      connection like:
      static inline int connAddr(connection *conn, char *ip, size_t ip_len, int *port, int remote) {
          if (conn && conn->type->addr) {
              return conn->type->addr(conn, ip, ip_len, port, remote);
          }
      
          return -1;
      }
      
      Also remove 'FD_TO_PEER_NAME' & 'FD_TO_SOCK_NAME', use a boolean type
      'remote' to get local/remote address of a connection.
      
      With these changes, it's possible to support the other connection
      types which does not use socket(Ex, RDMA).
      
      Thanks to Oran for suggestions!
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      bff7ecc7
    • zhenwei pi's avatar
      Reorder methods for ConnectionType · b9d77288
      zhenwei pi authored
      
      
      Reorder methods for CT_Socket & CT_TLS, also add comments to make the
      methods clear.
      
      Also move the CT_TLS to the end of file, other methods can be static
      in the next step.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      b9d77288
    • zhenwei pi's avatar
      Move 'connGetSocketError' to 'anetGetError' · 8045e26e
      zhenwei pi authored
      
      
      getsockopt is part of TCP, rename 'connGetSocketError' to
      'anetGetError', and move it into anet.c.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      8045e26e
  3. 05 Jul, 2022 1 commit
  4. 03 Jul, 2022 1 commit
  5. 22 Feb, 2022 1 commit
    • Andy Pan's avatar
      Reduce system calls of write for client->reply by introducing writev (#9934) · 496375fc
      Andy Pan authored
      There are scenarios where it results in many small objects in the reply list,
      such as commands heavily using deferred array replies (`addReplyDeferredLen`).
      E.g. what COMMAND command and CLUSTER SLOTS used to do (see #10056, #7123),
      but also in case of a transaction or a pipeline of commands that use just one differed array reply.
      
      We used to have to run multiple loops along with multiple calls to `write()` to send data back to
      peer based on the current code, but by means of `writev()`, we can gather those scattered
      objects in reply list and include the static reply buffer as well, then send it by one system call,
      that ought to achieve higher performance.
      
      In the case of TLS,  we simply check and concatenate buffers into one big buffer and send it
      away by one call to `connTLSWrite()`, if the amount of all buffers exceeds `NET_MAX_WRITES_PER_EVENT`,
      then invoke `connTLSWrite()` multiple times to avoid a huge massive of memory copies.
      
      Note that aside of reducing system calls, this change will also reduce the amount of
      small TCP packets sent.
      496375fc
  6. 15 Feb, 2022 1 commit
    • Yossi Gottlieb's avatar
      Fix OpenSSL 3.0.x related issues. (#10291) · 3881f785
      Yossi Gottlieb authored
      * Drop obsolete initialization calls.
      * Use decoder API for DH parameters.
      * Enable auto DH parameters if not explicitly used, which should be the
        preferred configuration going forward.
      3881f785
  7. 01 Dec, 2021 1 commit
    • yoav-steinberg's avatar
      Multiparam config set (#9748) · 0e5b813e
      yoav-steinberg authored
      We can now do: `config set maxmemory 10m repl-backlog-size 5m`
      
      ## Basic algorithm to support "transaction like" config sets:
      
      1. Backup all relevant current values (via get).
      2. Run "verify" and "set" on everything, if we fail run "restore".
      3. Run "apply" on everything (optional optimization: skip functions already run). If we fail run "restore".
      4. Return success.
      
      ### restore
      1. Run set on everything in backup. If we fail log it and continue (this puts us in an undefined
         state but we decided it's better than the alternative of panicking). This indicates either a bug
         or some unsupported external state.
      2. Run apply on everything in backup (optimization: skip functions already run). If we fail log
         it (see comment above).
      3. Return error.
      
      ## Implementation/design changes:
      * Apply function are idempotent (have no effect if they are run more than once for the same config).
      * No indication in set functions if we're reading the config or running from the `CONFIG SET` command
         (removed `update` argument).
      * Set function should set some config variable and assume an (optional) apply function will use that
         later to apply. If we know this setting can be safely applied immediately and can always be reverted
         and doesn't depend on any other configuration we can apply immediately from within the set function
         (and not store the setting anywhere). This is the case of this `dir` config, for example, which has no
         apply function. No apply function is need also in the case that setting the variable in the `server` struct
         is all that needs to be done to make the configuration take effect. Note that the original concept of `update_fn`,
         which received the old and new values was removed and replaced by the optional apply function.
      * Apply functions use settings written to the `server` struct and don't receive any inputs.
      * I take care that for the generic (non-special) configs if there's no change I avoid calling the setter (possible
         optimization: avoid calling the apply function as well).
      * Passing the same config parameter more than once to `config set` will fail. You can't do `config set my-setting
         value1 my-setting value2`.
      
      Note that getting `save` in the context of the conf file parsing to work here as before was a pain.
      The conf file supports an aggregate `save` definition, where each `save` line is added to the server's
      save params. This is unlike any other line in the config file where each line overwrites any previous
      configuration. Since we now support passing multiple save params in a single line (see top comments
      about `save` in https://github.com/redis/redis/pull/9644) we should deprecate the aggregate nature of
      this config line and perhaps reduce this ugly code in the future.
      0e5b813e
  8. 08 Nov, 2021 1 commit
    • Yossi Gottlieb's avatar
      Fix EINTR test failures. (#9751) · a1aba4bf
      Yossi Gottlieb authored
      * Clean up EINTR handling so EINTR will not change connection state to begin with.
      * On TLS, catch EINTR and return it as-is before going through OpenSSL error handling (which seems to not distinguish it from EAGAIN).
      a1aba4bf
  9. 11 Sep, 2021 1 commit
  10. 14 Jul, 2021 2 commits
  11. 10 Jun, 2021 1 commit
    • Binbin's avatar
      Fixed some typos, add a spell check ci and others minor fix (#8890) · 0bfccc55
      Binbin authored
      This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes.
      This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything,
      but at least it is also unlikely to cause false positives.
      
      Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use.
      
      Here's a summary of other changes:
      1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces).
      2. Outdated function / variable / argument names in comments
      3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString.
      4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751
      5. Some outdated https link URLs.
      6. Fix some outdated comment. Such as:
          - In README: about the rdb, we used to said create a `thread`, change to `process`
          - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey)
          - notifyKeyspaceEvent fucntion comment (add type arg)
          - Some others minor fix in comment (Most of them are incorrectly quoted by variable names)
      7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
      0bfccc55
  12. 22 Mar, 2021 1 commit
  13. 03 Mar, 2021 1 commit
  14. 08 Feb, 2021 1 commit
  15. 13 Dec, 2020 1 commit
  16. 11 Dec, 2020 1 commit
    • Yossi Gottlieb's avatar
      TLS: Add different client cert support. (#8076) · 8c291b97
      Yossi Gottlieb authored
      This adds a new `tls-client-cert-file` and `tls-client-key-file`
      configuration directives which make it possible to use different
      certificates for the TLS-server and TLS-client functions of Redis.
      
      This is an optional directive. If it is not specified the `tls-cert-file`
      and `tls-key-file` directives are used for TLS client functions as well.
      
      Also, `utils/gen-test-certs.sh` now creates additional server-only and client-only certs and will skip intensive operations if target files already exist.
      8c291b97
  17. 07 Dec, 2020 1 commit
  18. 11 Oct, 2020 1 commit
  19. 29 Sep, 2020 1 commit
    • Yossi Gottlieb's avatar
      TLS: Do not require CA config if not used. (#7862) · 1591e347
      Yossi Gottlieb authored
      The tls-ca-cert or tls-ca-cert-dir configuration parameters are only
      used when Redis needs to authenticate peer certificates, in one of these
      scenarios:
      
      1. Incoming clients or replicas, with `tls-auth-clients` enabled.
      2. A replica authenticating the master's peer certificate.
      3. Cluster nodes authenticating other nodes when establishing the bus
         protocol connection.
      1591e347
  20. 23 Sep, 2020 1 commit
  21. 22 Sep, 2020 1 commit
  22. 17 Aug, 2020 1 commit
  23. 28 Jul, 2020 2 commits
  24. 10 Jul, 2020 2 commits
  25. 27 May, 2020 1 commit
  26. 14 May, 2020 1 commit
  27. 10 May, 2020 1 commit
  28. 22 Apr, 2020 1 commit