1. 08 Nov, 2024 1 commit
    • Yuan Wang's avatar
      Dynamic event loop binding for connection structure (#13642) · 779af3ab
      Yuan Wang authored
      
      
      The IO thread has an independent event loop, so we can no longer
      hard-code the event loop to the connection, instead, we should
      dynamically select the event loop for the connection.
      
      - configure the event loop during connection creation.
      - add a new interface to allow dynamic event loop binding.
      
      For TLS connection, we need to check for any pending data on the
      connection and handle it accordingly when changing connection cross IO
      thread and main thread. This commit doesn't handle it, @sundb will
      overall support for TLS connection later.
      
      ---------
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      779af3ab
  2. 20 Mar, 2024 1 commit
  3. 06 Feb, 2024 1 commit
    • Gann's avatar
      Improve error handling in connSocketBlockingConnect for various connction failures (#13008) · 0777dc78
      Gann authored
      This commit addresses a problem in connSocketBlockingConnect where
      different types of connection failures, including timeouts and other
      errors, were not consistently handled. Previously, the function did not
      return C_ERR immediately after detecting a connection failure, which
      could lead to inconsistent states and misinterpretation of the
      connection status.
      
      With this update, connSocketBlockingConnect now correctly returns C_ERR
      upon encountering any connection error, ensuring that all types of
      connection failures are handled consistently and the behavior of the
      function aligns with expected outcomes in case of connection issues.
      
      Closes #12900
      0777dc78
  4. 02 Jan, 2024 1 commit
    • AshMosh's avatar
      Manage number of new connections per cycle (#12178) · c3f8b542
      AshMosh authored
      
      
      There are situations (especially in TLS) in which the engine gets too occupied managing a large number of new connections. Existing connections may time-out while the server is processing the new connections initial TLS handshakes, which may cause cause new connections to be established, perpetuating the problem. To better manage the tradeoff between new connection rate and other workloads, this change adds a new config to manage maximum number of new connections per event loop cycle, instead of using a predetermined number (currently 1000).
      
      This change introduces two new configurations, max-new-connections-per-cycle and max-new-tls-connections-per-cycle. The default value of the tcp connections is 10 per cycle and the default value of tls connections per cycle is 1.
      ---------
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      c3f8b542
  5. 28 May, 2023 1 commit
  6. 03 May, 2023 1 commit
    • Madelyn Olson's avatar
      Remove prototypes with empty declarations (#12020) · 5e3be1be
      Madelyn Olson authored
      Technically declaring a prototype with an empty declaration has been deprecated since the early days of C, but we never got a warning for it. C2x will apparently be introducing a breaking change if you are using this type of declarator, so Clang 15 has started issuing a warning with -pedantic. Although not apparently a problem for any of the compiler we build on, if feels like the right thing is to properly adhere to the C standard and use (void).
      5e3be1be
  7. 04 Apr, 2023 1 commit
  8. 04 Jan, 2023 1 commit
  9. 04 Nov, 2022 1 commit
    • Binbin's avatar
      Introduce socket shutdown into connection type, used if a fork is active (#11376) · fac188b4
      Binbin authored
      Introduce socket `shutdown()` into connection type, and use it
      on normal socket if a fork is active. This allows us to close
      client connections when there are child processes sharing the
      file descriptors.
      
      Fixes #10077. The reason is that since the `fork()` child is holding
      the file descriptors, the `close` in `unlinkClient -> connClose`
      isn't sufficient. The client will not realize that the connection is
      disconnected until the child process ends.
      
      Let's try to be conservative and only use shutdown when the fork is active.
      fac188b4
  10. 22 Aug, 2022 11 commits
    • zhenwei pi's avatar
      Introduce .listen into connection type · 0b27cfe3
      zhenwei pi authored
      
      
      Introduce listen method into connection type, this allows no hard code
      of listen logic. Originally, we initialize server during startup like
      this:
          if (server.port)
              listenToPort(server.port,&server.ipfd);
          if (server.tls_port)
              listenToPort(server.port,&server.tlsfd);
          if (server.unixsocket)
              anetUnixServer(...server.unixsocket...);
      
          ...
          if (createSocketAcceptHandler(&server.ipfd, acceptTcpHandler) != C_OK)
          if (createSocketAcceptHandler(&server.tlsfd, acceptTcpHandler) != C_OK)
          if (createSocketAcceptHandler(&server.sofd, acceptTcpHandler) != C_OK)
          ...
      
      If a new connection type gets supported, we have to add more hard code
      to setup listener.
      
      Introduce .listen and refactor listener, and Unix socket supports this.
      this allows to setup listener arguments and create listener in a loop.
      
      What's more, '.listen' is defined in connection.h, so we should include
      server.h to import 'struct socketFds', but server.h has already include
      'connection.h'. To avoid including loop(also to make code reasonable),
      define 'struct connListener' in connection.h instead of 'struct socketFds'
      in server.h. This leads this commit to get more changes.
      
      There are more fields in 'struct connListener', hence it's possible to
      simplify changeBindAddr & applyTLSPort() & updatePort() into a single
      logic: update the listener config from the server.xxx, and re-create
      the listener.
      
      Because of the new field 'priv' in struct connListener, we expect to pass
      this to the accept handler(even it's not used currently), this may be used
      in the future.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      0b27cfe3
    • zhenwei pi's avatar
      Use connection name of string · 45617385
      zhenwei pi authored
      
      
      Suggested by Oran, use an array to store all the connection types
      instead of a linked list, and use connection name of string. The index
      of a connection is dynamically allocated.
      
      Currently we support max 8 connection types, include:
      - tcp
      - unix socket
      - tls
      
      and RDMA is in the plan, then we have another 4 types to support, it
      should be enough in a long time.
      
      Introduce 3 functions to get connection type by a fast path:
      - connectionTypeTcp()
      - connectionTypeTls()
      - connectionTypeUnix()
      
      Note that connectionByType() is designed to use only in unlikely code path.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      45617385
    • zhenwei pi's avatar
      Abstract accept handler · 0ae02ce9
      zhenwei pi authored
      
      
      Abstract accept handler for socket&TLS, and add helper function
      'connAcceptHandler' to get accept handler by specified type.
      
      Also move acceptTcpHandler into socket.c, and move
      acceptTLSHandler into tls.c.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      0ae02ce9
    • zhenwei pi's avatar
      Fully abstract connection type · 1234e3a5
      zhenwei pi authored
      
      
      Abstract common interface of connection type, so Redis can hide the
      implementation and uplayer only calls connection API without macro.
      
                     uplayer
                        |
                 connection layer
                   /          \
                socket        TLS
      
      Currently, for both socket and TLS, all the methods of connection type
      are declared as static functions.
      
      It's possible to build TLS(even socket) as a shared library, and Redis
      loads it dynamically in the next step.
      
      Also add helper function connTypeOfCluster() and
      connTypeOfReplication() to simplify the code:
      link->conn = server.tls_cluster ? connCreateTLS() : connCreateSocket();
      -> link->conn = connCreate(connTypeOfCluster());
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      1234e3a5
    • zhenwei pi's avatar
      Introduce pending data for connection type · 709b55b0
      zhenwei pi authored
      
      
      Introduce .has_pending_data and .process_pending_data for connection
      type, and hide tlsHasPendingData() and tlsProcessPendingData(). Also
      set .has_pending_data and .process_pending_data as NULL explicitly in
      socket.c.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      709b55b0
    • zhenwei pi's avatar
      Introduce connection layer framework · 8234a512
      zhenwei pi authored
      
      
      Use connTypeRegister() to register a connection type into redis, and
      query connection by connectionByType() via type.
      
      With this change, we can hide TLS specified methods into connection
      type:
      - void tlsInit(void);
      - void tlsCleanup(void);
      - int tlsConfigure(redisTLSContextConfig *ctx_config);
      - int isTlsConfigured(void);
      
      Merge isTlsConfigured & tlsConfigure, use an argument *reconfigure*
      to distinguish:
         tlsConfigure(&server.tls_ctx_config)
      -> onnTypeConfigure(CONN_TYPE_TLS, &server.tls_ctx_config, 1)
      
         isTlsConfigured() && tlsConfigure(&server.tls_ctx_config)
      -> connTypeConfigure(CONN_TYPE_TLS, &server.tls_ctx_config, 0)
      
      Finally, we can remove USE_OPENSSL from config.c. If redis is built
      without TLS, and still run redis with TLS, then redis reports:
       # Missing implement of connection type 1
       # Failed to configure TLS. Check logs for more info.
      
      The log can be optimised, let's leave it in the future. Maybe we can
      use connection type as a string.
      
      Although uninitialized fields of a static struct are zero, we still
      set them as NULL explicitly in socket.c, let them clear to read & maintain:
          .init = NULL,
          .cleanup = NULL,
          .configure = NULL,
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      8234a512
    • zhenwei pi's avatar
      Introduce connAddr · bff7ecc7
      zhenwei pi authored
      
      
      Originally, connPeerToString is designed to get the address info from
      socket only(for both TCP & TLS), and the API 'connPeerToString' is
      oriented to operate a FD like:
      int connPeerToString(connection *conn, char *ip, size_t ip_len, int *port) {
          return anetFdToString(conn ? conn->fd : -1, ip, ip_len, port, FD_TO_PEER_NAME);
      }
      
      Introduce connAddr and implement .addr method for socket and TLS,
      thus the API 'connAddr' and 'connFormatAddr' become oriented to a
      connection like:
      static inline int connAddr(connection *conn, char *ip, size_t ip_len, int *port, int remote) {
          if (conn && conn->type->addr) {
              return conn->type->addr(conn, ip, ip_len, port, remote);
          }
      
          return -1;
      }
      
      Also remove 'FD_TO_PEER_NAME' & 'FD_TO_SOCK_NAME', use a boolean type
      'remote' to get local/remote address of a connection.
      
      With these changes, it's possible to support the other connection
      types which does not use socket(Ex, RDMA).
      
      Thanks to Oran for suggestions!
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      bff7ecc7
    • zhenwei pi's avatar
      Reorder methods for ConnectionType · b9d77288
      zhenwei pi authored
      
      
      Reorder methods for CT_Socket & CT_TLS, also add comments to make the
      methods clear.
      
      Also move the CT_TLS to the end of file, other methods can be static
      in the next step.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      b9d77288
    • zhenwei pi's avatar
      Move 'connGetSocketError' to 'anetGetError' · 8045e26e
      zhenwei pi authored
      
      
      getsockopt is part of TCP, rename 'connGetSocketError' to
      'anetGetError', and move it into anet.c.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      8045e26e
    • zhenwei pi's avatar
      Move several conn functions to connection.h · dca5c6ff
      zhenwei pi authored
      
      
      These functions are really short enough and they are the connection
      functions, separate them from the socket source.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      dca5c6ff
    • zhenwei pi's avatar
      Rename connection.c to socket.c · 22e74e47
      zhenwei pi authored
      
      
      ConnectionType CT_Socket is implemented in connection.c, so rename
      this file to socket.c.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      22e74e47
  11. 20 Jul, 2022 1 commit
  12. 22 Feb, 2022 1 commit
    • Andy Pan's avatar
      Reduce system calls of write for client->reply by introducing writev (#9934) · 496375fc
      Andy Pan authored
      There are scenarios where it results in many small objects in the reply list,
      such as commands heavily using deferred array replies (`addReplyDeferredLen`).
      E.g. what COMMAND command and CLUSTER SLOTS used to do (see #10056, #7123),
      but also in case of a transaction or a pipeline of commands that use just one differed array reply.
      
      We used to have to run multiple loops along with multiple calls to `write()` to send data back to
      peer based on the current code, but by means of `writev()`, we can gather those scattered
      objects in reply list and include the static reply buffer as well, then send it by one system call,
      that ought to achieve higher performance.
      
      In the case of TLS,  we simply check and concatenate buffers into one big buffer and send it
      away by one call to `connTLSWrite()`, if the amount of all buffers exceeds `NET_MAX_WRITES_PER_EVENT`,
      then invoke `connTLSWrite()` multiple times to avoid a huge massive of memory copies.
      
      Note that aside of reducing system calls, this change will also reduce the amount of
      small TCP packets sent.
      496375fc
  13. 08 Nov, 2021 1 commit
    • Yossi Gottlieb's avatar
      Fix EINTR test failures. (#9751) · a1aba4bf
      Yossi Gottlieb authored
      * Clean up EINTR handling so EINTR will not change connection state to begin with.
      * On TLS, catch EINTR and return it as-is before going through OpenSSL error handling (which seems to not distinguish it from EAGAIN).
      a1aba4bf
  14. 01 Mar, 2021 1 commit
  15. 25 Nov, 2020 1 commit
  16. 28 Oct, 2020 1 commit
    • yoav-steinberg's avatar
      Add local address to CLIENT LIST, and a CLIENT KILL filter. (#7913) · 84b3c18f
      yoav-steinberg authored
      Useful when you want to know through which bind address the client connected to
      the server in case of multiple bind addresses.
      
      - Adding `laddr` field to CLIENT list showing the local (bind) address.
      - Adding `LADDR` option to CLIENT KILL to kill all the clients connected
        to a specific local address.
      - Refactoring to share code.
      84b3c18f
  17. 22 Sep, 2020 2 commits
    • yixiang's avatar
      Fix connGetSocketError usage (#7811) · b96c3595
      yixiang authored
      b96c3595
    • Yossi Gottlieb's avatar
      Fix occasional hangs on replication reconnection. (#7830) · 1980f639
      Yossi Gottlieb authored
      This happens only on diskless replicas when attempting to reconnect after 
      failing to load an RDB file. It is more likely to occur with larger datasets.
      
      After reconnection is initiated, replicationEmptyDbCallback() may get called 
      and try to write to an unconnected socket. This triggered another issue where
      the connection is put into an error state and the connect handler never gets
      called. The problem is a regression introduced by commit c17e597d.
      1980f639
  18. 17 Aug, 2020 1 commit
  19. 28 Jul, 2020 1 commit
  20. 22 Mar, 2020 1 commit
    • Yossi Gottlieb's avatar
      Conns: Fix connClose() / connAccept() behavior. · fa9aa908
      Yossi Gottlieb authored
      We assume accept handlers may choose to reject a connection and close
      it, but connAccept() callers can't distinguish between this state and
      other error states requiring connClose().
      
      This makes it safe (and mandatory!) to always call connClose() if
      connAccept() fails, and safe for accept handlers to close connections
      (which will defer).
      fa9aa908
  21. 15 Oct, 2019 2 commits
  22. 07 Oct, 2019 3 commits
    • Oran Agra's avatar
      TLS: Implement support for write barrier. · 6b629480
      Oran Agra authored
      6b629480
    • Oran Agra's avatar
      diskless replication rdb transfer uses pipe, and writes to sockets form the parent process. · 5a477946
      Oran Agra authored
      misc:
      - handle SSL_has_pending by iterating though these in beforeSleep, and setting timeout of 0 to aeProcessEvents
      - fix issue with epoll signaling EPOLLHUP and EPOLLERR only to the write handlers. (needed to detect the rdb pipe was closed)
      - add key-load-delay config for testing
      - trim connShutdown which is no longer needed
      - rioFdsetWrite -> rioFdWrite - simplified since there's no longer need to write to multiple FDs
      - don't detect rdb child exited (don't call wait3) until we detect the pipe is closed
      - Cleanup bad optimization from rio.c, add another one
      5a477946
    • Yossi Gottlieb's avatar
      TLS: Connections refactoring and TLS support. · b087dd1d
      Yossi Gottlieb authored
      * Introduce a connection abstraction layer for all socket operations and
      integrate it across the code base.
      * Provide an optional TLS connections implementation based on OpenSSL.
      * Pull a newer version of hiredis with TLS support.
      * Tests, redis-cli updates for TLS support.
      b087dd1d