1. 27 Oct, 2020 40 commits
    • Oran Agra's avatar
      Allow requirepass config to clear the password (#7899) · b20af834
      Oran Agra authored
      This is a compatibility issue with redis 5.0 that was introduced by ACL.
      Before this commit, setting requirepass to an empty string will result
      in a server that needs an empty AUTH, unlike redis 5.0 which would
      accept connections without an AUTH.
      
      (cherry picked from commit 19418b6b)
      b20af834
    • Wang Yuan's avatar
      Remove temporary aof and rdb files in a background thread (#7905) · 14532772
      Wang Yuan authored
      If we fail or stop to rewrite aof, we need to remove temporary aof.
      We also remove temporary rdb when replicas abort to receive rdb.
      But currently we delete them in main thread, to avoid blocking,
      we should use bg_unlink to remove them in a background thread.
      
      Btw, we have already used this way to removed child process temporary rdb.
      
      (cherry picked from commit aaacb8c9)
      14532772
    • guybe7's avatar
      Minor improvements to module blocked on keys (#7903) · e4e24460
      guybe7 authored
      - Clarify some documentation comments
      - Make sure blocked-on-keys client privdata is accessible
        from withing the timeout callback
      - Handle blocked clients in beforeSleep - In case a key
        becomes "ready" outside of processCommand
      
      See #7879 #7880
      
      (cherry picked from commit addf47dc)
      e4e24460
    • Andreas Lind's avatar
      Support redis-cli -u rediss://... (#7900) · db9cffe7
      Andreas Lind authored
      
      (cherry picked from commit 8b497881)
      db9cffe7
    • Yossi Gottlieb's avatar
      Modules: fix RM_GetCommandKeys API. (#7901) · 9c475272
      Yossi Gottlieb authored
      This cleans up and simplifies the API by passing the command name as the
      first argument. Previously the command name was specified explicitly,
      but was still included in the argv.
      
      (cherry picked from commit 056a43e1)
      9c475272
    • Meir Shpilraien (Spielrein)'s avatar
      Add Module API for version and compatibility checks (#7865) · 6f2c894d
      Meir Shpilraien (Spielrein) authored
      
      
      * Introduce a new API's: RM_GetContextFlagsAll, and
      RM_GetKeyspaceNotificationFlagsAll that will return the
      full flags mask of each feature. The module writer can
      check base on this value if the Flags he needs are
      supported or not.
      
      * For each flag, introduce a new value on redismodule.h,
      this value represents the LAST value and should be there
      as a reminder to update it when a new value is added,
      also it will be used in the code to calculate the full
      flags mask (assuming flags are incrementally increasing).
      In addition, stated that the module writer should not use
      the LAST flag directly and he should use the GetFlagAll API's.
      
      * Introduce a new API: RM_IsSubEventSupported, that returns for a given
      event and subevent, whether or not the subevent supported.
      
      * Introduce a new macro RMAPI_FUNC_SUPPORTED(func) that returns whether
      or not a function API is supported by comparing it to NULL.
      
      * Introduce a new API: int RM_GetServerVersion();, that will return the
      current Redis version in the format 0x00MMmmpp; e.g. 0x00060008;
      
      * Changed unstable version from 999.999.999 to 255.255.255
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      (cherry picked from commit adc3183c)
      6f2c894d
    • Yossi Gottlieb's avatar
      Module API: Add RM_GetClientCertificate(). (#7866) · 4147a220
      Yossi Gottlieb authored
      This API function makes it possible to retrieve the X.509 certificate
      used by clients to authenticate TLS connections.
      
      (cherry picked from commit 0aec98dc)
      4147a220
    • Yossi Gottlieb's avatar
      Modules: Add RM_GetDetachedThreadSafeContext(). (#7886) · 28d1fe67
      Yossi Gottlieb authored
      The main motivation here is to provide a way for modules to create a
      single, global context that can be used for logging.
      
      Currently, it is possible to obtain a thread-safe context that is not
      attached to any blocked client by using `RM_GetThreadSafeContext`.
      However, the attached context is not linked to the module identity so
      log messages produced are not tagged with the module name.
      
      Ideally we'd fix this in `RM_GetThreadSafeContext` itself but as it
      doesn't accept the current context as an argument there's no way to do
      that in a backwards compatible manner.
      
      (cherry picked from commit 907da058)
      28d1fe67
    • Yossi Gottlieb's avatar
      Modules: add RM_GetCommandKeys(). · 116a2042
      Yossi Gottlieb authored
      This is essentially the same as calling COMMAND GETKEYS but provides a
      more efficient interface that can be used in every context (i.e. not a
      Redis command).
      
      (cherry picked from commit 7d117d75)
      116a2042
    • Yossi Gottlieb's avatar
      Introduce getKeysResult for getKeysFromCommand. · 945983ac
      Yossi Gottlieb authored
      Avoid using a static buffer for short key index responses, and make it
      caller's responsibility to stack-allocate a result type. Responses that
      don't fit are still allocated on the heap.
      
      (cherry picked from commit 9b7f8ba8)
      945983ac
    • Madelyn Olson's avatar
      Fixed excessive categories being displayed from acls (#7889) · e7c8002b
      Madelyn Olson authored
      
      (cherry picked from commit 2127f7c8)
      e7c8002b
    • Oran Agra's avatar
      Add some additional signal info to the crash log (#7891) · 86b5e1a7
      Oran Agra authored
      - si_code can be very useful info some day.
      - a clear indication that redis was killed by an external user
      
      (cherry picked from commit f659d236)
      86b5e1a7
    • Oran Agra's avatar
      Allow blocked XREAD on a cluster replica (#7881) · bab83457
      Oran Agra authored
      I suppose that it was overlooked, since till recently none of the blocked commands were readonly.
      
      other changes:
      - add test for the above.
      - add better support for additional (and deferring) clients for
        cluster tests
      - improve a test which left the client in MULTI state.
      
      (cherry picked from commit 216c1106)
      bab83457
    • Oran Agra's avatar
      memory reporting of clients argv (#7874) · 76f3a63d
      Oran Agra authored
      track and report memory used by clients argv.
      this is very usaful in case clients started sending a command and didn't
      complete it. in which case the first args of the command are already
      trimmed from the query buffer.
      
      in an effort to avoid cache misses and overheads while keeping track of
      these, i avoid calling sdsZmallocSize and instead use the sdslen /
      bulk-len which can at least give some insight into the problem.
      
      This memory is now added to the total clients memory usage, as well as
      the client list.
      
      (cherry picked from commit bea40e6a)
      76f3a63d
    • DvirDukhan's avatar
      redis-cli add control on raw format line delimiter (#7841) · 93b4c6b3
      DvirDukhan authored
      
      
      Adding -D option for redis-cli to control newline between command
      responses in raw mode.
      
      Also removing cleanup code before calling exit, just in order
      to avoid adding more adding more cleanup code (redis doesn't
      bother to release allocations before exit anyway)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 6418d767)
      93b4c6b3
    • Oran Agra's avatar
      Include internal sds fragmentation in MEMORY reporting (#7864) · 12055ed8
      Oran Agra authored
      The MEMORY command is used for debugging memory usage, so it should include internal
      fragmentation, same as used_memory
      
      (cherry picked from commit eb6241a3)
      12055ed8
    • Oran Agra's avatar
      Fix crash in script timeout during AOF loading (#7870) · c80f6219
      Oran Agra authored
      
      (cherry picked from commit dc803d25)
      c80f6219
    • Rafi Einstein's avatar
      Makefile: enable program suffixes via PROG_SUFFIX (#7868) · 801c7307
      Rafi Einstein authored
      (cherry picked from commit b8187d39)
      801c7307
    • nitaicaro's avatar
      Fixed Tracking test “The other connection is able to get invalidations” (#7871) · a7b95b7c
      nitaicaro authored
      
      
      PROBLEM:
      
      [$rd1 read] reads invalidation messages one by one, so it's never going to see the second invalidation message produced after INCR b, whether or not it exists. Adding another read will block incase no invalidation message is produced.
      
      FIX:
      
      We switch the order of "INCR a" and "INCR b" - now "INCR b" comes first. We still only read the first invalidation message produces. If an invalidation message is wrongly produces for b - then it will be produced before that of a, since "INCR b" comes before "INCR a".
      Co-authored-by: default avatarNitai Caro <caronita@amazon.com>
      (cherry picked from commit 8fb89a57)
      a7b95b7c
    • Yossi Gottlieb's avatar
      Modules: expose real client on conn events. · 919580a2
      Yossi Gottlieb authored
      When REDISMODULE_EVENT_CLIENT_CHANGE events are delivered, modules may
      want to mutate the client state (e.g. perform authentication).
      
      This change links the module context with the real client rather than a
      fake client for these events.
      
      (cherry picked from commit 67b43f75)
      919580a2
    • Yossi Gottlieb's avatar
      Module API: Fail ineffective auth calls. · e83d6b0c
      Yossi Gottlieb authored
      The client pointed to by the module context may in some cases be a fake
      client. RM_Authenticate*() calls in this case would be ineffective but
      appear to succeed, and this change fails them to make it easier to catch
      such cases.
      
      (cherry picked from commit cfccfbd6)
      e83d6b0c
    • Yossi Gottlieb's avatar
      TLS: Do not require CA config if not used. (#7862) · cb070acc
      Yossi Gottlieb authored
      The tls-ca-cert or tls-ca-cert-dir configuration parameters are only
      used when Redis needs to authenticate peer certificates, in one of these
      scenarios:
      
      1. Incoming clients or replicas, with `tls-auth-clients` enabled.
      2. A replica authenticating the master's peer certificate.
      3. Cluster nodes authenticating other nodes when establishing the bus
         protocol connection.
      
      (cherry picked from commit 1591e347)
      cb070acc
    • Oran Agra's avatar
      warning: comparison between signed and unsigned integer in 32bit build (#7838) · 9ce6fc73
      Oran Agra authored
      
      (cherry picked from commit ef33252c)
      9ce6fc73
    • David CARLIER's avatar
      Add support for Haiku OS (#7435) · a7b9cfbe
      David CARLIER authored
      
      (cherry picked from commit f971a5d8)
      a7b9cfbe
    • Gavrie Philipson's avatar
      Fix typo in module API docs (#7861) · 219bab2d
      Gavrie Philipson authored
      
      (cherry picked from commit 49383a4a)
      219bab2d
    • David CARLIER's avatar
      getting rss size implementation for netbsd (#7293) · d51bebd4
      David CARLIER authored
      
      (cherry picked from commit ce8bfc56)
      d51bebd4
    • Oran Agra's avatar
      Fix new obuf-limits tests to work with TLS (#7848) · 120c6b43
      Oran Agra authored
      Also stabilize new shutdown tests on slow machines (valgrind)
      
      (cherry picked from commit 8aa083bd)
      120c6b43
    • caozb's avatar
      ignore slaveof no one in redis.conf (#7842) · 63075b81
      caozb authored
      
      
      when slaveof config is "no one", reset any pre-existing config and resume.
      
      also solve a memory leak if slaveof appears twice.
      and fail loading if port number is out of range or not an integer.
      Co-authored-by: default avatarcaozhengbin <caozb@yidingyun.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit a295770e)
      63075b81
    • Wang Yuan's avatar
      Don't support Gopher if enable io threads to read queries (#7851) · 94180837
      Wang Yuan authored
      There's currently an issue with IO threads and gopher (issuing lookupKey from within the thread).
      simply fix is to just not support it for now.
      
      (cherry picked from commit c9f00bcc)
      94180837
    • Wang Yuan's avatar
      Set 'loading' and 'shutdown_asap' to volatile sig_atomic_t type (#7845) · 5d1ac2df
      Wang Yuan authored
      We may access and modify these two variables in signal handler function,
      to guarantee them async-signal-safe, so we should set them to volatile
      sig_atomic_t type.
      
      It doesn't look like this could have caused any real issue, and it seems that
      signals are handled in main thread on most platforms. But we want to follow C
      and POSIX standard in signal handler function.
      
      (cherry picked from commit f1863a1f)
      5d1ac2df
    • Uri Shachar's avatar
      Fix config rewrite file handling to make it really atomic (#7824) · 90555566
      Uri Shachar authored
      Make sure we handle short writes correctly, sync to disk after writing  and use
      rename to make sure the replacement is actually atomic.
      In any case of failure old configuration will remain in place.
      
      Also, add some additional logging to make it easier to diagnose rewrite problems.
      
      (cherry picked from commit c30bd02c)
      90555566
    • WuYunlong's avatar
      Add fsync to readSyncBulkPayload(). (#7839) · bb7e00b6
      WuYunlong authored
      We should sync temp DB file before renaming as rdb_fsync_range does not use
      flag `SYNC_FILE_RANGE_WAIT_AFTER`.
      
      Refer to `Linux Programmer's Manual`:
      SYNC_FILE_RANGE_WAIT_AFTER
          Wait upon write-out of all pages in the range after performing any write.
      
      (cherry picked from commit 0d62caab)
      bb7e00b6
    • Wen Hui's avatar
      rdb.c: handle fclose error case differently to avoid double fclose (#7307) · faa431c9
      Wen Hui authored
      When fclose would fail, the previous implementation would have attempted to do fclose again
      this can in theory lead to segfault.
      
      other changes:
      check for non-zero return value as failure rather than a specific error code.
      this doesn't fix a real bug, just a minor cleanup.
      
      (cherry picked from commit 323029ba)
      faa431c9
    • Wang Yuan's avatar
      Don't write replies if close the client ASAP (#7202) · 757ad7cd
      Wang Yuan authored
      
      
      Before this commit, we would have continued to add replies to the reply buffer even if client
      output buffer limit is reached, so the used memory would keep increasing over the configured limit.
      What's more, we shouldn’t write any reply to the client if it is set 'CLIENT_CLOSE_ASAP' flag
      because that doesn't conform to its definition and we will close all clients flagged with
      'CLIENT_CLOSE_ASAP' in ‘beforeSleep’.
      
      Because of code execution order, before this, we may firstly write to part of the replies to
      the socket before disconnecting it, but in fact, we may can’t send the full replies to clients
      since OS socket buffer is limited. But this unexpected behavior makes some commands work well,
      for instance ACL DELUSER, if the client deletes the current user, we need to send reply to client
      and close the connection, but before, we close the client firstly and write the reply to reply
      buffer. secondly, we shouldn't do this despite the fact it works well in most cases.
      
      We add a flag 'CLIENT_CLOSE_AFTER_COMMAND' to mark clients, this flag means we will close the
      client after executing commands and send all entire replies, so that we can write replies to
      reply buffer during executing commands, send replies to clients, and close them later.
      
      We also fix some implicit problems. If client output buffer limit is enforced in 'multi/exec',
      all commands will be executed completely in redis and clients will not read any reply instead of
      partial replies. Even more, if the client executes 'ACL deluser' the using user in 'multi/exec',
      it will not read the replies after 'ACL deluser' just like before executing 'client kill' itself
      in 'multi/exec'.
      
      We added some tests for output buffer limit breach during multi-exec and using a pipeline of
      many small commands rather than one with big response.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 57709c4b)
      757ad7cd
    • Guy Korland's avatar
      Fix RedisModule_HashGet examples (#6697) · 31491202
      Guy Korland authored
      
      (cherry picked from commit b464afb9)
      31491202
    • Oran Agra's avatar
      fix recently broken TLS build error, and add coverage for CI (#7833) · 26c1c601
      Oran Agra authored
      
      (cherry picked from commit a735bf5c)
      26c1c601
    • David CARLIER's avatar
      Further NetBSD update and build fixes. (#7831) · 2ccbba65
      David CARLIER authored
      mainly backtrace and register dump support.
      
      (cherry picked from commit c3edaa79)
      2ccbba65
    • WuYunlong's avatar
      Fix redundancy use of semicolon in do-while macros in ziplist.c. (#7832) · c73406c2
      WuYunlong authored
      this is very dangerous bug, but it looks like it didn't cause any harm.
      
      (cherry picked from commit 63cd4d4e)
      c73406c2
    • yixiang's avatar
      Fix connGetSocketError usage (#7811) · c1423c01
      yixiang authored
      
      (cherry picked from commit b96c3595)
      c1423c01
    • Oran Agra's avatar
      RM_GetContextFlags - document missing flags (#7821) · ecf51b0b
      Oran Agra authored
      
      (cherry picked from commit 26ca04a8)
      ecf51b0b