1. 15 Feb, 2018 5 commits
  2. 18 Jan, 2018 2 commits
    • Dvir Volk's avatar
      9f7e214e
    • Oran Agra's avatar
      fix processing of large bulks (above 2GB) · aefa9caa
      Oran Agra authored
      - protocol parsing (processMultibulkBuffer) was limitted to 32big positions in the buffer
        readQueryFromClient potential overflow
      - rioWriteBulkCount used int, although rioWriteBulkString gave it size_t
      - several places in sds.c that used int for string length or index.
      - bugfix in RM_SaveAuxField (return was 1 or -1 and not length)
      - RM_SaveStringBuffer was limitted to 32bit length
      aefa9caa
  3. 24 Nov, 2017 3 commits
  4. 23 Nov, 2017 2 commits
  5. 04 Oct, 2017 4 commits
  6. 23 Jul, 2017 3 commits
    • antirez's avatar
      Modules: don't crash when Lua calls a module blocking command. · 41e3617d
      antirez authored
      Lua scripting does not support calling blocking commands, however all
      the native Redis commands are flagged as "s" (no scripting flag), so
      this is not possible at all. With modules there is no such mechanism in
      order to flag a command as non callable by the Lua scripting engine,
      moreover we cannot trust the modules users from complying all the times:
      it is likely that modules will be released to have blocking commands
      without such commands being flagged correctly, even if we provide a way to
      signal this fact.
      
      This commit attempts to address the problem in a short term way, by
      detecting that a module is trying to block in the context of the Lua
      scripting engine client, and preventing to do this. The module will
      actually believe to block as usually, but what happens is that the Lua
      script receives an error immediately, and the background call is ignored
      by the Redis engine (if not for the cleanup callbacks, once it
      unblocks).
      
      Long term, the more likely solution, is to introduce a new call called
      RedisModule_GetClientFlags(), so that a command can detect if the caller
      is a Lua script, and return an error, or avoid blocking at all.
      
      Being the blocking API experimental right now, more work is needed in
      this regard in order to reach a level well blocking module commands and
      all the other Redis subsystems interact peacefully.
      
      Now the effect is like the following:
      
          127.0.0.1:6379> eval "redis.call('hello.block',1,5000)" 0
          (error) ERR Error running script (call to
          f_b5ba35ff97bc1ef23debc4d6e9fd802da187ed53): @user_script:1: ERR
          Blocking module command called from Lua script
      
      This commit fixes issue #4127 in the short term.
      41e3617d
    • antirez's avatar
      10370b20
    • antirez's avatar
      Fix two bugs in moduleTypeLookupModuleByID(). · 9a4f3d72
      antirez authored
      The function cache was not working at all, and the function returned
      wrong values if there where two or more modules exporting native data
      types.
      
      See issue #4131 for more details.
      9a4f3d72
  7. 14 Jul, 2017 4 commits
  8. 06 Jul, 2017 4 commits
  9. 27 Jun, 2017 1 commit
    • antirez's avatar
      RDB modules values serialization format version 2. · 5af0fc0c
      antirez authored
      The original RDB serialization format was not parsable without the
      module loaded, becuase the structure was managed only by the module
      itself. Moreover RDB is a streaming protocol in the sense that it is
      both produce di an append-only fashion, and is also sometimes directly
      sent to the socket (in the case of diskless replication).
      
      The fact that modules values cannot be parsed without the relevant
      module loaded is a problem in many ways: RDB checking tools must have
      loaded modules even for doing things not involving the value at all,
      like splitting an RDB into N RDBs by key or alike, or just checking the
      RDB for sanity.
      
      In theory module values could be just a blob of data with a prefixed
      length in order for us to be able to skip it. However prefixing the values
      with a length would mean one of the following:
      
      1. To be able to write some data at a previous offset. This breaks
      stremaing.
      2. To bufferize values before outputting them. This breaks performances.
      3. To have some chunked RDB output format. This breaks simplicity.
      
      Moreover, the above solution, still makes module values a totally opaque
      matter, with the fowllowing problems:
      
      1. The RDB check tool can just skip the value without being able to at
      least check the general structure. For datasets composed mostly of
      modules values this means to just check the outer level of the RDB not
      actually doing any checko on most of the data itself.
      2. It is not possible to do any recovering or processing of data for which a
      module no longer exists in the future, or is unknown.
      
      So this commit implements a different solution. The modules RDB
      serialization API is composed if well defined calls to store integers,
      floats, doubles or strings. After this commit, the parts generated by
      the module API have a one-byte prefix for each of the above emitted
      parts, and there is a final EOF byte as well. So even if we don't know
      exactly how to interpret a module value, we can always parse it at an
      high level, check the overall structure, understand the types used to
      store the information, and easily skip the whole value.
      
      The change is backward compatible: older RDB files can be still loaded
      since the new encoding has a new RDB type: MODULE_2 (of value 7).
      The commit also implements the ability to check RDB files for sanity
      taking advantage of the new feature.
      5af0fc0c
  10. 11 May, 2017 7 commits
  11. 18 Apr, 2017 3 commits
    • antirez's avatar
      Make more obvious why there was issue #3843. · a5b66da8
      antirez authored
      a5b66da8
    • antirez's avatar
      Fix modules blocking commands awake delay. · f60d6f09
      antirez authored
      If a thread unblocks a client blocked in a module command, by using the
      RedisMdoule_UnblockClient() API, the event loop may not be awaken until
      the next timeout of the multiplexing API or the next unrelated I/O
      operation on other clients. We actually want the client to be served
      ASAP, so a mechanism is needed in order for the unblocking API to inform
      Redis that there is a client to serve ASAP.
      
      This commit fixes the issue using the old trick of the pipe: when a
      client needs to be unblocked, a byte is written in a pipe. When we run
      the list of clients blocked in modules, we consume all the bytes
      written in the pipe. Writes and reads are performed inside the context
      of the mutex, so no race is possible in which we consume the bytes that
      are actually related to an awake request for a client that should still
      be put into the list of clients to unblock.
      
      It was verified that after the fix the server handles the blocked
      clients with the expected short delay.
      
      Thanks to @dvirsky for understanding there was such a problem and
      reporting it.
      f60d6f09
    • Dvir Volk's avatar
      17250409
  12. 21 Feb, 2017 1 commit
    • antirez's avatar
      Use SipHash hash function to mitigate HashDos attempts. · ba647598
      antirez authored
      This change attempts to switch to an hash function which mitigates
      the effects of the HashDoS attack (denial of service attack trying
      to force data structures to worst case behavior) while at the same time
      providing Redis with an hash function that does not expect the input
      data to be word aligned, a condition no longer true now that sds.c
      strings have a varialbe length header.
      
      Note that it is possible sometimes that even using an hash function
      for which collisions cannot be generated without knowing the seed,
      special implementation details or the exposure of the seed in an
      indirect way (for example the ability to add elements to a Set and
      check the return in which Redis returns them with SMEMBERS) may
      make the attacker's life simpler in the process of trying to guess
      the correct seed, however the next step would be to switch to a
      log(N) data structure when too many items in a single bucket are
      detected: this seems like an overkill in the case of Redis.
      
      SPEED REGRESION TESTS:
      
      In order to verify that switching from MurmurHash to SipHash had
      no impact on speed, a set of benchmarks involving fast insertion
      of 5 million of keys were performed.
      
      The result shows Redis with SipHash in high pipelining conditions
      to be about 4% slower compared to using the previous hash function.
      However this could partially be related to the fact that the current
      implementation does not attempt to hash whole words at a time but
      reads single bytes, in order to have an output which is endian-netural
      and at the same time working on systems where unaligned memory accesses
      are a problem.
      
      Further X86 specific optimizations should be tested, the function
      may easily get at the same level of MurMurHash2 if a few optimizations
      are performed.
      ba647598
  13. 12 Jan, 2017 1 commit