1. 22 Dec, 2020 1 commit
    • Oran Agra's avatar
      Remove read-only flag from non-keyspace cmds, different approach for EXEC to... · 411c18bb
      Oran Agra authored
      Remove read-only flag from non-keyspace cmds, different approach for EXEC to propagate MULTI (#8216)
      
      In the distant history there was only the read flag for commands, and whatever
      command that didn't have the read flag was a write one.
      Then we added the write flag, but some portions of the code still used !read
      Also some commands that don't work on the keyspace at all, still have the read
      flag.
      
      Changes in this commit:
      1. remove the read-only flag from TIME, ECHO, ROLE and LASTSAVE
      
      2. EXEC command used to decides if it should propagate a MULTI by looking at
         the command flags (!read & !admin).
         When i was about to change it to look at the write flag instead, i realized
         that this would cause it not to propagate a MULTI for PUBLISH, EVAL, and
         SCRIPT, all 3 are not marked as either a read command or a write one (as
         they should), but all 3 are calling forceCommandPropagation.
      
         So instead of introducing a new flag to denote a command that "writes" but
         not into the keyspace, and still needs propagation, i decided to rely on
         the forceCommandPropagation, and just fix the code to propagate MULTI when
         needed rather than depending on the command flags at all.
      
         The implication of my change then is that now it won't decide to propagate
         MULTI when it sees one of these: SELECT, PING, INFO, COMMAND, TIME and
         other commands which are neither read nor write.
      
      3. Changing getNodeByQuery and clusterRedirectBlockedClientIfNeeded in
         cluster.c to look at !write rather than read flag.
         This should have no implications, since these code paths are only reachable
         for commands which access keys, and these are always marked as either read
         or write.
      
      This commit improve MULTI propagation tests, for modules and a bunch of
      other special cases, all of which used to pass already before that commit.
      the only one that test change that uncovered a change of behavior is the
      one that DELs a non-existing key, it used to propagate an empty
      multi-exec block, and no longer does.
      411c18bb
  2. 13 Dec, 2020 3 commits
    • Yossi Gottlieb's avatar
      Several (mostly Solaris-related) cleanups (#8171) · 86e3395c
      Yossi Gottlieb authored
      * Allow runtest-moduleapi use a different 'make', for systems where GNU Make is 'gmake'.
      * Fix issue with builds on Solaris re-building everything from scratch due to CFLAGS/LDFLAGS not stored.
      * Fix compile failure on Solaris due to atomicvar and a bunch of warnings.
      * Fix garbled log timestamps on Solaris.
      86e3395c
    • Oran Agra's avatar
      Add module event for repl-diskless-load swapdb (#8153) · ab60dcf5
      Oran Agra authored
      When a replica uses the diskless-load swapdb approach, it backs up the old database,
      then attempts to load a new one, and in case of failure, it restores the backup.
      
      this means that modules with global out of keyspace data, must have an option to
      subscribe to events and backup/restore/discard their global data too.
      ab60dcf5
    • Yossi Gottlieb's avatar
      Modules: add defrag API support. (#8149) · 63c1303c
      Yossi Gottlieb authored
      Add a new set of defrag functions that take a defrag context and allow
      defragmenting memory blocks and RedisModuleStrings.
      
      Modules can register a defrag callback which will be invoked when the
      defrag process handles globals.
      
      Modules with custom data types can also register a datatype-specific
      defrag callback which is invoked for keys that require defragmentation.
      The callback and associated functions support both one-step and
      multi-step options, depending on the complexity of the key as exposed by
      the free_effort callback.
      63c1303c
  3. 09 Dec, 2020 1 commit
    • Yossi Gottlieb's avatar
      Add module data-type support for COPY. (#8112) · 4e064fba
      Yossi Gottlieb authored
      This adds a copy callback for module data types, in order to make
      modules compatible with the new COPY command.
      
      The callback is optional and COPY will fail for keys with data types
      that do not implement it.
      4e064fba
  4. 08 Dec, 2020 1 commit
    • Oran Agra's avatar
      Handle output buffer limits for Module blocked clients (#8141) · 48efc25f
      Oran Agra authored
      Module blocked clients cache the response in a temporary client,
      the reply list in this client would be affected by the recent fix
      in #7202, but when the reply is later copied into the real client,
      it would have bypassed all the checks for output buffer limit, which
      would have resulted in both: responding with a partial response to
      the client, and also not disconnecting it at all.
      48efc25f
  5. 06 Dec, 2020 1 commit
    • guybe7's avatar
      Make sure we do not propagate nested MULTI/EXEC (#8097) · 1df5bb56
      guybe7 authored
      One way this was happening is when a module issued an RM_Call which would inject MULTI.
      If the module command that does that was itself issued by something else that already did
      added MULTI (e.g. another module, or a Lua script), it would have caused nested MULTI.
      
      In fact the MULTI state in the client or the MULTI_EMITTED flag in the context isn't
      the right indication that we need to propagate MULTI or not, because on a nested calls
      (possibly a module action called by a keyspace event of another module action), these
      flags aren't retained / reflected.
      
      instead there's now a global propagate_in_transaction flag for that.
      
      in addition to that, we now have a global in_eval and in_exec flags, to serve the flags
      of RM_GetContextFlags, since their dependence on the current client is wrong for the same
      reasons mentioned above.
      1df5bb56
  6. 17 Nov, 2020 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Unified MULTI, LUA, and RM_Call with respect to blocking commands (#8025) · d87a0d02
      Meir Shpilraien (Spielrein) authored
      
      
      Blocking command should not be used with MULTI, LUA, and RM_Call. This is because,
      the caller, who executes the command in this context, expects a reply.
      
      Today, LUA and MULTI have a special (and different) treatment to blocking commands:
      
      LUA   - Most commands are marked with no-script flag which are checked when executing
      and command from LUA, commands that are not marked (like XREAD) verify that their
      blocking mode is not used inside LUA (by checking the CLIENT_LUA client flag).
      MULTI - Command that is going to block, first verify that the client is not inside
      multi (by checking the CLIENT_MULTI client flag). If the client is inside multi, they
      return a result which is a match to the empty key with no timeout (for example blpop
      inside MULTI will act as lpop)
      For modules that perform RM_Call with blocking command, the returned results type is
      REDISMODULE_REPLY_UNKNOWN and the caller can not really know what happened.
      
      Disadvantages of the current state are:
      
      No unified approach, LUA, MULTI, and RM_Call, each has a different treatment
      Module can not safely execute blocking command (and get reply or error).
      Though It is true that modules are not like LUA or MULTI and should be smarter not
      to execute blocking commands on RM_Call, sometimes you want to execute a command base
      on client input (for example if you create a module that provides a new scripting
      language like javascript or python).
      While modules (on modules command) can check for REDISMODULE_CTX_FLAGS_LUA or
      REDISMODULE_CTX_FLAGS_MULTI to know not to block the client, there is no way to
      check if the command came from another module using RM_Call. So there is no way
      for a module to know not to block another module RM_Call execution.
      
      This commit adds a way to unify the treatment for blocking clients by introducing
      a new CLIENT_DENY_BLOCKING client flag. On LUA, MULTI, and RM_Call the new flag
      turned on to signify that the client should not be blocked. A blocking command
      verifies that the flag is turned off before blocking. If a blocking command sees
      that the CLIENT_DENY_BLOCKING flag is on, it's not blocking and return results
      which are matches to empty key with no timeout (as MULTI does today).
      
      The new flag is checked on the following commands:
      
      List blocking commands: BLPOP, BRPOP, BRPOPLPUSH, BLMOVE,
      Zset blocking commands: BZPOPMIN, BZPOPMAX
      Stream blocking commands: XREAD, XREADGROUP
      SUBSCRIBE, PSUBSCRIBE, MONITOR
      In addition, the new flag is turned on inside the AOF client, we do not want to
      block the AOF client to prevent deadlocks and commands ordering issues (and there
      is also an existing assert in the code that verifies it).
      
      To keep backward compatibility on LUA, all the no-script flags on existing commands
      were kept untouched. In addition, a LUA special treatment on XREAD and XREADGROUP was kept.
      
      To keep backward compatibility on MULTI (which today allows SUBSCRIBE, and PSUBSCRIBE).
      We added a special treatment on those commands to allow executing them on MULTI.
      
      The only backward compatibility issue that this PR introduces is that now MONITOR
      is not allowed inside MULTI.
      
      Tests were added to verify blocking commands are not blocking the client on LUA, MULTI,
      or RM_Call. Tests were added to verify the module can check for CLIENT_DENY_BLOCKING flag.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarItamar Haber <itamar@redislabs.com>
      d87a0d02
  7. 16 Nov, 2020 1 commit
    • chenyangyang's avatar
      Modules callbacks for lazy free effort, and unlink (#7912) · c1aaad06
      chenyangyang authored
      Add two optional callbacks to the RedisModuleTypeMethods structure, which is `free_effort`
      and `unlink`. the `free_effort` callback indicates the effort required to free a module memory.
      Currently, if the effort exceeds LAZYFREE_THRESHOLD, the module memory may be released
      asynchronously. the `unlink` callback indicates the key has been removed from the DB by redis, and
      may soon be freed by a background thread.
      
      Add `lazyfreed_objects` info field, which represents the number of objects that have been
      lazyfreed since redis was started.
      
      Add `RM_GetTypeMethodVersion` API, which return the current redis-server runtime value of
      `REDISMODULE_TYPE_METHOD_VERSION`. You can use that when calling `RM_CreateDataType` to know
      which fields of RedisModuleTypeMethods are gonna be supported and which will be ignored.
      c1aaad06
  8. 11 Nov, 2020 1 commit
  9. 09 Nov, 2020 1 commit
  10. 11 Oct, 2020 5 commits
    • Yossi Gottlieb's avatar
      Modules: fix RM_GetCommandKeys API. (#7901) · 056a43e1
      Yossi Gottlieb authored
      This cleans up and simplifies the API by passing the command name as the
      first argument. Previously the command name was specified explicitly,
      but was still included in the argv.
      056a43e1
    • Meir Shpilraien (Spielrein)'s avatar
      Add Module API for version and compatibility checks (#7865) · adc3183c
      Meir Shpilraien (Spielrein) authored
      
      
      * Introduce a new API's: RM_GetContextFlagsAll, and
      RM_GetKeyspaceNotificationFlagsAll that will return the
      full flags mask of each feature. The module writer can
      check base on this value if the Flags he needs are
      supported or not.
      
      * For each flag, introduce a new value on redismodule.h,
      this value represents the LAST value and should be there
      as a reminder to update it when a new value is added,
      also it will be used in the code to calculate the full
      flags mask (assuming flags are incrementally increasing).
      In addition, stated that the module writer should not use
      the LAST flag directly and he should use the GetFlagAll API's.
      
      * Introduce a new API: RM_IsSubEventSupported, that returns for a given
      event and subevent, whether or not the subevent supported.
      
      * Introduce a new macro RMAPI_FUNC_SUPPORTED(func) that returns whether
      or not a function API is supported by comparing it to NULL.
      
      * Introduce a new API: int RM_GetServerVersion();, that will return the
      current Redis version in the format 0x00MMmmpp; e.g. 0x00060008;
      
      * Changed unstable version from 999.999.999 to 255.255.255
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      adc3183c
    • Yossi Gottlieb's avatar
      Module API: Add RM_GetClientCertificate(). (#7866) · 0aec98dc
      Yossi Gottlieb authored
      This API function makes it possible to retrieve the X.509 certificate
      used by clients to authenticate TLS connections.
      0aec98dc
    • Yossi Gottlieb's avatar
      Modules: Add RM_GetDetachedThreadSafeContext(). (#7886) · 907da058
      Yossi Gottlieb authored
      The main motivation here is to provide a way for modules to create a
      single, global context that can be used for logging.
      
      Currently, it is possible to obtain a thread-safe context that is not
      attached to any blocked client by using `RM_GetThreadSafeContext`.
      However, the attached context is not linked to the module identity so
      log messages produced are not tagged with the module name.
      
      Ideally we'd fix this in `RM_GetThreadSafeContext` itself but as it
      doesn't accept the current context as an argument there's no way to do
      that in a backwards compatible manner.
      907da058
    • Yossi Gottlieb's avatar
      Modules: add RM_GetCommandKeys(). · 7d117d75
      Yossi Gottlieb authored
      This is essentially the same as calling COMMAND GETKEYS but provides a
      more efficient interface that can be used in every context (i.e. not a
      Redis command).
      7d117d75
  11. 20 Sep, 2020 1 commit
  12. 10 Sep, 2020 1 commit
  13. 09 Sep, 2020 1 commit
  14. 17 Aug, 2020 1 commit
  15. 09 Aug, 2020 1 commit
  16. 23 Jul, 2020 1 commit
  17. 14 Jun, 2020 1 commit
    • meir@redislabs.com's avatar
      Fix RM_ScanKey module api not to return int encoded strings · a89bf734
      meir@redislabs.com authored
      The scan key module API provides the scan callback with the current
      field name and value (if it exists). Those arguments are RedisModuleString*
      which means it supposes to point to robj which is encoded as a string.
      Using createStringObjectFromLongLong function might return robj that
      points to an integer and so break a module that tries for example to
      use RedisModule_StringPtrLen on the given field/value.
      
      The PR introduces a fix that uses the createObject function and sdsfromlonglong function.
      Using those function promise that the field and value pass to the to the
      scan callback will be Strings.
      
      The PR also changes the Scan test module to use RedisModule_StringPtrLen
      to catch the issue. without this, the issue is hidden because
      RedisModule_ReplyWithString knows to handle integer encoding of the
      given robj (RedisModuleString).
      
      The PR also introduces a new test to verify the issue is solved.
      a89bf734
  18. 27 May, 2020 1 commit
  19. 01 Apr, 2020 1 commit
    • Guy Benoish's avatar
      Fix memory corruption in moduleHandleBlockedClients · c4dc5b80
      Guy Benoish authored
      By using a "circular BRPOPLPUSH"-like scenario it was
      possible the get the same client on db->blocking_keys
      twice (See comment in moduleTryServeClientBlockedOnKey)
      
      The fix was actually already implememnted in
      moduleTryServeClientBlockedOnKey but it had a bug:
      the funxction should return 0 or 1 (not OK or ERR)
      
      Other changes:
      1. Added two commands to blockonkeys.c test module (To
         reproduce the case described above)
      2. Simplify blockonkeys.c in order to make testing easier
      3. cast raxSize() to avoid warning with format spec
      c4dc5b80
  20. 31 Mar, 2020 2 commits
  21. 23 Feb, 2020 1 commit
  22. 06 Feb, 2020 1 commit
  23. 30 Jan, 2020 1 commit
    • Guy Benoish's avatar
      ld2string should fail if string contains \0 in the middle · 2deb5551
      Guy Benoish authored
      This bug affected RM_StringToLongDouble and HINCRBYFLOAT.
      I added tests for both cases.
      
      Main changes:
      1. Fixed string2ld to fail if string contains \0 in the middle
      2. Use string2ld in getLongDoubleFromObject - No point of
         having duplicated code here
      
      The two changes above broke RM_SaveLongDouble/RM_LoadLongDouble
      because the long double string was saved with length+1 (An innocent
      mistake, but it's actually a bug - The length passed to
      RM_SaveLongDouble should not include the last \0).
      2deb5551
  24. 30 Dec, 2019 1 commit
    • Guy Benoish's avatar
      Modules: Fix blocked-client-related memory leak · d7d13721
      Guy Benoish authored
      If a blocked module client times-out (or disconnects, unblocked
      by CLIENT command, etc.) we need to call moduleUnblockClient
      in order to free memory allocated by the module sub-system
      and blocked-client private data
      
      Other changes:
      Made blockedonkeys.tcl tests a bit more aggressive in order
      to smoke-out potential memory leaks
      d7d13721
  25. 17 Dec, 2019 3 commits
  26. 12 Dec, 2019 1 commit
    • Yossi Gottlieb's avatar
      Improve RM_ModuleTypeReplaceValue() API. · 0283db58
      Yossi Gottlieb authored
      With the previous API, a NULL return value was ambiguous and could
      represent either an old value of NULL or an error condition. The new API
      returns a status code and allows the old value to be returned
      by-reference.
      
      This commit also includes test coverage based on
      tests/modules/datatype.c which did not exist at the time of the original
      commit.
      0283db58
  27. 02 Dec, 2019 1 commit
  28. 11 Nov, 2019 2 commits
    • Oran Agra's avatar
      Add RM_ScanKey to scan hash, set, zset, changes to RM_Scan API · 0f8692b4
      Oran Agra authored
      - Adding RM_ScanKey
      - Adding tests for RM_ScanKey
      - Refactoring RM_Scan API
      
      Changes in RM_Scan
      - cleanup in docs and coding convention
      - Moving out of experimantal Api
      - Adding ctx to scan callback
      - Dont use cursor of -1 as an indication of done (can be a valid cursor)
      - Set errno when returning 0 for various reasons
      - Rename Cursor to ScanCursor
      - Test filters key that are not strings, and opens a key if NULL
      0f8692b4
    • meir@redislabs.com's avatar
      Added scan implementation to module api. · 11c6ce81
      meir@redislabs.com authored
      The implementation expose the following new functions:
      1. RedisModule_CursorCreate - allow to create a new cursor object for
      keys scanning
      2. RedisModule_CursorRestart - restart an existing cursor to restart the
      scan
      3. RedisModule_CursorDestroy - destroy an existing cursor
      4. RedisModule_Scan - scan keys
      
      The RedisModule_Scan function gets a cursor object, a callback and void*
      (used as user private data).
      The callback will be called for each key in the database proving the key
      name and the value as RedisModuleKey.
      11c6ce81
  29. 10 Nov, 2019 2 commits
    • Oran Agra's avatar
      rename RN_SetLRUOrLFU -> RM_SetLRU and RN_SetLFU · 28c20b4e
      Oran Agra authored
      - the API name was odd, separated to two apis one for LRU and one for LFU
      - the LRU idle time was in 1 second resolution, which might be ok for RDB
        and RESTORE, but i think modules may need higher resolution
      - adding tests for LFU and for handling maxmemory policy mismatch
      28c20b4e
    • Oran Agra's avatar
      fix leak in module api rdb test · 02f21113
      Oran Agra authored
      recently added more reads into that function, if a later read fails, i must
      either free what's already allocated, or return the pointer so that the free
      callback will release it.
      02f21113