1. 11 Oct, 2020 4 commits
    • Meir Shpilraien (Spielrein)'s avatar
      Add Module API for version and compatibility checks (#7865) · adc3183c
      Meir Shpilraien (Spielrein) authored
      
      
      * Introduce a new API's: RM_GetContextFlagsAll, and
      RM_GetKeyspaceNotificationFlagsAll that will return the
      full flags mask of each feature. The module writer can
      check base on this value if the Flags he needs are
      supported or not.
      
      * For each flag, introduce a new value on redismodule.h,
      this value represents the LAST value and should be there
      as a reminder to update it when a new value is added,
      also it will be used in the code to calculate the full
      flags mask (assuming flags are incrementally increasing).
      In addition, stated that the module writer should not use
      the LAST flag directly and he should use the GetFlagAll API's.
      
      * Introduce a new API: RM_IsSubEventSupported, that returns for a given
      event and subevent, whether or not the subevent supported.
      
      * Introduce a new macro RMAPI_FUNC_SUPPORTED(func) that returns whether
      or not a function API is supported by comparing it to NULL.
      
      * Introduce a new API: int RM_GetServerVersion();, that will return the
      current Redis version in the format 0x00MMmmpp; e.g. 0x00060008;
      
      * Changed unstable version from 999.999.999 to 255.255.255
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      adc3183c
    • Yossi Gottlieb's avatar
      Module API: Add RM_GetClientCertificate(). (#7866) · 0aec98dc
      Yossi Gottlieb authored
      This API function makes it possible to retrieve the X.509 certificate
      used by clients to authenticate TLS connections.
      0aec98dc
    • Yossi Gottlieb's avatar
      Modules: Add RM_GetDetachedThreadSafeContext(). (#7886) · 907da058
      Yossi Gottlieb authored
      The main motivation here is to provide a way for modules to create a
      single, global context that can be used for logging.
      
      Currently, it is possible to obtain a thread-safe context that is not
      attached to any blocked client by using `RM_GetThreadSafeContext`.
      However, the attached context is not linked to the module identity so
      log messages produced are not tagged with the module name.
      
      Ideally we'd fix this in `RM_GetThreadSafeContext` itself but as it
      doesn't accept the current context as an argument there's no way to do
      that in a backwards compatible manner.
      907da058
    • Yossi Gottlieb's avatar
      Modules: add RM_GetCommandKeys(). · 7d117d75
      Yossi Gottlieb authored
      This is essentially the same as calling COMMAND GETKEYS but provides a
      more efficient interface that can be used in every context (i.e. not a
      Redis command).
      7d117d75
  2. 20 Sep, 2020 1 commit
  3. 09 Sep, 2020 1 commit
  4. 06 Sep, 2020 1 commit
  5. 17 Aug, 2020 1 commit
  6. 09 Aug, 2020 1 commit
  7. 06 Aug, 2020 1 commit
    • Oran Agra's avatar
      Accelerate diskless master connections, and general re-connections (#6271) · c17e597d
      Oran Agra authored
      Diskless master has some inherent latencies.
      1) fork starts with delay from cron rather than immediately
      2) replica is put online only after an ACK. but the ACK
         was sent only once a second.
      3) but even if it would arrive immediately, it will not
         register in case cron didn't yet detect that the fork is done.
      
      Besides that, when a replica disconnects, it doesn't immediately
      attempts to re-connect, it waits for replication cron (one per second).
      in case it was already online, it may be important to try to re-connect
      as soon as possible, so that the backlog at the master doesn't vanish.
      
      In case it disconnected during rdb transfer, one can argue that it's
      not very important to re-connect immediately, but this is needed for the
      "diskless loading short read" test to be able to run 100 iterations in 5
      seconds, rather than 3 (waiting for replication cron re-connection)
      
      changes in this commit:
      1) sync command starts a fork immediately if no sync_delay is configured
      2) replica sends REPLCONF ACK when done reading the rdb (rather than on 1s cron)
      3) when a replica unexpectedly disconnets, it immediately tries to
         re-connect rather than waiting 1s
      4) when when a child exits, if there is another replica waiting, we spawn a new
         one right away, instead of waiting for 1s replicationCron.
      5) added a call to connectWithMaster from replicationSetMaster. which is called
         from the REPLICAOF command but also in 3 places in cluster.c, in all of
         these the connection attempt will now be immediate instead of delayed by 1
         second.
      
      side note:
      we can add a call to rdbPipeReadHandler in replconfCommand when getting
      a REPLCONF ACK from the replica to solve a race where the replica got
      the entire rdb and EOF marker before we detected that the pipe was
      closed.
      in the test i did see this race happens in one about of some 300 runs,
      but i concluded that this race is unlikely in real life (where the
      replica is on another host and we're more likely to first detect the
      pipe was closed.
      the test runs 100 iterations in 3 seconds, so in some cases it'll take 4
      seconds instead (waiting for another REPLCONF ACK).
      
      Removing unneeded startBgsaveForReplication from updateSlavesWaitingForBgsave
      Now that CheckChildrenDone is calling the new replicationStartPendingFork
      (extracted from serverCron) there's actually no need to call
      startBgsaveForReplication from updateSlavesWaitingForBgsave anymore,
      since as soon as updateSlavesWaitingForBgsave returns, CheckChildrenDone is
      calling replicationStartPendingFork that handles that anyway.
      The code in updateSlavesWaitingForBgsave had a bug in which it ignored
      repl-diskless-sync-delay, but removing that code shows that this bug was
      hiding another bug, which is that the max_idle should have used >= and
      not >, this one second delay has a big impact on my new test.
      c17e597d
  8. 31 Jul, 2020 1 commit
  9. 28 Jul, 2020 1 commit
    • Oran Agra's avatar
      Fix failing tests due to issues with wait_for_log_message (#7572) · 109b5ccd
      Oran Agra authored
      - the test now waits for specific set of log messages rather than wait for
        timeout looking for just one message.
      - we don't wanna sample the current length of the log after an action, due
        to a race, we need to start the search from the line number of the last
        message we where waiting for.
      - when attempting to trigger a full sync, use multi-exec to avoid a race
        where the replica manages to re-connect before we completed the set of
        actions that should force a full sync.
      - fix verify_log_message which was broken and unused
      109b5ccd
  10. 23 Jul, 2020 1 commit
  11. 10 Jul, 2020 1 commit
    • Oran Agra's avatar
      stabilize tests that look for log lines (#7367) · 8e76e134
      Oran Agra authored
      tests were sensitive to additional log lines appearing in the log
      causing the search to come empty handed.
      
      instead of just looking for the n last log lines, capture the log lines
      before performing the action, and then search from that offset.
      8e76e134
  12. 14 Jun, 2020 1 commit
    • meir@redislabs.com's avatar
      Fix RM_ScanKey module api not to return int encoded strings · a89bf734
      meir@redislabs.com authored
      The scan key module API provides the scan callback with the current
      field name and value (if it exists). Those arguments are RedisModuleString*
      which means it supposes to point to robj which is encoded as a string.
      Using createStringObjectFromLongLong function might return robj that
      points to an integer and so break a module that tries for example to
      use RedisModule_StringPtrLen on the given field/value.
      
      The PR introduces a fix that uses the createObject function and sdsfromlonglong function.
      Using those function promise that the field and value pass to the to the
      scan callback will be Strings.
      
      The PR also changes the Scan test module to use RedisModule_StringPtrLen
      to catch the issue. without this, the issue is hidden because
      RedisModule_ReplyWithString knows to handle integer encoding of the
      given robj (RedisModuleString).
      
      The PR also introduces a new test to verify the issue is solved.
      a89bf734
  13. 03 Apr, 2020 1 commit
    • Guy Benoish's avatar
      Try to fix time-sensitive tests in blockonkey.tcl · 1b0d30ae
      Guy Benoish authored
      There is an inherent race between the deferring client and the
      "main" client of the test: While the deferring client issues a blocking
      command, we can't know for sure that by the time the "main" client
      tries to issue another command (Usually one that unblocks the deferring
      client) the deferring client is even blocked...
      For lack of a better choice this commit uses TCL's 'after' in order
      to give some time for the deferring client to issues its blocking
      command before the "main" client does its thing.
      This problem probably exists in many other tests but this commit
      tries to fix blockonkeys.tcl
      1b0d30ae
  14. 01 Apr, 2020 1 commit
    • Guy Benoish's avatar
      Fix memory corruption in moduleHandleBlockedClients · c4dc5b80
      Guy Benoish authored
      By using a "circular BRPOPLPUSH"-like scenario it was
      possible the get the same client on db->blocking_keys
      twice (See comment in moduleTryServeClientBlockedOnKey)
      
      The fix was actually already implememnted in
      moduleTryServeClientBlockedOnKey but it had a bug:
      the funxction should return 0 or 1 (not OK or ERR)
      
      Other changes:
      1. Added two commands to blockonkeys.c test module (To
         reproduce the case described above)
      2. Simplify blockonkeys.c in order to make testing easier
      3. cast raxSize() to avoid warning with format spec
      c4dc5b80
  15. 31 Mar, 2020 2 commits
  16. 23 Feb, 2020 1 commit
  17. 30 Dec, 2019 1 commit
    • Guy Benoish's avatar
      Modules: Fix blocked-client-related memory leak · d7d13721
      Guy Benoish authored
      If a blocked module client times-out (or disconnects, unblocked
      by CLIENT command, etc.) we need to call moduleUnblockClient
      in order to free memory allocated by the module sub-system
      and blocked-client private data
      
      Other changes:
      Made blockedonkeys.tcl tests a bit more aggressive in order
      to smoke-out potential memory leaks
      d7d13721
  18. 17 Dec, 2019 1 commit
  19. 12 Dec, 2019 1 commit
    • Yossi Gottlieb's avatar
      Improve RM_ModuleTypeReplaceValue() API. · 0283db58
      Yossi Gottlieb authored
      With the previous API, a NULL return value was ambiguous and could
      represent either an old value of NULL or an error condition. The new API
      returns a status code and allows the old value to be returned
      by-reference.
      
      This commit also includes test coverage based on
      tests/modules/datatype.c which did not exist at the time of the original
      commit.
      0283db58
  20. 19 Nov, 2019 1 commit
    • Oran Agra's avatar
      try to fix an unstable test (module hook for loading progress) · ed226976
      Oran Agra authored
      there were two lssues, one is taht BGREWRITEAOF failed since the initial one was still in progress
      the solution for this one is to enable appendonly from the server startup so there's no initial aofrw.
      
      the other problem was 0 loading progress events, theory is that on some
      platforms a sleep of 1 will cause a much greater delay due to the context
      switch, but on other platform it doesn't. in theory a sleep of 100 micro
      for 1k keys whould take 100ms, and with hz of 500 we should be gettering
      50 events (one every 2ms). in practise it doesn't work like that, so trying
      to find a sleep that would be long enough but still not cause the test to take
      too long.
      ed226976
  21. 11 Nov, 2019 2 commits
    • Oran Agra's avatar
      Add RM_ScanKey to scan hash, set, zset, changes to RM_Scan API · 0f8692b4
      Oran Agra authored
      - Adding RM_ScanKey
      - Adding tests for RM_ScanKey
      - Refactoring RM_Scan API
      
      Changes in RM_Scan
      - cleanup in docs and coding convention
      - Moving out of experimantal Api
      - Adding ctx to scan callback
      - Dont use cursor of -1 as an indication of done (can be a valid cursor)
      - Set errno when returning 0 for various reasons
      - Rename Cursor to ScanCursor
      - Test filters key that are not strings, and opens a key if NULL
      0f8692b4
    • meir@redislabs.com's avatar
      Added scan implementation to module api. · 11c6ce81
      meir@redislabs.com authored
      The implementation expose the following new functions:
      1. RedisModule_CursorCreate - allow to create a new cursor object for
      keys scanning
      2. RedisModule_CursorRestart - restart an existing cursor to restart the
      scan
      3. RedisModule_CursorDestroy - destroy an existing cursor
      4. RedisModule_Scan - scan keys
      
      The RedisModule_Scan function gets a cursor object, a callback and void*
      (used as user private data).
      The callback will be called for each key in the database proving the key
      name and the value as RedisModuleKey.
      11c6ce81
  22. 10 Nov, 2019 1 commit
    • Oran Agra's avatar
      rename RN_SetLRUOrLFU -> RM_SetLRU and RN_SetLFU · 28c20b4e
      Oran Agra authored
      - the API name was odd, separated to two apis one for LRU and one for LFU
      - the LRU idle time was in 1 second resolution, which might be ok for RDB
        and RESTORE, but i think modules may need higher resolution
      - adding tests for LFU and for handling maxmemory policy mismatch
      28c20b4e
  23. 04 Nov, 2019 6 commits
  24. 03 Nov, 2019 1 commit
    • Oran Agra's avatar
      Add module api for looking into INFO fields · 4d580438
      Oran Agra authored
      - Add RM_GetServerInfo and friends
      - Add auto memory for new opaque struct
      - Add tests for new APIs
      
      other minor fixes:
      - add const in various char pointers
      - requested_section in modulesCollectInfo was actually not sds but char*
      - extract new string2d out of getDoubleFromObject for code reuse
      
      Add module API for
      4d580438
  25. 29 Oct, 2019 1 commit
    • Oran Agra's avatar
      Modules hooks: complete missing hooks for the initial set of hooks · 51c3ff8d
      Oran Agra authored
      * replication hooks: role change, master link status, replica online/offline
      * persistence hooks: saving, loading, loading progress
      * misc hooks: cron loop, shutdown, module loaded/unloaded
      * change the way hooks test work, and add tests for all of the above
      
      startLoading() now gets flag indicating what is loaded.
      stopLoading() now gets an indication of success or failure.
      adding startSaving() and stopSaving() with similar args and role.
      51c3ff8d
  26. 28 Oct, 2019 1 commit
    • Oran Agra's avatar
      Module api tests for RM_Call · 0399b5a2
      Oran Agra authored
      Adding a test for coverage for RM_Call in a new "misc" unit
      to be used for various short simple tests
      
      also solves compilation warnings in redismodule.h and fork.c
      0399b5a2
  27. 24 Oct, 2019 3 commits
  28. 03 Oct, 2019 1 commit