- 08 Dec, 2020 1 commit
-
-
Oran Agra authored
Module blocked clients cache the response in a temporary client, the reply list in this client would be affected by the recent fix in #7202, but when the reply is later copied into the real client, it would have bypassed all the checks for output buffer limit, which would have resulted in both: responding with a partial response to the client, and also not disconnecting it at all.
-
- 06 Dec, 2020 1 commit
-
-
guybe7 authored
One way this was happening is when a module issued an RM_Call which would inject MULTI. If the module command that does that was itself issued by something else that already did added MULTI (e.g. another module, or a Lua script), it would have caused nested MULTI. In fact the MULTI state in the client or the MULTI_EMITTED flag in the context isn't the right indication that we need to propagate MULTI or not, because on a nested calls (possibly a module action called by a keyspace event of another module action), these flags aren't retained / reflected. instead there's now a global propagate_in_transaction flag for that. in addition to that, we now have a global in_eval and in_exec flags, to serve the flags of RM_GetContextFlags, since their dependence on the current client is wrong for the same reasons mentioned above.
-
- 01 Dec, 2020 1 commit
-
-
Itamar Haber authored
Fixes #7923. This PR appropriates the special `&` symbol (because `@` and `*` are taken), followed by a literal value or pattern for describing the Pub/Sub patterns that an ACL user can interact with. It is similar to the existing key patterns mechanism in function (additive) and implementation (copy-pasta). It also adds the allchannels and resetchannels ACL keywords, naturally. The default user is given allchannels permissions, whereas new users get whatever is defined by the acl-pubsub-default configuration directive. For backward compatibility in 6.2, the default of this directive is allchannels but this is likely to be changed to resetchannels in the next major version for stronger default security settings. Unless allchannels is set for the user, channel access permissions are checked as follows : * Calls to both PUBLISH and SUBSCRIBE will fail unless a pattern matching the argumentative channel name(s) exists for the user. * Calls to PSUBSCRIBE will fail unless the pattern(s) provided as an argument literally exist(s) in the user's list. Such failures are logged to the ACL log. Runtime changes to channel permissions for a user with existing subscribing clients cause said clients to disconnect unless the new permissions permit the connections to continue. Note, however, that PSUBSCRIBErs' patterns are matched literally, so given the change bar:* -> b*, pattern subscribers to bar:* will be disconnected. Notes/questions: * UNSUBSCRIBE, PUNSUBSCRIBE and PUBSUB remain unprotected due to lack of reasons for touching them.
-
- 17 Nov, 2020 1 commit
-
-
Meir Shpilraien (Spielrein) authored
Blocking command should not be used with MULTI, LUA, and RM_Call. This is because, the caller, who executes the command in this context, expects a reply. Today, LUA and MULTI have a special (and different) treatment to blocking commands: LUA - Most commands are marked with no-script flag which are checked when executing and command from LUA, commands that are not marked (like XREAD) verify that their blocking mode is not used inside LUA (by checking the CLIENT_LUA client flag). MULTI - Command that is going to block, first verify that the client is not inside multi (by checking the CLIENT_MULTI client flag). If the client is inside multi, they return a result which is a match to the empty key with no timeout (for example blpop inside MULTI will act as lpop) For modules that perform RM_Call with blocking command, the returned results type is REDISMODULE_REPLY_UNKNOWN and the caller can not really know what happened. Disadvantages of the current state are: No unified approach, LUA, MULTI, and RM_Call, each has a different treatment Module can not safely execute blocking command (and get reply or error). Though It is true that modules are not like LUA or MULTI and should be smarter not to execute blocking commands on RM_Call, sometimes you want to execute a command base on client input (for example if you create a module that provides a new scripting language like javascript or python). While modules (on modules command) can check for REDISMODULE_CTX_FLAGS_LUA or REDISMODULE_CTX_FLAGS_MULTI to know not to block the client, there is no way to check if the command came from another module using RM_Call. So there is no way for a module to know not to block another module RM_Call execution. This commit adds a way to unify the treatment for blocking clients by introducing a new CLIENT_DENY_BLOCKING client flag. On LUA, MULTI, and RM_Call the new flag turned on to signify that the client should not be blocked. A blocking command verifies that the flag is turned off before blocking. If a blocking command sees that the CLIENT_DENY_BLOCKING flag is on, it's not blocking and return results which are matches to empty key with no timeout (as MULTI does today). The new flag is checked on the following commands: List blocking commands: BLPOP, BRPOP, BRPOPLPUSH, BLMOVE, Zset blocking commands: BZPOPMIN, BZPOPMAX Stream blocking commands: XREAD, XREADGROUP SUBSCRIBE, PSUBSCRIBE, MONITOR In addition, the new flag is turned on inside the AOF client, we do not want to block the AOF client to prevent deadlocks and commands ordering issues (and there is also an existing assert in the code that verifies it). To keep backward compatibility on LUA, all the no-script flags on existing commands were kept untouched. In addition, a LUA special treatment on XREAD and XREADGROUP was kept. To keep backward compatibility on MULTI (which today allows SUBSCRIBE, and PSUBSCRIBE). We added a special treatment on those commands to allow executing them on MULTI. The only backward compatibility issue that this PR introduces is that now MONITOR is not allowed inside MULTI. Tests were added to verify blocking commands are not blocking the client on LUA, MULTI, or RM_Call. Tests were added to verify the module can check for CLIENT_DENY_BLOCKING flag. Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Itamar Haber <itamar@redislabs.com>
-
- 16 Nov, 2020 1 commit
-
-
chenyangyang authored
Add two optional callbacks to the RedisModuleTypeMethods structure, which is `free_effort` and `unlink`. the `free_effort` callback indicates the effort required to free a module memory. Currently, if the effort exceeds LAZYFREE_THRESHOLD, the module memory may be released asynchronously. the `unlink` callback indicates the key has been removed from the DB by redis, and may soon be freed by a background thread. Add `lazyfreed_objects` info field, which represents the number of objects that have been lazyfreed since redis was started. Add `RM_GetTypeMethodVersion` API, which return the current redis-server runtime value of `REDISMODULE_TYPE_METHOD_VERSION`. You can use that when calling `RM_CreateDataType` to know which fields of RedisModuleTypeMethods are gonna be supported and which will be ignored.
-
- 11 Nov, 2020 1 commit
-
-
Yossi Gottlieb authored
-
- 11 Oct, 2020 4 commits
-
-
Meir Shpilraien (Spielrein) authored
* Introduce a new API's: RM_GetContextFlagsAll, and RM_GetKeyspaceNotificationFlagsAll that will return the full flags mask of each feature. The module writer can check base on this value if the Flags he needs are supported or not. * For each flag, introduce a new value on redismodule.h, this value represents the LAST value and should be there as a reminder to update it when a new value is added, also it will be used in the code to calculate the full flags mask (assuming flags are incrementally increasing). In addition, stated that the module writer should not use the LAST flag directly and he should use the GetFlagAll API's. * Introduce a new API: RM_IsSubEventSupported, that returns for a given event and subevent, whether or not the subevent supported. * Introduce a new macro RMAPI_FUNC_SUPPORTED(func) that returns whether or not a function API is supported by comparing it to NULL. * Introduce a new API: int RM_GetServerVersion();, that will return the current Redis version in the format 0x00MMmmpp; e.g. 0x00060008; * Changed unstable version from 999.999.999 to 255.255.255 Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Yossi Gottlieb <yossigo@gmail.com>
-
Yossi Gottlieb authored
This API function makes it possible to retrieve the X.509 certificate used by clients to authenticate TLS connections.
-
Yossi Gottlieb authored
The main motivation here is to provide a way for modules to create a single, global context that can be used for logging. Currently, it is possible to obtain a thread-safe context that is not attached to any blocked client by using `RM_GetThreadSafeContext`. However, the attached context is not linked to the module identity so log messages produced are not tagged with the module name. Ideally we'd fix this in `RM_GetThreadSafeContext` itself but as it doesn't accept the current context as an argument there's no way to do that in a backwards compatible manner.
-
Yossi Gottlieb authored
This is essentially the same as calling COMMAND GETKEYS but provides a more efficient interface that can be used in every context (i.e. not a Redis command).
-
- 20 Sep, 2020 1 commit
-
-
Wen Hui authored
-
- 09 Sep, 2020 1 commit
-
-
Roi Lipman authored
Co-authored-by:
Yossi Gottlieb <yossigo@gmail.com> Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 06 Sep, 2020 1 commit
-
-
Oran Agra authored
this is important when running a test with --loop
-
- 17 Aug, 2020 1 commit
-
-
Yossi Gottlieb authored
The `REDISMODULE_CLIENTINFO_FLAG_SSL` flag was already a part of the `RedisModuleClientInfo` structure but was not implemented.
-
- 09 Aug, 2020 1 commit
-
-
Meir Shpilraien (Spielrein) authored
Added RedisModule_HoldString that either returns a shallow copy of the given String (by increasing the String ref count) or a new deep copy of String in case its not possible to get a shallow copy. Co-authored-by:
Itamar Haber <itamar@redislabs.com>
-
- 06 Aug, 2020 1 commit
-
-
Oran Agra authored
Diskless master has some inherent latencies. 1) fork starts with delay from cron rather than immediately 2) replica is put online only after an ACK. but the ACK was sent only once a second. 3) but even if it would arrive immediately, it will not register in case cron didn't yet detect that the fork is done. Besides that, when a replica disconnects, it doesn't immediately attempts to re-connect, it waits for replication cron (one per second). in case it was already online, it may be important to try to re-connect as soon as possible, so that the backlog at the master doesn't vanish. In case it disconnected during rdb transfer, one can argue that it's not very important to re-connect immediately, but this is needed for the "diskless loading short read" test to be able to run 100 iterations in 5 seconds, rather than 3 (waiting for replication cron re-connection) changes in this commit: 1) sync command starts a fork immediately if no sync_delay is configured 2) replica sends REPLCONF ACK when done reading the rdb (rather than on 1s cron) 3) when a replica unexpectedly disconnets, it immediately tries to re-connect rather than waiting 1s 4) when when a child exits, if there is another replica waiting, we spawn a new one right away, instead of waiting for 1s replicationCron. 5) added a call to connectWithMaster from replicationSetMaster. which is called from the REPLICAOF command but also in 3 places in cluster.c, in all of these the connection attempt will now be immediate instead of delayed by 1 second. side note: we can add a call to rdbPipeReadHandler in replconfCommand when getting a REPLCONF ACK from the replica to solve a race where the replica got the entire rdb and EOF marker before we detected that the pipe was closed. in the test i did see this race happens in one about of some 300 runs, but i concluded that this race is unlikely in real life (where the replica is on another host and we're more likely to first detect the pipe was closed. the test runs 100 iterations in 3 seconds, so in some cases it'll take 4 seconds instead (waiting for another REPLCONF ACK). Removing unneeded startBgsaveForReplication from updateSlavesWaitingForBgsave Now that CheckChildrenDone is calling the new replicationStartPendingFork (extracted from serverCron) there's actually no need to call startBgsaveForReplication from updateSlavesWaitingForBgsave anymore, since as soon as updateSlavesWaitingForBgsave returns, CheckChildrenDone is calling replicationStartPendingFork that handles that anyway. The code in updateSlavesWaitingForBgsave had a bug in which it ignored repl-diskless-sync-delay, but removing that code shows that this bug was hiding another bug, which is that the max_idle should have used >= and not >, this one second delay has a big impact on my new test.
-
- 31 Jul, 2020 1 commit
-
-
Oran Agra authored
besides, hooks test was time sensitive. when the replica managed to reconnect quickly after the client kill, the test would fail
-
- 28 Jul, 2020 1 commit
-
-
Oran Agra authored
- the test now waits for specific set of log messages rather than wait for timeout looking for just one message. - we don't wanna sample the current length of the log after an action, due to a race, we need to start the search from the line number of the last message we where waiting for. - when attempting to trigger a full sync, use multi-exec to avoid a race where the replica manages to re-connect before we completed the set of actions that should force a full sync. - fix verify_log_message which was broken and unused
-
- 23 Jul, 2020 1 commit
-
-
Meir Shpilraien (Spielrein) authored
Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Itamar Haber <itamar@redislabs.com>
-
- 10 Jul, 2020 1 commit
-
-
Oran Agra authored
tests were sensitive to additional log lines appearing in the log causing the search to come empty handed. instead of just looking for the n last log lines, capture the log lines before performing the action, and then search from that offset.
-
- 14 Jun, 2020 1 commit
-
-
meir@redislabs.com authored
The scan key module API provides the scan callback with the current field name and value (if it exists). Those arguments are RedisModuleString* which means it supposes to point to robj which is encoded as a string. Using createStringObjectFromLongLong function might return robj that points to an integer and so break a module that tries for example to use RedisModule_StringPtrLen on the given field/value. The PR introduces a fix that uses the createObject function and sdsfromlonglong function. Using those function promise that the field and value pass to the to the scan callback will be Strings. The PR also changes the Scan test module to use RedisModule_StringPtrLen to catch the issue. without this, the issue is hidden because RedisModule_ReplyWithString knows to handle integer encoding of the given robj (RedisModuleString). The PR also introduces a new test to verify the issue is solved.
-
- 03 Apr, 2020 1 commit
-
-
Guy Benoish authored
There is an inherent race between the deferring client and the "main" client of the test: While the deferring client issues a blocking command, we can't know for sure that by the time the "main" client tries to issue another command (Usually one that unblocks the deferring client) the deferring client is even blocked... For lack of a better choice this commit uses TCL's 'after' in order to give some time for the deferring client to issues its blocking command before the "main" client does its thing. This problem probably exists in many other tests but this commit tries to fix blockonkeys.tcl
-
- 01 Apr, 2020 1 commit
-
-
Guy Benoish authored
By using a "circular BRPOPLPUSH"-like scenario it was possible the get the same client on db->blocking_keys twice (See comment in moduleTryServeClientBlockedOnKey) The fix was actually already implememnted in moduleTryServeClientBlockedOnKey but it had a bug: the funxction should return 0 or 1 (not OK or ERR) Other changes: 1. Added two commands to blockonkeys.c test module (To reproduce the case described above) 2. Simplify blockonkeys.c in order to make testing easier 3. cast raxSize() to avoid warning with format spec
-
- 31 Mar, 2020 2 commits
-
-
Guy Benoish authored
Makse sure call() doesn't wrap replicated commands with a redundant MULTI/EXEC Other, unrelated changes: 1. Formatting compiler warning in INFO CLIENTS 2. Use CLIENT_ID_AOF instead of UINT64_MAX
-
antirez authored
-
- 23 Feb, 2020 1 commit
-
-
Oran Agra authored
in some cases we were trying to kill the fork before it got created
-
- 30 Dec, 2019 1 commit
-
-
Guy Benoish authored
If a blocked module client times-out (or disconnects, unblocked by CLIENT command, etc.) we need to call moduleUnblockClient in order to free memory allocated by the module sub-system and blocked-client private data Other changes: Made blockedonkeys.tcl tests a bit more aggressive in order to smoke-out potential memory leaks
-
- 17 Dec, 2019 1 commit
-
-
Madelyn Olson authored
-
- 12 Dec, 2019 1 commit
-
-
Yossi Gottlieb authored
With the previous API, a NULL return value was ambiguous and could represent either an old value of NULL or an error condition. The new API returns a status code and allows the old value to be returned by-reference. This commit also includes test coverage based on tests/modules/datatype.c which did not exist at the time of the original commit.
-
- 19 Nov, 2019 1 commit
-
-
Oran Agra authored
there were two lssues, one is taht BGREWRITEAOF failed since the initial one was still in progress the solution for this one is to enable appendonly from the server startup so there's no initial aofrw. the other problem was 0 loading progress events, theory is that on some platforms a sleep of 1 will cause a much greater delay due to the context switch, but on other platform it doesn't. in theory a sleep of 100 micro for 1k keys whould take 100ms, and with hz of 500 we should be gettering 50 events (one every 2ms). in practise it doesn't work like that, so trying to find a sleep that would be long enough but still not cause the test to take too long.
-
- 11 Nov, 2019 2 commits
-
-
Oran Agra authored
- Adding RM_ScanKey - Adding tests for RM_ScanKey - Refactoring RM_Scan API Changes in RM_Scan - cleanup in docs and coding convention - Moving out of experimantal Api - Adding ctx to scan callback - Dont use cursor of -1 as an indication of done (can be a valid cursor) - Set errno when returning 0 for various reasons - Rename Cursor to ScanCursor - Test filters key that are not strings, and opens a key if NULL
-
meir@redislabs.com authored
The implementation expose the following new functions: 1. RedisModule_CursorCreate - allow to create a new cursor object for keys scanning 2. RedisModule_CursorRestart - restart an existing cursor to restart the scan 3. RedisModule_CursorDestroy - destroy an existing cursor 4. RedisModule_Scan - scan keys The RedisModule_Scan function gets a cursor object, a callback and void* (used as user private data). The callback will be called for each key in the database proving the key name and the value as RedisModuleKey.
-
- 10 Nov, 2019 1 commit
-
-
Oran Agra authored
- the API name was odd, separated to two apis one for LRU and one for LFU - the LRU idle time was in 1 second resolution, which might be ok for RDB and RESTORE, but i think modules may need higher resolution - adding tests for LFU and for handling maxmemory policy mismatch
-
- 04 Nov, 2019 6 commits
-
-
Yossi Gottlieb authored
Add two new functions that leverage the RedisModuleDataType mechanism for RDB serialization/deserialization and make it possible to use it to/from arbitrary strings: * RM_SaveDataTypeToString() * RM_LoadDataTypeFromString()
-
artix authored
-
Oran Agra authored
-
Guy Benoish authored
-
Oran Agra authored
rename RM_ServerInfoGetFieldNumerical RM_ServerInfoGetFieldSigned move string2ull to util.c fix leak in RM_GetServerInfo when duplicate info fields exist
-
Oran Agra authored
-
- 03 Nov, 2019 1 commit
-
-
Oran Agra authored
- Add RM_GetServerInfo and friends - Add auto memory for new opaque struct - Add tests for new APIs other minor fixes: - add const in various char pointers - requested_section in modulesCollectInfo was actually not sds but char* - extract new string2d out of getDoubleFromObject for code reuse Add module API for
-