- 23 Sep, 2021 1 commit
-
-
yoav-steinberg authored
### Description A mechanism for disconnecting clients when the sum of all connected clients is above a configured limit. This prevents eviction or OOM caused by accumulated used memory between all clients. It's a complimentary mechanism to the `client-output-buffer-limit` mechanism which takes into account not only a single client and not only output buffers but rather all memory used by all clients. #### Design The general design is as following: * We track memory usage of each client, taking into account all memory used by the client (query buffer, output buffer, parsed arguments, etc...). This is kept up to date after reading from the socket, after processing commands and after writing to the socket. * Based on the used memory we sort all clients into buckets. Each bucket contains all clients using up up to x2 memory of the clients in the bucket below it. For example up to 1m clients, up to 2m clients, up to 4m clients, ... * Before processing a command and before sleep we check if we're over the configured limit. If we are we start disconnecting clients from larger buckets downwards until we're under the limit. #### Config `maxmemory-clients` max memory all clients are allowed to consume, above this threshold we disconnect clients. This config can either be set to 0 (meaning no limit), a size in bytes (possibly with MB/GB suffix), or as a percentage of `maxmemory` by using the `%` suffix (e.g. setting it to `10%` would mean 10% of `maxmemory`). #### Important code changes * During the development I encountered yet more situations where our io-threads access global vars. And needed to fix them. I also had to handle keeps the clients sorted into the memory buckets (which are global) while their memory usage changes in the io-thread. To achieve this I decided to simplify how we check if we're in an io-thread and make it much more explicit. I removed the `CLIENT_PENDING_READ` flag used for checking if the client is in an io-thread (it wasn't used for anything else) and just used the global `io_threads_op` variable the same way to check during writes. * I optimized the cleanup of the client from the `clients_pending_read` list on client freeing. We now store a pointer in the `client` struct to this list so we don't need to search in it (`pending_read_list_node`). * Added `evicted_clients` stat to `INFO` command. * Added `CLIENT NO-EVICT ON|OFF` sub command to exclude a specific client from the client eviction mechanism. Added corrosponding 'e' flag in the client info string. * Added `multi-mem` field in the client info string to show how much memory is used up by buffered multi commands. * Client `tot-mem` now accounts for buffered multi-commands, pubsub patterns and channels (partially), tracking prefixes (partially). * CLIENT_CLOSE_ASAP flag is now handled in a new `beforeNextClient()` function so clients will be disconnected between processing different clients and not only before sleep. This new function can be used in the future for work we want to do outside the command processing loop but don't want to wait for all clients to be processed before we get to it. Specifically I wanted to handle output-buffer-limit related closing before we process client eviction in case the two race with each other. * Added a `DEBUG CLIENT-EVICTION` command to print out info about the client eviction buckets. * Each client now holds a pointer to the client eviction memory usage bucket it belongs to and listNode to itself in that bucket for quick removal. * Global `io_threads_op` variable now can contain a `IO_THREADS_OP_IDLE` value indicating no io-threading is currently being executed. * In order to track memory used by each clients in real-time we can't rely on updating these stats in `clientsCron()` alone anymore. So now I call `updateClientMemUsage()` (used to be `clientsCronTrackClientsMemUsage()`) after command processing, after writing data to pubsub clients, after writing the output buffer and after reading from the socket (and maybe other places too). The function is written to be fast. * Clients are evicted if needed (with appropriate log line) in `beforeSleep()` and before processing a command (before performing oom-checks and key-eviction). * All clients memory usage buckets are grouped as follows: * All clients using less than 64k. * 64K..128K * 128K..256K * ... * 2G..4G * All clients using 4g and up. * Added client-eviction.tcl with a bunch of tests for the new mechanism. * Extended maxmemory.tcl to test the interaction between maxmemory and maxmemory-clients settings. * Added an option to flag a numeric configuration variable as a "percent", this means that if we encounter a '%' after the number in the config file (or config set command) we consider it as valid. Such a number is store internally as a negative value. This way an integer value can be interpreted as either a percent (negative) or absolute value (positive). This is useful for example if some numeric configuration can optionally be set to a percentage of something else. Co-authored-by:Oran Agra <oran@redislabs.com>
-
- 23 Aug, 2021 1 commit
-
-
Wen Hui authored
This aims to solve the issue in CONFIG SET maxmemory can only set maxmemory to up to 9223372036854775807 (2^63) while the maxmemory should be ULLONG. Added a memtoull function to convert a string representing an amount of memory into the number of bytes (similar to memtoll but for ull). Also added ull2string to convert a ULLong to string (Similar to ll2string).
-
- 10 Mar, 2021 1 commit
-
-
sundb authored
1. Add `redis-server test all` support to run all tests. 2. Add redis test to daily ci. 3. Add `--accurate` option to run slow tests for more iterations (so that by default we run less cycles (shorter time, and less prints). 4. Move dict benchmark to REDIS_TEST. 5. fix some leaks in tests 6. make quicklist tests run on a specific fill set of options rather than huge ranges 7. move some prints in quicklist test outside their loops to reduce prints 8. removing sds.h from dict.c since it is now used in both redis-server and redis-cli (uses hiredis sds)
-
- 15 Feb, 2021 1 commit
-
-
Yossi Gottlieb authored
Fixes #8489
-
- 18 Jan, 2021 1 commit
-
-
Raghav Muddur authored
-
- 13 Dec, 2020 1 commit
-
-
Yossi Gottlieb authored
* Allow runtest-moduleapi use a different 'make', for systems where GNU Make is 'gmake'. * Fix issue with builds on Solaris re-building everything from scratch due to CFLAGS/LDFLAGS not stored. * Fix compile failure on Solaris due to atomicvar and a bunch of warnings. * Fix garbled log timestamps on Solaris.
-
- 06 May, 2020 1 commit
-
-
antirez authored
-
- 05 May, 2020 1 commit
-
-
Brad Dunbar authored
-
- 23 Apr, 2020 1 commit
-
-
antirez authored
Now that we have an interface to use this API directly, via ACL GENPASS, we are no longer sure what people could do with it. So why don't make it a strong primitive exported by Redis in order to create unique IDs and so forth? The implementation was tested against the test vectors that can be found in RFC4231.
-
- 30 Jan, 2020 1 commit
-
-
Guy Benoish authored
This bug affected RM_StringToLongDouble and HINCRBYFLOAT. I added tests for both cases. Main changes: 1. Fixed string2ld to fail if string contains \0 in the middle 2. Use string2ld in getLongDoubleFromObject - No point of having duplicated code here The two changes above broke RM_SaveLongDouble/RM_LoadLongDouble because the long double string was saved with length+1 (An innocent mistake, but it's actually a bug - The length passed to RM_SaveLongDouble should not include the last \0).
-
- 05 Nov, 2019 1 commit
-
-
Guy Benoish authored
-
- 04 Nov, 2019 1 commit
-
-
Oran Agra authored
rename RM_ServerInfoGetFieldNumerical RM_ServerInfoGetFieldSigned move string2ull to util.c fix leak in RM_GetServerInfo when duplicate info fields exist
-
- 03 Nov, 2019 2 commits
-
-
Oran Agra authored
looks like each platform implements long double differently (different bit count) so we can't save them as binary, and we also want to avoid creating a new RDB format version, so we save these are hex strings using "%La". This commit includes a change in the arguments of ld2string to support this. as well as tests for coverage and short reads. coded by @guybe7
-
Oran Agra authored
- Add RM_GetServerInfo and friends - Add auto memory for new opaque struct - Add tests for new APIs other minor fixes: - add const in various char pointers - requested_section in modulesCollectInfo was actually not sds but char* - extract new string2d out of getDoubleFromObject for code reuse Add module API for
-
- 28 Jan, 2019 1 commit
-
-
Guy Benoish authored
The string representation of `long double` may take up to ~5000 chars (see PR #3745). Before this fix HINCRBYFLOAT would never overflow (since the string could not exceed 256 chars). Now it can.
-
- 11 Dec, 2018 2 commits
- 15 Nov, 2018 1 commit
-
-
Weiliang Li authored
fix comment typo in util.c
-
- 26 Oct, 2018 2 commits
-
-
David Carlier authored
-
David Carlier authored
timezone global is a linux-ism whereas it is a function under BSD. Here a helper to get the timezone value in a more portable manner.
-
- 24 Jul, 2018 2 commits
- 23 Jul, 2018 2 commits
- 21 Jul, 2018 1 commit
-
-
dsomeshwar authored
-
- 03 Jul, 2018 1 commit
-
-
Jack Drogon authored
-
- 05 Apr, 2018 1 commit
-
-
antirez authored
-
- 12 Dec, 2017 1 commit
-
-
nashe authored
-
- 10 May, 2016 1 commit
-
-
antirez authored
-
- 04 Nov, 2015 1 commit
-
-
antirez authored
During the refactoring needed for lazy free, specifically the conversion of t_hash from struct robj to plain SDS strings, HINCRBFLOAT was accidentally moved away from long doubles to doubles for internal processing of increments and formatting. The diminished precision created more obvious artifacts in the way small numbers are formatted once we convert from decimal number in radix 10 to double and back to its string in radix 10. By using more precision, we now have less surprising results at least with small numbers like "1.23", exactly like in the previous versions of Redis. See issue #2846.
-
- 01 Oct, 2015 4 commits
- 27 Feb, 2015 2 commits
- 12 Feb, 2015 1 commit
-
-
antirez authored
Related to PR #2357.
-
- 22 Jan, 2015 3 commits