- 02 Jan, 2022 1 commit
-
-
Viktor Söderqvist authored
To avoid data loss, this commit adds a grace period for lagging replicas to catch up the replication offset. Done: * Wait for replicas when shutdown is triggered by SIGTERM and SIGINT. * Wait for replicas when shutdown is triggered by the SHUTDOWN command. A new blocked client type BLOCKED_SHUTDOWN is introduced, allowing multiple clients to call SHUTDOWN in parallel. Note that they don't expect a response unless an error happens and shutdown is aborted. * Log warning for each replica lagging behind when finishing shutdown. * CLIENT_PAUSE_WRITE while waiting for replicas. * Configurable grace period 'shutdown-timeout' in seconds (default 10). * New flags for the SHUTDOWN command: - NOW disables the grace period for lagging replicas. - FORCE ignores errors writing the RDB or AOF files which would normally prevent a shutdown. - ABORT cancels ongoing shutdown. Can't be combined with other flags. * New field in the output of the INFO command: 'shutdown_in_milliseconds'. The value is the remaining maximum time to wait for lagging replicas before finishing the shutdown. This field is present in the Server section **only** during shutdown. Not directly related: * When shutting down, if there is an AOF saving child, it is killed **even** if AOF is disabled. This can happen if BGREWRITEAOF is used when AOF is off. * Client pause now has end time and type (WRITE or ALL) per purpose. The different pause purposes are *CLIENT PAUSE command*, *failover* and *shutdown*. If clients are unpaused for one purpose, it doesn't affect client pause for other purposes. For example, the CLIENT UNPAUSE command doesn't affect client pause initiated by the failover or shutdown procedures. A completed failover or a failed shutdown doesn't unpause clients paused by the CLIENT PAUSE command. Notes: * DEBUG RESTART doesn't wait for replicas. * We already have a warning logged when a replica disconnects. This means that if any replica connection is lost during the shutdown, it is either logged as disconnected or as lagging at the time of exit. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 19 Dec, 2021 2 commits
-
-
Oran Agra authored
- add needs:debug flag for some tests - disable "save" in external tests (speedup?) - use debug_digest proc instead of debug command directly so it can be skipped - use OBJECT ENCODING instead of DEBUG OBJECT to get encoding - add a proc for OBJECT REFCOUNT so it can be skipped - move a bunch of tests in latency_monitor tests to happen later so that latency monitor has some values in it - add missing close_replication_stream calls - make sure to close the temp client if DEBUG LOG fails
-
YaacovHazan authored
Block sensitive configs and commands by default. * `enable-protected-configs` - block modification of configs with the new `PROTECTED_CONFIG` flag. Currently we add this flag to `dbfilename`, and `dir` configs, all of which are non-mutable configs that can set a file redis will write to. * `enable-debug-command` - block the `DEBUG` command * `enable-module-command` - block the `MODULE` command These have a default value set to `no`, so that these features are not exposed by default to client connections, and can only be set by modifying the config file. Users can change each of these to either `yes` (allow all access), or `local` (allow access from local TCP connections and unix domain connections) Note that this is a **breaking change** (specifically the part about MODULE command being disabled by default). I.e. we don't consider DEBUG command being blocked as an issue (people shouldn't have been using it), and the few configs we protected are unlikely to have been set at runtime anyway. On the other hand, it's likely to assume some users who use modules, load them from the config file anyway. Note that's the whole point of this PR, for redis to be more secure by default and reduce the attack surface on innocent users, so secure defaults will necessarily mean a breaking change.
-
- 29 Nov, 2021 1 commit
-
-
Binbin authored
Update CI so that warnings cause build failures. Also fix a warning in `test-sanitizer-address`: ``` In function ‘strncpy’, inlined from ‘clusterUpdateMyselfIp’ at cluster.c:545:13: /usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: error: ‘__builtin_strncpy’ specified bound 46 equals destination size [-Werror=stringop-truncation] 106 | return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest)); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ cc1: all warnings being treated as errors ```
-
- 28 Nov, 2021 1 commit
-
-
Meir Shpilraien (Spielrein) authored
msgpack lib missed using lua_checkstack and so on rare cases overflow the stack by at most 2 elements. This is a violation of the Lua C API. Notice that Lua allocates additional 5 more elements on top of lua->stack_last so Redis does not access an invalid memory. But it is an API violation and we should avoid it. This PR also added a new Lua compilation option. The new option can be enable using environment variable called LUA_DEBUG. If set to `yes` (by default `no`), Lua will be compiled without optimizations and with debug symbols (`-O0 -g`). In addition, in this new mode, Lua will be compiled with the `-DLUA_USE_APICHECK` flag that enables extended Lua C API validations. In addition, set LUA_DEBUG=yes on daily valgrind flow so we will be able to catch Lua C API violations in the future.
-
- 21 Nov, 2021 1 commit
-
-
Ozan Tezcan authored
specifically the ziplist and listpack unit tests and benchmarks run for too long with address sanitizer and --accurate
-
- 18 Nov, 2021 1 commit
-
-
Binbin authored
So that we can print server log on test failure.
-
- 16 Nov, 2021 1 commit
-
-
Oran Agra authored
Add --accurate to unit tests (new feature recently added) Add --no-latency to valgrind run (was present only for modules) add --no-latency to macos and freebsd runs (was not present for modules) add --timeout to freebsd (same one we have for valgrind)
-
- 11 Nov, 2021 3 commits
-
-
Ozan Tezcan authored
Was impossible to run the daily CI from private forks due to "redis/redis" repo check. Let's disable that check for manual triggers.
-
Ozan Tezcan authored
- Added sanitizer support. `address`, `undefined` and `thread` sanitizers are available. - To build Redis with desired sanitizer : `make SANITIZER=undefined` - There were some sanitizer findings, cleaned up codebase - Added tests with address and undefined behavior sanitizers to daily CI. - Added tests with address sanitizer to the per-PR CI (smoke out mem leaks sooner). Basically, there are three types of issues : **1- Unaligned load/store** : Most probably, this issue may cause a crash on a platform that does not support unaligned access. Redis does unaligned access only on supported platforms. **2- Signed integer overflow.** Although, signed overflow issue can be problematic time to time and change how compiler generates code, current findings mostly about signed shift or simple addition overflow. For most platforms Redis can be compiled for, this wouldn't cause any issue as far as I can tell (checked generated code on godbolt.org). **3 -Minor leak** (redis-cli), **use-after-free**(just before calling exit()); UB means nothing guaranteed and risky to reason about program behavior but I don't think any of the fixes here worth backporting. As sanitizers are now part of the CI, preventing new issues will be the real benefit.
-
yoav-steinberg authored
On test failure store the external redis server logs as CI artifacts so we can review them. Write test name to server log for external server tests. This is attempted and silently failed in case external server doesn't support it. Note that in non-external server mode we use a more robust method of writing to the log which doesn't depend on the server actually running/working. This isn't possible for externl servers and required for some complex tests which are skipped in external mode anyway. Cleanup: remove dup code.
-
- 10 Nov, 2021 2 commits
-
-
Oran Agra authored
First, avoid using --accurate on the freebsd CI, we only care about systematic issues there due to being different platform, but not accuracy Secondly, when looking at the test which timed out it seems silly and outdated: - it used KEYS to attempt to trigger lazy expiry, but KEYS doesn't do that anymore. - it used some hard coded sleeps rather than waiting for things to happen and exiting ASAP
-
Oran Agra authored
We saw some tests sporadically time out on valgrind (namely the ones from #9323). Increasing valgrind timeout from 20 mins to 40 mins in CI. And fixing an outdated help message.
-
- 09 Nov, 2021 1 commit
-
-
YaacovHazan authored
In both tests, "diskless loading short read" and "diskless loading short read with module", the timeout of waiting for the replica to respond to a short read and log it, is too short. Also, add --dump-logs in runtest-moduleapi for valgrind runs.
-
- 03 Nov, 2021 1 commit
-
-
perryitay authored
Redis lists are stored in quicklist, which is currently a linked list of ziplists. Ziplists are limited to storing elements no larger than 4GB, so when bigger items are added they're getting truncated. This PR changes quicklists so that they're capable of storing large items in quicklist nodes that are plain string buffers rather than ziplist. As part of the PR there were few other changes in redis: 1. new DEBUG sub-commands: - QUICKLIST-PACKED-THRESHOLD - set the threshold of for the node type to be plan or ziplist. default (1GB) - QUICKLIST <key> - Shows low level info about the quicklist encoding of <key> 2. rdb format change: - A new type was added - RDB_TYPE_LIST_QUICKLIST_2 . - container type (packed / plain) was added to the beginning of the rdb object (before the actual node list). 3. testing: - Tests that requires over 100MB will be by default skipped. a new flag was added to 'runtest' to run the large memory tests (not used by default) Co-authored-by:
sundb <sundbcn@gmail.com> Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 16 Oct, 2021 1 commit
-
-
Yossi Gottlieb authored
* Fix test modules linking on macOS 11.x. * Use macOS 10.x for FreeBSD VM as VirtualBox is not yet supported on 11.
-
- 11 Aug, 2021 1 commit
-
-
Yossi Gottlieb authored
Making sure Redis builds properly on older compiler is important given the wide range of systems it is built for. So far Ubuntu 16.04 has been used for this purpose, but as it's getting phased out we'll move to `oldoldstable` Debian as an "old system" precursor.
-
- 21 Jul, 2021 1 commit
-
-
Huang Zhw authored
On 32 bit platform, the bit position of GETBIT/SETBIT/BITFIELD/BITCOUNT,BITPOS may overflow (see CVE-2021-32761) (#9191) GETBIT, SETBIT may access wrong address because of wrap. BITCOUNT and BITPOS may return wrapped results. BITFIELD may access the wrong address but also allocate insufficient memory and segfault (see CVE-2021-32761). This commit uses `uint64_t` or `long long` instead of `size_t`. related https://github.com/redis/redis/pull/8096 At 32bit platform: > setbit bit 4294967295 1 (integer) 0 > config set proto-max-bulk-len 536870913 OK > append bit "\xFF" (integer) 536870913 > getbit bit 4294967296 (integer) 0 When the bit index is larger than 4294967295, size_t can't hold bit index. In the past, `proto-max-bulk-len` is limit to 536870912, so there is no problem. After this commit, bit position is stored in `uint64_t` or `long long`. So when `proto-max-bulk-len > 536870912`, 32bit platforms can still be correct. For 64bit platform, this problem still exists. The major reason is bit pos 8 times of byte pos. When proto-max-bulk-len is very larger, bit pos may overflow. But at 64bit platform, we don't have so long string. So this bug may never happen. Additionally this commit add a test cost `512MB` memory which is tag as `large-memory`. Make freebsd ci and valgrind ci ignore this test.
-
- 18 Jul, 2021 1 commit
-
-
Binbin authored
fix spell checker CI after recently merged old PR
-
- 05 Jul, 2021 2 commits
- 24 Jun, 2021 1 commit
-
-
Oran Agra authored
The freebsd and macos jobs were still broken. also add a few more skip jobs options.
-
- 23 Jun, 2021 2 commits
-
-
Oran Agra authored
-
Oran Agra authored
The daily CI was broken by #9119 seems that for cron scheduled tasks, these ifs aren't evaluated to false. But also it turns out that workflow_dispatch is only able to run CI on branches in the main repo (not on PRs). this is an attempt to overcome that by being able to checkout from any repo we want.
-
- 22 Jun, 2021 1 commit
-
-
Oran Agra authored
Hopefully now we'll be able to manually trigger extensive tests to PR branches without modifying them.
-
- 10 Jun, 2021 1 commit
-
-
Binbin authored
This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes. This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything, but at least it is also unlikely to cause false positives. Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use. Here's a summary of other changes: 1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces). 2. Outdated function / variable / argument names in comments 3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString. 4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751 5. Some outdated https link URLs. 6. Fix some outdated comment. Such as: - In README: about the rdb, we used to said create a `thread`, change to `process` - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey) - notifyKeyspaceEvent fucntion comment (add type arg) - Some others minor fix in comment (Most of them are incorrectly quoted by variable names) 7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
-
- 09 Jun, 2021 1 commit
-
-
Yossi Gottlieb authored
This commit revives the improves the ability to run the test suite against external servers, instead of launching and managing `redis-server` processes as part of the test fixture. This capability existed in the past, using the `--host` and `--port` options. However, it was quite limited and mostly useful when running a specific tests. Attempting to run larger chunks of the test suite experienced many issues: * Many tests depend on being able to start and control `redis-server` themselves, and there's no clear distinction between external server compatible and other tests. * Cluster mode is not supported (resulting with `CROSSSLOT` errors). This PR cleans up many things and makes it possible to run the entire test suite against an external server. It also provides more fine grained controls to handle cases where the external server supports a subset of the Redis commands, limited number of databases, cluster mode, etc. The tests directory now contains a `README.md` file that describes how this works. This commit also includes additional cleanups and fixes: * Tests can now be tagged. * Tag-based selection is now unified across `start_server`, `tags` and `test`. * More information is provided about skipped or ignored tests. * Repeated patterns in tests have been extracted to common procedures, both at a global level and on a per-test file basis. * Cleaned up some cases where test setup was based on a previous test executing (a major anti-pattern that repeats itself in many places). * Cleaned up some cases where test teardown was not part of a test (in the future we should have dedicated teardown code that executes even when tests fail). * Fixed some tests that were flaky running on external servers.
-
- 27 Apr, 2021 1 commit
-
-
yoav-steinberg authored
Specifically we had issues with NTP sync failure which was resolved here: https://github.com/vmactions/freebsd-vm/commit/457af7345642e154a79d219971a2d4a7c7fe2118
-
- 26 Apr, 2021 1 commit
-
-
Guy Korland authored
-
- 25 Apr, 2021 1 commit
-
-
Oran Agra authored
-
- 10 Mar, 2021 1 commit
-
-
sundb authored
1. Add `redis-server test all` support to run all tests. 2. Add redis test to daily ci. 3. Add `--accurate` option to run slow tests for more iterations (so that by default we run less cycles (shorter time, and less prints). 4. Move dict benchmark to REDIS_TEST. 5. fix some leaks in tests 6. make quicklist tests run on a specific fill set of options rather than huge ranges 7. move some prints in quicklist test outside their loops to reduce prints 8. removing sds.h from dict.c since it is now used in both redis-server and redis-cli (uses hiredis sds)
-
- 09 Mar, 2021 1 commit
-
-
Harkrishn Patro authored
Co-authored-by:
Harkrishn Patro <harkrisp@amazon.com>
-
- 25 Feb, 2021 1 commit
-
-
Yossi Gottlieb authored
* Add better control of malloc_usable_size() usage. * Use malloc_usable_size on alpine libc daily job. * Add no-malloc-usable-size daily jobs. * Fix zmalloc(0) when HAVE_MALLOC_SIZE is undefined. In order to align with the jemalloc behavior, this should never return NULL or OOM panic.
-
- 23 Feb, 2021 2 commits
-
-
Yossi Gottlieb authored
Also adds a new daily CI test, relying on the fact that we don't use malloc_size() on alpine libmusl. Fixes #8531
-
Yossi Gottlieb authored
* Remove linux/version.h dependency. This introduces unnecessary dependencies, and generally not a good idea as the platform we build on may be different than the platform we run on. To determine if sync_file_range exists we can simply rely on header file hints. * Fix setproctitle() on libmusl. The previous ifdef checks were a bit too strict for no apparent reason. * Fix tests failure on Linux with no backtrace. * Add alpine daily CI job.
-
- 10 Feb, 2021 1 commit
-
-
filipe oliveira authored
- removes time sensitive checks from block on background tests during leak checks. - fix uninitialized variable on RedisModuleBlockedClient() when calling RM_BlockedClientMeasureTimeEnd() without RM_BlockedClientMeasureTimeStart()
-
- 07 Feb, 2021 2 commits
-
-
Oran Agra authored
Github started shifting some repositoreis to use ubuntu 20.04 by default tcl8.5 is missing in these, but 8.6 exists in both 20.04 and 18.04
-
Yossi Gottlieb authored
Dump the entire server log if a test failed, to easy troubleshooting with no access to log files.
-
- 03 Feb, 2021 1 commit
-
-
Yossi Gottlieb authored
* Add bash temporarily to allow sentinel fd leaks test to run. * Use vmactions-freebsd rdist sync to work around bind permission denied and slow execution issues. * Upgrade to tcl8.6 to be aligned with latest Ubuntu envs. * Concat all command executions to avoid ignoring failures. * Skip intensive fuzzer on FreeBSD. For some yet unknown reason, generate_fuzzy_traffic_on_key causes TCL to significantly bloat on FreeBSD resulting with out of memory.
-
- 20 Jan, 2021 1 commit
-
-
Guy Korland authored
Co-authored-by:
Oran Agra <oran@redislabs.com>
-