Unverified Commit 8a86bca5 authored by Yossi Gottlieb's avatar Yossi Gottlieb Committed by GitHub
Browse files

Improve test suite to handle external servers better. (#9033)

This commit revives the improves the ability to run the test suite against
external servers, instead of launching and managing `redis-server` processes as
part of the test fixture.

This capability existed in the past, using the `--host` and `--port` options.
However, it was quite limited and mostly useful when running a specific tests.
Attempting to run larger chunks of the test suite experienced many issues:

* Many tests depend on being able to start and control `redis-server` themselves,
and there's no clear distinction between external server compatible and other
tests.
* Cluster mode is not supported (resulting with `CROSSSLOT` errors).

This PR cleans up many things and makes it possible to run the entire test suite
against an external server. It also provides more fine grained controls to
handle cases where the external server supports a subset of the Redis commands,
limited number of databases, cluster mode, etc.

The tests directory now contains a `README.md` file that describes how this
works.

This commit also includes additional cleanups and fixes:

* Tests can now be tagged.
* Tag-based selection is now unified across `start_server`, `tags` and `test`.
* More information is provided about skipped or ignored tests.
* Repeated patterns in tests have been extracted to common procedures, both at a
  global level and on a per-test file basis.
* Cleaned up some cases where test setup was based on a previous test executing
  (a major anti-pattern that repeats itself in many places).
* Cleaned up some cases where test teardown was not part of a test (in the
  future we should have dedicated teardown code that executes even when tests
  fail).
* Fixed some tests that were flaky running on external servers.
parent c396fd91
......@@ -15,7 +15,7 @@ jobs:
- name: test
run: |
sudo apt-get install tcl8.6 tclx
./runtest --verbose
./runtest --verbose --tags -slow
- name: module api test
run: ./runtest-moduleapi --verbose
......
name: External Server Tests
on:
pull_request:
push:
schedule:
- cron: '0 0 * * *'
jobs:
test-external-standalone:
runs-on: ubuntu-latest
timeout-minutes: 14400
steps:
- uses: actions/checkout@v2
- name: Build
run: make REDIS_CFLAGS=-Werror
- name: Start redis-server
run: ./src/redis-server --daemonize yes
- name: Run external test
run: |
./runtest \
--host 127.0.0.1 --port 6379 \
--tags -slow
test-external-cluster:
runs-on: ubuntu-latest
timeout-minutes: 14400
steps:
- uses: actions/checkout@v2
- name: Build
run: make REDIS_CFLAGS=-Werror
- name: Start redis-server
run: ./src/redis-server --cluster-enabled yes --daemonize yes
- name: Create a single node cluster
run: ./src/redis-cli cluster addslots $(for slot in {0..16383}; do echo $slot; done); sleep 5
- name: Run external test
run: |
./runtest \
--host 127.0.0.1 --port 6379 \
--cluster-mode \
--tags -slow
Redis Test Suite
================
The normal execution mode of the test suite involves starting and manipulating
local `redis-server` instances, inspecting process state, log files, etc.
The test suite also supports execution against an external server, which is
enabled using the `--host` and `--port` parameters. When executing against an
external server, tests tagged `external:skip` are skipped.
There are additional runtime options that can further adjust the test suite to
match different external server configurations:
| Option | Impact |
| -------------------- | -------------------------------------------------------- |
| `--singledb` | Only use database 0, don't assume others are supported. |
| `--ignore-encoding` | Skip all checks for specific encoding. |
| `--ignore-digest` | Skip key value digest validations. |
| `--cluster-mode` | Run in strict Redis Cluster compatibility mode. |
Tags
----
Tags are applied to tests to classify them according to the subsystem they test,
but also to indicate compatibility with different run modes and required
capabilities.
Tags can be applied in different context levels:
* `start_server` context
* `tags` context that bundles several tests together
* A single test context.
The following compatibility and capability tags are currently used:
| Tag | Indicates |
| --------------------- | --------- |
| `external:skip` | Not compatible with external servers. |
| `cluster:skip` | Not compatible with `--cluster-mode`. |
| `needs:repl` | Uses replication and needs to be able to `SYNC` from server. |
| `needs:debug` | Uses the `DEBUG` command or other debugging focused commands (like `OBJECT`). |
| `needs:pfdebug` | Uses the `PFDEBUG` command. |
| `needs:config-maxmemory` | Uses `CONFIG SET` to manipulate memory limit, eviction policies, etc. |
| `needs:config-resetstat` | Uses `CONFIG RESETSTAT` to reset statistics. |
| `needs:reset` | Uses `RESET` to reset client connections. |
| `needs:save` | Uses `SAVE` to create an RDB file. |
When using an external server (`--host` and `--port`), filtering using the
`external:skip` tags is done automatically.
When using `--cluster-mode`, filtering using the `cluster:skip` tag is done
automatically.
In addition, it is possible to specify additional configuration. For example, to
run tests on a server that does not permit `SYNC` use:
./runtest --host <host> --port <port> --tags -needs:repl
......@@ -22,7 +22,7 @@ proc start_server_aof {overrides code} {
kill_server $srv
}
tags {"aof"} {
tags {"aof external:skip"} {
## Server can start when aof-load-truncated is set to yes and AOF
## is truncated, with an incomplete MULTI block.
create_aof {
......
......@@ -11,7 +11,7 @@ proc stop_bg_block_op {handle} {
catch {exec /bin/kill -9 $handle}
}
start_server {tags {"repl"}} {
start_server {tags {"repl" "external:skip"}} {
start_server {} {
set master [srv -1 client]
set master_host [srv -1 host]
......
tags {"external:skip"} {
# Copy RDB with zipmap encoded hash to server path
set server_path [tmpdir "server.convert-zipmap-hash-on-load"]
......@@ -33,3 +35,5 @@ start_server [list overrides [list "dir" $server_path "dbfilename" "hash-zipmap.
assert_match {v1 v2} [r hmget hash f1 f2]
}
}
}
# tests of corrupt ziplist payload with valid CRC
tags {"dump" "corruption"} {
tags {"dump" "corruption" "external:skip"} {
# catch sigterm so that in case one of the random command hangs the test,
# usually due to redis not putting a response in the output buffers,
......
......@@ -6,7 +6,7 @@
# * some tests set sanitize-dump-payload to no and some to yet, depending on
# what we want to test
tags {"dump" "corruption"} {
tags {"dump" "corruption" "external:skip"} {
set corrupt_payload_7445 "\x0E\x01\x1D\x1D\x00\x00\x00\x16\x00\x00\x00\x03\x00\x00\x04\x43\x43\x43\x43\x06\x04\x42\x42\x42\x42\x06\x3F\x41\x41\x41\x41\xFF\x09\x00\x88\xA5\xCA\xA8\xC5\x41\xF4\x35"
......
start_server {tags {"failover"}} {
start_server {tags {"failover external:skip"}} {
start_server {} {
start_server {} {
set node_0 [srv 0 client]
......
tags {"external:skip"} {
set system_name [string tolower [exec uname -s]]
set system_supported 0
......@@ -49,3 +51,5 @@ if {$system_supported} {
}
}
}
......@@ -5,7 +5,7 @@
# We keep these tests just because they reproduce edge cases in the replication
# logic in hope they'll be able to spot some problem in the future.
start_server {tags {"psync2"}} {
start_server {tags {"psync2 external:skip"}} {
start_server {} {
# Config
set debug_msg 0 ; # Enable additional debug messages
......@@ -74,7 +74,7 @@ start_server {} {
}}
start_server {tags {"psync2"}} {
start_server {tags {"psync2 external:skip"}} {
start_server {} {
start_server {} {
start_server {} {
......@@ -180,7 +180,7 @@ start_server {} {
}
}}}}}
start_server {tags {"psync2"}} {
start_server {tags {"psync2 external:skip"}} {
start_server {} {
start_server {} {
......
......@@ -4,7 +4,7 @@
# redis-benchmark. At the end we check that the data is the same
# everywhere.
start_server {tags {"psync2"}} {
start_server {tags {"psync2 external:skip"}} {
start_server {} {
start_server {} {
# Config
......
......@@ -70,7 +70,7 @@ proc show_cluster_status {} {
}
}
start_server {tags {"psync2"}} {
start_server {tags {"psync2 external:skip"}} {
start_server {} {
start_server {} {
start_server {} {
......
tags {"rdb"} {
tags {"rdb external:skip"} {
set server_path [tmpdir "server.rdb-encoding-test"]
......
......@@ -5,7 +5,7 @@ proc cmdstat {cmd} {
return [cmdrstat $cmd r]
}
start_server {tags {"benchmark network"}} {
start_server {tags {"benchmark network external:skip"}} {
start_server {} {
set master_host [srv 0 host]
set master_port [srv 0 port]
......
source tests/support/cli.tcl
if {$::singledb} {
set ::dbnum 0
} else {
set ::dbnum 9
}
start_server {tags {"cli"}} {
proc open_cli {{opts "-n 9"} {infile ""}} {
proc open_cli {{opts ""} {infile ""}} {
if { $opts == "" } {
set opts "-n $::dbnum"
}
set ::env(TERM) dumb
set cmdline [rediscli [srv host] [srv port] $opts]
if {$infile ne ""} {
......@@ -65,7 +74,7 @@ start_server {tags {"cli"}} {
}
proc _run_cli {opts args} {
set cmd [rediscli [srv host] [srv port] [list -n 9 {*}$args]]
set cmd [rediscli [srv host] [srv port] [list -n $::dbnum {*}$args]]
foreach {key value} $opts {
if {$key eq "pipe"} {
set cmd "sh -c \"$value | $cmd\""
......@@ -269,7 +278,7 @@ start_server {tags {"cli"}} {
assert_match "OK" [r config set repl-diskless-sync yes]
assert_match "OK" [r config set repl-diskless-sync-delay 0]
test_redis_cli_rdb_dump
}
} {} {needs:repl}
test "Scan mode" {
r flushdb
......@@ -302,13 +311,18 @@ start_server {tags {"cli"}} {
}
close_cli $fd
}
} {} {needs:repl}
test "Piping raw protocol" {
set cmds [tmpfile "cli_cmds"]
set cmds_fd [open $cmds "w"]
set cmds_count 2101
if {!$::singledb} {
puts $cmds_fd [formatCommand select 9]
incr cmds_count
}
puts $cmds_fd [formatCommand del test-counter]
for {set i 0} {$i < 1000} {incr i} {
......@@ -326,7 +340,7 @@ start_server {tags {"cli"}} {
set output [read_cli $cli_fd]
assert_equal {1000} [r get test-counter]
assert_match {*All data transferred*errors: 0*replies: 2102*} $output
assert_match "*All data transferred*errors: 0*replies: ${cmds_count}*" $output
file delete $cmds
}
......
start_server {tags {"repl"}} {
start_server {tags {"repl external:skip"}} {
start_server {} {
test {First server should have role slave after SLAVEOF} {
r -1 slaveof [srv 0 host] [srv 0 port]
......
start_server {tags {"repl"}} {
start_server {tags {"repl external:skip"}} {
start_server {} {
test {First server should have role slave after SLAVEOF} {
r -1 slaveof [srv 0 host] [srv 0 port]
......@@ -45,7 +45,7 @@ start_server {tags {"repl"}} {
}
}
start_server {tags {"repl"}} {
start_server {tags {"repl external:skip"}} {
start_server {} {
test {First server should have role slave after SLAVEOF} {
r -1 slaveof [srv 0 host] [srv 0 port]
......
start_server {tags {"repl network"}} {
start_server {tags {"repl network external:skip"}} {
start_server {} {
set master [srv -1 client]
......@@ -39,7 +39,7 @@ start_server {tags {"repl network"}} {
}
}
start_server {tags {"repl"}} {
start_server {tags {"repl external:skip"}} {
start_server {} {
set master [srv -1 client]
set master_host [srv -1 host]
......@@ -85,7 +85,7 @@ start_server {tags {"repl"}} {
}
}
start_server {tags {"repl"}} {
start_server {tags {"repl external:skip"}} {
start_server {} {
set master [srv -1 client]
set master_host [srv -1 host]
......
......@@ -117,6 +117,7 @@ proc test_psync {descr duration backlog_size backlog_ttl delay cond mdl sdl reco
}
}
tags {"external:skip"} {
foreach mdl {no yes} {
foreach sdl {disabled swapdb} {
test_psync {no reconnection, just sync} 6 1000000 3600 0 {
......@@ -139,3 +140,4 @@ foreach mdl {no yes} {
} $mdl $sdl 1
}
}
}
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment