1. 20 Jun, 2023 1 commit
    • Wen Hui's avatar
      adding geo command edge cases tests (#12274) · 66ea178c
      Wen Hui authored
      For geosearch and georadius we have already test coverage for wrong type, but we dont have for geodist, geohash, geopos commands. So adding the wrong type test cases for geodist, geohash, geopos commands.
      
      Existing code, we have verify_geo_edge_response_bymember function for wrong type test cases which has member as an option. But the function is being called in other test cases where the output is not inline with these commnds(geodist, geohash, geopos). So I could not include these commands(geodist, geohash, geopos) as part of existing function, hence implemented a new function verify_geo_edge_response_generic and called from the test case.
      66ea178c
  2. 11 Mar, 2023 1 commit
    • guybe7's avatar
      Add reply_schema to command json files (internal for now) (#10273) · 4ba47d2d
      guybe7 authored
      Work in progress towards implementing a reply schema as part of COMMAND DOCS, see #9845
      Since ironing the details of the reply schema of each and every command can take a long time, we
      would like to merge this PR when the infrastructure is ready, and let this mature in the unstable branch.
      Meanwhile the changes of this PR are internal, they are part of the repo, but do not affect the produced build.
      
      ### Background
      In #9656 we add a lot of information about Redis commands, but we are missing information about the replies
      
      ### Motivation
      1. Documentation. This is the primary goal.
      2. It should be possible, based on the output of COMMAND, to be able to generate client code in typed
        languages. In order to do that, we need Redis to tell us, in detail, what each reply looks like.
      3. We would like to build a fuzzer that verifies the reply structure (for now we use the existing
        testsuite, see the "Testing" section)
      
      ### Schema
      The idea is to supply some sort of schema for the various replies of each command.
      The schema will describe the conceptual structure of the reply (for generated clients), as defined in RESP3.
      Note that the reply structure itself may change, depending on the arguments (e.g. `XINFO STREAM`, with
      and without the `FULL` modifier)
      We decided to use the standard json-schema (see https://json-schema.org/) as the reply-schema.
      
      Example for `BZPOPMIN`:
      ```
      "reply_schema": {
          "oneOf": [
              {
                  "description": "Timeout reached and no elements were popped.",
                  "type": "null"
              },
              {
                  "description": "The keyname, popped member, and its score.",
                  "type": "array",
                  "minItems": 3,
                  "maxItems": 3,
                  "items": [
                      {
                          "description": "Keyname",
                          "type": "string"
                      },
                      {
                          "description": "Member",
                          "type": "string"
                      },
                      {
                          "description": "Score",
                          "type": "number"
                      }
                  ]
              }
          ]
      }
      ```
      
      #### Notes
      1.  It is ok that some commands' reply structure depends on the arguments and it's the caller's responsibility
        to know which is the relevant one. this comes after looking at other request-reply systems like OpenAPI,
        where the reply schema can also be oneOf and the caller is responsible to know which schema is the relevant one.
      2. The reply schemas will describe RESP3 replies only. even though RESP3 is structured, we want to use reply
        schema for documentation (and possibly to create a fuzzer that validates the replies)
      3. For documentation, the description field will include an explanation of the scenario in which the reply is sent,
        including any relation to arguments. for example, for `ZRANGE`'s two schemas we will need to state that one
        is with `WITHSCORES` and the other is without.
      4. For documentation, there will be another optional field "notes" in which we will add a short description of
        the representation in RESP2, in case it's not trivial (RESP3's `ZRANGE`'s nested array vs. RESP2's flat
        array, for example)
      
      Given the above:
      1. We can generate the "return" section of all commands in [redis-doc](https://redis.io/commands/)
        (given that "description" and "notes" are comprehensive enough)
      2. We can generate a client in a strongly typed language (but the return type could be a conceptual
        `union` and the caller needs to know which schema is relevant). see the section below for RESP2 support.
      3. We can create a fuzzer for RESP3.
      
      ### Limitations (because we are using the standard json-schema)
      The problem is that Redis' replies are more diverse than what the json format allows. This means that,
      when we convert the reply to a json (in order to validate the schema against it), we lose information (see
      the "Testing" section below).
      The other option would have been to extend the standard json-schema (and json format) to include stuff
      like sets, bulk-strings, error-string, etc. but that would mean also extending the schema-validator - and that
      seemed like too much work, so we decided to compromise.
      
      Examples:
      1. We cannot tell the difference between an "array" and a "set"
      2. We cannot tell the difference between simple-string and bulk-string
      3. we cannot verify true uniqueness of items in commands like ZRANGE: json-schema doesn't cover the
        case of two identical members with different scores (e.g. `[["m1",6],["m1",7]]`) because `uniqueItems`
        compares (member,score) tuples and not just the member name. 
      
      ### Testing
      This commit includes some changes inside Redis in order to verify the schemas (existing and future ones)
      are indeed correct (i.e. describe the actual response of Redis).
      To do that, we added a debugging feature to Redis that causes it to produce a log of all the commands
      it executed and their replies.
      For that, Redis needs to be compiled with `-DLOG_REQ_RES` and run with
      `--reg-res-logfile <file> --client-default-resp 3` (the testsuite already does that if you run it with
      `--log-req-res --force-resp3`)
      You should run the testsuite with the above args (and `--dont-clean`) in order to make Redis generate
      `.reqres` files (same dir as the `stdout` files) which contain request-response pairs.
      These files are later on processed by `./utils/req-res-log-validator.py` which does:
      1. Goes over req-res files, generated by redis-servers, spawned by the testsuite (see logreqres.c)
      2. For each request-response pair, it validates the response against the request's reply_schema
        (obtained from the extended COMMAND DOCS)
      5. In order to get good coverage of the Redis commands, and all their different replies, we chose to use
        the existing redis test suite, rather than attempt to write a fuzzer.
      
      #### Notes about RESP2
      1. We will not be able to use the testing tool to verify RESP2 replies (we are ok with that, it's time to
        accept RESP3 as the future RESP)
      2. Since the majority of the test suite is using RESP2, and we want the server to reply with RESP3
        so that we can validate it, we will need to know how to convert the actual reply to the one expected.
         - number and boolean are always strings in RESP2 so the conversion is easy
         - objects (maps) are always a flat array in RESP2
         - others (nested array in RESP3's `ZRANGE` and others) will need some special per-command
           handling (so the client will not be totally auto-generated)
      
      Example for ZRANGE:
      ```
      "reply_schema": {
          "anyOf": [
              {
                  "description": "A list of member elements",
                  "type": "array",
                  "uniqueItems": true,
                  "items": {
                      "type": "string"
                  }
              },
              {
                  "description": "Members and their scores. Returned in case `WITHSCORES` was used.",
                  "notes": "In RESP2 this is returned as a flat array",
                  "type": "array",
                  "uniqueItems": true,
                  "items": {
                      "type": "array",
                      "minItems": 2,
                      "maxItems": 2,
                      "items": [
                          {
                              "description": "Member",
                              "type": "string"
                          },
                          {
                              "description": "Score",
                              "type": "number"
                          }
                      ]
                  }
              }
          ]
      }
      ```
      
      ### Other changes
      1. Some tests that behave differently depending on the RESP are now being tested for both RESP,
        regardless of the special log-req-res mode ("Pub/Sub PING" for example)
      2. Update the history field of CLIENT LIST
      3. Added basic tests for commands that were not covered at all by the testsuite
      
      ### TODO
      
      - [x] (maybe a different PR) add a "condition" field to anyOf/oneOf schemas that refers to args. e.g.
        when `SET` return NULL, the condition is `arguments.get||arguments.condition`, for `OK` the condition
        is `!arguments.get`, and for `string` the condition is `arguments.get` - https://github.com/redis/redis/issues/11896
      - [x] (maybe a different PR) also run `runtest-cluster` in the req-res logging mode
      - [x] add the new tests to GH actions (i.e. compile with `-DLOG_REQ_RES`, run the tests, and run the validator)
      - [x] (maybe a different PR) figure out a way to warn about (sub)schemas that are uncovered by the output
        of the tests - https://github.com/redis/redis/issues/11897
      - [x] (probably a separate PR) add all missing schemas
      - [x] check why "SDOWN is triggered by misconfigured instance replying with errors" fails with --log-req-res
      - [x] move the response transformers to their own file (run both regular, cluster, and sentinel tests - need to
        fight with the tcl including mechanism a bit)
      - [x] issue: module API - https://github.com/redis/redis/issues/11898
      - [x] (probably a separate PR): improve schemas: add `required` to `object`s - https://github.com/redis/redis/issues/11899
      
      Co-authored-by: default avatarOzan Tezcan <ozantezcan@gmail.com>
      Co-authored-by: default avatarHanna Fadida <hanna.fadida@redislabs.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarShaya Potter <shaya@redislabs.com>
      4ba47d2d
  3. 15 Dec, 2022 1 commit
  4. 04 Dec, 2022 1 commit
  5. 01 Nov, 2022 1 commit
  6. 14 Apr, 2022 1 commit
    • Madelyn Olson's avatar
      Fix incorrect error code for eval scripts and fix test error checking (#10575) · effa707e
      Madelyn Olson authored
      By the convention of errors, there is supposed to be a space between the code and the name.
      While looking at some lua stuff I noticed that interpreter errors were not adding the space,
      so some clients will try to map the detailed error message into the error.
      
      We have tests that hit this condition, but they were just checking that the string "starts" with ERR.
      I updated some other tests with similar incorrect string checking. This isn't complete though, as
      there are other ways we check for ERR I didn't fix.
      
      Produces some fun output like:
      ```
      # Errorstats
      errorstat_ERR:count=1
      errorstat_ERRuser_script_1_:count=1
      ```
      effa707e
  7. 21 Feb, 2022 1 commit
  8. 01 Aug, 2021 1 commit
    • Binbin's avatar
      GEO* STORE with empty src key delete the dest key and return 0, not empty array (#9271) · 86555ae0
      Binbin authored
      
      
      With an empty src key, we need to deal with two situations:
      1. non-STORE: We should return emptyarray.
      2. STORE: Try to delete the store key and return 0.
      
      This applies to both GEOSEARCHSTORE (new to v6.2), and
      also GEORADIUS STORE (which was broken since forever)
      
      This pr try to fix #9261. i.e. both STORE variants would have behaved
      like the non-STORE variants when the source key was missing,
      returning an empty array and not deleting the destination key,
      instead of returning 0, and deleting the destination key.
      
      Also add more tests for some commands.
      - GEORADIUS: wrong type src key, non existing src key, empty search,
        store with non existing src key, store with empty search
      - GEORADIUSBYMEMBER: wrong type src key, non existing src key,
        non existing member, store with non existing src key
      - GEOSEARCH: wrong type src key, non existing src key, empty search,
        frommember with non existing member
      - GEOSEARCHSTORE: wrong type key, non existing src key,
        fromlonlat with empty search, frommember with non existing member
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      86555ae0
  9. 09 Jun, 2021 1 commit
    • Yossi Gottlieb's avatar
      Improve test suite to handle external servers better. (#9033) · 8a86bca5
      Yossi Gottlieb authored
      This commit revives the improves the ability to run the test suite against
      external servers, instead of launching and managing `redis-server` processes as
      part of the test fixture.
      
      This capability existed in the past, using the `--host` and `--port` options.
      However, it was quite limited and mostly useful when running a specific tests.
      Attempting to run larger chunks of the test suite experienced many issues:
      
      * Many tests depend on being able to start and control `redis-server` themselves,
      and there's no clear distinction between external server compatible and other
      tests.
      * Cluster mode is not supported (resulting with `CROSSSLOT` errors).
      
      This PR cleans up many things and makes it possible to run the entire test suite
      against an external server. It also provides more fine grained controls to
      handle cases where the external server supports a subset of the Redis commands,
      limited number of databases, cluster mode, etc.
      
      The tests directory now contains a `README.md` file that describes how this
      works.
      
      This commit also includes additional cleanups and fixes:
      
      * Tests can now be tagged.
      * Tag-based selection is now unified across `start_server`, `tags` and `test`.
      * More information is provided about skipped or ignored tests.
      * Repeated patterns in tests have been extracted to common procedures, both at a
        global level and on a per-test file basis.
      * Cleaned up some cases where test setup was based on a previous test executing
        (a major anti-pattern that repeats itself in many places).
      * Cleaned up some cases where test teardown was not part of a test (in the
        future we should have dedicated teardown code that executes even when tests
        fail).
      * Fixed some tests that were flaky running on external servers.
      8a86bca5
  10. 04 Feb, 2021 2 commits
    • Yang Bodong's avatar
      Fix GEOSEARCH tcl test error (#8451) · b7b23a0f
      Yang Bodong authored
      Issue with new test due to longitude wraparound.
      b7b23a0f
    • Yang Bodong's avatar
      GEOSEARCH bybox bug fixes and new fuzzy tester (#8445) · ded1655d
      Yang Bodong authored
      
      
      Fix errors of GEOSEARCH bybox search due to:
      1. projection of the box to a trapezoid (when the meter box is converted to long / lat it's no longer a box).
      2. width and height mismatch
      
      Changes:
      - New GEOSEARCH point in rectangle algorithm
      - Fix GEOSEARCH bybox width and height mismatch bug
      - Add GEOSEARCH bybox testing to the existing "GEOADD + GEORANGE randomized test"
      - Add new fuzzy test to stress test the bybox corners and edges
      - Add some tests for edge cases of the bybox algorithm
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ded1655d
  11. 08 Jan, 2021 1 commit
  12. 03 Jan, 2021 1 commit
  13. 14 Dec, 2020 1 commit
  14. 12 Dec, 2020 1 commit
    • 杨博东's avatar
      Add GEOSEARCH / GEOSEARCHSTORE commands (#8094) · 4d06d99b
      杨博东 authored
      Add commands to query geospatial data with bounding box.
      
      Two new commands that replace the existing 4 GEORADIUS* commands.
      
      GEOSEARCH key [FROMMEMBER member] [FROMLOC long lat] [BYRADIUS radius
      unit] [BYBOX width height unit] [WITHCORD] [WITHDIST] [WITHASH] [COUNT
      count] [ASC|DESC]
      
      GEOSEARCHSTORE dest_key src_key [FROMMEMBER member] [FROMLOC long lat]
      [BYRADIUS radius unit] [BYBOX width height unit] [WITHCORD] [WITHDIST]
      [WITHASH] [COUNT count] [ASC|DESC] [STOREDIST]
      
      - Add two types of CIRCULAR_TYPE and RECTANGLE_TYPE to achieve different searches
      - Judge whether the point is within the rectangle, refer to:
      geohashGetDistanceIfInRectangle
      4d06d99b
  15. 18 Dec, 2019 1 commit
  16. 08 Oct, 2019 1 commit
    • antirez's avatar
      Geo: output 10 chars of geohash, not 11. · 009862ab
      antirez authored
      This does not limit the actual precision, because the last digit bits were
      garbage, and the shift value became even negative in the last iteration.
      009862ab
  17. 15 Jan, 2019 1 commit
  18. 03 Jul, 2017 1 commit
    • antirez's avatar
      Fix GEORADIUS edge case with huge radius. · b2cd9fca
      antirez authored
      This commit closes issue #3698, at least for now, since the root cause
      was not fixed: the bounding box function, for huge radiuses, does not
      return a correct bounding box, there are points still within the radius
      that are left outside.
      
      So when using GEORADIUS queries with radiuses in the order of 5000 km or
      more, it was possible to see, at the edge of the area, certain points
      not correctly reported.
      
      Because the bounding box for now was used just as an optimization, and
      such huge radiuses are not common, for now the optimization is just
      switched off when the radius is near such magnitude.
      
      Three test cases found by the Continuous Integration test were added, so
      that we can easily trigger the bug again, both for regression testing
      and in order to properly fix it as some point in the future.
      b2cd9fca
  19. 20 Dec, 2016 1 commit
  20. 05 Dec, 2016 1 commit
    • antirez's avatar
      Geo: improve fuzz test. · b1fc06f7
      antirez authored
      The test now uses more diverse radius sizes, especially sizes near or
      greater the whole earth surface are used, that are known to trigger edge
      cases. Moreover the PRNG seeding was probably resulting into the same
      sequence tested over and over again, now seeding unsing the current unix
      time in milliseconds.
      
      Related to #3631.
      b1fc06f7
  21. 27 Jul, 2016 1 commit
    • antirez's avatar
      Multiple GEORADIUS bugs fixed. · 356a6304
      antirez authored
      By grepping the continuous integration errors log a number of GEORADIUS
      tests failures were detected.
      
      Fortunately when a GEORADIUS failure happens, the test suite logs enough
      information in order to reproduce the problem: the PRNG seed,
      coordinates and radius of the query.
      
      By reproducing the issues, three different bugs were discovered and
      fixed in this commit. This commit also improves the already good
      reporting of the fuzzer and adds the failure vectors as regression
      tests.
      
      The issues found:
      
      1. We need larger squares around the poles in order to cover the area
      requested by the user. There were already checks in order to use a
      smaller step (larger squares) but the limit set (+/- 67 degrees) is not
      enough in certain edge cases, so 66 is used now.
      
      2. Even near the equator, when the search area center is very near the
      edge of the square, the north, south, west or ovest square may not be
      able to fully cover the specified radius. Now a test is performed at the
      edge of the initial guessed search area, and larger squares are used in
      case the test fails.
      
      3. Because of rounding errors between Redis and Tcl, sometimes the test
      signaled false positives. This is now addressed.
      
      Whenever possible the original code was improved a bit in other ways. A
      debugging example stanza was added in order to make the next debugging
      session simpler when the next bug is found.
      356a6304
  22. 18 Feb, 2016 1 commit
  23. 06 Oct, 2015 1 commit
  24. 14 Sep, 2015 1 commit
    • antirez's avatar
      GEORADIUS: Don't report duplicates when radius is huge. · 3c23b5ff
      antirez authored
      Georadius works by computing the center + neighbors squares covering all
      the area of the specified position and radius. Then a distance filter is
      used to remove elements which are actually outside the range.
      
      When a huge radius is used, like 5000 km or more, adjacent neighbors may
      collide and be the same, leading to the reporting of the same element
      multiple times. This only happens in the edge case of huge radius but is
      not ideal.
      
      A robust but slow solution would involve qsorting the range to remove
      all the duplicates. However since the collisions are only in adjacent
      boxes, for the way they are ordered in the code, it is much faster to
      just check if the current box is the same as the previous one processed.
      
      This commit adds a regression test for the bug.
      
      Fixes #2767.
      3c23b5ff
  25. 09 Jul, 2015 1 commit
    • antirez's avatar
      GEOENCODE / GEODECODE commands removed. · b96af595
      antirez authored
      Rationale:
      
      1. The commands look like internals exposed without a real strong use
      case.
      2. Whatever there is an use case, the client would implement the
      commands client side instead of paying RTT just to use a simple to
      reimplement library.
      3. They add complexity to an otherwise quite straightforward API.
      
      So for now KILLED ;-)
      b96af595
  26. 29 Jun, 2015 4 commits
  27. 27 Jun, 2015 1 commit
  28. 26 Jun, 2015 1 commit
    • antirez's avatar
      Geo: from lat,lon API to lon,lat API according to GIS standard · fa9d62d3
      antirez authored
      The GIS standard and all the major DBs implementing GIS related
      functions take coordinates as x,y that is longitude,latitude.
      It was a bad start for Redis to do things differently, so even if this
      means that existing users of the Geo module will be required to change
      their code, Redis now conforms to the standard.
      
      Usually Redis is very backward compatible, but this is not an exception
      to this rule, since this is the first Geo implementation entering the
      official Redis source code. It is not wise to try to be backward
      compatible with code forks... :-)
      
      Close #2637.
      fa9d62d3
  29. 24 Jun, 2015 2 commits
    • antirez's avatar
      Geo: GEOHASH command test. · 5fd756bf
      antirez authored
      5fd756bf
    • antirez's avatar
      Geo: GEORADIUS fuzzy testing by reimplementing it in Tcl. · cf89a19f
      antirez authored
      We set random points in the world, pick a random position, and check if
      the returned points by Redis match the ones computed by Tcl by brute
      forcing all the points using the distance between two points formula.
      
      This approach is sounding since immediately resulted in finding a bug in
      the original implementation.
      cf89a19f
  30. 23 Jun, 2015 1 commit
  31. 22 Jun, 2015 3 commits
    • antirez's avatar
      Geo: fix tests after distance precision change · 575e247a
      antirez authored
      575e247a
    • antirez's avatar
      Geo: removed JSON failing test (false positive) · 73134f6a
      antirez authored
      Server output is matched to a pre-computed output. The last digits
      differ because of rouding errors.
      73134f6a
    • Matt Stancliff's avatar
      [In-Progress] Add Geo Commands · 7f4ac3d1
      Matt Stancliff authored
      Current todo:
        - replace functions in zset.{c,h} with a new unified Redis
          zset access API.
      
      Once we get the zset interface fixed, we can squash
      relevant commits in this branch and have one nice commit
      to merge into unstable.
      
      This commit adds:
        - Geo commands
        - Tests; runnable with: ./runtest --single unit/geo
        - Geo helpers in deps/geohash-int/
        - src/geo.{c,h} and src/geojson.{c,h} implementing geo commands
        - Updated build configurations to get everything working
        - TEMPORARY: src/zset.{c,h} implementing zset score and zset
          range reading without writing to client output buffers.
        - Modified linkage of one t_zset.c function for use in zset.c
      
      Conflicts:
      	src/Makefile
      	src/redis.c
      7f4ac3d1