1. 31 Jan, 2023 1 commit
    • uriyage's avatar
      Optimization: sdsRemoveFreeSpace to avoid realloc on noop (#11766) · 46393f98
      uriyage authored
      
      
      In #7875 (Redis 6.2), we changed the sds alloc to be the usable allocation
      size in order to:
      
      > reduce the need for realloc calls by making the sds implicitly take over
      the internal fragmentation
      
      This change was done most sds functions, excluding `sdsRemoveFreeSpace` and
      `sdsResize`, the reason is that in some places (e.g. clientsCronResizeQueryBuffer)
      we call sdsRemoveFreeSpace when we see excessive free space and want to trim it.
      so if we don't trim it exactly to size, the caller may still see excessive free space and
      call it again and again.
      
      However, this resulted in some excessive calls to realloc, even when there's no need
      and it's gonna be a no-op (e.g. when reducing 15 bytes allocation to 13).
      
      It turns out that a call for realloc with jemalloc can be expensive even if it ends up
      doing nothing, so this PR adds a check using `je_nallocx`, which is cheap to avoid
      the call for realloc.
      
      in addition to that this PR unifies sdsResize and sdsRemoveFreeSpace into common
      code. the difference between them was that sdsResize would avoid using SDS_TYPE_5,
      since it want to keep the string ready to be resized again, while sdsRemoveFreeSpace
      would permit using SDS_TYPE_5 and get an optimal memory consumption.
      now both methods take a `would_regrow` argument that makes it more explicit.
      
      the only actual impact of that is that in clientsCronResizeQueryBuffer we call both sdsResize
      and sdsRemoveFreeSpace for in different cases, and we now prevent the use of SDS_TYPE_5 in both.
      
      The new test that was added to cover this concern used to pass before this PR as well,
      this PR is just a performance optimization and cleanup.
      
      Benchmark:
      `redis-benchmark -c 100 -t set  -d 512 -P 10  -n  100000000`
      on i7-9850H with jemalloc, shows improvement from 1021k ops/sec to 1067k (average of 3 runs).
      some 4.5% improvement.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      46393f98
  2. 18 Jan, 2022 1 commit
    • Yossi Gottlieb's avatar
      Fix additional AOF filename issues. (#10110) · 25e6d4d4
      Yossi Gottlieb authored
      This extends the previous fix (#10049) to address any form of
      non-printable or whitespace character (including newlines, quotes,
      non-printables, etc.)
      
      Also, removes the limitation on appenddirname, to align with the way
      filenames are handled elsewhere in Redis.
      25e6d4d4
  3. 16 Nov, 2021 1 commit
  4. 05 Jul, 2021 1 commit
    • Oran Agra's avatar
      Query buffer shrinking improvements (#5013) · ec582cc7
      Oran Agra authored
      
      
      when tracking the peak, don't reset the peak to 0, reset it to the
      maximum of the current used, and the planned to be used by the current
      arg.
      
      when shrining, split the two separate conditions.
      the idle time shrinking will remove all free space.
      but the peak based shrinking will keep room for the current arg.
      
      when we resize due to a peak (rahter than idle time), don't trim all
      unused space, let the qbuf keep a size that's sufficient for the
      currently process bulklen, and the current peak.
      Co-authored-by: default avatarsundb <sundbcn@gmail.com>
      Co-authored-by: default avataryoav-steinberg <yoav@monfort.co.il>
      ec582cc7
  5. 22 Jun, 2021 1 commit
    • Oran Agra's avatar
      Adjustments to recent RM_StringTruncate fix (#3718) (#9125) · ae418eca
      Oran Agra authored
      - Introduce a new sdssubstr api as a building block for sdsrange.
        The API of sdsrange is many times hard to work with and also has
        corner case that cause bugs. sdsrange is easy to work with and also
        simplifies the implementation of sdsrange.
      - Revert the fix to RM_StringTruncate and just use sdssubstr instead of
        sdsrange.
      - Solve valgrind warnings from the new tests introduced by the previous
        PR.
      ae418eca
  6. 15 Jun, 2021 1 commit
    • sundb's avatar
      Fix the wrong reisze of querybuf (#9003) · e5d8a5eb
      sundb authored
      The initialize memory of `querybuf` is `PROTO_IOBUF_LEN(1024*16) * 2` (due to sdsMakeRoomFor being greedy), under `jemalloc`, the allocated memory will be 40k.
      This will most likely result in the `querybuf` being resized when call `clientsCronResizeQueryBuffer` unless the client requests it fast enough.
      
      Note that this bug existed even before #7875, since the condition for resizing includes the sds headers (32k+6).
      
      ## Changes
      1. Use non-greedy sdsMakeRoomFor when allocating the initial query buffer (of 16k).
      1. Also use non-greedy allocation when working with BIG_ARG (we won't use that extra space anyway)
      2. in case we did use a greedy allocation, read as much as we can into the buffer we got (including internal frag), to reduce system calls.
      3. introduce a dedicated constant for the shrinking (same value as before)
      3. Add test for querybuf.
      4. improve a maxmemory test by ignoring the effect of replica query buffers (can accumulate many ACKs on slow env)
      5. improve a maxmemory by disabling slowlog (it will cause slight memory growth on slow env).
      e5d8a5eb
  7. 10 Mar, 2021 1 commit
    • sundb's avatar
      Add run all test support with define REDIS_TEST (#8570) · 95d6297d
      sundb authored
      1. Add `redis-server test all` support to run all tests.
      2. Add redis test to daily ci.
      3. Add `--accurate` option to run slow tests for more iterations (so that
         by default we run less cycles (shorter time, and less prints).
      4. Move dict benchmark to REDIS_TEST.
      5. fix some leaks in tests
      6. make quicklist tests run on a specific fill set of options rather than huge ranges
      7. move some prints in quicklist test outside their loops to reduce prints
      8. removing sds.h from dict.c since it is now used in both redis-server and
         redis-cli (uses hiredis sds)
      95d6297d
  8. 28 Jan, 2021 1 commit
  9. 06 Dec, 2020 1 commit
  10. 21 Dec, 2019 1 commit
  11. 29 Dec, 2017 1 commit
    • Oran Agra's avatar
      fix processing of large bulks (above 2GB) · 60a4f12f
      Oran Agra authored
      - protocol parsing (processMultibulkBuffer) was limitted to 32big positions in the buffer
        readQueryFromClient potential overflow
      - rioWriteBulkCount used int, although rioWriteBulkString gave it size_t
      - several places in sds.c that used int for string length or index.
      - bugfix in RM_SaveAuxField (return was 1 or -1 and not length)
      - RM_SaveStringBuffer was limitted to 32bit length
      60a4f12f
  12. 23 Feb, 2017 1 commit
  13. 17 Nov, 2015 1 commit
    • antirez's avatar
      Lua debugger: use sds_malloc() to allocate eval cli array. · e57cccde
      antirez authored
      Redis-cli handles the debugger "eval" command in a special way since
      sdssplitargs() would not be ok: we need to send the Redis debugger the
      whole Lua script without any parsing. However in order to later free the
      argument vector inside redis-cli using just sdsfreesplitres(), we need
      to allocate the array of SDS pointers using the same allocator SDS is
      using, that may differ to what Redis is using.
      
      So now a newer version of SDS exports sds_malloc() and other allocator
      functions to give access, to the program it is linked to, the allocator
      used internally by SDS.
      e57cccde
  14. 25 Jul, 2015 4 commits
  15. 15 Jul, 2015 1 commit
    • antirez's avatar
      SDS: New sds type 5 implemented. · 0ab27a45
      antirez authored
      This is an attempt to use the refcount feature of the sds.c fork
      provided in the Pull Request #2509. A new type, SDS_TYPE_5 is introduced
      having a one byte header with just the string length, without
      information about the available additional length at the end of the
      string (this means that sdsMakeRoomFor() will be required each time
      we want to append something, since the string will always report to have
      0 bytes available).
      
      More work needed in order to avoid common SDS functions will pay the
      cost of this type. For example both sdscatprintf() and sdscatfmt()
      should try to upgrade to SDS_TYPE_8 ASAP when appending chars.
      0ab27a45
  16. 14 Jul, 2015 1 commit
  17. 08 Jan, 2015 1 commit
  18. 02 Jan, 2015 1 commit
    • Matt Stancliff's avatar
      Add sdsnative() · e1619772
      Matt Stancliff authored
      Use the existing memory space for an SDS to convert it to a regular
      character buffer so we don't need to allocate duplicate space just
      to extract a usable buffer for native operations.
      e1619772
  19. 23 Dec, 2014 1 commit
    • Matt Stancliff's avatar
      Allow all code tests to run using Redis args · 8febcffd
      Matt Stancliff authored
      Previously, many files had individual main() functions for testing,
      but each required being compiled with their own testing flags.
      That gets difficult when you have 8 different flags you need
      to set just to run all tests (plus, some test files required
      other files to be compiled aaginst them, and it seems some didn't
      build at all without including the rest of Redis).
      
      Now all individual test main() funcions are renamed to a test
      function for the file itself and one global REDIS_TEST define enables
      testing across the entire codebase.
      
      Tests can now be run with:
        - `./redis-server test <test>`
      
        e.g. ./redis-server test ziplist
      
      If REDIS_TEST is not defined, then no tests get included and no
      tests are included in the final redis-server binary.
      8febcffd
  20. 11 Dec, 2014 2 commits
  21. 13 Aug, 2014 1 commit
  22. 28 Apr, 2014 1 commit
  23. 12 Aug, 2013 1 commit
  24. 24 Jul, 2013 1 commit
    • antirez's avatar
      sdsrange() does not need to return a value. · 6ea8e094
      antirez authored
      Actaully the string is modified in-place and a reallocation is never
      needed, so there is no need to return the new sds string pointer as
      return value of the function, that is now just "void".
      6ea8e094
  25. 04 Jul, 2013 1 commit
  26. 06 Mar, 2013 1 commit
  27. 30 Mar, 2012 1 commit
  28. 14 Mar, 2012 1 commit
  29. 16 Jan, 2012 1 commit
  30. 21 Nov, 2011 1 commit
  31. 02 Nov, 2011 1 commit
  32. 13 Sep, 2011 1 commit
  33. 25 May, 2011 1 commit
  34. 05 May, 2011 1 commit
  35. 07 Apr, 2011 1 commit
  36. 10 Dec, 2010 1 commit