1. 15 Sep, 2024 1 commit
  2. 10 Jul, 2023 1 commit
  3. 18 Jun, 2023 1 commit
    • Oran Agra's avatar
      Set Jemalloc --disable-cache-oblivious to reduce memory overhead (#12315) · 07c14672
      Oran Agra authored
      Apparently for large size classes Jemalloc allocate some extra
      memory (can be up to 25% overhead for allocations of 16kb).
      see https://github.com/jemalloc/jemalloc/issues/1098#issuecomment-1589870476
      
      p.s. from Redis's perspective that looks like external fragmentation,
      (i.e. allocated bytes will be low, and active pages bytes will be large)
      which  can cause active-defrag to eat CPU cycles in vain.
      
      Some details about this mechanism we disable:
      ---------------------------------------------------------------
      Disabling this mechanism only affects large allocations (above 16kb)
      Not only that it isn't expected to cause any performance regressions,
      it's actually recommended, unless you have a specific workload pattern
      and hardware that benefit from this feature -- by default it's enabled and
      adds address randomization to all large buffers, by over allocating 1 page
      per large size class, and offsetting into that page to make the starting
      address of the user buffer randomized. Workloads such as scientific
      computation often handle multiple big matrixes at the same time, and the
      randomization makes sure that the cacheline level accesses don't suffer
      bad conflicts (when they all start from page-aligned addresses).
      
      However the downsize is also quite noticeable, like you observed that extra
      page per large size can cause memory overhead, plus the extra TLB entry.
      The other factor is, hardware in the last few years started doing the
      randomization at the hardware level, i.e. the address to cacheline mapping isn't
      a direct mapping anymore. So there's debate to disable the randomization by default,
      but we are still hesitant because when it matters, it could matter a lot, and having
      it enabled by default limits that worst case behavior, even though it means the
      majority of workloads suffers a regression.
      
      So in short, it's safe and offers better performance in most cases.
      07c14672
  4. 01 May, 2023 1 commit
    • Oran Agra's avatar
      Upgrade to jemalloc 5.3.0 · 0897c8af
      Oran Agra authored
      * Regenerate configure script sccording to deps/README
      * update iget_defrag_hint by following changes to arena_dalloc_no_tcache
      0897c8af
  5. 02 Feb, 2023 1 commit
  6. 15 Oct, 2022 1 commit
    • filipe oliveira's avatar
      optimizing d2string() and addReplyDouble() with grisu2: double to string... · 29380ff7
      filipe oliveira authored
      optimizing d2string() and addReplyDouble() with grisu2: double to string conversion based on Florian Loitsch's Grisu-algorithm (#10587)
      
      All commands / use cases that heavily rely on double to a string representation conversion,
      (e.g. meaning take a double-precision floating-point number like 1.5 and return a string like "1.5" ),
      could benefit from a performance boost by swapping snprintf(buf,len,"%.17g",value) by the
      equivalent [fpconv_dtoa](https://github.com/night-shift/fpconv) or any other algorithm that ensures
      100% coverage of conversion.
      
      This is a well-studied topic and Projects like MongoDB. RedPanda, PyTorch leverage libraries
      ( fmtlib ) that use the optimized double to string conversion underneath.
      
      
      The positive impact can be substantial. This PR uses the grisu2 approach ( grisu explained on
      https://www.cs.tufts.edu/~nr/cs257/archive/florian-loitsch/printf.pdf section 5 ). 
      
      test suite changes:
      Despite being compatible, in some cases it produces a different result from printf, and some tests
      had to be adjusted.
      one case is that `%.17g` (which means %e or %f which ever is shorter), chose to use `5000000000`
      instead of 5e+9, which sounds like a bug?
      In other cases, we changed TCL to compare numbers instead of strings to ignore minor rounding
      issues (`expr 0.8 == 0.79999999999999999`) 
      29380ff7
  7. 23 Aug, 2022 1 commit
    • Oran Agra's avatar
      Build TLS as a loadable module · 4faddf18
      Oran Agra authored
      
      
      * Support BUILD_TLS=module to be loaded as a module via config file or
        command line. e.g. redis-server --loadmodule redis-tls.so
      * Updates to redismodule.h to allow it to be used side by side with
        server.h by defining REDISMODULE_CORE_MODULE
      * Changes to server.h, redismodule.h and module.c to avoid repeated
        type declarations (gcc 4.8 doesn't like these)
      * Add a mechanism for non-ABI neutral modules (ones who include
        server.h) to refuse loading if they detect not being built together with
        redis (release.c)
      * Fix wrong signature of RedisModuleDefragFunc, this could break
        compilation of a module, but not the ABI
      * Move initialization of listeners in server.c to be after loading
        the modules
      * Config TLS after initialization of listeners
      * Init cluster after initialization of listeners
      * Add TLS module to CI
      * Fix a test suite race conditions:
        Now that the listeners are initialized later, it's not sufficient to
        wait for the PID message in the log, we need to wait for the "Server
        Initialized" message.
      * Fix issues with moduleconfigs test as a result from start_server
        waiting for "Server Initialized"
      * Fix issues with modules/infra test as a result of an additional module
        present
      
      Notes about Sentinel:
      Sentinel can't really rely on the tls module, since it uses hiredis to
      initiate connections and depends on OpenSSL (won't be able to use any
      other connection modules for that), so it was decided that when TLS is
      built as a module, sentinel does not support TLS at all.
      This means that it keeps using redis_tls_ctx and redis_tls_client_ctx directly.
      
      Example code of config in redis-tls.so(may be use in the future):
      RedisModuleString *tls_cfg = NULL;
      
      void tlsInfo(RedisModuleInfoCtx *ctx, int for_crash_report) {
          UNUSED(for_crash_report);
          RedisModule_InfoAddSection(ctx, "");
          RedisModule_InfoAddFieldLongLong(ctx, "var", 42);
      }
      
      int tlsCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
      {
          if (argc != 2) return RedisModule_WrongArity(ctx);
          return RedisModule_ReplyWithString(ctx, argv[1]);
      }
      
      RedisModuleString *getStringConfigCommand(const char *name, void *privdata) {
          REDISMODULE_NOT_USED(name);
          REDISMODULE_NOT_USED(privdata);
          return tls_cfg;
      }
      
      int setStringConfigCommand(const char *name, RedisModuleString *new, void *privdata, RedisModuleString **err) {
          REDISMODULE_NOT_USED(name);
          REDISMODULE_NOT_USED(err);
          REDISMODULE_NOT_USED(privdata);
          if (tls_cfg) RedisModule_FreeString(NULL, tls_cfg);
          RedisModule_RetainString(NULL, new);
          tls_cfg = new;
          return REDISMODULE_OK;
      }
      
      int RedisModule_OnLoad(void *ctx, RedisModuleString **argv, int argc)
      {
          ....
          if (RedisModule_CreateCommand(ctx,"tls",tlsCommand,"",0,0,0) == REDISMODULE_ERR)
              return REDISMODULE_ERR;
      
          if (RedisModule_RegisterStringConfig(ctx, "cfg", "", REDISMODULE_CONFIG_DEFAULT, getStringConfigCommand, setStringConfigCommand, NULL, NULL) == REDISMODULE_ERR)
              return REDISMODULE_ERR;
      
          if (RedisModule_LoadConfigs(ctx) == REDISMODULE_ERR) {
              if (tls_cfg) {
                  RedisModule_FreeString(ctx, tls_cfg);
                  tls_cfg = NULL;
              }
              return REDISMODULE_ERR;
          }
          ...
      }
      Co-authored-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      4faddf18
  8. 28 Nov, 2021 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Fix Lua C API violation on lua msgpack lib. (#9832) · a8c1253b
      Meir Shpilraien (Spielrein) authored
      msgpack lib missed using lua_checkstack and so on rare
      cases overflow the stack by at most 2 elements. This is a
      violation of the Lua C API. Notice that Lua allocates
      additional 5 more elements on top of lua->stack_last
      so Redis does not access an invalid memory. But it is an
      API violation and we should avoid it.
      
      This PR also added a new Lua compilation option. The new
      option can be enable using environment variable called
      LUA_DEBUG. If set to `yes` (by default `no`), Lua will be
      compiled without optimizations and with debug symbols (`-O0 -g`).
      In addition, in this new mode, Lua will be compiled with the
      `-DLUA_USE_APICHECK` flag that enables extended Lua C API
      validations.
      
      In addition, set LUA_DEBUG=yes on daily valgrind flow so we
      will be able to catch Lua C API violations in the future.
      a8c1253b
  9. 24 Oct, 2021 1 commit
  10. 10 Oct, 2021 1 commit
  11. 29 Sep, 2020 1 commit
    • YoongHM's avatar
      Fix compilation warnings in Lua and jemalloc dependencies (#7785) · 448c435b
      YoongHM authored
      
      
      - The argument `u` in for `ar` is ignored (and generates warnings since `D` became the default.
        All it does is avoid updating unchanged objects (shouldn't have any impact on our build)
      - Enable `LUA_USE_MKSTEMP` to force the use of `mkstemp()` instead of `tmpname()` (which is dead
        code in redis anyway).
      - Remove unused variable `c` in `f_parser()`
      - Removed misleadingly indented space in `luaL_loadfile()` and ``addfield()`
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      448c435b
  12. 21 Sep, 2020 1 commit
    • YoongHM's avatar
      Fix warning from jemalloc configure script (#7790) · d3faed87
      YoongHM authored
      jemalloc configure shows this:
          configure: WARNING: unrecognized options: --enable-cc-silence
      
      The changelog of jemalloc 4.0 has:
        - Replace --enable-cc-silence with --disable-cc-silence to suppress spurious
          warnings by default.
      d3faed87
  13. 25 Aug, 2020 1 commit
  14. 07 Oct, 2019 1 commit
    • Yossi Gottlieb's avatar
      TLS: Connections refactoring and TLS support. · b087dd1d
      Yossi Gottlieb authored
      * Introduce a connection abstraction layer for all socket operations and
      integrate it across the code base.
      * Provide an optional TLS connections implementation based on OpenSSL.
      * Pull a newer version of hiredis with TLS support.
      * Tests, redis-cli updates for TLS support.
      b087dd1d
  15. 24 May, 2018 1 commit
  16. 06 Jul, 2016 1 commit
    • antirez's avatar
      geohash.c and geohash_helper.c are part of Redis. · eaa713e9
      antirez authored
      They were under /deps since they originate from a different source tree,
      however at this point they are very modified and we took ownership of
      both the files making changes, fixing bugs, so there is no upgrade path
      from the original code tree.
      
      Given that, better to move the code under /src with proper dependencies
      and with a more simpler editing experience.
      eaa713e9
  17. 18 May, 2016 1 commit
  18. 22 Jun, 2015 1 commit
    • Matt Stancliff's avatar
      [In-Progress] Add Geo Commands · 7f4ac3d1
      Matt Stancliff authored
      Current todo:
        - replace functions in zset.{c,h} with a new unified Redis
          zset access API.
      
      Once we get the zset interface fixed, we can squash
      relevant commits in this branch and have one nice commit
      to merge into unstable.
      
      This commit adds:
        - Geo commands
        - Tests; runnable with: ./runtest --single unit/geo
        - Geo helpers in deps/geohash-int/
        - src/geo.{c,h} and src/geojson.{c,h} implementing geo commands
        - Updated build configurations to get everything working
        - TEMPORARY: src/zset.{c,h} implementing zset score and zset
          range reading without writing to client output buffers.
        - Modified linkage of one t_zset.c function for use in zset.c
      
      Conflicts:
      	src/Makefile
      	src/redis.c
      7f4ac3d1
  19. 02 Jan, 2015 1 commit
  20. 14 Nov, 2014 1 commit
    • Matt Stancliff's avatar
      Lua: Upgrade cjson to 2.1.0 (2012-03-01) · 4fdcd213
      Matt Stancliff authored
      The new cjson has some improvements over our current version including
      increased platform compatability, a new resource limit to restrict
      decode depth, and better invalid number handling.
      
      One minor change was required to deps/Makefile because this version
      of cjson doesn't export itself globally, so we added a quick little
      define of -DENABLE_CJSON_GLOBAL.
      
      cjson now has an optional higher performing float parsing interface,
      but we are not including it (g_fmt.c, dtoa.c) because it requires
      endianness declaration during compile time.
      
      This commit is exactly lua_cjson.c from 2.1.0 with one minor
      change of altering the two Lua includes for local search
      instead of system-wide importing.
      4fdcd213
  21. 12 Aug, 2014 1 commit
  22. 16 Mar, 2013 1 commit
  23. 01 Nov, 2012 1 commit
    • antirez's avatar
      32 bit build fixed on Linux. · d36fb95a
      antirez authored
      It failed because of the way jemalloc was compiled (without passing the
      right flags to make, but just to configure). Now the same set of flags
      are also passed to the make command, fixing the issue.
      
      This fixes issue #744
      d36fb95a
  24. 11 Apr, 2012 1 commit
    • Pieter Noordhuis's avatar
      Clean up Makefiles · 0a08d2b0
      Pieter Noordhuis authored
      Remove unused variables. Instead of overriding non-standard variables
      such as ARCH and PROF, use standard variables CFLAGS and LDFLAGS to
      override Makefile settings. Move dependencies generated by `make dep` to
      a separate file.
      0a08d2b0
  25. 15 Nov, 2011 1 commit
    • Pieter Noordhuis's avatar
      Rebuild deps/ and src/ when ARCH changes · 4b8a6394
      Pieter Noordhuis authored
      This change moves the build instructions for dependencies to a separate
      Makefile in deps/. The ARCH environment variable is stored in a
      .make-arch file in the same directory as the Makefile. The contents of
      this file is read and compared to the current ARCH, and, on a mismatch
      triggers rebuilding the entire source tree.
      
      When file .make-arch exists and matches with ARCH from the environment,
      the dependencies are assumed to already be built.
      
      The new "clean" target only cleans the Redis source tree, not its
      dependencies. To clear the dependencies as well, the "distclean" target
      can be used.
      4b8a6394