1. 31 May, 2022 1 commit
    • DarrenJiang13's avatar
      Adds isolated netstats for replication. (#10062) · bb1de082
      DarrenJiang13 authored
      
      
      The amount of `server.stat_net_output_bytes/server.stat_net_input_bytes`
      is actually the sum of replication flow and users' data flow. 
      It may cause confusions like this:
      "Why does my server get such a large output_bytes while I am doing nothing? ". 
      
      After discussions and revisions, now here is the change about what this
      PR brings (final version before merge):
      - 2 server variables to count the network bytes during replication,
           including fullsync and propagate bytes.
           - `server.stat_net_repl_output_bytes`/`server.stat_net_repl_input_bytes`
      - 3 info fields to print the input and output of repl bytes and instantaneous
           value of total repl bytes.
           - `total_net_repl_input_bytes` / `total_net_repl_output_bytes`
           - `instantaneous_repl_total_kbps`
      - 1 new API `rioCheckType()` to check the type of rio. So we can use this
           to distinguish between diskless and diskbased replication
      - 2 new counting items to keep network statistics consistent between master
           and slave
          - rdb portion during diskless replica. in `rdbLoadProgressCallback()`
          - first line of the full sync payload. in `readSyncBulkPayload()`
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      bb1de082
  2. 31 Dec, 2021 1 commit
  3. 04 Nov, 2021 1 commit
  4. 30 Aug, 2021 1 commit
    • Wang Yuan's avatar
      Use sync_file_range to optimize fsync if possible (#9409) · 9a0c0617
      Wang Yuan authored
      We implement incremental data sync in rio.c by call fsync, on slow disk, that may cost a lot of time,
      sync_file_range could provide async fsync, so we could serialize key/value and sync file data at the same time.
      
      > one tip for sync_file_range usage: http://lkml.iu.edu/hypermail/linux/kernel/1005.2/01845.html
      
      Additionally, this change avoids a single large write to be used, which can result in a mass of dirty
      pages in the kernel (increasing the risk of someone else's write to block).
      
      On HDD, current solution could reduce approximate half of dumping RDB time,
      this PR costs 50s for dump 7.7G rdb but unstable branch costs 93s.
      On NVME SSD, this PR can't reduce much time,  this PR costs 40s, unstable branch costs 48s.
      
      Moreover, I find calling data sync every 4MB is better than 32MB.
      9a0c0617
  5. 30 Jul, 2021 1 commit
  6. 10 Jun, 2021 1 commit
    • Binbin's avatar
      Fixed some typos, add a spell check ci and others minor fix (#8890) · 0bfccc55
      Binbin authored
      This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes.
      This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything,
      but at least it is also unlikely to cause false positives.
      
      Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use.
      
      Here's a summary of other changes:
      1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces).
      2. Outdated function / variable / argument names in comments
      3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString.
      4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751
      5. Some outdated https link URLs.
      6. Fix some outdated comment. Such as:
          - In README: about the rdb, we used to said create a `thread`, change to `process`
          - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey)
          - notifyKeyspaceEvent fucntion comment (add type arg)
          - Some others minor fix in comment (Most of them are incorrectly quoted by variable names)
      7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
      0bfccc55
  7. 01 Apr, 2021 1 commit
    • Wang Yuan's avatar
      Handle remaining fsync errors (#8419) · 1eb85249
      Wang Yuan authored
      In `aof.c`, we call fsync when stop aof, and now print a log to let user know that if fail.
      In `cluster.c`, we now return error, the calling function already handles these write errors.
      In `redis-cli.c`, users hope to save rdb, we now print a message if fsync failed.
      In `rio.c`, we now treat fsync errors like we do for write errors. 
      In `server.c`, we try to fsync aof file when shutdown redis, we only can print one log if fail.
      In `bio.c`, if failing to fsync aof file, we will set `aof_bio_fsync_status` to error , and reject writing just like last writing aof error,  moreover also set INFO command field `aof_last_write_status` to error.
      1eb85249
  8. 14 Mar, 2021 1 commit
  9. 24 Jul, 2020 1 commit
  10. 23 Jul, 2020 1 commit
    • Oran Agra's avatar
      Fix harmless bug in rioConnRead (#7557) · 40d7fca3
      Oran Agra authored
      this code is in use only if the master is disk-based, and the replica is
      diskless. In this case we use a buffered reader, but we must avoid reading
      past the rdb file, into the command stream. which Luckly rdb.c doesn't
      really attempt to do (it knows how much it should read).
      
      When rioConnRead detects that the extra buffering attempt reaches beyond
      the read limit it should read less, but if the caller actually requested
      more, then it should return with an error rather than a short read. the
      bug would have resulted in short read.
      
      in order to fix it, the code must consider the real requested size, and
      not the extra buffering size.
      40d7fca3
  11. 07 Oct, 2019 2 commits
    • Oran Agra's avatar
      diskless replication rdb transfer uses pipe, and writes to sockets form the parent process. · 5a477946
      Oran Agra authored
      misc:
      - handle SSL_has_pending by iterating though these in beforeSleep, and setting timeout of 0 to aeProcessEvents
      - fix issue with epoll signaling EPOLLHUP and EPOLLERR only to the write handlers. (needed to detect the rdb pipe was closed)
      - add key-load-delay config for testing
      - trim connShutdown which is no longer needed
      - rioFdsetWrite -> rioFdWrite - simplified since there's no longer need to write to multiple FDs
      - don't detect rdb child exited (don't call wait3) until we detect the pipe is closed
      - Cleanup bad optimization from rio.c, add another one
      5a477946
    • Yossi Gottlieb's avatar
      TLS: Connections refactoring and TLS support. · b087dd1d
      Yossi Gottlieb authored
      * Introduce a connection abstraction layer for all socket operations and
      integrate it across the code base.
      * Provide an optional TLS connections implementation based on OpenSSL.
      * Pull a newer version of hiredis with TLS support.
      * Tests, redis-cli updates for TLS support.
      b087dd1d
  12. 17 Jul, 2019 1 commit
  13. 10 Jul, 2019 1 commit
  14. 08 Jul, 2019 3 commits
    • antirez's avatar
      dfcbeaf1
    • antirez's avatar
      81b18fa3
    • Oran Agra's avatar
      diskless replication on slave side (don't store rdb to file), plus some other related fixes · 2de544cf
      Oran Agra authored
      The implementation of the diskless replication was currently diskless only on the master side.
      The slave side was still storing the received rdb file to the disk before loading it back in and parsing it.
      
      This commit adds two modes to load rdb directly from socket:
      1) when-empty
      2) using "swapdb"
      the third mode of using diskless slave by flushdb is risky and currently not included.
      
      other changes:
      --------------
      distinguish between aof configuration and state so that we can re-enable aof only when sync eventually
      succeeds (and not when exiting from readSyncBulkPayload after a failed attempt)
      also a CONFIG GET and INFO during rdb loading would have lied
      
      When loading rdb from the network, don't kill the server on short read (that can be a network error)
      
      Fix rdb check when performed on preamble AOF
      
      tests:
      run replication tests for diskless slave too
      make replication test a bit more aggressive
      Add test for diskless load swapdb
      2de544cf
  15. 15 Mar, 2018 1 commit
  16. 29 Dec, 2017 1 commit
    • Oran Agra's avatar
      fix processing of large bulks (above 2GB) · 60a4f12f
      Oran Agra authored
      - protocol parsing (processMultibulkBuffer) was limitted to 32big positions in the buffer
        readQueryFromClient potential overflow
      - rioWriteBulkCount used int, although rioWriteBulkString gave it size_t
      - several places in sds.c that used int for string length or index.
      - bugfix in RM_SaveAuxField (return was 1 or -1 and not length)
      - RM_SaveStringBuffer was limitted to 32bit length
      60a4f12f
  17. 25 Apr, 2016 1 commit
  18. 27 Jul, 2015 1 commit
  19. 26 Jul, 2015 2 commits
  20. 22 Oct, 2014 1 commit
  21. 17 Oct, 2014 3 commits
  22. 14 Oct, 2014 1 commit
    • antirez's avatar
      rio.c fdset target: tolerate (and report) a subset of FDs in error. · 2a436aae
      antirez authored
      Fdset target is used when we want to write an RDB file directly to
      slave's sockets. In this setup as long as there is a single slave that
      is still receiving our payload, we want to continue sennding instead of
      aborting. However rio calls should abort of no FD is ok.
      
      Also we want the errors reported so that we can signal the parent who is
      ok and who is broken, so there is a new set integers with the state of
      each fd. Zero is ok, non-zero is the errno of the failure, if avaialble,
      or a generic EIO.
      2a436aae
  23. 10 Oct, 2014 2 commits
  24. 22 Jan, 2014 1 commit
    • antirez's avatar
      Use fflush() before fsync() in rio.c. · 172f14d4
      antirez authored
      Incremental flushing in rio.c is only used to avoid huge kernel buffers
      synched to slow disks creating big latency spikes, so this fix has no
      durability implications, however it is certainly more correct to make
      sure that the FILE buffers are flushed to the kernel before calling
      fsync on the file descriptor.
      
      Thanks to Li Shao Kai for reporting this issue in the Redis mailing
      list.
      172f14d4
  25. 16 Jul, 2013 1 commit
  26. 24 Apr, 2013 1 commit
  27. 29 Nov, 2012 1 commit
    • antirez's avatar
      Introduced the Build ID in INFO and --version output. · 2f62c966
      antirez authored
      The idea is to be able to identify a build in a unique way, so for
      instance after a bug report we can recognize that the build is the one
      of a popular Linux distribution and perform the debugging in the same
      environment.
      2f62c966
  28. 08 Nov, 2012 1 commit
  29. 09 Apr, 2012 3 commits
  30. 23 Oct, 2011 1 commit
  31. 22 Sep, 2011 1 commit