Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
ruanhaishen
redis
Commits
6854c7b9
Commit
6854c7b9
authored
Jul 20, 2016
by
antirez
Browse files
LFU: make counter log factor and decay time configurable.
parent
6416ab19
Changes
5
Hide whitespace changes
Inline
Side-by-side
redis.conf
View file @
6854c7b9
...
...
@@ -492,7 +492,7 @@ slave-priority 100
############################## MEMORY MANAGEMENT ################################
#
Don't use more memory than
the specified amount of bytes.
#
Set a memory usage limit to
the specified amount of bytes.
# When the memory limit is reached Redis will try to remove keys
# according to the eviction policy selected (see maxmemory-policy).
#
...
...
@@ -501,8 +501,8 @@ slave-priority 100
# that would use more memory, like SET, LPUSH, and so on, and will continue
# to reply to read-only commands like GET.
#
# This option is usually useful when using Redis as an LRU cache, or to
set
# a hard memory limit for an instance (using the 'noeviction' policy).
# This option is usually useful when using Redis as an LRU
or LFU
cache, or to
#
set
a hard memory limit for an instance (using the 'noeviction' policy).
#
# WARNING: If you have slaves attached to an instance with maxmemory on,
# the size of the output buffers needed to feed the slaves are subtracted
...
...
@@ -520,12 +520,20 @@ slave-priority 100
# MAXMEMORY POLICY: how Redis will select what to remove when maxmemory
# is reached. You can select among five behaviors:
#
# volatile-lru -> remove the key with an expire set using an LRU algorithm
# allkeys-lru -> remove any key according to the LRU algorithm
# volatile-random -> remove a random key with an expire set
# allkeys-random -> remove a random key, any key
# volatile-ttl -> remove the key with the nearest expire time (minor TTL)
# noeviction -> don't expire at all, just return an error on write operations
# volatile-lru -> Evict using approximated LRU among the keys with an expire set.
# allkeys-lru -> Evict any key using approximated LRU.
# volatile-lfu -> Evict using approximated LFU among the keys with an expire set.
# allkeys-lfu -> Evict any key using approximated LFU.
# volatile-random -> Remove a random key among the ones with an expire set.
# allkeys-random -> Remove a random key, any key.
# volatile-ttl -> Remove the key with the nearest expire time (minor TTL)
# noeviction -> Don't evict anything, just return an error on write operations.
#
# LRU means Least Recently Used
# LFU means Least Frequently Used
#
# Both LRU, LFU and volatile-ttl are implemented using approximated
# randomized algorithms.
#
# Note: with any of the above policies, Redis will return an error on write
# operations, when there are no suitable keys for eviction.
...
...
@@ -540,14 +548,14 @@ slave-priority 100
#
# maxmemory-policy noeviction
# LRU and minimal TTL algorithms are not precise algorithms but approximated
# LRU
, LFU
and minimal TTL algorithms are not precise algorithms but approximated
# algorithms (in order to save memory), so you can tune it for speed or
# accuracy. For default Redis will check five keys and pick the one that was
# used less recently, you can change the sample size using the following
# configuration directive.
#
# The default of 5 produces good enough results. 10 Approximates very closely
# true LRU but costs
a bit
more CPU. 3 is
very
fast but not very accurate.
# true LRU but costs more CPU. 3 is fast
er
but not very accurate.
#
# maxmemory-samples 5
...
...
@@ -1113,3 +1121,55 @@ hz 10
# in order to commit the file to the disk more incrementally and avoid
# big latency spikes.
aof
-
rewrite
-
incremental
-
fsync
yes
# Redis LFU eviction (see maxmemory setting) can be tuned. However it is a good
# idea to start with the default settings and only change them after investigating
# how to improve the performances and how the keys LFU change over time, which
# is possible to inspect via the OBJECT FREQ command.
#
# There are two tunable parameters in the Redis LFU implementation: the
# counter logarithm factor and the counter decay time. It is important to
# understand what the two parameters mean before changing them.
#
# The LFU counter is just 8 bits per key, it's maximum value is 255, so Redis
# uses a probabilistic increment with logarithmic behavior. Given the value
# of the old counter, when a key is accessed, the counter is incremented in
# this way:
#
# 1. A random number R between 0 and 1 is extracted.
# 2. A probability P is calculated as 1/(old_value*lfu_log_factor+1).
# 3. The counter is incremented only if R < P.
#
# The default lfu-log-factor is 10. This is a table of how the frequency
# counter changes with a different number of accesses with different
# logarithmic factors:
#
# +--------+------------+------------+------------+------------+------------+
# | factor | 100 hits | 1000 hits | 100K hits | 1M hits | 10M hits |
# +--------+------------+------------+------------+------------+------------+
# | 0 | 104 | 255 | 255 | 255 | 255 |
# +--------+------------+------------+------------+------------+------------+
# | 1 | 18 | 49 | 255 | 255 | 255 |
# +--------+------------+------------+------------+------------+------------+
# | 10 | 10 | 18 | 142 | 255 | 255 |
# +--------+------------+------------+------------+------------+------------+
# | 100 | 8 | 11 | 49 | 143 | 255 |
# +--------+------------+------------+------------+------------+------------+
#
# NOTE: The above table was obtained by running the following commands:
#
# redis-benchmark -n 1000000 incr foo
# redis-cli object freq foo
#
# NOTE 2: The counter initial value is 5 in order to give new objects a chance
# to accumulate hits.
#
# The counter decay time is the time, in minutes, that must elapse in order
# for the key counter to be divided by two (or decremented if it has a value
# less <= 10).
#
# The default value for the lfu-decay-time is 1. A Special value of 0 means to
# decay the counter every time it happens to be scanned.
#
# lfu-log-factor 10
# lfu-decay-time 1
src/config.c
View file @
6854c7b9
...
...
@@ -324,6 +324,18 @@ void loadServerConfigFromString(char *config) {
err
=
"maxmemory-samples must be 1 or greater"
;
goto
loaderr
;
}
}
else
if
(
!
strcasecmp
(
argv
[
0
],
"lfu-log-factor"
)
&&
argc
==
2
)
{
server
.
lfu_log_factor
=
atoi
(
argv
[
1
]);
if
(
server
.
maxmemory_samples
<
0
)
{
err
=
"lfu-log-factor must be 0 or greater"
;
goto
loaderr
;
}
}
else
if
(
!
strcasecmp
(
argv
[
0
],
"lfu-decay-time"
)
&&
argc
==
2
)
{
server
.
lfu_decay_time
=
atoi
(
argv
[
1
]);
if
(
server
.
maxmemory_samples
<
1
)
{
err
=
"lfu-decay-time must be 0 or greater"
;
goto
loaderr
;
}
}
else
if
(
!
strcasecmp
(
argv
[
0
],
"slaveof"
)
&&
argc
==
3
)
{
slaveof_linenum
=
linenum
;
server
.
masterhost
=
sdsnew
(
argv
[
1
]);
...
...
@@ -955,6 +967,10 @@ void configSetCommand(client *c) {
"tcp-keepalive"
,
server
.
tcpkeepalive
,
0
,
LLONG_MAX
)
{
}
config_set_numerical_field
(
"maxmemory-samples"
,
server
.
maxmemory_samples
,
1
,
LLONG_MAX
)
{
}
config_set_numerical_field
(
"lfu-log-factor"
,
server
.
lfu_log_factor
,
0
,
LLONG_MAX
)
{
}
config_set_numerical_field
(
"lfu-decay-time"
,
server
.
lfu_decay_time
,
0
,
LLONG_MAX
)
{
}
config_set_numerical_field
(
"timeout"
,
server
.
maxidletime
,
0
,
LONG_MAX
)
{
}
config_set_numerical_field
(
...
...
src/evict.c
View file @
6854c7b9
...
...
@@ -287,13 +287,12 @@ unsigned long LFUTimeElapsed(unsigned long ldt) {
/* Logarithmically increment a counter. The greater is the current counter value
* the less likely is that it gets really implemented. Saturate it at 255. */
#define LFU_LOG_FACTOR 10
uint8_t
LFULogIncr
(
uint8_t
counter
)
{
if
(
counter
==
255
)
return
255
;
double
r
=
(
double
)
rand
()
/
RAND_MAX
;
double
baseval
=
counter
-
LFU_INIT_VAL
;
if
(
baseval
<
0
)
baseval
=
0
;
double
p
=
1
.
0
/
(
baseval
*
LFU_LOG_FACTOR
+
1
);
double
p
=
1
.
0
/
(
baseval
*
server
.
lfu_log_factor
+
1
);
if
(
r
<
p
)
counter
++
;
return
counter
;
}
...
...
@@ -308,7 +307,7 @@ uint8_t LFULogIncr(uint8_t counter) {
unsigned
long
LFUDecrAndReturn
(
robj
*
o
)
{
unsigned
long
ldt
=
o
->
lru
>>
8
;
unsigned
long
counter
=
o
->
lru
&
255
;
if
(
LFUTimeElapsed
(
ldt
)
>=
LFU_DECR_INTERVAL
&&
counter
)
{
if
(
LFUTimeElapsed
(
ldt
)
>=
server
.
lfu_decay_time
&&
counter
)
{
if
(
counter
>
LFU_INIT_VAL
*
2
)
{
counter
/=
2
;
if
(
counter
<
LFU_INIT_VAL
*
2
)
counter
=
LFU_INIT_VAL
*
2
;
...
...
src/server.c
View file @
6854c7b9
...
...
@@ -1341,6 +1341,8 @@ void initServerConfig(void) {
server
.
maxmemory
=
CONFIG_DEFAULT_MAXMEMORY
;
server
.
maxmemory_policy
=
CONFIG_DEFAULT_MAXMEMORY_POLICY
;
server
.
maxmemory_samples
=
CONFIG_DEFAULT_MAXMEMORY_SAMPLES
;
server
.
lfu_log_factor
=
CONFIG_DEFAULT_LFU_LOG_FACTOR
;
server
.
lfu_decay_time
=
CONFIG_DEFAULT_LFU_DECAY_TIME
;
server
.
hash_max_ziplist_entries
=
OBJ_HASH_MAX_ZIPLIST_ENTRIES
;
server
.
hash_max_ziplist_value
=
OBJ_HASH_MAX_ZIPLIST_VALUE
;
server
.
list_max_ziplist_size
=
OBJ_LIST_MAX_ZIPLIST_SIZE
;
...
...
src/server.h
View file @
6854c7b9
...
...
@@ -129,6 +129,8 @@ typedef long long mstime_t; /* millisecond time type. */
#define CONFIG_DEFAULT_REPL_DISABLE_TCP_NODELAY 0
#define CONFIG_DEFAULT_MAXMEMORY 0
#define CONFIG_DEFAULT_MAXMEMORY_SAMPLES 5
#define CONFIG_DEFAULT_LFU_LOG_FACTOR 10
#define CONFIG_DEFAULT_LFU_DECAY_TIME 1
#define CONFIG_DEFAULT_AOF_FILENAME "appendonly.aof"
#define CONFIG_DEFAULT_AOF_NO_FSYNC_ON_REWRITE 0
#define CONFIG_DEFAULT_AOF_LOAD_TRUNCATED 1
...
...
@@ -981,6 +983,8 @@ struct redisServer {
unsigned
long
long
maxmemory
;
/* Max number of memory bytes to use */
int
maxmemory_policy
;
/* Policy for key eviction */
int
maxmemory_samples
;
/* Pricision of random sampling */
unsigned
int
lfu_log_factor
;
/* LFU logarithmic counter factor. */
unsigned
int
lfu_decay_time
;
/* LFU counter decay factor. */
/* Blocked clients */
unsigned
int
bpop_blocked_clients
;
/* Number of clients blocked by lists */
list
*
unblocked_clients
;
/* list of clients to unblock before next loop */
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment