Unverified Commit 64f61596 authored by Oran Agra's avatar Oran Agra Committed by GitHub
Browse files

Merge Redis Functions PR (#9780)

# Redis Function
This PR added the Redis Functions capabilities that were suggested on #8693.
The PR also introduce a big refactoring to the current Lua implementation
(i.e `scripting.c`). The main purpose of the refactoring is to have better
code sharing between the Lua implementation that exists today on Redis
(`scripting.c`) and the new Lua engine that is introduced on this PR.
The refactoring includes code movements and file name changes as well as some
logic changes that need to be carefully reviewed. To make the review easier,
the PR was split into multiple commits. Each commit is deeply described later on
but the main concept is that some commits are just moving code around without
making any logical changes, those commits are less likely to cause any issues
or regressions and can be reviewed fast. Other commits, which perform code and
logic changes, need to be reviewed carefully, but those commits were created
after the code movements so it's pretty easy to see what was changed. To sum up,
it is highly recommended to review this PR commit by commit as it will be easier
to see the changes, it is also recommended to read each commit description
(written below) to understand what was changed on the commit and whether or not
it's just a huge code movement or a logic changes.

## Terminology
Currently, the terminology in Redis is not clearly defined. Scripts refer to Lua
scripts and eval also refers only to Lua. Introducing Redis Function requires
redefining those terms to be able to clearly understand what is been discussed
on each context.
* eval - legacy Lua script implementation.
* Function - new scripting implementation (currently implemented in Lua but in
  the future, it might be other languages like javascript).
* Engine - the component that is responsible for executing functions.
* Script - Function or legacy Lua (executed with `eval` or `evalsha`)

## Refactoring New Structure
Today, the entire scripting logic is located on `scripting.c`. This logic can
be split into 3 main groups:
1. Script management - responsible for storing the scripts that were sent to
   Redis and retrieving them when they need to be run (base on the script sha
   on the current implementation).
2. Script invocation - invoke the script given on `eval` or `evalsha` command
   (this part includes finding the relevant script, preparing the arguments, ..)
3. Interact back with Redis (command invocation)

Those 3 groups are tightly coupled on `scripting.c`. Redis Functions also need
to use those groups logics, for example,  to interact back with Redis or to
execute Lua code. The refactoring attempts to split those 3 groups and define
APIs so that we can reuse the code both on legacy Lua scripts and Redis Functions.

In order to do so we define the following units:
1. script.c: responsible for interaction with Redis from within a script.
2. script_lua.c: responsible to execute Lua code, uses `script.c` to interact
   with Redis from within the Lua code.
3. function_lua.c: contains the Lua engine implementation, uses `script_lua.c`
   to execute the Lua code.
4. functions.c: Contains Redis Functions implementation (`FUNCTION` command,),
   uses `functions_lua.c` if the function it wants to invoke needs the Lua
   engine.
4. eval.c: the original `scripting.c` contains the Lua legacy implementation and
   was refactored to use `script_lua.c` to invoke the Lua code.

## Commits breakdown
Notice: Some small commits are omitted from this list as they are small and
insignificant (for example build fixes)

### First commit - code movements
This commit rename `scripting.c` -> `eval.c` and introduce the new `script_lua.c`
unit. The commit moves relevant code from `eval.c` (`scripting.c`) to
`script_lua.c`, the purpose of moving the code is so that later we will be able
to re-use the code on the Lua engine (`function_lua.c`). The commit only moves
the code without modifying even a single line, so there is a very low risk of
breaking anything and it also makes it much easier to see the changes on the
following commits.
Because the commit does not change the code (only moves it), it does not compile.
But we do not care about it as the only purpose here is to make the review
processes simpler.

### Second commit - move legacy Lua variables into `eval.c`
Today, all Lua-related variables are located on the server struct. The commit
attempt to identify those variable and take them out from the server struct,
leaving only script related variables (variables that later need to be used
also by engines)
The following variable where renamed and left on the server struct:
   * lua_caller 			-> script_caller
   * lua_time_limit 		-> script_time_limit
   * lua_timedout 		-> script_timedout
   * lua_oom 			-> script_oom
   * lua_disable_deny_script 	-> script_disable_deny_script
   * in_eval			-> in_script

The following variables where moved to lctx under eval.c
   * lua
   * lua_client
   * lua_cur_script
   * lua_scripts
   * lua_scripts_mem
   * lua_replicate_commands
   * lua_write_dirty
   * lua_random_dirty
   * lua_multi_emitted
   * lua_repl
   * lua_kill
   * lua_time_start
   * lua_time_snapshot

This commit is in a low risk of introducing any issues and it is just moving
variables around and not changing any logic.

### Third commit - introducing script unit
This commit introduces the `script.c` unit. Its purpose (as described above) is
to provide an API for scripts to interact with Redis. Interaction includes
mostly executing commands, but also other functionalities. The interaction is
done using a `ScriptRunCtx` object that needs to be created by the user and
initialized using `scriptPrepareForRun`. A detailed list of functionalities
expose by the unit:
1. Calling commands (including all the validation checks such as
   acl, cluster, read only run, ...)
2. Set Resp
3. Set Replication method (AOF/REPLICATION/NONE)
4. Call Redis back on long-running scripts to allow Redis to reply to clients
   and perform script kill

The commit introduces the new unit and uses it on eval commands to interact with
Redis.

### Fourth commit - Moved functionality of invoke Lua code to `script_lua.c`
This commit moves the logic of invoking the Lua code into `script_lua.c` so
later it can be used also by Lua engine (`function_lua.c`). The code is located
on `callFunction` function and assumes the Lua function already located on the
top of the Lua stack. This commit also change `eval.c` to use the new
functionality to invoke Lua code.

### Fith commit - Added Redis Functions unit (`functions.c`) and Lua engine
(`function_lua.c`)
Added Redis Functions unit under `functions.c`, included:
1. FUNCTION command:
     * FUNCTION CREATE
     * FUNCTION CALL
     * FUNCTION DELETE
     * FUNCTION KILL
     * FUNCTION INFO
     * FUNCTION STATS
2. Register engines

In addition, this commit introduces the first engine that uses the Redis
Functions capabilities, the Lua engine (`function_lua.c`)

## API Changes
### `lua-time-limit`
configuration was renamed to `script-time-limit` (keep `lua-time-limit` as alias
for backward compatibility).

### Error log changes
When integrating with Redis from within a Lua script, the `Lua` term was removed
from all the error messages and instead we write only `script`. For example:
`Wrong number of args calling Redis command From Lua script` -> `Wrong number
of args calling Redis command From script`

### `info memory` changes:
Before stating all the changes made to memory stats we will try to explain the
reason behind them and what we want to see on those metrics:
* memory metrics should show both totals (for all scripting frameworks), as well
  as a breakdown per framework / vm.
* The totals metrics should have "human" metrics while the breakdown shouldn't.
* We did try to maintain backward compatibility in some way, that said we did
  make some repurpose to existing metrics where it looks reasonable.
* We separate between memory used by the script framework (part of redis's
  used_memory), and memory used by the VM (not part of redis's used_memory)

A full breakdown of `info memory` changes:
* `used_memory_lua` and `used_memory_lua_human` was deprecated,
  `used_memory_vm_eval` has the same meaning as `used_memory_lua`
* `used_memory_scripts` was renamed to `used_memory_scripts_eval`
* `used_memory_scripts` and `used_memory_scripts_human` were repurposed and now
  return the total memory used by functions and eval (not including vm memory,
  only code cache, and structs).
* `used_memory_vm_function` was added and represents the total memory used by
  functions vm's
* `used_memory_functions` was added and represents the total memory by functions
  (not including vm memory, only code cache, and structs)
* `used_memory_vm_total` and `used_memory_vm_total_human` was added and
  represents the total memory used by vm's (functions and eval combined)

### `functions.caches`
`functions.caches` field was added to `memory stats`, representing the memory
used by engines that are not functions (this memory includes data structures
like dictionaries, arrays, ...)

## New API
### FUNCTION CREATE

Usage: FUNCTION CREATE `ENGINE` `NAME` `[REPLACE]` `[DESC <DESCRIPTION>]` `<CODE>`

* `ENGINE` - The name of the engine to use to create the script.
* `NAME` - the name of the function that can be used later to call the function
  using `FUNCTION CALL` command.
* `REPLACE` - if given, replace the given function with the existing function
  (if exists).
* `DESCRIPTION` - optional argument describing the function and what it does
* `CODE` - function code.

The command will return `OK` if created successfully or error in the following
cases:
* The given engine name does not exist
* The function name is already taken and `REPLACE` was not used.
* The given function failed on the compilation.

### FCALL and FCALL_RO

Usage: FCALL/FCALL_RO `NAME` `NUM_KEYS key1 key2` … ` arg1 arg2`

Call and execute the function specified by `NAME`. The function will receive
all arguments given after `NUM_KEYS`. The return value from the function will
be returned to the user as a result.

* `NAME` - Name of the function to run.
* The rest is as today with EVALSHA command.

The command will return an error in the following cases:
* `NAME` does not exist
* The function itself returned an error.

The `FCALL_RO` is equivalent to `EVAL_RO` and allows only read-only commands to
be invoked from the script.

### FUNCTION DELETE

Usage: FUNCTION DELETE `NAME`

Delete a function identified by `NAME`. Return `OK` on success or error on one
of the following:
* The given function does not exist

### FUNCTION INFO

Usage: FUNCTION INFO `NAME` [WITHCODE]

Return information about a function by function name:
* Function name
* Engine name
* Description
* Raw code (only if WITHCODE argument is given)

### FUNCTION LIST

Usage: FUNCTION LIST

Return general information about all the functions:
* Function name
* Engine name
* Description

### FUNCTION STATS

Usage: FUNCTION STATS

Return information about the current running function:
* Function name
* Command that was used to invoke the function
* Duration in MS that the function is already running

If no function is currently running, this section is just a RESP nil.

Additionally, return a list of all the available engines.

### FUNCTION KILL

Usage: `FUNCTION KILL`

Kill the currently executing function. The command will fail if the function
already initiated a write command.

## Notes
Note: Function creation/deletion is replicated to AOF but AOFRW is not
implemented sense its going to be removed: #9794
parents e57a4db5 cbd46317
...@@ -443,6 +443,16 @@ This file also implements both the `SYNC` and `PSYNC` commands that are ...@@ -443,6 +443,16 @@ This file also implements both the `SYNC` and `PSYNC` commands that are
used in order to perform the first synchronization between masters and used in order to perform the first synchronization between masters and
replicas, or to continue the replication after a disconnection. replicas, or to continue the replication after a disconnection.
Script
---
The script unit is compose of 3 units
* `script.c` - integration of scripts with Redis (commands execution, set replication/resp, ..)
* `script_lua.c` - responsible to execute Lua code, uses script.c to interact with Redis from within the Lua code.
* `function_lua.c` - contains the Lua engine implementation, uses script_lua.c to execute the Lua code.
* `functions.c` - Contains Redis Functions implementation (FUNCTION command), uses functions_lua.c if the function it wants to invoke needs the Lua engine.
* `eval.c` - Contains the `eval` implementation using `script_lua.c` to invoke the Lua code.
Other C files Other C files
--- ---
...@@ -451,7 +461,6 @@ Other C files ...@@ -451,7 +461,6 @@ Other C files
* `sds.c` is the Redis string library, check https://github.com/antirez/sds for more information. * `sds.c` is the Redis string library, check https://github.com/antirez/sds for more information.
* `anet.c` is a library to use POSIX networking in a simpler way compared to the raw interface exposed by the kernel. * `anet.c` is a library to use POSIX networking in a simpler way compared to the raw interface exposed by the kernel.
* `dict.c` is an implementation of a non-blocking hash table which rehashes incrementally. * `dict.c` is an implementation of a non-blocking hash table which rehashes incrementally.
* `scripting.c` implements Lua scripting. It is completely self-contained and isolated from the rest of the Redis implementation and is simple enough to understand if you are familiar with the Lua API.
* `cluster.c` implements the Redis Cluster. Probably a good read only after being very familiar with the rest of the Redis code base. If you want to read `cluster.c` make sure to read the [Redis Cluster specification][4]. * `cluster.c` implements the Redis Cluster. Probably a good read only after being very familiar with the rest of the Redis code base. If you want to read `cluster.c` make sure to read the [Redis Cluster specification][4].
[4]: https://redis.io/topics/cluster-spec [4]: https://redis.io/topics/cluster-spec
......
...@@ -309,7 +309,7 @@ endif ...@@ -309,7 +309,7 @@ endif
REDIS_SERVER_NAME=redis-server$(PROG_SUFFIX) REDIS_SERVER_NAME=redis-server$(PROG_SUFFIX)
REDIS_SENTINEL_NAME=redis-sentinel$(PROG_SUFFIX) REDIS_SENTINEL_NAME=redis-sentinel$(PROG_SUFFIX)
REDIS_SERVER_OBJ=adlist.o quicklist.o ae.o anet.o dict.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crcspeed.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o redis-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o lolwut.o lolwut5.o lolwut6.o acl.o tracking.o connection.o tls.o sha256.o timeout.o setcpuaffinity.o monotonic.o mt19937-64.o resp_parser.o call_reply.o REDIS_SERVER_OBJ=adlist.o quicklist.o ae.o anet.o dict.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o eval.o bio.o rio.o rand.o memtest.o crcspeed.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o redis-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o lolwut.o lolwut5.o lolwut6.o acl.o tracking.o connection.o tls.o sha256.o timeout.o setcpuaffinity.o monotonic.o mt19937-64.o resp_parser.o call_reply.o script_lua.o script.o functions.o function_lua.o
REDIS_CLI_NAME=redis-cli$(PROG_SUFFIX) REDIS_CLI_NAME=redis-cli$(PROG_SUFFIX)
REDIS_CLI_OBJ=anet.o adlist.o dict.o redis-cli.o zmalloc.o release.o ae.o redisassert.o crcspeed.o crc64.o siphash.o crc16.o monotonic.o cli_common.o mt19937-64.o REDIS_CLI_OBJ=anet.o adlist.o dict.o redis-cli.o zmalloc.o release.o ae.o redisassert.o crcspeed.o crc64.o siphash.o crc16.o monotonic.o cli_common.o mt19937-64.o
REDIS_BENCHMARK_NAME=redis-benchmark$(PROG_SUFFIX) REDIS_BENCHMARK_NAME=redis-benchmark$(PROG_SUFFIX)
......
...@@ -1897,7 +1897,7 @@ void addACLLogEntry(client *c, int reason, int context, int argpos, sds username ...@@ -1897,7 +1897,7 @@ void addACLLogEntry(client *c, int reason, int context, int argpos, sds username
} }
client *realclient = c; client *realclient = c;
if (realclient->flags & CLIENT_LUA) realclient = server.lua_caller; if (realclient->flags & CLIENT_SCRIPT) realclient = server.script_caller;
le->cinfo = catClientInfoString(sdsempty(),realclient); le->cinfo = catClientInfoString(sdsempty(),realclient);
le->context = context; le->context = context;
......
...@@ -30,6 +30,7 @@ ...@@ -30,6 +30,7 @@
#include "server.h" #include "server.h"
#include "bio.h" #include "bio.h"
#include "rio.h" #include "rio.h"
#include "functions.h"
#include <signal.h> #include <signal.h>
#include <fcntl.h> #include <fcntl.h>
...@@ -754,7 +755,8 @@ int loadAppendOnlyFile(char *filename) { ...@@ -754,7 +755,8 @@ int loadAppendOnlyFile(char *filename) {
serverLog(LL_NOTICE,"Reading RDB preamble from AOF file..."); serverLog(LL_NOTICE,"Reading RDB preamble from AOF file...");
if (fseek(fp,0,SEEK_SET) == -1) goto readerr; if (fseek(fp,0,SEEK_SET) == -1) goto readerr;
rioInitWithFile(&rdb,fp); rioInitWithFile(&rdb,fp);
if (rdbLoadRio(&rdb,RDBFLAGS_AOF_PREAMBLE,NULL,server.db) != C_OK) {
if (rdbLoadRio(&rdb,RDBFLAGS_AOF_PREAMBLE,NULL) != C_OK) {
serverLog(LL_WARNING,"Error reading the RDB preamble of the AOF file, AOF loading aborted"); serverLog(LL_WARNING,"Error reading the RDB preamble of the AOF file, AOF loading aborted");
goto readerr; goto readerr;
} else { } else {
......
...@@ -2650,7 +2650,7 @@ standardConfig configs[] = { ...@@ -2650,7 +2650,7 @@ standardConfig configs[] = {
createULongConfig("acllog-max-len", NULL, MODIFIABLE_CONFIG, 0, LONG_MAX, server.acllog_max_len, 128, INTEGER_CONFIG, NULL, NULL), createULongConfig("acllog-max-len", NULL, MODIFIABLE_CONFIG, 0, LONG_MAX, server.acllog_max_len, 128, INTEGER_CONFIG, NULL, NULL),
/* Long Long configs */ /* Long Long configs */
createLongLongConfig("lua-time-limit", NULL, MODIFIABLE_CONFIG, 0, LONG_MAX, server.lua_time_limit, 5000, INTEGER_CONFIG, NULL, NULL),/* milliseconds */ createLongLongConfig("script-time-limit", "lua-time-limit", MODIFIABLE_CONFIG, 0, LONG_MAX, server.script_time_limit, 5000, INTEGER_CONFIG, NULL, NULL),/* milliseconds */
createLongLongConfig("cluster-node-timeout", NULL, MODIFIABLE_CONFIG, 0, LLONG_MAX, server.cluster_node_timeout, 15000, INTEGER_CONFIG, NULL, NULL), createLongLongConfig("cluster-node-timeout", NULL, MODIFIABLE_CONFIG, 0, LLONG_MAX, server.cluster_node_timeout, 15000, INTEGER_CONFIG, NULL, NULL),
createLongLongConfig("slowlog-log-slower-than", NULL, MODIFIABLE_CONFIG, -1, LLONG_MAX, server.slowlog_log_slower_than, 10000, INTEGER_CONFIG, NULL, NULL), createLongLongConfig("slowlog-log-slower-than", NULL, MODIFIABLE_CONFIG, -1, LLONG_MAX, server.slowlog_log_slower_than, 10000, INTEGER_CONFIG, NULL, NULL),
createLongLongConfig("latency-monitor-threshold", NULL, MODIFIABLE_CONFIG, 0, LLONG_MAX, server.latency_monitor_threshold, 0, INTEGER_CONFIG, NULL, NULL), createLongLongConfig("latency-monitor-threshold", NULL, MODIFIABLE_CONFIG, 0, LLONG_MAX, server.latency_monitor_threshold, 0, INTEGER_CONFIG, NULL, NULL),
......
...@@ -31,6 +31,7 @@ ...@@ -31,6 +31,7 @@
#include "cluster.h" #include "cluster.h"
#include "atomicvar.h" #include "atomicvar.h"
#include "latency.h" #include "latency.h"
#include "script.h"
#include <signal.h> #include <signal.h>
#include <ctype.h> #include <ctype.h>
...@@ -88,7 +89,7 @@ robj *lookupKey(redisDb *db, robj *key, int flags) { ...@@ -88,7 +89,7 @@ robj *lookupKey(redisDb *db, robj *key, int flags) {
* commands is to make writable replicas behave consistently. It * commands is to make writable replicas behave consistently. It
* shall not be used in readonly commands. Modules are accepted so * shall not be used in readonly commands. Modules are accepted so
* that we don't break old modules. */ * that we don't break old modules. */
client *c = server.in_eval ? server.lua_client : server.current_client; client *c = server.in_script ? scriptGetClient() : server.current_client;
serverAssert(!c || !c->cmd || (c->cmd->flags & (CMD_WRITE|CMD_MODULE))); serverAssert(!c || !c->cmd || (c->cmd->flags & (CMD_WRITE|CMD_MODULE)));
} }
if (expireIfNeeded(db, key, force_delete_expired)) { if (expireIfNeeded(db, key, force_delete_expired)) {
...@@ -1501,8 +1502,8 @@ int keyIsExpired(redisDb *db, robj *key) { ...@@ -1501,8 +1502,8 @@ int keyIsExpired(redisDb *db, robj *key) {
* only the first time it is accessed and not in the middle of the * only the first time it is accessed and not in the middle of the
* script execution, making propagation to slaves / AOF consistent. * script execution, making propagation to slaves / AOF consistent.
* See issue #1525 on Github for more information. */ * See issue #1525 on Github for more information. */
if (server.lua_caller) { if (server.script_caller) {
now = server.lua_time_snapshot; now = evalTimeSnapshot();
} }
/* If we are in the middle of a command execution, we still want to use /* If we are in the middle of a command execution, we still want to use
* a reference time that does not change: in that case we just use the * a reference time that does not change: in that case we just use the
...@@ -1743,6 +1744,11 @@ int evalGetKeys(struct redisCommand *cmd, robj **argv, int argc, getKeysResult * ...@@ -1743,6 +1744,11 @@ int evalGetKeys(struct redisCommand *cmd, robj **argv, int argc, getKeysResult *
return genericGetKeys(0, 2, 3, 1, argv, argc, result); return genericGetKeys(0, 2, 3, 1, argv, argc, result);
} }
int functionGetKeys(struct redisCommand *cmd, robj **argv, int argc, getKeysResult *result) {
UNUSED(cmd);
return genericGetKeys(0, 2, 3, 1, argv, argc, result);
}
int lmpopGetKeys(struct redisCommand *cmd, robj **argv, int argc, getKeysResult *result) { int lmpopGetKeys(struct redisCommand *cmd, robj **argv, int argc, getKeysResult *result) {
UNUSED(cmd); UNUSED(cmd);
return genericGetKeys(0, 1, 2, 1, argv, argc, result); return genericGetKeys(0, 1, 2, 1, argv, argc, result);
......
...@@ -919,7 +919,7 @@ NULL ...@@ -919,7 +919,7 @@ NULL
addReplyStatus(c,"Apparently Redis did not crash: test passed"); addReplyStatus(c,"Apparently Redis did not crash: test passed");
} else if (!strcasecmp(c->argv[1]->ptr,"set-disable-deny-scripts") && c->argc == 3) } else if (!strcasecmp(c->argv[1]->ptr,"set-disable-deny-scripts") && c->argc == 3)
{ {
server.lua_disable_deny_script = atoi(c->argv[2]->ptr);; server.script_disable_deny_script = atoi(c->argv[2]->ptr);;
addReply(c,shared.ok); addReply(c,shared.ok);
} else if (!strcasecmp(c->argv[1]->ptr,"config-rewrite-force-all") && c->argc == 2) } else if (!strcasecmp(c->argv[1]->ptr,"config-rewrite-force-all") && c->argc == 2)
{ {
......
...@@ -939,7 +939,7 @@ long defragOtherGlobals() { ...@@ -939,7 +939,7 @@ long defragOtherGlobals() {
/* there are many more pointers to defrag (e.g. client argv, output / aof buffers, etc. /* there are many more pointers to defrag (e.g. client argv, output / aof buffers, etc.
* but we assume most of these are short lived, we only need to defrag allocations * but we assume most of these are short lived, we only need to defrag allocations
* that remain static for a long time */ * that remain static for a long time */
defragged += activeDefragSdsDict(server.lua_scripts, DEFRAG_SDS_DICT_VAL_IS_STROB); defragged += activeDefragSdsDict(evalScriptsDict(), DEFRAG_SDS_DICT_VAL_IS_STROB);
defragged += activeDefragSdsListAndDict(server.repl_scriptcache_fifo, server.repl_scriptcache_dict, DEFRAG_SDS_DICT_NO_VAL); defragged += activeDefragSdsListAndDict(server.repl_scriptcache_fifo, server.repl_scriptcache_dict, DEFRAG_SDS_DICT_NO_VAL);
defragged += moduleDefragGlobals(); defragged += moduleDefragGlobals();
return defragged; return defragged;
......
...@@ -33,6 +33,7 @@ ...@@ -33,6 +33,7 @@
#include "cluster.h" #include "cluster.h"
#include "monotonic.h" #include "monotonic.h"
#include "resp_parser.h" #include "resp_parser.h"
#include "script_lua.h"
#include <lua.h> #include <lua.h>
#include <lauxlib.h> #include <lauxlib.h>
...@@ -40,32 +41,22 @@ ...@@ -40,32 +41,22 @@
#include <ctype.h> #include <ctype.h>
#include <math.h> #include <math.h>
static void redisProtocolToLuaType_Int(void *ctx, long long val, const char *proto, size_t proto_len);
static void redisProtocolToLuaType_BulkString(void *ctx, const char *str, size_t len, const char *proto, size_t proto_len);
static void redisProtocolToLuaType_NullBulkString(void *ctx, const char *proto, size_t proto_len);
static void redisProtocolToLuaType_NullArray(void *ctx, const char *proto, size_t proto_len);
static void redisProtocolToLuaType_Status(void *ctx, const char *str, size_t len, const char *proto, size_t proto_len);
static void redisProtocolToLuaType_Error(void *ctx, const char *str, size_t len, const char *proto, size_t proto_len);
static void redisProtocolToLuaType_Array(struct ReplyParser *parser, void *ctx, size_t len, const char *proto);
static void redisProtocolToLuaType_Map(struct ReplyParser *parser, void *ctx, size_t len, const char *proto);
static void redisProtocolToLuaType_Set(struct ReplyParser *parser, void *ctx, size_t len, const char *proto);
static void redisProtocolToLuaType_Null(void *ctx, const char *proto, size_t proto_len);
static void redisProtocolToLuaType_Bool(void *ctx, int val, const char *proto, size_t proto_len);
static void redisProtocolToLuaType_Double(void *ctx, double d, const char *proto, size_t proto_len);
static void redisProtocolToLuaType_BigNumber(void *ctx, const char *str, size_t len, const char *proto, size_t proto_len);
static void redisProtocolToLuaType_VerbatimString(void *ctx, const char *format, const char *str, size_t len, const char *proto, size_t proto_len);
static void redisProtocolToLuaType_Attribute(struct ReplyParser *parser, void *ctx, size_t len, const char *proto);
int redis_math_random (lua_State *L);
int redis_math_randomseed (lua_State *L);
void ldbInit(void); void ldbInit(void);
void ldbDisable(client *c); void ldbDisable(client *c);
void ldbEnable(client *c); void ldbEnable(client *c);
void evalGenericCommandWithDebugging(client *c, int evalsha); void evalGenericCommandWithDebugging(client *c, int evalsha);
void luaLdbLineHook(lua_State *lua, lua_Debug *ar);
void ldbLog(sds entry);
void ldbLogRedisReply(char *reply);
sds ldbCatStackValue(sds s, lua_State *lua, int idx); sds ldbCatStackValue(sds s, lua_State *lua, int idx);
/* Lua context */
struct luaCtx {
lua_State *lua; /* The Lua interpreter. We use just one for all clients */
client *lua_client; /* The "fake client" to query Redis from Lua */
char *lua_cur_script; /* SHA1 of the script currently running, or NULL */
dict *lua_scripts; /* A dictionary of SHA1 -> Lua scripts */
unsigned long long lua_scripts_mem; /* Cached scripts' memory + oh */
int lua_replicate_commands; /* True if we are doing single commands repl. */
} lctx;
/* Debugger shared state is stored inside this global structure. */ /* Debugger shared state is stored inside this global structure. */
#define LDB_BREAKPOINTS_MAX 64 /* Max number of breakpoints. */ #define LDB_BREAKPOINTS_MAX 64 /* Max number of breakpoints. */
#define LDB_MAX_LEN_DEFAULT 256 /* Default len limit for replies / var dumps. */ #define LDB_MAX_LEN_DEFAULT 256 /* Default len limit for replies / var dumps. */
...@@ -115,1035 +106,6 @@ void sha1hex(char *digest, char *script, size_t len) { ...@@ -115,1035 +106,6 @@ void sha1hex(char *digest, char *script, size_t len) {
digest[40] = '\0'; digest[40] = '\0';
} }
/* ---------------------------------------------------------------------------
* Redis reply to Lua type conversion functions.
* ------------------------------------------------------------------------- */
/* Take a Redis reply in the Redis protocol format and convert it into a
* Lua type. Thanks to this function, and the introduction of not connected
* clients, it is trivial to implement the redis() lua function.
*
* Basically we take the arguments, execute the Redis command in the context
* of a non connected client, then take the generated reply and convert it
* into a suitable Lua type. With this trick the scripting feature does not
* need the introduction of a full Redis internals API. The script
* is like a normal client that bypasses all the slow I/O paths.
*
* Note: in this function we do not do any sanity check as the reply is
* generated by Redis directly. This allows us to go faster.
*
* Errors are returned as a table with a single 'err' field set to the
* error string.
*/
static const ReplyParserCallbacks DefaultLuaTypeParserCallbacks = {
.null_array_callback = redisProtocolToLuaType_NullArray,
.bulk_string_callback = redisProtocolToLuaType_BulkString,
.null_bulk_string_callback = redisProtocolToLuaType_NullBulkString,
.error_callback = redisProtocolToLuaType_Error,
.simple_str_callback = redisProtocolToLuaType_Status,
.long_callback = redisProtocolToLuaType_Int,
.array_callback = redisProtocolToLuaType_Array,
.set_callback = redisProtocolToLuaType_Set,
.map_callback = redisProtocolToLuaType_Map,
.bool_callback = redisProtocolToLuaType_Bool,
.double_callback = redisProtocolToLuaType_Double,
.null_callback = redisProtocolToLuaType_Null,
.big_number_callback = redisProtocolToLuaType_BigNumber,
.verbatim_string_callback = redisProtocolToLuaType_VerbatimString,
.attribute_callback = redisProtocolToLuaType_Attribute,
.error = NULL,
};
void redisProtocolToLuaType(lua_State *lua, char* reply) {
ReplyParser parser = {.curr_location = reply, .callbacks = DefaultLuaTypeParserCallbacks};
parseReply(&parser, lua);
}
static void redisProtocolToLuaType_Int(void *ctx, long long val, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
if (!ctx) {
return;
}
lua_State *lua = ctx;
if (!lua_checkstack(lua, 1)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_pushnumber(lua,(lua_Number)val);
}
static void redisProtocolToLuaType_NullBulkString(void *ctx, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
if (!ctx) {
return;
}
lua_State *lua = ctx;
if (!lua_checkstack(lua, 1)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_pushboolean(lua,0);
}
static void redisProtocolToLuaType_NullArray(void *ctx, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
if (!ctx) {
return;
}
lua_State *lua = ctx;
if (!lua_checkstack(lua, 1)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_pushboolean(lua,0);
}
static void redisProtocolToLuaType_BulkString(void *ctx, const char *str, size_t len, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
if (!ctx) {
return;
}
lua_State *lua = ctx;
if (!lua_checkstack(lua, 1)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_pushlstring(lua,str,len);
}
static void redisProtocolToLuaType_Status(void *ctx, const char *str, size_t len, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
if (!ctx) {
return;
}
lua_State *lua = ctx;
if (!lua_checkstack(lua, 3)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_newtable(lua);
lua_pushstring(lua,"ok");
lua_pushlstring(lua,str,len);
lua_settable(lua,-3);
}
static void redisProtocolToLuaType_Error(void *ctx, const char *str, size_t len, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
if (!ctx) {
return;
}
lua_State *lua = ctx;
if (!lua_checkstack(lua, 3)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_newtable(lua);
lua_pushstring(lua,"err");
lua_pushlstring(lua,str,len);
lua_settable(lua,-3);
}
static void redisProtocolToLuaType_Map(struct ReplyParser *parser, void *ctx, size_t len, const char *proto) {
UNUSED(proto);
lua_State *lua = ctx;
if (lua) {
if (!lua_checkstack(lua, 3)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_newtable(lua);
lua_pushstring(lua, "map");
lua_newtable(lua);
}
for (size_t j = 0; j < len; j++) {
parseReply(parser,lua);
parseReply(parser,lua);
if (lua) lua_settable(lua,-3);
}
if (lua) lua_settable(lua,-3);
}
static void redisProtocolToLuaType_Set(struct ReplyParser *parser, void *ctx, size_t len, const char *proto) {
UNUSED(proto);
lua_State *lua = ctx;
if (lua) {
if (!lua_checkstack(lua, 3)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_newtable(lua);
lua_pushstring(lua, "set");
lua_newtable(lua);
}
for (size_t j = 0; j < len; j++) {
parseReply(parser,lua);
if (lua) {
if (!lua_checkstack(lua, 1)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic.
* Notice that here we need to check the stack again because the recursive
* call to redisProtocolToLuaType might have use the room allocated in the stack*/
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_pushboolean(lua,1);
lua_settable(lua,-3);
}
}
if (lua) lua_settable(lua,-3);
}
static void redisProtocolToLuaType_Array(struct ReplyParser *parser, void *ctx, size_t len, const char *proto) {
UNUSED(proto);
lua_State *lua = ctx;
if (lua){
if (!lua_checkstack(lua, 2)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_newtable(lua);
}
for (size_t j = 0; j < len; j++) {
if (lua) lua_pushnumber(lua,j+1);
parseReply(parser,lua);
if (lua) lua_settable(lua,-3);
}
}
static void redisProtocolToLuaType_Attribute(struct ReplyParser *parser, void *ctx, size_t len, const char *proto) {
UNUSED(proto);
/* Parse the attribute reply.
* Currently, we do not expose the attribute to the Lua script so
* we just need to continue parsing and ignore it (the NULL ensures that the
* reply will be ignored). */
for (size_t j = 0; j < len; j++) {
parseReply(parser,NULL);
parseReply(parser,NULL);
}
/* Parse the reply itself. */
parseReply(parser,ctx);
}
static void redisProtocolToLuaType_VerbatimString(void *ctx, const char *format, const char *str, size_t len, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
if (!ctx) {
return;
}
lua_State *lua = ctx;
if (!lua_checkstack(lua, 5)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_newtable(lua);
lua_pushstring(lua,"verbatim_string");
lua_newtable(lua);
lua_pushstring(lua,"string");
lua_pushlstring(lua,str,len);
lua_settable(lua,-3);
lua_pushstring(lua,"format");
lua_pushlstring(lua,format,3);
lua_settable(lua,-3);
lua_settable(lua,-3);
}
static void redisProtocolToLuaType_BigNumber(void *ctx, const char *str, size_t len, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
if (!ctx) {
return;
}
lua_State *lua = ctx;
if (!lua_checkstack(lua, 3)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_newtable(lua);
lua_pushstring(lua,"big_number");
lua_pushlstring(lua,str,len);
lua_settable(lua,-3);
}
static void redisProtocolToLuaType_Null(void *ctx, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
if (!ctx) {
return;
}
lua_State *lua = ctx;
if (!lua_checkstack(lua, 1)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_pushnil(lua);
}
static void redisProtocolToLuaType_Bool(void *ctx, int val, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
if (!ctx) {
return;
}
lua_State *lua = ctx;
if (!lua_checkstack(lua, 1)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_pushboolean(lua,val);
}
static void redisProtocolToLuaType_Double(void *ctx, double d, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
if (!ctx) {
return;
}
lua_State *lua = ctx;
if (!lua_checkstack(lua, 3)) {
/* Increase the Lua stack if needed, to make sure there is enough room
* to push elements to the stack. On failure, exit with panic. */
serverPanic("lua stack limit reach when parsing redis.call reply");
}
lua_newtable(lua);
lua_pushstring(lua,"double");
lua_pushnumber(lua,d);
lua_settable(lua,-3);
}
/* This function is used in order to push an error on the Lua stack in the
* format used by redis.pcall to return errors, which is a lua table
* with a single "err" field set to the error string. Note that this
* table is never a valid reply by proper commands, since the returned
* tables are otherwise always indexed by integers, never by strings. */
void luaPushError(lua_State *lua, char *error) {
lua_Debug dbg;
/* If debugging is active and in step mode, log errors resulting from
* Redis commands. */
if (ldb.active && ldb.step) {
ldbLog(sdscatprintf(sdsempty(),"<error> %s",error));
}
lua_newtable(lua);
lua_pushstring(lua,"err");
/* Attempt to figure out where this function was called, if possible */
if(lua_getstack(lua, 1, &dbg) && lua_getinfo(lua, "nSl", &dbg)) {
sds msg = sdscatprintf(sdsempty(), "%s: %d: %s",
dbg.source, dbg.currentline, error);
lua_pushstring(lua, msg);
sdsfree(msg);
} else {
lua_pushstring(lua, error);
}
lua_settable(lua,-3);
}
/* In case the error set into the Lua stack by luaPushError() was generated
* by the non-error-trapping version of redis.pcall(), which is redis.call(),
* this function will raise the Lua error so that the execution of the
* script will be halted. */
int luaRaiseError(lua_State *lua) {
lua_pushstring(lua,"err");
lua_gettable(lua,-2);
return lua_error(lua);
}
/* Sort the array currently in the stack. We do this to make the output
* of commands like KEYS or SMEMBERS something deterministic when called
* from Lua (to play well with AOf/replication).
*
* The array is sorted using table.sort itself, and assuming all the
* list elements are strings. */
void luaSortArray(lua_State *lua) {
/* Initial Stack: array */
lua_getglobal(lua,"table");
lua_pushstring(lua,"sort");
lua_gettable(lua,-2); /* Stack: array, table, table.sort */
lua_pushvalue(lua,-3); /* Stack: array, table, table.sort, array */
if (lua_pcall(lua,1,0,0)) {
/* Stack: array, table, error */
/* We are not interested in the error, we assume that the problem is
* that there are 'false' elements inside the array, so we try
* again with a slower function but able to handle this case, that
* is: table.sort(table, __redis__compare_helper) */
lua_pop(lua,1); /* Stack: array, table */
lua_pushstring(lua,"sort"); /* Stack: array, table, sort */
lua_gettable(lua,-2); /* Stack: array, table, table.sort */
lua_pushvalue(lua,-3); /* Stack: array, table, table.sort, array */
lua_getglobal(lua,"__redis__compare_helper");
/* Stack: array, table, table.sort, array, __redis__compare_helper */
lua_call(lua,2,0);
}
/* Stack: array (sorted), table */
lua_pop(lua,1); /* Stack: array (sorted) */
}
/* ---------------------------------------------------------------------------
* Lua reply to Redis reply conversion functions.
* ------------------------------------------------------------------------- */
/* Reply to client 'c' converting the top element in the Lua stack to a
* Redis reply. As a side effect the element is consumed from the stack. */
void luaReplyToRedisReply(client *c, lua_State *lua) {
int t = lua_type(lua,-1);
if (!lua_checkstack(lua, 4)) {
/* Increase the Lua stack if needed to make sure there is enough room
* to push 4 elements to the stack. On failure, return error.
* Notice that we need, in the worst case, 4 elements because returning a map might
* require push 4 elements to the Lua stack.*/
addReplyErrorFormat(c, "reached lua stack limit");
lua_pop(lua,1); /* pop the element from the stack */
return;
}
switch(t) {
case LUA_TSTRING:
addReplyBulkCBuffer(c,(char*)lua_tostring(lua,-1),lua_strlen(lua,-1));
break;
case LUA_TBOOLEAN:
if (server.lua_client->resp == 2)
addReply(c,lua_toboolean(lua,-1) ? shared.cone :
shared.null[c->resp]);
else
addReplyBool(c,lua_toboolean(lua,-1));
break;
case LUA_TNUMBER:
addReplyLongLong(c,(long long)lua_tonumber(lua,-1));
break;
case LUA_TTABLE:
/* We need to check if it is an array, an error, or a status reply.
* Error are returned as a single element table with 'err' field.
* Status replies are returned as single element table with 'ok'
* field. */
/* Handle error reply. */
/* we took care of the stack size on function start */
lua_pushstring(lua,"err");
lua_gettable(lua,-2);
t = lua_type(lua,-1);
if (t == LUA_TSTRING) {
addReplyErrorFormat(c,"-%s",lua_tostring(lua,-1));
lua_pop(lua,2);
return;
}
lua_pop(lua,1); /* Discard field name pushed before. */
/* Handle status reply. */
lua_pushstring(lua,"ok");
lua_gettable(lua,-2);
t = lua_type(lua,-1);
if (t == LUA_TSTRING) {
sds ok = sdsnew(lua_tostring(lua,-1));
sdsmapchars(ok,"\r\n"," ",2);
addReplySds(c,sdscatprintf(sdsempty(),"+%s\r\n",ok));
sdsfree(ok);
lua_pop(lua,2);
return;
}
lua_pop(lua,1); /* Discard field name pushed before. */
/* Handle double reply. */
lua_pushstring(lua,"double");
lua_gettable(lua,-2);
t = lua_type(lua,-1);
if (t == LUA_TNUMBER) {
addReplyDouble(c,lua_tonumber(lua,-1));
lua_pop(lua,2);
return;
}
lua_pop(lua,1); /* Discard field name pushed before. */
/* Handle big number reply. */
lua_pushstring(lua,"big_number");
lua_gettable(lua,-2);
t = lua_type(lua,-1);
if (t == LUA_TSTRING) {
sds big_num = sdsnewlen(lua_tostring(lua,-1), lua_strlen(lua,-1));
sdsmapchars(big_num,"\r\n"," ",2);
addReplyBigNum(c,big_num,sdslen(big_num));
sdsfree(big_num);
lua_pop(lua,2);
return;
}
lua_pop(lua,1); /* Discard field name pushed before. */
/* Handle verbatim reply. */
lua_pushstring(lua,"verbatim_string");
lua_gettable(lua,-2);
t = lua_type(lua,-1);
if (t == LUA_TTABLE) {
lua_pushstring(lua,"format");
lua_gettable(lua,-2);
t = lua_type(lua,-1);
if (t == LUA_TSTRING){
char* format = (char*)lua_tostring(lua,-1);
lua_pushstring(lua,"string");
lua_gettable(lua,-3);
t = lua_type(lua,-1);
if (t == LUA_TSTRING){
size_t len;
char* str = (char*)lua_tolstring(lua,-1,&len);
addReplyVerbatim(c, str, len, format);
lua_pop(lua,4);
return;
}
lua_pop(lua,1);
}
lua_pop(lua,1);
}
lua_pop(lua,1); /* Discard field name pushed before. */
/* Handle map reply. */
lua_pushstring(lua,"map");
lua_gettable(lua,-2);
t = lua_type(lua,-1);
if (t == LUA_TTABLE) {
int maplen = 0;
void *replylen = addReplyDeferredLen(c);
/* we took care of the stack size on function start */
lua_pushnil(lua); /* Use nil to start iteration. */
while (lua_next(lua,-2)) {
/* Stack now: table, key, value */
lua_pushvalue(lua,-2); /* Dup key before consuming. */
luaReplyToRedisReply(c, lua); /* Return key. */
luaReplyToRedisReply(c, lua); /* Return value. */
/* Stack now: table, key. */
maplen++;
}
setDeferredMapLen(c,replylen,maplen);
lua_pop(lua,2);
return;
}
lua_pop(lua,1); /* Discard field name pushed before. */
/* Handle set reply. */
lua_pushstring(lua,"set");
lua_gettable(lua,-2);
t = lua_type(lua,-1);
if (t == LUA_TTABLE) {
int setlen = 0;
void *replylen = addReplyDeferredLen(c);
/* we took care of the stack size on function start */
lua_pushnil(lua); /* Use nil to start iteration. */
while (lua_next(lua,-2)) {
/* Stack now: table, key, true */
lua_pop(lua,1); /* Discard the boolean value. */
lua_pushvalue(lua,-1); /* Dup key before consuming. */
luaReplyToRedisReply(c, lua); /* Return key. */
/* Stack now: table, key. */
setlen++;
}
setDeferredSetLen(c,replylen,setlen);
lua_pop(lua,2);
return;
}
lua_pop(lua,1); /* Discard field name pushed before. */
/* Handle the array reply. */
void *replylen = addReplyDeferredLen(c);
int j = 1, mbulklen = 0;
while(1) {
/* we took care of the stack size on function start */
lua_pushnumber(lua,j++);
lua_gettable(lua,-2);
t = lua_type(lua,-1);
if (t == LUA_TNIL) {
lua_pop(lua,1);
break;
}
luaReplyToRedisReply(c, lua);
mbulklen++;
}
setDeferredArrayLen(c,replylen,mbulklen);
break;
default:
addReplyNull(c);
}
lua_pop(lua,1);
}
/* ---------------------------------------------------------------------------
* Lua redis.* functions implementations.
* ------------------------------------------------------------------------- */
#define LUA_CMD_OBJCACHE_SIZE 32
#define LUA_CMD_OBJCACHE_MAX_LEN 64
int luaRedisGenericCommand(lua_State *lua, int raise_error) {
int j, argc = lua_gettop(lua);
struct redisCommand *cmd;
client *c = server.lua_client;
sds reply;
/* Cached across calls. */
static robj **argv = NULL;
static int argv_size = 0;
static robj *cached_objects[LUA_CMD_OBJCACHE_SIZE];
static size_t cached_objects_len[LUA_CMD_OBJCACHE_SIZE];
static int inuse = 0; /* Recursive calls detection. */
/* By using Lua debug hooks it is possible to trigger a recursive call
* to luaRedisGenericCommand(), which normally should never happen.
* To make this function reentrant is futile and makes it slower, but
* we should at least detect such a misuse, and abort. */
if (inuse) {
char *recursion_warning =
"luaRedisGenericCommand() recursive call detected. "
"Are you doing funny stuff with Lua debug hooks?";
serverLog(LL_WARNING,"%s",recursion_warning);
luaPushError(lua,recursion_warning);
return 1;
}
inuse++;
/* Require at least one argument */
if (argc == 0) {
luaPushError(lua,
"Please specify at least one argument for redis.call()");
inuse--;
return raise_error ? luaRaiseError(lua) : 1;
}
/* Build the arguments vector */
if (argv_size < argc) {
argv = zrealloc(argv,sizeof(robj*)*argc);
argv_size = argc;
}
for (j = 0; j < argc; j++) {
char *obj_s;
size_t obj_len;
char dbuf[64];
if (lua_type(lua,j+1) == LUA_TNUMBER) {
/* We can't use lua_tolstring() for number -> string conversion
* since Lua uses a format specifier that loses precision. */
lua_Number num = lua_tonumber(lua,j+1);
obj_len = snprintf(dbuf,sizeof(dbuf),"%.17g",(double)num);
obj_s = dbuf;
} else {
obj_s = (char*)lua_tolstring(lua,j+1,&obj_len);
if (obj_s == NULL) break; /* Not a string. */
}
/* Try to use a cached object. */
if (j < LUA_CMD_OBJCACHE_SIZE && cached_objects[j] &&
cached_objects_len[j] >= obj_len)
{
sds s = cached_objects[j]->ptr;
argv[j] = cached_objects[j];
cached_objects[j] = NULL;
memcpy(s,obj_s,obj_len+1);
sdssetlen(s, obj_len);
} else {
argv[j] = createStringObject(obj_s, obj_len);
}
}
/* Check if one of the arguments passed by the Lua script
* is not a string or an integer (lua_isstring() return true for
* integers as well). */
if (j != argc) {
j--;
while (j >= 0) {
decrRefCount(argv[j]);
j--;
}
luaPushError(lua,
"Lua redis() command arguments must be strings or integers");
inuse--;
return raise_error ? luaRaiseError(lua) : 1;
}
/* Pop all arguments from the stack, we do not need them anymore
* and this way we guaranty we will have room on the stack for the result. */
lua_pop(lua, argc);
/* Setup our fake client for command execution */
c->argv = argv;
c->argc = argc;
c->user = server.lua_caller->user;
/* Process module hooks */
moduleCallCommandFilters(c);
argv = c->argv;
argc = c->argc;
/* Log the command if debugging is active. */
if (ldb.active && ldb.step) {
sds cmdlog = sdsnew("<redis>");
for (j = 0; j < c->argc; j++) {
if (j == 10) {
cmdlog = sdscatprintf(cmdlog," ... (%d more)",
c->argc-j-1);
break;
} else {
cmdlog = sdscatlen(cmdlog," ",1);
cmdlog = sdscatsds(cmdlog,c->argv[j]->ptr);
}
}
ldbLog(cmdlog);
}
/* Command lookup */
cmd = lookupCommand(argv,argc);
if (!cmd || ((cmd->arity > 0 && cmd->arity != argc) ||
(argc < -cmd->arity)))
{
if (cmd)
luaPushError(lua,
"Wrong number of args calling Redis command From Lua script");
else
luaPushError(lua,"Unknown Redis command called from Lua script");
goto cleanup;
}
c->cmd = c->lastcmd = cmd;
/* There are commands that are not allowed inside scripts. */
if (!server.lua_disable_deny_script && (cmd->flags & CMD_NOSCRIPT)) {
luaPushError(lua, "This Redis command is not allowed from scripts");
goto cleanup;
}
/* This check is for EVAL_RO, EVALSHA_RO. We want to allow only read only commands */
if ((server.lua_caller->cmd->proc == evalRoCommand ||
server.lua_caller->cmd->proc == evalShaRoCommand) &&
(cmd->flags & CMD_WRITE))
{
luaPushError(lua, "Write commands are not allowed from read-only scripts");
goto cleanup;
}
/* Check the ACLs. */
int acl_errpos;
int acl_retval = ACLCheckAllPerm(c,&acl_errpos);
if (acl_retval != ACL_OK) {
addACLLogEntry(c,acl_retval,ACL_LOG_CTX_LUA,acl_errpos,NULL,NULL);
switch (acl_retval) {
case ACL_DENIED_CMD:
luaPushError(lua, "The user executing the script can't run this "
"command or subcommand");
break;
case ACL_DENIED_KEY:
luaPushError(lua, "The user executing the script can't access "
"at least one of the keys mentioned in the "
"command arguments");
break;
case ACL_DENIED_CHANNEL:
luaPushError(lua, "The user executing the script can't publish "
"to the channel mentioned in the command");
break;
default:
luaPushError(lua, "The user executing the script is lacking the "
"permissions for the command");
break;
}
goto cleanup;
}
/* Write commands are forbidden against read-only slaves, or if a
* command marked as non-deterministic was already called in the context
* of this script. */
if (cmd->flags & CMD_WRITE) {
int deny_write_type = writeCommandsDeniedByDiskError();
if (server.lua_random_dirty && !server.lua_replicate_commands) {
luaPushError(lua,
"Write commands not allowed after non deterministic commands. Call redis.replicate_commands() at the start of your script in order to switch to single commands replication mode.");
goto cleanup;
} else if (server.masterhost && server.repl_slave_ro &&
server.lua_caller->id != CLIENT_ID_AOF &&
!(server.lua_caller->flags & CLIENT_MASTER))
{
luaPushError(lua, shared.roslaveerr->ptr);
goto cleanup;
} else if (deny_write_type != DISK_ERROR_TYPE_NONE) {
if (deny_write_type == DISK_ERROR_TYPE_RDB) {
luaPushError(lua, shared.bgsaveerr->ptr);
} else {
sds aof_write_err = sdscatfmt(sdsempty(),
"-MISCONF Errors writing to the AOF file: %s\r\n",
strerror(server.aof_last_write_errno));
luaPushError(lua, aof_write_err);
sdsfree(aof_write_err);
}
goto cleanup;
}
}
/* If we reached the memory limit configured via maxmemory, commands that
* could enlarge the memory usage are not allowed, but only if this is the
* first write in the context of this script, otherwise we can't stop
* in the middle. */
if (server.maxmemory && /* Maxmemory is actually enabled. */
server.lua_caller->id != CLIENT_ID_AOF && /* Don't care about mem if loading from AOF. */
!server.masterhost && /* Slave must execute the script. */
server.lua_write_dirty == 0 && /* Script had no side effects so far. */
server.lua_oom && /* Detected OOM when script start. */
(cmd->flags & CMD_DENYOOM))
{
luaPushError(lua, shared.oomerr->ptr);
goto cleanup;
}
if (cmd->flags & CMD_RANDOM) server.lua_random_dirty = 1;
if (cmd->flags & CMD_WRITE) server.lua_write_dirty = 1;
/* If this is a Redis Cluster node, we need to make sure Lua is not
* trying to access non-local keys, with the exception of commands
* received from our master or when loading the AOF back in memory. */
if (server.cluster_enabled && server.lua_caller->id != CLIENT_ID_AOF &&
!(server.lua_caller->flags & CLIENT_MASTER))
{
int error_code;
/* Duplicate relevant flags in the lua client. */
c->flags &= ~(CLIENT_READONLY|CLIENT_ASKING);
c->flags |= server.lua_caller->flags & (CLIENT_READONLY|CLIENT_ASKING);
if (getNodeByQuery(c,c->cmd,c->argv,c->argc,NULL,&error_code) !=
server.cluster->myself)
{
if (error_code == CLUSTER_REDIR_DOWN_RO_STATE) {
luaPushError(lua,
"Lua script attempted to execute a write command while the "
"cluster is down and readonly");
} else if (error_code == CLUSTER_REDIR_DOWN_STATE) {
luaPushError(lua,
"Lua script attempted to execute a command while the "
"cluster is down");
} else {
luaPushError(lua,
"Lua script attempted to access a non local key in a "
"cluster node");
}
goto cleanup;
}
}
/* If we are using single commands replication, we need to wrap what
* we propagate into a MULTI/EXEC block, so that it will be atomic like
* a Lua script in the context of AOF and slaves. */
if (server.lua_replicate_commands &&
!server.lua_multi_emitted &&
!(server.lua_caller->flags & CLIENT_MULTI) &&
server.lua_write_dirty &&
server.lua_repl != PROPAGATE_NONE)
{
execCommandPropagateMulti(server.lua_caller->db->id);
server.lua_multi_emitted = 1;
/* Now we are in the MULTI context, the lua_client should be
* flag as CLIENT_MULTI. */
c->flags |= CLIENT_MULTI;
}
/* Run the command */
int call_flags = CMD_CALL_SLOWLOG | CMD_CALL_STATS;
if (server.lua_replicate_commands) {
/* Set flags according to redis.set_repl() settings. */
if (server.lua_repl & PROPAGATE_AOF)
call_flags |= CMD_CALL_PROPAGATE_AOF;
if (server.lua_repl & PROPAGATE_REPL)
call_flags |= CMD_CALL_PROPAGATE_REPL;
}
call(c,call_flags);
serverAssert((c->flags & CLIENT_BLOCKED) == 0);
/* Convert the result of the Redis command into a suitable Lua type.
* The first thing we need is to create a single string from the client
* output buffers. */
if (listLength(c->reply) == 0 && (size_t)c->bufpos < c->buf_usable_size) {
/* This is a fast path for the common case of a reply inside the
* client static buffer. Don't create an SDS string but just use
* the client buffer directly. */
c->buf[c->bufpos] = '\0';
reply = c->buf;
c->bufpos = 0;
} else {
reply = sdsnewlen(c->buf,c->bufpos);
c->bufpos = 0;
while(listLength(c->reply)) {
clientReplyBlock *o = listNodeValue(listFirst(c->reply));
reply = sdscatlen(reply,o->buf,o->used);
listDelNode(c->reply,listFirst(c->reply));
}
}
if (raise_error && reply[0] != '-') raise_error = 0;
redisProtocolToLuaType(lua,reply);
/* If the debugger is active, log the reply from Redis. */
if (ldb.active && ldb.step)
ldbLogRedisReply(reply);
/* Sort the output array if needed, assuming it is a non-null multi bulk
* reply as expected. */
if ((cmd->flags & CMD_SORT_FOR_SCRIPT) &&
(server.lua_replicate_commands == 0) &&
(reply[0] == '*' && reply[1] != '-')) {
luaSortArray(lua);
}
if (reply != c->buf) sdsfree(reply);
c->reply_bytes = 0;
cleanup:
/* Clean up. Command code may have changed argv/argc so we use the
* argv/argc of the client instead of the local variables. */
for (j = 0; j < c->argc; j++) {
robj *o = c->argv[j];
/* Try to cache the object in the cached_objects array.
* The object must be small, SDS-encoded, and with refcount = 1
* (we must be the only owner) for us to cache it. */
if (j < LUA_CMD_OBJCACHE_SIZE &&
o->refcount == 1 &&
(o->encoding == OBJ_ENCODING_RAW ||
o->encoding == OBJ_ENCODING_EMBSTR) &&
sdslen(o->ptr) <= LUA_CMD_OBJCACHE_MAX_LEN)
{
sds s = o->ptr;
if (cached_objects[j]) decrRefCount(cached_objects[j]);
cached_objects[j] = o;
cached_objects_len[j] = sdsalloc(s);
} else {
decrRefCount(o);
}
}
if (c->argv != argv) {
zfree(c->argv);
argv = NULL;
argv_size = 0;
}
c->user = NULL;
if (raise_error) {
/* If we are here we should have an error in the stack, in the
* form of a table with an "err" field. Extract the string to
* return the plain error. */
inuse--;
return luaRaiseError(lua);
}
inuse--;
return 1;
}
/* redis.call() */
int luaRedisCallCommand(lua_State *lua) {
return luaRedisGenericCommand(lua,1);
}
/* redis.pcall() */
int luaRedisPCallCommand(lua_State *lua) {
return luaRedisGenericCommand(lua,0);
}
/* This adds redis.sha1hex(string) to Lua scripts using the same hashing
* function used for sha1ing lua scripts. */
int luaRedisSha1hexCommand(lua_State *lua) {
int argc = lua_gettop(lua);
char digest[41];
size_t len;
char *s;
if (argc != 1) {
lua_pushstring(lua, "wrong number of arguments");
return lua_error(lua);
}
s = (char*)lua_tolstring(lua,1,&len);
sha1hex(digest,s,len);
lua_pushstring(lua,digest);
return 1;
}
/* Returns a table with a single field 'field' set to the string value
* passed as argument. This helper function is handy when returning
* a Redis Protocol error or status reply from Lua:
*
* return redis.error_reply("ERR Some Error")
* return redis.status_reply("ERR Some Error")
*/
int luaRedisReturnSingleFieldTable(lua_State *lua, char *field) {
if (lua_gettop(lua) != 1 || lua_type(lua,-1) != LUA_TSTRING) {
luaPushError(lua, "wrong number or type of arguments");
return 1;
}
lua_newtable(lua);
lua_pushstring(lua, field);
lua_pushvalue(lua, -3);
lua_settable(lua, -3);
return 1;
}
/* redis.error_reply() */
int luaRedisErrorReplyCommand(lua_State *lua) {
return luaRedisReturnSingleFieldTable(lua,"err");
}
/* redis.status_reply() */
int luaRedisStatusReplyCommand(lua_State *lua) {
return luaRedisReturnSingleFieldTable(lua,"ok");
}
/* redis.replicate_commands()
*
* Turn on single commands replication if the script never called
* a write command so far, and returns true. Otherwise if the script
* already started to write, returns false and stick to whole scripts
* replication, which is our default. */
int luaRedisReplicateCommandsCommand(lua_State *lua) {
if (server.lua_write_dirty) {
lua_pushboolean(lua,0);
} else {
server.lua_replicate_commands = 1;
/* When we switch to single commands replication, we can provide
* different math.random() sequences at every call, which is what
* the user normally expects. */
redisSrand48(rand());
lua_pushboolean(lua,1);
}
return 1;
}
/* redis.breakpoint() /* redis.breakpoint()
* *
* Allows to stop execution during a debugging session from within * Allows to stop execution during a debugging session from within
...@@ -1176,165 +138,26 @@ int luaRedisDebugCommand(lua_State *lua) { ...@@ -1176,165 +138,26 @@ int luaRedisDebugCommand(lua_State *lua) {
return 0; return 0;
} }
/* redis.set_repl() /* redis.replicate_commands()
* *
* Set the propagation of write commands executed in the context of the * Turn on single commands replication if the script never called
* script to on/off for AOF and slaves. */ * a write command so far, and returns true. Otherwise if the script
int luaRedisSetReplCommand(lua_State *lua) { * already started to write, returns false and stick to whole scripts
int argc = lua_gettop(lua); * replication, which is our default. */
int flags; int luaRedisReplicateCommandsCommand(lua_State *lua) {
scriptRunCtx* rctx = luaGetFromRegistry(lua, REGISTRY_RUN_CTX_NAME);
if (server.lua_replicate_commands == 0) { if (rctx->flags & SCRIPT_WRITE_DIRTY) {
lua_pushstring(lua, "You can set the replication behavior only after turning on single commands replication with redis.replicate_commands()."); lua_pushboolean(lua,0);
return lua_error(lua); } else {
} else if (argc != 1) { lctx.lua_replicate_commands = 1;
lua_pushstring(lua, "redis.set_repl() requires two arguments."); rctx->flags &= ~SCRIPT_EVAL_REPLICATION;
return lua_error(lua); /* When we switch to single commands replication, we can provide
} * different math.random() sequences at every call, which is what
* the user normally expects. */
flags = lua_tonumber(lua,-1); redisSrand48(rand());
if ((flags & ~(PROPAGATE_AOF|PROPAGATE_REPL)) != 0) { lua_pushboolean(lua,1);
lua_pushstring(lua, "Invalid replication flags. Use REPL_AOF, REPL_REPLICA, REPL_ALL or REPL_NONE.");
return lua_error(lua);
}
server.lua_repl = flags;
return 0;
}
/* redis.log() */
int luaLogCommand(lua_State *lua) {
int j, argc = lua_gettop(lua);
int level;
sds log;
if (argc < 2) {
lua_pushstring(lua, "redis.log() requires two arguments or more.");
return lua_error(lua);
} else if (!lua_isnumber(lua,-argc)) {
lua_pushstring(lua, "First argument must be a number (log level).");
return lua_error(lua);
}
level = lua_tonumber(lua,-argc);
if (level < LL_DEBUG || level > LL_WARNING) {
lua_pushstring(lua, "Invalid debug level.");
return lua_error(lua);
}
if (level < server.verbosity) return 0;
/* Glue together all the arguments */
log = sdsempty();
for (j = 1; j < argc; j++) {
size_t len;
char *s;
s = (char*)lua_tolstring(lua,(-argc)+j,&len);
if (s) {
if (j != 1) log = sdscatlen(log," ",1);
log = sdscatlen(log,s,len);
}
}
serverLogRaw(level,log);
sdsfree(log);
return 0;
}
/* redis.setresp() */
int luaSetResp(lua_State *lua) {
int argc = lua_gettop(lua);
if (argc != 1) {
lua_pushstring(lua, "redis.setresp() requires one argument.");
return lua_error(lua);
}
int resp = lua_tonumber(lua,-argc);
if (resp != 2 && resp != 3) {
lua_pushstring(lua, "RESP version must be 2 or 3.");
return lua_error(lua);
} }
return 1;
server.lua_client->resp = resp;
return 0;
}
/* ---------------------------------------------------------------------------
* Lua engine initialization and reset.
* ------------------------------------------------------------------------- */
void luaLoadLib(lua_State *lua, const char *libname, lua_CFunction luafunc) {
lua_pushcfunction(lua, luafunc);
lua_pushstring(lua, libname);
lua_call(lua, 1, 0);
}
LUALIB_API int (luaopen_cjson) (lua_State *L);
LUALIB_API int (luaopen_struct) (lua_State *L);
LUALIB_API int (luaopen_cmsgpack) (lua_State *L);
LUALIB_API int (luaopen_bit) (lua_State *L);
void luaLoadLibraries(lua_State *lua) {
luaLoadLib(lua, "", luaopen_base);
luaLoadLib(lua, LUA_TABLIBNAME, luaopen_table);
luaLoadLib(lua, LUA_STRLIBNAME, luaopen_string);
luaLoadLib(lua, LUA_MATHLIBNAME, luaopen_math);
luaLoadLib(lua, LUA_DBLIBNAME, luaopen_debug);
luaLoadLib(lua, "cjson", luaopen_cjson);
luaLoadLib(lua, "struct", luaopen_struct);
luaLoadLib(lua, "cmsgpack", luaopen_cmsgpack);
luaLoadLib(lua, "bit", luaopen_bit);
#if 0 /* Stuff that we don't load currently, for sandboxing concerns. */
luaLoadLib(lua, LUA_LOADLIBNAME, luaopen_package);
luaLoadLib(lua, LUA_OSLIBNAME, luaopen_os);
#endif
}
/* Remove a functions that we don't want to expose to the Redis scripting
* environment. */
void luaRemoveUnsupportedFunctions(lua_State *lua) {
lua_pushnil(lua);
lua_setglobal(lua,"loadfile");
lua_pushnil(lua);
lua_setglobal(lua,"dofile");
}
/* This function installs metamethods in the global table _G that prevent
* the creation of globals accidentally.
*
* It should be the last to be called in the scripting engine initialization
* sequence, because it may interact with creation of globals. */
void scriptingEnableGlobalsProtection(lua_State *lua) {
char *s[32];
sds code = sdsempty();
int j = 0;
/* strict.lua from: http://metalua.luaforge.net/src/lib/strict.lua.html.
* Modified to be adapted to Redis. */
s[j++]="local dbg=debug\n";
s[j++]="local mt = {}\n";
s[j++]="setmetatable(_G, mt)\n";
s[j++]="mt.__newindex = function (t, n, v)\n";
s[j++]=" if dbg.getinfo(2) then\n";
s[j++]=" local w = dbg.getinfo(2, \"S\").what\n";
s[j++]=" if w ~= \"main\" and w ~= \"C\" then\n";
s[j++]=" error(\"Script attempted to create global variable '\"..tostring(n)..\"'\", 2)\n";
s[j++]=" end\n";
s[j++]=" end\n";
s[j++]=" rawset(t, n, v)\n";
s[j++]="end\n";
s[j++]="mt.__index = function (t, n)\n";
s[j++]=" if dbg.getinfo(2) and dbg.getinfo(2, \"S\").what ~= \"C\" then\n";
s[j++]=" error(\"Script attempted to access nonexistent global variable '\"..tostring(n)..\"'\", 2)\n";
s[j++]=" end\n";
s[j++]=" return rawget(t, n)\n";
s[j++]="end\n";
s[j++]="debug = nil\n";
s[j++]=NULL;
for (j = 0; s[j] != NULL; j++) code = sdscatlen(code,s[j],strlen(s[j]));
luaL_loadbuffer(lua,code,sdslen(code),"@enable_strict_lua");
lua_pcall(lua,0,0,0);
sdsfree(code);
} }
/* Initialize the scripting environment. /* Initialize the scripting environment.
...@@ -1351,104 +174,23 @@ void scriptingInit(int setup) { ...@@ -1351,104 +174,23 @@ void scriptingInit(int setup) {
lua_State *lua = lua_open(); lua_State *lua = lua_open();
if (setup) { if (setup) {
server.lua_client = NULL; lctx.lua_client = NULL;
server.lua_caller = NULL; server.script_caller = NULL;
server.lua_cur_script = NULL; lctx.lua_cur_script = NULL;
server.lua_timedout = 0; server.script_disable_deny_script = 0;
server.lua_disable_deny_script = 0;
ldbInit(); ldbInit();
} }
luaLoadLibraries(lua);
luaRemoveUnsupportedFunctions(lua);
/* Initialize a dictionary we use to map SHAs to scripts. /* Initialize a dictionary we use to map SHAs to scripts.
* This is useful for replication, as we need to replicate EVALSHA * This is useful for replication, as we need to replicate EVALSHA
* as EVAL, so we need to remember the associated script. */ * as EVAL, so we need to remember the associated script. */
server.lua_scripts = dictCreate(&shaScriptObjectDictType); lctx.lua_scripts = dictCreate(&shaScriptObjectDictType);
server.lua_scripts_mem = 0; lctx.lua_scripts_mem = 0;
/* Register the redis commands table and fields */
lua_newtable(lua);
/* redis.call */
lua_pushstring(lua,"call");
lua_pushcfunction(lua,luaRedisCallCommand);
lua_settable(lua,-3);
/* redis.pcall */
lua_pushstring(lua,"pcall");
lua_pushcfunction(lua,luaRedisPCallCommand);
lua_settable(lua,-3);
/* redis.log and log levels. */
lua_pushstring(lua,"log");
lua_pushcfunction(lua,luaLogCommand);
lua_settable(lua,-3);
/* redis.setresp */
lua_pushstring(lua,"setresp");
lua_pushcfunction(lua,luaSetResp);
lua_settable(lua,-3);
lua_pushstring(lua,"LOG_DEBUG");
lua_pushnumber(lua,LL_DEBUG);
lua_settable(lua,-3);
lua_pushstring(lua,"LOG_VERBOSE"); luaRegisterRedisAPI(lua);
lua_pushnumber(lua,LL_VERBOSE);
lua_settable(lua,-3);
lua_pushstring(lua,"LOG_NOTICE");
lua_pushnumber(lua,LL_NOTICE);
lua_settable(lua,-3);
lua_pushstring(lua,"LOG_WARNING");
lua_pushnumber(lua,LL_WARNING);
lua_settable(lua,-3);
/* redis.sha1hex */
lua_pushstring(lua, "sha1hex");
lua_pushcfunction(lua, luaRedisSha1hexCommand);
lua_settable(lua, -3);
/* redis.error_reply and redis.status_reply */ /* register debug commands */
lua_pushstring(lua, "error_reply"); lua_getglobal(lua,"redis");
lua_pushcfunction(lua, luaRedisErrorReplyCommand);
lua_settable(lua, -3);
lua_pushstring(lua, "status_reply");
lua_pushcfunction(lua, luaRedisStatusReplyCommand);
lua_settable(lua, -3);
/* redis.replicate_commands */
lua_pushstring(lua, "replicate_commands");
lua_pushcfunction(lua, luaRedisReplicateCommandsCommand);
lua_settable(lua, -3);
/* redis.set_repl and associated flags. */
lua_pushstring(lua,"set_repl");
lua_pushcfunction(lua,luaRedisSetReplCommand);
lua_settable(lua,-3);
lua_pushstring(lua,"REPL_NONE");
lua_pushnumber(lua,PROPAGATE_NONE);
lua_settable(lua,-3);
lua_pushstring(lua,"REPL_AOF");
lua_pushnumber(lua,PROPAGATE_AOF);
lua_settable(lua,-3);
lua_pushstring(lua,"REPL_SLAVE");
lua_pushnumber(lua,PROPAGATE_REPL);
lua_settable(lua,-3);
lua_pushstring(lua,"REPL_REPLICA");
lua_pushnumber(lua,PROPAGATE_REPL);
lua_settable(lua,-3);
lua_pushstring(lua,"REPL_ALL");
lua_pushnumber(lua,PROPAGATE_AOF|PROPAGATE_REPL);
lua_settable(lua,-3);
/* redis.breakpoint */ /* redis.breakpoint */
lua_pushstring(lua,"breakpoint"); lua_pushstring(lua,"breakpoint");
...@@ -1460,21 +202,12 @@ void scriptingInit(int setup) { ...@@ -1460,21 +202,12 @@ void scriptingInit(int setup) {
lua_pushcfunction(lua,luaRedisDebugCommand); lua_pushcfunction(lua,luaRedisDebugCommand);
lua_settable(lua,-3); lua_settable(lua,-3);
/* Finally set the table as 'redis' global var. */ /* redis.replicate_commands */
lua_setglobal(lua,"redis"); lua_pushstring(lua, "replicate_commands");
lua_pushcfunction(lua, luaRedisReplicateCommandsCommand);
/* Replace math.random and math.randomseed with our implementations. */ lua_settable(lua, -3);
lua_getglobal(lua,"math");
lua_pushstring(lua,"random");
lua_pushcfunction(lua,redis_math_random);
lua_settable(lua,-3);
lua_pushstring(lua,"randomseed");
lua_pushcfunction(lua,redis_math_randomseed);
lua_settable(lua,-3);
lua_setglobal(lua,"math"); lua_setglobal(lua,"redis");
/* Add a helper function that we use to sort the multi bulk output of non /* Add a helper function that we use to sort the multi bulk output of non
* deterministic commands, when containing 'false' elements. */ * deterministic commands, when containing 'false' elements. */
...@@ -1513,31 +246,31 @@ void scriptingInit(int setup) { ...@@ -1513,31 +246,31 @@ void scriptingInit(int setup) {
* inside the Lua interpreter. * inside the Lua interpreter.
* Note: there is no need to create it again when this function is called * Note: there is no need to create it again when this function is called
* by scriptingReset(). */ * by scriptingReset(). */
if (server.lua_client == NULL) { if (lctx.lua_client == NULL) {
server.lua_client = createClient(NULL); lctx.lua_client = createClient(NULL);
server.lua_client->flags |= CLIENT_LUA; lctx.lua_client->flags |= CLIENT_SCRIPT;
/* We do not want to allow blocking commands inside Lua */ /* We do not want to allow blocking commands inside Lua */
server.lua_client->flags |= CLIENT_DENY_BLOCKING; lctx.lua_client->flags |= CLIENT_DENY_BLOCKING;
} }
/* Lua beginners often don't use "local", this is likely to introduce /* Lua beginners often don't use "local", this is likely to introduce
* subtle bugs in their code. To prevent problems we protect accesses * subtle bugs in their code. To prevent problems we protect accesses
* to global variables. */ * to global variables. */
scriptingEnableGlobalsProtection(lua); luaEnableGlobalsProtection(lua, 1);
server.lua = lua; lctx.lua = lua;
} }
/* Release resources related to Lua scripting. /* Release resources related to Lua scripting.
* This function is used in order to reset the scripting environment. */ * This function is used in order to reset the scripting environment. */
void scriptingRelease(int async) { void scriptingRelease(int async) {
if (async) if (async)
freeLuaScriptsAsync(server.lua_scripts); freeLuaScriptsAsync(lctx.lua_scripts);
else else
dictRelease(server.lua_scripts); dictRelease(lctx.lua_scripts);
server.lua_scripts_mem = 0; lctx.lua_scripts_mem = 0;
lua_close(server.lua); lua_close(lctx.lua);
} }
void scriptingReset(int async) { void scriptingReset(int async) {
...@@ -1545,62 +278,6 @@ void scriptingReset(int async) { ...@@ -1545,62 +278,6 @@ void scriptingReset(int async) {
scriptingInit(0); scriptingInit(0);
} }
/* Set an array of Redis String Objects as a Lua array (table) stored into a
* global variable. */
void luaSetGlobalArray(lua_State *lua, char *var, robj **elev, int elec) {
int j;
lua_newtable(lua);
for (j = 0; j < elec; j++) {
lua_pushlstring(lua,(char*)elev[j]->ptr,sdslen(elev[j]->ptr));
lua_rawseti(lua,-2,j+1);
}
lua_setglobal(lua,var);
}
/* ---------------------------------------------------------------------------
* Redis provided math.random
* ------------------------------------------------------------------------- */
/* We replace math.random() with our implementation that is not affected
* by specific libc random() implementations and will output the same sequence
* (for the same seed) in every arch. */
/* The following implementation is the one shipped with Lua itself but with
* rand() replaced by redisLrand48(). */
int redis_math_random (lua_State *L) {
/* the `%' avoids the (rare) case of r==1, and is needed also because on
some systems (SunOS!) `rand()' may return a value larger than RAND_MAX */
lua_Number r = (lua_Number)(redisLrand48()%REDIS_LRAND48_MAX) /
(lua_Number)REDIS_LRAND48_MAX;
switch (lua_gettop(L)) { /* check number of arguments */
case 0: { /* no arguments */
lua_pushnumber(L, r); /* Number between 0 and 1 */
break;
}
case 1: { /* only upper limit */
int u = luaL_checkint(L, 1);
luaL_argcheck(L, 1<=u, 1, "interval is empty");
lua_pushnumber(L, floor(r*u)+1); /* int between 1 and `u' */
break;
}
case 2: { /* lower and upper limits */
int l = luaL_checkint(L, 1);
int u = luaL_checkint(L, 2);
luaL_argcheck(L, l<=u, 2, "interval is empty");
lua_pushnumber(L, floor(r*(u-l+1))+l); /* int between `l' and `u' */
break;
}
default: return luaL_error(L, "wrong number of arguments");
}
return 1;
}
int redis_math_randomseed (lua_State *L) {
redisSrand48(luaL_checkint(L, 1));
return 0;
}
/* --------------------------------------------------------------------------- /* ---------------------------------------------------------------------------
* EVAL and SCRIPT commands implementation * EVAL and SCRIPT commands implementation
* ------------------------------------------------------------------------- */ * ------------------------------------------------------------------------- */
...@@ -1622,7 +299,7 @@ int redis_math_randomseed (lua_State *L) { ...@@ -1622,7 +299,7 @@ int redis_math_randomseed (lua_State *L) {
* *
* If 'c' is not NULL, on error the client is informed with an appropriate * If 'c' is not NULL, on error the client is informed with an appropriate
* error describing the nature of the problem and the Lua interpreter error. */ * error describing the nature of the problem and the Lua interpreter error. */
sds luaCreateFunction(client *c, lua_State *lua, robj *body) { sds luaCreateFunction(client *c, robj *body) {
char funcname[43]; char funcname[43];
dictEntry *de; dictEntry *de;
...@@ -1631,7 +308,7 @@ sds luaCreateFunction(client *c, lua_State *lua, robj *body) { ...@@ -1631,7 +308,7 @@ sds luaCreateFunction(client *c, lua_State *lua, robj *body) {
sha1hex(funcname+2,body->ptr,sdslen(body->ptr)); sha1hex(funcname+2,body->ptr,sdslen(body->ptr));
sds sha = sdsnewlen(funcname+2,40); sds sha = sdsnewlen(funcname+2,40);
if ((de = dictFind(server.lua_scripts,sha)) != NULL) { if ((de = dictFind(lctx.lua_scripts,sha)) != NULL) {
sdsfree(sha); sdsfree(sha);
return dictGetKey(de); return dictGetKey(de);
} }
...@@ -1643,25 +320,25 @@ sds luaCreateFunction(client *c, lua_State *lua, robj *body) { ...@@ -1643,25 +320,25 @@ sds luaCreateFunction(client *c, lua_State *lua, robj *body) {
funcdef = sdscatlen(funcdef,body->ptr,sdslen(body->ptr)); funcdef = sdscatlen(funcdef,body->ptr,sdslen(body->ptr));
funcdef = sdscatlen(funcdef,"\nend",4); funcdef = sdscatlen(funcdef,"\nend",4);
if (luaL_loadbuffer(lua,funcdef,sdslen(funcdef),"@user_script")) { if (luaL_loadbuffer(lctx.lua,funcdef,sdslen(funcdef),"@user_script")) {
if (c != NULL) { if (c != NULL) {
addReplyErrorFormat(c, addReplyErrorFormat(c,
"Error compiling script (new function): %s\n", "Error compiling script (new function): %s\n",
lua_tostring(lua,-1)); lua_tostring(lctx.lua,-1));
} }
lua_pop(lua,1); lua_pop(lctx.lua,1);
sdsfree(sha); sdsfree(sha);
sdsfree(funcdef); sdsfree(funcdef);
return NULL; return NULL;
} }
sdsfree(funcdef); sdsfree(funcdef);
if (lua_pcall(lua,0,0,0)) { if (lua_pcall(lctx.lua,0,0,0)) {
if (c != NULL) { if (c != NULL) {
addReplyErrorFormat(c,"Error running script (new function): %s\n", addReplyErrorFormat(c,"Error running script (new function): %s\n",
lua_tostring(lua,-1)); lua_tostring(lctx.lua,-1));
} }
lua_pop(lua,1); lua_pop(lctx.lua,1);
sdsfree(sha); sdsfree(sha);
return NULL; return NULL;
} }
...@@ -1669,92 +346,40 @@ sds luaCreateFunction(client *c, lua_State *lua, robj *body) { ...@@ -1669,92 +346,40 @@ sds luaCreateFunction(client *c, lua_State *lua, robj *body) {
/* We also save a SHA1 -> Original script map in a dictionary /* We also save a SHA1 -> Original script map in a dictionary
* so that we can replicate / write in the AOF all the * so that we can replicate / write in the AOF all the
* EVALSHA commands as EVAL using the original script. */ * EVALSHA commands as EVAL using the original script. */
int retval = dictAdd(server.lua_scripts,sha,body); int retval = dictAdd(lctx.lua_scripts,sha,body);
serverAssertWithInfo(c ? c : server.lua_client,NULL,retval == DICT_OK); serverAssertWithInfo(c ? c : lctx.lua_client,NULL,retval == DICT_OK);
server.lua_scripts_mem += sdsZmallocSize(sha) + getStringObjectSdsUsedMemory(body); lctx.lua_scripts_mem += sdsZmallocSize(sha) + getStringObjectSdsUsedMemory(body);
incrRefCount(body); incrRefCount(body);
return sha; return sha;
} }
/* This is the Lua script "count" hook that we use to detect scripts timeout. */
void luaMaskCountHook(lua_State *lua, lua_Debug *ar) {
long long elapsed = elapsedMs(server.lua_time_start);
UNUSED(ar);
UNUSED(lua);
/* Set the timeout condition if not already set and the maximum
* execution time was reached. */
if (elapsed >= server.lua_time_limit && server.lua_timedout == 0) {
serverLog(LL_WARNING,
"Lua slow script detected: still in execution after %lld milliseconds. "
"You can try killing the script using the SCRIPT KILL command. "
"Script SHA1 is: %s",
elapsed, server.lua_cur_script);
server.lua_timedout = 1;
blockingOperationStarts();
/* Once the script timeouts we reenter the event loop to permit others
* to call SCRIPT KILL or SHUTDOWN NOSAVE if needed. For this reason
* we need to mask the client executing the script from the event loop.
* If we don't do that the client may disconnect and could no longer be
* here when the EVAL command will return. */
protectClient(server.lua_caller);
}
if (server.lua_timedout) processEventsWhileBlocked();
if (server.lua_kill) {
serverLog(LL_WARNING,"Lua script killed by user with SCRIPT KILL.");
/*
* Set the hook to invoke all the time so the user
         * will not be able to catch the error with pcall and invoke
         * pcall again which will prevent the script from ever been killed
*/
lua_sethook(lua, luaMaskCountHook, LUA_MASKLINE, 0);
lua_pushstring(lua,"Script killed by user with SCRIPT KILL...");
lua_error(lua);
}
}
void prepareLuaClient(void) { void prepareLuaClient(void) {
/* Select the right DB in the context of the Lua client */ /* Select the right DB in the context of the Lua client */
selectDb(server.lua_client,server.lua_caller->db->id); selectDb(lctx.lua_client,server.script_caller->db->id);
server.lua_client->resp = 2; /* Default is RESP2, scripts can change it. */ lctx.lua_client->resp = 2; /* Default is RESP2, scripts can change it. */
/* If we are in MULTI context, flag Lua client as CLIENT_MULTI. */ /* If we are in MULTI context, flag Lua client as CLIENT_MULTI. */
if (server.lua_caller->flags & CLIENT_MULTI) { if (server.script_caller->flags & CLIENT_MULTI) {
server.lua_client->flags |= CLIENT_MULTI; lctx.lua_client->flags |= CLIENT_MULTI;
} }
} }
void resetLuaClient(void) { void resetLuaClient(void) {
/* After the script done, remove the MULTI state. */ /* After the script done, remove the MULTI state. */
server.lua_client->flags &= ~CLIENT_MULTI; lctx.lua_client->flags &= ~CLIENT_MULTI;
} }
void evalGenericCommand(client *c, int evalsha) { void evalGenericCommand(client *c, int evalsha) {
lua_State *lua = server.lua; lua_State *lua = lctx.lua;
char funcname[43]; char funcname[43];
long long numkeys; long long numkeys;
long long initial_server_dirty = server.dirty; long long initial_server_dirty = server.dirty;
int delhook = 0, err;
/* When we replicate whole scripts, we want the same PRNG sequence at /* When we replicate whole scripts, we want the same PRNG sequence at
* every call so that our PRNG is not affected by external state. */ * every call so that our PRNG is not affected by external state. */
redisSrand48(0); redisSrand48(0);
/* We set this flag to zero to remember that so far no random command lctx.lua_replicate_commands = server.lua_always_replicate_commands;
* was called. This way we can allow the user to call commands like
* SRANDMEMBER or RANDOMKEY from Lua scripts as far as no write command
* is called (otherwise the replication and AOF would end with non
* deterministic sequences).
*
* Thanks to this flag we'll raise an error every time a write command
* is called after a random command was used. */
server.lua_random_dirty = 0;
server.lua_write_dirty = 0;
server.lua_replicate_commands = server.lua_always_replicate_commands;
server.lua_multi_emitted = 0;
server.lua_repl = PROPAGATE_AOF|PROPAGATE_REPL;
/* Get the number of arguments that are keys */ /* Get the number of arguments that are keys */
if (getLongLongFromObjectOrReply(c,c->argv[2],&numkeys,NULL) != C_OK) if (getLongLongFromObjectOrReply(c,c->argv[2],&numkeys,NULL) != C_OK)
...@@ -1803,7 +428,7 @@ void evalGenericCommand(client *c, int evalsha) { ...@@ -1803,7 +428,7 @@ void evalGenericCommand(client *c, int evalsha) {
addReplyErrorObject(c, shared.noscripterr); addReplyErrorObject(c, shared.noscripterr);
return; return;
} }
if (luaCreateFunction(c,lua,c->argv[1]) == NULL) { if (luaCreateFunction(c,c->argv[1]) == NULL) {
lua_pop(lua,1); /* remove the error handler from the stack. */ lua_pop(lua,1); /* remove the error handler from the stack. */
/* The error is sent to the client by luaCreateFunction() /* The error is sent to the client by luaCreateFunction()
* itself when it returns NULL. */ * itself when it returns NULL. */
...@@ -1814,92 +439,24 @@ void evalGenericCommand(client *c, int evalsha) { ...@@ -1814,92 +439,24 @@ void evalGenericCommand(client *c, int evalsha) {
serverAssert(!lua_isnil(lua,-1)); serverAssert(!lua_isnil(lua,-1));
} }
/* Populate the argv and keys table accordingly to the arguments that lctx.lua_cur_script = funcname + 2;
* EVAL received. */
luaSetGlobalArray(lua,"KEYS",c->argv+3,numkeys);
luaSetGlobalArray(lua,"ARGV",c->argv+3+numkeys,c->argc-3-numkeys);
/* Set a hook in order to be able to stop the script execution if it scriptRunCtx rctx;
* is running for too much time. scriptPrepareForRun(&rctx, lctx.lua_client, c, lctx.lua_cur_script);
* We set the hook only if the time limit is enabled as the hook will rctx.flags |= SCRIPT_EVAL_MODE; /* mark the current run as legacy so we
* make the Lua script execution slower. will get legacy error messages and logs */
* if (!lctx.lua_replicate_commands) rctx.flags |= SCRIPT_EVAL_REPLICATION;
* If we are debugging, we set instead a "line" hook so that the /* This check is for EVAL_RO, EVALSHA_RO. We want to allow only read only commands */
* debugger is call-back at every line executed by the script. */ if ((server.script_caller->cmd->proc == evalRoCommand ||
server.in_eval = 1; server.script_caller->cmd->proc == evalShaRoCommand)) {
server.lua_caller = c; rctx.flags |= SCRIPT_READ_ONLY;
server.lua_cur_script = funcname + 2;
server.lua_time_start = getMonotonicUs();
server.lua_time_snapshot = mstime();
server.lua_kill = 0;
if (server.lua_time_limit > 0 && ldb.active == 0) {
lua_sethook(lua,luaMaskCountHook,LUA_MASKCOUNT,100000);
delhook = 1;
} else if (ldb.active) {
lua_sethook(server.lua,luaLdbLineHook,LUA_MASKLINE|LUA_MASKCOUNT,100000);
delhook = 1;
}
prepareLuaClient();
/* At this point whether this script was never seen before or if it was
* already defined, we can call it. We have zero arguments and expect
* a single return value. */
err = lua_pcall(lua,0,1,-2);
resetLuaClient();
/* Perform some cleanup that we need to do both on error and success. */
if (delhook) lua_sethook(lua,NULL,0,0); /* Disable hook */
if (server.lua_timedout) {
server.lua_timedout = 0;
blockingOperationEnds();
/* Restore the client that was protected when the script timeout
* was detected. */
unprotectClient(c);
if (server.masterhost && server.master)
queueClientForReprocessing(server.master);
}
server.in_eval = 0;
server.lua_caller = NULL;
server.lua_cur_script = NULL;
/* Call the Lua garbage collector from time to time to avoid a
* full cycle performed by Lua, which adds too latency.
*
* The call is performed every LUA_GC_CYCLE_PERIOD executed commands
* (and for LUA_GC_CYCLE_PERIOD collection steps) because calling it
* for every command uses too much CPU. */
#define LUA_GC_CYCLE_PERIOD 50
{
static long gc_count = 0;
gc_count++;
if (gc_count == LUA_GC_CYCLE_PERIOD) {
lua_gc(lua,LUA_GCSTEP,LUA_GC_CYCLE_PERIOD);
gc_count = 0;
}
} }
if (err) { luaCallFunction(&rctx, lua, c->argv+3, numkeys, c->argv+3+numkeys, c->argc-3-numkeys, ldb.active);
addReplyErrorFormat(c,"Error running script (call to %s): %s\n",
funcname, lua_tostring(lua,-1));
lua_pop(lua,2); /* Consume the Lua reply and remove error handler. */
} else {
/* On success convert the Lua return value into Redis protocol, and
* send it to * the client. */
luaReplyToRedisReply(c,lua); /* Convert and consume the reply. */
lua_pop(lua,1); /* Remove the error handler. */ lua_pop(lua,1); /* Remove the error handler. */
} scriptResetRun(&rctx);
/* If we are using single commands replication, emit EXEC if there lctx.lua_cur_script = NULL;
* was at least a write. */
if (server.lua_replicate_commands) {
preventCommandPropagation(c);
if (server.lua_multi_emitted) {
execCommandPropagateExec(c->db->id);
}
}
/* EVALSHA should be propagated to Slave and AOF file as full EVAL, unless /* EVALSHA should be propagated to Slave and AOF file as full EVAL, unless
* we are sure that the script was already in the context of all the * we are sure that the script was already in the context of all the
...@@ -1911,12 +468,12 @@ void evalGenericCommand(client *c, int evalsha) { ...@@ -1911,12 +468,12 @@ void evalGenericCommand(client *c, int evalsha) {
* For replication, every time a new slave attaches to the master, we need to * For replication, every time a new slave attaches to the master, we need to
* flush our cache of scripts that can be replicated as EVALSHA, while * flush our cache of scripts that can be replicated as EVALSHA, while
* for AOF we need to do so every time we rewrite the AOF file. */ * for AOF we need to do so every time we rewrite the AOF file. */
if (evalsha && !server.lua_replicate_commands) { if (evalsha && !lctx.lua_replicate_commands) {
if (!replicationScriptCacheExists(c->argv[1]->ptr)) { if (!replicationScriptCacheExists(c->argv[1]->ptr)) {
/* This script is not in our script cache, replicate it as /* This script is not in our script cache, replicate it as
* EVAL, then add it into the script cache, as from now on * EVAL, then add it into the script cache, as from now on
* slaves and AOF know about it. */ * slaves and AOF know about it. */
robj *script = dictFetchValue(server.lua_scripts,c->argv[1]->ptr); robj *script = dictFetchValue(lctx.lua_scripts,c->argv[1]->ptr);
replicationScriptCacheAdd(c->argv[1]->ptr); replicationScriptCacheAdd(c->argv[1]->ptr);
serverAssertWithInfo(c,NULL,script != NULL); serverAssertWithInfo(c,NULL,script != NULL);
...@@ -2018,27 +575,18 @@ NULL ...@@ -2018,27 +575,18 @@ NULL
addReplyArrayLen(c, c->argc-2); addReplyArrayLen(c, c->argc-2);
for (j = 2; j < c->argc; j++) { for (j = 2; j < c->argc; j++) {
if (dictFind(server.lua_scripts,c->argv[j]->ptr)) if (dictFind(lctx.lua_scripts,c->argv[j]->ptr))
addReply(c,shared.cone); addReply(c,shared.cone);
else else
addReply(c,shared.czero); addReply(c,shared.czero);
} }
} else if (c->argc == 3 && !strcasecmp(c->argv[1]->ptr,"load")) { } else if (c->argc == 3 && !strcasecmp(c->argv[1]->ptr,"load")) {
sds sha = luaCreateFunction(c,server.lua,c->argv[2]); sds sha = luaCreateFunction(c,c->argv[2]);
if (sha == NULL) return; /* The error was sent by luaCreateFunction(). */ if (sha == NULL) return; /* The error was sent by luaCreateFunction(). */
addReplyBulkCBuffer(c,sha,40); addReplyBulkCBuffer(c,sha,40);
forceCommandPropagation(c,PROPAGATE_REPL|PROPAGATE_AOF); forceCommandPropagation(c,PROPAGATE_REPL|PROPAGATE_AOF);
} else if (c->argc == 2 && !strcasecmp(c->argv[1]->ptr,"kill")) { } else if (c->argc == 2 && !strcasecmp(c->argv[1]->ptr,"kill")) {
if (server.lua_caller == NULL) { scriptKill(c, 1);
addReplyError(c,"-NOTBUSY No scripts in execution right now.");
} else if (server.lua_caller->flags & CLIENT_MASTER) {
addReplyError(c,"-UNKILLABLE The busy script was sent by a master instance in the context of replication and cannot be killed.");
} else if (server.lua_write_dirty) {
addReplyError(c,"-UNKILLABLE Sorry the script already executed write commands against the dataset. You can either wait the script termination or kill the server in a hard way using the SHUTDOWN NOSAVE command.");
} else {
server.lua_kill = 1;
addReply(c,shared.ok);
}
} else if (c->argc == 3 && !strcasecmp(c->argv[1]->ptr,"debug")) { } else if (c->argc == 3 && !strcasecmp(c->argv[1]->ptr,"debug")) {
if (clientHasPendingReplies(c)) { if (clientHasPendingReplies(c)) {
addReplyError(c,"SCRIPT DEBUG must be called outside a pipeline"); addReplyError(c,"SCRIPT DEBUG must be called outside a pipeline");
...@@ -2063,6 +611,26 @@ NULL ...@@ -2063,6 +611,26 @@ NULL
} }
} }
unsigned long evalMemory() {
return luaMemory(lctx.lua);
}
dict* evalScriptsDict() {
return lctx.lua_scripts;
}
unsigned long evalScriptsMemory() {
return lctx.lua_scripts_mem +
dictSize(lctx.lua_scripts) * sizeof(dictEntry) +
dictSlots(lctx.lua_scripts) * sizeof(dictEntry*);
}
/* Returns the time when the script invocation started */
mstime_t evalTimeSnapshot() {
return scriptTimeSnapshot();
}
/* --------------------------------------------------------------------------- /* ---------------------------------------------------------------------------
* LDB: Redis Lua debugging facilities * LDB: Redis Lua debugging facilities
* ------------------------------------------------------------------------- */ * ------------------------------------------------------------------------- */
...@@ -2087,6 +655,10 @@ void ldbFlushLog(list *log) { ...@@ -2087,6 +655,10 @@ void ldbFlushLog(list *log) {
listDelNode(log,ln); listDelNode(log,ln);
} }
int ldbIsEnabled(){
return ldb.active && ldb.step;
}
/* Enable debug mode of Lua scripts for this client. */ /* Enable debug mode of Lua scripts for this client. */
void ldbEnable(client *c) { void ldbEnable(client *c) {
c->flags |= CLIENT_LUA_DEBUG; c->flags |= CLIENT_LUA_DEBUG;
...@@ -2817,7 +1389,7 @@ void ldbEval(lua_State *lua, sds *argv, int argc) { ...@@ -2817,7 +1389,7 @@ void ldbEval(lua_State *lua, sds *argv, int argc) {
* implementation, with ldb.step enabled, so as a side effect the Redis command * implementation, with ldb.step enabled, so as a side effect the Redis command
* and its reply are logged. */ * and its reply are logged. */
void ldbRedis(lua_State *lua, sds *argv, int argc) { void ldbRedis(lua_State *lua, sds *argv, int argc) {
int j, saved_rc = server.lua_replicate_commands; int j, saved_rc = lctx.lua_replicate_commands;
if (!lua_checkstack(lua, argc + 1)) { if (!lua_checkstack(lua, argc + 1)) {
/* Increase the Lua stack if needed to make sure there is enough room /* Increase the Lua stack if needed to make sure there is enough room
...@@ -2836,10 +1408,10 @@ void ldbRedis(lua_State *lua, sds *argv, int argc) { ...@@ -2836,10 +1408,10 @@ void ldbRedis(lua_State *lua, sds *argv, int argc) {
for (j = 1; j < argc; j++) for (j = 1; j < argc; j++)
lua_pushlstring(lua,argv[j],sdslen(argv[j])); lua_pushlstring(lua,argv[j],sdslen(argv[j]));
ldb.step = 1; /* Force redis.call() to log. */ ldb.step = 1; /* Force redis.call() to log. */
server.lua_replicate_commands = 1; lctx.lua_replicate_commands = 1;
lua_pcall(lua,argc-1,1,0); /* Stack: redis, result */ lua_pcall(lua,argc-1,1,0); /* Stack: redis, result */
ldb.step = 0; /* Disable logging. */ ldb.step = 0; /* Disable logging. */
server.lua_replicate_commands = saved_rc; lctx.lua_replicate_commands = saved_rc;
lua_pop(lua,2); /* Discard the result and clean the stack. */ lua_pop(lua,2); /* Discard the result and clean the stack. */
} }
...@@ -3015,6 +1587,7 @@ ldbLog(sdsnew(" next line of code.")); ...@@ -3015,6 +1587,7 @@ ldbLog(sdsnew(" next line of code."));
/* This is the core of our Lua debugger, called each time Lua is about /* This is the core of our Lua debugger, called each time Lua is about
* to start executing a new line. */ * to start executing a new line. */
void luaLdbLineHook(lua_State *lua, lua_Debug *ar) { void luaLdbLineHook(lua_State *lua, lua_Debug *ar) {
scriptRunCtx* rctx = luaGetFromRegistry(lua, REGISTRY_RUN_CTX_NAME);
lua_getstack(lua,0,ar); lua_getstack(lua,0,ar);
lua_getinfo(lua,"Sl",ar); lua_getinfo(lua,"Sl",ar);
ldb.currentline = ar->currentline; ldb.currentline = ar->currentline;
...@@ -3027,9 +1600,9 @@ void luaLdbLineHook(lua_State *lua, lua_Debug *ar) { ...@@ -3027,9 +1600,9 @@ void luaLdbLineHook(lua_State *lua, lua_Debug *ar) {
/* Check if a timeout occurred. */ /* Check if a timeout occurred. */
if (ar->event == LUA_HOOKCOUNT && ldb.step == 0 && bp == 0) { if (ar->event == LUA_HOOKCOUNT && ldb.step == 0 && bp == 0) {
mstime_t elapsed = elapsedMs(server.lua_time_start); mstime_t elapsed = elapsedMs(rctx->start_time);
mstime_t timelimit = server.lua_time_limit ? mstime_t timelimit = server.script_time_limit ?
server.lua_time_limit : 5000; server.script_time_limit : 5000;
if (elapsed >= timelimit) { if (elapsed >= timelimit) {
timeout = 1; timeout = 1;
ldb.step = 1; ldb.step = 1;
...@@ -3057,7 +1630,7 @@ void luaLdbLineHook(lua_State *lua, lua_Debug *ar) { ...@@ -3057,7 +1630,7 @@ void luaLdbLineHook(lua_State *lua, lua_Debug *ar) {
lua_pushstring(lua, "timeout during Lua debugging with client closing connection"); lua_pushstring(lua, "timeout during Lua debugging with client closing connection");
lua_error(lua); lua_error(lua);
} }
server.lua_time_start = getMonotonicUs(); rctx->start_time = getMonotonicUs();
server.lua_time_snapshot = mstime(); rctx->snapshot_time = mstime();
} }
} }
...@@ -33,6 +33,7 @@ ...@@ -33,6 +33,7 @@
#include "server.h" #include "server.h"
#include "bio.h" #include "bio.h"
#include "atomicvar.h" #include "atomicvar.h"
#include "script.h"
#include <math.h> #include <math.h>
/* ---------------------------------------------------------------------------- /* ----------------------------------------------------------------------------
...@@ -472,7 +473,7 @@ static int evictionTimeProc( ...@@ -472,7 +473,7 @@ static int evictionTimeProc(
static int isSafeToPerformEvictions(void) { static int isSafeToPerformEvictions(void) {
/* - There must be no script in timeout condition. /* - There must be no script in timeout condition.
* - Nor we are loading data right now. */ * - Nor we are loading data right now. */
if (server.lua_timedout || server.loading) return 0; if (scriptIsTimedout() || server.loading) return 0;
/* By default replicas should ignore maxmemory /* By default replicas should ignore maxmemory
* and just be masters exact copies. */ * and just be masters exact copies. */
......
/*
* Copyright (c) 2021, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/*
* function_lua.c unit provides the Lua engine functionality.
* Including registering the engine and implementing the engine
* callbacks:
* * Create a function from blob (usually text)
* * Invoke a function
* * Free function memory
* * Get memory usage
*
* Uses script_lua.c to run the Lua code.
*/
#include "functions.h"
#include "script_lua.h"
#include <lua.h>
#include <lauxlib.h>
#include <lualib.h>
#define LUA_ENGINE_NAME "LUA"
#define REGISTRY_ENGINE_CTX_NAME "__ENGINE_CTX__"
#define REGISTRY_ERROR_HANDLER_NAME "__ERROR_HANDLER__"
/* Lua engine ctx */
typedef struct luaEngineCtx {
lua_State *lua;
} luaEngineCtx;
/* Lua function ctx */
typedef struct luaFunctionCtx {
/* Special ID that allows getting the Lua function object from the Lua registry */
int lua_function_ref;
} luaFunctionCtx;
/*
* Compile a given blob and save it on the registry.
* Return a function ctx with Lua ref that allows to later retrieve the
* function from the registry.
*
* Return NULL on compilation error and set the error to the err variable
*/
static void* luaEngineCreate(void *engine_ctx, sds blob, sds *err) {
luaEngineCtx *lua_engine_ctx = engine_ctx;
lua_State *lua = lua_engine_ctx->lua;
if (luaL_loadbuffer(lua, blob, sdslen(blob), "@user_function")) {
*err = sdsempty();
*err = sdscatprintf(*err, "Error compiling function: %s",
lua_tostring(lua, -1));
lua_pop(lua, 1);
return NULL;
}
serverAssert(lua_isfunction(lua, -1));
int lua_function_ref = luaL_ref(lua, LUA_REGISTRYINDEX);
luaFunctionCtx *f_ctx = zmalloc(sizeof(*f_ctx));
*f_ctx = (luaFunctionCtx ) { .lua_function_ref = lua_function_ref, };
return f_ctx;
}
/*
* Invole the give function with the given keys and args
*/
static void luaEngineCall(scriptRunCtx *run_ctx,
void *engine_ctx,
void *compiled_function,
robj **keys,
size_t nkeys,
robj **args,
size_t nargs)
{
luaEngineCtx *lua_engine_ctx = engine_ctx;
lua_State *lua = lua_engine_ctx->lua;
luaFunctionCtx *f_ctx = compiled_function;
/* Push error handler */
lua_pushstring(lua, REGISTRY_ERROR_HANDLER_NAME);
lua_gettable(lua, LUA_REGISTRYINDEX);
lua_rawgeti(lua, LUA_REGISTRYINDEX, f_ctx->lua_function_ref);
serverAssert(lua_isfunction(lua, -1));
luaCallFunction(run_ctx, lua, keys, nkeys, args, nargs, 0);
lua_pop(lua, 1); /* Pop error handler */
}
static size_t luaEngineGetUsedMemoy(void *engine_ctx) {
luaEngineCtx *lua_engine_ctx = engine_ctx;
return luaMemory(lua_engine_ctx->lua);
}
static size_t luaEngineFunctionMemoryOverhead(void *compiled_function) {
return zmalloc_size(compiled_function);
}
static size_t luaEngineMemoryOverhead(void *engine_ctx) {
luaEngineCtx *lua_engine_ctx = engine_ctx;
return zmalloc_size(lua_engine_ctx);
}
static void luaEngineFreeFunction(void *engine_ctx, void *compiled_function) {
luaEngineCtx *lua_engine_ctx = engine_ctx;
lua_State *lua = lua_engine_ctx->lua;
luaFunctionCtx *f_ctx = compiled_function;
lua_unref(lua, f_ctx->lua_function_ref);
zfree(f_ctx);
}
/* Initialize Lua engine, should be called once on start. */
int luaEngineInitEngine() {
luaEngineCtx *lua_engine_ctx = zmalloc(sizeof(*lua_engine_ctx));
lua_engine_ctx->lua = lua_open();
luaRegisterRedisAPI(lua_engine_ctx->lua);
/* Save error handler to registry */
lua_pushstring(lua_engine_ctx->lua, REGISTRY_ERROR_HANDLER_NAME);
char *errh_func = "local dbg = debug\n"
"local error_handler = function (err)\n"
" local i = dbg.getinfo(2,'nSl')\n"
" if i and i.what == 'C' then\n"
" i = dbg.getinfo(3,'nSl')\n"
" end\n"
" if i then\n"
" return i.source .. ':' .. i.currentline .. ': ' .. err\n"
" else\n"
" return err\n"
" end\n"
"end\n"
"return error_handler";
luaL_loadbuffer(lua_engine_ctx->lua, errh_func, strlen(errh_func), "@err_handler_def");
lua_pcall(lua_engine_ctx->lua,0,1,0);
lua_settable(lua_engine_ctx->lua, LUA_REGISTRYINDEX);
/* save the engine_ctx on the registry so we can get it from the Lua interpreter */
luaSaveOnRegistry(lua_engine_ctx->lua, REGISTRY_ENGINE_CTX_NAME, lua_engine_ctx);
luaEnableGlobalsProtection(lua_engine_ctx->lua, 0);
engine *lua_engine = zmalloc(sizeof(*lua_engine));
*lua_engine = (engine) {
.engine_ctx = lua_engine_ctx,
.create = luaEngineCreate,
.call = luaEngineCall,
.get_used_memory = luaEngineGetUsedMemoy,
.get_function_memory_overhead = luaEngineFunctionMemoryOverhead,
.get_engine_memory_overhead = luaEngineMemoryOverhead,
.free_function = luaEngineFreeFunction,
};
return functionsRegisterEngine(LUA_ENGINE_NAME, lua_engine);
}
/*
* Copyright (c) 2021, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "functions.h"
#include "sds.h"
#include "dict.h"
#include "adlist.h"
#include "atomicvar.h"
static size_t engine_cache_memory = 0;
/* Forward declaration */
static void engineFunctionDispose(dict *d, void *obj);
struct functionsCtx {
dict *functions; /* Function name -> Function object that can be used to run the function */
size_t cache_memory /* Overhead memory (structs, dictionaries, ..) used by all the functions */;
};
dictType engineDictType = {
dictSdsCaseHash, /* hash function */
dictSdsDup, /* key dup */
NULL, /* val dup */
dictSdsKeyCaseCompare, /* key compare */
dictSdsDestructor, /* key destructor */
NULL, /* val destructor */
NULL /* allow to expand */
};
dictType functionDictType = {
dictSdsHash, /* hash function */
dictSdsDup, /* key dup */
NULL, /* val dup */
dictSdsKeyCompare, /* key compare */
dictSdsDestructor, /* key destructor */
engineFunctionDispose,/* val destructor */
NULL /* allow to expand */
};
/* Dictionary of engines */
static dict *engines = NULL;
/* Functions Ctx.
* Contains the dictionary that map a function name to
* function object and the cache memory used by all the functions */
static functionsCtx *functions_ctx = NULL;
static size_t functionMallocSize(functionInfo *fi) {
return zmalloc_size(fi) + sdsZmallocSize(fi->name)
+ (fi->desc ? sdsZmallocSize(fi->desc) : 0)
+ sdsZmallocSize(fi->code)
+ fi->ei->engine->get_function_memory_overhead(fi->function);
}
/* Dispose function memory */
static void engineFunctionDispose(dict *d, void *obj) {
UNUSED(d);
functionInfo *fi = obj;
sdsfree(fi->code);
sdsfree(fi->name);
if (fi->desc) {
sdsfree(fi->desc);
}
engine *engine = fi->ei->engine;
engine->free_function(engine->engine_ctx, fi->function);
zfree(fi);
}
/* Free function memory and detele it from the functions dictionary */
static void engineFunctionFree(functionInfo *fi, functionsCtx *functions) {
functions->cache_memory -= functionMallocSize(fi);
dictDelete(functions->functions, fi->name);
}
/* Clear all the functions from the given functions ctx */
void functionsCtxClear(functionsCtx *functions_ctx) {
dictEmpty(functions_ctx->functions, NULL);
functions_ctx->cache_memory = 0;
}
/* Free the given functions ctx */
void functionsCtxFree(functionsCtx *functions_ctx) {
functionsCtxClear(functions_ctx);
dictRelease(functions_ctx->functions);
zfree(functions_ctx);
}
/* Swap the current functions ctx with the given one.
* Free the old functions ctx. */
void functionsCtxSwapWithCurrent(functionsCtx *new_functions_ctx) {
functionsCtxFree(functions_ctx);
functions_ctx = new_functions_ctx;
}
/* return the current functions ctx */
functionsCtx* functionsCtxGetCurrent() {
return functions_ctx;
}
/* Create a new functions ctx */
functionsCtx* functionsCtxCreate() {
functionsCtx *ret = zmalloc(sizeof(functionsCtx));
ret->functions = dictCreate(&functionDictType);
ret->cache_memory = 0;
return ret;
}
/*
* Register a function info to functions dictionary
* 1. Set the function client
* 2. Add function to functions dictionary
* 3. update cache memory
*/
static void engineFunctionRegister(functionInfo *fi, functionsCtx *functions) {
int res = dictAdd(functions->functions, fi->name, fi);
serverAssert(res == DICT_OK);
functions->cache_memory += functionMallocSize(fi);
}
/*
* Creating a function info object and register it.
* Return the created object
*/
static functionInfo* engineFunctionCreate(sds name, void *function, engineInfo *ei,
sds desc, sds code, functionsCtx *functions)
{
functionInfo *fi = zmalloc(sizeof(*fi));
*fi = (functionInfo ) {
.name = sdsdup(name),
.function = function,
.ei = ei,
.code = sdsdup(code),
.desc = desc ? sdsdup(desc) : NULL,
};
engineFunctionRegister(fi, functions);
return fi;
}
/* Register an engine, should be called once by the engine on startup and give the following:
*
* - engine_name - name of the engine to register
* - engine_ctx - the engine ctx that should be used by Redis to interact with the engine */
int functionsRegisterEngine(const char *engine_name, engine *engine) {
sds engine_name_sds = sdsnew(engine_name);
if (dictFetchValue(engines, engine_name_sds)) {
serverLog(LL_WARNING, "Same engine was registered twice");
sdsfree(engine_name_sds);
return C_ERR;
}
client *c = createClient(NULL);
c->flags |= (CLIENT_DENY_BLOCKING | CLIENT_SCRIPT);
engineInfo *ei = zmalloc(sizeof(*ei));
*ei = (engineInfo ) { .name = engine_name_sds, .engine = engine, .c = c,};
dictAdd(engines, engine_name_sds, ei);
engine_cache_memory += zmalloc_size(ei) + sdsZmallocSize(ei->name) +
zmalloc_size(engine) +
engine->get_engine_memory_overhead(engine->engine_ctx);
return C_OK;
}
/*
* FUNCTION STATS
*/
void functionsStatsCommand(client *c) {
if (scriptIsRunning() && scriptIsEval()) {
addReplyErrorObject(c, shared.slowevalerr);
return;
}
addReplyMapLen(c, 2);
addReplyBulkCString(c, "running_script");
if (!scriptIsRunning()) {
addReplyNull(c);
} else {
addReplyMapLen(c, 3);
addReplyBulkCString(c, "name");
addReplyBulkCString(c, scriptCurrFunction());
addReplyBulkCString(c, "command");
client *script_client = scriptGetCaller();
addReplyArrayLen(c, script_client->argc);
for (int i = 0 ; i < script_client->argc ; ++i) {
addReplyBulkCBuffer(c, script_client->argv[i]->ptr, sdslen(script_client->argv[i]->ptr));
}
addReplyBulkCString(c, "duration_ms");
addReplyLongLong(c, scriptRunDuration());
}
addReplyBulkCString(c, "engines");
addReplyArrayLen(c, dictSize(engines));
dictIterator *iter = dictGetIterator(engines);
dictEntry *entry = NULL;
while ((entry = dictNext(iter))) {
engineInfo *ei = dictGetVal(entry);
addReplyBulkCString(c, ei->name);
}
dictReleaseIterator(iter);
}
/*
* FUNCTION LIST
*/
void functionsListCommand(client *c) {
/* general information on all the functions */
addReplyArrayLen(c, dictSize(functions_ctx->functions));
dictIterator *iter = dictGetIterator(functions_ctx->functions);
dictEntry *entry = NULL;
while ((entry = dictNext(iter))) {
functionInfo *fi = dictGetVal(entry);
addReplyMapLen(c, 3);
addReplyBulkCString(c, "name");
addReplyBulkCBuffer(c, fi->name, sdslen(fi->name));
addReplyBulkCString(c, "engine");
addReplyBulkCBuffer(c, fi->ei->name, sdslen(fi->ei->name));
addReplyBulkCString(c, "description");
if (fi->desc) {
addReplyBulkCBuffer(c, fi->desc, sdslen(fi->desc));
} else {
addReplyNull(c);
}
}
dictReleaseIterator(iter);
}
/*
* FUNCTION INFO <FUNCTION NAME> [WITHCODE]
*/
void functionsInfoCommand(client *c) {
if (c->argc > 4) {
addReplyErrorFormat(c,"wrong number of arguments for '%s' command or subcommand", c->cmd->name);
return;
}
/* dedicated information on specific function */
robj *function_name = c->argv[2];
int with_code = 0;
if (c->argc == 4) {
robj *with_code_arg = c->argv[3];
if (!strcasecmp(with_code_arg->ptr, "withcode")) {
with_code = 1;
}
}
functionInfo *fi = dictFetchValue(functions_ctx->functions, function_name->ptr);
if (!fi) {
addReplyError(c, "Function does not exists");
return;
}
addReplyMapLen(c, with_code? 4 : 3);
addReplyBulkCString(c, "name");
addReplyBulkCBuffer(c, fi->name, sdslen(fi->name));
addReplyBulkCString(c, "engine");
addReplyBulkCBuffer(c, fi->ei->name, sdslen(fi->ei->name));
addReplyBulkCString(c, "description");
if (fi->desc) {
addReplyBulkCBuffer(c, fi->desc, sdslen(fi->desc));
} else {
addReplyNull(c);
}
if (with_code) {
addReplyBulkCString(c, "code");
addReplyBulkCBuffer(c, fi->code, sdslen(fi->code));
}
}
/*
* FUNCTION DELETE <FUNCTION NAME>
*/
void functionsDeleteCommand(client *c) {
if (server.masterhost && server.repl_slave_ro && !(c->flags & CLIENT_MASTER)) {
addReplyError(c, "Can not delete a function on a read only replica");
return;
}
robj *function_name = c->argv[2];
functionInfo *fi = dictFetchValue(functions_ctx->functions, function_name->ptr);
if (!fi) {
addReplyError(c, "Function not found");
return;
}
engineFunctionFree(fi, functions_ctx);
forceCommandPropagation(c, PROPAGATE_REPL | PROPAGATE_AOF);
addReply(c, shared.ok);
}
void functionsKillCommand(client *c) {
scriptKill(c, 0);
}
static void fcallCommandGeneric(client *c, int ro) {
robj *function_name = c->argv[1];
functionInfo *fi = dictFetchValue(functions_ctx->functions, function_name->ptr);
if (!fi) {
addReplyError(c, "Function not found");
return;
}
engine *engine = fi->ei->engine;
long long numkeys;
/* Get the number of arguments that are keys */
if (getLongLongFromObject(c->argv[2], &numkeys) != C_OK) {
addReplyError(c, "Bad number of keys provided");
return;
}
if (numkeys > (c->argc - 3)) {
addReplyError(c, "Number of keys can't be greater than number of args");
return;
} else if (numkeys < 0) {
addReplyError(c, "Number of keys can't be negative");
return;
}
scriptRunCtx run_ctx;
scriptPrepareForRun(&run_ctx, fi->ei->c, c, fi->name);
if (ro) {
run_ctx.flags |= SCRIPT_READ_ONLY;
}
engine->call(&run_ctx, engine->engine_ctx, fi->function, c->argv + 3, numkeys,
c->argv + 3 + numkeys, c->argc - 3 - numkeys);
scriptResetRun(&run_ctx);
}
/*
* FCALL <FUNCTION NAME> nkeys <key1 .. keyn> <arg1 .. argn>
*/
void fcallCommand(client *c) {
fcallCommandGeneric(c, 0);
}
/*
* FCALL_RO <FUNCTION NAME> nkeys <key1 .. keyn> <arg1 .. argn>
*/
void fcallCommandReadOnly(client *c) {
fcallCommandGeneric(c, 1);
}
void functionsHelpCommand(client *c) {
const char *help[] = {
"CREATE <ENGINE NAME> <FUNCTION NAME> [REPLACE] [DESC <FUNCTION DESCRIPTION>] <FUNCTION CODE>",
" Create a new function with the given function name and code.",
"DELETE <FUNCTION NAME>",
" Delete the given function.",
"INFO <FUNCTION NAME> [WITHCODE]",
" For each function, print the following information about the function:",
" * Function name",
" * The engine used to run the function",
" * Function description",
" * Function code (only if WITHCODE is given)",
"LIST",
" Return general information on all the functions:",
" * Function name",
" * The engine used to run the function",
" * Function description",
"STATS",
" Return information about the current function running:",
" * Function name",
" * Command used to run the function",
" * Duration in MS that the function is running",
" If not function is running, return nil",
" In addition, returns a list of available engines.",
"KILL",
" Kill the current running function.",
NULL };
addReplyHelp(c, help);
}
/* Compile and save the given function, return C_OK on success and C_ERR on failure.
* In case on failure the err out param is set with relevant error message */
int functionsCreateWithFunctionCtx(sds function_name,sds engine_name, sds desc, sds code,
int replace, sds* err, functionsCtx *functions) {
engineInfo *ei = dictFetchValue(engines, engine_name);
if (!ei) {
*err = sdsnew("Engine not found");
return C_ERR;
}
engine *engine = ei->engine;
functionInfo *fi = dictFetchValue(functions->functions, function_name);
if (fi && !replace) {
*err = sdsnew("Function already exists");
return C_ERR;
}
void *function = engine->create(engine->engine_ctx, code, err);
if (*err) {
return C_ERR;
}
if (fi) {
/* free the already existing function as we are going to replace it */
engineFunctionFree(fi, functions);
}
engineFunctionCreate(function_name, function, ei, desc, code, functions);
return C_OK;
}
/*
* FUNCTION CREATE <ENGINE NAME> <FUNCTION NAME>
* [REPLACE] [DESC <FUNCTION DESCRIPTION>] <FUNCTION CODE>
*
* ENGINE NAME - name of the engine to use the run the function
* FUNCTION NAME - name to use to invoke the function
* REPLACE - optional, replace existing function
* DESCRIPTION - optional, function description
* FUNCTION CODE - function code to pass to the engine
*/
void functionsCreateCommand(client *c) {
if (server.masterhost && server.repl_slave_ro && !(c->flags & CLIENT_MASTER)) {
addReplyError(c, "Can not create a function on a read only replica");
return;
}
robj *engine_name = c->argv[2];
robj *function_name = c->argv[3];
int replace = 0;
int argc_pos = 4;
sds desc = NULL;
while (argc_pos < c->argc - 1) {
robj *next_arg = c->argv[argc_pos++];
if (!strcasecmp(next_arg->ptr, "replace")) {
replace = 1;
continue;
}
if (!strcasecmp(next_arg->ptr, "description")) {
if (argc_pos >= c->argc) {
addReplyError(c, "Bad function description");
return;
}
desc = c->argv[argc_pos++]->ptr;
continue;
}
}
if (argc_pos >= c->argc) {
addReplyError(c, "Function code is missing");
return;
}
robj *code = c->argv[argc_pos];
sds err = NULL;
if (functionsCreateWithFunctionCtx(function_name->ptr, engine_name->ptr,
desc, code->ptr, replace, &err, functions_ctx) != C_OK)
{
addReplyErrorSds(c, err);
return;
}
forceCommandPropagation(c, PROPAGATE_REPL | PROPAGATE_AOF);
addReply(c, shared.ok);
}
/* Return memory usage of all the engines combine */
unsigned long functionsMemory() {
dictIterator *iter = dictGetIterator(engines);
dictEntry *entry = NULL;
size_t engines_nemory = 0;
while ((entry = dictNext(iter))) {
engineInfo *ei = dictGetVal(entry);
engine *engine = ei->engine;
engines_nemory += engine->get_used_memory(engine->engine_ctx);
}
dictReleaseIterator(iter);
return engines_nemory;
}
/* Return memory overhead of all the engines combine */
unsigned long functionsMemoryOverhead() {
size_t memory_overhead = dictSize(engines) * sizeof(dictEntry) +
dictSlots(engines) * sizeof(dictEntry*);
memory_overhead += dictSize(functions_ctx->functions) * sizeof(dictEntry) +
dictSlots(functions_ctx->functions) * sizeof(dictEntry*) + sizeof(functionsCtx);
memory_overhead += functions_ctx->cache_memory;
memory_overhead += engine_cache_memory;
return memory_overhead;
}
/* Returns the number of functions */
unsigned long functionsNum() {
return dictSize(functions_ctx->functions);
}
dict* functionsGet() {
return functions_ctx->functions;
}
/* Initialize engine data structures.
* Should be called once on server initialization */
int functionsInit() {
engines = dictCreate(&engineDictType);
functions_ctx = functionsCtxCreate();
if (luaEngineInitEngine() != C_OK) {
return C_ERR;
}
return C_OK;
}
/*
* Copyright (c) 2021, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef __FUNCTIONS_H_
#define __FUNCTIONS_H_
/*
* functions.c unit provides the Redis Functions API:
* * FUNCTION CREATE
* * FUNCTION CALL
* * FUNCTION DELETE
* * FUNCTION KILL
* * FUNCTION INFO
*
* Also contains implementation for:
* * Save/Load function from rdb
* * Register engines
*/
#include "server.h"
#include "script.h"
#include "redismodule.h"
typedef struct engine {
/* engine specific context */
void *engine_ctx;
/* Create function callback, get the engine_ctx, and function code.
* returns NULL on error and set sds to be the error message */
void* (*create)(void *engine_ctx, sds code, sds *err);
/* Invoking a function, r_ctx is an opaque object (from engine POV).
* The r_ctx should be used by the engine to interaction with Redis,
* such interaction could be running commands, set resp, or set
* replication mode
*/
void (*call)(scriptRunCtx *r_ctx, void *engine_ctx, void *compiled_function,
robj **keys, size_t nkeys, robj **args, size_t nargs);
/* get current used memory by the engine */
size_t (*get_used_memory)(void *engine_ctx);
/* Return memory overhead for a given function,
* such memory is not counted as engine memory but as general
* structs memory that hold different information */
size_t (*get_function_memory_overhead)(void *compiled_function);
/* Return memory overhead for engine (struct size holding the engine)*/
size_t (*get_engine_memory_overhead)(void *engine_ctx);
/* free the given function */
void (*free_function)(void *engine_ctx, void *compiled_function);
} engine;
/* Hold information about an engine.
* Used on rdb.c so it must be declared here. */
typedef struct engineInfo {
sds name; /* Name of the engine */
engine *engine; /* engine callbacks that allows to interact with the engine */
client *c; /* Client that is used to run commands */
} engineInfo;
/* Hold information about the specific function.
* Used on rdb.c so it must be declared here. */
typedef struct functionInfo {
sds name; /* Function name */
void *function; /* Opaque object that set by the function's engine and allow it
to run the function, usually it's the function compiled code. */
engineInfo *ei; /* Pointer to the function engine */
sds code; /* Function code */
sds desc; /* Function description */
} functionInfo;
int functionsRegisterEngine(const char *engine_name, engine *engine_ctx);
int functionsCreateWithFunctionCtx(sds function_name, sds engine_name, sds desc, sds code,
int replace, sds* err, functionsCtx *functions);
void functionsCreateCommand(client *c);
void fcallCommand(client *c);
void fcallCommandReadOnly(client *c);
void functionsDeleteCommand(client *c);
void functionsKillCommand(client *c);
void functionsStatsCommand(client *c);
void functionsInfoCommand(client *c);
void functionsListCommand(client *c);
void functionsHelpCommand(client *c);
unsigned long functionsMemory();
unsigned long functionsMemoryOverhead();
int functionsLoad(rio *rdb, int ver);
unsigned long functionsNum();
dict* functionsGet();
functionsCtx* functionsCtxGetCurrent();
functionsCtx* functionsCtxCreate();
void functionsCtxFree(functionsCtx *functions_ctx);
void functionsCtxClear(functionsCtx *functions_ctx);
void functionsCtxSwapWithCurrent(functionsCtx *functions_ctx);
int luaEngineInitEngine();
int functionsInit();
#endif /* __FUNCTIONS_H_ */
...@@ -629,7 +629,7 @@ void moduleHandlePropagationAfterCommandCallback(RedisModuleCtx *ctx) { ...@@ -629,7 +629,7 @@ void moduleHandlePropagationAfterCommandCallback(RedisModuleCtx *ctx) {
/* If this command is executed from with Lua or MULTI/EXEC we do not /* If this command is executed from with Lua or MULTI/EXEC we do not
* need to propagate EXEC */ * need to propagate EXEC */
if (server.in_eval || server.in_exec) return; if (server.in_script || server.in_exec) return;
/* Handle the replication of the final EXEC, since whatever a command /* Handle the replication of the final EXEC, since whatever a command
* emits is always wrapped around MULTI/EXEC. */ * emits is always wrapped around MULTI/EXEC. */
...@@ -2333,7 +2333,7 @@ int RM_ReplyWithLongDouble(RedisModuleCtx *ctx, long double ld) { ...@@ -2333,7 +2333,7 @@ int RM_ReplyWithLongDouble(RedisModuleCtx *ctx, long double ld) {
void moduleReplicateMultiIfNeeded(RedisModuleCtx *ctx) { void moduleReplicateMultiIfNeeded(RedisModuleCtx *ctx) {
/* Skip this if client explicitly wrap the command with MULTI, or if /* Skip this if client explicitly wrap the command with MULTI, or if
* the module command was called by a script. */ * the module command was called by a script. */
if (server.in_eval || server.in_exec) return; if (server.in_script || server.in_exec) return;
/* If we already emitted MULTI return ASAP. */ /* If we already emitted MULTI return ASAP. */
if (server.propagate_in_transaction) return; if (server.propagate_in_transaction) return;
/* If this is a thread safe context, we do not want to wrap commands /* If this is a thread safe context, we do not want to wrap commands
...@@ -2709,7 +2709,7 @@ int RM_GetContextFlags(RedisModuleCtx *ctx) { ...@@ -2709,7 +2709,7 @@ int RM_GetContextFlags(RedisModuleCtx *ctx) {
} }
} }
if (server.in_eval) if (server.in_script)
flags |= REDISMODULE_CTX_FLAGS_LUA; flags |= REDISMODULE_CTX_FLAGS_LUA;
if (server.in_exec) if (server.in_exec)
...@@ -6215,7 +6215,7 @@ void unblockClientFromModule(client *c) { ...@@ -6215,7 +6215,7 @@ void unblockClientFromModule(client *c) {
*/ */
RedisModuleBlockedClient *moduleBlockClient(RedisModuleCtx *ctx, RedisModuleCmdFunc reply_callback, RedisModuleCmdFunc timeout_callback, void (*free_privdata)(RedisModuleCtx*,void*), long long timeout_ms, RedisModuleString **keys, int numkeys, void *privdata) { RedisModuleBlockedClient *moduleBlockClient(RedisModuleCtx *ctx, RedisModuleCmdFunc reply_callback, RedisModuleCmdFunc timeout_callback, void (*free_privdata)(RedisModuleCtx*,void*), long long timeout_ms, RedisModuleString **keys, int numkeys, void *privdata) {
client *c = ctx->client; client *c = ctx->client;
int islua = server.in_eval; int islua = server.in_script;
int ismulti = server.in_exec; int ismulti = server.in_exec;
c->bpop.module_blocked_handle = zmalloc(sizeof(RedisModuleBlockedClient)); c->bpop.module_blocked_handle = zmalloc(sizeof(RedisModuleBlockedClient));
......
...@@ -30,6 +30,7 @@ ...@@ -30,6 +30,7 @@
#include "server.h" #include "server.h"
#include "atomicvar.h" #include "atomicvar.h"
#include "cluster.h" #include "cluster.h"
#include "script.h"
#include <sys/socket.h> #include <sys/socket.h>
#include <sys/uio.h> #include <sys/uio.h>
#include <math.h> #include <math.h>
...@@ -260,7 +261,7 @@ void clientInstallWriteHandler(client *c) { ...@@ -260,7 +261,7 @@ void clientInstallWriteHandler(client *c) {
int prepareClientToWrite(client *c) { int prepareClientToWrite(client *c) {
/* If it's the Lua client we always return ok without installing any /* If it's the Lua client we always return ok without installing any
* handler since there is no socket at all. */ * handler since there is no socket at all. */
if (c->flags & (CLIENT_LUA|CLIENT_MODULE)) return C_OK; if (c->flags & (CLIENT_SCRIPT|CLIENT_MODULE)) return C_OK;
/* If CLIENT_CLOSE_ASAP flag is set, we need not write anything. */ /* If CLIENT_CLOSE_ASAP flag is set, we need not write anything. */
if (c->flags & CLIENT_CLOSE_ASAP) return C_ERR; if (c->flags & CLIENT_CLOSE_ASAP) return C_ERR;
...@@ -1491,7 +1492,7 @@ void freeClientAsync(client *c) { ...@@ -1491,7 +1492,7 @@ void freeClientAsync(client *c) {
* may access the list while Redis uses I/O threads. All the other accesses * may access the list while Redis uses I/O threads. All the other accesses
* are in the context of the main thread while the other threads are * are in the context of the main thread while the other threads are
* idle. */ * idle. */
if (c->flags & CLIENT_CLOSE_ASAP || c->flags & CLIENT_LUA) return; if (c->flags & CLIENT_CLOSE_ASAP || c->flags & CLIENT_SCRIPT) return;
c->flags |= CLIENT_CLOSE_ASAP; c->flags |= CLIENT_CLOSE_ASAP;
if (server.io_threads_num == 1) { if (server.io_threads_num == 1) {
/* no need to bother with locking if there's just one thread (the main thread) */ /* no need to bother with locking if there's just one thread (the main thread) */
...@@ -2199,7 +2200,7 @@ int processInputBuffer(client *c) { ...@@ -2199,7 +2200,7 @@ int processInputBuffer(client *c) {
* condition on the slave. We want just to accumulate the replication * condition on the slave. We want just to accumulate the replication
* stream (instead of replying -BUSY like we do with other clients) and * stream (instead of replying -BUSY like we do with other clients) and
* later resume the processing. */ * later resume the processing. */
if (server.lua_timedout && c->flags & CLIENT_MASTER) break; if (scriptIsTimedout() && c->flags & CLIENT_MASTER) break;
/* CLIENT_CLOSE_AFTER_REPLY closes the connection once the reply is /* CLIENT_CLOSE_AFTER_REPLY closes the connection once the reply is
* written to the client. Make sure to not let the reply grow after * written to the client. Make sure to not let the reply grow after
......
...@@ -29,6 +29,7 @@ ...@@ -29,6 +29,7 @@
*/ */
#include "server.h" #include "server.h"
#include "functions.h"
#include <math.h> #include <math.h>
#include <ctype.h> #include <ctype.h>
...@@ -1203,9 +1204,7 @@ struct redisMemOverhead *getMemoryOverheadData(void) { ...@@ -1203,9 +1204,7 @@ struct redisMemOverhead *getMemoryOverheadData(void) {
mh->aof_buffer = mem; mh->aof_buffer = mem;
mem_total+=mem; mem_total+=mem;
mem = server.lua_scripts_mem; mem = evalScriptsMemory();
mem += dictSize(server.lua_scripts) * sizeof(dictEntry) +
dictSlots(server.lua_scripts) * sizeof(dictEntry*);
mem += dictSize(server.repl_scriptcache_dict) * sizeof(dictEntry) + mem += dictSize(server.repl_scriptcache_dict) * sizeof(dictEntry) +
dictSlots(server.repl_scriptcache_dict) * sizeof(dictEntry*); dictSlots(server.repl_scriptcache_dict) * sizeof(dictEntry*);
if (listLength(server.repl_scriptcache_fifo) > 0) { if (listLength(server.repl_scriptcache_fifo) > 0) {
...@@ -1214,6 +1213,8 @@ struct redisMemOverhead *getMemoryOverheadData(void) { ...@@ -1214,6 +1213,8 @@ struct redisMemOverhead *getMemoryOverheadData(void) {
} }
mh->lua_caches = mem; mh->lua_caches = mem;
mem_total+=mem; mem_total+=mem;
mh->functions_caches = functionsMemoryOverhead();
mem_total+=mh->functions_caches;
for (j = 0; j < server.dbnum; j++) { for (j = 0; j < server.dbnum; j++) {
redisDb *db = server.db+j; redisDb *db = server.db+j;
...@@ -1325,7 +1326,7 @@ sds getMemoryDoctorReport(void) { ...@@ -1325,7 +1326,7 @@ sds getMemoryDoctorReport(void) {
} }
/* Too many scripts are cached? */ /* Too many scripts are cached? */
if (dictSize(server.lua_scripts) > 1000) { if (dictSize(evalScriptsDict()) > 1000) {
many_scripts = 1; many_scripts = 1;
num_reports++; num_reports++;
} }
...@@ -1529,7 +1530,7 @@ NULL ...@@ -1529,7 +1530,7 @@ NULL
} else if (!strcasecmp(c->argv[1]->ptr,"stats") && c->argc == 2) { } else if (!strcasecmp(c->argv[1]->ptr,"stats") && c->argc == 2) {
struct redisMemOverhead *mh = getMemoryOverheadData(); struct redisMemOverhead *mh = getMemoryOverheadData();
addReplyMapLen(c,25+mh->num_dbs); addReplyMapLen(c,26+mh->num_dbs);
addReplyBulkCString(c,"peak.allocated"); addReplyBulkCString(c,"peak.allocated");
addReplyLongLong(c,mh->peak_allocated); addReplyLongLong(c,mh->peak_allocated);
...@@ -1555,6 +1556,9 @@ NULL ...@@ -1555,6 +1556,9 @@ NULL
addReplyBulkCString(c,"lua.caches"); addReplyBulkCString(c,"lua.caches");
addReplyLongLong(c,mh->lua_caches); addReplyLongLong(c,mh->lua_caches);
addReplyBulkCString(c,"functions.caches");
addReplyLongLong(c,mh->functions_caches);
for (size_t j = 0; j < mh->num_dbs; j++) { for (size_t j = 0; j < mh->num_dbs; j++) {
char dbname[32]; char dbname[32];
snprintf(dbname,sizeof(dbname),"db.%zd",mh->db[j].dbid); snprintf(dbname,sizeof(dbname),"db.%zd",mh->db[j].dbid);
......
...@@ -32,6 +32,7 @@ ...@@ -32,6 +32,7 @@
#include "zipmap.h" #include "zipmap.h"
#include "endianconv.h" #include "endianconv.h"
#include "stream.h" #include "stream.h"
#include "functions.h"
#include <math.h> #include <math.h>
#include <fcntl.h> #include <fcntl.h>
...@@ -1239,6 +1240,25 @@ int rdbSaveRio(rio *rdb, int *error, int rdbflags, rdbSaveInfo *rsi) { ...@@ -1239,6 +1240,25 @@ int rdbSaveRio(rio *rdb, int *error, int rdbflags, rdbSaveInfo *rsi) {
if (rdbSaveInfoAuxFields(rdb,rdbflags,rsi) == -1) goto werr; if (rdbSaveInfoAuxFields(rdb,rdbflags,rsi) == -1) goto werr;
if (rdbSaveModulesAux(rdb, REDISMODULE_AUX_BEFORE_RDB) == -1) goto werr; if (rdbSaveModulesAux(rdb, REDISMODULE_AUX_BEFORE_RDB) == -1) goto werr;
/* save functions */
dict *functions = functionsGet();
dictIterator *iter = dictGetIterator(functions);
dictEntry *entry = NULL;
while ((entry = dictNext(iter))) {
rdbSaveType(rdb, RDB_OPCODE_FUNCTION);
functionInfo* fi = dictGetVal(entry);
if (rdbSaveRawString(rdb, (unsigned char *) fi->name, sdslen(fi->name)) == -1) goto werr;
if (rdbSaveRawString(rdb, (unsigned char *) fi->ei->name, sdslen(fi->ei->name)) == -1) goto werr;
if (fi->desc) {
if (rdbSaveLen(rdb, 1) == -1) goto werr; /* desc exists */
if (rdbSaveRawString(rdb, (unsigned char *) fi->desc, sdslen(fi->desc)) == -1) goto werr;
} else {
if (rdbSaveLen(rdb, 0) == -1) goto werr; /* desc not exists */
}
if (rdbSaveRawString(rdb, (unsigned char *) fi->code, sdslen(fi->code)) == -1) goto werr;
}
dictReleaseIterator(iter);
for (j = 0; j < server.dbnum; j++) { for (j = 0; j < server.dbnum; j++) {
redisDb *db = server.db+j; redisDb *db = server.db+j;
dict *d = db->dict; dict *d = db->dict;
...@@ -1303,8 +1323,8 @@ int rdbSaveRio(rio *rdb, int *error, int rdbflags, rdbSaveInfo *rsi) { ...@@ -1303,8 +1323,8 @@ int rdbSaveRio(rio *rdb, int *error, int rdbflags, rdbSaveInfo *rsi) {
* the script cache as well: on successful PSYNC after a restart, we need * the script cache as well: on successful PSYNC after a restart, we need
* to be able to process any EVALSHA inside the replication backlog the * to be able to process any EVALSHA inside the replication backlog the
* master will send us. */ * master will send us. */
if (rsi && dictSize(server.lua_scripts)) { if (rsi && dictSize(evalScriptsDict())) {
di = dictGetIterator(server.lua_scripts); di = dictGetIterator(evalScriptsDict());
while((de = dictNext(di)) != NULL) { while((de = dictNext(di)) != NULL) {
robj *body = dictGetVal(de); robj *body = dictGetVal(de);
if (rdbSaveAuxField(rdb,"lua",3,body->ptr,sdslen(body->ptr)) == -1) if (rdbSaveAuxField(rdb,"lua",3,body->ptr,sdslen(body->ptr)) == -1)
...@@ -2687,12 +2707,80 @@ void rdbLoadProgressCallback(rio *r, const void *buf, size_t len) { ...@@ -2687,12 +2707,80 @@ void rdbLoadProgressCallback(rio *r, const void *buf, size_t len) {
} }
} }
static int rdbFunctionLoad(rio *rdb, int ver, functionsCtx* functions_ctx) {
UNUSED(ver);
sds name = NULL;
sds engine_name = NULL;
sds desc = NULL;
sds blob = NULL;
sds err = NULL;
uint64_t has_desc;
int res = C_ERR;
if (!(name = rdbGenericLoadStringObject(rdb, RDB_LOAD_SDS, NULL))) {
serverLog(LL_WARNING, "Failed loading function name");
goto error;
}
if (!(engine_name = rdbGenericLoadStringObject(rdb, RDB_LOAD_SDS, NULL))) {
serverLog(LL_WARNING, "Failed loading engine name");
goto error;
}
if ((has_desc = rdbLoadLen(rdb, NULL)) == RDB_LENERR) {
serverLog(LL_WARNING, "Failed loading function desc indicator");
goto error;
}
if (has_desc && !(desc = rdbGenericLoadStringObject(rdb, RDB_LOAD_SDS, NULL))) {
serverLog(LL_WARNING, "Failed loading function desc");
goto error;
}
if (!(blob = rdbGenericLoadStringObject(rdb, RDB_LOAD_SDS, NULL))) {
serverLog(LL_WARNING, "Failed loading function blob");
goto error;
}
if (functionsCreateWithFunctionCtx(name, engine_name, desc, blob, 0, &err, functions_ctx) != C_OK) {
serverLog(LL_WARNING, "Failed compiling and saving the function %s", err);
goto error;
}
res = C_OK;
error:
if (name) sdsfree(name);
if (engine_name) sdsfree(engine_name);
if (desc) sdsfree(desc);
if (blob) sdsfree(blob);
if (err) sdsfree(err);
return res;
}
/* Load an RDB file from the rio stream 'rdb'. On success C_OK is returned, /* Load an RDB file from the rio stream 'rdb'. On success C_OK is returned,
* otherwise C_ERR is returned and 'errno' is set accordingly. */ * otherwise C_ERR is returned and 'errno' is set accordingly. */
int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi, redisDb *dbarray) { int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi) {
functionsCtx* functions_ctx = functionsCtxGetCurrent();
functionsCtxClear(functions_ctx);
rdbLoadingCtx loading_ctx = { .dbarray = server.db, .functions_ctx = functions_ctx };
int retval = rdbLoadRioWithLoadingCtx(rdb,rdbflags,rsi,&loading_ctx);
if (retval != C_OK) {
/* Loading failed, clear the function ctx */
functionsCtxClear(functions_ctx);
}
return retval;
}
/* Load an RDB file from the rio stream 'rdb'. On success C_OK is returned,
* otherwise C_ERR is returned and 'errno' is set accordingly.
* The rdb_loading_ctx argument holds objects to which the rdb will be loaded to,
* currently it only allow to set db object and functionsCtx to which the data
* will be loaded (in the future it might contains more such objects). */
int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadingCtx *rdb_loading_ctx) {
uint64_t dbid = 0; uint64_t dbid = 0;
int type, rdbver; int type, rdbver;
redisDb *db = dbarray+0; redisDb *db = rdb_loading_ctx->dbarray+0;
char buf[1024]; char buf[1024];
int error; int error;
long long empty_keys_skipped = 0; long long empty_keys_skipped = 0;
...@@ -2764,7 +2852,7 @@ int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi, redisDb *dbarray) { ...@@ -2764,7 +2852,7 @@ int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi, redisDb *dbarray) {
"databases. Exiting\n", server.dbnum); "databases. Exiting\n", server.dbnum);
exit(1); exit(1);
} }
db = dbarray+dbid; db = rdb_loading_ctx->dbarray+dbid;
continue; /* Read next opcode. */ continue; /* Read next opcode. */
} else if (type == RDB_OPCODE_RESIZEDB) { } else if (type == RDB_OPCODE_RESIZEDB) {
/* RESIZEDB: Hint about the size of the keys in the currently /* RESIZEDB: Hint about the size of the keys in the currently
...@@ -2808,7 +2896,7 @@ int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi, redisDb *dbarray) { ...@@ -2808,7 +2896,7 @@ int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi, redisDb *dbarray) {
if (rsi) rsi->repl_offset = strtoll(auxval->ptr,NULL,10); if (rsi) rsi->repl_offset = strtoll(auxval->ptr,NULL,10);
} else if (!strcasecmp(auxkey->ptr,"lua")) { } else if (!strcasecmp(auxkey->ptr,"lua")) {
/* Load the script back in memory. */ /* Load the script back in memory. */
if (luaCreateFunction(NULL,server.lua,auxval) == NULL) { if (luaCreateFunction(NULL, auxval) == NULL) {
rdbReportCorruptRDB( rdbReportCorruptRDB(
"Can't load Lua script from RDB file! " "Can't load Lua script from RDB file! "
"BODY: %s", (char*)auxval->ptr); "BODY: %s", (char*)auxval->ptr);
...@@ -2895,6 +2983,12 @@ int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi, redisDb *dbarray) { ...@@ -2895,6 +2983,12 @@ int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi, redisDb *dbarray) {
decrRefCount(aux); decrRefCount(aux);
continue; /* Read next opcode. */ continue; /* Read next opcode. */
} }
} else if (type == RDB_OPCODE_FUNCTION) {
if (rdbFunctionLoad(rdb, rdbver, rdb_loading_ctx->functions_ctx) != C_OK) {
serverLog(LL_WARNING,"Failed loading function");
goto eoferr;
}
continue;
} }
/* Read key */ /* Read key */
...@@ -3044,7 +3138,9 @@ int rdbLoad(char *filename, rdbSaveInfo *rsi, int rdbflags) { ...@@ -3044,7 +3138,9 @@ int rdbLoad(char *filename, rdbSaveInfo *rsi, int rdbflags) {
if ((fp = fopen(filename,"r")) == NULL) return C_ERR; if ((fp = fopen(filename,"r")) == NULL) return C_ERR;
startLoadingFile(fp, filename,rdbflags); startLoadingFile(fp, filename,rdbflags);
rioInitWithFile(&rdb,fp); rioInitWithFile(&rdb,fp);
retval = rdbLoadRio(&rdb,rdbflags,rsi,server.db);
retval = rdbLoadRio(&rdb,rdbflags,rsi);
fclose(fp); fclose(fp);
stopLoading(retval==C_OK); stopLoading(retval==C_OK);
return retval; return retval;
......
...@@ -100,6 +100,7 @@ ...@@ -100,6 +100,7 @@
#define rdbIsObjectType(t) ((t >= 0 && t <= 7) || (t >= 9 && t <= 18)) #define rdbIsObjectType(t) ((t >= 0 && t <= 7) || (t >= 9 && t <= 18))
/* Special RDB opcodes (saved/loaded with rdbSaveType/rdbLoadType). */ /* Special RDB opcodes (saved/loaded with rdbSaveType/rdbLoadType). */
#define RDB_OPCODE_FUNCTION 246 /* engine data */
#define RDB_OPCODE_MODULE_AUX 247 /* Module auxiliary data. */ #define RDB_OPCODE_MODULE_AUX 247 /* Module auxiliary data. */
#define RDB_OPCODE_IDLE 248 /* LRU idle time. */ #define RDB_OPCODE_IDLE 248 /* LRU idle time. */
#define RDB_OPCODE_FREQ 249 /* LFU frequency. */ #define RDB_OPCODE_FREQ 249 /* LFU frequency. */
...@@ -166,7 +167,8 @@ int rdbSaveBinaryDoubleValue(rio *rdb, double val); ...@@ -166,7 +167,8 @@ int rdbSaveBinaryDoubleValue(rio *rdb, double val);
int rdbLoadBinaryDoubleValue(rio *rdb, double *val); int rdbLoadBinaryDoubleValue(rio *rdb, double *val);
int rdbSaveBinaryFloatValue(rio *rdb, float val); int rdbSaveBinaryFloatValue(rio *rdb, float val);
int rdbLoadBinaryFloatValue(rio *rdb, float *val); int rdbLoadBinaryFloatValue(rio *rdb, float *val);
int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi, redisDb *db); int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi);
int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadingCtx *rdb_loading_ctx);
int rdbSaveRio(rio *rdb, int *error, int rdbflags, rdbSaveInfo *rsi); int rdbSaveRio(rio *rdb, int *error, int rdbflags, rdbSaveInfo *rsi);
rdbSaveInfo *rdbPopulateSaveInfo(rdbSaveInfo *rsi); rdbSaveInfo *rdbPopulateSaveInfo(rdbSaveInfo *rsi);
......
...@@ -32,6 +32,7 @@ ...@@ -32,6 +32,7 @@
#include "server.h" #include "server.h"
#include "cluster.h" #include "cluster.h"
#include "bio.h" #include "bio.h"
#include "functions.h"
#include <memory.h> #include <memory.h>
#include <sys/time.h> #include <sys/time.h>
...@@ -546,7 +547,7 @@ void replicationFeedMonitors(client *c, list *monitors, int dictid, robj **argv, ...@@ -546,7 +547,7 @@ void replicationFeedMonitors(client *c, list *monitors, int dictid, robj **argv,
gettimeofday(&tv,NULL); gettimeofday(&tv,NULL);
cmdrepr = sdscatprintf(cmdrepr,"%ld.%06ld ",(long)tv.tv_sec,(long)tv.tv_usec); cmdrepr = sdscatprintf(cmdrepr,"%ld.%06ld ",(long)tv.tv_sec,(long)tv.tv_usec);
if (c->flags & CLIENT_LUA) { if (c->flags & CLIENT_SCRIPT) {
cmdrepr = sdscatprintf(cmdrepr,"[%d lua] ",dictid); cmdrepr = sdscatprintf(cmdrepr,"[%d lua] ",dictid);
} else if (c->flags & CLIENT_UNIX_SOCKET) { } else if (c->flags & CLIENT_UNIX_SOCKET) {
cmdrepr = sdscatprintf(cmdrepr,"[%d unix:%s] ",dictid,server.unixsocket); cmdrepr = sdscatprintf(cmdrepr,"[%d unix:%s] ",dictid,server.unixsocket);
...@@ -1738,6 +1739,7 @@ void readSyncBulkPayload(connection *conn) { ...@@ -1738,6 +1739,7 @@ void readSyncBulkPayload(connection *conn) {
ssize_t nread, readlen, nwritten; ssize_t nread, readlen, nwritten;
int use_diskless_load = useDisklessLoad(); int use_diskless_load = useDisklessLoad();
redisDb *diskless_load_tempDb = NULL; redisDb *diskless_load_tempDb = NULL;
functionsCtx* temp_functions_ctx = NULL;
int empty_db_flags = server.repl_slave_lazy_flush ? EMPTYDB_ASYNC : int empty_db_flags = server.repl_slave_lazy_flush ? EMPTYDB_ASYNC :
EMPTYDB_NO_FLAGS; EMPTYDB_NO_FLAGS;
off_t left; off_t left;
...@@ -1913,6 +1915,7 @@ void readSyncBulkPayload(connection *conn) { ...@@ -1913,6 +1915,7 @@ void readSyncBulkPayload(connection *conn) {
if (use_diskless_load && server.repl_diskless_load == REPL_DISKLESS_LOAD_SWAPDB) { if (use_diskless_load && server.repl_diskless_load == REPL_DISKLESS_LOAD_SWAPDB) {
/* Initialize empty tempDb dictionaries. */ /* Initialize empty tempDb dictionaries. */
diskless_load_tempDb = disklessLoadInitTempDb(); diskless_load_tempDb = disklessLoadInitTempDb();
temp_functions_ctx = functionsCtxCreate();
moduleFireServerEvent(REDISMODULE_EVENT_REPL_ASYNC_LOAD, moduleFireServerEvent(REDISMODULE_EVENT_REPL_ASYNC_LOAD,
REDISMODULE_SUBEVENT_REPL_ASYNC_LOAD_STARTED, REDISMODULE_SUBEVENT_REPL_ASYNC_LOAD_STARTED,
...@@ -1935,6 +1938,7 @@ void readSyncBulkPayload(connection *conn) { ...@@ -1935,6 +1938,7 @@ void readSyncBulkPayload(connection *conn) {
if (use_diskless_load) { if (use_diskless_load) {
rio rdb; rio rdb;
redisDb *dbarray; redisDb *dbarray;
functionsCtx* functions_ctx;
int asyncLoading = 0; int asyncLoading = 0;
if (server.repl_diskless_load == REPL_DISKLESS_LOAD_SWAPDB) { if (server.repl_diskless_load == REPL_DISKLESS_LOAD_SWAPDB) {
...@@ -1947,8 +1951,11 @@ void readSyncBulkPayload(connection *conn) { ...@@ -1947,8 +1951,11 @@ void readSyncBulkPayload(connection *conn) {
asyncLoading = 1; asyncLoading = 1;
} }
dbarray = diskless_load_tempDb; dbarray = diskless_load_tempDb;
functions_ctx = temp_functions_ctx;
} else { } else {
dbarray = server.db; dbarray = server.db;
functions_ctx = functionsCtxGetCurrent();
functionsCtxClear(functions_ctx);
} }
rioInitWithConn(&rdb,conn,server.repl_transfer_size); rioInitWithConn(&rdb,conn,server.repl_transfer_size);
...@@ -1960,7 +1967,8 @@ void readSyncBulkPayload(connection *conn) { ...@@ -1960,7 +1967,8 @@ void readSyncBulkPayload(connection *conn) {
startLoading(server.repl_transfer_size, RDBFLAGS_REPLICATION, asyncLoading); startLoading(server.repl_transfer_size, RDBFLAGS_REPLICATION, asyncLoading);
int loadingFailed = 0; int loadingFailed = 0;
if (rdbLoadRio(&rdb,RDBFLAGS_REPLICATION,&rsi,dbarray) != C_OK) { rdbLoadingCtx loadingCtx = { .dbarray = dbarray, .functions_ctx = functions_ctx };
if (rdbLoadRioWithLoadingCtx(&rdb,RDBFLAGS_REPLICATION,&rsi,&loadingCtx) != C_OK) {
/* RDB loading failed. */ /* RDB loading failed. */
serverLog(LL_WARNING, serverLog(LL_WARNING,
"Failed trying to load the MASTER synchronization DB " "Failed trying to load the MASTER synchronization DB "
...@@ -1988,6 +1996,7 @@ void readSyncBulkPayload(connection *conn) { ...@@ -1988,6 +1996,7 @@ void readSyncBulkPayload(connection *conn) {
NULL); NULL);
disklessLoadDiscardTempDb(diskless_load_tempDb); disklessLoadDiscardTempDb(diskless_load_tempDb);
functionsCtxFree(temp_functions_ctx);
serverLog(LL_NOTICE, "MASTER <-> REPLICA sync: Discarding temporary DB in background"); serverLog(LL_NOTICE, "MASTER <-> REPLICA sync: Discarding temporary DB in background");
} else { } else {
/* Remove the half-loaded data in case we started with an empty replica. */ /* Remove the half-loaded data in case we started with an empty replica. */
...@@ -2010,6 +2019,9 @@ void readSyncBulkPayload(connection *conn) { ...@@ -2010,6 +2019,9 @@ void readSyncBulkPayload(connection *conn) {
serverLog(LL_NOTICE, "MASTER <-> REPLICA sync: Swapping active DB with loaded DB"); serverLog(LL_NOTICE, "MASTER <-> REPLICA sync: Swapping active DB with loaded DB");
swapMainDbWithTempDb(diskless_load_tempDb); swapMainDbWithTempDb(diskless_load_tempDb);
/* swap existing functions ctx with the temporary one */
functionsCtxSwapWithCurrent(temp_functions_ctx);
moduleFireServerEvent(REDISMODULE_EVENT_REPL_ASYNC_LOAD, moduleFireServerEvent(REDISMODULE_EVENT_REPL_ASYNC_LOAD,
REDISMODULE_SUBEVENT_REPL_ASYNC_LOAD_COMPLETED, REDISMODULE_SUBEVENT_REPL_ASYNC_LOAD_COMPLETED,
NULL); NULL);
......
/*
* Copyright (c) 2009-2021, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "server.h"
#include "script.h"
#include "cluster.h"
/* On script invocation, holding the current run context */
static scriptRunCtx *curr_run_ctx = NULL;
static void exitScriptTimedoutMode(scriptRunCtx *run_ctx) {
serverAssert(run_ctx == curr_run_ctx);
serverAssert(scriptIsTimedout());
run_ctx->flags &= ~SCRIPT_TIMEDOUT;
blockingOperationEnds();
/* if we are a replica and we have an active master, set it for continue processing */
if (server.masterhost && server.master) queueClientForReprocessing(server.master);
}
static void enterScriptTimedoutMode(scriptRunCtx *run_ctx) {
serverAssert(run_ctx == curr_run_ctx);
serverAssert(!scriptIsTimedout());
/* Mark script as timedout */
run_ctx->flags |= SCRIPT_TIMEDOUT;
blockingOperationStarts();
}
int scriptIsTimedout() {
return scriptIsRunning() && (curr_run_ctx->flags & SCRIPT_TIMEDOUT);
}
client* scriptGetClient() {
serverAssert(scriptIsRunning());
return curr_run_ctx->c;
}
client* scriptGetCaller() {
serverAssert(scriptIsRunning());
return curr_run_ctx->original_client;
}
/* interrupt function for scripts, should be call
* from time to time to reply some special command (like ping)
* and also check if the run should be terminated. */
int scriptInterrupt(scriptRunCtx *run_ctx) {
if (run_ctx->flags & SCRIPT_TIMEDOUT) {
/* script already timedout
we just need to precess some events and return */
processEventsWhileBlocked();
return (run_ctx->flags & SCRIPT_KILLED) ? SCRIPT_KILL : SCRIPT_CONTINUE;
}
long long elapsed = elapsedMs(run_ctx->start_time);
if (elapsed < server.script_time_limit) {
return SCRIPT_CONTINUE;
}
serverLog(LL_WARNING,
"Slow script detected: still in execution after %lld milliseconds. "
"You can try killing the script using the %s command.",
elapsed, (run_ctx->flags & SCRIPT_EVAL_MODE) ? "SCRIPT KILL" : "FUNCTION KILL");
enterScriptTimedoutMode(run_ctx);
/* Once the script timeouts we reenter the event loop to permit others
* some commands execution. For this reason
* we need to mask the client executing the script from the event loop.
* If we don't do that the client may disconnect and could no longer be
* here when the EVAL command will return. */
protectClient(run_ctx->original_client);
processEventsWhileBlocked();
return (run_ctx->flags & SCRIPT_KILLED) ? SCRIPT_KILL : SCRIPT_CONTINUE;
}
/* Prepare the given run ctx for execution */
void scriptPrepareForRun(scriptRunCtx *run_ctx, client *engine_client, client *caller, const char *funcname) {
serverAssert(!curr_run_ctx);
/* set the curr_run_ctx so we can use it to kill the script if needed */
curr_run_ctx = run_ctx;
run_ctx->c = engine_client;
run_ctx->original_client = caller;
run_ctx->funcname = funcname;
client *script_client = run_ctx->c;
client *curr_client = run_ctx->original_client;
server.script_caller = curr_client;
/* Select the right DB in the context of the Lua client */
selectDb(script_client, curr_client->db->id);
script_client->resp = 2; /* Default is RESP2, scripts can change it. */
/* If we are in MULTI context, flag Lua client as CLIENT_MULTI. */
if (curr_client->flags & CLIENT_MULTI) {
script_client->flags |= CLIENT_MULTI;
}
server.in_script = 1;
run_ctx->start_time = getMonotonicUs();
run_ctx->snapshot_time = mstime();
run_ctx->flags = 0;
run_ctx->repl_flags = PROPAGATE_AOF | PROPAGATE_REPL;
}
/* Reset the given run ctx after execution */
void scriptResetRun(scriptRunCtx *run_ctx) {
serverAssert(curr_run_ctx);
/* After the script done, remove the MULTI state. */
run_ctx->c->flags &= ~CLIENT_MULTI;
server.in_script = 0;
server.script_caller = NULL;
if (scriptIsTimedout()) {
exitScriptTimedoutMode(run_ctx);
/* Restore the client that was protected when the script timeout
* was detected. */
unprotectClient(run_ctx->original_client);
}
if (!(run_ctx->flags & SCRIPT_EVAL_REPLICATION)) {
preventCommandPropagation(run_ctx->original_client);
if (run_ctx->flags & SCRIPT_MULTI_EMMITED) {
execCommandPropagateExec(run_ctx->original_client->db->id);
}
}
/* unset curr_run_ctx so we will know there is no running script */
curr_run_ctx = NULL;
}
/* return true if a script is currently running */
int scriptIsRunning() {
return curr_run_ctx != NULL;
}
const char* scriptCurrFunction() {
serverAssert(scriptIsRunning());
return curr_run_ctx->funcname;
}
int scriptIsEval() {
serverAssert(scriptIsRunning());
return curr_run_ctx->flags & SCRIPT_EVAL_MODE;
}
/* Kill the current running script */
void scriptKill(client *c, int is_eval) {
if (!curr_run_ctx) {
addReplyError(c, "-NOTBUSY No scripts in execution right now.");
return;
}
if (curr_run_ctx->original_client->flags & CLIENT_MASTER) {
addReplyError(c,
"-UNKILLABLE The busy script was sent by a master instance in the context of replication and cannot be killed.");
}
if (curr_run_ctx->flags & SCRIPT_WRITE_DIRTY) {
addReplyError(c,
"-UNKILLABLE Sorry the script already executed write "
"commands against the dataset. You can either wait the "
"script termination or kill the server in a hard way "
"using the SHUTDOWN NOSAVE command.");
return;
}
if (is_eval && !(curr_run_ctx->flags & SCRIPT_EVAL_MODE)) {
/* Kill a function with 'SCRIPT KILL' is not allow */
addReplyErrorObject(c, shared.slowscripterr);
return;
}
if (!is_eval && (curr_run_ctx->flags & SCRIPT_EVAL_MODE)) {
/* Kill an eval with 'FUNCTION KILL' is not allow */
addReplyErrorObject(c, shared.slowevalerr);
return;
}
curr_run_ctx->flags |= SCRIPT_KILLED;
addReply(c, shared.ok);
}
static int scriptVerifyCommandArity(struct redisCommand *cmd, int argc, sds *err) {
if (!cmd || ((cmd->arity > 0 && cmd->arity != argc) || (argc < cmd->arity))) {
if (cmd)
*err = sdsnew("Wrong number of args calling Redis command from script");
else
*err = sdsnew("Unknown Redis command called from script");
return C_ERR;
}
return C_OK;
}
static int scriptVerifyACL(client *c, sds *err) {
/* Check the ACLs. */
int acl_errpos;
int acl_retval = ACLCheckAllPerm(c, &acl_errpos);
if (acl_retval != ACL_OK) {
addACLLogEntry(c,acl_retval,ACL_LOG_CTX_LUA,acl_errpos,NULL,NULL);
switch (acl_retval) {
case ACL_DENIED_CMD:
*err = sdsnew("The user executing the script can't run this "
"command or subcommand");
break;
case ACL_DENIED_KEY:
*err = sdsnew("The user executing the script can't access "
"at least one of the keys mentioned in the "
"command arguments");
break;
case ACL_DENIED_CHANNEL:
*err = sdsnew("The user executing the script can't publish "
"to the channel mentioned in the command");
break;
default:
*err = sdsnew("The user executing the script is lacking the "
"permissions for the command");
break;
}
return C_ERR;
}
return C_OK;
}
static int scriptVerifyWriteCommandAllow(scriptRunCtx *run_ctx, char **err) {
if (!(run_ctx->c->cmd->flags & CMD_WRITE)) {
return C_OK;
}
if (run_ctx->flags & SCRIPT_READ_ONLY) {
/* We know its a write command, on a read only run we do not allow it. */
*err = sdsnew("Write commands are not allowed from read-only scripts.");
return C_ERR;
}
if ((run_ctx->flags & SCRIPT_RANDOM_DIRTY) && (run_ctx->flags & SCRIPT_EVAL_REPLICATION)) {
*err = sdsnew("Write commands not allowed after non deterministic commands. Call redis.replicate_commands() at the start of your script in order to switch to single commands replication mode.");
return C_ERR;
}
/* Write commands are forbidden against read-only slaves, or if a
* command marked as non-deterministic was already called in the context
* of this script. */
int deny_write_type = writeCommandsDeniedByDiskError();
if (server.masterhost && server.repl_slave_ro && run_ctx->original_client->flags != CLIENT_ID_AOF
&& !(run_ctx->original_client->flags & CLIENT_MASTER))
{
*err = sdsdup(shared.roslaveerr->ptr);
return C_ERR;
}
if (deny_write_type != DISK_ERROR_TYPE_NONE) {
if (deny_write_type == DISK_ERROR_TYPE_RDB) {
*err = sdsdup(shared.bgsaveerr->ptr);
} else {
*err = sdsempty();
*err = sdscatfmt(*err,
"MISCONF Errors writing to the AOF file: %s\r\n",
strerror(server.aof_last_write_errno));
}
return C_ERR;
}
return C_OK;
}
static int scriptVerifyOOM(scriptRunCtx *run_ctx, char **err) {
/* If we reached the memory limit configured via maxmemory, commands that
* could enlarge the memory usage are not allowed, but only if this is the
* first write in the context of this script, otherwise we can't stop
* in the middle. */
if (server.maxmemory && /* Maxmemory is actually enabled. */
run_ctx->original_client->id != CLIENT_ID_AOF && /* Don't care about mem if loading from AOF. */
!server.masterhost && /* Slave must execute the script. */
!(run_ctx->flags & SCRIPT_WRITE_DIRTY) && /* Script had no side effects so far. */
server.script_oom && /* Detected OOM when script start. */
(run_ctx->c->cmd->flags & CMD_DENYOOM))
{
*err = sdsdup(shared.oomerr->ptr);
return C_ERR;
}
return C_OK;
}
static int scriptVerifyClusterState(client *c, client *original_c, sds *err) {
if (!server.cluster_enabled || original_c->id == CLIENT_ID_AOF || (original_c->flags & CLIENT_MASTER)) {
return C_OK;
}
/* If this is a Redis Cluster node, we need to make sure the script is not
* trying to access non-local keys, with the exception of commands
* received from our master or when loading the AOF back in memory. */
int error_code;
/* Duplicate relevant flags in the script client. */
c->flags &= ~(CLIENT_READONLY | CLIENT_ASKING);
c->flags |= original_c->flags & (CLIENT_READONLY | CLIENT_ASKING);
if (getNodeByQuery(c, c->cmd, c->argv, c->argc, NULL, &error_code) != server.cluster->myself) {
if (error_code == CLUSTER_REDIR_DOWN_RO_STATE) {
*err = sdsnew(
"Script attempted to execute a write command while the "
"cluster is down and readonly");
} else if (error_code == CLUSTER_REDIR_DOWN_STATE) {
*err = sdsnew("Script attempted to execute a command while the "
"cluster is down");
} else {
*err = sdsnew("Script attempted to access a non local key in a "
"cluster node");
}
return C_ERR;
}
return C_OK;
}
static void scriptEmitMultiIfNeeded(scriptRunCtx *run_ctx) {
/* If we are using single commands replication, we need to wrap what
* we propagate into a MULTI/EXEC block, so that it will be atomic like
* a Lua script in the context of AOF and slaves. */
client *c = run_ctx->c;
if (!(run_ctx->flags & SCRIPT_EVAL_REPLICATION)
&& !(run_ctx->flags & SCRIPT_MULTI_EMMITED)
&& !(run_ctx->original_client->flags & CLIENT_MULTI)
&& (run_ctx->flags & SCRIPT_WRITE_DIRTY)
&& ((run_ctx->repl_flags & PROPAGATE_AOF)
|| (run_ctx->repl_flags & PROPAGATE_REPL)))
{
execCommandPropagateMulti(run_ctx->original_client->db->id);
run_ctx->flags |= SCRIPT_MULTI_EMMITED;
/* Now we are in the MULTI context, the lua_client should be
* flag as CLIENT_MULTI. */
c->flags |= CLIENT_MULTI;
}
}
/* set RESP for a given run_ctx */
int scriptSetResp(scriptRunCtx *run_ctx, int resp) {
if (resp != 2 && resp != 3) {
return C_ERR;
}
run_ctx->c->resp = resp;
return C_OK;
}
/* set Repl for a given run_ctx
* either: PROPAGATE_AOF | PROPAGATE_REPL*/
int scriptSetRepl(scriptRunCtx *run_ctx, int repl) {
if ((repl & ~(PROPAGATE_AOF | PROPAGATE_REPL)) != 0) {
return C_ERR;
}
run_ctx->repl_flags = repl;
return C_OK;
}
/* Call a Redis command.
* The reply is written to the run_ctx client and it is
* up to the engine to take and parse.
* The err out variable is set only if error occurs and describe the error.
* If err is set on reply is written to the run_ctx client. */
void scriptCall(scriptRunCtx *run_ctx, robj* *argv, int argc, sds *err) {
client *c = run_ctx->c;
/* Setup our fake client for command execution */
c->argv = argv;
c->argc = argc;
c->user = run_ctx->original_client->user;
/* Process module hooks */
moduleCallCommandFilters(c);
argv = c->argv;
argc = c->argc;
struct redisCommand *cmd = lookupCommand(argv, argc);
if (scriptVerifyCommandArity(cmd, argc, err) != C_OK) {
return;
}
c->cmd = c->lastcmd = cmd;
/* There are commands that are not allowed inside scripts. */
if (!server.script_disable_deny_script && (cmd->flags & CMD_NOSCRIPT)) {
*err = sdsnew("This Redis command is not allowed from script");
return;
}
if (scriptVerifyACL(c, err) != C_OK) {
return;
}
if (scriptVerifyWriteCommandAllow(run_ctx, err) != C_OK) {
return;
}
if (scriptVerifyOOM(run_ctx, err) != C_OK) {
return;
}
if (cmd->flags & CMD_WRITE) {
/* signify that we already change the data in this execution */
run_ctx->flags |= SCRIPT_WRITE_DIRTY;
}
if (cmd->flags & CMD_RANDOM) {
/* signify that we already perform a random command in this execution */
run_ctx->flags |= SCRIPT_RANDOM_DIRTY;
}
if (scriptVerifyClusterState(c, run_ctx->original_client, err) != C_OK) {
return;
}
scriptEmitMultiIfNeeded(run_ctx);
int call_flags = CMD_CALL_SLOWLOG | CMD_CALL_STATS;
if (!(run_ctx->flags & SCRIPT_EVAL_REPLICATION)) {
if (run_ctx->repl_flags & PROPAGATE_AOF) {
call_flags |= CMD_CALL_PROPAGATE_AOF;
}
if (run_ctx->repl_flags & PROPAGATE_REPL) {
call_flags |= CMD_CALL_PROPAGATE_REPL;
}
}
call(c, call_flags);
serverAssert((c->flags & CLIENT_BLOCKED) == 0);
}
/* Returns the time when the script invocation started */
mstime_t scriptTimeSnapshot() {
serverAssert(!curr_run_ctx);
return curr_run_ctx->snapshot_time;
}
long long scriptRunDuration() {
serverAssert(scriptIsRunning());
return elapsedMs(curr_run_ctx->start_time);
}
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment