Unverified Commit 885f6b5c authored by Meir Shpilraien (Spielrein)'s avatar Meir Shpilraien (Spielrein) Committed by GitHub
Browse files

Redis Function Libraries (#10004)

# Redis Function Libraries

This PR implements Redis Functions Libraries as describe on: https://github.com/redis/redis/issues/9906.

Libraries purpose is to provide a better code sharing between functions by allowing to create multiple
functions in a single command. Functions that were created together can safely share code between
each other without worrying about compatibility issues and versioning.

Creating a new library is done using 'FUNCTION LOAD' command (full API is described below)

This PR introduces a new struct called libraryInfo, libraryInfo holds information about a library:
* name - name of the library
* engine - engine used to create the library
* code - library code
* description - library description
* functions - the functions exposed by the library

When Redis gets the `FUNCTION LOAD` command it creates a new empty libraryInfo.
Redis passes the `CODE` to the relevant engine alongside the empty libraryInfo.
As a result, the engine will create one or more functions by calling 'libraryCreateFunction'.
The new funcion will be added to the newly created libraryInfo. So far Everything is happening
locally on the libraryInfo so it is easy to abort the operation (in case of an error) by simply
freeing the libraryInfo. After the library info is fully constructed we start the joining phase by
which we will join the new library to the other libraries currently exist on Redis.
The joining phase make sure there is no function collision and add the library to the
librariesCtx (renamed from functionCtx). LibrariesCtx is used all around the code in the exact
same way as functionCtx was used (with respect to RDB loading, replicatio, ...).
The only difference is that apart from function dictionary (maps function name to functionInfo
object), the librariesCtx contains also a libraries dictionary that maps library name to libraryInfo object.

## New API
### FUNCTION LOAD
`FUNCTION LOAD <ENGINE> <LIBRARY NAME> [REPLACE] [DESCRIPTION <DESCRIPTION>] <CODE>`
Create a new library with the given parameters:
* ENGINE - REPLACE Engine name to use to create the library.
* LIBRARY NAME - The new library name.
* REPLACE - If the library already exists, replace it.
* DESCRIPTION - Library description.
* CODE - Library code.

Return "OK" on success, or error on the following cases:
* Library name already taken and REPLACE was not used
* Name collision with another existing library (even if replace was uses)
* Library registration failed by the engine (usually compilation error)

## Changed API
### FUNCTION LIST
`FUNCTION LIST [LIBRARYNAME <LIBRARY NAME PATTERN>] [WITHCODE]`
Command was modified to also allow getting libraries code (so `FUNCTION INFO` command is no longer
needed and removed). In addition the command gets an option argument, `LIBRARYNAME` allows you to
only get libraries that match the given `LIBRARYNAME` pattern. By default, it returns all libraries.

### INFO MEMORY
Added number of libraries to `INFO MEMORY`

### Commands flags
`DENYOOM` flag was set on `FUNCTION LOAD` and `FUNCTION RESTORE`. We consider those commands
as commands that add new data to the dateset (functions are data) and so we want to disallows
to run those commands on OOM.

## Removed API
* FUNCTION CREATE - Decided on https://github.com/redis/redis/issues/9906
* FUNCTION INFO - Decided on https://github.com/redis/redis/issues/9899

## Lua engine changes
When the Lua engine gets the code given on `FUNCTION LOAD` command, it immediately runs it, we call
this run the loading run. Loading run is not a usual script run, it is not possible to invoke any
Redis command from within the load run.
Instead there is a new API provided by `library` object. The new API's: 
* `redis.log` - behave the same as `redis.log`
* `redis.register_function` - register a new function to the library

The loading run purpose is to register functions using the new `redis.register_function` API.
Any attempt to use any other API will result in an error. In addition, the load run is has a time
limit of 500ms, error is raise on timeout and the entire operation is aborted.

### `redis.register_function`
`redis.register_function(<function_name>, <callback>, [<description>])`
This new API allows users to register a new function that will be linked to the newly created library.
This API can only be called during the load run (see definition above). Any attempt to use it outside
of the load run will result in an error.
The parameters pass to the API are:
* function_name - Function name (must be a Lua string)
* callback - Lua function object that will be called when the function is invokes using fcall/fcall_ro
* description - Function description, optional (must be a Lua string).

### Example
The following example creates a library called `lib` with 2 functions, `f1` and `f1`, returns 1 and 2 respectively:
```
local function f1(keys, args)
    return 1
end

local function f2(keys, args)
    return 2
end

redis.register_function('f1', f1)
redis.register_function('f2', f2)
```

Notice: Unlike `eval`, functions inside a library get the KEYS and ARGV as arguments to the
functions and not as global.

### Technical Details

On the load run we only want the user to be able to call a white list on API's. This way, in
the future, if new API's will be added, the new API's will not be available to the load run
unless specifically added to this white list. We put the while list on the `library` object and
make sure the `library` object is only available to the load run by using [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv) API. This API allows us to set
the `globals` of a function (and all the function it creates). Before starting the load run we
create a new fresh Lua table (call it `g`) that only contains the `library` API (we make sure
to set global protection on this table just like the general global protection already exists
today), then we use [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv)
to set `g` as the global table of the load run. After the load run finished we update `g`
metatable and set `__index` and `__newindex` functions to be `_G` (Lua default globals),
we also pop out the `library` object as we do not need it anymore.
This way, any function that was created on the load run (and will be invoke using `fcall`) will
see the default globals as it expected to see them and will not have the `library` API anymore.

An important outcome of this new approach is that now we can achieve a distinct global table
for each library (it is not yet like that but it is very easy to achieve it now). In the future we can
decide to remove global protection because global on different libraries will not collide or we
can chose to give different API to different libraries base on some configuration or input.

Notice that this technique was meant to prevent errors and was not meant to prevent malicious
user from exploit it. For example, the load run can still save the `library` object on some local
variable and then using in `fcall` context. To prevent such a malicious use, the C code also make
sure it is running in the right context and if not raise an error.
parent 568c2e03
...@@ -3144,24 +3144,6 @@ struct redisCommandArg FCALL_RO_Args[] = { ...@@ -3144,24 +3144,6 @@ struct redisCommandArg FCALL_RO_Args[] = {
{0} {0}
}; };
/********** FUNCTION CREATE ********************/
/* FUNCTION CREATE history */
#define FUNCTION_CREATE_History NULL
/* FUNCTION CREATE hints */
#define FUNCTION_CREATE_Hints NULL
/* FUNCTION CREATE argument table */
struct redisCommandArg FUNCTION_CREATE_Args[] = {
{"engine-name",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{"function-name",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{"replace",ARG_TYPE_PURE_TOKEN,-1,"REPLACE",NULL,NULL,CMD_ARG_OPTIONAL},
{"function-description",ARG_TYPE_STRING,-1,"DESC",NULL,NULL,CMD_ARG_OPTIONAL},
{"function-code",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{0}
};
/********** FUNCTION DELETE ********************/ /********** FUNCTION DELETE ********************/
/* FUNCTION DELETE history */ /* FUNCTION DELETE history */
...@@ -3213,21 +3195,6 @@ struct redisCommandArg FUNCTION_FLUSH_Args[] = { ...@@ -3213,21 +3195,6 @@ struct redisCommandArg FUNCTION_FLUSH_Args[] = {
/* FUNCTION HELP hints */ /* FUNCTION HELP hints */
#define FUNCTION_HELP_Hints NULL #define FUNCTION_HELP_Hints NULL
/********** FUNCTION INFO ********************/
/* FUNCTION INFO history */
#define FUNCTION_INFO_History NULL
/* FUNCTION INFO hints */
#define FUNCTION_INFO_Hints NULL
/* FUNCTION INFO argument table */
struct redisCommandArg FUNCTION_INFO_Args[] = {
{"function-name",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{"withcode",ARG_TYPE_PURE_TOKEN,-1,"WITHCODE",NULL,NULL,CMD_ARG_OPTIONAL},
{0}
};
/********** FUNCTION KILL ********************/ /********** FUNCTION KILL ********************/
/* FUNCTION KILL history */ /* FUNCTION KILL history */
...@@ -3244,6 +3211,31 @@ struct redisCommandArg FUNCTION_INFO_Args[] = { ...@@ -3244,6 +3211,31 @@ struct redisCommandArg FUNCTION_INFO_Args[] = {
/* FUNCTION LIST hints */ /* FUNCTION LIST hints */
#define FUNCTION_LIST_Hints NULL #define FUNCTION_LIST_Hints NULL
/* FUNCTION LIST argument table */
struct redisCommandArg FUNCTION_LIST_Args[] = {
{"library-name-pattern",ARG_TYPE_STRING,-1,"LIBRARYNAME",NULL,NULL,CMD_ARG_OPTIONAL},
{"withcode",ARG_TYPE_PURE_TOKEN,-1,"WITHCODE",NULL,NULL,CMD_ARG_OPTIONAL},
{0}
};
/********** FUNCTION LOAD ********************/
/* FUNCTION LOAD history */
#define FUNCTION_LOAD_History NULL
/* FUNCTION LOAD hints */
#define FUNCTION_LOAD_Hints NULL
/* FUNCTION LOAD argument table */
struct redisCommandArg FUNCTION_LOAD_Args[] = {
{"engine-name",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{"library-name",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{"replace",ARG_TYPE_PURE_TOKEN,-1,"REPLACE",NULL,NULL,CMD_ARG_OPTIONAL},
{"library-description",ARG_TYPE_STRING,-1,"DESC",NULL,NULL,CMD_ARG_OPTIONAL},
{"function-code",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{0}
};
/********** FUNCTION RESTORE ********************/ /********** FUNCTION RESTORE ********************/
/* FUNCTION RESTORE history */ /* FUNCTION RESTORE history */
...@@ -3277,15 +3269,14 @@ struct redisCommandArg FUNCTION_RESTORE_Args[] = { ...@@ -3277,15 +3269,14 @@ struct redisCommandArg FUNCTION_RESTORE_Args[] = {
/* FUNCTION command table */ /* FUNCTION command table */
struct redisCommand FUNCTION_Subcommands[] = { struct redisCommand FUNCTION_Subcommands[] = {
{"create","Create a function with the given arguments (name, code, description)","O(1) (considering compilation time is redundant)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_CREATE_History,FUNCTION_CREATE_Hints,functionCreateCommand,-5,CMD_NOSCRIPT|CMD_WRITE,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_CREATE_Args},
{"delete","Delete a function by name","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_DELETE_History,FUNCTION_DELETE_Hints,functionDeleteCommand,3,CMD_NOSCRIPT|CMD_WRITE,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_DELETE_Args}, {"delete","Delete a function by name","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_DELETE_History,FUNCTION_DELETE_Hints,functionDeleteCommand,3,CMD_NOSCRIPT|CMD_WRITE,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_DELETE_Args},
{"dump","Dump all functions into a serialized binary payload","O(N) where N is the number of functions","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_DUMP_History,FUNCTION_DUMP_Hints,functionDumpCommand,2,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING}, {"dump","Dump all functions into a serialized binary payload","O(N) where N is the number of functions","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_DUMP_History,FUNCTION_DUMP_Hints,functionDumpCommand,2,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING},
{"flush","Deleting all functions","O(N) where N is the number of functions deleted","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_FLUSH_History,FUNCTION_FLUSH_Hints,functionFlushCommand,-2,CMD_NOSCRIPT|CMD_WRITE,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_FLUSH_Args}, {"flush","Deleting all functions","O(N) where N is the number of functions deleted","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_FLUSH_History,FUNCTION_FLUSH_Hints,functionFlushCommand,-2,CMD_NOSCRIPT|CMD_WRITE,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_FLUSH_Args},
{"help","Show helpful text about the different subcommands","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_HELP_History,FUNCTION_HELP_Hints,functionHelpCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SCRIPTING}, {"help","Show helpful text about the different subcommands","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_HELP_History,FUNCTION_HELP_Hints,functionHelpCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SCRIPTING},
{"info","Return information about a function by function name","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_INFO_History,FUNCTION_INFO_Hints,functionInfoCommand,-3,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_INFO_Args},
{"kill","Kill the function currently in execution.","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_KILL_History,FUNCTION_KILL_Hints,functionKillCommand,2,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING}, {"kill","Kill the function currently in execution.","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_KILL_History,FUNCTION_KILL_Hints,functionKillCommand,2,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING},
{"list","List information about all the functions","O(N) where N is the number of functions","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_LIST_History,FUNCTION_LIST_Hints,functionListCommand,2,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING}, {"list","List information about all the functions","O(N) where N is the number of functions","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_LIST_History,FUNCTION_LIST_Hints,functionListCommand,-2,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_LIST_Args},
{"restore","Restore all the functions on the given payload","O(N) where N is the number of functions on the payload","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_RESTORE_History,FUNCTION_RESTORE_Hints,functionRestoreCommand,-3,CMD_NOSCRIPT|CMD_WRITE,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_RESTORE_Args}, {"load","Create a function with the given arguments (name, code, description)","O(1) (considering compilation time is redundant)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_LOAD_History,FUNCTION_LOAD_Hints,functionLoadCommand,-5,CMD_NOSCRIPT|CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_LOAD_Args},
{"restore","Restore all the functions on the given payload","O(N) where N is the number of functions on the payload","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_RESTORE_History,FUNCTION_RESTORE_Hints,functionRestoreCommand,-3,CMD_NOSCRIPT|CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_RESTORE_Args},
{"stats","Return information about the function currently running (name, description, duration)","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_STATS_History,FUNCTION_STATS_Hints,functionStatsCommand,2,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING}, {"stats","Return information about the function currently running (name, description, duration)","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_STATS_History,FUNCTION_STATS_Hints,functionStatsCommand,2,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING},
{0} {0}
}; };
......
{
"INFO": {
"summary": "Return information about a function by function name",
"complexity": "O(1)",
"group": "scripting",
"since": "7.0.0",
"arity": -3,
"container": "FUNCTION",
"function": "functionInfoCommand",
"command_flags": [
"NOSCRIPT"
],
"acl_categories": [
"SCRIPTING"
],
"arguments": [
{
"name": "function-name",
"type": "string"
},
{
"name": "withcode",
"type": "pure-token",
"token": "WITHCODE",
"optional": true
}
]
}
}
...@@ -4,7 +4,7 @@ ...@@ -4,7 +4,7 @@
"complexity": "O(N) where N is the number of functions", "complexity": "O(N) where N is the number of functions",
"group": "scripting", "group": "scripting",
"since": "7.0.0", "since": "7.0.0",
"arity": 2, "arity": -2,
"container": "FUNCTION", "container": "FUNCTION",
"function": "functionListCommand", "function": "functionListCommand",
"command_flags": [ "command_flags": [
...@@ -12,6 +12,20 @@ ...@@ -12,6 +12,20 @@
], ],
"acl_categories": [ "acl_categories": [
"SCRIPTING" "SCRIPTING"
],
"arguments": [
{
"name": "library-name-pattern",
"type": "string",
"token": "LIBRARYNAME",
"optional": true
},
{
"name": "withcode",
"type": "pure-token",
"token": "WITHCODE",
"optional": true
}
] ]
} }
} }
{ {
"CREATE": { "LOAD": {
"summary": "Create a function with the given arguments (name, code, description)", "summary": "Create a function with the given arguments (name, code, description)",
"complexity": "O(1) (considering compilation time is redundant)", "complexity": "O(1) (considering compilation time is redundant)",
"group": "scripting", "group": "scripting",
"since": "7.0.0", "since": "7.0.0",
"arity": -5, "arity": -5,
"container": "FUNCTION", "container": "FUNCTION",
"function": "functionCreateCommand", "function": "functionLoadCommand",
"command_flags": [ "command_flags": [
"NOSCRIPT", "NOSCRIPT",
"WRITE" "WRITE",
"DENYOOM"
], ],
"acl_categories": [ "acl_categories": [
"SCRIPTING" "SCRIPTING"
...@@ -20,7 +21,7 @@ ...@@ -20,7 +21,7 @@
"type": "string" "type": "string"
}, },
{ {
"name": "function-name", "name": "library-name",
"type": "string" "type": "string"
}, },
{ {
...@@ -30,7 +31,7 @@ ...@@ -30,7 +31,7 @@
"optional": true "optional": true
}, },
{ {
"name": "function-description", "name": "library-description",
"type": "string", "type": "string",
"token": "DESC", "token": "DESC",
"optional": true "optional": true
......
...@@ -9,7 +9,8 @@ ...@@ -9,7 +9,8 @@
"function": "functionRestoreCommand", "function": "functionRestoreCommand",
"command_flags": [ "command_flags": [
"NOSCRIPT", "NOSCRIPT",
"WRITE" "WRITE",
"DENYOOM"
], ],
"acl_categories": [ "acl_categories": [
"SCRIPTING" "SCRIPTING"
......
...@@ -461,7 +461,7 @@ long long emptyData(int dbnum, int flags, void(callback)(dict*)) { ...@@ -461,7 +461,7 @@ long long emptyData(int dbnum, int flags, void(callback)(dict*)) {
if (with_functions) { if (with_functions) {
serverAssert(dbnum == -1); serverAssert(dbnum == -1);
functionsCtxClearCurrent(async); functionsLibCtxClearCurrent(async);
} }
/* Also fire the end event. Note that this event will fire almost /* Also fire the end event. Note that this event will fire almost
......
...@@ -48,6 +48,9 @@ ...@@ -48,6 +48,9 @@
#define LUA_ENGINE_NAME "LUA" #define LUA_ENGINE_NAME "LUA"
#define REGISTRY_ENGINE_CTX_NAME "__ENGINE_CTX__" #define REGISTRY_ENGINE_CTX_NAME "__ENGINE_CTX__"
#define REGISTRY_ERROR_HANDLER_NAME "__ERROR_HANDLER__" #define REGISTRY_ERROR_HANDLER_NAME "__ERROR_HANDLER__"
#define REGISTRY_LOAD_CTX_NAME "__LIBRARY_CTX__"
#define LIBRARY_API_NAME "__LIBRARY_API__"
#define LOAD_TIMEOUT_MS 500
/* Lua engine ctx */ /* Lua engine ctx */
typedef struct luaEngineCtx { typedef struct luaEngineCtx {
...@@ -60,6 +63,27 @@ typedef struct luaFunctionCtx { ...@@ -60,6 +63,27 @@ typedef struct luaFunctionCtx {
int lua_function_ref; int lua_function_ref;
} luaFunctionCtx; } luaFunctionCtx;
typedef struct loadCtx {
functionLibInfo *li;
monotime start_time;
} loadCtx;
/* Hook for FUNCTION LOAD execution.
* Used to cancel the execution in case of a timeout (500ms).
* This execution should be fast and should only register
* functions so 500ms should be more than enough. */
static void luaEngineLoadHook(lua_State *lua, lua_Debug *ar) {
UNUSED(ar);
loadCtx *load_ctx = luaGetFromRegistry(lua, REGISTRY_LOAD_CTX_NAME);
uint64_t duration = elapsedMs(load_ctx->start_time);
if (duration > LOAD_TIMEOUT_MS) {
lua_sethook(lua, luaEngineLoadHook, LUA_MASKLINE, 0);
lua_pushstring(lua,"FUNCTION LOAD timeout");
lua_error(lua);
}
}
/* /*
* Compile a given blob and save it on the registry. * Compile a given blob and save it on the registry.
* Return a function ctx with Lua ref that allows to later retrieve the * Return a function ctx with Lua ref that allows to later retrieve the
...@@ -67,25 +91,88 @@ typedef struct luaFunctionCtx { ...@@ -67,25 +91,88 @@ typedef struct luaFunctionCtx {
* *
* Return NULL on compilation error and set the error to the err variable * Return NULL on compilation error and set the error to the err variable
*/ */
static void* luaEngineCreate(void *engine_ctx, sds blob, sds *err) { static int luaEngineCreate(void *engine_ctx, functionLibInfo *li, sds blob, sds *err) {
luaEngineCtx *lua_engine_ctx = engine_ctx; luaEngineCtx *lua_engine_ctx = engine_ctx;
lua_State *lua = lua_engine_ctx->lua; lua_State *lua = lua_engine_ctx->lua;
/* Each library will have its own global distinct table.
* We will create a new fresh Lua table and use
* lua_setfenv to set the table as the library globals
* (https://www.lua.org/manual/5.1/manual.html#lua_setfenv)
*
* At first, populate this new table with only the 'library' API
* to make sure only 'library' API is available at start. After the
* initial run is finished and all functions are registered, add
* all the default globals to the library global table and delete
* the library API.
*
* There are 2 ways to achieve the last part (add default
* globals to the new table):
*
* 1. Initialize the new table with all the default globals
* 2. Inheritance using metatable (https://www.lua.org/pil/14.3.html)
*
* For now we are choosing the second, we can change it in the future to
* achieve a better isolation between functions. */
lua_newtable(lua); /* Global table for the library */
lua_pushstring(lua, REDIS_API_NAME);
lua_pushstring(lua, LIBRARY_API_NAME);
lua_gettable(lua, LUA_REGISTRYINDEX); /* get library function from registry */
lua_settable(lua, -3); /* push the library table to the new global table */
/* Set global protection on the new global table */
luaSetGlobalProtection(lua_engine_ctx->lua);
/* compile the code */
if (luaL_loadbuffer(lua, blob, sdslen(blob), "@user_function")) { if (luaL_loadbuffer(lua, blob, sdslen(blob), "@user_function")) {
*err = sdsempty(); *err = sdscatprintf(sdsempty(), "Error compiling function: %s", lua_tostring(lua, -1));
*err = sdscatprintf(*err, "Error compiling function: %s", lua_pop(lua, 2); /* pops the error and globals table */
lua_tostring(lua, -1)); return C_ERR;
lua_pop(lua, 1);
return NULL;
} }
serverAssert(lua_isfunction(lua, -1)); serverAssert(lua_isfunction(lua, -1));
int lua_function_ref = luaL_ref(lua, LUA_REGISTRYINDEX); loadCtx load_ctx = {
.li = li,
.start_time = getMonotonicUs(),
};
luaSaveOnRegistry(lua, REGISTRY_LOAD_CTX_NAME, &load_ctx);
luaFunctionCtx *f_ctx = zmalloc(sizeof(*f_ctx)); /* set the function environment so only 'library' API can be accessed. */
*f_ctx = (luaFunctionCtx ) { .lua_function_ref = lua_function_ref, }; lua_pushvalue(lua, -2); /* push global table to the front */
lua_setfenv(lua, -2);
return f_ctx; lua_sethook(lua,luaEngineLoadHook,LUA_MASKCOUNT,100000);
/* Run the compiled code to allow it to register functions */
if (lua_pcall(lua,0,0,0)) {
*err = sdscatprintf(sdsempty(), "Error registering functions: %s", lua_tostring(lua, -1));
lua_pop(lua, 2); /* pops the error and globals table */
lua_sethook(lua,NULL,0,0); /* Disable hook */
luaSaveOnRegistry(lua, REGISTRY_LOAD_CTX_NAME, NULL);
return C_ERR;
}
lua_sethook(lua,NULL,0,0); /* Disable hook */
luaSaveOnRegistry(lua, REGISTRY_LOAD_CTX_NAME, NULL);
/* stack contains the global table, lets rearrange it to contains the entire API. */
/* delete 'redis' API */
lua_pushstring(lua, REDIS_API_NAME);
lua_pushnil(lua);
lua_settable(lua, -3);
/* create metatable */
lua_newtable(lua);
lua_pushstring(lua, "__index");
lua_pushvalue(lua, LUA_GLOBALSINDEX); /* push original globals */
lua_settable(lua, -3);
lua_pushstring(lua, "__newindex");
lua_pushvalue(lua, LUA_GLOBALSINDEX); /* push original globals */
lua_settable(lua, -3);
lua_setmetatable(lua, -2);
lua_pop(lua, 1); /* pops the global table */
return C_OK;
} }
/* /*
...@@ -137,6 +224,64 @@ static void luaEngineFreeFunction(void *engine_ctx, void *compiled_function) { ...@@ -137,6 +224,64 @@ static void luaEngineFreeFunction(void *engine_ctx, void *compiled_function) {
zfree(f_ctx); zfree(f_ctx);
} }
static int luaRegisterFunction(lua_State *lua) {
int argc = lua_gettop(lua);
if (argc < 2 || argc > 3) {
luaPushError(lua, "wrong number of arguments to redis.register_function");
return luaRaiseError(lua);
}
loadCtx *load_ctx = luaGetFromRegistry(lua, REGISTRY_LOAD_CTX_NAME);
if (!load_ctx) {
luaPushError(lua, "redis.register_function can only be called on FUNCTION LOAD command");
return luaRaiseError(lua);
}
if (!lua_isstring(lua, 1)) {
luaPushError(lua, "first argument to redis.register_function must be a string");
return luaRaiseError(lua);
}
if (!lua_isfunction(lua, 2)) {
luaPushError(lua, "second argument to redis.register_function must be a function");
return luaRaiseError(lua);
}
if (argc == 3 && !lua_isstring(lua, 3)) {
luaPushError(lua, "third argument to redis.register_function must be a string");
return luaRaiseError(lua);
}
size_t function_name_len;
const char *function_name = lua_tolstring(lua, 1, &function_name_len);
sds function_name_sds = sdsnewlen(function_name, function_name_len);
sds desc_sds = NULL;
if (argc == 3){
size_t desc_len;
const char *desc = lua_tolstring(lua, 3, &desc_len);
desc_sds = sdsnewlen(desc, desc_len);
lua_pop(lua, 1); /* pop out the description */
}
int lua_function_ref = luaL_ref(lua, LUA_REGISTRYINDEX);
luaFunctionCtx *lua_f_ctx = zmalloc(sizeof(*lua_f_ctx));
*lua_f_ctx = (luaFunctionCtx ) { .lua_function_ref = lua_function_ref, };
sds err = NULL;
if (functionLibCreateFunction(function_name_sds, lua_f_ctx, load_ctx->li, desc_sds, &err) != C_OK) {
sdsfree(function_name_sds);
if (desc_sds) sdsfree(desc_sds);
lua_unref(lua, lua_f_ctx->lua_function_ref);
zfree(lua_f_ctx);
luaPushError(lua, err);
sdsfree(err);
return luaRaiseError(lua);
}
return 0;
}
/* Initialize Lua engine, should be called once on start. */ /* Initialize Lua engine, should be called once on start. */
int luaEngineInitEngine() { int luaEngineInitEngine() {
luaEngineCtx *lua_engine_ctx = zmalloc(sizeof(*lua_engine_ctx)); luaEngineCtx *lua_engine_ctx = zmalloc(sizeof(*lua_engine_ctx));
...@@ -144,6 +289,18 @@ int luaEngineInitEngine() { ...@@ -144,6 +289,18 @@ int luaEngineInitEngine() {
luaRegisterRedisAPI(lua_engine_ctx->lua); luaRegisterRedisAPI(lua_engine_ctx->lua);
/* Register the library commands table and fields and store it to registry */
lua_pushstring(lua_engine_ctx->lua, LIBRARY_API_NAME);
lua_newtable(lua_engine_ctx->lua);
lua_pushstring(lua_engine_ctx->lua, "register_function");
lua_pushcfunction(lua_engine_ctx->lua, luaRegisterFunction);
lua_settable(lua_engine_ctx->lua, -3);
luaRegisterLogFunction(lua_engine_ctx->lua);
lua_settable(lua_engine_ctx->lua, LUA_REGISTRYINDEX);
/* Save error handler to registry */ /* Save error handler to registry */
lua_pushstring(lua_engine_ctx->lua, REGISTRY_ERROR_HANDLER_NAME); lua_pushstring(lua_engine_ctx->lua, REGISTRY_ERROR_HANDLER_NAME);
char *errh_func = "local dbg = debug\n" char *errh_func = "local dbg = debug\n"
...@@ -163,11 +320,16 @@ int luaEngineInitEngine() { ...@@ -163,11 +320,16 @@ int luaEngineInitEngine() {
lua_pcall(lua_engine_ctx->lua,0,1,0); lua_pcall(lua_engine_ctx->lua,0,1,0);
lua_settable(lua_engine_ctx->lua, LUA_REGISTRYINDEX); lua_settable(lua_engine_ctx->lua, LUA_REGISTRYINDEX);
/* save the engine_ctx on the registry so we can get it from the Lua interpreter */ /* Save global protection to registry */
luaSaveOnRegistry(lua_engine_ctx->lua, REGISTRY_ENGINE_CTX_NAME, lua_engine_ctx); luaRegisterGlobalProtectionFunction(lua_engine_ctx->lua);
luaEnableGlobalsProtection(lua_engine_ctx->lua, 0); /* Set global protection on globals */
lua_pushvalue(lua_engine_ctx->lua, LUA_GLOBALSINDEX);
luaSetGlobalProtection(lua_engine_ctx->lua);
lua_pop(lua_engine_ctx->lua, 1);
/* save the engine_ctx on the registry so we can get it from the Lua interpreter */
luaSaveOnRegistry(lua_engine_ctx->lua, REGISTRY_ENGINE_CTX_NAME, lua_engine_ctx);
engine *lua_engine = zmalloc(sizeof(*lua_engine)); engine *lua_engine = zmalloc(sizeof(*lua_engine));
*lua_engine = (engine) { *lua_engine = (engine) {
......
This diff is collapsed.
...@@ -47,13 +47,15 @@ ...@@ -47,13 +47,15 @@
#include "script.h" #include "script.h"
#include "redismodule.h" #include "redismodule.h"
typedef struct functionLibInfo functionLibInfo;
typedef struct engine { typedef struct engine {
/* engine specific context */ /* engine specific context */
void *engine_ctx; void *engine_ctx;
/* Create function callback, get the engine_ctx, and function code. /* Create function callback, get the engine_ctx, and function code.
* returns NULL on error and set sds to be the error message */ * returns NULL on error and set sds to be the error message */
void* (*create)(void *engine_ctx, sds code, sds *err); int (*create)(void *engine_ctx, functionLibInfo *li, sds code, sds *err);
/* Invoking a function, r_ctx is an opaque object (from engine POV). /* Invoking a function, r_ctx is an opaque object (from engine POV).
* The r_ctx should be used by the engine to interaction with Redis, * The r_ctx should be used by the engine to interaction with Redis,
...@@ -89,29 +91,40 @@ typedef struct engineInfo { ...@@ -89,29 +91,40 @@ typedef struct engineInfo {
/* Hold information about the specific function. /* Hold information about the specific function.
* Used on rdb.c so it must be declared here. */ * Used on rdb.c so it must be declared here. */
typedef struct functionInfo { typedef struct functionInfo {
sds name; /* Function name */ sds name; /* Function name */
void *function; /* Opaque object that set by the function's engine and allow it void *function; /* Opaque object that set by the function's engine and allow it
to run the function, usually it's the function compiled code. */ to run the function, usually it's the function compiled code. */
engineInfo *ei; /* Pointer to the function engine */ functionLibInfo* li; /* Pointer to the library created the function */
sds code; /* Function code */ sds desc; /* Function description */
sds desc; /* Function description */
} functionInfo; } functionInfo;
/* Hold information about the specific library.
* Used on rdb.c so it must be declared here. */
struct functionLibInfo {
sds name; /* Library name */
dict *functions; /* Functions dictionary */
engineInfo *ei; /* Pointer to the function engine */
sds code; /* Library code */
sds desc; /* Library description */
};
int functionsRegisterEngine(const char *engine_name, engine *engine_ctx); int functionsRegisterEngine(const char *engine_name, engine *engine_ctx);
int functionsCreateWithFunctionCtx(sds function_name, sds engine_name, sds desc, sds code, int functionsCreateWithLibraryCtx(sds lib_name, sds engine_name, sds desc, sds code,
int replace, sds* err, functionsCtx *functions); int replace, sds* err, functionsLibCtx *lib_ctx);
unsigned long functionsMemory(); unsigned long functionsMemory();
unsigned long functionsMemoryOverhead(); unsigned long functionsMemoryOverhead();
int functionsLoad(rio *rdb, int ver);
unsigned long functionsNum(); unsigned long functionsNum();
dict* functionsGet(); unsigned long functionsLibNum();
size_t functionsLen(functionsCtx *functions_ctx); dict* functionsLibGet();
functionsCtx* functionsCtxGetCurrent(); size_t functionsLibCtxfunctionsLen(functionsLibCtx *functions_ctx);
functionsCtx* functionsCtxCreate(); functionsLibCtx* functionsLibCtxGetCurrent();
void functionsCtxClearCurrent(int async); functionsLibCtx* functionsLibCtxCreate();
void functionsCtxFree(functionsCtx *functions_ctx); void functionsLibCtxClearCurrent(int async);
void functionsCtxClear(functionsCtx *functions_ctx); void functionsLibCtxFree(functionsLibCtx *lib_ctx);
void functionsCtxSwapWithCurrent(functionsCtx *functions_ctx); void functionsLibCtxClear(functionsLibCtx *lib_ctx);
void functionsLibCtxSwapWithCurrent(functionsLibCtx *lib_ctx);
int functionLibCreateFunction(sds name, void *function, functionLibInfo *li, sds desc, sds *err);
int luaEngineInitEngine(); int luaEngineInitEngine();
int functionsInit(); int functionsInit();
......
...@@ -49,9 +49,9 @@ void lazyFreeLuaScripts(void *args[]) { ...@@ -49,9 +49,9 @@ void lazyFreeLuaScripts(void *args[]) {
/* Release the functions ctx. */ /* Release the functions ctx. */
void lazyFreeFunctionsCtx(void *args[]) { void lazyFreeFunctionsCtx(void *args[]) {
functionsCtx *f_ctx = args[0]; functionsLibCtx *functions_lib_ctx = args[0];
size_t len = functionsLen(f_ctx); size_t len = functionsLibCtxfunctionsLen(functions_lib_ctx);
functionsCtxFree(f_ctx); functionsLibCtxFree(functions_lib_ctx);
atomicDecr(lazyfree_objects,len); atomicDecr(lazyfree_objects,len);
atomicIncr(lazyfreed_objects,len); atomicIncr(lazyfreed_objects,len);
} }
...@@ -204,12 +204,12 @@ void freeLuaScriptsAsync(dict *lua_scripts) { ...@@ -204,12 +204,12 @@ void freeLuaScriptsAsync(dict *lua_scripts) {
} }
/* Free functions ctx, if the functions ctx contains enough functions, free it in async way. */ /* Free functions ctx, if the functions ctx contains enough functions, free it in async way. */
void freeFunctionsAsync(functionsCtx *f_ctx) { void freeFunctionsAsync(functionsLibCtx *functions_lib_ctx) {
if (functionsLen(f_ctx) > LAZYFREE_THRESHOLD) { if (functionsLibCtxfunctionsLen(functions_lib_ctx) > LAZYFREE_THRESHOLD) {
atomicIncr(lazyfree_objects,functionsLen(f_ctx)); atomicIncr(lazyfree_objects,functionsLibCtxfunctionsLen(functions_lib_ctx));
bioCreateLazyFreeJob(lazyFreeFunctionsCtx,1,f_ctx); bioCreateLazyFreeJob(lazyFreeFunctionsCtx,1,functions_lib_ctx);
} else { } else {
functionsCtxFree(f_ctx); functionsLibCtxFree(functions_lib_ctx);
} }
} }
......
...@@ -1215,7 +1215,7 @@ ssize_t rdbSaveSingleModuleAux(rio *rdb, int when, moduleType *mt) { ...@@ -1215,7 +1215,7 @@ ssize_t rdbSaveSingleModuleAux(rio *rdb, int when, moduleType *mt) {
} }
ssize_t rdbSaveFunctions(rio *rdb) { ssize_t rdbSaveFunctions(rio *rdb) {
dict *functions = functionsGet(); dict *functions = functionsLibGet();
dictIterator *iter = dictGetIterator(functions); dictIterator *iter = dictGetIterator(functions);
dictEntry *entry = NULL; dictEntry *entry = NULL;
ssize_t written = 0; ssize_t written = 0;
...@@ -1223,23 +1223,23 @@ ssize_t rdbSaveFunctions(rio *rdb) { ...@@ -1223,23 +1223,23 @@ ssize_t rdbSaveFunctions(rio *rdb) {
while ((entry = dictNext(iter))) { while ((entry = dictNext(iter))) {
if ((ret = rdbSaveType(rdb, RDB_OPCODE_FUNCTION)) < 0) goto werr; if ((ret = rdbSaveType(rdb, RDB_OPCODE_FUNCTION)) < 0) goto werr;
written += ret; written += ret;
functionInfo *fi = dictGetVal(entry); functionLibInfo *li = dictGetVal(entry);
if ((ret = rdbSaveRawString(rdb, (unsigned char *) fi->name, sdslen(fi->name))) < 0) goto werr; if ((ret = rdbSaveRawString(rdb, (unsigned char *) li->name, sdslen(li->name))) < 0) goto werr;
written += ret; written += ret;
if ((ret = rdbSaveRawString(rdb, (unsigned char *) fi->ei->name, sdslen(fi->ei->name))) < 0) goto werr; if ((ret = rdbSaveRawString(rdb, (unsigned char *) li->ei->name, sdslen(li->ei->name))) < 0) goto werr;
written += ret; written += ret;
if (fi->desc) { if (li->desc) {
/* desc exists */ /* desc exists */
if ((ret = rdbSaveLen(rdb, 1)) < 0) goto werr; if ((ret = rdbSaveLen(rdb, 1)) < 0) goto werr;
written += ret; written += ret;
if ((ret = rdbSaveRawString(rdb, (unsigned char *) fi->desc, sdslen(fi->desc))) < 0) goto werr; if ((ret = rdbSaveRawString(rdb, (unsigned char *) li->desc, sdslen(li->desc))) < 0) goto werr;
written += ret; written += ret;
} else { } else {
/* desc not exists */ /* desc not exists */
if ((ret = rdbSaveLen(rdb, 0)) < 0) goto werr; if ((ret = rdbSaveLen(rdb, 0)) < 0) goto werr;
written += ret; written += ret;
} }
if ((ret = rdbSaveRawString(rdb, (unsigned char *) fi->code, sdslen(fi->code))) < 0) goto werr; if ((ret = rdbSaveRawString(rdb, (unsigned char *) li->code, sdslen(li->code))) < 0) goto werr;
written += ret; written += ret;
} }
dictReleaseIterator(iter); dictReleaseIterator(iter);
...@@ -2746,7 +2746,7 @@ void rdbLoadProgressCallback(rio *r, const void *buf, size_t len) { ...@@ -2746,7 +2746,7 @@ void rdbLoadProgressCallback(rio *r, const void *buf, size_t len) {
* The err output parameter is optional and will be set with relevant error * The err output parameter is optional and will be set with relevant error
* message on failure, it is the caller responsibility to free the error * message on failure, it is the caller responsibility to free the error
* message on failure. */ * message on failure. */
int rdbFunctionLoad(rio *rdb, int ver, functionsCtx* functions_ctx, int rdbflags, sds *err) { int rdbFunctionLoad(rio *rdb, int ver, functionsLibCtx* lib_ctx, int rdbflags, sds *err) {
UNUSED(ver); UNUSED(ver);
sds name = NULL; sds name = NULL;
sds engine_name = NULL; sds engine_name = NULL;
...@@ -2756,7 +2756,7 @@ int rdbFunctionLoad(rio *rdb, int ver, functionsCtx* functions_ctx, int rdbflags ...@@ -2756,7 +2756,7 @@ int rdbFunctionLoad(rio *rdb, int ver, functionsCtx* functions_ctx, int rdbflags
sds error = NULL; sds error = NULL;
int res = C_ERR; int res = C_ERR;
if (!(name = rdbGenericLoadStringObject(rdb, RDB_LOAD_SDS, NULL))) { if (!(name = rdbGenericLoadStringObject(rdb, RDB_LOAD_SDS, NULL))) {
error = sdsnew("Failed loading function name"); error = sdsnew("Failed loading library name");
goto error; goto error;
} }
...@@ -2766,23 +2766,23 @@ int rdbFunctionLoad(rio *rdb, int ver, functionsCtx* functions_ctx, int rdbflags ...@@ -2766,23 +2766,23 @@ int rdbFunctionLoad(rio *rdb, int ver, functionsCtx* functions_ctx, int rdbflags
} }
if ((has_desc = rdbLoadLen(rdb, NULL)) == RDB_LENERR) { if ((has_desc = rdbLoadLen(rdb, NULL)) == RDB_LENERR) {
error = sdsnew("Failed loading function description indicator"); error = sdsnew("Failed loading library description indicator");
goto error; goto error;
} }
if (has_desc && !(desc = rdbGenericLoadStringObject(rdb, RDB_LOAD_SDS, NULL))) { if (has_desc && !(desc = rdbGenericLoadStringObject(rdb, RDB_LOAD_SDS, NULL))) {
error = sdsnew("Failed loading function description"); error = sdsnew("Failed loading library description");
goto error; goto error;
} }
if (!(blob = rdbGenericLoadStringObject(rdb, RDB_LOAD_SDS, NULL))) { if (!(blob = rdbGenericLoadStringObject(rdb, RDB_LOAD_SDS, NULL))) {
error = sdsnew("Failed loading function blob"); error = sdsnew("Failed loading library blob");
goto error; goto error;
} }
if (functionsCreateWithFunctionCtx(name, engine_name, desc, blob, rdbflags & RDBFLAGS_ALLOW_DUP, &error, functions_ctx) != C_OK) { if (functionsCreateWithLibraryCtx(name, engine_name, desc, blob, rdbflags & RDBFLAGS_ALLOW_DUP, &error, lib_ctx) != C_OK) {
if (!error) { if (!error) {
error = sdsnew("Failed creating the function"); error = sdsnew("Failed creating the library");
} }
goto error; goto error;
} }
...@@ -2808,8 +2808,8 @@ error: ...@@ -2808,8 +2808,8 @@ error:
/* Load an RDB file from the rio stream 'rdb'. On success C_OK is returned, /* Load an RDB file from the rio stream 'rdb'. On success C_OK is returned,
* otherwise C_ERR is returned and 'errno' is set accordingly. */ * otherwise C_ERR is returned and 'errno' is set accordingly. */
int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi) { int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi) {
functionsCtx* functions_ctx = functionsCtxGetCurrent(); functionsLibCtx* functions_lib_ctx = functionsLibCtxGetCurrent();
rdbLoadingCtx loading_ctx = { .dbarray = server.db, .functions_ctx = functions_ctx }; rdbLoadingCtx loading_ctx = { .dbarray = server.db, .functions_lib_ctx = functions_lib_ctx };
int retval = rdbLoadRioWithLoadingCtx(rdb,rdbflags,rsi,&loading_ctx); int retval = rdbLoadRioWithLoadingCtx(rdb,rdbflags,rsi,&loading_ctx);
return retval; return retval;
} }
...@@ -2818,7 +2818,7 @@ int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi) { ...@@ -2818,7 +2818,7 @@ int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi) {
/* Load an RDB file from the rio stream 'rdb'. On success C_OK is returned, /* Load an RDB file from the rio stream 'rdb'. On success C_OK is returned,
* otherwise C_ERR is returned and 'errno' is set accordingly. * otherwise C_ERR is returned and 'errno' is set accordingly.
* The rdb_loading_ctx argument holds objects to which the rdb will be loaded to, * The rdb_loading_ctx argument holds objects to which the rdb will be loaded to,
* currently it only allow to set db object and functionsCtx to which the data * currently it only allow to set db object and functionLibCtx to which the data
* will be loaded (in the future it might contains more such objects). */ * will be loaded (in the future it might contains more such objects). */
int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadingCtx *rdb_loading_ctx) { int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadingCtx *rdb_loading_ctx) {
uint64_t dbid = 0; uint64_t dbid = 0;
...@@ -3023,8 +3023,8 @@ int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadin ...@@ -3023,8 +3023,8 @@ int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadin
} }
} else if (type == RDB_OPCODE_FUNCTION) { } else if (type == RDB_OPCODE_FUNCTION) {
sds err = NULL; sds err = NULL;
if (rdbFunctionLoad(rdb, rdbver, rdb_loading_ctx->functions_ctx, rdbflags, &err) != C_OK) { if (rdbFunctionLoad(rdb, rdbver, rdb_loading_ctx->functions_lib_ctx, rdbflags, &err) != C_OK) {
serverLog(LL_WARNING,"Failed loading function, %s", err); serverLog(LL_WARNING,"Failed loading library, %s", err);
sdsfree(err); sdsfree(err);
goto eoferr; goto eoferr;
} }
......
...@@ -169,7 +169,7 @@ int rdbSaveBinaryFloatValue(rio *rdb, float val); ...@@ -169,7 +169,7 @@ int rdbSaveBinaryFloatValue(rio *rdb, float val);
int rdbLoadBinaryFloatValue(rio *rdb, float *val); int rdbLoadBinaryFloatValue(rio *rdb, float *val);
int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi); int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi);
int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadingCtx *rdb_loading_ctx); int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadingCtx *rdb_loading_ctx);
int rdbFunctionLoad(rio *rdb, int ver, functionsCtx* functions_ctx, int rdbflags, sds *err); int rdbFunctionLoad(rio *rdb, int ver, functionsLibCtx* lib_ctx, int rdbflags, sds *err);
int rdbSaveRio(int req, rio *rdb, int *error, int rdbflags, rdbSaveInfo *rsi); int rdbSaveRio(int req, rio *rdb, int *error, int rdbflags, rdbSaveInfo *rsi);
ssize_t rdbSaveFunctions(rio *rdb); ssize_t rdbSaveFunctions(rio *rdb);
rdbSaveInfo *rdbPopulateSaveInfo(rdbSaveInfo *rsi); rdbSaveInfo *rdbPopulateSaveInfo(rdbSaveInfo *rsi);
......
...@@ -1792,7 +1792,7 @@ void readSyncBulkPayload(connection *conn) { ...@@ -1792,7 +1792,7 @@ void readSyncBulkPayload(connection *conn) {
ssize_t nread, readlen, nwritten; ssize_t nread, readlen, nwritten;
int use_diskless_load = useDisklessLoad(); int use_diskless_load = useDisklessLoad();
redisDb *diskless_load_tempDb = NULL; redisDb *diskless_load_tempDb = NULL;
functionsCtx* temp_functions_ctx = NULL; functionsLibCtx* temp_functions_lib_ctx = NULL;
int empty_db_flags = server.repl_slave_lazy_flush ? EMPTYDB_ASYNC : int empty_db_flags = server.repl_slave_lazy_flush ? EMPTYDB_ASYNC :
EMPTYDB_NO_FLAGS; EMPTYDB_NO_FLAGS;
off_t left; off_t left;
...@@ -1968,7 +1968,7 @@ void readSyncBulkPayload(connection *conn) { ...@@ -1968,7 +1968,7 @@ void readSyncBulkPayload(connection *conn) {
if (use_diskless_load && server.repl_diskless_load == REPL_DISKLESS_LOAD_SWAPDB) { if (use_diskless_load && server.repl_diskless_load == REPL_DISKLESS_LOAD_SWAPDB) {
/* Initialize empty tempDb dictionaries. */ /* Initialize empty tempDb dictionaries. */
diskless_load_tempDb = disklessLoadInitTempDb(); diskless_load_tempDb = disklessLoadInitTempDb();
temp_functions_ctx = functionsCtxCreate(); temp_functions_lib_ctx = functionsLibCtxCreate();
moduleFireServerEvent(REDISMODULE_EVENT_REPL_ASYNC_LOAD, moduleFireServerEvent(REDISMODULE_EVENT_REPL_ASYNC_LOAD,
REDISMODULE_SUBEVENT_REPL_ASYNC_LOAD_STARTED, REDISMODULE_SUBEVENT_REPL_ASYNC_LOAD_STARTED,
...@@ -1991,7 +1991,7 @@ void readSyncBulkPayload(connection *conn) { ...@@ -1991,7 +1991,7 @@ void readSyncBulkPayload(connection *conn) {
if (use_diskless_load) { if (use_diskless_load) {
rio rdb; rio rdb;
redisDb *dbarray; redisDb *dbarray;
functionsCtx* functions_ctx; functionsLibCtx* functions_lib_ctx;
int asyncLoading = 0; int asyncLoading = 0;
if (server.repl_diskless_load == REPL_DISKLESS_LOAD_SWAPDB) { if (server.repl_diskless_load == REPL_DISKLESS_LOAD_SWAPDB) {
...@@ -2004,11 +2004,11 @@ void readSyncBulkPayload(connection *conn) { ...@@ -2004,11 +2004,11 @@ void readSyncBulkPayload(connection *conn) {
asyncLoading = 1; asyncLoading = 1;
} }
dbarray = diskless_load_tempDb; dbarray = diskless_load_tempDb;
functions_ctx = temp_functions_ctx; functions_lib_ctx = temp_functions_lib_ctx;
} else { } else {
dbarray = server.db; dbarray = server.db;
functions_ctx = functionsCtxGetCurrent(); functions_lib_ctx = functionsLibCtxGetCurrent();
functionsCtxClear(functions_ctx); functionsLibCtxClear(functions_lib_ctx);
} }
rioInitWithConn(&rdb,conn,server.repl_transfer_size); rioInitWithConn(&rdb,conn,server.repl_transfer_size);
...@@ -2020,7 +2020,7 @@ void readSyncBulkPayload(connection *conn) { ...@@ -2020,7 +2020,7 @@ void readSyncBulkPayload(connection *conn) {
startLoading(server.repl_transfer_size, RDBFLAGS_REPLICATION, asyncLoading); startLoading(server.repl_transfer_size, RDBFLAGS_REPLICATION, asyncLoading);
int loadingFailed = 0; int loadingFailed = 0;
rdbLoadingCtx loadingCtx = { .dbarray = dbarray, .functions_ctx = functions_ctx }; rdbLoadingCtx loadingCtx = { .dbarray = dbarray, .functions_lib_ctx = functions_lib_ctx };
if (rdbLoadRioWithLoadingCtx(&rdb,RDBFLAGS_REPLICATION,&rsi,&loadingCtx) != C_OK) { if (rdbLoadRioWithLoadingCtx(&rdb,RDBFLAGS_REPLICATION,&rsi,&loadingCtx) != C_OK) {
/* RDB loading failed. */ /* RDB loading failed. */
serverLog(LL_WARNING, serverLog(LL_WARNING,
...@@ -2049,7 +2049,7 @@ void readSyncBulkPayload(connection *conn) { ...@@ -2049,7 +2049,7 @@ void readSyncBulkPayload(connection *conn) {
NULL); NULL);
disklessLoadDiscardTempDb(diskless_load_tempDb); disklessLoadDiscardTempDb(diskless_load_tempDb);
functionsCtxFree(temp_functions_ctx); functionsLibCtxFree(temp_functions_lib_ctx);
serverLog(LL_NOTICE, "MASTER <-> REPLICA sync: Discarding temporary DB in background"); serverLog(LL_NOTICE, "MASTER <-> REPLICA sync: Discarding temporary DB in background");
} else { } else {
/* Remove the half-loaded data in case we started with an empty replica. */ /* Remove the half-loaded data in case we started with an empty replica. */
...@@ -2073,7 +2073,7 @@ void readSyncBulkPayload(connection *conn) { ...@@ -2073,7 +2073,7 @@ void readSyncBulkPayload(connection *conn) {
swapMainDbWithTempDb(diskless_load_tempDb); swapMainDbWithTempDb(diskless_load_tempDb);
/* swap existing functions ctx with the temporary one */ /* swap existing functions ctx with the temporary one */
functionsCtxSwapWithCurrent(temp_functions_ctx); functionsLibCtxSwapWithCurrent(temp_functions_lib_ctx);
moduleFireServerEvent(REDISMODULE_EVENT_REPL_ASYNC_LOAD, moduleFireServerEvent(REDISMODULE_EVENT_REPL_ASYNC_LOAD,
REDISMODULE_SUBEVENT_REPL_ASYNC_LOAD_COMPLETED, REDISMODULE_SUBEVENT_REPL_ASYNC_LOAD_COMPLETED,
......
...@@ -80,6 +80,9 @@ void* luaGetFromRegistry(lua_State* lua, const char* name) { ...@@ -80,6 +80,9 @@ void* luaGetFromRegistry(lua_State* lua, const char* name) {
lua_pushstring(lua, name); lua_pushstring(lua, name);
lua_gettable(lua, LUA_REGISTRYINDEX); lua_gettable(lua, LUA_REGISTRYINDEX);
if (lua_isnil(lua, -1)) {
return NULL;
}
/* must be light user data */ /* must be light user data */
serverAssert(lua_islightuserdata(lua, -1)); serverAssert(lua_islightuserdata(lua, -1));
...@@ -427,7 +430,7 @@ static void redisProtocolToLuaType_Double(void *ctx, double d, const char *proto ...@@ -427,7 +430,7 @@ static void redisProtocolToLuaType_Double(void *ctx, double d, const char *proto
* with a single "err" field set to the error string. Note that this * with a single "err" field set to the error string. Note that this
* table is never a valid reply by proper commands, since the returned * table is never a valid reply by proper commands, since the returned
* tables are otherwise always indexed by integers, never by strings. */ * tables are otherwise always indexed by integers, never by strings. */
static void luaPushError(lua_State *lua, char *error) { void luaPushError(lua_State *lua, char *error) {
lua_Debug dbg; lua_Debug dbg;
/* If debugging is active and in step mode, log errors resulting from /* If debugging is active and in step mode, log errors resulting from
...@@ -455,7 +458,7 @@ static void luaPushError(lua_State *lua, char *error) { ...@@ -455,7 +458,7 @@ static void luaPushError(lua_State *lua, char *error) {
* by the non-error-trapping version of redis.pcall(), which is redis.call(), * by the non-error-trapping version of redis.pcall(), which is redis.call(),
* this function will raise the Lua error so that the execution of the * this function will raise the Lua error so that the execution of the
* script will be halted. */ * script will be halted. */
static int luaRaiseError(lua_State *lua) { int luaRaiseError(lua_State *lua) {
lua_pushstring(lua,"err"); lua_pushstring(lua,"err");
lua_gettable(lua,-2); lua_gettable(lua,-2);
return lua_error(lua); return lua_error(lua);
...@@ -656,6 +659,10 @@ static void luaReplyToRedisReply(client *c, client* script_client, lua_State *lu ...@@ -656,6 +659,10 @@ static void luaReplyToRedisReply(client *c, client* script_client, lua_State *lu
static int luaRedisGenericCommand(lua_State *lua, int raise_error) { static int luaRedisGenericCommand(lua_State *lua, int raise_error) {
int j, argc = lua_gettop(lua); int j, argc = lua_gettop(lua);
scriptRunCtx* rctx = luaGetFromRegistry(lua, REGISTRY_RUN_CTX_NAME); scriptRunCtx* rctx = luaGetFromRegistry(lua, REGISTRY_RUN_CTX_NAME);
if (!rctx) {
luaPushError(lua, "redis.call/pcall can only be called inside a script invocation");
return luaRaiseError(lua);
}
sds err = NULL; sds err = NULL;
client* c = rctx->c; client* c = rctx->c;
sds reply; sds reply;
...@@ -911,6 +918,10 @@ static int luaRedisSetReplCommand(lua_State *lua) { ...@@ -911,6 +918,10 @@ static int luaRedisSetReplCommand(lua_State *lua) {
int flags, argc = lua_gettop(lua); int flags, argc = lua_gettop(lua);
scriptRunCtx* rctx = luaGetFromRegistry(lua, REGISTRY_RUN_CTX_NAME); scriptRunCtx* rctx = luaGetFromRegistry(lua, REGISTRY_RUN_CTX_NAME);
if (!rctx) {
lua_pushstring(lua, "redis.set_repl can only be called inside a script invocation");
return lua_error(lua);
}
if (argc != 1) { if (argc != 1) {
lua_pushstring(lua, "redis.set_repl() requires two arguments."); lua_pushstring(lua, "redis.set_repl() requires two arguments.");
...@@ -966,6 +977,11 @@ static int luaLogCommand(lua_State *lua) { ...@@ -966,6 +977,11 @@ static int luaLogCommand(lua_State *lua) {
/* redis.setresp() */ /* redis.setresp() */
static int luaSetResp(lua_State *lua) { static int luaSetResp(lua_State *lua) {
scriptRunCtx* rctx = luaGetFromRegistry(lua, REGISTRY_RUN_CTX_NAME);
if (!rctx) {
lua_pushstring(lua, "redis.setresp can only be called inside a script invocation");
return lua_error(lua);
}
int argc = lua_gettop(lua); int argc = lua_gettop(lua);
if (argc != 1) { if (argc != 1) {
...@@ -978,7 +994,6 @@ static int luaSetResp(lua_State *lua) { ...@@ -978,7 +994,6 @@ static int luaSetResp(lua_State *lua) {
lua_pushstring(lua, "RESP version must be 2 or 3."); lua_pushstring(lua, "RESP version must be 2 or 3.");
return lua_error(lua); return lua_error(lua);
} }
scriptRunCtx* rctx = luaGetFromRegistry(lua, REGISTRY_RUN_CTX_NAME);
scriptSetResp(rctx, resp); scriptSetResp(rctx, resp);
return 0; return 0;
} }
...@@ -1031,8 +1046,8 @@ static void luaRemoveUnsupportedFunctions(lua_State *lua) { ...@@ -1031,8 +1046,8 @@ static void luaRemoveUnsupportedFunctions(lua_State *lua) {
* sequence, because it may interact with creation of globals. * sequence, because it may interact with creation of globals.
* *
* On Legacy Lua (eval) we need to check 'w ~= \"main\"' otherwise we will not be able * On Legacy Lua (eval) we need to check 'w ~= \"main\"' otherwise we will not be able
* to create the global 'function <sha> ()' variable. On Lua engine we do not use this trick * to create the global 'function <sha> ()' variable. On Functions Lua engine we do not use
* so its not needed. */ * this trick so it's not needed. */
void luaEnableGlobalsProtection(lua_State *lua, int is_eval) { void luaEnableGlobalsProtection(lua_State *lua, int is_eval) {
char *s[32]; char *s[32];
sds code = sdsempty(); sds code = sdsempty();
...@@ -1067,33 +1082,72 @@ void luaEnableGlobalsProtection(lua_State *lua, int is_eval) { ...@@ -1067,33 +1082,72 @@ void luaEnableGlobalsProtection(lua_State *lua, int is_eval) {
sdsfree(code); sdsfree(code);
} }
void luaRegisterRedisAPI(lua_State* lua) { /* Create a global protection function and put it to registry.
luaLoadLibraries(lua); * This need to be called once in the lua_State lifetime.
luaRemoveUnsupportedFunctions(lua); * After called it is possible to use luaSetGlobalProtection
* to set global protection on a give table.
/* Register the redis commands table and fields */ *
lua_newtable(lua); * The function assumes the Lua stack have a least enough
* space to push 2 element, its up to the caller to verify
/* redis.call */ * this before calling this function.
lua_pushstring(lua,"call"); *
lua_pushcfunction(lua,luaRedisCallCommand); * Notice, the difference between this and luaEnableGlobalsProtection
lua_settable(lua,-3); * is that luaEnableGlobalsProtection is enabling global protection
* on the current Lua globals. This registering a global protection
* function that later can be applied on any table. */
void luaRegisterGlobalProtectionFunction(lua_State *lua) {
lua_pushstring(lua, REGISTRY_SET_GLOBALS_PROTECTION_NAME);
char *global_protection_func = "local dbg = debug\n"
"local globals_protection = function (t)\n"
" local mt = {}\n"
" setmetatable(t, mt)\n"
" mt.__newindex = function (t, n, v)\n"
" if dbg.getinfo(2) then\n"
" local w = dbg.getinfo(2, \"S\").what\n"
" if w ~= \"C\" then\n"
" error(\"Script attempted to create global variable '\"..tostring(n)..\"'\", 2)\n"
" end"
" end"
" rawset(t, n, v)\n"
" end\n"
" mt.__index = function (t, n)\n"
" if dbg.getinfo(2) and dbg.getinfo(2, \"S\").what ~= \"C\" then\n"
" error(\"Script attempted to access nonexistent global variable '\"..tostring(n)..\"'\", 2)\n"
" end\n"
" return rawget(t, n)\n"
" end\n"
"end\n"
"return globals_protection";
int res = luaL_loadbuffer(lua, global_protection_func, strlen(global_protection_func), "@global_protection_def");
serverAssert(res == 0);
res = lua_pcall(lua,0,1,0);
serverAssert(res == 0);
lua_settable(lua, LUA_REGISTRYINDEX);
}
/* redis.pcall */ /* Set global protection on a given table.
lua_pushstring(lua,"pcall"); * The table need to be located on the top of the lua stack.
lua_pushcfunction(lua,luaRedisPCallCommand); * After called, it will no longer be possible to set
lua_settable(lua,-3); * new items on the table. The function is not removing
* the table from the top of the stack!
*
* The function assumes the Lua stack have a least enough
* space to push 2 element, its up to the caller to verify
* this before calling this function. */
void luaSetGlobalProtection(lua_State *lua) {
lua_pushstring(lua, REGISTRY_SET_GLOBALS_PROTECTION_NAME);
lua_gettable(lua, LUA_REGISTRYINDEX);
lua_pushvalue(lua, -2);
int res = lua_pcall(lua, 1, 0, 0);
serverAssert(res == 0);
}
void luaRegisterLogFunction(lua_State* lua) {
/* redis.log and log levels. */ /* redis.log and log levels. */
lua_pushstring(lua,"log"); lua_pushstring(lua,"log");
lua_pushcfunction(lua,luaLogCommand); lua_pushcfunction(lua,luaLogCommand);
lua_settable(lua,-3); lua_settable(lua,-3);
/* redis.setresp */
lua_pushstring(lua,"setresp");
lua_pushcfunction(lua,luaSetResp);
lua_settable(lua,-3);
lua_pushstring(lua,"LOG_DEBUG"); lua_pushstring(lua,"LOG_DEBUG");
lua_pushnumber(lua,LL_DEBUG); lua_pushnumber(lua,LL_DEBUG);
lua_settable(lua,-3); lua_settable(lua,-3);
...@@ -1109,6 +1163,31 @@ void luaRegisterRedisAPI(lua_State* lua) { ...@@ -1109,6 +1163,31 @@ void luaRegisterRedisAPI(lua_State* lua) {
lua_pushstring(lua,"LOG_WARNING"); lua_pushstring(lua,"LOG_WARNING");
lua_pushnumber(lua,LL_WARNING); lua_pushnumber(lua,LL_WARNING);
lua_settable(lua,-3); lua_settable(lua,-3);
}
void luaRegisterRedisAPI(lua_State* lua) {
luaLoadLibraries(lua);
luaRemoveUnsupportedFunctions(lua);
/* Register the redis commands table and fields */
lua_newtable(lua);
/* redis.call */
lua_pushstring(lua,"call");
lua_pushcfunction(lua,luaRedisCallCommand);
lua_settable(lua,-3);
/* redis.pcall */
lua_pushstring(lua,"pcall");
lua_pushcfunction(lua,luaRedisPCallCommand);
lua_settable(lua,-3);
luaRegisterLogFunction(lua);
/* redis.setresp */
lua_pushstring(lua,"setresp");
lua_pushcfunction(lua,luaSetResp);
lua_settable(lua,-3);
/* redis.sha1hex */ /* redis.sha1hex */
lua_pushstring(lua, "sha1hex"); lua_pushstring(lua, "sha1hex");
...@@ -1149,7 +1228,7 @@ void luaRegisterRedisAPI(lua_State* lua) { ...@@ -1149,7 +1228,7 @@ void luaRegisterRedisAPI(lua_State* lua) {
lua_settable(lua,-3); lua_settable(lua,-3);
/* Finally set the table as 'redis' global var. */ /* Finally set the table as 'redis' global var. */
lua_setglobal(lua,"redis"); lua_setglobal(lua,REDIS_API_NAME);
/* Replace math.random and math.randomseed with our implementations. */ /* Replace math.random and math.randomseed with our implementations. */
lua_getglobal(lua,"math"); lua_getglobal(lua,"math");
...@@ -1167,7 +1246,7 @@ void luaRegisterRedisAPI(lua_State* lua) { ...@@ -1167,7 +1246,7 @@ void luaRegisterRedisAPI(lua_State* lua) {
/* Set an array of Redis String Objects as a Lua array (table) stored into a /* Set an array of Redis String Objects as a Lua array (table) stored into a
* global variable. */ * global variable. */
static void luaSetGlobalArray(lua_State *lua, char *var, robj **elev, int elec) { static void luaCreateArray(lua_State *lua, robj **elev, int elec) {
int j; int j;
lua_newtable(lua); lua_newtable(lua);
...@@ -1175,7 +1254,6 @@ static void luaSetGlobalArray(lua_State *lua, char *var, robj **elev, int elec) ...@@ -1175,7 +1254,6 @@ static void luaSetGlobalArray(lua_State *lua, char *var, robj **elev, int elec)
lua_pushlstring(lua,(char*)elev[j]->ptr,sdslen(elev[j]->ptr)); lua_pushlstring(lua,(char*)elev[j]->ptr,sdslen(elev[j]->ptr));
lua_rawseti(lua,-2,j+1); lua_rawseti(lua,-2,j+1);
} }
lua_setglobal(lua,var);
} }
/* --------------------------------------------------------------------------- /* ---------------------------------------------------------------------------
...@@ -1189,6 +1267,11 @@ static void luaSetGlobalArray(lua_State *lua, char *var, robj **elev, int elec) ...@@ -1189,6 +1267,11 @@ static void luaSetGlobalArray(lua_State *lua, char *var, robj **elev, int elec)
/* The following implementation is the one shipped with Lua itself but with /* The following implementation is the one shipped with Lua itself but with
* rand() replaced by redisLrand48(). */ * rand() replaced by redisLrand48(). */
static int redis_math_random (lua_State *L) { static int redis_math_random (lua_State *L) {
scriptRunCtx* rctx = luaGetFromRegistry(L, REGISTRY_RUN_CTX_NAME);
if (!rctx) {
return luaL_error(L, "math.random can only be called inside a script invocation");
}
/* the `%' avoids the (rare) case of r==1, and is needed also because on /* the `%' avoids the (rare) case of r==1, and is needed also because on
some systems (SunOS!) `rand()' may return a value larger than RAND_MAX */ some systems (SunOS!) `rand()' may return a value larger than RAND_MAX */
lua_Number r = (lua_Number)(redisLrand48()%REDIS_LRAND48_MAX) / lua_Number r = (lua_Number)(redisLrand48()%REDIS_LRAND48_MAX) /
...@@ -1217,6 +1300,10 @@ static int redis_math_random (lua_State *L) { ...@@ -1217,6 +1300,10 @@ static int redis_math_random (lua_State *L) {
} }
static int redis_math_randomseed (lua_State *L) { static int redis_math_randomseed (lua_State *L) {
scriptRunCtx* rctx = luaGetFromRegistry(L, REGISTRY_RUN_CTX_NAME);
if (!rctx) {
return luaL_error(L, "math.randomseed can only be called inside a script invocation");
}
redisSrand48(luaL_checkint(L, 1)); redisSrand48(luaL_checkint(L, 1));
return 0; return 0;
} }
...@@ -1260,13 +1347,24 @@ void luaCallFunction(scriptRunCtx* run_ctx, lua_State *lua, robj** keys, size_t ...@@ -1260,13 +1347,24 @@ void luaCallFunction(scriptRunCtx* run_ctx, lua_State *lua, robj** keys, size_t
/* Populate the argv and keys table accordingly to the arguments that /* Populate the argv and keys table accordingly to the arguments that
* EVAL received. */ * EVAL received. */
luaSetGlobalArray(lua,"KEYS",keys,nkeys); luaCreateArray(lua,keys,nkeys);
luaSetGlobalArray(lua,"ARGV",args,nargs); /* On eval, keys and arguments are globals. */
if (run_ctx->flags & SCRIPT_EVAL_MODE) lua_setglobal(lua,"KEYS");
luaCreateArray(lua,args,nargs);
if (run_ctx->flags & SCRIPT_EVAL_MODE) lua_setglobal(lua,"ARGV");
/* At this point whether this script was never seen before or if it was /* At this point whether this script was never seen before or if it was
* already defined, we can call it. We have zero arguments and expect * already defined, we can call it.
* a single return value. */ * On eval mode, we have zero arguments and expect a single return value.
int err = lua_pcall(lua,0,1,-2); * In addition the error handler is located on position -2 on the Lua stack.
* On function mode, we pass 2 arguments (the keys and args tables),
* and the error handler is located on position -4 (stack: error_handler, callback, keys, args) */
int err;
if (run_ctx->flags & SCRIPT_EVAL_MODE) {
err = lua_pcall(lua,0,1,-2);
} else {
err = lua_pcall(lua,2,1,-4);
}
/* Call the Lua garbage collector from time to time to avoid a /* Call the Lua garbage collector from time to time to avoid a
* full cycle performed by Lua, which adds too latency. * full cycle performed by Lua, which adds too latency.
......
...@@ -55,9 +55,16 @@ ...@@ -55,9 +55,16 @@
#include <lualib.h> #include <lualib.h>
#define REGISTRY_RUN_CTX_NAME "__RUN_CTX__" #define REGISTRY_RUN_CTX_NAME "__RUN_CTX__"
#define REGISTRY_SET_GLOBALS_PROTECTION_NAME "__GLOBAL_PROTECTION__"
#define REDIS_API_NAME "redis"
void luaRegisterRedisAPI(lua_State* lua); void luaRegisterRedisAPI(lua_State* lua);
void luaEnableGlobalsProtection(lua_State *lua, int is_eval); void luaEnableGlobalsProtection(lua_State *lua, int is_eval);
void luaRegisterGlobalProtectionFunction(lua_State *lua);
void luaSetGlobalProtection(lua_State *lua);
void luaRegisterLogFunction(lua_State* lua);
void luaPushError(lua_State *lua, char *error);
int luaRaiseError(lua_State *lua);
void luaSaveOnRegistry(lua_State* lua, const char* name, void* ptr); void luaSaveOnRegistry(lua_State* lua, const char* name, void* ptr);
void* luaGetFromRegistry(lua_State* lua, const char* name); void* luaGetFromRegistry(lua_State* lua, const char* name);
void luaCallFunction(scriptRunCtx* r_ctx, lua_State *lua, robj** keys, size_t nkeys, robj** args, size_t nargs, int debug_enabled); void luaCallFunction(scriptRunCtx* r_ctx, lua_State *lua, robj** keys, size_t nkeys, robj** args, size_t nargs, int debug_enabled);
......
...@@ -4886,6 +4886,7 @@ sds genRedisInfoString(const char *section) { ...@@ -4886,6 +4886,7 @@ sds genRedisInfoString(const char *section) {
"used_memory_scripts_eval:%lld\r\n" "used_memory_scripts_eval:%lld\r\n"
"number_of_cached_scripts:%lu\r\n" "number_of_cached_scripts:%lu\r\n"
"number_of_functions:%lu\r\n" "number_of_functions:%lu\r\n"
"number_of_libraries:%lu\r\n"
"used_memory_vm_functions:%lld\r\n" "used_memory_vm_functions:%lld\r\n"
"used_memory_vm_total:%lld\r\n" "used_memory_vm_total:%lld\r\n"
"used_memory_vm_total_human:%s\r\n" "used_memory_vm_total_human:%s\r\n"
...@@ -4936,6 +4937,7 @@ sds genRedisInfoString(const char *section) { ...@@ -4936,6 +4937,7 @@ sds genRedisInfoString(const char *section) {
(long long) mh->lua_caches, (long long) mh->lua_caches,
dictSize(evalScriptsDict()), dictSize(evalScriptsDict()),
functionsNum(), functionsNum(),
functionsLibNum(),
memory_functions, memory_functions,
memory_functions + memory_lua, memory_functions + memory_lua,
used_memory_vm_total_hmem, used_memory_vm_total_hmem,
......
...@@ -873,7 +873,7 @@ typedef struct redisDb { ...@@ -873,7 +873,7 @@ typedef struct redisDb {
} redisDb; } redisDb;
/* forward declaration for functions ctx */ /* forward declaration for functions ctx */
typedef struct functionsCtx functionsCtx; typedef struct functionsLibCtx functionsLibCtx;
/* Holding object that need to be populated during /* Holding object that need to be populated during
* rdb loading. On loading end it is possible to decide * rdb loading. On loading end it is possible to decide
...@@ -882,7 +882,7 @@ typedef struct functionsCtx functionsCtx; ...@@ -882,7 +882,7 @@ typedef struct functionsCtx functionsCtx;
* successful loading and dropped on failure. */ * successful loading and dropped on failure. */
typedef struct rdbLoadingCtx { typedef struct rdbLoadingCtx {
redisDb* dbarray; redisDb* dbarray;
functionsCtx* functions_ctx; functionsLibCtx* functions_lib_ctx;
}rdbLoadingCtx; }rdbLoadingCtx;
/* Client MULTI/EXEC state */ /* Client MULTI/EXEC state */
...@@ -3017,7 +3017,7 @@ int ldbPendingChildren(void); ...@@ -3017,7 +3017,7 @@ int ldbPendingChildren(void);
sds luaCreateFunction(client *c, robj *body); sds luaCreateFunction(client *c, robj *body);
void luaLdbLineHook(lua_State *lua, lua_Debug *ar); void luaLdbLineHook(lua_State *lua, lua_Debug *ar);
void freeLuaScriptsAsync(dict *lua_scripts); void freeLuaScriptsAsync(dict *lua_scripts);
void freeFunctionsAsync(functionsCtx *f_ctx); void freeFunctionsAsync(functionsLibCtx *lib_ctx);
int ldbIsEnabled(); int ldbIsEnabled();
void ldbLog(sds entry); void ldbLog(sds entry);
void ldbLogRedisReply(char *reply); void ldbLogRedisReply(char *reply);
...@@ -3279,11 +3279,10 @@ void evalShaRoCommand(client *c); ...@@ -3279,11 +3279,10 @@ void evalShaRoCommand(client *c);
void scriptCommand(client *c); void scriptCommand(client *c);
void fcallCommand(client *c); void fcallCommand(client *c);
void fcallroCommand(client *c); void fcallroCommand(client *c);
void functionCreateCommand(client *c); void functionLoadCommand(client *c);
void functionDeleteCommand(client *c); void functionDeleteCommand(client *c);
void functionKillCommand(client *c); void functionKillCommand(client *c);
void functionStatsCommand(client *c); void functionStatsCommand(client *c);
void functionInfoCommand(client *c);
void functionListCommand(client *c); void functionListCommand(client *c);
void functionHelpCommand(client *c); void functionHelpCommand(client *c);
void functionFlushCommand(client *c); void functionFlushCommand(client *c);
......
...@@ -322,7 +322,7 @@ if {!$::tls} { ;# fake_redis_node doesn't support TLS ...@@ -322,7 +322,7 @@ if {!$::tls} { ;# fake_redis_node doesn't support TLS
set dir [lindex [r config get dir] 1] set dir [lindex [r config get dir] 1]
assert_equal "OK" [r debug populate 100000 key 1000] assert_equal "OK" [r debug populate 100000 key 1000]
assert_equal "OK" [r function create lua func1 "return 123"] assert_equal "OK" [r function load lua lib1 "redis.register_function('func1', function() return 123 end)"]
if {$functions_only} { if {$functions_only} {
set args "--functions-rdb $dir/cli.rdb" set args "--functions-rdb $dir/cli.rdb"
} else { } else {
...@@ -335,11 +335,10 @@ if {!$::tls} { ;# fake_redis_node doesn't support TLS ...@@ -335,11 +335,10 @@ if {!$::tls} { ;# fake_redis_node doesn't support TLS
file rename "$dir/cli.rdb" "$dir/dump.rdb" file rename "$dir/cli.rdb" "$dir/dump.rdb"
assert_equal "OK" [r set should-not-exist 1] assert_equal "OK" [r set should-not-exist 1]
assert_equal "OK" [r function create lua should_not_exist_func "return 456"] assert_equal "OK" [r function load lua should_not_exist_func "redis.register_function('should_not_exist_func', function() return 456 end)"]
assert_equal "OK" [r debug reload nosave] assert_equal "OK" [r debug reload nosave]
assert_equal {} [r get should-not-exist] assert_equal {} [r get should-not-exist]
assert_error "ERR Function does not exists" {r function info should_not_exist_func} assert_equal {{library_name lib1 engine LUA description {} functions {{name func1 description {}}}}} [r function list]
assert_equal "func1" [dict get [r function info func1] name]
if {$functions_only} { if {$functions_only} {
assert_equal 0 [r dbsize] assert_equal 0 [r dbsize]
} else { } else {
......
...@@ -522,10 +522,10 @@ foreach testType {Successful Aborted} { ...@@ -522,10 +522,10 @@ foreach testType {Successful Aborted} {
$replica set mykey myvalue $replica set mykey myvalue
# Set a function value on replica to check status during loading, on failure and after swapping db # Set a function value on replica to check status during loading, on failure and after swapping db
$replica function create LUA test {return 'hello1'} $replica function load LUA test {redis.register_function('test', function() return 'hello1' end)}
# Set a function value on master to check it reaches the replica when replication ends # Set a function value on master to check it reaches the replica when replication ends
$master function create LUA test {return 'hello2'} $master function load LUA test {redis.register_function('test', function() return 'hello2' end)}
# Force the replica to try another full sync (this time it will have matching master replid) # Force the replica to try another full sync (this time it will have matching master replid)
$master multi $master multi
...@@ -658,7 +658,7 @@ test {diskless loading short read} { ...@@ -658,7 +658,7 @@ test {diskless loading short read} {
set start [clock clicks -milliseconds] set start [clock clicks -milliseconds]
# Set a function value to check short read handling on functions # Set a function value to check short read handling on functions
r function create LUA test {return 'hello1'} r function load LUA test {redis.register_function('test', function() return 'hello1' end)}
for {set k 0} {$k < 3} {incr k} { for {set k 0} {$k < 3} {incr k} {
for {set i 0} {$i < 10} {incr i} { for {set i 0} {$i < 10} {incr i} {
......
...@@ -182,7 +182,7 @@ start_server [list overrides $base_conf] { ...@@ -182,7 +182,7 @@ start_server [list overrides $base_conf] {
# upload a function to all the cluster # upload a function to all the cluster
exec src/redis-cli --cluster-yes --cluster call 127.0.0.1:[srv 0 port] \ exec src/redis-cli --cluster-yes --cluster call 127.0.0.1:[srv 0 port] \
FUNCTION CREATE LUA TEST {return 'hello'} FUNCTION LOAD LUA TEST {redis.register_function('test', function() return 'hello' end)}
# adding node to the cluster # adding node to the cluster
exec src/redis-cli --cluster-yes --cluster add-node \ exec src/redis-cli --cluster-yes --cluster add-node \
...@@ -199,13 +199,13 @@ start_server [list overrides $base_conf] { ...@@ -199,13 +199,13 @@ start_server [list overrides $base_conf] {
} }
# make sure 'test' function was added to the new node # make sure 'test' function was added to the new node
assert_equal {{name TEST engine LUA description {}}} [$node4_rd FUNCTION LIST] assert_equal {{library_name TEST engine LUA description {} functions {{name test description {}}}}} [$node4_rd FUNCTION LIST]
# add function to node 5 # add function to node 5
assert_equal {OK} [$node5_rd FUNCTION CREATE LUA TEST {return 'hello1'}] assert_equal {OK} [$node5_rd FUNCTION LOAD LUA TEST {redis.register_function('test', function() return 'hello' end)}]
# make sure functions was added to node 5 # make sure functions was added to node 5
assert_equal {{name TEST engine LUA description {}}} [$node5_rd FUNCTION LIST] assert_equal {{library_name TEST engine LUA description {} functions {{name test description {}}}}} [$node5_rd FUNCTION LIST]
# adding node 5 to the cluster should failed because it already contains the 'test' function # adding node 5 to the cluster should failed because it already contains the 'test' function
catch { catch {
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment