Unverified Commit 885f6b5c authored by Meir Shpilraien (Spielrein)'s avatar Meir Shpilraien (Spielrein) Committed by GitHub
Browse files

Redis Function Libraries (#10004)

# Redis Function Libraries

This PR implements Redis Functions Libraries as describe on: https://github.com/redis/redis/issues/9906.

Libraries purpose is to provide a better code sharing between functions by allowing to create multiple
functions in a single command. Functions that were created together can safely share code between
each other without worrying about compatibility issues and versioning.

Creating a new library is done using 'FUNCTION LOAD' command (full API is described below)

This PR introduces a new struct called libraryInfo, libraryInfo holds information about a library:
* name - name of the library
* engine - engine used to create the library
* code - library code
* description - library description
* functions - the functions exposed by the library

When Redis gets the `FUNCTION LOAD` command it creates a new empty libraryInfo.
Redis passes the `CODE` to the relevant engine alongside the empty libraryInfo.
As a result, the engine will create one or more functions by calling 'libraryCreateFunction'.
The new funcion will be added to the newly created libraryInfo. So far Everything is happening
locally on the libraryInfo so it is easy to abort the operation (in case of an error) by simply
freeing the libraryInfo. After the library info is fully constructed we start the joining phase by
which we will join the new library to the other libraries currently exist on Redis.
The joining phase make sure there is no function collision and add the library to the
librariesCtx (renamed from functionCtx). LibrariesCtx is used all around the code in the exact
same way as functionCtx was used (with respect to RDB loading, replicatio, ...).
The only difference is that apart from function dictionary (maps function name to functionInfo
object), the librariesCtx contains also a libraries dictionary that maps library name to libraryInfo object.

## New API
### FUNCTION LOAD
`FUNCTION LOAD <ENGINE> <LIBRARY NAME> [REPLACE] [DESCRIPTION <DESCRIPTION>] <CODE>`
Create a new library with the given parameters:
* ENGINE - REPLACE Engine name to use to create the library.
* LIBRARY NAME - The new library name.
* REPLACE - If the library already exists, replace it.
* DESCRIPTION - Library description.
* CODE - Library code.

Return "OK" on success, or error on the following cases:
* Library name already taken and REPLACE was not used
* Name collision with another existing library (even if replace was uses)
* Library registration failed by the engine (usually compilation error)

## Changed API
### FUNCTION LIST
`FUNCTION LIST [LIBRARYNAME <LIBRARY NAME PATTERN>] [WITHCODE]`
Command was modified to also allow getting libraries code (so `FUNCTION INFO` command is no longer
needed and removed). In addition the command gets an option argument, `LIBRARYNAME` allows you to
only get libraries that match the given `LIBRARYNAME` pattern. By default, it returns all libraries.

### INFO MEMORY
Added number of libraries to `INFO MEMORY`

### Commands flags
`DENYOOM` flag was set on `FUNCTION LOAD` and `FUNCTION RESTORE`. We consider those commands
as commands that add new data to the dateset (functions are data) and so we want to disallows
to run those commands on OOM.

## Removed API
* FUNCTION CREATE - Decided on https://github.com/redis/redis/issues/9906
* FUNCTION INFO - Decided on https://github.com/redis/redis/issues/9899

## Lua engine changes
When the Lua engine gets the code given on `FUNCTION LOAD` command, it immediately runs it, we call
this run the loading run. Loading run is not a usual script run, it is not possible to invoke any
Redis command from within the load run.
Instead there is a new API provided by `library` object. The new API's: 
* `redis.log` - behave the same as `redis.log`
* `redis.register_function` - register a new function to the library

The loading run purpose is to register functions using the new `redis.register_function` API.
Any attempt to use any other API will result in an error. In addition, the load run is has a time
limit of 500ms, error is raise on timeout and the entire operation is aborted.

### `redis.register_function`
`redis.register_function(<function_name>, <callback>, [<description>])`
This new API allows users to register a new function that will be linked to the newly created library.
This API can only be called during the load run (see definition above). Any attempt to use it outside
of the load run will result in an error.
The parameters pass to the API are:
* function_name - Function name (must be a Lua string)
* callback - Lua function object that will be called when the function is invokes using fcall/fcall_ro
* description - Function description, optional (must be a Lua string).

### Example
The following example creates a library called `lib` with 2 functions, `f1` and `f1`, returns 1 and 2 respectively:
```
local function f1(keys, args)
    return 1
end

local function f2(keys, args)
    return 2
end

redis.register_function('f1', f1)
redis.register_function('f2', f2)
```

Notice: Unlike `eval`, functions inside a library get the KEYS and ARGV as arguments to the
functions and not as global.

### Technical Details

On the load run we only want the user to be able to call a white list on API's. This way, in
the future, if new API's will be added, the new API's will not be available to the load run
unless specifically added to this white list. We put the while list on the `library` object and
make sure the `library` object is only available to the load run by using [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv) API. This API allows us to set
the `globals` of a function (and all the function it creates). Before starting the load run we
create a new fresh Lua table (call it `g`) that only contains the `library` API (we make sure
to set global protection on this table just like the general global protection already exists
today), then we use [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv)
to set `g` as the global table of the load run. After the load run finished we update `g`
metatable and set `__index` and `__newindex` functions to be `_G` (Lua default globals),
we also pop out the `library` object as we do not need it anymore.
This way, any function that was created on the load run (and will be invoke using `fcall`) will
see the default globals as it expected to see them and will not have the `library` API anymore.

An important outcome of this new approach is that now we can achieve a distinct global table
for each library (it is not yet like that but it is very easy to achieve it now). In the future we can
decide to remove global protection because global on different libraries will not collide or we
can chose to give different API to different libraries base on some configuration or input.

Notice that this technique was meant to prevent errors and was not meant to prevent malicious
user from exploit it. For example, the load run can still save the `library` object on some local
variable and then using in `fcall` context. To prevent such a malicious use, the C code also make
sure it is running in the right context and if not raise an error.
parent 568c2e03
......@@ -3144,24 +3144,6 @@ struct redisCommandArg FCALL_RO_Args[] = {
{0}
};
/********** FUNCTION CREATE ********************/
/* FUNCTION CREATE history */
#define FUNCTION_CREATE_History NULL
/* FUNCTION CREATE hints */
#define FUNCTION_CREATE_Hints NULL
/* FUNCTION CREATE argument table */
struct redisCommandArg FUNCTION_CREATE_Args[] = {
{"engine-name",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{"function-name",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{"replace",ARG_TYPE_PURE_TOKEN,-1,"REPLACE",NULL,NULL,CMD_ARG_OPTIONAL},
{"function-description",ARG_TYPE_STRING,-1,"DESC",NULL,NULL,CMD_ARG_OPTIONAL},
{"function-code",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{0}
};
/********** FUNCTION DELETE ********************/
/* FUNCTION DELETE history */
......@@ -3213,21 +3195,6 @@ struct redisCommandArg FUNCTION_FLUSH_Args[] = {
/* FUNCTION HELP hints */
#define FUNCTION_HELP_Hints NULL
/********** FUNCTION INFO ********************/
/* FUNCTION INFO history */
#define FUNCTION_INFO_History NULL
/* FUNCTION INFO hints */
#define FUNCTION_INFO_Hints NULL
/* FUNCTION INFO argument table */
struct redisCommandArg FUNCTION_INFO_Args[] = {
{"function-name",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{"withcode",ARG_TYPE_PURE_TOKEN,-1,"WITHCODE",NULL,NULL,CMD_ARG_OPTIONAL},
{0}
};
/********** FUNCTION KILL ********************/
/* FUNCTION KILL history */
......@@ -3244,6 +3211,31 @@ struct redisCommandArg FUNCTION_INFO_Args[] = {
/* FUNCTION LIST hints */
#define FUNCTION_LIST_Hints NULL
/* FUNCTION LIST argument table */
struct redisCommandArg FUNCTION_LIST_Args[] = {
{"library-name-pattern",ARG_TYPE_STRING,-1,"LIBRARYNAME",NULL,NULL,CMD_ARG_OPTIONAL},
{"withcode",ARG_TYPE_PURE_TOKEN,-1,"WITHCODE",NULL,NULL,CMD_ARG_OPTIONAL},
{0}
};
/********** FUNCTION LOAD ********************/
/* FUNCTION LOAD history */
#define FUNCTION_LOAD_History NULL
/* FUNCTION LOAD hints */
#define FUNCTION_LOAD_Hints NULL
/* FUNCTION LOAD argument table */
struct redisCommandArg FUNCTION_LOAD_Args[] = {
{"engine-name",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{"library-name",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{"replace",ARG_TYPE_PURE_TOKEN,-1,"REPLACE",NULL,NULL,CMD_ARG_OPTIONAL},
{"library-description",ARG_TYPE_STRING,-1,"DESC",NULL,NULL,CMD_ARG_OPTIONAL},
{"function-code",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE},
{0}
};
/********** FUNCTION RESTORE ********************/
/* FUNCTION RESTORE history */
......@@ -3277,15 +3269,14 @@ struct redisCommandArg FUNCTION_RESTORE_Args[] = {
/* FUNCTION command table */
struct redisCommand FUNCTION_Subcommands[] = {
{"create","Create a function with the given arguments (name, code, description)","O(1) (considering compilation time is redundant)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_CREATE_History,FUNCTION_CREATE_Hints,functionCreateCommand,-5,CMD_NOSCRIPT|CMD_WRITE,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_CREATE_Args},
{"delete","Delete a function by name","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_DELETE_History,FUNCTION_DELETE_Hints,functionDeleteCommand,3,CMD_NOSCRIPT|CMD_WRITE,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_DELETE_Args},
{"dump","Dump all functions into a serialized binary payload","O(N) where N is the number of functions","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_DUMP_History,FUNCTION_DUMP_Hints,functionDumpCommand,2,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING},
{"flush","Deleting all functions","O(N) where N is the number of functions deleted","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_FLUSH_History,FUNCTION_FLUSH_Hints,functionFlushCommand,-2,CMD_NOSCRIPT|CMD_WRITE,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_FLUSH_Args},
{"help","Show helpful text about the different subcommands","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_HELP_History,FUNCTION_HELP_Hints,functionHelpCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SCRIPTING},
{"info","Return information about a function by function name","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_INFO_History,FUNCTION_INFO_Hints,functionInfoCommand,-3,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_INFO_Args},
{"kill","Kill the function currently in execution.","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_KILL_History,FUNCTION_KILL_Hints,functionKillCommand,2,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING},
{"list","List information about all the functions","O(N) where N is the number of functions","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_LIST_History,FUNCTION_LIST_Hints,functionListCommand,2,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING},
{"restore","Restore all the functions on the given payload","O(N) where N is the number of functions on the payload","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_RESTORE_History,FUNCTION_RESTORE_Hints,functionRestoreCommand,-3,CMD_NOSCRIPT|CMD_WRITE,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_RESTORE_Args},
{"list","List information about all the functions","O(N) where N is the number of functions","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_LIST_History,FUNCTION_LIST_Hints,functionListCommand,-2,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_LIST_Args},
{"load","Create a function with the given arguments (name, code, description)","O(1) (considering compilation time is redundant)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_LOAD_History,FUNCTION_LOAD_Hints,functionLoadCommand,-5,CMD_NOSCRIPT|CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_LOAD_Args},
{"restore","Restore all the functions on the given payload","O(N) where N is the number of functions on the payload","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_RESTORE_History,FUNCTION_RESTORE_Hints,functionRestoreCommand,-3,CMD_NOSCRIPT|CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_SCRIPTING,.args=FUNCTION_RESTORE_Args},
{"stats","Return information about the function currently running (name, description, duration)","O(1)","7.0.0",CMD_DOC_NONE,NULL,NULL,COMMAND_GROUP_SCRIPTING,FUNCTION_STATS_History,FUNCTION_STATS_Hints,functionStatsCommand,2,CMD_NOSCRIPT,ACL_CATEGORY_SCRIPTING},
{0}
};
......
{
"INFO": {
"summary": "Return information about a function by function name",
"complexity": "O(1)",
"group": "scripting",
"since": "7.0.0",
"arity": -3,
"container": "FUNCTION",
"function": "functionInfoCommand",
"command_flags": [
"NOSCRIPT"
],
"acl_categories": [
"SCRIPTING"
],
"arguments": [
{
"name": "function-name",
"type": "string"
},
{
"name": "withcode",
"type": "pure-token",
"token": "WITHCODE",
"optional": true
}
]
}
}
......@@ -4,7 +4,7 @@
"complexity": "O(N) where N is the number of functions",
"group": "scripting",
"since": "7.0.0",
"arity": 2,
"arity": -2,
"container": "FUNCTION",
"function": "functionListCommand",
"command_flags": [
......@@ -12,6 +12,20 @@
],
"acl_categories": [
"SCRIPTING"
],
"arguments": [
{
"name": "library-name-pattern",
"type": "string",
"token": "LIBRARYNAME",
"optional": true
},
{
"name": "withcode",
"type": "pure-token",
"token": "WITHCODE",
"optional": true
}
]
}
}
{
"CREATE": {
"LOAD": {
"summary": "Create a function with the given arguments (name, code, description)",
"complexity": "O(1) (considering compilation time is redundant)",
"group": "scripting",
"since": "7.0.0",
"arity": -5,
"container": "FUNCTION",
"function": "functionCreateCommand",
"function": "functionLoadCommand",
"command_flags": [
"NOSCRIPT",
"WRITE"
"WRITE",
"DENYOOM"
],
"acl_categories": [
"SCRIPTING"
......@@ -20,7 +21,7 @@
"type": "string"
},
{
"name": "function-name",
"name": "library-name",
"type": "string"
},
{
......@@ -30,7 +31,7 @@
"optional": true
},
{
"name": "function-description",
"name": "library-description",
"type": "string",
"token": "DESC",
"optional": true
......
......@@ -9,7 +9,8 @@
"function": "functionRestoreCommand",
"command_flags": [
"NOSCRIPT",
"WRITE"
"WRITE",
"DENYOOM"
],
"acl_categories": [
"SCRIPTING"
......
......@@ -461,7 +461,7 @@ long long emptyData(int dbnum, int flags, void(callback)(dict*)) {
if (with_functions) {
serverAssert(dbnum == -1);
functionsCtxClearCurrent(async);
functionsLibCtxClearCurrent(async);
}
/* Also fire the end event. Note that this event will fire almost
......
......@@ -48,6 +48,9 @@
#define LUA_ENGINE_NAME "LUA"
#define REGISTRY_ENGINE_CTX_NAME "__ENGINE_CTX__"
#define REGISTRY_ERROR_HANDLER_NAME "__ERROR_HANDLER__"
#define REGISTRY_LOAD_CTX_NAME "__LIBRARY_CTX__"
#define LIBRARY_API_NAME "__LIBRARY_API__"
#define LOAD_TIMEOUT_MS 500
/* Lua engine ctx */
typedef struct luaEngineCtx {
......@@ -60,6 +63,27 @@ typedef struct luaFunctionCtx {
int lua_function_ref;
} luaFunctionCtx;
typedef struct loadCtx {
functionLibInfo *li;
monotime start_time;
} loadCtx;
/* Hook for FUNCTION LOAD execution.
* Used to cancel the execution in case of a timeout (500ms).
* This execution should be fast and should only register
* functions so 500ms should be more than enough. */
static void luaEngineLoadHook(lua_State *lua, lua_Debug *ar) {
UNUSED(ar);
loadCtx *load_ctx = luaGetFromRegistry(lua, REGISTRY_LOAD_CTX_NAME);
uint64_t duration = elapsedMs(load_ctx->start_time);
if (duration > LOAD_TIMEOUT_MS) {
lua_sethook(lua, luaEngineLoadHook, LUA_MASKLINE, 0);
lua_pushstring(lua,"FUNCTION LOAD timeout");
lua_error(lua);
}
}
/*
* Compile a given blob and save it on the registry.
* Return a function ctx with Lua ref that allows to later retrieve the
......@@ -67,25 +91,88 @@ typedef struct luaFunctionCtx {
*
* Return NULL on compilation error and set the error to the err variable
*/
static void* luaEngineCreate(void *engine_ctx, sds blob, sds *err) {
static int luaEngineCreate(void *engine_ctx, functionLibInfo *li, sds blob, sds *err) {
luaEngineCtx *lua_engine_ctx = engine_ctx;
lua_State *lua = lua_engine_ctx->lua;
/* Each library will have its own global distinct table.
* We will create a new fresh Lua table and use
* lua_setfenv to set the table as the library globals
* (https://www.lua.org/manual/5.1/manual.html#lua_setfenv)
*
* At first, populate this new table with only the 'library' API
* to make sure only 'library' API is available at start. After the
* initial run is finished and all functions are registered, add
* all the default globals to the library global table and delete
* the library API.
*
* There are 2 ways to achieve the last part (add default
* globals to the new table):
*
* 1. Initialize the new table with all the default globals
* 2. Inheritance using metatable (https://www.lua.org/pil/14.3.html)
*
* For now we are choosing the second, we can change it in the future to
* achieve a better isolation between functions. */
lua_newtable(lua); /* Global table for the library */
lua_pushstring(lua, REDIS_API_NAME);
lua_pushstring(lua, LIBRARY_API_NAME);
lua_gettable(lua, LUA_REGISTRYINDEX); /* get library function from registry */
lua_settable(lua, -3); /* push the library table to the new global table */
/* Set global protection on the new global table */
luaSetGlobalProtection(lua_engine_ctx->lua);
/* compile the code */
if (luaL_loadbuffer(lua, blob, sdslen(blob), "@user_function")) {
*err = sdsempty();
*err = sdscatprintf(*err, "Error compiling function: %s",
lua_tostring(lua, -1));
lua_pop(lua, 1);
return NULL;
*err = sdscatprintf(sdsempty(), "Error compiling function: %s", lua_tostring(lua, -1));
lua_pop(lua, 2); /* pops the error and globals table */
return C_ERR;
}
serverAssert(lua_isfunction(lua, -1));
int lua_function_ref = luaL_ref(lua, LUA_REGISTRYINDEX);
loadCtx load_ctx = {
.li = li,
.start_time = getMonotonicUs(),
};
luaSaveOnRegistry(lua, REGISTRY_LOAD_CTX_NAME, &load_ctx);
luaFunctionCtx *f_ctx = zmalloc(sizeof(*f_ctx));
*f_ctx = (luaFunctionCtx ) { .lua_function_ref = lua_function_ref, };
/* set the function environment so only 'library' API can be accessed. */
lua_pushvalue(lua, -2); /* push global table to the front */
lua_setfenv(lua, -2);
return f_ctx;
lua_sethook(lua,luaEngineLoadHook,LUA_MASKCOUNT,100000);
/* Run the compiled code to allow it to register functions */
if (lua_pcall(lua,0,0,0)) {
*err = sdscatprintf(sdsempty(), "Error registering functions: %s", lua_tostring(lua, -1));
lua_pop(lua, 2); /* pops the error and globals table */
lua_sethook(lua,NULL,0,0); /* Disable hook */
luaSaveOnRegistry(lua, REGISTRY_LOAD_CTX_NAME, NULL);
return C_ERR;
}
lua_sethook(lua,NULL,0,0); /* Disable hook */
luaSaveOnRegistry(lua, REGISTRY_LOAD_CTX_NAME, NULL);
/* stack contains the global table, lets rearrange it to contains the entire API. */
/* delete 'redis' API */
lua_pushstring(lua, REDIS_API_NAME);
lua_pushnil(lua);
lua_settable(lua, -3);
/* create metatable */
lua_newtable(lua);
lua_pushstring(lua, "__index");
lua_pushvalue(lua, LUA_GLOBALSINDEX); /* push original globals */
lua_settable(lua, -3);
lua_pushstring(lua, "__newindex");
lua_pushvalue(lua, LUA_GLOBALSINDEX); /* push original globals */
lua_settable(lua, -3);
lua_setmetatable(lua, -2);
lua_pop(lua, 1); /* pops the global table */
return C_OK;
}
/*
......@@ -137,6 +224,64 @@ static void luaEngineFreeFunction(void *engine_ctx, void *compiled_function) {
zfree(f_ctx);
}
static int luaRegisterFunction(lua_State *lua) {
int argc = lua_gettop(lua);
if (argc < 2 || argc > 3) {
luaPushError(lua, "wrong number of arguments to redis.register_function");
return luaRaiseError(lua);
}
loadCtx *load_ctx = luaGetFromRegistry(lua, REGISTRY_LOAD_CTX_NAME);
if (!load_ctx) {
luaPushError(lua, "redis.register_function can only be called on FUNCTION LOAD command");
return luaRaiseError(lua);
}
if (!lua_isstring(lua, 1)) {
luaPushError(lua, "first argument to redis.register_function must be a string");
return luaRaiseError(lua);
}
if (!lua_isfunction(lua, 2)) {
luaPushError(lua, "second argument to redis.register_function must be a function");
return luaRaiseError(lua);
}
if (argc == 3 && !lua_isstring(lua, 3)) {
luaPushError(lua, "third argument to redis.register_function must be a string");
return luaRaiseError(lua);
}
size_t function_name_len;
const char *function_name = lua_tolstring(lua, 1, &function_name_len);
sds function_name_sds = sdsnewlen(function_name, function_name_len);
sds desc_sds = NULL;
if (argc == 3){
size_t desc_len;
const char *desc = lua_tolstring(lua, 3, &desc_len);
desc_sds = sdsnewlen(desc, desc_len);
lua_pop(lua, 1); /* pop out the description */
}
int lua_function_ref = luaL_ref(lua, LUA_REGISTRYINDEX);
luaFunctionCtx *lua_f_ctx = zmalloc(sizeof(*lua_f_ctx));
*lua_f_ctx = (luaFunctionCtx ) { .lua_function_ref = lua_function_ref, };
sds err = NULL;
if (functionLibCreateFunction(function_name_sds, lua_f_ctx, load_ctx->li, desc_sds, &err) != C_OK) {
sdsfree(function_name_sds);
if (desc_sds) sdsfree(desc_sds);
lua_unref(lua, lua_f_ctx->lua_function_ref);
zfree(lua_f_ctx);
luaPushError(lua, err);
sdsfree(err);
return luaRaiseError(lua);
}
return 0;
}
/* Initialize Lua engine, should be called once on start. */
int luaEngineInitEngine() {
luaEngineCtx *lua_engine_ctx = zmalloc(sizeof(*lua_engine_ctx));
......@@ -144,6 +289,18 @@ int luaEngineInitEngine() {
luaRegisterRedisAPI(lua_engine_ctx->lua);
/* Register the library commands table and fields and store it to registry */
lua_pushstring(lua_engine_ctx->lua, LIBRARY_API_NAME);
lua_newtable(lua_engine_ctx->lua);
lua_pushstring(lua_engine_ctx->lua, "register_function");
lua_pushcfunction(lua_engine_ctx->lua, luaRegisterFunction);
lua_settable(lua_engine_ctx->lua, -3);
luaRegisterLogFunction(lua_engine_ctx->lua);
lua_settable(lua_engine_ctx->lua, LUA_REGISTRYINDEX);
/* Save error handler to registry */
lua_pushstring(lua_engine_ctx->lua, REGISTRY_ERROR_HANDLER_NAME);
char *errh_func = "local dbg = debug\n"
......@@ -163,11 +320,16 @@ int luaEngineInitEngine() {
lua_pcall(lua_engine_ctx->lua,0,1,0);
lua_settable(lua_engine_ctx->lua, LUA_REGISTRYINDEX);
/* save the engine_ctx on the registry so we can get it from the Lua interpreter */
luaSaveOnRegistry(lua_engine_ctx->lua, REGISTRY_ENGINE_CTX_NAME, lua_engine_ctx);
/* Save global protection to registry */
luaRegisterGlobalProtectionFunction(lua_engine_ctx->lua);
luaEnableGlobalsProtection(lua_engine_ctx->lua, 0);
/* Set global protection on globals */
lua_pushvalue(lua_engine_ctx->lua, LUA_GLOBALSINDEX);
luaSetGlobalProtection(lua_engine_ctx->lua);
lua_pop(lua_engine_ctx->lua, 1);
/* save the engine_ctx on the registry so we can get it from the Lua interpreter */
luaSaveOnRegistry(lua_engine_ctx->lua, REGISTRY_ENGINE_CTX_NAME, lua_engine_ctx);
engine *lua_engine = zmalloc(sizeof(*lua_engine));
*lua_engine = (engine) {
......
This diff is collapsed.
......@@ -47,13 +47,15 @@
#include "script.h"
#include "redismodule.h"
typedef struct functionLibInfo functionLibInfo;
typedef struct engine {
/* engine specific context */
void *engine_ctx;
/* Create function callback, get the engine_ctx, and function code.
* returns NULL on error and set sds to be the error message */
void* (*create)(void *engine_ctx, sds code, sds *err);
int (*create)(void *engine_ctx, functionLibInfo *li, sds code, sds *err);
/* Invoking a function, r_ctx is an opaque object (from engine POV).
* The r_ctx should be used by the engine to interaction with Redis,
......@@ -89,29 +91,40 @@ typedef struct engineInfo {
/* Hold information about the specific function.
* Used on rdb.c so it must be declared here. */
typedef struct functionInfo {
sds name; /* Function name */
void *function; /* Opaque object that set by the function's engine and allow it
to run the function, usually it's the function compiled code. */
engineInfo *ei; /* Pointer to the function engine */
sds code; /* Function code */
sds desc; /* Function description */
sds name; /* Function name */
void *function; /* Opaque object that set by the function's engine and allow it
to run the function, usually it's the function compiled code. */
functionLibInfo* li; /* Pointer to the library created the function */
sds desc; /* Function description */
} functionInfo;
/* Hold information about the specific library.
* Used on rdb.c so it must be declared here. */
struct functionLibInfo {
sds name; /* Library name */
dict *functions; /* Functions dictionary */
engineInfo *ei; /* Pointer to the function engine */
sds code; /* Library code */
sds desc; /* Library description */
};
int functionsRegisterEngine(const char *engine_name, engine *engine_ctx);
int functionsCreateWithFunctionCtx(sds function_name, sds engine_name, sds desc, sds code,
int replace, sds* err, functionsCtx *functions);
int functionsCreateWithLibraryCtx(sds lib_name, sds engine_name, sds desc, sds code,
int replace, sds* err, functionsLibCtx *lib_ctx);
unsigned long functionsMemory();
unsigned long functionsMemoryOverhead();
int functionsLoad(rio *rdb, int ver);
unsigned long functionsNum();
dict* functionsGet();
size_t functionsLen(functionsCtx *functions_ctx);
functionsCtx* functionsCtxGetCurrent();
functionsCtx* functionsCtxCreate();
void functionsCtxClearCurrent(int async);
void functionsCtxFree(functionsCtx *functions_ctx);
void functionsCtxClear(functionsCtx *functions_ctx);
void functionsCtxSwapWithCurrent(functionsCtx *functions_ctx);
unsigned long functionsLibNum();
dict* functionsLibGet();
size_t functionsLibCtxfunctionsLen(functionsLibCtx *functions_ctx);
functionsLibCtx* functionsLibCtxGetCurrent();
functionsLibCtx* functionsLibCtxCreate();
void functionsLibCtxClearCurrent(int async);
void functionsLibCtxFree(functionsLibCtx *lib_ctx);
void functionsLibCtxClear(functionsLibCtx *lib_ctx);
void functionsLibCtxSwapWithCurrent(functionsLibCtx *lib_ctx);
int functionLibCreateFunction(sds name, void *function, functionLibInfo *li, sds desc, sds *err);
int luaEngineInitEngine();
int functionsInit();
......
......@@ -49,9 +49,9 @@ void lazyFreeLuaScripts(void *args[]) {
/* Release the functions ctx. */
void lazyFreeFunctionsCtx(void *args[]) {
functionsCtx *f_ctx = args[0];
size_t len = functionsLen(f_ctx);
functionsCtxFree(f_ctx);
functionsLibCtx *functions_lib_ctx = args[0];
size_t len = functionsLibCtxfunctionsLen(functions_lib_ctx);
functionsLibCtxFree(functions_lib_ctx);
atomicDecr(lazyfree_objects,len);
atomicIncr(lazyfreed_objects,len);
}
......@@ -204,12 +204,12 @@ void freeLuaScriptsAsync(dict *lua_scripts) {
}
/* Free functions ctx, if the functions ctx contains enough functions, free it in async way. */
void freeFunctionsAsync(functionsCtx *f_ctx) {
if (functionsLen(f_ctx) > LAZYFREE_THRESHOLD) {
atomicIncr(lazyfree_objects,functionsLen(f_ctx));
bioCreateLazyFreeJob(lazyFreeFunctionsCtx,1,f_ctx);
void freeFunctionsAsync(functionsLibCtx *functions_lib_ctx) {
if (functionsLibCtxfunctionsLen(functions_lib_ctx) > LAZYFREE_THRESHOLD) {
atomicIncr(lazyfree_objects,functionsLibCtxfunctionsLen(functions_lib_ctx));
bioCreateLazyFreeJob(lazyFreeFunctionsCtx,1,functions_lib_ctx);
} else {
functionsCtxFree(f_ctx);
functionsLibCtxFree(functions_lib_ctx);
}
}
......
......@@ -1215,7 +1215,7 @@ ssize_t rdbSaveSingleModuleAux(rio *rdb, int when, moduleType *mt) {
}
ssize_t rdbSaveFunctions(rio *rdb) {
dict *functions = functionsGet();
dict *functions = functionsLibGet();
dictIterator *iter = dictGetIterator(functions);
dictEntry *entry = NULL;
ssize_t written = 0;
......@@ -1223,23 +1223,23 @@ ssize_t rdbSaveFunctions(rio *rdb) {
while ((entry = dictNext(iter))) {
if ((ret = rdbSaveType(rdb, RDB_OPCODE_FUNCTION)) < 0) goto werr;
written += ret;
functionInfo *fi = dictGetVal(entry);
if ((ret = rdbSaveRawString(rdb, (unsigned char *) fi->name, sdslen(fi->name))) < 0) goto werr;
functionLibInfo *li = dictGetVal(entry);
if ((ret = rdbSaveRawString(rdb, (unsigned char *) li->name, sdslen(li->name))) < 0) goto werr;
written += ret;
if ((ret = rdbSaveRawString(rdb, (unsigned char *) fi->ei->name, sdslen(fi->ei->name))) < 0) goto werr;
if ((ret = rdbSaveRawString(rdb, (unsigned char *) li->ei->name, sdslen(li->ei->name))) < 0) goto werr;
written += ret;
if (fi->desc) {
if (li->desc) {
/* desc exists */
if ((ret = rdbSaveLen(rdb, 1)) < 0) goto werr;
written += ret;
if ((ret = rdbSaveRawString(rdb, (unsigned char *) fi->desc, sdslen(fi->desc))) < 0) goto werr;
if ((ret = rdbSaveRawString(rdb, (unsigned char *) li->desc, sdslen(li->desc))) < 0) goto werr;
written += ret;
} else {
/* desc not exists */
if ((ret = rdbSaveLen(rdb, 0)) < 0) goto werr;
written += ret;
}
if ((ret = rdbSaveRawString(rdb, (unsigned char *) fi->code, sdslen(fi->code))) < 0) goto werr;
if ((ret = rdbSaveRawString(rdb, (unsigned char *) li->code, sdslen(li->code))) < 0) goto werr;
written += ret;
}
dictReleaseIterator(iter);
......@@ -2746,7 +2746,7 @@ void rdbLoadProgressCallback(rio *r, const void *buf, size_t len) {
* The err output parameter is optional and will be set with relevant error
* message on failure, it is the caller responsibility to free the error
* message on failure. */
int rdbFunctionLoad(rio *rdb, int ver, functionsCtx* functions_ctx, int rdbflags, sds *err) {
int rdbFunctionLoad(rio *rdb, int ver, functionsLibCtx* lib_ctx, int rdbflags, sds *err) {
UNUSED(ver);
sds name = NULL;
sds engine_name = NULL;
......@@ -2756,7 +2756,7 @@ int rdbFunctionLoad(rio *rdb, int ver, functionsCtx* functions_ctx, int rdbflags
sds error = NULL;
int res = C_ERR;
if (!(name = rdbGenericLoadStringObject(rdb, RDB_LOAD_SDS, NULL))) {
error = sdsnew("Failed loading function name");
error = sdsnew("Failed loading library name");
goto error;
}
......@@ -2766,23 +2766,23 @@ int rdbFunctionLoad(rio *rdb, int ver, functionsCtx* functions_ctx, int rdbflags
}
if ((has_desc = rdbLoadLen(rdb, NULL)) == RDB_LENERR) {
error = sdsnew("Failed loading function description indicator");
error = sdsnew("Failed loading library description indicator");
goto error;
}
if (has_desc && !(desc = rdbGenericLoadStringObject(rdb, RDB_LOAD_SDS, NULL))) {
error = sdsnew("Failed loading function description");
error = sdsnew("Failed loading library description");
goto error;
}
if (!(blob = rdbGenericLoadStringObject(rdb, RDB_LOAD_SDS, NULL))) {
error = sdsnew("Failed loading function blob");
error = sdsnew("Failed loading library blob");
goto error;
}
if (functionsCreateWithFunctionCtx(name, engine_name, desc, blob, rdbflags & RDBFLAGS_ALLOW_DUP, &error, functions_ctx) != C_OK) {
if (functionsCreateWithLibraryCtx(name, engine_name, desc, blob, rdbflags & RDBFLAGS_ALLOW_DUP, &error, lib_ctx) != C_OK) {
if (!error) {
error = sdsnew("Failed creating the function");
error = sdsnew("Failed creating the library");
}
goto error;
}
......@@ -2808,8 +2808,8 @@ error:
/* Load an RDB file from the rio stream 'rdb'. On success C_OK is returned,
* otherwise C_ERR is returned and 'errno' is set accordingly. */
int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi) {
functionsCtx* functions_ctx = functionsCtxGetCurrent();
rdbLoadingCtx loading_ctx = { .dbarray = server.db, .functions_ctx = functions_ctx };
functionsLibCtx* functions_lib_ctx = functionsLibCtxGetCurrent();
rdbLoadingCtx loading_ctx = { .dbarray = server.db, .functions_lib_ctx = functions_lib_ctx };
int retval = rdbLoadRioWithLoadingCtx(rdb,rdbflags,rsi,&loading_ctx);
return retval;
}
......@@ -2818,7 +2818,7 @@ int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi) {
/* Load an RDB file from the rio stream 'rdb'. On success C_OK is returned,
* otherwise C_ERR is returned and 'errno' is set accordingly.
* The rdb_loading_ctx argument holds objects to which the rdb will be loaded to,
* currently it only allow to set db object and functionsCtx to which the data
* currently it only allow to set db object and functionLibCtx to which the data
* will be loaded (in the future it might contains more such objects). */
int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadingCtx *rdb_loading_ctx) {
uint64_t dbid = 0;
......@@ -3023,8 +3023,8 @@ int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadin
}
} else if (type == RDB_OPCODE_FUNCTION) {
sds err = NULL;
if (rdbFunctionLoad(rdb, rdbver, rdb_loading_ctx->functions_ctx, rdbflags, &err) != C_OK) {
serverLog(LL_WARNING,"Failed loading function, %s", err);
if (rdbFunctionLoad(rdb, rdbver, rdb_loading_ctx->functions_lib_ctx, rdbflags, &err) != C_OK) {
serverLog(LL_WARNING,"Failed loading library, %s", err);
sdsfree(err);
goto eoferr;
}
......
......@@ -169,7 +169,7 @@ int rdbSaveBinaryFloatValue(rio *rdb, float val);
int rdbLoadBinaryFloatValue(rio *rdb, float *val);
int rdbLoadRio(rio *rdb, int rdbflags, rdbSaveInfo *rsi);
int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadingCtx *rdb_loading_ctx);
int rdbFunctionLoad(rio *rdb, int ver, functionsCtx* functions_ctx, int rdbflags, sds *err);
int rdbFunctionLoad(rio *rdb, int ver, functionsLibCtx* lib_ctx, int rdbflags, sds *err);
int rdbSaveRio(int req, rio *rdb, int *error, int rdbflags, rdbSaveInfo *rsi);
ssize_t rdbSaveFunctions(rio *rdb);
rdbSaveInfo *rdbPopulateSaveInfo(rdbSaveInfo *rsi);
......
......@@ -1792,7 +1792,7 @@ void readSyncBulkPayload(connection *conn) {
ssize_t nread, readlen, nwritten;
int use_diskless_load = useDisklessLoad();
redisDb *diskless_load_tempDb = NULL;
functionsCtx* temp_functions_ctx = NULL;
functionsLibCtx* temp_functions_lib_ctx = NULL;
int empty_db_flags = server.repl_slave_lazy_flush ? EMPTYDB_ASYNC :
EMPTYDB_NO_FLAGS;
off_t left;
......@@ -1968,7 +1968,7 @@ void readSyncBulkPayload(connection *conn) {
if (use_diskless_load && server.repl_diskless_load == REPL_DISKLESS_LOAD_SWAPDB) {
/* Initialize empty tempDb dictionaries. */
diskless_load_tempDb = disklessLoadInitTempDb();
temp_functions_ctx = functionsCtxCreate();
temp_functions_lib_ctx = functionsLibCtxCreate();
moduleFireServerEvent(REDISMODULE_EVENT_REPL_ASYNC_LOAD,
REDISMODULE_SUBEVENT_REPL_ASYNC_LOAD_STARTED,
......@@ -1991,7 +1991,7 @@ void readSyncBulkPayload(connection *conn) {
if (use_diskless_load) {
rio rdb;
redisDb *dbarray;
functionsCtx* functions_ctx;
functionsLibCtx* functions_lib_ctx;
int asyncLoading = 0;
if (server.repl_diskless_load == REPL_DISKLESS_LOAD_SWAPDB) {
......@@ -2004,11 +2004,11 @@ void readSyncBulkPayload(connection *conn) {
asyncLoading = 1;
}
dbarray = diskless_load_tempDb;
functions_ctx = temp_functions_ctx;
functions_lib_ctx = temp_functions_lib_ctx;
} else {
dbarray = server.db;
functions_ctx = functionsCtxGetCurrent();
functionsCtxClear(functions_ctx);
functions_lib_ctx = functionsLibCtxGetCurrent();
functionsLibCtxClear(functions_lib_ctx);
}
rioInitWithConn(&rdb,conn,server.repl_transfer_size);
......@@ -2020,7 +2020,7 @@ void readSyncBulkPayload(connection *conn) {
startLoading(server.repl_transfer_size, RDBFLAGS_REPLICATION, asyncLoading);
int loadingFailed = 0;
rdbLoadingCtx loadingCtx = { .dbarray = dbarray, .functions_ctx = functions_ctx };
rdbLoadingCtx loadingCtx = { .dbarray = dbarray, .functions_lib_ctx = functions_lib_ctx };
if (rdbLoadRioWithLoadingCtx(&rdb,RDBFLAGS_REPLICATION,&rsi,&loadingCtx) != C_OK) {
/* RDB loading failed. */
serverLog(LL_WARNING,
......@@ -2049,7 +2049,7 @@ void readSyncBulkPayload(connection *conn) {
NULL);
disklessLoadDiscardTempDb(diskless_load_tempDb);
functionsCtxFree(temp_functions_ctx);
functionsLibCtxFree(temp_functions_lib_ctx);
serverLog(LL_NOTICE, "MASTER <-> REPLICA sync: Discarding temporary DB in background");
} else {
/* Remove the half-loaded data in case we started with an empty replica. */
......@@ -2073,7 +2073,7 @@ void readSyncBulkPayload(connection *conn) {
swapMainDbWithTempDb(diskless_load_tempDb);
/* swap existing functions ctx with the temporary one */
functionsCtxSwapWithCurrent(temp_functions_ctx);
functionsLibCtxSwapWithCurrent(temp_functions_lib_ctx);
moduleFireServerEvent(REDISMODULE_EVENT_REPL_ASYNC_LOAD,
REDISMODULE_SUBEVENT_REPL_ASYNC_LOAD_COMPLETED,
......
......@@ -80,6 +80,9 @@ void* luaGetFromRegistry(lua_State* lua, const char* name) {
lua_pushstring(lua, name);
lua_gettable(lua, LUA_REGISTRYINDEX);
if (lua_isnil(lua, -1)) {
return NULL;
}
/* must be light user data */
serverAssert(lua_islightuserdata(lua, -1));
......@@ -427,7 +430,7 @@ static void redisProtocolToLuaType_Double(void *ctx, double d, const char *proto
* with a single "err" field set to the error string. Note that this
* table is never a valid reply by proper commands, since the returned
* tables are otherwise always indexed by integers, never by strings. */
static void luaPushError(lua_State *lua, char *error) {
void luaPushError(lua_State *lua, char *error) {
lua_Debug dbg;
/* If debugging is active and in step mode, log errors resulting from
......@@ -455,7 +458,7 @@ static void luaPushError(lua_State *lua, char *error) {
* by the non-error-trapping version of redis.pcall(), which is redis.call(),
* this function will raise the Lua error so that the execution of the
* script will be halted. */
static int luaRaiseError(lua_State *lua) {
int luaRaiseError(lua_State *lua) {
lua_pushstring(lua,"err");
lua_gettable(lua,-2);
return lua_error(lua);
......@@ -656,6 +659,10 @@ static void luaReplyToRedisReply(client *c, client* script_client, lua_State *lu
static int luaRedisGenericCommand(lua_State *lua, int raise_error) {
int j, argc = lua_gettop(lua);
scriptRunCtx* rctx = luaGetFromRegistry(lua, REGISTRY_RUN_CTX_NAME);
if (!rctx) {
luaPushError(lua, "redis.call/pcall can only be called inside a script invocation");
return luaRaiseError(lua);
}
sds err = NULL;
client* c = rctx->c;
sds reply;
......@@ -911,6 +918,10 @@ static int luaRedisSetReplCommand(lua_State *lua) {
int flags, argc = lua_gettop(lua);
scriptRunCtx* rctx = luaGetFromRegistry(lua, REGISTRY_RUN_CTX_NAME);
if (!rctx) {
lua_pushstring(lua, "redis.set_repl can only be called inside a script invocation");
return lua_error(lua);
}
if (argc != 1) {
lua_pushstring(lua, "redis.set_repl() requires two arguments.");
......@@ -966,6 +977,11 @@ static int luaLogCommand(lua_State *lua) {
/* redis.setresp() */
static int luaSetResp(lua_State *lua) {
scriptRunCtx* rctx = luaGetFromRegistry(lua, REGISTRY_RUN_CTX_NAME);
if (!rctx) {
lua_pushstring(lua, "redis.setresp can only be called inside a script invocation");
return lua_error(lua);
}
int argc = lua_gettop(lua);
if (argc != 1) {
......@@ -978,7 +994,6 @@ static int luaSetResp(lua_State *lua) {
lua_pushstring(lua, "RESP version must be 2 or 3.");
return lua_error(lua);
}
scriptRunCtx* rctx = luaGetFromRegistry(lua, REGISTRY_RUN_CTX_NAME);
scriptSetResp(rctx, resp);
return 0;
}
......@@ -1031,8 +1046,8 @@ static void luaRemoveUnsupportedFunctions(lua_State *lua) {
* sequence, because it may interact with creation of globals.
*
* On Legacy Lua (eval) we need to check 'w ~= \"main\"' otherwise we will not be able
* to create the global 'function <sha> ()' variable. On Lua engine we do not use this trick
* so its not needed. */
* to create the global 'function <sha> ()' variable. On Functions Lua engine we do not use
* this trick so it's not needed. */
void luaEnableGlobalsProtection(lua_State *lua, int is_eval) {
char *s[32];
sds code = sdsempty();
......@@ -1067,33 +1082,72 @@ void luaEnableGlobalsProtection(lua_State *lua, int is_eval) {
sdsfree(code);
}
void luaRegisterRedisAPI(lua_State* lua) {
luaLoadLibraries(lua);
luaRemoveUnsupportedFunctions(lua);
/* Register the redis commands table and fields */
lua_newtable(lua);
/* redis.call */
lua_pushstring(lua,"call");
lua_pushcfunction(lua,luaRedisCallCommand);
lua_settable(lua,-3);
/* Create a global protection function and put it to registry.
* This need to be called once in the lua_State lifetime.
* After called it is possible to use luaSetGlobalProtection
* to set global protection on a give table.
*
* The function assumes the Lua stack have a least enough
* space to push 2 element, its up to the caller to verify
* this before calling this function.
*
* Notice, the difference between this and luaEnableGlobalsProtection
* is that luaEnableGlobalsProtection is enabling global protection
* on the current Lua globals. This registering a global protection
* function that later can be applied on any table. */
void luaRegisterGlobalProtectionFunction(lua_State *lua) {
lua_pushstring(lua, REGISTRY_SET_GLOBALS_PROTECTION_NAME);
char *global_protection_func = "local dbg = debug\n"
"local globals_protection = function (t)\n"
" local mt = {}\n"
" setmetatable(t, mt)\n"
" mt.__newindex = function (t, n, v)\n"
" if dbg.getinfo(2) then\n"
" local w = dbg.getinfo(2, \"S\").what\n"
" if w ~= \"C\" then\n"
" error(\"Script attempted to create global variable '\"..tostring(n)..\"'\", 2)\n"
" end"
" end"
" rawset(t, n, v)\n"
" end\n"
" mt.__index = function (t, n)\n"
" if dbg.getinfo(2) and dbg.getinfo(2, \"S\").what ~= \"C\" then\n"
" error(\"Script attempted to access nonexistent global variable '\"..tostring(n)..\"'\", 2)\n"
" end\n"
" return rawget(t, n)\n"
" end\n"
"end\n"
"return globals_protection";
int res = luaL_loadbuffer(lua, global_protection_func, strlen(global_protection_func), "@global_protection_def");
serverAssert(res == 0);
res = lua_pcall(lua,0,1,0);
serverAssert(res == 0);
lua_settable(lua, LUA_REGISTRYINDEX);
}
/* redis.pcall */
lua_pushstring(lua,"pcall");
lua_pushcfunction(lua,luaRedisPCallCommand);
lua_settable(lua,-3);
/* Set global protection on a given table.
* The table need to be located on the top of the lua stack.
* After called, it will no longer be possible to set
* new items on the table. The function is not removing
* the table from the top of the stack!
*
* The function assumes the Lua stack have a least enough
* space to push 2 element, its up to the caller to verify
* this before calling this function. */
void luaSetGlobalProtection(lua_State *lua) {
lua_pushstring(lua, REGISTRY_SET_GLOBALS_PROTECTION_NAME);
lua_gettable(lua, LUA_REGISTRYINDEX);
lua_pushvalue(lua, -2);
int res = lua_pcall(lua, 1, 0, 0);
serverAssert(res == 0);
}
void luaRegisterLogFunction(lua_State* lua) {
/* redis.log and log levels. */
lua_pushstring(lua,"log");
lua_pushcfunction(lua,luaLogCommand);
lua_settable(lua,-3);
/* redis.setresp */
lua_pushstring(lua,"setresp");
lua_pushcfunction(lua,luaSetResp);
lua_settable(lua,-3);
lua_pushstring(lua,"LOG_DEBUG");
lua_pushnumber(lua,LL_DEBUG);
lua_settable(lua,-3);
......@@ -1109,6 +1163,31 @@ void luaRegisterRedisAPI(lua_State* lua) {
lua_pushstring(lua,"LOG_WARNING");
lua_pushnumber(lua,LL_WARNING);
lua_settable(lua,-3);
}
void luaRegisterRedisAPI(lua_State* lua) {
luaLoadLibraries(lua);
luaRemoveUnsupportedFunctions(lua);
/* Register the redis commands table and fields */
lua_newtable(lua);
/* redis.call */
lua_pushstring(lua,"call");
lua_pushcfunction(lua,luaRedisCallCommand);
lua_settable(lua,-3);
/* redis.pcall */
lua_pushstring(lua,"pcall");
lua_pushcfunction(lua,luaRedisPCallCommand);
lua_settable(lua,-3);
luaRegisterLogFunction(lua);
/* redis.setresp */
lua_pushstring(lua,"setresp");
lua_pushcfunction(lua,luaSetResp);
lua_settable(lua,-3);
/* redis.sha1hex */
lua_pushstring(lua, "sha1hex");
......@@ -1149,7 +1228,7 @@ void luaRegisterRedisAPI(lua_State* lua) {
lua_settable(lua,-3);
/* Finally set the table as 'redis' global var. */
lua_setglobal(lua,"redis");
lua_setglobal(lua,REDIS_API_NAME);
/* Replace math.random and math.randomseed with our implementations. */
lua_getglobal(lua,"math");
......@@ -1167,7 +1246,7 @@ void luaRegisterRedisAPI(lua_State* lua) {
/* Set an array of Redis String Objects as a Lua array (table) stored into a
* global variable. */
static void luaSetGlobalArray(lua_State *lua, char *var, robj **elev, int elec) {
static void luaCreateArray(lua_State *lua, robj **elev, int elec) {
int j;
lua_newtable(lua);
......@@ -1175,7 +1254,6 @@ static void luaSetGlobalArray(lua_State *lua, char *var, robj **elev, int elec)
lua_pushlstring(lua,(char*)elev[j]->ptr,sdslen(elev[j]->ptr));
lua_rawseti(lua,-2,j+1);
}
lua_setglobal(lua,var);
}
/* ---------------------------------------------------------------------------
......@@ -1189,6 +1267,11 @@ static void luaSetGlobalArray(lua_State *lua, char *var, robj **elev, int elec)
/* The following implementation is the one shipped with Lua itself but with
* rand() replaced by redisLrand48(). */
static int redis_math_random (lua_State *L) {
scriptRunCtx* rctx = luaGetFromRegistry(L, REGISTRY_RUN_CTX_NAME);
if (!rctx) {
return luaL_error(L, "math.random can only be called inside a script invocation");
}
/* the `%' avoids the (rare) case of r==1, and is needed also because on
some systems (SunOS!) `rand()' may return a value larger than RAND_MAX */
lua_Number r = (lua_Number)(redisLrand48()%REDIS_LRAND48_MAX) /
......@@ -1217,6 +1300,10 @@ static int redis_math_random (lua_State *L) {
}
static int redis_math_randomseed (lua_State *L) {
scriptRunCtx* rctx = luaGetFromRegistry(L, REGISTRY_RUN_CTX_NAME);
if (!rctx) {
return luaL_error(L, "math.randomseed can only be called inside a script invocation");
}
redisSrand48(luaL_checkint(L, 1));
return 0;
}
......@@ -1260,13 +1347,24 @@ void luaCallFunction(scriptRunCtx* run_ctx, lua_State *lua, robj** keys, size_t
/* Populate the argv and keys table accordingly to the arguments that
* EVAL received. */
luaSetGlobalArray(lua,"KEYS",keys,nkeys);
luaSetGlobalArray(lua,"ARGV",args,nargs);
luaCreateArray(lua,keys,nkeys);
/* On eval, keys and arguments are globals. */
if (run_ctx->flags & SCRIPT_EVAL_MODE) lua_setglobal(lua,"KEYS");
luaCreateArray(lua,args,nargs);
if (run_ctx->flags & SCRIPT_EVAL_MODE) lua_setglobal(lua,"ARGV");
/* At this point whether this script was never seen before or if it was
* already defined, we can call it. We have zero arguments and expect
* a single return value. */
int err = lua_pcall(lua,0,1,-2);
* already defined, we can call it.
* On eval mode, we have zero arguments and expect a single return value.
* In addition the error handler is located on position -2 on the Lua stack.
* On function mode, we pass 2 arguments (the keys and args tables),
* and the error handler is located on position -4 (stack: error_handler, callback, keys, args) */
int err;
if (run_ctx->flags & SCRIPT_EVAL_MODE) {
err = lua_pcall(lua,0,1,-2);
} else {
err = lua_pcall(lua,2,1,-4);
}
/* Call the Lua garbage collector from time to time to avoid a
* full cycle performed by Lua, which adds too latency.
......
......@@ -55,9 +55,16 @@
#include <lualib.h>
#define REGISTRY_RUN_CTX_NAME "__RUN_CTX__"
#define REGISTRY_SET_GLOBALS_PROTECTION_NAME "__GLOBAL_PROTECTION__"
#define REDIS_API_NAME "redis"
void luaRegisterRedisAPI(lua_State* lua);
void luaEnableGlobalsProtection(lua_State *lua, int is_eval);
void luaRegisterGlobalProtectionFunction(lua_State *lua);
void luaSetGlobalProtection(lua_State *lua);
void luaRegisterLogFunction(lua_State* lua);
void luaPushError(lua_State *lua, char *error);
int luaRaiseError(lua_State *lua);
void luaSaveOnRegistry(lua_State* lua, const char* name, void* ptr);
void* luaGetFromRegistry(lua_State* lua, const char* name);
void luaCallFunction(scriptRunCtx* r_ctx, lua_State *lua, robj** keys, size_t nkeys, robj** args, size_t nargs, int debug_enabled);
......
......@@ -4886,6 +4886,7 @@ sds genRedisInfoString(const char *section) {
"used_memory_scripts_eval:%lld\r\n"
"number_of_cached_scripts:%lu\r\n"
"number_of_functions:%lu\r\n"
"number_of_libraries:%lu\r\n"
"used_memory_vm_functions:%lld\r\n"
"used_memory_vm_total:%lld\r\n"
"used_memory_vm_total_human:%s\r\n"
......@@ -4936,6 +4937,7 @@ sds genRedisInfoString(const char *section) {
(long long) mh->lua_caches,
dictSize(evalScriptsDict()),
functionsNum(),
functionsLibNum(),
memory_functions,
memory_functions + memory_lua,
used_memory_vm_total_hmem,
......
......@@ -873,7 +873,7 @@ typedef struct redisDb {
} redisDb;
/* forward declaration for functions ctx */
typedef struct functionsCtx functionsCtx;
typedef struct functionsLibCtx functionsLibCtx;
/* Holding object that need to be populated during
* rdb loading. On loading end it is possible to decide
......@@ -882,7 +882,7 @@ typedef struct functionsCtx functionsCtx;
* successful loading and dropped on failure. */
typedef struct rdbLoadingCtx {
redisDb* dbarray;
functionsCtx* functions_ctx;
functionsLibCtx* functions_lib_ctx;
}rdbLoadingCtx;
/* Client MULTI/EXEC state */
......@@ -3017,7 +3017,7 @@ int ldbPendingChildren(void);
sds luaCreateFunction(client *c, robj *body);
void luaLdbLineHook(lua_State *lua, lua_Debug *ar);
void freeLuaScriptsAsync(dict *lua_scripts);
void freeFunctionsAsync(functionsCtx *f_ctx);
void freeFunctionsAsync(functionsLibCtx *lib_ctx);
int ldbIsEnabled();
void ldbLog(sds entry);
void ldbLogRedisReply(char *reply);
......@@ -3279,11 +3279,10 @@ void evalShaRoCommand(client *c);
void scriptCommand(client *c);
void fcallCommand(client *c);
void fcallroCommand(client *c);
void functionCreateCommand(client *c);
void functionLoadCommand(client *c);
void functionDeleteCommand(client *c);
void functionKillCommand(client *c);
void functionStatsCommand(client *c);
void functionInfoCommand(client *c);
void functionListCommand(client *c);
void functionHelpCommand(client *c);
void functionFlushCommand(client *c);
......
......@@ -322,7 +322,7 @@ if {!$::tls} { ;# fake_redis_node doesn't support TLS
set dir [lindex [r config get dir] 1]
assert_equal "OK" [r debug populate 100000 key 1000]
assert_equal "OK" [r function create lua func1 "return 123"]
assert_equal "OK" [r function load lua lib1 "redis.register_function('func1', function() return 123 end)"]
if {$functions_only} {
set args "--functions-rdb $dir/cli.rdb"
} else {
......@@ -335,11 +335,10 @@ if {!$::tls} { ;# fake_redis_node doesn't support TLS
file rename "$dir/cli.rdb" "$dir/dump.rdb"
assert_equal "OK" [r set should-not-exist 1]
assert_equal "OK" [r function create lua should_not_exist_func "return 456"]
assert_equal "OK" [r function load lua should_not_exist_func "redis.register_function('should_not_exist_func', function() return 456 end)"]
assert_equal "OK" [r debug reload nosave]
assert_equal {} [r get should-not-exist]
assert_error "ERR Function does not exists" {r function info should_not_exist_func}
assert_equal "func1" [dict get [r function info func1] name]
assert_equal {{library_name lib1 engine LUA description {} functions {{name func1 description {}}}}} [r function list]
if {$functions_only} {
assert_equal 0 [r dbsize]
} else {
......
......@@ -522,10 +522,10 @@ foreach testType {Successful Aborted} {
$replica set mykey myvalue
# Set a function value on replica to check status during loading, on failure and after swapping db
$replica function create LUA test {return 'hello1'}
$replica function load LUA test {redis.register_function('test', function() return 'hello1' end)}
# Set a function value on master to check it reaches the replica when replication ends
$master function create LUA test {return 'hello2'}
$master function load LUA test {redis.register_function('test', function() return 'hello2' end)}
# Force the replica to try another full sync (this time it will have matching master replid)
$master multi
......@@ -658,7 +658,7 @@ test {diskless loading short read} {
set start [clock clicks -milliseconds]
# Set a function value to check short read handling on functions
r function create LUA test {return 'hello1'}
r function load LUA test {redis.register_function('test', function() return 'hello1' end)}
for {set k 0} {$k < 3} {incr k} {
for {set i 0} {$i < 10} {incr i} {
......
......@@ -182,7 +182,7 @@ start_server [list overrides $base_conf] {
# upload a function to all the cluster
exec src/redis-cli --cluster-yes --cluster call 127.0.0.1:[srv 0 port] \
FUNCTION CREATE LUA TEST {return 'hello'}
FUNCTION LOAD LUA TEST {redis.register_function('test', function() return 'hello' end)}
# adding node to the cluster
exec src/redis-cli --cluster-yes --cluster add-node \
......@@ -199,13 +199,13 @@ start_server [list overrides $base_conf] {
}
# make sure 'test' function was added to the new node
assert_equal {{name TEST engine LUA description {}}} [$node4_rd FUNCTION LIST]
assert_equal {{library_name TEST engine LUA description {} functions {{name test description {}}}}} [$node4_rd FUNCTION LIST]
# add function to node 5
assert_equal {OK} [$node5_rd FUNCTION CREATE LUA TEST {return 'hello1'}]
assert_equal {OK} [$node5_rd FUNCTION LOAD LUA TEST {redis.register_function('test', function() return 'hello' end)}]
# make sure functions was added to node 5
assert_equal {{name TEST engine LUA description {}}} [$node5_rd FUNCTION LIST]
assert_equal {{library_name TEST engine LUA description {} functions {{name test description {}}}}} [$node5_rd FUNCTION LIST]
# adding node 5 to the cluster should failed because it already contains the 'test' function
catch {
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment