• Meir Shpilraien (Spielrein)'s avatar
    Redis Function Libraries (#10004) · 885f6b5c
    Meir Shpilraien (Spielrein) authored
    # Redis Function Libraries
    
    This PR implements Redis Functions Libraries as describe on: https://github.com/redis/redis/issues/9906.
    
    Libraries purpose is to provide a better code sharing between functions by allowing to create multiple
    functions in a single command. Functions that were created together can safely share code between
    each other without worrying about compatibility issues and versioning.
    
    Creating a new library is done using 'FUNCTION LOAD' command (full API is described below)
    
    This PR introduces a new struct called libraryInfo, libraryInfo holds information about a library:
    * name - name of the library
    * engine - engine used to create the library
    * code - library code
    * description - library description
    * functions - the functions exposed by the library
    
    When Redis gets the `FUNCTION LOAD` command it creates a new empty libraryInfo.
    Redis passes the `CODE` to the relevant engine alongside the empty libraryInfo.
    As a result, the engine will create one or more functions by calling 'libraryCreateFunction'.
    The new funcion will be added to the newly created libraryInfo. So far Everything is happening
    locally on the libraryInfo so it is easy to abort the operation (in case of an error) by simply
    freeing the libraryInfo. After the library info is fully constructed we start the joining phase by
    which we will join the new library to the other libraries currently exist on Redis.
    The joining phase make sure there is no function collision and add the library to the
    librariesCtx (renamed from functionCtx). LibrariesCtx is used all around the code in the exact
    same way as functionCtx was used (with respect to RDB loading, replicatio, ...).
    The only difference is that apart from function dictionary (maps function name to functionInfo
    object), the librariesCtx contains also a libraries dictionary that maps library name to libraryInfo object.
    
    ## New API
    ### FUNCTION LOAD
    `FUNCTION LOAD <ENGINE> <LIBRARY NAME> [REPLACE] [DESCRIPTION <DESCRIPTION>] <CODE>`
    Create a new library with the given parameters:
    * ENGINE - REPLACE Engine name to use to create the library.
    * LIBRARY NAME - The new library name.
    * REPLACE - If the library already exists, replace it.
    * DESCRIPTION - Library description.
    * CODE - Library code.
    
    Return "OK" on success, or error on the following cases:
    * Library name already taken and REPLACE was not used
    * Name collision with another existing library (even if replace was uses)
    * Library registration failed by the engine (usually compilation error)
    
    ## Changed API
    ### FUNCTION LIST
    `FUNCTION LIST [LIBRARYNAME <LIBRARY NAME PATTERN>] [WITHCODE]`
    Command was modified to also allow getting libraries code (so `FUNCTION INFO` command is no longer
    needed and removed). In addition the command gets an option argument, `LIBRARYNAME` allows you to
    only get libraries that match the given `LIBRARYNAME` pattern. By default, it returns all libraries.
    
    ### INFO MEMORY
    Added number of libraries to `INFO MEMORY`
    
    ### Commands flags
    `DENYOOM` flag was set on `FUNCTION LOAD` and `FUNCTION RESTORE`. We consider those commands
    as commands that add new data to the dateset (functions are data) and so we want to disallows
    to run those commands on OOM.
    
    ## Removed API
    * FUNCTION CREATE - Decided on https://github.com/redis/redis/issues/9906
    * FUNCTION INFO - Decided on https://github.com/redis/redis/issues/9899
    
    ## Lua engine changes
    When the Lua engine gets the code given on `FUNCTION LOAD` command, it immediately runs it, we call
    this run the loading run. Loading run is not a usual script run, it is not possible to invoke any
    Redis command from within the load run.
    Instead there is a new API provided by `library` object. The new API's: 
    * `redis.log` - behave the same as `redis.log`
    * `redis.register_function` - register a new function to the library
    
    The loading run purpose is to register functions using the new `redis.register_function` API.
    Any attempt to use any other API will result in an error. In addition, the load run is has a time
    limit of 500ms, error is raise on timeout and the entire operation is aborted.
    
    ### `redis.register_function`
    `redis.register_function(<function_name>, <callback>, [<description>])`
    This new API allows users to register a new function that will be linked to the newly created library.
    This API can only be called during the load run (see definition above). Any attempt to use it outside
    of the load run will result in an error.
    The parameters pass to the API are:
    * function_name - Function name (must be a Lua string)
    * callback - Lua function object that will be called when the function is invokes using fcall/fcall_ro
    * description - Function description, optional (must be a Lua string).
    
    ### Example
    The following example creates a library called `lib` with 2 functions, `f1` and `f1`, returns 1 and 2 respectively:
    ```
    local function f1(keys, args)
        return 1
    end
    
    local function f2(keys, args)
        return 2
    end
    
    redis.register_function('f1', f1)
    redis.register_function('f2', f2)
    ```
    
    Notice: Unlike `eval`, functions inside a library get the KEYS and ARGV as arguments to the
    functions and not as global.
    
    ### Technical Details
    
    On the load run we only want the user to be able to call a white list on API's. This way, in
    the future, if new API's will be added, the new API's will not be available to the load run
    unless specifically added to this white list. We put the while list on the `library` object and
    make sure the `library` object is only available to the load run by using [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv) API. This API allows us to set
    the `globals` of a function (and all the function it creates). Before starting the load run we
    create a new fresh Lua table (call it `g`) that only contains the `library` API (we make sure
    to set global protection on this table just like the general global protection already exists
    today), then we use [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv)
    to set `g` as the global table of the load run. After the load run finished we update `g`
    metatable and set `__index` and `__newindex` functions to be `_G` (Lua default globals),
    we also pop out the `library` object as we do not need it anymore.
    This way, any function that was created on the load run (and will be invoke using `fcall`) will
    see the default globals as it expected to see them and will not have the `library` API anymore.
    
    An important outcome of this new approach is that now we can achieve a distinct global table
    for each library (it is not yet like that but it is very easy to achieve it now). In the future we can
    decide to remove global protection because global on different libraries will not collide or we
    can chose to give different API to different libraries base on some configuration or input.
    
    Notice that this technique was meant to prevent errors and was not meant to prevent malicious
    user from exploit it. For example, the load run can still save the `library` object on some local
    variable and then using in `fcall` context. To prevent such a malicious use, the C code also make
    sure it is running in the right context and if not raise an error.
    885f6b5c
redis-cli.tcl 14.8 KB