1. 16 Jul, 2024 1 commit
    • debing.sun's avatar
      Trigger Lua GC after script loading (#13407) · 88af96c7
      debing.sun authored
      Nowdays we do not trigger LUA GC after loading lua script. This means
      that when a large number of scripts are loaded, such as when functions
      are propagating from the master to the replica, if the LUA scripts are
      never touched on the replica, the garbage might remain there
      indefinitely.
      
      Before this PR, we would share a gc_count between scripts and functions.
      This means that, under certain circumstances, the GC trigger for scripts
      and functions was not fair.
      For example, loading a large number of scripts followed by a small
      number of functions could result in the functions triggering GC.
      In this PR, we assign a unique `gc_count` to each of them, so the GC
      triggers between them will no longer affect each other.
      
      on the other hand, this PR will to bring regession for script loading
      commands(`FUNCTION LOAD` and `SCRIPT LOAD`), but they are not hot path,
      we can ignore it, and it will be replaced
      https://github.com/redis/redis/pull/13375
      
       in the future.
      
      ---------
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      88af96c7
  2. 10 Jul, 2024 1 commit
    • debing.sun's avatar
      Rebuild function engines for function flush command (#13383) · ffff7fea
      debing.sun authored
      
      
      ### Issue
      The current implementation of `FUNCTION FLUSH` command uses
      `lua_unref()` to unreference script closures in Lua vm. However,
      invoking `lua_unref()` during lazy free (`ASYNC` argument) is risky
      since it is not thread-safe.
      
      Another issue is that using `lua_unref()` to unreference references does
      not trigger GC, This can result in the Lua VM leaves a significant
      amount of garbage, which may never be cleaned up if not properly GC.
      
      ### Solution
      The proposed solution is to completely rebuild the engines, resulting in
      a brand new Lua VM.
      
      ---------
      Co-authored-by: default avatarmeir <meir@redis.com>
      ffff7fea
  3. 23 Jan, 2024 1 commit
    • Binbin's avatar
      Some cleanups around function (#12940) · 628c0dea
      Binbin authored
      This PR did some cleanups around function:
      - drop the comment about Libraries Ctx, since we do have comment
        in functionsLibCtx, no need to maintain multiple copies.
      - remove outdated comment about the dropped Library description.
      - remove unused desc and code vars in functionExtractLibMetaData.
      - fix engines_nemory typo, changed it to engines_memory.
      - remove outdated comment about FUNCTION CREATE and FUNCTION INFO,
        FUNCTION CREATE was renamed to FUNCTION LOAD.
      - Check in initServer whether the return of functionsInit is OK.
      628c0dea
  4. 16 Aug, 2022 1 commit
  5. 14 Aug, 2022 1 commit
    • sundb's avatar
      Add missing lua_pop in luaGetFromRegistry (#11097) · 8aad2ac3
      sundb authored
      This pr mainly has the following four changes:
      
      1. Add missing lua_pop in `luaGetFromRegistry`.
          This bug affects `redis.register_function`, where `luaGetFromRegistry` in
          `luaRegisterFunction` will return null when we call `redis.register_function` nested.
          .e.g
          ```
          FUNCTION LOAD "#!lua name=mylib \n local lib=redis \n lib.register_function('f2', function(keys, args) lib.register_function('f1', function () end) end)"
          fcall f2 0
          ````
          But since we exit when luaGetFromRegistry returns null, it does not cause the stack to grow indefinitely.
      
      3. When getting `REGISTRY_RUN_CTX_NAME` from the registry, use `serverAssert`
          instead of error return. Since none of these lua functions are registered at the time
          of function load, scriptRunCtx will never be NULL.
      4. Add `serverAssert` for `luaLdbLineHook`, `luaEngineLoadHook`.
      5. Remove `luaGetFromRegistry` from `redis_math_random` and
          `redis_math_randomseed`, it looks like they are redundant.
      8aad2ac3
  6. 03 Aug, 2022 1 commit
  7. 01 Jun, 2022 1 commit
    • Oran Agra's avatar
      Expose script flags to processCommand for better handling (#10744) · df558618
      Oran Agra authored
      The important part is that read-only scripts (not just EVAL_RO
      and FCALL_RO, but also ones with `no-writes` executed by normal EVAL or
      FCALL), will now be permitted to run during CLIENT PAUSE WRITE (unlike
      before where only the _RO commands would be processed).
      
      Other than that, some errors like OOM, READONLY, MASTERDOWN are now
      handled by processCommand, rather than the command itself affects the
      error string (and even error code in some cases), and command stats.
      
      Besides that, now the `may-replicate` commands, PFCOUNT and PUBLISH, will
      be considered `write` commands in scripts and will be blocked in all
      read-only scripts just like other write commands.
      They'll also be blocked in EVAL_RO (i.e. even for scripts without the
      `no-writes` shebang flag.
      
      This commit also hides the `may_replicate` flag from the COMMAND command
      output. this is a **breaking change**.
      
      background about may_replicate:
      We don't want to expose a no-may-replicate flag or alike to scripts, since we
      consider the may-replicate thing an internal concern of redis, that we may
      some day get rid of.
      In fact, the may-replicate flag was initially introduced to flag EVAL: since
      we didn't know what it's gonna do ahead of execution, before function-flags
      existed). PUBLISH and PFCOUNT, both of which because they have side effects
      which may some day be fixed differently.
      
      code changes:
      The changes in eval.c are mostly code re-ordering:
      - evalCalcFunctionName is extracted out of evalGenericCommand
      - evalExtractShebangFlags is extracted luaCreateFunction
      - evalGetCommandFlags is new code
      df558618
  8. 22 May, 2022 1 commit
    • Oran Agra's avatar
      Scripts that declare the `no-writes` flag are implicitly `allow-oom` too. (#10699) · b0e18f80
      Oran Agra authored
      Scripts that have the `no-writes` flag, cannot execute write commands,
      and since all `deny-oom` commands are write commands, we now act
      as if the `allow-oom` flag is implicitly set for scripts that set the `no-writes` flag.
      this also implicitly means that the EVAL*_RO and FCALL_RO commands can
      never fails with OOM error.
      
      Note about a bug that's no longer relevant:
      There was an issue with EVAL*_RO using shebang not being blocked correctly
      in OOM state:
      When an EVAL script declares a shebang, it was by default not allowed to run in
      OOM state.
      but this depends on a flag that is updated before the command is executed, which
      was not updated in case of the `_RO` variants.
      the result is that if the previous cached state was outdated (either true or false),
      the script will either unjustly fail with OOM, or unjustly allowed to run despite
      the OOM state.
      It doesn't affect scripts without a shebang since these depend on the actual
      commands they run, and since these are only read commands, they don't care
      for that cached oom state flag.
      it did affect scripts with shebang and no allow-oom flag, bug after the change in
      this PR, scripts that are run with eval_ro would implicitly have that flag so again
      the cached state doesn't matter.
      
      p.s. this isn't a breaking change since all it does is allow scripts to run when they
      should / could rather than blocking them.
      b0e18f80
  9. 10 May, 2022 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Fix #10705, avoid relinking the same library twice. (#10706) · 442e73ea
      Meir Shpilraien (Spielrein) authored
      Set `old_li` to NULL to avoid linking it again on error.
      Before the fix, loading an already existing library will cause the existing library to be added again. This cause not harm other then wrong statistics. The statistics that are effected  by the issue are:
      * `libraries_count` and `functions_count` returned by `function stats` command
      * `used_memory_functions` returned on `info memory` command
      * `functions.caches` returned on `memory stats` command
      442e73ea
  10. 26 Apr, 2022 2 commits
    • meir's avatar
      Protect globals of both evals scripts and functions. · 3731580b
      meir authored
      Use the new `lua_enablereadonlytable` Lua API to protect the global tables of
      both evals scripts and functions. For eval scripts, the implemetation is easy,
      We simply call `lua_enablereadonlytable` on the global table to turn it into
      a readonly table.
      
      On functions its more complecated, we want to be able to switch globals between
      load run and function run. To achieve this, we create a new empty table that
      acts as the globals table for function, we control the actual globals using metatable
      manipulation. Notice that even if the user gets a pointer to the original tables, all
      the tables are set to be readonly (using `lua_enablereadonlytable` Lua API) so he can
      not change them. The following inlustration better explain the solution:
      
      ```
      Global table {} <- global table metatable {.__index = __real_globals__}
      ```
      
      The `__real_globals__` is set depends on the run context (function load or function call).
      
      Why this solution is needed and its not enough to simply switch globals?
      When we run in the context of function load and create our functions, our function gets
      the current globals that was set when they were created. Replacing the globals after
      the creation will not effect them. This is why this trick it mandatory.
      3731580b
    • Madelyn Olson's avatar
      By default prevent cross slot operations in functions and scripts with # (#10615) · efcd1bf3
      Madelyn Olson authored
      Adds the `allow-cross-slot-keys` flag to Eval scripts and Functions to allow
      scripts to access keys from multiple slots.
      The default behavior is now that they are not allowed to do that (unlike before).
      This is a breaking change for 7.0 release candidates (to be part of 7.0.0), but
      not for previous redis releases since EVAL without shebang isn't doing this check.
      
      Note that the check is done on both the keys declared by the EVAL / FCALL command
      arguments, and also the ones used by the script when making a `redis.call`.
      
      A note about the implementation, there seems to have been some confusion
      about allowing access to non local keys. I thought I missed something in our
      wider conversation, but Redis scripts do block access to non-local keys.
      So the issue was just about cross slots being accessed.
      efcd1bf3
  11. 25 Apr, 2022 1 commit
    • guybe7's avatar
      Fix regression not aborting transaction on error, and re-edit some error responses (#10612) · df787764
      guybe7 authored
      1. Disk error and slave count checks didn't flag the transactions or counted correctly in command stats (regression from #10372  , 7.0 RC3)
      2. RM_Call will reply the same way Redis does, in case of non-exisitng command or arity error
      3. RM_WrongArtiy will consider the full command name
      4. Use lowercase 'u' in "unknonw subcommand" (to align with "unknown command")
      
      Followup work of #10127
      df787764
  12. 19 Apr, 2022 1 commit
  13. 05 Apr, 2022 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Functions: Move library meta data to be part of the library payload. (#10500) · ae020e3d
      Meir Shpilraien (Spielrein) authored
      ## Move library meta data to be part of the library payload.
      
      Following the discussion on https://github.com/redis/redis/issues/10429 and the intention to add (in the future) library versioning support, we believe that the entire library metadata (like name and engine) should be part of the library payload and not provided by the `FUNCTION LOAD` command. The reasoning behind this is that the programmer who developed the library should be the one who set those values (name, engine, and in the future also version). **It is not the responsibility of the admin who load the library into the database.**
      
      The PR moves all the library metadata (engine and function name) to be part of the library payload. The metadata needs to be provided on the first line of the payload using the shebang format (`#!<engine> name=<name>`), example:
      
      ```lua
      #!lua name=test
      redis.register_function('foo', function() return 1 end)
      ```
      
      The above script will run on the Lua engine and will create a library called `test`.
      
      ## API Changes (compare to 7.0 rc2)
      
      * `FUNCTION LOAD` command was change and now it simply gets the library payload and extract the engine and name from the payload. In addition, the command will now return the function name which can later be used on `FUNCTION DELETE` and `FUNCTION LIST`.
      * The description field was completely removed from`FUNCTION LOAD`, and `FUNCTION LIST`
      
      
      ## Breaking Changes (compare to 7.0 rc2)
      
      * Library description was removed (we can re-add it in the future either as part of the shebang line or an additional line).
      * Loading an AOF file that was generated by either 7.0 rc1 or 7.0 rc2 will fail because the old command syntax is invalid.
      
      ## Notes
      
      * Loading an RDB file that was generated by rc1 / rc2 **is** supported, Redis will automatically add the shebang to the libraries payloads (we can probably delete that code after 7.0.3 or so since there's no need to keep supporting upgrades from an RC build).
      ae020e3d
  14. 21 Mar, 2022 1 commit
  15. 31 Jan, 2022 1 commit
  16. 25 Jan, 2022 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Added engine stats to FUNCTION STATS command. (#10179) · 5a38ccc2
      Meir Shpilraien (Spielrein) authored
      Added the following statistics (per engine) to FUNCTION STATS command:
      * number of functions
      * number of libraries
      
      Output example:
      ```
      > FUNCTION stats
      1) "running_script"
      2) (nil)
      3) "engines"
      4) 1) "LUA"
         2) 1) "libraries_count"
            2) (integer) 1
            3) "functions_count"
            4) (integer) 1
      ```
      
      To collect the stats, added a new dictionary to libraries_ctx that contains
      for each engine, the engine statistics representing the current libraries_ctx.
      Update the stats on:
      1. Link library to libraries_ctx
      2. Unlink library from libraries_ctx
      3. Flushing libraries_ctx
      5a38ccc2
  17. 24 Jan, 2022 1 commit
    • yoav-steinberg's avatar
      Support function flags in script EVAL via shebang header (#10126) · 7eadc5ee
      yoav-steinberg authored
      In #10025 we added a mechanism for flagging certain properties for Redis Functions.
      This lead us to think we'd like to "port" this mechanism to Redis Scripts (`EVAL`) as well. 
      
      One good reason for this, other than the added functionality is because it addresses the
      poor behavior we currently have in `EVAL` in case the script performs a (non DENY_OOM) write operation
      during OOM state. See #8478 (And a previous attempt to handle it via #10093) for details.
      Note that in Redis Functions **all** write operations (including DEL) will return an error during OOM state
      unless the function is flagged as `allow-oom` in which case no OOM checking is performed at all.
      
      This PR:
      - Enables setting `EVAL` (and `SCRIPT LOAD`) script flags as defined in #10025.
      - Provides a syntactical framework via [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) for
        additional script annotations and even engine selection (instead of just lua) for scripts.
      - Provides backwards compatibility so scripts without the new annotations will behave as they did before.
      - Appropriate tests.
      - Changes `EVAL[SHA]/_RO` to be flagged as `STALE` commands. This makes it possible to flag individual
        scripts as `allow-stale` or not flag them as such. In backwards compatibility mode these commands will
        return the `MASTERDOWN` error as before.
      - Changes `SCRIPT LOAD` to be flagged as a `STALE` command. This is mainly to make it logically
        compatible with the change to `EVAL` in the previous point. It enables loading a script on a stale server
        which is technically okay it doesn't relate directly to the server's dataset. Running the script does, but that
        won't work unless the script is explicitly marked as `allow-stale`.
      
      Note that even though the LUA syntax doesn't support hash tag comments `.lua` files do support a shebang
      tag on the top so they can be executed on Unix systems like any shell script. LUA's `luaL_loadfile` handles
      this as part of the LUA library. In the case of `luaL_loadbuffer`, which is what Redis uses, I needed to fix the
      input script in case of a shebang manually. I did this the same way `luaL_loadfile` does, by replacing the
      first line with a single line feed character.
      7eadc5ee
  18. 23 Jan, 2022 1 commit
    • Binbin's avatar
      sub-command support for ACL CAT and COMMAND LIST. redisCommand always stores fullname (#10127) · 23325c13
      Binbin authored
      
      
      Summary of changes:
      1. Rename `redisCommand->name` to `redisCommand->declared_name`, it is a
        const char * for native commands and SDS for module commands.
      2. Store the [sub]command fullname in `redisCommand->fullname` (sds).
      3. List subcommands in `ACL CAT`
      4. List subcommands in `COMMAND LIST`
      5. `moduleUnregisterCommands` now will also free the module subcommands.
      6. RM_GetCurrentCommandName returns full command name
      
      Other changes:
      1. Add `addReplyErrorArity` and `addReplyErrorExpireTime`
      2. Remove `getFullCommandName` function that now is useless.
      3. Some cleanups about `fullname` since now it is SDS.
      4. Delete `populateSingleCommand` function from server.h that is useless.
      5. Added tests to cover this change.
      6. Add some module unload tests and fix the leaks
      7. Make error messages uniform, make sure they always contain the full command
        name and that it's quoted.
      7. Fixes some typos
      
      see the history in #9504, fixes #10124
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarguybe7 <guy.benoish@redislabs.com>
      23325c13
  19. 20 Jan, 2022 1 commit
    • perryitay's avatar
      Adding module api for processing commands during busy jobs and allow flagging... · c4b78823
      perryitay authored
      
      Adding module api for processing commands during busy jobs and allow flagging the commands that should be handled at this status (#9963)
      
      Some modules might perform a long-running logic in different stages of Redis lifetime, for example:
      * command execution
      * RDB loading
      * thread safe context
      
      During this long-running logic Redis is not responsive.
      
      This PR offers 
      1. An API to process events while a busy command is running (`RM_Yield`)
      2. A new flag (`ALLOW_BUSY`) to mark the commands that should be handled during busy
        jobs which can also be used by modules (`allow-busy`)
      3. In slow commands and thread safe contexts, this flag will start rejecting commands with -BUSY only
        after `busy-reply-threshold`
      4. During loading (`rdb_load` callback), it'll process events right away (not wait for `busy-reply-threshold`),
        but either way, the processing is throttled to the server hz rate.
      5. Allow modules to Yield to redis background tasks, but not to client commands
      
      * rename `script-time-limit` to `busy-reply-threshold` (an alias to the pre-7.0 `lua-time-limit`)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      c4b78823
  20. 18 Jan, 2022 1 commit
  21. 14 Jan, 2022 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Function Flags support (no-writes, no-cluster, allow-state, allow-oom) (#10066) · 4db4b434
      Meir Shpilraien (Spielrein) authored
      # Redis Functions Flags
      
      Following the discussion on #10025 Added Functions Flags support.
      The PR is divided to 2 sections:
      * Add named argument support to `redis.register_function` API.
      * Add support for function flags
      
      ## `redis.register_function` named argument support
      
      The first part of the PR adds support for named argument on `redis.register_function`, example:
      ```
      redis.register_function{
          function_name='f1',
          callback=function()
              return 'hello'
          end,
          description='some desc'
      }
      ```
      
      The positional arguments is also kept, which means that it still possible to write:
      ```
      redis.register_function('f1', function() return 'hello' end)
      ```
      
      But notice that it is no longer possible to pass the optional description argument on the positional
      argument version. Positional argument was change to allow passing only the mandatory arguments
      (function name and callback). To pass more arguments the user must use the named argument version.
      
      As with positional arguments, the `function_name` and `callback` is mandatory and an error will be
      raise if those are missing. Also, an error will be raise if an unknown argument name is given or the
      arguments type is wrong.
      
      Tests was added to verify the new syntax.
      
      ## Functions Flags
      
      The second part of the PR is adding functions flags support.
      Flags are given to Redis when the engine calls `functionLibCreateFunction`, supported flags are:
      
      * `no-writes` - indicating the function perform no writes which means that it is OK to run it on:
         * read-only replica
         * Using FCALL_RO
         * If disk error detected
         
         It will not be possible to run a function in those situations unless the function turns on the `no-writes` flag
      
      * `allow-oom` - indicate that its OK to run the function even if Redis is in OOM state, if the function will
        not turn on this flag it will not be possible to run it if OOM reached (even if the function declares `no-writes`
        and even if `fcall_ro` is used). If this flag is set, any command will be allow on OOM (even those that is
        marked with CMD_DENYOOM). The assumption is that this flag is for advance users that knows its
        meaning and understand what they are doing, and Redis trust them to not increase the memory usage.
        (e.g. it could be an INCR or a modification on an existing key, or a DEL command)
      
      * `allow-state` - indicate that its OK to run the function on stale replica, in this case we will also make
        sure the function is only perform `stale` commands and raise an error if not.
      
      * `no-cluster` - indicate to disallow running the function if cluster is enabled.
      
      Default behaviure of functions (if no flags is given):
      1. Allow functions to read and write
      2. Do not run functions on OOM
      3. Do not run functions on stale replica
      4. Allow functions on cluster
      
      ### Lua API for functions flags
      
      On Lua engine, it is possible to give functions flags as `flags` named argument:
      
      ```
      redis.register_function{function_name='f1', callback=function() return 1 end, flags={'no-writes', 'allow-oom'}, description='description'}
      ```
      
      The function flags argument must be a Lua table that contains all the requested flags, The following
      will result in an error:
      * Unknown flag
      * Wrong flag type
      
      Default behaviour is the same as if no flags are used.
      
      Tests were added to verify all flags functionality
      
      ## Additional changes
      * mark FCALL and FCALL_RO with CMD_STALE flag (unlike EVAL), so that they can run if the function was
        registered with the `allow-stale` flag.
      * Verify `CMD_STALE` on `scriptCall` (`redis.call`), so it will not be possible to call commands from script while
        stale unless the command is marked with the `CMD_STALE` flags. so that even if the function is allowed while
        stale we do not allow it to bypass the `CMD_STALE` flag of commands.
      * Flags section was added to `FUNCTION LIST` command to provide the set of flags for each function:
      ```
      > FUNCTION list withcode
      1)  1) "library_name"
          2) "test"
          3) "engine"
          4) "LUA"
          5) "description"
          6) (nil)
          7) "functions"
          8) 1) 1) "name"
                2) "f1"
                3) "description"
                4) (nil)
                5) "flags"
                6) (empty array)
          9) "library_code"
         10) "redis.register_function{function_name='f1', callback=function() return 1 end}"
      ```
      * Added API to get Redis version from within a script, The redis version can be provided using:
         1. `redis.REDIS_VERSION` - string representation of the redis version in the format of MAJOR.MINOR.PATH
         2. `redis.REDIS_VERSION_NUM` - number representation of the redis version in the format of `0x00MMmmpp`
            (`MM` - major, `mm` - minor,  `pp` - patch). The number version can be used to check if version is greater or less 
            another version. The string version can be used to return to the user or print as logs.
      
         This new API is provided to eval scripts and functions, it also possible to use this API during functions loading phase.
      4db4b434
  22. 06 Jan, 2022 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Redis Function Libraries (#10004) · 885f6b5c
      Meir Shpilraien (Spielrein) authored
      # Redis Function Libraries
      
      This PR implements Redis Functions Libraries as describe on: https://github.com/redis/redis/issues/9906.
      
      Libraries purpose is to provide a better code sharing between functions by allowing to create multiple
      functions in a single command. Functions that were created together can safely share code between
      each other without worrying about compatibility issues and versioning.
      
      Creating a new library is done using 'FUNCTION LOAD' command (full API is described below)
      
      This PR introduces a new struct called libraryInfo, libraryInfo holds information about a library:
      * name - name of the library
      * engine - engine used to create the library
      * code - library code
      * description - library description
      * functions - the functions exposed by the library
      
      When Redis gets the `FUNCTION LOAD` command it creates a new empty libraryInfo.
      Redis passes the `CODE` to the relevant engine alongside the empty libraryInfo.
      As a result, the engine will create one or more functions by calling 'libraryCreateFunction'.
      The new funcion will be added to the newly created libraryInfo. So far Everything is happening
      locally on the libraryInfo so it is easy to abort the operation (in case of an error) by simply
      freeing the libraryInfo. After the library info is fully constructed we start the joining phase by
      which we will join the new library to the other libraries currently exist on Redis.
      The joining phase make sure there is no function collision and add the library to the
      librariesCtx (renamed from functionCtx). LibrariesCtx is used all around the code in the exact
      same way as functionCtx was used (with respect to RDB loading, replicatio, ...).
      The only difference is that apart from function dictionary (maps function name to functionInfo
      object), the librariesCtx contains also a libraries dictionary that maps library name to libraryInfo object.
      
      ## New API
      ### FUNCTION LOAD
      `FUNCTION LOAD <ENGINE> <LIBRARY NAME> [REPLACE] [DESCRIPTION <DESCRIPTION>] <CODE>`
      Create a new library with the given parameters:
      * ENGINE - REPLACE Engine name to use to create the library.
      * LIBRARY NAME - The new library name.
      * REPLACE - If the library already exists, replace it.
      * DESCRIPTION - Library description.
      * CODE - Library code.
      
      Return "OK" on success, or error on the following cases:
      * Library name already taken and REPLACE was not used
      * Name collision with another existing library (even if replace was uses)
      * Library registration failed by the engine (usually compilation error)
      
      ## Changed API
      ### FUNCTION LIST
      `FUNCTION LIST [LIBRARYNAME <LIBRARY NAME PATTERN>] [WITHCODE]`
      Command was modified to also allow getting libraries code (so `FUNCTION INFO` command is no longer
      needed and removed). In addition the command gets an option argument, `LIBRARYNAME` allows you to
      only get libraries that match the given `LIBRARYNAME` pattern. By default, it returns all libraries.
      
      ### INFO MEMORY
      Added number of libraries to `INFO MEMORY`
      
      ### Commands flags
      `DENYOOM` flag was set on `FUNCTION LOAD` and `FUNCTION RESTORE`. We consider those commands
      as commands that add new data to the dateset (functions are data) and so we want to disallows
      to run those commands on OOM.
      
      ## Removed API
      * FUNCTION CREATE - Decided on https://github.com/redis/redis/issues/9906
      * FUNCTION INFO - Decided on https://github.com/redis/redis/issues/9899
      
      ## Lua engine changes
      When the Lua engine gets the code given on `FUNCTION LOAD` command, it immediately runs it, we call
      this run the loading run. Loading run is not a usual script run, it is not possible to invoke any
      Redis command from within the load run.
      Instead there is a new API provided by `library` object. The new API's: 
      * `redis.log` - behave the same as `redis.log`
      * `redis.register_function` - register a new function to the library
      
      The loading run purpose is to register functions using the new `redis.register_function` API.
      Any attempt to use any other API will result in an error. In addition, the load run is has a time
      limit of 500ms, error is raise on timeout and the entire operation is aborted.
      
      ### `redis.register_function`
      `redis.register_function(<function_name>, <callback>, [<description>])`
      This new API allows users to register a new function that will be linked to the newly created library.
      This API can only be called during the load run (see definition above). Any attempt to use it outside
      of the load run will result in an error.
      The parameters pass to the API are:
      * function_name - Function name (must be a Lua string)
      * callback - Lua function object that will be called when the function is invokes using fcall/fcall_ro
      * description - Function description, optional (must be a Lua string).
      
      ### Example
      The following example creates a library called `lib` with 2 functions, `f1` and `f1`, returns 1 and 2 respectively:
      ```
      local function f1(keys, args)
          return 1
      end
      
      local function f2(keys, args)
          return 2
      end
      
      redis.register_function('f1', f1)
      redis.register_function('f2', f2)
      ```
      
      Notice: Unlike `eval`, functions inside a library get the KEYS and ARGV as arguments to the
      functions and not as global.
      
      ### Technical Details
      
      On the load run we only want the user to be able to call a white list on API's. This way, in
      the future, if new API's will be added, the new API's will not be available to the load run
      unless specifically added to this white list. We put the while list on the `library` object and
      make sure the `library` object is only available to the load run by using [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv) API. This API allows us to set
      the `globals` of a function (and all the function it creates). Before starting the load run we
      create a new fresh Lua table (call it `g`) that only contains the `library` API (we make sure
      to set global protection on this table just like the general global protection already exists
      today), then we use [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv)
      to set `g` as the global table of the load run. After the load run finished we update `g`
      metatable and set `__index` and `__newindex` functions to be `_G` (Lua default globals),
      we also pop out the `library` object as we do not need it anymore.
      This way, any function that was created on the load run (and will be invoke using `fcall`) will
      see the default globals as it expected to see them and will not have the `library` API anymore.
      
      An important outcome of this new approach is that now we can achieve a distinct global table
      for each library (it is not yet like that but it is very easy to achieve it now). In the future we can
      decide to remove global protection because global on different libraries will not collide or we
      can chose to give different API to different libraries base on some configuration or input.
      
      Notice that this technique was meant to prevent errors and was not meant to prevent malicious
      user from exploit it. For example, the load run can still save the `library` object on some local
      variable and then using in `fcall` context. To prevent such a malicious use, the C code also make
      sure it is running in the right context and if not raise an error.
      885f6b5c
  23. 26 Dec, 2021 2 commits
    • Meir Shpilraien (Spielrein)'s avatar
      Add FUNCTION DUMP and RESTORE. (#9938) · 365cbf46
      Meir Shpilraien (Spielrein) authored
      Follow the conclusions to support Functions in redis cluster (#9899)
      
      Added 2 new FUNCTION sub-commands:
      1. `FUNCTION DUMP` - dump a binary payload representation of all the functions.
      2. `FUNCTION RESTORE <PAYLOAD> [FLUSH|APPEND|REPLACE]` - give the binary payload extracted
         using `FUNCTION DUMP`, restore all the functions on the given payload. Restore policy can be given to
         control how to handle existing functions (default is APPEND):
         * FLUSH: delete all existing functions.
         * APPEND: appends the restored functions to the existing functions. On collision, abort.
         * REPLACE: appends the restored functions to the existing functions. On collision,
           replace the old function with the new function.
      
      Modify `redis-cli --cluster add-node` to use `FUNCTION DUMP` to get existing functions from
      one of the nodes in the cluster, and `FUNCTION RESTORE` to load the same set of functions
      to the new node. `redis-cli` will execute this step before sending the `CLUSTER MEET` command
      to the new node. If `FUNCTION DUMP` returns an error, assume the current Redis version do not
      support functions and skip `FUNCTION RESTORE`. If `FUNCTION RESTORE` fails, abort and do not send
      the `CLUSTER MEET` command. If the new node already contains functions (before the `FUNCTION RESTORE`
      is sent), abort and do not add the node to the cluster. Test was added to verify
      `redis-cli --cluster add-node` works as expected. 
      365cbf46
    • Meir Shpilraien (Spielrein)'s avatar
      Changed fuction name to be case insensitive. (#9984) · 08ff606b
      Meir Shpilraien (Spielrein) authored
      Use case insensitive string comparison for function names (like we do for commands and configs)
      In addition, add verification that the functions only use the following characters: [a-zA-Z0-9_]
      08ff606b
  24. 21 Dec, 2021 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Change FUNCTION CREATE, DELETE and FLUSH to be WRITE commands instead of MAY_REPLICATE. (#9953) · 3bcf1084
      Meir Shpilraien (Spielrein) authored
      The issue with MAY_REPLICATE is that all automatic mechanisms to handle
      write commands will not work. This require have a special treatment for:
      * Not allow those commands to be executed on RO replica.
      * Allow those commands to be executed on RO replica from primary connection.
      * Allow those commands to be executed on the RO replica from AOF.
      
      By setting those commands as WRITE commands we are getting all those properties from Redis.
      Test was added to verify that those properties work as expected.
      
      In addition, rearrange when and where functions are flushed. Before this PR functions were
      flushed manually on `rdbLoadRio` and cleaned manually on failure. This contradicts the
      assumptions that functions are data and need to be created/deleted alongside with the
      data. A side effect of this, for example, `debug reload noflush` did not flush the data but
      did flush the functions, `debug loadaof` flush the data but not the functions.
      This PR move functions deletion into `emptyDb`. `emptyDb` (renamed to `emptyData`) will
      now accept an additional flag, `NOFUNCTIONS` which specifically indicate that we do not
      want to flush the functions (on all other cases, functions will be flushed). Used the new flag
      on FLUSHALL and FLUSHDB only! Tests were added to `debug reload` and `debug loadaof`
      to verify that functions behave the same as the data.
      
      Notice that because now functions will be deleted along side with the data we can not allow
      `CLUSTER RESET` to be called from within a function (it will cause the function to be released
      while running), this PR adds `NO_SCRIPT` flag to `CLUSTER RESET`  so it will not be possible
      to be called from within a function. The other cluster commands are allowed from within a
      function (there are use-cases that uses `GETKEYSINSLOT` to iterate over all the keys on a
      given slot). Tests was added to verify `CLUSTER RESET` is denied from within a script.
      
      Another small change on this PR is that `RDBFLAGS_ALLOW_DUP` is also applicable on functions.
      When loading functions, if this flag is set, we will replace old functions with new ones on collisions. 
      3bcf1084
  25. 19 Dec, 2021 1 commit
    • Oran Agra's avatar
      Add external test that runs without debug command (#9964) · 6add1b72
      Oran Agra authored
      - add needs:debug flag for some tests
      - disable "save" in external tests (speedup?)
      - use debug_digest proc instead of debug command directly so it can be skipped
      - use OBJECT ENCODING instead of DEBUG OBJECT to get encoding
      - add a proc for OBJECT REFCOUNT so it can be skipped
      - move a bunch of tests in latency_monitor tests to happen later so that latency monitor has some values in it
      - add missing close_replication_stream calls
      - make sure to close the temp client if DEBUG LOG fails
      6add1b72
  26. 16 Dec, 2021 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Add FUNCTION FLUSH command to flush all functions (#9936) · 687210f1
      Meir Shpilraien (Spielrein) authored
      Added `FUNCTION FLUSH` command. The new sub-command allows delete all the functions.
      An optional `[SYNC|ASYNC]` argument can be given to control whether or not to flush the
      functions synchronously or asynchronously. if not given the default flush mode is chosen by
      `lazyfree-lazy-user-flush` configuration values.
      
      Add the missing `functions.tcl` test to the list of tests that are executed in test_helper.tcl,
      and call FUNCTION FLUSH in between servers in external mode
      687210f1
  27. 02 Dec, 2021 1 commit
    • meir@redislabs.com's avatar
      Redis Functions - Added redis function unit and Lua engine · cbd46317
      meir@redislabs.com authored
      Redis function unit is located inside functions.c
      and contains Redis Function implementation:
      1. FUNCTION commands:
        * FUNCTION CREATE
        * FCALL
        * FCALL_RO
        * FUNCTION DELETE
        * FUNCTION KILL
        * FUNCTION INFO
      2. Register engine
      
      In addition, this commit introduce the first engine
      that uses the Redis Function capabilities, the
      Lua engine.
      cbd46317