FFI Tutorial

+This page is intended to give you an overview of the features of the FFI +library by presenting a few use cases and guidelines. +

+This page makes no attempt to explain all of the FFI library, though. +You'll want to have a look at the ffi.* API +function reference and the FFI +semantics to learn more. +

+ +

Loading the FFI Library

+The FFI library is built into LuaJIT by default, but it's not loaded +and initialized by default. The suggested way to use the FFI library +is to add the following to the start of every Lua file that needs one +of its functions: +

+local ffi = require("ffi")
+

+Please note, this doesn't define an ffi variable in the table +of globals — you really need to use the local variable. The +require function ensures the library is only loaded once. +

+Note: If you want to experiment with the FFI from the interactive prompt +of the command line executable, omit the local, as it doesn't +preserve local variables across lines. +

+ +

Accessing Standard System Functions

+The following code explains how to access standard system functions. +We slowly print two lines of dots by sleeping for 10 milliseconds +after each dot: +

+ 
+①
+
+
+
+
+
+②
+③
+④
+
+
+
+⑤
+
+
+
+
+
+⑥local ffi = require("ffi")
+ffi.cdef[[
+void Sleep(int ms);
+int poll(struct pollfd *fds, unsigned long nfds, int timeout);
+]]
+
+local sleep
+if ffi.os == "Windows" then
+  function sleep(s)
+    ffi.C.Sleep(s*1000)
+  end
+else
+  function sleep(s)
+    ffi.C.poll(nil, 0, s*1000)
+  end
+end
+
+for i=1,160 do
+  io.write("."); io.flush()
+  sleep(0.01)
+end
+io.write("\n")
+

+Here's the step-by-step explanation: +

+① This defines the +C library functions we're going to use. The part inside the +double-brackets (in green) is just standard C syntax. You can +usually get this info from the C header files or the +documentation provided by each C library or C compiler. +

+② The difficulty we're +facing here, is that there are different standards to choose from. +Windows has a simple Sleep() function. On other systems there +are a variety of functions available to achieve sub-second sleeps, but +with no clear consensus. Thankfully poll() can be used for +this task, too, and it's present on most non-Windows systems. The +check for ffi.os makes sure we use the Windows-specific +function only on Windows systems. +

+③ Here we're wrapping the +call to the C function in a Lua function. This isn't strictly +necessary, but it's helpful to deal with system-specific issues only +in one part of the code. The way we're wrapping it ensures the check +for the OS is only done during initialization and not for every call. +

+④ A more subtle point is +that we defined our sleep() function (for the sake of this +example) as taking the number of seconds, but accepting fractional +seconds. Multiplying this by 1000 gets us milliseconds, but that still +leaves it a Lua number, which is a floating-point value. Alas, the +Sleep() function only accepts an integer value. Luckily for +us, the FFI library automatically performs the conversion when calling +the function (truncating the FP value towards zero, like in C). +

+Some readers will notice that Sleep() is part of +KERNEL32.DLL and is also a stdcall function. So how +can this possibly work? The FFI library provides the ffi.C +default C library namespace, which allows calling functions from +the default set of libraries, like a C compiler would. Also, the +FFI library automatically detects stdcall functions, so you +don't need to declare them as such. +

+⑤ The poll() +function takes a couple more arguments we're not going to use. You can +simply use nil to pass a NULL pointer and 0 +for the nfds parameter. Please note, that the +number 0 does not convert to a pointer value, +unlike in C++. You really have to pass pointers to pointer arguments +and numbers to number arguments. +

+The page on FFI semantics has all +of the gory details about +conversions between Lua +objects and C types. For the most part you don't have to deal +with this, as it's performed automatically and it's carefully designed +to bridge the semantic differences between Lua and C. +

+⑥ Now that we have defined +our own sleep() function, we can just call it from plain Lua +code. That wasn't so bad, huh? Turning these boring animated dots into +a fascinating best-selling game is left as an exercise for the reader. +:-) +

+ +

Accessing the zlib Compression Library

+The following code shows how to access the » zlib compression library from Lua code. +We'll define two convenience wrapper functions that take a string and +compress or uncompress it to another string: +

+ 
+①
+
+
+
+
+
+
+②
+
+
+③
+
+④
+
+
+⑤
+
+
+⑥
+
+
+
+
+
+
+
+⑦local ffi = require("ffi")
+ffi.cdef[[
+unsigned long compressBound(unsigned long sourceLen);
+int compress2(uint8_t *dest, unsigned long *destLen,
+	      const uint8_t *source, unsigned long sourceLen, int level);
+int uncompress(uint8_t *dest, unsigned long *destLen,
+	       const uint8_t *source, unsigned long sourceLen);
+]]
+local zlib = ffi.load(ffi.os == "Windows" and "zlib1" or "z")
+
+local function compress(txt)
+  local n = zlib.compressBound(#txt)
+  local buf = ffi.new("uint8_t[?]", n)
+  local buflen = ffi.new("unsigned long[1]", n)
+  local res = zlib.compress2(buf, buflen, txt, #txt, 9)
+  assert(res == 0)
+  return ffi.string(buf, buflen[0])
+end
+
+local function uncompress(comp, n)
+  local buf = ffi.new("uint8_t[?]", n)
+  local buflen = ffi.new("unsigned long[1]", n)
+  local res = zlib.uncompress(buf, buflen, comp, #comp)
+  assert(res == 0)
+  return ffi.string(buf, buflen[0])
+end
+
+-- Simple test code.
+local txt = string.rep("abcd", 1000)
+print("Uncompressed size: ", #txt)
+local c = compress(txt)
+print("Compressed size: ", #c)
+local txt2 = uncompress(c, #txt)
+assert(txt2 == txt)
+

+Here's the step-by-step explanation: +

+① This defines some of the +C functions provided by zlib. For the sake of this example, some +type indirections have been reduced and it uses the predefined +fixed-size integer types, while still adhering to the zlib API/ABI. +

+② This loads the zlib shared +library. On POSIX systems, it's named libz.so and usually +comes pre-installed. Since ffi.load() automatically adds any +missing standard prefixes/suffixes, we can simply load the +"z" library. On Windows it's named zlib1.dll and +you'll have to download it first from the +» zlib site. The check for +ffi.os makes sure we pass the right name to +ffi.load(). +

+③ First, the maximum size of +the compression buffer is obtained by calling the +zlib.compressBound function with the length of the +uncompressed string. The next line allocates a byte buffer of this +size. The [?] in the type specification indicates a +variable-length array (VLA). The actual number of elements of this +array is given as the 2nd argument to ffi.new(). +

+④ This may look strange at +first, but have a look at the declaration of the compress2 +function from zlib: the destination length is defined as a pointer! +This is because you pass in the maximum buffer size and get back the +actual length that was used. +

+In C you'd pass in the address of a local variable +(&buflen). But since there's no address-of operator in +Lua, we'll just pass in a one-element array. Conveniently, it can be +initialized with the maximum buffer size in one step. Calling the +actual zlib.compress2 function is then straightforward. +

+⑤ We want to return the +compressed data as a Lua string, so we'll use ffi.string(). +It needs a pointer to the start of the data and the actual length. The +length has been returned in the buflen array, so we'll just +get it from there. +

+Note that since the function returns now, the buf and +buflen variables will eventually be garbage collected. This +is fine, because ffi.string() has copied the contents to a +newly created (interned) Lua string. If you plan to call this function +lots of times, consider reusing the buffers and/or handing back the +results in buffers instead of strings. This will reduce the overhead +for garbage collection and string interning. +

+⑥ The uncompress +functions does the exact opposite of the compress function. +The compressed data doesn't include the size of the original string, +so this needs to be passed in. Otherwise, no surprises here. +

+⑦ The code, that makes use +of the functions we just defined, is just plain Lua code. It doesn't +need to know anything about the LuaJIT FFI — the convenience +wrapper functions completely hide it. +

+One major advantage of the LuaJIT FFI is that you are now able to +write those wrappers in Lua. And at a fraction of the time it +would cost you to create an extra C module using the Lua/C API. +Many of the simpler C functions can probably be used directly +from your Lua code, without any wrappers. +

+Side note: the zlib API uses the long type for passing +lengths and sizes around. But all those zlib functions actually only +deal with 32 bit values. This is an unfortunate choice for a +public API, but may be explained by zlib's history — we'll just +have to deal with it. +

+First, you should know that a long is a 64 bit type e.g. +on POSIX/x64 systems, but a 32 bit type on Windows/x64 and on +32 bit systems. Thus a long result can be either a plain +Lua number or a boxed 64 bit integer cdata object, depending on +the target system. +

+Ok, so the ffi.* functions generally accept cdata objects +wherever you'd want to use a number. That's why we get a away with +passing n to ffi.string() above. But other Lua +library functions or modules don't know how to deal with this. So for +maximum portability, one needs to use tonumber() on returned +long results before passing them on. Otherwise the +application might work on some systems, but would fail in a POSIX/x64 +environment. +

+ +

Defining Metamethods for a C Type

+The following code explains how to define metamethods for a C type. +We define a simple point type and add some operations to it: +

+ 
+①
+
+
+
+②
+
+③
+
+④
+
+
+
+⑤
+
+⑥local ffi = require("ffi")
+ffi.cdef[[
+typedef struct { double x, y; } point_t;
+]]
+
+local point
+local mt = {
+  __add = function(a, b) return point(a.x+b.x, a.y+b.y) end,
+  __len = function(a) return math.sqrt(a.x*a.x + a.y*a.y) end,
+  __index = {
+    area = function(a) return a.x*a.x + a.y*a.y end,
+  },
+}
+point = ffi.metatype("point_t", mt)
+
+local a = point(3, 4)
+print(a.x, a.y)  --> 3  4
+print(#a)        --> 5
+print(a:area())  --> 25
+local b = a + point(0.5, 8)
+print(#b)        --> 12.5
+

+Here's the step-by-step explanation: +

+① This defines the C type for a +two-dimensional point object. +

+② We have to declare the variable +holding the point constructor first, because it's used inside of a +metamethod. +

+③ Let's define an __add +metamethod which adds the coordinates of two points and creates a new +point object. For simplicity, this function assumes that both arguments +are points. But it could be any mix of objects, if at least one operand +is of the required type (e.g. adding a point plus a number or vice +versa). Our __len metamethod returns the distance of a point to +the origin. +

+④ If we run out of operators, we can +define named methods, too. Here, the __index table defines an +area function. For custom indexing needs, one might want to +define __index and __newindex functions instead. +

+⑤ This associates the metamethods with +our C type. This only needs to be done once. For convenience, a +constructor is returned by +ffi.metatype(). +We're not required to use it, though. The original C type can still +be used e.g. to create an array of points. The metamethods automatically +apply to any and all uses of this type. +

+Please note, that the association with a metatable is permanent and +the metatable must not be modified afterwards! Ditto for the +__index table. +

+⑥ Here are some simple usage examples +for the point type and their expected results. The predefined +operations (such as a.x) can be freely mixed with the newly +defined metamethods. Note that area is a method and must be +called with the Lua syntax for methods: a:area(), not +a.area(). +

+The C type metamethod mechanism is most useful when used in +conjunction with C libraries that are written in an object-oriented +style. Creators return a pointer to a new instance, and methods take an +instance pointer as the first argument. Sometimes you can just point +__index to the library namespace and __gc to the +destructor and you're done. But often enough you'll want to add +convenience wrappers, e.g. to return actual Lua strings or when +returning multiple values. +

+Some C libraries only declare instance pointers as an opaque +void * type. In this case you can use a fake type for all +declarations, e.g. a pointer to a named (incomplete) struct will do: +typedef struct foo_type *foo_handle. The C side doesn't +know what you declare with the LuaJIT FFI, but as long as the underlying +types are compatible, everything still works. +

+ +

Translating C Idioms

+Here's a list of common C idioms and their translation to the +LuaJIT FFI: +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Idiom	C code	Lua code
Pointer dereference `int *p;`	`x = p; p = y;`	`x = p[0] p[0] = y`
Pointer indexing `int i, *p;`	`x = p[i]; p[i+1] = y;`	`x = p[i] p[i+1] = y`
Array indexing `int i, a[];`	`x = a[i]; a[i+1] = y;`	`x = a[i] a[i+1] = y`
`struct`/`union` dereference `struct foo s;`	`x = s.field; s.field = y;`	`x = s.field s.field = y`
`struct`/`union` pointer deref. `struct foo *sp;`	`x = sp->field; sp->field = y;`	`x = s.field s.field = y`
Pointer arithmetic `int i, *p;`	`x = p + i; y = p - i;`	`x = p + i y = p - i`
Pointer difference `int p1, p2;`	`x = p1 - p2;`	`x = p1 - p2`
Array element pointer `int i, a[];`	`x = &a[i];`	`x = a+i`
Cast pointer to address `int *p;`	`x = (intptr_t)p;`	`x = tonumber( ffi.cast("intptr_t", p))`
Functions with outargs `void foo(int *inoutlen);`	`int len = x; foo(&len); y = len;`	`local len = ffi.new("int[1]", x) foo(len) y = len[0]`
Vararg conversions `int printf(char *fmt, ...);`	`printf("%g", 1.0); printf("%d", 1);`	`printf("%g", 1); printf("%d", ffi.new("int", 1))`

+ +

To Cache or Not to Cache

+It's a common Lua idiom to cache library functions in local variables +or upvalues, e.g.: +

+local byte, char = string.byte, string.char
+local function foo(x)
+  return char(byte(x)+1)
+end
+

+This replaces several hash-table lookups with a (faster) direct use of +a local or an upvalue. This is less important with LuaJIT, since the +JIT compiler optimizes hash-table lookups a lot and is even able to +hoist most of them out of the inner loops. It can't eliminate +all of them, though, and it saves some typing for often-used +functions. So there's still a place for this, even with LuaJIT. +

+The situation is a bit different with C function calls via the +FFI library. The JIT compiler has special logic to eliminate all +of the lookup overhead for functions resolved from a +C library namespace! +Thus it's not helpful and actually counter-productive to cache +individual C functions like this: +

+local funca, funcb = ffi.C.funca, ffi.C.funcb -- Not helpful!
+local function foo(x, n)
+  for i=1,n do funcb(funca(x, i), 1) end
+end
+

+This turns them into indirect calls and generates bigger and slower +machine code. Instead, you'll want to cache the namespace itself and +rely on the JIT compiler to eliminate the lookups: +

+local C = ffi.C          -- Instead use this!
+local function foo(x, n)
+  for i=1,n do C.funcb(C.funca(x, i), 1) end
+end
+

+This generates both shorter and faster code. So don't cache +C functions, but do cache namespaces! Most often the +namespace is already in a local variable at an outer scope, e.g. from +local lib = ffi.load(...). Note that copying +it to a local variable in the function scope is unnecessary. +

+
+