1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
|
<!DOCTYPE html>
<html>
<head>
<title>String Buffer Library</title>
<meta charset="utf-8">
<meta name="Copyright" content="Copyright (C) 2005-2022">
<meta name="Language" content="en">
<link rel="stylesheet" type="text/css" href="bluequad.css" media="screen">
<link rel="stylesheet" type="text/css" href="bluequad-print.css" media="print">
<style type="text/css">
.lib {
vertical-align: middle;
margin-left: 5px;
padding: 0 5px;
font-size: 60%;
border-radius: 5px;
background: #c5d5ff;
color: #000;
}
</style>
</head>
<body>
<div id="site">
<a href="https://luajit.org"><span>Lua<span id="logo">JIT</span></span></a>
</div>
<div id="head">
<h1>String Buffer Library</h1>
</div>
<div id="nav">
<ul><li>
<a href="luajit.html">LuaJIT</a>
<ul><li>
<a href="https://luajit.org/download.html">Download <span class="ext">»</span></a>
</li><li>
<a href="install.html">Installation</a>
</li><li>
<a href="running.html">Running</a>
</li></ul>
</li><li>
<a href="extensions.html">Extensions</a>
<ul><li>
<a href="ext_ffi.html">FFI Library</a>
<ul><li>
<a href="ext_ffi_tutorial.html">FFI Tutorial</a>
</li><li>
<a href="ext_ffi_api.html">ffi.* API</a>
</li><li>
<a href="ext_ffi_semantics.html">FFI Semantics</a>
</li></ul>
</li><li>
<a class="current" href="ext_buffer.html">String Buffers</a>
</li><li>
<a href="ext_jit.html">jit.* Library</a>
</li><li>
<a href="ext_c_api.html">Lua/C API</a>
</li><li>
<a href="ext_profiler.html">Profiler</a>
</li></ul>
</li><li>
<a href="status.html">Status</a>
</li><li>
<a href="faq.html">FAQ</a>
</li><li>
<a href="https://luajit.org/list.html">Mailing List <span class="ext">»</span></a>
</li></ul>
</div>
<div id="main">
<p>
The string buffer library allows <b>high-performance manipulation of
string-like data</b>.
</p>
<p>
Unlike Lua strings, which are constants, string buffers are
<b>mutable</b> sequences of 8-bit (binary-transparent) characters. Data
can be stored, formatted and encoded into a string buffer and later
converted, extracted or decoded.
</p>
<p>
The convenient string buffer API simplifies common string manipulation
tasks, that would otherwise require creating many intermediate strings.
String buffers improve performance by eliminating redundant memory
copies, object creation, string interning and garbage collection
overhead. In conjunction with the FFI library, they allow zero-copy
operations.
</p>
<p>
The string buffer library also includes a high-performance
<a href="serialize">serializer</a> for Lua objects.
</p>
<h2 id="wip" style="color:#ff0000">Work in Progress</h2>
<p>
<b style="color:#ff0000">This library is a work in progress. More
functionality will be added soon.</b>
</p>
<h2 id="use">Using the String Buffer Library</h2>
<p>
The string buffer library is built into LuaJIT by default, but it's not
loaded by default. Add this to the start of every Lua file that needs
one of its functions:
</p>
<pre class="code">
local buffer = require("string.buffer")
</pre>
<p>
The convention for the syntax shown on this page is that <tt>buffer</tt>
refers to the buffer library and <tt>buf</tt> refers to an individual
buffer object.
</p>
<p>
Please note the difference between a Lua function call, e.g.
<tt>buffer.new()</tt> (with a dot) and a Lua method call, e.g.
<tt>buf:reset()</tt> (with a colon).
</p>
<h3 id="buffer_object">Buffer Objects</h3>
<p>
A buffer object is a garbage-collected Lua object. After creation with
<tt>buffer.new()</tt>, it can (and should) be reused for many operations.
When the last reference to a buffer object is gone, it will eventually
be freed by the garbage collector, along with the allocated buffer
space.
</p>
<p>
Buffers operate like a FIFO (first-in first-out) data structure. Data
can be appended (written) to the end of the buffer and consumed (read)
from the front of the buffer. These operations may be freely mixed.
</p>
<p>
The buffer space that holds the characters is managed automatically
— it grows as needed and already consumed space is recycled. Use
<tt>buffer.new(size)</tt> and <tt>buf:free()</tt>, if you need more
control.
</p>
<p>
The maximum size of a single buffer is the same as the maximum size of a
Lua string, which is slightly below two gigabytes. For huge data sizes,
neither strings nor buffers are the right data structure — use the
FFI library to directly map memory or files up to the virtual memory
limit of your OS.
</p>
<h3 id="buffer_overview">Buffer Method Overview</h3>
<ul>
<li>
The <tt>buf:put*()</tt>-like methods append (write) characters to the
end of the buffer.
</li>
<li>
The <tt>buf:get*()</tt>-like methods consume (read) characters from the
front of the buffer.
</li>
<li>
Other methods, like <tt>buf:tostring()</tt> only read the buffer
contents, but don't change the buffer.
</li>
<li>
The <tt>buf:set()</tt> method allows zero-copy consumption of a string
or an FFI cdata object as a buffer.
</li>
<li>
The FFI-specific methods allow zero-copy read/write-style operations or
modifying the buffer contents in-place. Please check the
<a href="#ffi_caveats">FFI caveats</a> below, too.
</li>
<li>
Methods that don't need to return anything specific, return the buffer
object itself as a convenience. This allows method chaining, e.g.:
<tt>buf:reset():encode(obj)</tt> or <tt>buf:skip(len):get()</tt>
</li>
</ul>
<h2 id="create">Buffer Creation and Management</h2>
<h3 id="buffer_new"><tt>local buf = buffer.new([size [,options]])<br>
local buf = buffer.new([options])</tt></h3>
<p>
Creates a new buffer object.
</p>
<p>
The optional <tt>size</tt> argument ensures a minimum initial buffer
size. This is strictly an optimization when the required buffer size is
known beforehand. The buffer space will grow as needed, in any case.
</p>
<p>
The optional table <tt>options</tt> sets various
<a href="#serialize_options">serialization options</a>.
</p>
<h3 id="buffer_reset"><tt>buf = buf:reset()</tt></h3>
<p>
Reset (empty) the buffer. The allocated buffer space is not freed and
may be reused.
</p>
<h3 id="buffer_free"><tt>buf = buf:free()</tt></h3>
<p>
The buffer space of the buffer object is freed. The object itself
remains intact, empty and may be reused.
</p>
<p>
Note: you normally don't need to use this method. The garbage collector
automatically frees the buffer space, when the buffer object is
collected. Use this method, if you need to free the associated memory
immediately.
</p>
<h2 id="write">Buffer Writers</h2>
<h3 id="buffer_put"><tt>buf = buf:put([str|num|obj] [,…])</tt></h3>
<p>
Appends a string <tt>str</tt>, a number <tt>num</tt> or any object
<tt>obj</tt> with a <tt>__tostring</tt> metamethod to the buffer.
Multiple arguments are appended in the given order.
</p>
<p>
Appending a buffer to a buffer is possible and short-circuited
internally. But it still involves a copy. Better combine the buffer
writes to use a single buffer.
</p>
<h3 id="buffer_putf"><tt>buf = buf:putf(format, …)</tt></h3>
<p>
Appends the formatted arguments to the buffer. The <tt>format</tt>
string supports the same options as <tt>string.format()</tt>.
</p>
<h3 id="buffer_putcdata"><tt>buf = buf:putcdata(cdata, len)</tt><span class="lib">FFI</span></h3>
<p>
Appends the given <tt>len</tt> number of bytes from the memory pointed
to by the FFI <tt>cdata</tt> object to the buffer. The object needs to
be convertible to a (constant) pointer.
</p>
<h3 id="buffer_set"><tt>buf = buf:set(str)<br>
buf = buf:set(cdata, len)</tt><span class="lib">FFI</span></h3>
<p>
This method allows zero-copy consumption of a string or an FFI cdata
object as a buffer. It stores a reference to the passed string
<tt>str</tt> or the FFI <tt>cdata</tt> object in the buffer. Any buffer
space originally allocated is freed. This is <i>not</i> an append
operation, unlike the <tt>buf:put*()</tt> methods.
</p>
<p>
After calling this method, the buffer behaves as if
<tt>buf:free():put(str)</tt> or <tt>buf:free():put(cdata, len)</tt>
had been called. However, the data is only referenced and not copied, as
long as the buffer is only consumed.
</p>
<p>
In case the buffer is written to later on, the referenced data is copied
and the object reference is removed (copy-on-write semantics).
</p>
<p>
The stored reference is an anchor for the garbage collector and keeps the
originally passed string or FFI cdata object alive.
</p>
<h3 id="buffer_reserve"><tt>ptr, len = buf:reserve(size)</tt><span class="lib">FFI</span><br>
<tt>buf = buf:commit(used)</tt><span class="lib">FFI</span></h3>
<p>
The <tt>reserve</tt> method reserves at least <tt>size</tt> bytes of
write space in the buffer. It returns an <tt>uint8_t *</tt> FFI
cdata pointer <tt>ptr</tt> that points to this space.
</p>
<p>
The available length in bytes is returned in <tt>len</tt>. This is at
least <tt>size</tt> bytes, but may be more to facilitate efficient
buffer growth. You can either make use of the additional space or ignore
<tt>len</tt> and only use <tt>size</tt> bytes.
</p>
<p>
The <tt>commit</tt> method appends the <tt>used</tt> bytes of the
previously returned write space to the buffer data.
</p>
<p>
This pair of methods allows zero-copy use of C read-style APIs:
</p>
<pre class="code">
local MIN_SIZE = 65536
repeat
local ptr, len = buf:reserve(MIN_SIZE)
local n = C.read(fd, ptr, len)
if n == 0 then break end -- EOF.
if n < 0 then error("read error") end
buf:commit(n)
until false
</pre>
<p>
The reserved write space is <i>not</i> initialized. At least the
<tt>used</tt> bytes <b>must</b> be written to before calling the
<tt>commit</tt> method. There's no need to call the <tt>commit</tt>
method, if nothing is added to the buffer (e.g. on error).
</p>
<h2 id="read">Buffer Readers</h2>
<h3 id="buffer_length"><tt>len = #buf</tt></h3>
<p>
Returns the current length of the buffer data in bytes.
</p>
<h3 id="buffer_concat"><tt>res = str|num|buf .. str|num|buf […]</tt></h3>
<p>
The Lua concatenation operator <tt>..</tt> also accepts buffers, just
like strings or numbers. It always returns a string and not a buffer.
</p>
<p>
Note that although this is supported for convenience, this thwarts one
of the main reasons to use buffers, which is to avoid string
allocations. Rewrite it with <tt>buf:put()</tt> and <tt>buf:get()</tt>.
</p>
<p>
Mixing this with unrelated objects that have a <tt>__concat</tt>
metamethod may not work, since these probably only expect strings.
</p>
<h3 id="buffer_skip"><tt>buf = buf:skip(len)</tt></h3>
<p>
Skips (consumes) <tt>len</tt> bytes from the buffer up to the current
length of the buffer data.
</p>
<h3 id="buffer_get"><tt>str, … = buf:get([len|nil] [,…])</tt></h3>
<p>
Consumes the buffer data and returns one or more strings. If called
without arguments, the whole buffer data is consumed. If called with a
number, up to <tt>len</tt> bytes are consumed. A <tt>nil</tt> argument
consumes the remaining buffer space (this only makes sense as the last
argument). Multiple arguments consume the buffer data in the given
order.
</p>
<p>
Note: a zero length or no remaining buffer data returns an empty string
and not <tt>nil</tt>.
</p>
<h3 id="buffer_tostring"><tt>str = buf:tostring()<br>
str = tostring(buf)</tt></h3>
<p>
Creates a string from the buffer data, but doesn't consume it. The
buffer remains unchanged.
</p>
<p>
Buffer objects also define a <tt>__tostring</tt> metamethod. This means
buffers can be passed to the global <tt>tostring()</tt> function and
many other functions that accept this in place of strings. The important
internal uses in functions like <tt>io.write()</tt> are short-circuited
to avoid the creation of an intermediate string object.
</p>
<h3 id="buffer_ref"><tt>ptr, len = buf:ref()</tt><span class="lib">FFI</span></h3>
<p>
Returns an <tt>uint8_t *</tt> FFI cdata pointer <tt>ptr</tt> that
points to the buffer data. The length of the buffer data in bytes is
returned in <tt>len</tt>.
</p>
<p>
The returned pointer can be directly passed to C functions that expect a
buffer and a length. You can also do bytewise reads
(<tt>local x = ptr[i]</tt>) or writes
(<tt>ptr[i] = 0x40</tt>) of the buffer data.
</p>
<p>
In conjunction with the <tt>skip</tt> method, this allows zero-copy use
of C write-style APIs:
</p>
<pre class="code">
repeat
local ptr, len = buf:ref()
if len == 0 then break end
local n = C.write(fd, ptr, len)
if n < 0 then error("write error") end
buf:skip(n)
until n >= len
</pre>
<p>
Unlike Lua strings, buffer data is <i>not</i> implicitly
zero-terminated. It's not safe to pass <tt>ptr</tt> to C functions that
expect zero-terminated strings. If you're not using <tt>len</tt>, then
you're doing something wrong.
</p>
<h2 id="serialize">Serialization of Lua Objects</h2>
<p>
The following functions and methods allow <b>high-speed serialization</b>
(encoding) of a Lua object into a string and decoding it back to a Lua
object. This allows convenient storage and transport of <b>structured
data</b>.
</p>
<p>
The encoded data is in an <a href="#serialize_format">internal binary
format</a>. The data can be stored in files, binary-transparent
databases or transmitted to other LuaJIT instances across threads,
processes or networks.
</p>
<p>
Encoding speed can reach up to 1 Gigabyte/second on a modern desktop- or
server-class system, even when serializing many small objects. Decoding
speed is mostly constrained by object creation cost.
</p>
<p>
The serializer handles most Lua types, common FFI number types and
nested structures. Functions, thread objects, other FFI cdata and full
userdata cannot be serialized (yet).
</p>
<p>
The encoder serializes nested structures as trees. Multiple references
to a single object will be stored separately and create distinct objects
after decoding. Circular references cause an error.
</p>
<h3 id="serialize_methods">Serialization Functions and Methods</h3>
<h3 id="buffer_encode"><tt>str = buffer.encode(obj)<br>
buf = buf:encode(obj)</tt></h3>
<p>
Serializes (encodes) the Lua object <tt>obj</tt>. The stand-alone
function returns a string <tt>str</tt>. The buffer method appends the
encoding to the buffer.
</p>
<p>
<tt>obj</tt> can be any of the supported Lua types — it doesn't
need to be a Lua table.
</p>
<p>
This function may throw an error when attempting to serialize
unsupported object types, circular references or deeply nested tables.
</p>
<h3 id="buffer_decode"><tt>obj = buffer.decode(str)<br>
obj = buf:decode()</tt></h3>
<p>
The stand-alone function deserializes (decodes) the string
<tt>str</tt>, the buffer method deserializes one object from the
buffer. Both return a Lua object <tt>obj</tt>.
</p>
<p>
The returned object may be any of the supported Lua types —
even <tt>nil</tt>.
</p>
<p>
This function may throw an error when fed with malformed or incomplete
encoded data. The stand-alone function throws when there's left-over
data after decoding a single top-level object. The buffer method leaves
any left-over data in the buffer.
</p>
<p>
Attempting to deserialize an FFI type will throw an error, if the FFI
library is not built-in or has not been loaded, yet.
</p>
<h3 id="serialize_options">Serialization Options</h3>
<p>
The <tt>options</tt> table passed to <tt>buffer.new()</tt> may contain
the following members (all optional):
</p>
<ul>
<li>
<tt>dict</tt> is a Lua table holding a <b>dictionary of strings</b> that
commonly occur as table keys of objects you are serializing. These keys
are compactly encoded as indexes during serialization. A well-chosen
dictionary saves space and improves serialization performance.
</li>
<li>
<tt>metatable</tt> is a Lua table holding a <b>dictionary of metatables</b>
for the table objects you are serializing.
</li>
</ul>
<p>
<tt>dict</tt> needs to be an array of strings and <tt>metatable</tt> needs
to be an array of tables. Both starting at index 1 and without holes (no
<tt>nil</tt> in between). The tables are anchored in the buffer object and
internally modified into a two-way index (don't do this yourself, just pass
a plain array). The tables must not be modified after they have been passed
to <tt>buffer.new()</tt>.
</p>
<p>
The <tt>dict</tt> and <tt>metatable</tt> tables used by the encoder and
decoder must be the same. Put the most common entries at the front. Extend
at the end to ensure backwards-compatibility — older encodings can
then still be read. You may also set some indexes to <tt>false</tt> to
explicitly drop backwards-compatibility. Old encodings that use these
indexes will throw an error when decoded.
</p>
<p>
Metatables that are not found in the <tt>metatable</tt> dictionary are
ignored when encoding. Decoding returns a table with a <tt>nil</tt>
metatable.
</p>
<p>
Note: parsing and preparation of the options table is somewhat
expensive. Create a buffer object only once and recycle it for multiple
uses. Avoid mixing encoder and decoder buffers, since the
<tt>buf:set()</tt> method frees the already allocated buffer space:
</p>
<pre class="code">
local options = {
dict = { "commonly", "used", "string", "keys" },
}
local buf_enc = buffer.new(options)
local buf_dec = buffer.new(options)
local function encode(obj)
return buf_enc:reset():encode(obj):get()
end
local function decode(str)
return buf_dec:set(str):decode()
end
</pre>
<h3 id="serialize_stream">Streaming Serialization</h3>
<p>
In some contexts, it's desirable to do piecewise serialization of large
datasets, also known as <i>streaming</i>.
</p>
<p>
This serialization format can be safely concatenated and supports streaming.
Multiple encodings can simply be appended to a buffer and later decoded
individually:
</p>
<pre class="code">
local buf = buffer.new()
buf:encode(obj1)
buf:encode(obj2)
local copy1 = buf:decode()
local copy2 = buf:decode()
</pre>
<p>
Here's how to iterate over a stream:
</p>
<pre class="code">
while #buf ~= 0 do
local obj = buf:decode()
-- Do something with obj.
end
</pre>
<p>
Since the serialization format doesn't prepend a length to its encoding,
network applications may need to transmit the length, too.
</p>
<h3 id="serialize_format">Serialization Format Specification</h3>
<p>
This serialization format is designed for <b>internal use</b> by LuaJIT
applications. Serialized data is upwards-compatible and portable across
all supported LuaJIT platforms.
</p>
<p>
It's an <b>8-bit binary format</b> and not human-readable. It uses e.g.
embedded zeroes and stores embedded Lua string objects unmodified, which
are 8-bit-clean, too. Encoded data can be safely concatenated for
streaming and later decoded one top-level object at a time.
</p>
<p>
The encoding is reasonably compact, but tuned for maximum performance,
not for minimum space usage. It compresses well with any of the common
byte-oriented data compression algorithms.
</p>
<p>
Although documented here for reference, this format is explicitly
<b>not</b> intended to be a 'public standard' for structured data
interchange across computer languages (like JSON or MessagePack). Please
do not use it as such.
</p>
<p>
The specification is given below as a context-free grammar with a
top-level <tt>object</tt> as the starting point. Alternatives are
separated by the <tt>|</tt> symbol and <tt>*</tt> indicates repeats.
Grouping is implicit or indicated by <tt>{…}</tt>. Terminals are
either plain hex numbers, encoded as bytes, or have a <tt>.format</tt>
suffix.
</p>
<pre>
object → nil | false | true
| null | lightud32 | lightud64
| int | num | tab | tab_mt
| int64 | uint64 | complex
| string
nil → 0x00
false → 0x01
true → 0x02
null → 0x03 // NULL lightuserdata
lightud32 → 0x04 data.I // 32 bit lightuserdata
lightud64 → 0x05 data.L // 64 bit lightuserdata
int → 0x06 int.I // int32_t
num → 0x07 double.L
tab → 0x08 // Empty table
| 0x09 h.U h*{object object} // Key/value hash
| 0x0a a.U a*object // 0-based array
| 0x0b a.U a*object h.U h*{object object} // Mixed
| 0x0c a.U (a-1)*object // 1-based array
| 0x0d a.U (a-1)*object h.U h*{object object} // Mixed
tab_mt → 0x0e (index-1).U tab // Metatable dict entry
int64 → 0x10 int.L // FFI int64_t
uint64 → 0x11 uint.L // FFI uint64_t
complex → 0x12 re.L im.L // FFI complex
string → (0x20+len).U len*char.B
| 0x0f (index-1).U // String dict entry
.B = 8 bit
.I = 32 bit little-endian
.L = 64 bit little-endian
.U = prefix-encoded 32 bit unsigned number n:
0x00..0xdf → n.B
0xe0..0x1fdf → (0xe0|(((n-0xe0)>>8)&0x1f)).B ((n-0xe0)&0xff).B
0x1fe0.. → 0xff n.I
</pre>
<h2 id="error">Error handling</h2>
<p>
Many of the buffer methods can throw an error. Out-of-memory or usage
errors are best caught with an outer wrapper for larger parts of code.
There's not much one can do after that, anyway.
</p>
<p>
OTOH, you may want to catch some errors individually. Buffer methods need
to receive the buffer object as the first argument. The Lua colon-syntax
<tt>obj:method()</tt> does that implicitly. But to wrap a method with
<tt>pcall()</tt>, the arguments need to be passed like this:
</p>
<pre class="code">
local ok, err = pcall(buf.encode, buf, obj)
if not ok then
-- Handle error in err.
end
</pre>
<h2 id="ffi_caveats">FFI caveats</h2>
<p>
The string buffer library has been designed to work well together with
the FFI library. But due to the low-level nature of the FFI library,
some care needs to be taken:
</p>
<p>
First, please remember that FFI pointers are zero-indexed. The space
returned by <tt>buf:reserve()</tt> and <tt>buf:ref()</tt> starts at the
returned pointer and ends before <tt>len</tt> bytes after that.
</p>
<p>
I.e. the first valid index is <tt>ptr[0]</tt> and the last valid index
is <tt>ptr[len-1]</tt>. If the returned length is zero, there's no valid
index at all. The returned pointer may even be <tt>NULL</tt>.
</p>
<p>
The space pointed to by the returned pointer is only valid as long as
the buffer is not modified in any way (neither append, nor consume, nor
reset, etc.). The pointer is also not a GC anchor for the buffer object
itself.
</p>
<p>
Buffer data is only guaranteed to be byte-aligned. Casting the returned
pointer to a data type with higher alignment may cause unaligned
accesses. It depends on the CPU architecture whether this is allowed or
not (it's always OK on x86/x64 and mostly OK on other modern
architectures).
</p>
<p>
FFI pointers or references do not count as GC anchors for an underlying
object. E.g. an <tt>array</tt> allocated with <tt>ffi.new()</tt> is
anchored by <tt>buf:set(array, len)</tt>, but not by
<tt>buf:set(array+offset, len)</tt>. The addition of the offset
creates a new pointer, even when the offset is zero. In this case, you
need to make sure there's still a reference to the original array as
long as its contents are in use by the buffer.
</p>
<p>
Even though each LuaJIT VM instance is single-threaded (but you can
create multiple VMs), FFI data structures can be accessed concurrently.
Be careful when reading/writing FFI cdata from/to buffers to avoid
concurrent accesses or modifications. In particular, the memory
referenced by <tt>buf:set(cdata, len)</tt> must not be modified
while buffer readers are working on it. Shared, but read-only memory
mappings of files are OK, but only if the file does not change.
</p>
<br class="flush">
</div>
<div id="foot">
<hr class="hide">
Copyright © 2005-2022
<span class="noprint">
·
<a href="contact.html">Contact</a>
</span>
</div>
</body>
</html>
|
