The string buffer library allows high-performance manipulation of string-like data.
Unlike Lua strings, which are constants, string buffers are mutable sequences of 8-bit (binary-transparent) characters. Data can be stored, formatted and encoded into a string buffer and later converted, decoded or extracted.
The convenient string buffer API simplifies common string manipulation tasks, that would otherwise require creating many intermediate strings. String buffers improve performance by eliminating redundant memory copies, object creation, string interning and garbage collection overhead. In conjunction with the FFI library, they allow zero-copy operations.
Using the String Buffer Library
The string buffer library is built into LuaJIT by default, but it's not loaded by default. Add this to the start of every Lua file that needs one of its functions:
local buffer = require("string.buffer")
Work in Progress
This library is a work in progress. More functions will be added soon.
Serialization of Lua Objects
The following functions and methods allow high-speed serialization (encoding) of a Lua object into a string and decoding it back to a Lua object. This allows convenient storage and transport of structured data.
The encoded data is in an internal binary format. The data can be stored in files, binary-transparent databases or transmitted to other LuaJIT instances across threads, processes or networks.
Encoding speed can reach up to 1 Gigabyte/second on a modern desktop- or server-class system, even when serializing many small objects. Decoding speed is mostly constrained by object creation cost.
The serializer handles most Lua types, common FFI number types and nested structures. Functions, thread objects, other FFI cdata, full userdata and associated metatables cannot be serialized (yet).
The encoder serializes nested structures as trees. Multiple references to a single object will be stored separately and create distinct objects after decoding. Circular references cause an error.
str = buffer.encode(obj)
Serializes (encodes) the Lua object obj into the string str.
obj can be any of the supported Lua types — it doesn't need to be a Lua table.
This function may throw an error when attempting to serialize unsupported object types, circular references or deeply nested tables.
obj = buffer.decode(str)
De-serializes (decodes) the string str into the Lua object obj.
The returned object may be any of the supported Lua types — even nil.
This function may throw an error when fed with malformed or incomplete encoded data. The standalone function throws when there's left-over data after decoding a single top-level object.
Serialization Format Specification
This serialization format is designed for internal use by LuaJIT applications. Serialized data is upwards-compatible and portable across all supported LuaJIT platforms.
It's an 8-bit binary format and not human-readable. It uses e.g. embedded zeroes and stores embedded Lua string objects unmodified, which are 8-bit-clean, too. Encoded data can be safely concatenated for streaming and later decoded one top-level object at a time.
The encoding is reasonably compact, but tuned for maximum performance, not for minimum space usage. It compresses well with any of the common byte-oriented data compression algorithms.
Although documented here for reference, this format is explicitly not intended to be a 'public standard' for structured data interchange across computer languages (like JSON or MessagePack). Please do not use it as such.
The specification is given below as a context-free grammar with a top-level object as the starting point. Alternatives are separated by the | symbol and * indicates repeats. Grouping is implicit or indicated by {…}. Terminals are either plain hex numbers, encoded as bytes, or have a .format suffix.
object → nil | false | true
| null | lightud32 | lightud64
| int | num | tab
| int64 | uint64 | complex
| string
nil → 0x00
false → 0x01
true → 0x02
null → 0x03 // NULL lightuserdata
lightud32 → 0x04 data.I // 32 bit lightuserdata
lightud64 → 0x05 data.L // 64 bit lightuserdata
int → 0x06 int.I // int32_t
num → 0x07 double.L
tab → 0x08 // Empty table
| 0x09 h.U h*{object object} // Key/value hash
| 0x0a a.U a*object // 0-based array
| 0x0b a.U a*object h.U h*{object object} // Mixed
| 0x0c a.U (a-1)*object // 1-based array
| 0x0d a.U (a-1)*object h.U h*{object object} // Mixed
int64 → 0x10 int.L // FFI int64_t
uint64 → 0x11 uint.L // FFI uint64_t
complex → 0x12 re.L im.L // FFI complex
string → (0x20+len).U len*char.B
.B = 8 bit
.I = 32 bit little-endian
.L = 64 bit little-endian
.U = prefix-encoded 32 bit unsigned number n:
0x00..0xdf → n.B
0xe0..0x1fdf → (0xe0|(((n-0xe0)>>8)&0x1f)).B ((n-0xe0)&0xff).B
0x1fe0.. → 0xff n.I