Memory limit in Lua sandbox on C++

Limiting Memory and Resources in a Lua Sandbox with C++

When integrating Lua into C++ applications, there's often a need to isolate script execution from the main system. This is especially critical in scenarios where third-party code might be malicious or unstable—for example, in game engines, plugin systems, or sandbox environments. One key aspect of such isolation is controlling memory consumption. This article covers how to implement a strict memory usage limit in Lua using a custom allocator integrated into the interpreter's state.

How Lua Manages Memory

Lua delegates all memory operations to an external allocator. By default, it uses the standard realloc/free, but the language allows replacing it with your own implementation. This is achieved by passing a pointer to the function when creating lua_State. Importantly: all allocation, resizing, and freeing requests go through this allocator—including garbage collector operations, module loading, and creation of tables, strings, functions, and so on.

The allocator function has the following signature:

void *luaAlloc(void *ud, void *ptr, size_t currSize, size_t newSize);

• ud — user data (typically a pointer to the memory controller state);
• ptr — pointer to the current block (may be NULL when allocating new);
• currSize — current block size or object type (in Lua 5.2+ when ptr == NULL);
• newSize — requested new size (0 means free).

This gives full control over memory allocation, enabling tracking and limiting its usage.

Implementing a Limited Allocator

To track the amount of allocated memory, we introduce a state structure:

struct LimitedAllocatorState {
    size_t used {};
    size_t limit {1 * 1024 * 1024}; // 1 MB by default
    bool limitReached {false};
    bool overflow {false};

    bool isLimitEnabled() { return limit > 0; }
    void disableLimit() { limit = 0; }
    void resetErrorFlags() { limitReached = overflow = false; }
};

The allocator checks whether the new request (used - currSize + newSize) exceeds the set limit. If it does, it returns nullptr, which triggers an error in Lua (e.g., not enough memory).

Here are the key implementation points:

• When newSize == 0 — memory is freed, and used decreases by currSize;
• When ptr == NULL — a new block is allocated, currSize is ignored;
• Protection against integer overflow is mandatory;
• The allocator state must outlive lua_State.

Example function:

void *limitedAlloc(void *ud, void *ptr, size_t currSize, size_t newSize) {
    auto *allocState = static_cast<LimitedAllocatorState*>(ud);
    if (!allocState) return nullptr;

    if (ptr == nullptr) currSize = 0;
    if (newSize == 0) {
        if (ptr != nullptr) {
            allocState->used = (allocState->used >= currSize)
                ? allocState->used - currSize : 0;
        }
        std::free(ptr);
        return nullptr;
    }

    const size_t usedBase = (allocState->used >= currSize)
        ? allocState->used - currSize : 0;

    if (newSize > SIZE_MAX - usedBase) {
        allocState->overflow = true;
        return nullptr;
    }

    const size_t newUsed = usedBase + newSize;
    if (allocState->isLimitEnabled() && newUsed > allocState->limit) {
        allocState->limitReached = true;
        return nullptr;
    }

    void *newPtr = std::realloc(ptr, newSize);
    if (newPtr) allocState->used = newUsed;
    return newPtr;
}

Integration with C++ Runtime

When using the sol2 library (a popular wrapper over the Lua C API), creating a state with a custom allocator looks like this:

sol::state lua(sol::default_at_panic, limitedAlloc, &allocState);

It's important to follow the order of field declarations in the class: the allocator state must be destroyed after sol::state; otherwise, state destruction will access already-freed memory.

It's recommended to wrap all the logic in a LuaRuntime class that:

• Manages the state lifecycle;
• Provides methods for resetting errors and changing the limit on the fly;
• Allows checking whether the quota has been exceeded.

Example interface:

class LuaRuntime {
private:
    lua::memory::LimitedAllocatorState allocatorState;
    sol::state state;

public:
    LuaRuntime(size_t memoryLimit)
        : allocatorState({.limit = memoryLimit}),
          state(sol::default_at_panic, lua::memory::limitedAlloc, &allocatorState) {}

    bool hasAllocError() {
        return allocatorState.limitReached || allocatorState.overflow;
    }

    void resetAllocErrors() {
        allocatorState.resetErrorFlags();
    }

    bool setMemoryLimit(size_t limit) {
        allocatorState.limit = limit;
        return true;
    }
};

Limitations and Practical Recommendations

Several important notes when working with a limited allocator:

• The limit is shared across the entire state. If you use multiple sandboxes on one lua_State, they share a single memory budget. For strict isolation—use one state per sandbox.
• The garbage collector also consumes memory. Even after deleting objects in Lua, memory usage may temporarily increase during GC runs.
• Allocation errors don't always trigger a panic. Lua generates an exception, which can be caught via pcall or a panic handler.
• Don't forget to reset error flags. After handling an error, limitReached remains set until explicitly reset.

Also keep in mind that certain operations (e.g., concatenating long strings or deep recursion) can cause sharp spikes in memory consumption. Testing with realistic workloads is essential.

Key Takeaways

• Lua allows full control over memory allocation through a custom allocator.
• The memory limit applies to the entire state, not to individual scripts.
• When the limit is exceeded, the allocator returns nullptr, triggering an error in Lua.
• The allocator state must outlive lua_State.
• Integer overflow protection and correct arithmetic are essential for a reliable implementation.

— Editorial Team