Memory Management

Buffer Manager

The BufferManager is responsible for:

  1. Pooled Buffers: preallocated fixed-size buffers of memory that must be reference counted
  2. Unpooled Buffers: variable sized buffers that are allocated on-the-fly. They are also subject to reference counting.

The BufferManager stores the pooled buffers as MemorySegment(s). When a component asks for a Pooled buffer, the BufferManager retrieves an available buffer (it blocks the calling thread, if no buffer is available). It then hands out a TupleBuffer that is constructed through the pointer stored inside a MemorySegment. This is necessary because the BufferManager must keep all buffers stored to ensure that when its destructor is called, all buffers that it has ever created are deallocated. Note the BufferManager will check also that no reference counter is non-zero and will throw a fatal exception if a component has not returned every buffer. This is necessary to avoid memory leaks.

Unpooled buffers are either allocated on the spot or served via a previously allocated, unpooled buffer that has been returned to the BufferManager by some component.

TupleBuffer Abstraction

The TupleBuffer is the NES API that allows runtime components to access memory to store records. Its purpose is to avoid memory allocations and enable batching. To avoid memory allocation, a BufferManager keeps a fixed set of fixed size buffers that can be handed to the components. When a component is done with a buffer, it can be returned to the BufferManager. However, the multi-threaded nature of the NES execution engine introduces the challenge of reference counting, i.e., if a tuple buffer is shared among threads, we need to safely determine when the TupleBuffer can be returned to the BufferManager.

Our solution follows the same principle behind std::shared_ptr. We use the TupleBuffer as a normal value. Note that TupleBuffer* is forbidden and operator& of TupleBuffer is deleted. We increase the reference counter of the TupleBuffer every time it gets passed via a copy ctor or a copy assignment operator. When the dtor of a TupleBuffer is called, then the reference counter is decreased. To pass the TupleBuffer within a thread, a TupleBuffer& must be used. To pass the TupleBuffer among thread, use TupleBuffer. Ensure the copy semantics is used. The move semantics is enabled but that does not increase the internal counter. Important note: when a component is done with a TupleBuffer, it must be released. Not returning a TupleBuffer will result in a runtime error that the BufferManager will raise by the termination of the NES program.

Debugging Memory Leaks: you may want to define NES_DEBUG_TUPLE_BUFFER_LEAKS=1 through cmake to enable extra stats regarding memory leaks.

Dynamic Memory Layout

The idea behind the dynamic memory layout is to have an abstraction via the underlying TupleBuffers. The memory layout should be flexible for different use-cases, such as

  1. VarLength Fields e.g., Text, Sparse Matrix, Compressed Images
  2. Columnar Fields e.g., manipulations on single fields
  3. Compressed Fields e.g., run length encoding on numeric fields

The requirements are a high-level interface to read, write records and fields. The performance should be the same with an C++ struct without any knowledge required of underlying tuple structure.

It is worth mentioning that this is an ongoing project that is actively updated for better performance. Further iterations may change the API and will improve the performance.


Generally speaking, all operations exist for both layout types (row and column). A good starting point for real C++ code is to have a look at tests/UnitTests/DynamicMemoryTest.cpp.

	auto currentNumberOfRecords = mappedLayout->getNumberOfRecords();
	auto maxNumberOfRecords = mappedLayout->getCapacity();
Creating Layout

Creating a layout is usually the first step as a layout is needed for the next steps.

	auto rowLayout = DynamicRowLayout::create(schema, checkBoundaries);
	auto colLayout = DynamicColumnLayout::create(schema, checkBoundaries);

checkBoundaries is a boolean value that checks if accesses to this layout should be checked and an error should be thrown.

Binding Layout to TupleBuffer

After a layout is created, a tupleBuffer can be binded to a memory layout. For now the function is called map but this will change with the next release. A binded layout can read and write to a tupleBuffer.

	auto bindedRowLayout = rowLayout->bind(tupleBuffer0);
	auto bindedColumnLayout = columnLayout->bind(tupleBuffer1);
Read and Write Whole Records

It is possible to read and write whole records. For a record representation, std::tuple has been chosen. pushRecord<>() returns the success of its operation. Additionally, it is possible to write a tuple at a given recordIndex.

	std::tuple<uint8_t, uint16_t, uint32_t> writeRecord(1, 2, 3);
	// Writing
	bool success = bindedRowLayout->pushRecord<checkBoundaries>(writeRecord);
	success = bindedColumnLayout->pushRecord<checkBoundaries>(writeRecord);
	success = bindedColumnLayout->pushRecord<checkBoundaries>(writeRecord, recordIndex);
	// Reading
	std::tuple<uint8_t, uint16_t, uint32_t> readRecord = bindedRowLayout->readRecord<checkBoundaries, uint8_t, uint16_t, uint32_t>(recordIndex);
	std::tuple<uint8_t, uint16_t, uint32_t> readRecord = bindedColumnLayout->readRecord<checkBoundaries, uint8_t, uint16_t, uint32_t>(recordIndex);

checkBoundaries is a boolean value that checks if accesses to this layout should be checked and an error should be thrown. As it is a template, there is no runtime overhead.

Read and Write Fields

For some applications, having direct access to fields may be beneficial. Before access is possible, one has to create a field handler. This can be done via the fieldIndex or via the fieldName. Afterwards, accessing fields is possible via an operator overloading of [].

	// Creating a row and col field handler of type ''int32_t''
	auto rowFieldHandler0 = DynamicRowLayoutField<int32_t, checkBoundaries>::create(fieldIndex, bindedRowLayout);
	auto colFieldHandler = DynamicColumnLayoutField<int32_t, checkBoundaries>::create(fieldName, bindedColumnLayout);
	// Writing
	rowFieldHandler0[recordIndex] = 42;
	colFieldHandler[recordIndex] = 42;
	// Reading
	int32_t rowField0 = rowFieldHandler0[recordIndex];
	int32_t colField0 = colFieldHandler[recordIndex];
memory_management.txt · Last modified: 2022/02/23 07:33 by
Recent changes RSS feed Creative Commons License Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki