Query Compilation

NebulaStream relies on query compilation to translate a logical query plan (LQP) into an executable query plan.
This process was in general introduced in databases to improve execution performance and lately applied to stream processing.

To generate code NebulaStream relies on a tiered approach and transforms LQP in multiple steps to executable code,
e.g. lowering to physical operators, pipelining, lowering to generatable operators, and code-generation.
This approach enables NebulaStream to provide a high level of extendability and testability.

In particular query compilation in NebulaStream consist of the following steps:

1. Logical Query Plan

Initially, the query compiler receives an LQP, which represents a particular query or subquery with a set of logical operators.
Typically, the coordinator performs high-level query optimizations, e.g., merging, reordering, and sends LQPs to individual worker nodes.
On these workers, the query compiler handles the further processing of the query.

The LQP consists of logical operators in an acyclic graph. It can contain multiple data sources and data sinks.
In general, we differentiate between unary and binary operators. However, each operator can have an arbitrary amount of input edges.

2. Physical Query Plan

From the LQP, NebulaStream derives the physical query plan (PQP).
The PQP consists of physical operators in an acyclic graph. It can contain multiple data sources and data sinks.
In contrast to logical operators, physical operators represent a concrete physical aspect of a logical operator.
For instance, a logical Join operator is lowered to two JoinBuildOperators (one per join side) and one JoinSinkOperator.
Furthermore, the PQP defines that unary operators only can exactly have one input edge and binary operators exactly two input edges.
The only exceptions are the MultiplexOperator, which has n input edges, and the DemultiplexOperator, which has n output edges.

3. Pipeline Plan

Based on the PQP, NES performs pipelining to enable operator fusion. Depending on a particular pipelining policy NES decides to combine multiple operators in one pipeline.

4. Generatable Operators

5. Executable Query Plan


query_compiler.txt · Last modified: 2021/05/25 09:26 by nes
Recent changes RSS feed Creative Commons License Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki