Within a NES-cluster there can be several workers. These workers are either directly connected to the coordinator or transitively connected to the coordinator via another worker. The job of a worker is to locally manage the lifecycle of a query provided to it by the coordinator. Coordinator is providing a sub-query derived from a larger query to the worker.

A worker can also contain a data source, i.e., it is responsible for providing the stream of data that the query is interested in. Therefore, a worker with a data source contains all the information about the schema, the source type, the connection configuration, etc., that are necessary for identifying and consuming a data source. These configurations are supplied while starting a worker node as runtime arguments.

The information which worker contains which data source is managed centrally at the coordinator in a catalog, and is used while scheduling the execution of a query.

worker.txt · Last modified: 2022/02/23 07:09 by
Recent changes RSS feed Creative Commons License Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki