Deployment Overview
In this document, we describe the deployment components of NebulaStream. Currently, NES is structured in a master/worker hierarchical pattern, where a client is communicating to the NES-Coordinator to manage the deployment of queries. Operators of these queries are then transmitted from the Coordinator to automatically chosen worker nodes. The transmission of all coordination messages in NES happens via GRPC.
NES Coordinator
The NES Coordinator maintains the information of the following components in NES:
The NES Coordinator currently accepts the following requests:
Starting the coordinator where in the background a GRPC server is created and running
Stopping the coordinator and the corresponding GRPC server
Getting data of the NES Topology
Getting data of the NES StreamingCatalog
Getting data of the NES QueryService
Getting query statistics based on a query ID
Requesting monitoring information from a remote worker node
The NES Coordinator can be started using the following parameters:
–serverIp
(Set the IP address of the NESCoordinator (default: 127.0.0.1))
–coordinatorPort
(Set the RPC Server port of the NESCoordinator (default: 4000))
–restPort
(Set the port exposed for REST endpoints (default: 8081))
–numberOfSlots
(Set the number of computing slots in the NESCoordinator (default: number of processor in the host))
–help
(Display help message)
NES Worker
The NES worker maintains the information of the following components in NES:
NodeEngine
PhysicalStreamConfig where information are managed of the underlying properties of physical streams
The NES worker currently accepts the following requests:
Starting a NES-Worker and underlying GRPC server, so that workers can accept commands to deploy queries
Stopping a NES-Worker and the underlying GRPC server
Connecting and disconnecting from a NES-Coordinator
Registration of physical and logical streams
The NES worker can be started using the following parameters:
–coordinatorIp
(Set the server IP of the NES Coordinator to which the NESWorker should connect (default: 127.0.0.1))
–coordinatorPort
(Set the RPC server Port of the NES Coordinator to which the NESWorker connected (default: 0))
–rpcPort
(Set the RPC server of the NESWorker (default: 0))
–dataPort
(Set the data port of the NESWorker (default: 0))
–sourceType
(Set the type of the Source (must be either CSVSource or DefaultSource))
–sourceConfig
(Set the configuration of the source (e.g., path to the CSV file))
–sourceFrequency
(Set the sampling frequency of the source)
–physicalStreamName
(Set the physical name of the stream)
–numberOfBuffersToProduce
(Set the number of buffers to produce)
–logicalStreamName
(Set the logical stream name where this stream is added to)
–parentId
(Set the parentId of the NESWorker node)
–localWorkerIp
(Set worker ip (default: 127.0.0.1))
–numberOfSlots
(Set the number of computing slots in the NESWorker (default: number of processor in the host))
–help
(Display help message)
Deployment Phases
In NES the deployment is of queries is intended to take place in the following chronological order:
Starting a NES-Coordinator
Starting NES-Workers
Registering NES-Workers at the NES-Coordinator
Registration of physical streams
Creation of logical streams
Registration of a query
Deployment of a query
Data management operations like monitoring of a running query or according worker nodes