Deployment Overview

In this document, we describe the deployment components of NebulaStream. Currently, NES is structured in a master/worker hierarchical pattern, where a client is communicating to the NES-Coordinator to manage the deployment of queries. Operators of these queries are then transmitted from the Coordinator to automatically chosen worker nodes. The transmission of all coordination messages in NES happens via GRPC.

The NES Coordinator maintains the information of the following components in NES:

• Topology
• Streaming catalog where we have the mapping between logical and physical streams
• QueryService for query registration and deployment

The NES Coordinator currently accepts the following requests:

• Starting the coordinator where in the background a GRPC server is created and running
• Stopping the coordinator and the corresponding GRPC server
• Getting data of the NES Topology
• Getting data of the NES StreamingCatalog
• Getting data of the NES QueryService
• Getting query statistics based on a query ID
• Requesting monitoring information from a remote worker node

The NES Coordinator can be started using the following parameters:

• –serverIp (Set the IP address of the NESCoordinator (default: 127.0.0.1))
• –coordinatorPort (Set the RPC Server port of the NESCoordinator (default: 4000))
• –restPort (Set the port exposed for REST endpoints (default: 8081))
• –numberOfSlots (Set the number of computing slots in the NESCoordinator (default: number of processor in the host))
• –help (Display help message)

The NES worker maintains the information of the following components in NES:

1. NodeEngine
2. PhysicalStreamConfig where information are managed of the underlying properties of physical streams

The NES worker currently accepts the following requests:

1. Starting a NES-Worker and underlying GRPC server, so that workers can accept commands to deploy queries
2. Stopping a NES-Worker and the underlying GRPC server
3. Connecting and disconnecting from a NES-Coordinator
4. Registration of physical and logical streams

The NES worker can be started using the following parameters:

• –coordinatorIp (Set the server IP of the NES Coordinator to which the NESWorker should connect (default: 127.0.0.1))
• –coordinatorPort (Set the RPC server Port of the NES Coordinator to which the NESWorker connected (default: 0))
• –rpcPort (Set the RPC server of the NESWorker (default: 0))
• –dataPort (Set the data port of the NESWorker (default: 0))
• –sourceType (Set the type of the Source (must be either CSVSource or DefaultSource))
• –sourceConfig (Set the configuration of the source (e.g., path to the CSV file))
• –sourceFrequency (Set the sampling frequency of the source)
• –physicalStreamName (Set the physical name of the stream)
• –numberOfBuffersToProduce (Set the number of buffers to produce)
• –logicalStreamName (Set the logical stream name where this stream is added to)
• –parentId (Set the parentId of the NESWorker node)
• –localWorkerIp (Set worker ip (default: 127.0.0.1))
• –numberOfSlots (Set the number of computing slots in the NESWorker (default: number of processor in the host))
• –help (Display help message)

In NES the deployment is of queries is intended to take place in the following chronological order:

1. Starting a NES-Coordinator
2. Starting NES-Workers
3. Registering NES-Workers at the NES-Coordinator
4. Registration of physical streams
5. Creation of logical streams
6. Registration of a query
7. Deployment of a query
8. Data management operations like monitoring of a running query or according worker nodes