Configure NES Coordinator

NebulaStream provides two possibilities to configure Coordinator.

  1. Providing configuration over CLI.
  2. Providing configuration using YAML file.

CLI Configurations

In the following table, we provide available configuration keys, their default values, and the corresponding descriptions.

Key Default Description
–restIp 127.0.0.1 Set NES ip of the REST server
–coordinatorIp 127.0.0.1 Set RPC IP address of NES Coordinator
–rpcPort 0 Set the RPC Server port of the NES Coordinator. It must be set otherwise coordinator will fail.
–restPort 0 Set the port exposed for REST endpoints. It must be set otherwise coordinator will fail.
–dataPort 0 Set NES data server port. If set to 0, the operating system will choose one for us.
–numberOfSlots Number of processors installed in the host machine Set the number of computing slots in the NES Coordinator
–logLevel “LOG_DEBUG” The log level (LOG_NONE, LOG_WARNING, LOG_DEBUG, LOG_INFO, LOG_TRACE)
–numberOfBuffersInGlobalBufferManager 1024 Number buffers in global buffer pool
–numberOfBuffersPerWorker 128 Number buffers in task local buffer pool
–numberOfBuffersInSourceLocalBufferPool 64 Number buffers in source local buffer pool
–bufferSizeInBytes 4096 Size of buffer to pass data through the system
–numWorkerThreads 1 Number of threads available for the worker node. Defines how many queries can run in parallel.
–enableMonitoring true Enable monitoring - collection of metrics
–enableQueryReconfiguration false Enable query reconfiguration without redeploying them
–optimizer.queryMergerRule “SyntaxBasedCompleteQueryMergerRule” The rule to be used for performing query merging
–optimizer.memoryLayoutPolicy FORCE_ROW_LAYOUT Sets the memory layout selection policy. Can be [FORCE_ROW_LAYOUT/FORCE_COLUMN_LAYOUT] to select either always a row or a column layout.
–optimizer.performOnlySourceOperatorExpansion false Perform only source operator duplication when applying Logical Source Expansion Rewrite Rule.
–optimizer.performDistributedWindowOptimization true Perform distributed window optimizations.
–optimizer.distributedWindowChildThreshold 2 Number of child operators to a window before applying distributed window optimizations.
–optimizer.distributedWindowCombinerThreshold 4 Threshold for the insertion of pre-aggregation operators.
–optimizer.advanceSemanticValidation false Enable advance semantic query validation feature
–optimizer.queryBatchSize 1 The number of queries to be processed together

YAML Configuration

A user may create a configuration file for starting the Coordinator and pass it as input using CLI using –configPath=<path to config file>. A template Coordinator configurations file looks as follows:

# ~~~ Configurations for the NES Coordinator ~~~
 
# ~~ IPs and Ports ~~
# Set NES ip of the REST server
restIp: 127.0.0.1
 
# Set RPC IP address of NES Coordinator
coordinatorIp: 127.0.0.1
 
# Set RPC Server port of NES Coordinator
rpcPort: 4000
 
# Set port exposed for REST endpoints
restPort: 8081
 
# Set the data port of the network manager
dataPort: 3001
 
 
# ~~ Processing Configurations ~~
# Set the number of computing slots in NES Coordinator
numberOfSlots: 65535
 
# The log level (LOG_NONE, LOG_WARNING, LOG_DEBUG, LOG_INFO, LOG_TRACE)
logLevel: LOG_DEBUG
 
# Number buffers in global buffer pool
numberOfBuffersInGlobalBufferManager: 1024
 
# Number buffers in task local buffer pool
numberOfBuffersPerWorker: 128
 
# Number buffers in source local buffer pool
numberOfBuffersInSourceLocalBufferPool: 64
 
# Size of buffer to pass data through the system
bufferSizeInBytes: 4096
 
# Number of threads available for the worker node. Defines how many queries can run in parallel.
numWorkerThreads: 1
 
# The number of queries to be processed together
queryBatchSize: 1
 
# Enable monitoring
enableMonitoring: false

optimizer:
  # The rule to be used for performing query merging
  queryMergerRule: DefaultQueryMergerRule
 
  # Advance Semantic Query Validation Feature
  advanceSemanticValidation: false
 
  # Selects the memory layout selection policy can be [FORCE_ROW_LAYOUT|FORCE_COLUMN_LAYOUT]
  memoryLayoutPolicy: FORCE_ROW_LAYOUT
 
  # Enable reconfiguration of running query plans. Default: false
  enableQueryReconfiguration: false
 
  # The number of queries to be processed together
  queryBatchSize: 1
 
  # Perform only source operator duplication when applying Logical Source Expansion Rewrite Rule.
  performOnlySourceOperatorExpansion: false
 
coordinator-configurations.txt · Last modified: 2022/02/25 17:33 by 134.96.191.189
 
Recent changes RSS feed Creative Commons License Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki