Requirements:

OS:

Ubuntu 20.04+

IDE:

Clion/Eclipse

Automatic setup:

Execute

 scripts/dependencies/ubuntu/install.sh 

to install necessary dependencies (it may take up to 60 minutes).

Manual setup:

NES dependencies should be downloaded automatically now. In case they are not:

Standard Dependencies:

apt-get install -qq \
  clang \
  libdwarf-dev \
  libdwarf1 \
  llvm-dev \
  binutils-dev \
  libdw-dev \
  libboost-all-dev \
  liblog4cxx-dev \
  libcpprest-dev \
  libmbedtls-dev \
  libssl-dev \
  clang-format \
  librdkafka1 \
  librdkafka++1 \
  librdkafka-dev \
  libeigen3-dev \
  libzmqpp-dev \
  git \
  wget \
  z3 \
  tar

Cmake Dependency

Remove any previous cmake installation and install cmake-3.21.4 using following commands:

wget https://github.com/Kitware/CMake/releases/download/v3.21.4/cmake-3.21.4.tar.gz

tar -zxvf cmake-3.21.4.tar.gz

cd cmake-3.21.4

./bootstrap

make -j3

sudo make install

Kafka Dependences:

wget https://github.com/mfontanini/cppkafka/archive/v0.3.1.tar.gz -O cppkafka-0.3.1.tar.gz

tar -xzf cppkafka-0.3.1.tar.gz

mkdir -p cppkafka-0.3.1/build

cd cppkafka-0.3.1/build

cmake -DCPPKAFKA_BUILD_SHARED=ON -DCPPKAFKA_DISABLE_TESTS=ON -DCPPKAFKA_DISABLE_EXAMPLES=ON -DCPPKAFKA_RDKAFKA_STATIC_LIB=OFF ..

make -j2 -w 

sudo make install

OPC:

add-apt-repository ppa:open62541-team/ppa && \
apt-get update && \
apt-get install libopen62541-1-dev
CMAKE command:
 -DNES_USE_OPC=1

GRPC

git clone --branch v1.28.1 https://github.com/grpc/grpc.git
cd grpc && git submodule update --init --jobs 1 && mkdir -p build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release && \
  make -j && sudo make install && cd .. && cd .. && rm -rf grpc

MQTT

cd ${HOME} && git clone https://github.com/eclipse/paho.mqtt.c.git && \
  cd paho.mqtt.c && git checkout v1.3.8 && \
  cmake -Bbuild -H. -DPAHO_ENABLE_TESTING=OFF \
  -DPAHO_BUILD_STATIC=ON \
  -DPAHO_WITH_SSL=ON \
  -DPAHO_HIGH_PERFORMANCE=ON && \
  cmake --build build/ --target install && \
  ldconfig && cd ${HOME} && rm -rf paho.mqtt.c && \
  git clone https://github.com/eclipse/paho.mqtt.cpp && cd paho.mqtt.cpp && \
  cmake -Bbuild -H. -DPAHO_BUILD_STATIC=ON -DPAHO_BUILD_DOCUMENTATION=TRUE -DPAHO_BUILD_SAMPLES=TRUE && \
  cmake --build build/ --target install && ldconfig && cd ${HOME} && rm -rf paho.mqtt.cpp
CMAKE command:
 -DNES_USE_MQTT=1

MacOS setup:

With Docker installed run the following command to load and start the development image:

docker run -d --cap-add sys_ptrace -p127.0.0.1:2222:22 --entrypoint /usr/bin/env --name nebula_dev nebulastream/nes-dev-image:latest /usr/sbin/sshd -D

Build, Execution, Deployment settings in CLion

This command does not take the option of mounting any directory unto the container. Instead, we leverage CLions remote toolchain to sync the files into the docker container before the build. Go to Settings / Preferences | Build, Execution, Deployment. To connect CLion to your locally running container, set up a new remote toolchain with the following SSH credentials: - Host: 127.0.0.1 - Port: 2222 - Username: nes - Password: Dockerfile-NES-Dev file contains the password to be used for connection setup.

Then, create a new (debug) CMake setup using the toolchain and the following Cmake option (or others useful for your development):

-DCMAKE_BUILD_TYPE=Debug -DNES_SPECIFIC_FLAGS=-fstandalone-debug -DNES_SELF_HOSTING=1 -DNES_USE_KAFKA=0 -DNES_USE_OPC=0 -DNES_USE_MQTT=1 -DNES_USE_ADAPTIVE=0

The following changes to “Preferences | Build, Execution, Deployment | Deployment” have proven useful in keeping the synchronisation and build on the container smooth:

  • Connection → Root path: /
  • Mappings → Local path: /your/local/path/nebulastream
  • Mappings → Deployment path: /home/nes/nebulastream
  • Mappings → Web path: /
  • Excluded Paths → Add new local path: /your/local/path/nebulastream/cmake-build-debug

Utilities

When a new version of the development image becomes available stop the container and run:

docker rm -f nebula_dev
docker rmi nebulastream/nes-dev-image:latest
docker pull nebulastream/nes-dev-image:latest
docker run -d --cap-add sys_ptrace -p127.0.0.1:2222:22 --entrypoint /usr/bin/env --name nebula_dev nebulastream/nes-dev-image:latest /usr/sbin/sshd -D

You may verify if everything went well by running the command:

docker ps -a

MacOS setup using Docker-Machine:

It is recommended to use the streamlined process without Docker-Machine described above. Howver, Docker-Machine has previously been necessary and the setup details are still given as a backup:

In order to run NES on your macOS machine, first, download and install docker. e.g. by using HomeBrew in your terminal:

brew install docker docker-machine

or via their website https://docs.docker.com/docker-for-mac/install/

Next, install VirtualBox as:

brew install virtualbox

NES can't run natively on macOS therefore you need a VM. You can find it here: https://www.virtualbox.org/wiki/Downloads

After having installed Virtual Box and docker, run these commands in your terminal:

docker-machine create --driver=virtualbox default

It is recommended to navigate to docker's settings and increase the available RAM to ~8GB. This makes the build process much more stable. I another case it was necessary to increase the memory of the “default” machine in the settings of VirtualBox. (Run “docker-machine stop” before.)

Afterwards, you can run the following commands every time you want to work on NES:

docker-machine restart default
eval "$(docker-machine env default)"
docker rm -f $(docker ps -a -q)
docker-machine ssh default -f -N -L 2222:127.0.0.1:2222
docker run -v $(pwd):/nebulastream -d --cap-add sys_ptrace -p127.0.0.1:2222:22 --entrypoint /usr/bin/env --name nebula_dev nebulastream/nes-dev-image:latest /usr/sbin/sshd -D

Once this is up and running, you can connect your CLion with Cmake: https://www.jetbrains.com/help/clion/remote-projects-support.html

If you are experiencing issues with VirtualBox on Big Sur, please, refer to: http://biercoff.com/how-to-fix-docker-machine-installation-on-mac-os-x/ It is usually because you need to upgrade virtualbox and allow its trusted execution.

Windows 11 setup:

To support Windows as a development environment for NebulaStream we rely on WSL 2.

wsl --install -d Ubuntu

Build Command

git clone https://github.com/nebulastream/nebulastream.git

cd nebulastream

mkdir -p build

cd build

cmake -DCMAKE_BUILD_TYPE=Release -DBoost_NO_SYSTEM_PATHS=TRUE -DBoost_INCLUDE_DIR="/usr/include" -DBoost_LIBRARY_DIR="/usr/lib/x86_64-linux-gnu" \
-DCPPRESTSDK_DIR="/usr/lib/x86_64-linux-gnu/cmake/" -DCMAKE_CXX_FLAGS_RELEASE:STRING=" -Wno-unused-variable -Wno-unused-parameter -Wno-ignored-qualifiers
\ -Wno-sign-compare" ..

make -j2

Utilities

Include What You Use

IWYU is a clang tool which analyses the includes and proposes adding forward declarations, adding required includes (that would otherwise not be apparent due to the include order).

If installed and enabled (see below), `IWYU` is used to issue warnings if includes are missing or whenever constructs can be forward declared. Warnings that IWYU issues can be automatically fixed. Note, that this process is not entirely automatic, as recommendations of IWYU might lead to compile time errors, IWYU includes c-style headers, and utilizes “quotation” includes.

enable IWYU Enable the flag `ENABLE_IWYU` when running `CMake` via `-DENABLE_IWYU=ON`

Automatically apply IWYU's suggestions

Note, that this process is not entirely automatic, as recommendations of IWYU might lead to compile time errors, IWYU includes c-style headers, and utilizes “quotation” includes. Pipe the cmake output into a file, and execute

python fix_includes.py < /tmp/iwyu.out

Notes: When installing IWYU, you likely run into this issue. The code snipplet below lays out the problem and how I fixed it:

> clang -print-resource-dir
/usr/lib/llvm-10/lib/clang/10.0.0
> include-what-you-use -print-resource-dir
/usr/lib/clang/9.0.1
> sudo apt install libclang-9-dev

Sanitizers

For debugging Threading issues, memory leaks, UB and other kinds of undesired behavior, the are designated build types that you can utilize to run the corresponding Sanitizer. The specific flags are set by switching the `CMAKE_BUILD_TYPE` from Debug / Release to a name associated to the sanitizer you would like to use. In IDEs, these can be registered as `CMake Profiles`, when compiling set the `cmake` flags and `environment variables` manually. For the sake of providing a complete, copy-pastable chunk of options and environment variables, you can find the complete cmake options below (that repeat those that we use for Release builds).

Note: You can only enable one sanitizer configuration at a time.

UBSAN

Detects a lot of undefined behavior ( Comprehensive list).

Additional (potentially useful) Flags

Slowdown and memory overhead of this cmake profile is reported to be negligible. If you run into errors, try supplying the LLVM root directory `-DLLVM_ROOT_DIR='/usr/'`.

-DCMAKE_BUILD_TYPE=UB [NORMAL CMAKE FLAGS without BUILD_TYPE]
TSAN_OPTIONS=halt_on_error=0 UBSAN_OPTIONS=print_stacktrace=1 ASAN_OPTIONS=symbolize=1,fast_unwind_on_malloc=0,halt_on_error=0

ASAN

Leak and address sanitizers (report both memory errors and leak detection) Out of bounds accesses to heap, stack, global, Use after free, Use after return, Use after scope, Double-free, invalid free, Memory leaks.

Additional (potentially useful) Flags

Slowdown: 2x

Memory overhead: 3x Attention:

  1. The Address Sanitizer maps (but does not reserve) a lot of virtual address space. This means that tools like ulimit may not work as usually expected.
  2. Probably will produce error prone results because of non-instrumented dependencies.
  3. Libc/libstdc++ static linking is not supported.
-DCMAKE_BUILD_TYPE=Add -DASAN_OPTIONS=detect_leaks=1 [NORMAL CMAKE FLAGS without BUILD_TYPE]
TSAN_OPTIONS=halt_on_error=0 UBSAN_OPTIONS=print_stacktrace=1 ASAN_OPTIONS=symbolize=1,fast_unwind_on_malloc=0,halt_on_error=0

Memory Sanitizer

Detects uninitialized reads and conditional jumps based on uninitialized memory.

Additional (potentially useful) Flags

Slowdown: 3x - 6x (with origin tracking enabled).

Memory overhead: 2x - 3x

Attention:

  1. Make sure that llvm-symbolizer binary is in PATH, or set environment variable MSAN_SYMBOLIZER_PATH to point to it
  2. Probably will produce error prone results because of non-instrumented dependencies.
  3. Ordered List ItemLibc/libstdc++ static linking is not supported.
-DCMAKE_BUILD_TYPE=Mem [NORMAL CMAKE FLAGS without BUILD_TYPE]
TSAN_OPTIONS=halt_on_error=0 UBSAN_OPTIONS=print_stacktrace=1 ASAN_OPTIONS=symbolize=1,fast_unwind_on_malloc=0,halt_on_error=0

Thread Sanitizer

Detects race conditions. Additional (potentially useful) Flags.

Slowdown: 5x - 15x.

Memory overhead: 5x - 10x.

Attention

  1. ThreadSanitizer maps (but does not reserve) a lot of virtual address space. This means that tools like ulimit may not work as usually expected.
  2. Libc/libstdc++ static linking is not supported.
  3. Non-position-independent executables are not supported (clang++ acts as though -fPIE and -fPIC had been supplied).

Thread Sanitizer:

-DCMAKE_BUILD_TYPE=Thread [NORMAL CMAKE FLAGS without BUILD_TYPE]
TSAN_OPTIONS=halt_on_error=0 UBSAN_OPTIONS=print_stacktrace=1 ASAN_OPTIONS=symbolize=1,fast_unwind_on_malloc=0,halt_on_error=0

Testing

cd tests

ln -s ../nesCoordinator .

ln -s ../nesWorker .

make test_debug

Using Clion IDE - local development

Add the following configuration to “Cmake Options” configuration in the IDE

Release:

-DCMAKE_BUILD_TYPE=Release -DBoost_NO_SYSTEM_PATHS=TRUE -DBoost_INCLUDE_DIR="/usr/include" -DBoost_LIBRARY_DIR="/usr/lib/x86_64-linux-gnu" -DCPPRESTSDK_DIR="/usr/lib/x86_64-linux-gnu/cmake/" 

Debug:

-DCMAKE_BUILD_TYPE=Debug   -DBoost_NO_SYSTEM_PATHS=TRUE -DBoost_INCLUDE_DIR="/usr/include" -DBoost_LIBRARY_DIR="/usr/lib/x86_64-linux-gnu" -DCPPRESTSDK_DIR="/usr/lib/x86_64-linux-gnu/cmake/" 

Using Clion IDE with remote server

Compiling NebulaStream can be quite resource intensive. To enable development on low-end devices we provide a remote environment for TUB students.

For Windows

When working on a Windows client, keep in mind the following:

Due to the IntelliJ platform issue, 
you need to set the property value idea.case.sensitive.fs=true in the idea.properties file 
(to access the property file, select Help | Edit Custom Properties... on the main menu), 
then restart CLion with cache reset (File | Invalidate Caches / Restart...).

For files synchronization on Windows,
CLion relies on its own Remote Host Access and compression on the host side using the tar utility.
This mechanism causes the synchronization to perform slower than rsync on macOS and Linux.

Tutorial:

1. Ask NebulaStream team for an account on a remote server.

2. Clone NebulaStream repro on your local computer.

git clone git@github.com:nebulastream/nebulastream

3. Open local project in Clion

File/Open/$path

4. Add remote toolchain to Clion

4.1 Go to Settings / Preferences | Build, Execution, Deployment | Toolchains and click plus icon and select Remote Host from the drop-down menu to create a new toolchain.

4.2 Click the settings icon next to the Credentials field. In the dialog that opens, create an SSH configuration and provide the credentials for accessing your remote machine.

4.3 Go to Settings / Preferences | Build, Execution, Deployment | CMake, click plus icon to create a new CMake profile, and connect it to your remote toolchain using the Toolchain field.

Use the following CMake options CMake options

-DCMAKE_BUILD_TYPE=DEBUG -DBoost_NO_SYSTEM_PATHS=TRUE -DBoost_INCLUDE_DIR="/usr/include" -DBoost_LIBRARY_DIR="/usr/lib/x86_64-linux-gnu" -DCPPRESTSDK_DIR="/usr/lib/x86_64-linux-gnu/cmake/" -DLLVM_ROOT_DIR=/usr/

4.4 (optional)

If you want to avoid rebuilding the entire project when restarting the docker container:

  • Go to Settings / Preferences | Build, Execution, Deployment | Deployment
  • In the Connection tab: Set Root path = /nebulatream
  • In the Mappings tab: Set Deployment path = build-tmp
  • In the Excluded Paths tab: Add local path <path-to-nebulastream-repo>/build-tmp

4.5 Rebuild Project, Clion will start to upload the project to the remote server. This may take some time

Further Infors: https://www.jetbrains.com/help/clion/remote-projects-support.html#remote-toolchain

Build Using Docker

Standard Dependences:

docker-ce

Build and Test Command

docker run -v <path_to_project>/:/nebulastream nebulastream/nes-build-image

or

docker run -v <path_to_project>/:/nebulastream -d nebulastream/nes-build-image

for building in the background.

Build and Test Cross-Compiled

docker run -v <path_to_project>/:/nebulastream nebulastream/cross-compile

or

docker run -v <path_to_project>/:/nebulastream -d nebulastream/cross-compile` for building in the background. 

Additionally, in CLion set the following CMake options for a better debugging experience

-DCMAKE_BUILD_TYPE=Debug -DCPPRESTSDK_DIR="/usr/lib/x86_64-linux-gnu/cmake/" -DNES_USE_OPC=0 -DNES_USE_ADAPTIVE=0 -DNES_BUILD_BENCHMARKS=0 -DNES_SPECIFIC_FLAGS=-fstandalone-debug -DNES_LOGGING_LEVEL=NES_DEBUG -NES_BENCHMARKS_FLAG_MODE=0 -DCMAKE_ENABLE_JEMALLOC=0

Special CMAKE Flags

NES_USE_KAFKA 

: NES will be built using Kafka (0/1)

We introduced jemalloc as a more efficient implementation of malloc [1]. However, our default behaviour is to fallback on libc allocator. If one wants to use jemalloc, it is necessary to install

libjemalloc-dev

and add the following cmake flag

-DCMAKE_ENABLE_JEMALLOC=ON

[1] http://jemalloc.net/

 
how_to_build_nes.txt · Last modified: 2021/11/10 12:17 by 134.96.191.189
 
Recent changes RSS feed Creative Commons License Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki