I had the pleasure of participating to the EBSIS summer school and give a lecture about the data streaming processing paradigm and its use in Fog architectures. The lecture introduces the data analysis challenges and trade-offs in Fog architectures, overviews the data streaming processing paradigm (with links to SPEs such as Apache Storm, Apache Flink and Apache Spark) and introduces several use cases focusing on distributed analysis, parallel analysis and deterministic processing (among others). You can find the slides here.

Continuing my effort of keeping track of useful and short guides (more for myself than others), here you have the one to setup Apache Flink quickly.

Note: this installation does not require sudo, is for a single-node cluster (with JAVA_HOME set and Java up to date) with out-of-the-box configuration and DOES NOT use HDFS.

Connect to your server
ssh -L 8081:localhost:8081 username@server

Create a directory for storm and enter it
mkdir flink
cd flink

Download Apache Flink and unzip it
wget http://apache.mirrors.spacedump.net/flink/flink-1.0.3/flink-1.0.3-bin-hadoop1-scala_2.10.tgz (or the appropriate version)
tar -xvf flink-1.0.3-bin-hadoop1-scala_2.10.tgz

Start cluster
./bin/start-cluster

Open your web browser and go to localhost:8081. Do you see the UI? good…

Notes:

• Have a look at conf/flink-conf.yaml and conf/slaves and  to understand (i) what is going on and (ii) why the installation is so straightforward (read: what parameters are already set for you)
• Read https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/cluster_setup.html!

Very good news! Our paper “Highly Concurrent Stream Synchronization in Many-core Embedded Systems” has been accepted at the fourth ACM International Workshop on Manycore Embedded Systems (MES 2016).

The abstract follows:
Embedded many-core architectures are expected to serve as significant components in the infrastructure of upcoming technologies like networks for the Internet of Things (IoT), facing real-time and stream processing challenges. In this work we explore the applicability of ScaleGate, a synchronization object from the massive data stream processing domain, on many-core embedded systems. We propose a new implementation of ScaleGate on the Epiphany architecture, a scalable embedded many-core co-processor, and study communication patterns that appear in the context of a baseband signal processing application. Our experimental evaluation shows significant improvements over standard barrier-based approaches, due to the asynchrony exploited by the use of ScaleGate.