Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

From the Chip to the Cloud with Apache Software

This talk gives an overview over a lot of the Apache projects which one would consider important for building IoT and especially Industrial IoT (IIoT) solutions.

  • Be the first to comment

From the Chip to the Cloud with Apache Software

  1. 1. APACHE IOTAPACHE IOT A trip from the chip to the cloud Christofer Dutz < > Building IoT 2020 (Essen) 2020-03-03 christofer.dutz@codecentric.de 1
  2. 2. WHO AM I?WHO AM I? Christofer Dutz Senior IT Consultant codecentric AG Open-Source Enthusiast Committer of 10 Apache projects Member of the Apache Foundation VP of Apache PLC4X Twitter: @ChristoferDutz 2
  3. 3. FIRST STEPS IN IOTFIRST STEPS IN IOT Figure 1. https://www.youtube.com/watch?v=bF7_2FS3j1M 3
  4. 4. FIRST STEPS IN IOTFIRST STEPS IN IOT Figure 2. https://klaus.merkert.info/computer/ft/ 4
  5. 5. FIRST STEPS IN IOTFIRST STEPS IN IOT Figure 3. From the official manual. 5
  6. 6. INTRODUCTIONINTRODUCTION IoT is everywhere Things have become quite a bit easier Open-Source, but also Open-Hardware New Projects being started every day All the major foundations have "IoT Divisions" Hard to keep track of everything that’s happening 6
  7. 7. AGENDAAGENDA Take you on a journey from the chip via the edge through the fog into the cloud Focus on Apache projects Outreach to Other projects and foundations Discussion 7
  8. 8. PROJECTS WE’LL BE TALKING ABOUTPROJECTS WE’LL BE TALKING ABOUT Apache MyNewt Apache PLC4X Apache Camel Apache NiFi Apache Spark Apache Flink Apache Beam Apache Hadoop Apache Cassandra Apache IoTDB Apache Kafka Apache StreamPipes 8
  9. 9. APACHE MYNEWTAPACHE MYNEWT 9
  10. 10. APACHE MYNEWTAPACHE MYNEWT First Apache (Rt)OS "Linux for platforms too small to run Linux" Multiple layers of abstraction Allows developing software for a wide range of chips Built for Wireless Bluetooth (Mesh), LoRaWan, … Modular Drivers Built with security as main principle Secure Bootloader 10
  11. 11. APACHE MYNEWTAPACHE MYNEWT Figure 4. Source Amazon 11
  12. 12. APACHE MYNEWTAPACHE MYNEWT Figure 5. Source Ruuvi 12
  13. 13. APACHE PLC4XAPACHE PLC4X 13
  14. 14. APACHE PLC4XAPACHE PLC4X — Apache PLC4X Project Statement PLC4X is a set of libraries for communicating with industrial programmable logic controllers (PLCs) using a variety of protocols but with a shared API. 14
  15. 15. APACHE PLC4XAPACHE PLC4X Develop only using the API module Develop software for any type of PLC Highly inspired by JDBC Integration modules to other open-source SW Strongly growing number of supported protocols Soon also non Java drivers 15
  16. 16. APACHE PLC4XAPACHE PLC4X 16
  17. 17. APACHE CAMELAPACHE CAMEL 17
  18. 18. APACHE CAMELAPACHE CAMEL Integration framework Message oriented middleware Based on "Enterprise Integration Patterns" Book Integration into all major component frameworks Build pipelines Producers Transformations Consumers 18
  19. 19. APACHE NIFIAPACHE NIFI 19
  20. 20. APACHE NIFIAPACHE NIFI System for routing and processing streams of data Web-UI for modeling data streams Tracking of data through the pipelines Handling of back pressure Secure 20
  21. 21. APACHE NIFIAPACHE NIFI 21
  22. 22. APACHE MINIFIAPACHE MINIFI Runtime for running parts of NiFi streams on small edge devices Operations on edge devices Aggregation Filtering Translation … Secure communication with NiFi cluster No UI 22
  23. 23. APACHE SPARKAPACHE SPARK 23
  24. 24. APACHE SPARKAPACHE SPARK Batch processing Stream processing Runs in almost all cluster environments 24
  25. 25. APACHE FLINKAPACHE FLINK 25
  26. 26. APACHE FLINKAPACHE FLINK Stateful stream analytics Bounded and unbounded data streams Runs in almost all cluster environments 26
  27. 27. APACHE BEAMAPACHE BEAM 27
  28. 28. APACHE BEAMAPACHE BEAM Framework for running batch jobs Framework for running streaming "jobs" Abstracts from the actual runtime Runs "jobs" in: Spark Flink Hadoop Google Cloud … 28
  29. 29. APACHE HADOOPAPACHE HADOOP 29
  30. 30. APACHE HADOOPAPACHE HADOOP Big Data storage Unstructured Data Distributed storage and processing Processing of huge datasets MapReduce 30
  31. 31. APACHE CASSANDRAAPACHE CASSANDRA 31
  32. 32. APACHE CASSANDRAAPACHE CASSANDRA Big Data storage Table storage Linear scalability Distributed storage Managing of huge datasets Decentralized 32
  33. 33. APACHE IOTDB (INCUBATING)APACHE IOTDB (INCUBATING) 33
  34. 34. APACHE IOTDB (INCUBATING)APACHE IOTDB (INCUBATING) Time series DB Lightweight High performance Compressed data Ability to deal with IoT data Ingest data out of sequence Ingest data while being offline Ingest data on extremely small devices Integration with Hadoop & Spark 34
  35. 35. APACHE KAFKAAPACHE KAFKA 35
  36. 36. APACHE KAFKAAPACHE KAFKA Streaming Framework Able to route enormous amounts of data Fault tolerant Kafka Streams Processing of data Kafka Connect Ingestion of massive amounts of data (Output massive amounts of data) 36
  37. 37. APACHE KAFKAAPACHE KAFKA However doesn’t like: Bad networks Small (Edge) devices Solution Use MQTT on Edge devices Ingest MQTT data into Kafka cluster 37
  38. 38. APACHE STREAMPIPES (INCUBATING)APACHE STREAMPIPES (INCUBATING) 38
  39. 39. APACHE STREAMPIPES (INCUBATING)APACHE STREAMPIPES (INCUBATING) Self-Service IoT Data Analytics Platform Focus on data streams Similar to Apache NiFi Focus on (production) Industry Focus on being run by OT Focus on non-technical users (domain experts) Hides the complexity of modern IT solutions Uses Machine-learning to guess semantics of data 39
  40. 40. APACHE STREAMPIPES (INCUBATING)APACHE STREAMPIPES (INCUBATING) 40
  41. 41. APACHE STREAMPIPES (INCUBATING)APACHE STREAMPIPES (INCUBATING) 41
  42. 42. OUTREACH TO OTHER FOUNDATIONSOUTREACH TO OTHER FOUNDATIONS Eclipse Foundation Appears rather unstructured 36 projects in total Linux Foundation LF-Edge Initiative 7 Projects in total EdgeX Foundry EvE HiveMQ 42
  43. 43. THANKS FOR LISTENINGTHANKS FOR LISTENING Questions? Suggestions? Discussions! For news: Subscribe to mailing list: Follow me on Twitter: @ChristoferDutz iot@apache.org 43

×