Streaming Data Architectures: An Introduction to Streaming Data in Spark

Skillsoft issued completion badges are earned based on viewing the percentage required or receiving a passing score when assessment is required. Spark, an analytics engine built on Hadoop, can be used for working with big data, data science, and processing batch and streaming data. Explore the fundamentals of working with streams using Spark in this 9-video course. Key concepts covered here include the differences between batch and streaming data and the types of streaming data sources; processing streaming data, transformation of streams, and materialization of the results of the transformation; and how use of a message transport decouples a streaming application from the sources of streaming data. Next, learn about techniques used in Spark 1.x to work with streaming data and how it contrasts with processing batch data; how structured streaming in Spark 2.x is able to ease the task of stream processing for the app developer; and how streaming processing works in both Spark 1.x and 2.x. Finally, learn how triggers can be set up to periodically process streaming data and the various output modes available to publish the results of stream processing; and the key aspects of working with structured streaming in Spark.