Citam si dokumentaciu k Apache Sparku a vrta mi hlavou, ako je vnutorne naimplementavana tato vec, pozorne si precitajte ten kod, nie je tazke to pochopit co to robi.
Ide ale o to, ze na konci je volanie metody scc.start() co znamena, ze vsetko co som napisal za kod predtym defakto vobec nebezal, pretoze tymto start() ho spustim.
Ako ale dokazem toto vnutorne naimplementovat? Ono to potom znamena, ze mi to vrati vzdy nejaky medzi objekt na ktorom sa vola ta dalsia metoda? Nie je to nahodou skorej tak, ze si "ukladam" dovnutra ake metody spustim, ked zavolam ten start()?
val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount")
val ssc = new StreamingContext(conf, Seconds(1))
// Create a DStream that will connect to hostname:port, like localhost:9999
val lines = ssc.socketTextStream("localhost", 9999)
// Split each line into words
val words = lines.flatMap(_.split(" "))
val pairs = words.map(word => (word, 1))
val wordCounts = pairs.reduceByKey(_ + _)
Note that when these lines are executed, Spark Streaming only sets up the computation it will perform when it is started, and no real processing has started yet. To start the processing after all the transformations have been setup, we finally call
// Print the first ten elements of each RDD generated in this DStream to the console
wordCounts.print()
ssc.start() // Start the computation
ssc.awaitTermination() // Wait for the computation to terminate
4 Odpovědí
1096 Zhlédnutí