配置spark-streaming读取flume数据

int32位 posted @ Mar 22, 2016 03:46:23 PM in hadoop/spark , 2596 阅读

转载请注明：http://krystism.is-programmer.com/若有错误，请多多指正，谢谢！

1.依赖配置

由于我们使用sbt构建项目，因此所有依赖库写入build.sbt的libraryDependencies即可，格式为groupId % artifactId % version，具体字段含义建议参考maven.

我们这次代码除了需要spark-core外，还需要第三方库spark-streaming-flume,因此build.sbt 大致内容为:

name := "FlumeEventCount"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.0"

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.0"

libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.10" % "1.4.0"

注意build.sbt行与行之间要有空行，这是语法要求！

2.测试代码

通过调用FlumeUtils.createStream()方法创建flume流，本次测试仅仅统计每次(每隔2秒)获取的数据行数(事件数)，代码为:

package com.huawei.test

import org.apache.spark.SparkConf
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming._
import org.apache.spark.streaming.flume._
import org.apache.spark.util.IntParam

/**
 *  Produces a count of events received from Flume.
 *
 *  This should be used in conjunction with an AvroSink in Flume. It will start
 *  an Avro server on at the request host:port address and listen for requests.
 *  Your Flume AvroSink should be pointed to this address.
 *
 *  Usage: FlumeEventCount <host> <port>
 *    <host> is the host the Flume receiver will be started on - a receiver
 *           creates a server and listens for flume events.
 *    <port> is the port the Flume receiver will listen on.
 *
 *  To run this example:
 *    `$ bin/run-example org.apache.spark.examples.streaming.FlumeEventCount <host> <port> `
 */
object FlumeEventCount{
  def main(args: Array[String]) {
    if (args.length < 2) {
      System.err.println(
        "Usage: FlumeEventCount <host> <port>")
      System.exit(1)
    }

    val host = args(0)
    val port = args(1).toInt

    val batchInterval = Milliseconds(2000)

    // Create the context and set the batch size
    val sparkConf = new SparkConf().setAppName("FlumeEventCount")
    val ssc = new StreamingContext(sparkConf, batchInterval)

    // Create a flume stream
    val stream = FlumeUtils.createStream(ssc, host, port, StorageLevel.MEMORY_ONLY_SER_2)

    // Print out the count of events received from this server in each batch
    stream.count().map(cnt => "Received " + cnt + " flume events." ).print()

    ssc.start()
    ssc.awaitTermination()
  }
}

3.配置flume

只需要把sink配置成SparkSink即可

agent.sinks = spark
agent.sinks.spark.type = org.apache.spark.streaming.flume.sink.SparkSink
agent.sinks.spark.hostname = <hostname of the local machine>
agent.sinks.spark.port = <port to listen on for connection from Spark>
agent.sinks.spark.channel = memoryChannel

## 4.打包程序

cd $PROJECT_ROOT # PROJECT_ROOT为项目根路径，即build.sbt的位置
ant package

## 5.运行

注意：除了spark-core依赖包会由spark-submit自动引入，其他依赖包比如 spark-streaming-flume必须手动引入:

设置CLASSPATH,把依赖包放入CLASSPATH中
使用--jars参数手动加入

此次测试采用后种方法，即使用--jars参数。这个工程只差spark-streaming-flume包，sbt编译时已经自动下载到本地，位于~/.ivy2/cache/org.apache.spark/spark-streaming-flume_2.10/jars, 把所有的jar包复制到工程的lib目录下.

cp ~/.ivy2/cache/org.apache.spark/spark-streaming-flume_2.10/jars/*.jar lib

使用spark-submit 提交程序，为了避免每次重复输入命令，写一个脚本用于提交:

#!/bin/sh
spark-submit --master local[*] --class com.huawei.test.FlumeEventCount\
--jars lib/*.jar\
target/scala-2.10/flumeeventcount_2.10-1.0.jar localhost 50000

其中localhost表示flume写入的主机名，50000表示flume写入端口

6.运行结果

当flume有数据流时，程序会捕捉事件，统计每次的事件总数。

6.运行结果

当flume有数据流时，程序会捕捉事件，统计每次的事件总数。

-------------------------------------------
Time: 1436942874000 ms
-------------------------------------------
Received 1345 flume events.

-------------------------------------------
Time: 1436942876000 ms
-------------------------------------------
Received 2132 flume events.

-------------------------------------------
Time: 1436942878000 ms
-------------------------------------------
Received 0 flume events.

转载请注明：http://krystism.is-programmer.com/若有错误，请多多指正，谢谢！

相关文章
全局相关文章

无匹配

无匹配

[回复]

Alyssa 说:
2022年12月26日 21:48

There are a few steps to configuring spark-streaming to read flume data. The first is to set up a flume agent on each node that will be sending data. The next step is to define a engagement rings spark streaming context that will read data from the flume agent. Finally, it is necessary to define a flume sink that will write the data to spark.

[回复]

Bushra 说:
2023年4月04日 00:23

i love reading this article so beautiful!!great job! Relevant research

[回复]

Bushra 说:
2023年4月04日 00:37

i love reading this article so beautiful!!great job! Read Much More

[回复]

Bushra 说:
2023年4月08日 23:55

Thanks for your information, it was really very helpfull.. pressure washing services near Gibsonia PA

[回复]

Bushra 说:
2023年4月09日 04:02

thanks for this usefull article, waiting for this article like this again. Read Much More

[回复]

Bushra 说:
2023年4月09日 04:24

This is very educational content and written well for a change. It's nice to see that some people still understand how to write a quality post! Click Through to the Following Page

[回复]

amarnath helicopter 说:
2024年6月05日 07:21

Cool stuff you have and you keep overhaul every one of us

[回复]

tech 说:
2024年6月06日 21:20

Writing with style and getting good compliments on the article is quite hard, to be honest.But you've done it so calmly and with so cool feeling and you've nailed the job. This article is possessed with style and I am giving good compliment. Best! تأشيرة تركيا للمواطنين الجامايكي

[回复]

tech 说:
2024年6月23日 19:35

Great survey. I'm sure you're getting a great response. Wikipedia Links

[回复]

tech 说:
2024年6月30日 19:21

Cool stuff you have and you keep overhaul every one of us Edu Backlinks

[回复]

tech 说:
2024年8月31日 10:24

Good website, where did you find the details for this post? Still, I'm glad I found it, and I'll be returning soon to see what other postings you have. أهلية التأشيرة الهندية لمواطني ليختنشتاين

[回复]

tech 说:
2024年9月23日 12:58

You truly have a gift for creating original stuff. I appreciate the way you articulate your ideas in this piece and the way you think. Your writing style really strikes me. I appreciate you adding to the beauty of my experience. obywatele Czarnogóry uprawnieni do indyjskiej wizy elektronicznej

[回复]

tech 说:
2024年10月16日 05:16

I appreciate your great blog. Where else might I get information conveyed in such a flawless manner? I have a project that I am currently working on, and I have been watching for such info. elegibilitat de visat a l'Índia visat a l'Índia per a ciutadans de Kirguizistan

[回复]

seo 说:
2024年11月16日 19:59

New York fake IDs are often sought after, but using them risks serious legal consequences and fines. PENNSYLVANIA FAKE ID

[回复]

tech 说:
2024年12月02日 00:34

Thank you for writing; I understand precisely what you mean. Furthermore, I'm overjoyed to have found this page on Google. indian e visa eligibility timorese citizens

[回复]

seo 说:
2024年12月04日 15:55

Nexus Market offers innovative solutions, connecting businesses with a diverse range of products for efficient growth nexus market

[回复]

tech 说:
2025年1月04日 06:18

More readers would be interested in the works of authors if more of them took the time to create excellent material, as you do. I've gained much too much knowledge from your post. nl Cambodja visum voor Uruguayaanse burgers

[回复]

tech 说:
2025年1月10日 00:06

The article's emphasis on dispelling myths surrounding Bitcoin mixers promotes a more accurate understanding of their purpose and legality. Tonic Greens

[回复]

tech 说:
2025年1月23日 00:35

It's great to be returning to your blog after months away. It's appropriate that I've been waiting so long for this piece. This article and yours are on the same topic, thus I want to use it to complete my professor project. Many thanks for your very helpful and kind assistance. vízum na Srí Lanku pre občanov Nového Zélandu

[回复]

tech 说:
2025年2月12日 04:49

Your blog has truly captured my interest. I can see why, as you've done an excellent job of making it engaging. Your efforts are truly commendable. https: www.indiasvisa.org india visa online

[回复]

tech 说:
2025年2月18日 00:33

Your blog has truly captured my interest. I can see why, as you've done an excellent job of making it engaging. Your efforts are truly commendable. https: www.indiasvisa.org india visa online

[回复]

tech 说:
2025年3月21日 09:29

Can’t wait to see more from you, this is incredible how to avoid and identify travel visa scams

[回复]

tech 说:
2025年5月18日 06:21

This typography is absolutely stunning 토닥이

[回复]

tech 说:
2025年5月20日 19:10

The way you’ve balanced the colors is perfect olxtoto login

[回复]

pestoto login 说:
2025年5月20日 20:46

I high appreciate this post. It’s hard to find the good from the bad sometimes, but I think you’ve nailed it! would you mind updating your blog with more information?

[回复]

여성전용마사지 说:
2025年5月22日 16:49

힐링이 필요할땐 여성전용마사지를 찾아보세요 여성전용으로 토닥이 서비스와 마사지를 함께 느끼실 수 있습니다 간편하면서도 빠른 예약 지금 해보세요 <a href="https://todaki.isweb.co.kr/">여성전용마사지</a>

[回复]

여성전용마사지 说:
2025年5月22日 16:49

힐링이 필요할땐 여성전용마사지를 찾아보세요 여성전용으로 토닥이 서비스와 마사지를 함께 느끼실 수 있습니다 간편하면서도 빠른 예약 지금 해보세요

[回复]

olxtoto 说:
2025年5月26日 21:11

You've done a fantastic job writing this piece. It is quite sophisticated and easily reading. Even better, you've made it simple to read and easy to grasp. You're a really talented writer. I'm grateful.

[回复]

hargatoto 说:
2025年5月27日 17:30

This type of message always inspiring and I prefer to read quality content, so happy to find good place to many here in the post, the writing is just great, thanks for the post.

[回复]

tech 说:
2025年5月27日 21:10

We much appreciate your willingness to share information with us. You have made my task as easy as ABC, therefore we will always be in awe of what you have accomplished here. hargatoto

[回复]

link olxtoto 说:
2025年5月31日 17:50

This is just the information I am finding everywhere. Thanks for your blog, I just subscribe your blog. This is a nice blog..

[回复]

tech 说:
2025年6月10日 20:43

구글 아이디나 구글 계정 판매사이트를 찾으시나요? 이 사이트에서 저렴한 가격에 대량 소량 구입 하실 수 있습니다! 구글 아이디 판매사이트

[回复]

tech 说:
2025年6月14日 19:04

인천 지역의 여성들이 편하게 이용할 수 있는 프리미엄 인천여성전용마사지 입니다. 인천여성전용마사지

[回复]

부산여성전용마사지 说:
2025年6月15日 20:20

부산에 위치한 부산여성전용마사지를 이용해보세요 프리미엄 대한민국 최고의 부산 여성전용 힐링 출장 마사지 공간입니다. 강력추천 <a href="https://busantodaki.isweb.co.kr/">부산여성전용마사지</a>

[回复]

부산여성전용마사지 说:
2025年6月15日 20:21

부산에 위치한 부산여성전용마사지를 이용해보세요 프리미엄 대한민국 최고의 부산 여성전용 힐링 출장 마사지 공간입니다. 강력추천

[回复]

소액결제현금화 说:
2025年6月25日 05:29

소액결제현금화는 현금이 급한분들에게 안성맞춤이며, 다양한 서비스로 정보이용료 또는 대출이 불가능할때 급하게 쓰는 서비스입니다.

[回复]

saad 说:
2025年6月25日 20:45

Very efficiently written information. It will be beneficial to anybody who utilizes it, including me. Keep up the good work. For sure i will check out more posts. This site seems to get a good amount of visitors. Facebook

[回复]

비아그라 구매 说:
2025年6月29日 21:05

다음은 비아그라 구매와 관련된 주요 법규 사항과 위반 시 제재 내용입니다 의약품관리법 준수: 대한민국에서는 「의약품관리법」을 통해 의약품의 제조·유통·판매를 엄격히 규제하고 있습니다

[回复]

구리가라오케 说:
2025年7月01日 00:50

구리지역의 최고의 가라오케를 소개합니다, 유명한 구리가라오케를 검색하신 고객여러분 환영합니다, 최고의 음향시설을 느껴보세요!

[回复]

tech 说:
2025年7月02日 01:53

I recently started a blog, and the information you provide on your website has been really helpful and encouraging. I appreciate all of your time and effort. vivogut

[回复]

다산룸싸롱 说:
2025年7月12日 07:18

다산룸싸롱 수준의 품격과 노래방의 순수한 즐거움을 모두 갖춘 저희 다산가라오케를 방문해보세요. 최고의 시설과 서비스로 언제나 여러분을 맞이할 준비가 되어 있습니다

[回复]

olxtoto 说:
2025年7月12日 09:49

Excellent work! I want everyone to be aware of how excellent the information in your piece is. In regards to this matter, your opinions and mine are very similar. I know, so I'll check out your blog every day. For me, that may be really advantageous.

[回复]

비아그라 구매 说:
2025年8月11日 21:49

Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed, but also entertained!

[回复]