博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
flume 的source 、channel和sink 多种组合
阅读量:4186 次
发布时间:2019-05-26

本文共 3113 字,大约阅读时间需要 10 分钟。

flume 有三大组件source 、channel和sink,各个组件之间都可以相互组合使用,各组件间耦合度低。使用灵活,方便。

1.多sink

channel 的内容只输出一次,同一个event 如果sink1 输出,sink2 不输出; 最终 sink1+sink2=channel 中的数据。

配置文件如下:

a1.sources = r1a1.sinks = k1 k2a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = execa1.sources.r1.shell = /bin/bash -ca1.sources.r1.channels = c1a1.sources.r1.command = tail -F /opt/apps/logs/tail4.log# channela1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100#sink1a1.sinks.k1.channel = c1a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.kafka.topic = mytopica1.sinks.k1.kafka.bootstrap.servers = localhost:9092a1.sinks.k1.kafka.flumeBatchSize = 20a1.sinks.k1.kafka.producer.acks = 1a1.sinks.k1.kafka.producer.linger.ms = 1a1.sinks.ki.kafka.producer.compression.type = snappy#sink2a1.sinks.k2.type = file_rolla1.sinks.k2.channel = c1#a1.sinks.k2.sink.rollInterval=0a1.sinks.k2.sink.directory = /opt/apps/tmp

  • 2.多channel 多sink ,每个sink 输出内容一致

memory channel 用于kafka操作,实时性高,file channel 用于 sink file 数据安全性高) 

(多channel 单 sink 的情况没有举例,个人感觉用处不广泛。)

配置文件如下:

a1.sources = r1a1.sinks = k1 k2a1.channels = c1 c2# Describe/configure the sourcea1.sources.r1.type = execa1.sources.r1.shell = /bin/bash -ca1.sources.r1.channels = c1 c2a1.sources.r1.command = tail -F /opt/apps/logs/tail4.log#多个channel 的数据相同a1.sources.r1.selector.type=replicating# channel1a1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100#channel2a1.channels.c2.type = filea1.channels.c2.checkpointDir = /opt/apps/flume-1.7.0/checkpointa1.channels.c2.dataDirs = /opt/apps/flume-1.7.0/data#sink1a1.sinks.k1.channel = c1a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.kafka.topic = mytopica1.sinks.k1.kafka.bootstrap.servers = localhost:9092a1.sinks.k1.kafka.flumeBatchSize = 20a1.sinks.k1.kafka.producer.acks = 1a1.sinks.k1.kafka.producer.linger.ms = 1a1.sinks.ki.kafka.producer.compression.type = snappy#sink2a1.sinks.k2.type = file_rolla1.sinks.k2.channel = c2#a1.sinks.k2.sink.rollInterval=0a1.sinks.k2.sink.directory = /opt/apps/tmp

3. 多source 单 channel 单 sink

多个source 可以读取多种信息放在一个channel 然后输出到同一个地方 

配置文件如下:

a1.sources = r1 r2a1.sinks = k1a1.channels = c1# source1a1.sources.r1.type = execa1.sources.r1.shell = /bin/bash -ca1.sources.r1.channels = c1a1.sources.r1.command = tail -F /opt/apps/logs/tail4.log# source2a1.sources.r2.type = execa1.sources.r2.shell = /bin/bash -ca1.sources.r2.channels = c1a1.sources.r2.command = tail -F /opt/apps/logs/tail2.log# channel1  in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100#sink1a1.sinks.k1.channel = c1a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.kafka.topic = mytopica1.sinks.k1.kafka.bootstrap.servers = localhost:9092a1.sinks.k1.kafka.flumeBatchSize = 20a1.sinks.k1.kafka.producer.acks = 1a1.sinks.k1.kafka.producer.linger.ms = 1a1.sinks.ki.kafka.producer.compression.type = snappy

flume 像乐高积木一样可以自己随心所欲将不同的组件进行搭配使用,耦合度低。

版权声明:本文为博主编写文章,未经博主允许转载,转载请注明出处。 https://blog.csdn.net/u012373815/article/details/54351323
你可能感兴趣的文章
商务智能-基本方法-数据钻取
查看>>
openstack ice resize 详解(三)
查看>>
事务与锁(转)
查看>>
Namenode HA原理详解(脑裂)
查看>>
Differences between VMware FT and HA(转)
查看>>
Cloud Prizefight: OpenStack vs. VMware(转)
查看>>
亚马逊Auto Scaling
查看>>
openstack-instance-high-availability-Evacuate
查看>>
evacuate-instance-automatically
查看>>
pycharm常用设置(keymap设置及eclipse常用快捷键总结)
查看>>
关于在openstack的环境变量.bashrc自定自己简化命令
查看>>
Openstack Heat Project介绍(转)
查看>>
How to Perform an Upgrade from Icehouse to Juno(ice升级到juno)
查看>>
高扩展性网站的50条原则(转)-思维导图
查看>>
解决openstack novnc一段时间后自动挂断登录不上问题,novncproxy dead but pid file exists
查看>>
构建OpenStack的云基础架构:ManageIQ(转)
查看>>
云管理软件 ManageIQ(转)
查看>>
CentOS 7.0,启用iptables防火墙(转)
查看>>
svn忽略ignore文件记住方式(转)
查看>>
web缓存相关知识(转)
查看>>