發布時間： 2019-09-18 17:06:15
?、邸hannel：angent 內部的數據傳輸通道，用于從 source
flume 接收telnet網絡數據
vi netcat-logger.conf # 定義這個agent中各組件的名字 a1.sources = r1 a1.sinks = k1 a1.channels = c1 # 描述和配置source組件：r1 a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444 # 描述和配置sink組件：k1 a1.sinks.k1.type = logger # 描述和配置channel組件，此處使用是內存緩存的方式 a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # 描述和配置source channel sink之間的連接關系 a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 |
啟動Flume
?[hd@Mas?ter flume]# flume-ng agent -c /home/hd/apps/flume/conf/ -f example/netcat-logger.conf -n a1 -Dflume.root.logger=INFO,console
?Flume采集日志文件到HDFS
# Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source #監聽目錄,spoolDir指定目錄, fileHeader要不要給文件夾前墜名 a1.sources.r1.type = spooldir a1.sources.r1.spoolDir = /home/hadoop/flumespool a1.sources.r1.fileHeader = true # Describe the sink a1.sinks.k1.type = logger # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 |
啟動Flume
?[hd@ma?ster flume]# flume-ng agent -c /home/hd/apps/flume/conf/ -f example/spool-logger.conf -n a1 -Dflume.root.logger=INFO,console
上一篇： 人工智能AI培訓_機器學習之特征工程
下一篇： 大數據培訓_Hive 常用函數
微信
公眾號
