log4j直接输出日志到flume 此jar是由Cloudera的CDH发行版提供的一个工具类，通过配置，可以将log4j的日志直接输出到flume，方便日志的采集。 在CDH5.3.0版本中是：fl

log4j直接输出日志到flume

        此jar是由Cloudera的CDH发行版提供的一个工具类，通过配置，可以将log4j的日志直接输出到flume，方便日志的采集。

        在CDH5.3.0版本中是：flume-ng-log4jappender-1.5.0-cdh5.3.0-jar-with-dependencies.jar

        所在目录是：/opt/cloudera/parcels/CDH/lib/flume-ng/tools/

具体使用示例

log4j配置（log4j.properties)

log4j.category.com.xxx=INFO,console,flumelog4j.appender.console=org.apache.log4j.ConsoleAppenderlog4j.appender.console.target=System.outlog4j.appender.console.layout=org.apache.log4j.PatternLayoutlog4j.appender.console.layout.ConversionPattern="%d{yyyy-MM-dd HH:mm:ss} %p [%c:%L] - %m%nlog4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppenderlog4j.appender.flume.Hostname = localhostlog4j.appender.flume.Port = 4444log4j.appender.flume.UnsafeMode = truelog4j.appender.flume.layout=org.apache.log4j.PatternLayoutlog4j.appender.flume.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %p [%c:%L] - %m%n

配置classpath

        在classpath中要包含log4j.properties和flume-ng-log4jappender-1.5.0-cdh5.3.0-jar-with-dependencies.jar

编写JAVA测试类并导出为test.jar 

import java.util.Date;import org.apache.commons.logging.Log;import org.apache.commons.logging.LogFactory;public class WriteLog {    protected static final Log logger = LogFactory.getLog(WriteLog.class);    public static void main(String[] args) throws InterruptedException {        // TODO Auto-generated method stub        while (true) {            // 每隔两秒log输出一下当前系统时间戳            logger.info(new Date().getTime());            Thread.sleep(2000);            try {                throw new Exception("exception msg");            }            catch (Exception e) {                logger.error("error:" + e.getMessage());            }        }    }}

编写flume agent配置文件

a1.sources = r1a1.sinks = k1a1.channels = c1 # Describe/configure the sourcea1.sources.r1.type = avroa1.sources.r1.bind = localhosta1.sources.r1.port = 4444 # Describe the sinka1.sinks.k1.type = file_rolla1.sinks.k1.sink.directory = /data/soft/flume/tmpa1.sinks.k1.sink.rollInterval=86400a1.sinks.k1.sink.batchSize=100a1.sinks.k1.sink.serializer=texta1.sinks.k1.sink.serializer.appendNewline = false # Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 1000 # Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1

运行程序

        将相关程序上传到服务器，并先启动agent

        进入flume安装目录后，执行

      ‍  bin/flume-ng agent -c conf -f conf/avro.conf --name a1 -Dflume.root.logger=INFO,console‍

        执行测试程序

        java -classpath ./:flume-ng-log4jappender-1.5.0-cdh5.3.0-jar-with-dependencies.jar:test.jar com.xxx.WriteLog

检查运行结果

        tail -f /data/soft/flume/tmp/1436164166461-1 

2015-07-06 14:51:36 ERROR [com.xxx.WriteLog:27] - error:exception msg2015-07-06 14:51:36 ERROR [com.xxx.WriteLog:28] - error:stack2015-07-06 14:51:36 INFO [com.xxx.WriteLog:21] - 14361654969752015-07-06 14:51:38 ERROR [com.xxx.WriteLog:27] - error:exception msg2015-07-06 14:51:38 ERROR [com.xxx.WriteLog:28] - error:stack2015-07-06 14:51:38 INFO [com.xxx.WriteLog:21] - 1436165498977