想把hadoop的进程日志导入hive表进行分析，遂做了以下的尝试。
关于hadoop进程日志的解析
使用正则表达式获取四个字段，一个是日期时间，一个是日志级别，一个是类，最后一个是详细信息， 然后在hive中建一个表，可以用来方便查询。2015-12-18 22:23:23,357 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 32652 for container-id container_1448915696877_26289_01_000158: 110.6 MB of 2 GB physical memory used; 2.1 GB of 4.2 GB virtual memory used
2015-12-18 22:23:23,426 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 32615 for container-id container_1448915696877_26289_01_000102: 104.6 MB of 2 GB physical memory used; 2.1 GB of 4.2 GB virtual memory used 2015-12-18 22:23:23,467 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Uncaught exception in ContainerMemoryManager while managing memory of container_1448915696877_26289_01_000270 java.lang.IllegalArgumentException: disparate values at sun.misc.FDBigInt.quoRemIteration(FloatingDecimal.java:2931) at sun.misc.FormattedFloatingDecimal.dtoa(FormattedFloatingDecimal.java:922) at sun.misc.FormattedFloatingDecimal.<init>(FormattedFloatingDecimal.java:542) at java.util.Formatter$FormatSpecifier.print(Formatter.java:3264) at java.util.Formatter$FormatSpecifier.print(Formatter.java:3202) at java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2769) at java.util.Formatter$FormatSpecifier.print(Formatter.java:2720) at java.util.Formatter.format(Formatter.java:2500) at java.util.Formatter.format(Formatter.java:2435) at java.lang.String.format(String.java:2148) at org.apache.hadoop.util.StringUtils.format(StringUtils.java:123) at org.apache.hadoop.util.StringUtils$TraditionalBinaryPrefix.long2String(StringUtils.java:758) at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.formatUsageString(ContainersMonitorImpl.java:487) at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:399) 2015-12-18 22:23:23,498 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Uncaught exception in ContainerMemoryManager while managing memory of container_1448915696877_26289_01_000214
DROP TABLE IF EXISTS hadoop_log; CREATE TABLE hadoop_log (date1 STRING, time1 STRING, msgtype STRING, classname STRING, msgtext STRING ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES ( "input.regex" = "^(\\d{4}-\\d{2}-\\d{2})\\s+(\\d{2}.\\d{2}.\\d{2}.\\d{3})\\s+(\\S+)\\s+(\\S+)\\s+(.*)$", "output.format.string" = "%1$s %2$s %3$s %4$s %5$s")STORED AS TEXTFILE; LOAD DATA LOCAL INPATH "/home/student/hadooplog" INTO TABLE hadoop_log; SELECT date1, time1, msgtext FROM hadoop_log WHERE msgtype='ERROR' OR msgtype='WARN' LIMIT 5; LOAD DATA LOCAL INPATH "/home/student/hadooplog3" OVERWRITE INTO TABLE hadoop_log; 需要注意的一点是，hive以\n做为行分隔符，所以需要对原有的日志文件进行处理，因为原有的日志文件中可能有异常或错误发生，这个时候是多行的。否则hive中会有很
多空的记录。
可以写一段bash shell或python来完成需要的功能。下面是我刚学python写的，很简陋。
import rep=re.compile(r"^\d{4}-\d{2}-\d{2}\s+\d{2}.\d{2}.\d{2}.\d{3} INFO|WARN|ERROR|DEBUG")str=""f2=open('/home/student/hadooplog4','w')with open('/app/cdh23502/logs/hadoop-student-datanode-nn1.log','r') as f: for l in f: if(str==""): str=l.rstrip() continue if(str!="" and len(p.findall(l))>0): print "\n"+str f2.write(str+"\n") str=l.rstrip() else: str=str+l.rstrip() print "\n" + str f2.write(str+"\n")f2.flush()f2.close()