最近公司做大数据项目要求兼容2003和2007 框架是struts 我用poi提供的方法 可以解析比较小的excel 但是解析50W条左右的数据量比较大的文件会出现oom 然后我想拆分excel 但是操作的时候会损坏文件 本人菜鸟 求大神指点
以下是我上传的代码OutputStream bos = null;
InputStream stream = null;
String a = "";
String fileName [] = null;
Integer i = 0;
try{
stream = file.getInputStream();//把文件读入
String filePath = request.getRealPath("/files"+"/"+agentType);//上传到指定的upload包中
File f = new File(filePath);
if(!f.exists()){
f.mkdir();
}
bos = new FileOutputStream(filePath + "\\"+uuid+file.getFileName());
a = filePath + "\\"+uuid+file.getFileName();
int bytesRead = 0;
byte[] buffer = new byte[8192];
while ( (bytesRead = stream.read(buffer, 0, 8192)) != -1) {
if(i%200 == 0){
bos.close();
bos = new FileOutputStream(filePath + "\\"+uuid+"文本"+i+"@"+ file.getFileName());
a += ","+filePath + "\\"+uuid+"文本"+i+ file.getFileName();
}
bos.write(buffer, 0, bytesRead);//将文件写入服务器
i++;
}
fileName = a.split(",");
}catch (Exception e) {
e.printStackTrace();
}
finally{
bos.close();
stream.close();
}
return fileName;
出现过以下两个问题
java.io.EOFException: Unexpected end of ZLIB input stream
严重: Servlet.service() for servlet action threw exception
org.apache.poi.openxml4j.exceptions.InvalidFormatException: Package should contain a content type part [M1.13]
at org.apache.poi.openxml4j.opc.ZipPackage.getPartsImpl(ZipPackage.java:199)
at org.apache.poi.openxml4j.opc.OPCPackage.getParts(OPCPackage.java:665)
at org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:274)
at org.apache.poi.util.PackageHelper.open(PackageHelper.java:39)
at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:218)
at com.sssoft.data.struts.action.ExcelDataAction.upExT1(ExcelDataAction.java:123)
at com.sssoft.data.struts.action.ExcelDataAction.upExcel(ExcelDataAction.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.struts.actions.DispatchAction.dispatchMethod(DispatchAction.java:274)
at org.apache.struts.actions.DispatchAction.execute(DispatchAction.java:194)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.sssoft.framework.web.util.CharacterEncodingFilter.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
------解决方案--------------------
你这样等于把文件直接撕碎了，当然不行。。。
POI支持基于事件的解析，可以尝试一下，理论上需要的内存会比较小，
http://poi.apache.org/spreadsheet/how-to.html
------解决方案--------------------
这个应该类似于解析XML中的SAX解析，理论上不会有上限，当然，这也取决于你的程序怎么处理解析出来的数据，比如你还是要把解析出来的数据都放在内存里，那还是会遇到上限的。
------解决方案--------------------
解析文件应该不会oom的，应该是你解析的过程中往内存里塞了太多东西所以才oom，思路没错，检查代码去。
读一点处理一点，不要读多了往内存塞
------解决方案--------------------