当前位置: 代码迷 >> 综合 >> elasticsearch批量插入数据的时候出现java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection
  详细解决方案

elasticsearch批量插入数据的时候出现java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection

热度:38   发布时间:2023-12-25 05:57:39.0

问题:

elasticsearch每次都批量插入几万数据量,然后就会出现下列问题。看这个问题应该是配置的问题

ERROR [https-jsse-nio-443-exec-4] 2020-07-09 23:31:54 (EsMiniDaansouDataInfoWithBLOBsUtil.java:80) java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1611)at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1581)at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1551)at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:531)at cn.dxszxw.elasticsearch.esUtils.EsMiniDaansouDataInfoWithBLOBsUtil.increaseEsData(EsMiniDaansouDataInfoWithBLOBsUtil.java:76)at cn.dxszxw.controller.ElasticsearchController.createMiniDaansouDataInfoWithBLOBsAllIndexByRange(ElasticsearchController.java:146)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)... 1 more

 

解决办法(我仅处理elasticsearch7.8.0版本的,可以看到下面的方法出现的是RestHighLevelClient高级客户端的):

创建下面列举的配置类即可,然后相应修改下自己的需要的时间

import org.apache.http.HttpHost;
import org.apache.http.client.config.RequestConfig;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;@Configuration
public class ElasticSearchClientConfig {@Bean@Qualifier("highLevelClient")public RestHighLevelClient restHighLevelClient() {// RestHighLevelClient highLevelClient = new RestHighLevelClient(// RestClient.builder(new HttpHost("127.0.0.1", 9200, "http")));RestHighLevelClient highLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("127.0.0.1", 9200, "http")).setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {// 该方法接收一个RequestConfig.Builder对象,对该对象进行修改后然后返回。@Overridepublic RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {return requestConfigBuilder.setConnectTimeout(5000 * 1000) // 连接超时(默认为1秒).setSocketTimeout(6000 * 1000);// 套接字超时(默认为30秒)//更改客户端的超时限制默认30秒现在改为100*1000分钟}}));// 调整最大重试超时时间(默认为30秒).setMaxRetryTimeoutMillis(60000);return highLevelClient;}}

RestClientBuilder支持同时提供一个RequestConfigCallback和一个HttpClientConfigCallback,你可以定制 the Apache Async Http Client 公开的配置。这两个回调函数可以修改某些特定的行为,而不会覆盖RestClient初始化的所有其他默认配置。 

使用elasticsearch查询百亿级数据必然会影响查询的速度但是es与客户端的的默认响应时间只有30秒,查询大量的数据时30秒的时间一般都不够用。