Java Api实现Elasticsearch的滚动查询

解决ES每次只能查询一万条数据的问题

java 复制代码
	@Override
    public List<ESHandleDto> getVisitorsNum(String startTime, String endTime, String schoolName, String typeFunction) throws IOException {
        List<ESHandleDto> esHandleDtos = new ArrayList<>();
        SearchRequest searchRequest = new SearchRequest();
        searchRequest.indices(ElasticEnum.FUNCTIONLOG_INDEX.getValue());

        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
        if (StringUtils.hasText(schoolName)) {
            boolQueryBuilder.must(QueryBuilders.termQuery("schoolName.keyword", schoolName));
        }
        if (StringUtils.hasText(typeFunction)) {
            boolQueryBuilder.must(QueryBuilders.termQuery("typeFunction.keyword", typeFunction));
        }
        if (StringUtils.hasText(startTime) && StringUtils.hasText(endTime)) {
            boolQueryBuilder.must(QueryBuilders.rangeQuery("createDate").gte(startTime + "T00:00:00.000Z").lte(endTime + "T23:59:59.000Z"));
        }
        BoolQueryBuilder shouldQuery = QueryBuilders.boolQuery();
        shouldQuery.should().add(QueryBuilders.termQuery("description.keyword", "查询学生信息表"));
        shouldQuery.should().add(QueryBuilders.termQuery("description.keyword", "获取学校访客数据"));
        boolQueryBuilder.must(shouldQuery);

        sourceBuilder.sort("_doc", SortOrder.DESC);
        sourceBuilder.size(10000);
        sourceBuilder.query(boolQueryBuilder);
        searchRequest.scroll(TimeValue.timeValueMinutes(1));
        searchRequest.source(sourceBuilder);
//        System.out.println(sourceBuilder.toString());
        SearchResponse searchResponse = null;
        try {
            searchResponse = esConfig.restHighLevelClient().search(searchRequest, RequestOptions.DEFAULT);
        } catch (Throwable e) {
            throw new RuntimeException(e);
        }
        scrollHandle(searchResponse, esConfig, esHandleDtos, startTime, endTime);
        return esHandleDtos;
    }

    public static void scrollHandle(SearchResponse searchResponse, ESConfig esConfig, List<ESHandleDto> esHandleDtos, String startTime, String endTime) {
        SearchHits hits = searchResponse.getHits();
        String scrollId = searchResponse.getScrollId();
        SearchHit[] searchHits = hits.getHits();
        //对结果集处理
        List<FunctionLogElasticEntity> functionLogs = ESUtil.convertToFunctionLog(searchHits);
        visitorsResultHandle(functionLogs, esHandleDtos, ESUtil.isMoreThanMonth(startTime, endTime));

		//滚动查询部分,将从第10001笔数据开始
        while (searchHits != null && searchHits.length > 0) {
            SearchScrollRequest searchScrollRequest = new SearchScrollRequest(scrollId);
            searchScrollRequest.scroll(TimeValue.timeValueMinutes(1));
            try {
                searchResponse = esConfig.restHighLevelClient().scroll(searchScrollRequest, RequestOptions.DEFAULT);
            } catch (Throwable e) {
                throw new RuntimeException(e);
            }
            scrollId = searchResponse.getScrollId();
            hits = searchResponse.getHits();
            searchHits = hits.getHits();
            //对结果集处理
            functionLogs = ESUtil.convertToFunctionLog(searchHits);
            visitorsResultHandle(functionLogs, esHandleDtos, ESUtil.isMoreThanMonth(startTime, endTime));
        }

        //清除滚动,否则影响下次查询
        ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
        clearScrollRequest.addScrollId(scrollId);
        ClearScrollResponse clearScrollResponse = null;
        try {
            clearScrollResponse = esConfig.restHighLevelClient().clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
        boolean succeeded = clearScrollResponse.isSucceeded();
        System.out.println(succeeded);
    }
相关推荐
纪莫21 分钟前
A公司一面:类加载的过程是怎么样的? 双亲委派的优点和缺点? 产生fullGC的情况有哪些? spring的动态代理有哪些?区别是什么? 如何排查CPU使用率过高?
java·java面试⑧股
JavaGuide1 小时前
JDK 25(长期支持版) 发布,新特性解读!
java·后端
用户3721574261351 小时前
Java 轻松批量替换 Word 文档文字内容
java
白鲸开源1 小时前
教你数分钟内创建并运行一个 DolphinScheduler Workflow!
java
Java中文社群2 小时前
有点意思!Java8后最有用新特性排行榜!
java·后端·面试
代码匠心2 小时前
从零开始学Flink:数据源
java·大数据·后端·flink
间彧2 小时前
Spring Boot项目中如何自定义线程池
java
间彧2 小时前
Java线程池详解与实战指南
java
用户298698530143 小时前
Java 使用 Spire.PDF 将PDF文档转换为Word格式
java·后端
渣哥3 小时前
ConcurrentHashMap 1.7 vs 1.8:分段锁到 CAS+红黑树的演进与性能差异
java