SpringBoot集成ElasticSearch框架详解--项目搭建及实战必备手册
一:ElasticSearch介绍
ES 是对Lucene进行了封装,开源的、高度可拓展的、可用于大数据存储的全文搜索、分析引擎。 ES 主要特点:分布式、高可用、异步写入。 ES 核心概念:实时|集群|节点(保存数据)|索引|分片(将索引分片)|副本(分片可设置多个副本) ES 使用案例: 维基百科、Stack Overflow、Github。
二:版本兼容关系
Spring Boot Version (x) | Spring Data Elasticsearch Version (y) | Elasticsearch Version (z) |
---|---|---|
x <= 1.3.5 | y <= 1.3.4 | z <= 1.7.2* |
x >= 1.4.x | 2.0.0 <=y < 5.0.0** | 2.0.0 <= z < 5.0.0** |
三:SpringBoot1.x集成ElasticSearch5.x框架详解
(1)如果框架定型SpringBoot1.5.X以下,那就不能用spring-boot-starter-data-elasticsearch,需要用TransportClient来实现elasticSearch5.X的集成。
(2)下载elasticsearch5.2.2
https://www.elastic.co/downloads/past-releases
(3) 配置文件
config/elasticsearch.yml 主配置文件 config/jvm.options jvm参数配置文件 cofnig/log4j2.properties 日志配置文件
(4) 关键属性配置
属性 | 配置值 | 说明 |
---|---|---|
cluster.name | hsyt | 同一个服务上不同节点通过同一个集群名建立联系 |
node.name | node-1 | 默认的elasticsearch 随机生成uuid的前7个字符作为节点id,可以手工设置 |
path.conf | /path/to/conf | 配置文件的存储路径,默认是es根目录下的config文件夹 |
path.logs | /path/to/logs | 日志文件的存储路径,默认是es根目录下的logs文件夹 |
bootstrap.memory_lock | true | 当jvm开始swap写入交换空间时es的效率会降低,所以要保证它不swap,这对节点健康极其重要。一种方法是将其设置为true |
http.cors.enabled | true | 使用head等监控集群时需要配置 |
http.cors.allow-origin | “*” | 使用head等监控集群时需要配置 |
http.cors.allow-credential | true | 使用head等监控集群时需要配置 |
(5) SpringBoot框架搭建,如下图结构(详见之前我的博客步骤,此处省略)
https://blog.csdn.net/For_niu/article/details/87875470
(5) 服务提供者和服务消费者pom.xml文件配置如下依赖
<properties> <elasticsearch.version>5.2.2</elasticsearch.version> </properties> <dependencies> <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> <version>${elasticsearch.< 4000 /span>version}</version> </dependency> <dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>transport</artifactId> <version>${elasticsearch.version}</version> </dependency> </dependencies>
(6) 项目需要的一个数据表结构如下(dao层提供一个全表查询接口,此处省略)
CREATE TABLE `employee` ( `id` bigint(15) NOT NULL AUTO_INCREMENT COMMENT 'ID', `name` varchar(100) DEFAULT NULL COMMENT '名称', `sharding_id` bigint(15) DEFAULT NULL COMMENT '分区因子', PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=1098470660121956355 DEFAULT CHARSET=utf8 COMMENT='mycat数据表';
(7) 创建索引为例介绍微服务实现类代码
@Component @Service(version = "0.0.1", interfaceClass = TestService.class, registry = "sb", group = "sbTest", timeout = 3000) public class TestServiceImpl implements TestService { private static final Logger LOGGER = LoggerFactory.getLogger(TestServiceImpl.class); @Resource private TestDao testDao; private static TransportClient client; private static Object lock = new Object(); @Value("${elasticSearch.clusterName}") private String clusterName; @Value("${elasticSearch.hostList}") private String hostList; @Value("${elasticSearch.esIndex}") private String esIndex; @Override public Integer count() { Integer num = 0; try { num = testDao.count(); } catch (Exception e) { LoggerUtil.error(LOGGER, "统计异常", e.getMessage()); } return num; } @Override public Integer createEs() { TransportClient esClient = getClient(); try { esClient.admin().indices().prepareCreate(esIndex).execute().actionGet(); PutMappingRequest putMapping = Requests.putMappingRequest(esIndex).type(esIndex).source(createMapping(esIndex)); esClient.admin().indices().putMapping(putMapping).actionGet(); } catch (Exception e) { LoggerUtil.error(LOGGER, "prepareCreate error : {0}", e.getMessage()); } BulkRequestBuilder bulkRequest = null; try { bulkRequest = esClient.prepareBulk(); List<Test> list = testDao.selTest(); for (Test test : list) { IndexRequest request = esClient.prepareIndex(esIndex, esIndex, test.getId()).setSource(obj2JsonData(test)).request(); bulkRequest.add(request); } bulkRequest.execute().actionGet(); } catch (Exception e) { LoggerUtil.error(LOGGER, "addIndexData error : {0}", e.getMessage()); } return bulkRequest.numberOfActions(); } public TransportClient getClient( 20000 ) { try { if (client == null) { synchronized (lock) { String[] hostAndPorts = hostList.split(","); Settings settings = Settings.builder().put("cluster.name", clusterName).put("client.transport.sniff", true).build(); Map<String, Integer> hostInfos = parseHostAndPorts(hostAndPorts); client = new PreBuiltTransportClient(settings); for (Map.Entry<String, Integer> entry : hostInfos.entrySet()) { client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(entry.getKey()), entry.getValue())); } } } } catch (Exception e) { LoggerUtil.error(LOGGER, "构建服务失败:{0}", e.getMessage()); } return client; } public static Map<String, Integer> parseHostAndPorts(String[] hostAndPorts) { Map<String, Integer> hostMap = new HashMap<String, Integer>(); if (hostAndPorts == null) { return hostMap; } for (String hostAndPortStr : hostAndPorts) { String[] hostAndPort = hostAndPortStr.split(":"); if (hostAndPort.length != 2 || !StringUtils.isNumeric(hostAndPort[1])) { throw new IllegalStateException("端口号信息不正确"); } hostMap.put(hostAndPort[0], Integer.parseInt(hostAndPort[1])); } return hostMap; } public static XContentBuilder createMapping(String indexType) { XContentBuilder mapping = null; try { mapping = XContentFactory.jsonBuilder().startObject() .startObject(indexType).startObject("properties") .startObject("id").field("type", "text").field("fielddata", "true").endObject() .startObject("name").field("type", "text").endObject() .startObject("shardingId").field("type", "text").endObject() .endObject().endObject().endObject(); } catch (Exception e) { LoggerUtil.error(LOGGER, "createMapping error : {0}", e.getMessage()); } return mapping; } public static String obj2JsonData(Test test) { String jsonData = null; try { XContentBuilder jsonBuild = XContentFactory.jsonBuilder().startObject() .field("id", test.getId()) .field("OrgName", test.getName()) .field("shardingId", test.getShardingId()) .endObject(); jsonData = jsonBuild.string(); } catch (Exception e) { LoggerUtil.error(LOGGER, "obj2JsonData error : {0}", e.getMessage()); } return jsonData; }
(8)属性文件application-dev.properties(因为项目做了环境区分)配置如下:
elasticSearch.clusterName=yt-spring-boot-index elasticSearch.hostList=localhost:9300 elasticSearch.esIndex=employee
(8)启动服务
首先启动zookeeper和elasticsearch5.2.2服务,然后启动微服务,最后启动web服务
使用Restful web service发送请求并创建索引
(9)使用elasticsearch-head-master可视化工具来监控及操作elasticsearch
下载地址:https://github.com/mobz/elasticsearch-head
用自己习惯的浏览器打开index.html,访问索引,如下图。
四:SpringBoot + Spring Data ElasticSearch框架
(1)前面已经提到SpringBoot1.x如果通过SpringDataElasticSearch内部自带的ElasticSearch来集成,版本太低已不能满足项目的需要,本章节将介绍SpringBoot2.x+SpringDataElasticSearch来实现ElasticSearch高版本的全文检索功能。
(2)实体类模块和服务提供者和服务消费者引用配置(注意:由于SpringBoot引用的2.1.3.RELEASE版本,通过源码可以看到其配搭集成的ElasticSearch版本是6.2.2版本,所以需要下载比6.2.2更高版本的ElasticSearch服务版本!!!)
https://www.elastic.co/downloads/past-releases 建议下载elasticsearch 6.6.1版本。
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-elasticsearch</artifactId> </dependency>
(3)实体类增加注解@Document,很重要!(配置索引名称及类型)
@Getter @Setter @EqualsAndHashCode @NoArgsConstructor @Document(indexName = "wangze", type = "wangze") public class Test implements Serializable{ private static final long serialVersionUID = -911018888247361501L; private String id; private String name; private String lat; private String lon; private String address; private String shardingId; }
(4)新建xxxRepository.java接口再继承ElasticsearchRepository类,很重要!(内部集成了CRUD及分页等功能,可以满足绝大部分业务需求功能;另外ElasticSearchTemplate是对ElasticsearchRepository接口的补充,更多的是提供了底层接口的方法,比如千万级别以上的批量索引创建等)
import com.wangze.sb.api.entity.test.Test; import org.springframework.data.elasticsearch.repository.ElasticsearchRepository; import org.springframework.stereotype.Component; @Component public interface TestRepository extends ElasticsearchRepository<Test, String> { // 可以为空,也可以自定义方法,比如findByAlipayCount等等,其中AlipayCount为实体表字段名称驼峰格式。 }
(5)业务代码可以实现新增、批量新增、根据名称查询、查询全部索引、根据名称删除、删除全部索引,根据距离筛选后排序再分页查询等等接口,如下代码:
TestService.java代码如下:
public interface TestService { void addEs(Test test); void addEsBulk(List<Test> list); List<Test> searchEs(String name); List<Test> findAllEs(); void deleteEs(Test test); void deleteALl(); List<Test> queryByDistanceAndSort(); }
TestServiceImpl.java代码如下:
@Component @Service(version = "0.0.1", interfaceClass = TestService.class, registry = "sb", group = "sbTest", timeout = 3000) public class TestServiceImpl implements TestService { private static final Logger LOGGER = LoggerFactory.getLogger(TestServiceImpl.class); private static final String TEST_INDEX_NAME = "wangze"; @Resource private TestDao testDao; @Resource private TestRepository testRepository; @Autowired private ElasticsearchTemplate elasticsearchTemplate; @Override public void addEs(Test test) { try { testRepository.save(test); } catch (Exception e) { LoggerUtil.error(LOGGER, "addEs error", e.getMessage()); } } @Override public void addEsBulk(List<Test> tests) { int counter = 0; try { if (!elasticsearchTemplate.indexExists(TEST_INDEX_NAME)) { elasticsearchTemplate.createIndex(TEST_INDEX_NAME); } List<IndexQuery> queries = new ArrayList<>(); for (Test test : tests) { IndexQuery indexQuery = new IndexQuery(); indexQuery.setId(String.valueOf(test.getId())); indexQuery.setObject(test); indexQuery.setIndexName(TEST_INDEX_NAME); indexQuery.setType(TEST_INDEX_NAME); queries.add(indexQuery); if (counter % 500 == 0) { elasticsearchTemplate.bulkIndex(queries); queries.clear(); } counter++; } if (queries.size() > 0) { elasticsearchTemplate.bulkIndex(queries); } } catch (Exception e) { LoggerUtil.error(LOGGER, "addEsBulk error", e.getMessage()); } } @Override public List<Test> searchEs(String name) { Pageable pageRequest = new PageRequest(0, 50); Iterable<Test> test = testRepository.search(QueryBuilders.matchQuery("name", name), pageRequest); return Lists.newArrayList(test); } @Override public List<Test> findAllEs() { Iterable<Test> test = testRepository.findAll(); return Lists.newArrayList(test); } @Override public void deleteEs(Test test) { testRepository.delete(test); } @Override public void deleteALl() { testRepository.deleteAll(); } @Override public List<Test> queryByDistanceAndSort() { List<Test> personList = null; try { double lat = 39.929986; double lon = 116.395645; Long nowTime = System.currentTimeMillis(); GeoDistanceQueryBuilder filterBuilder = QueryBuilders.geoDistanceQuery("address").point(lat, lon).distance(100000, DistanceUnit.KILOMETERS); GeoDistanceSortBuilder sortBuilder = SortBuilders.geoDistanceSort("address", new GeoPoint(lat, lon)).unit(DistanceUnit.METERS).order(SortOrder.ASC); Pageable pageable = new PageRequest(0, 50); NativeSearchQueryBuilder builder1 = new NativeSearchQueryBuilder().withFilter(filterBuilder).withSort(sortBuilder).withPageable(pageable); personList = elasticsearchTemplate.queryForList(builder1.build(), Test.class); LoggerUtil.info(LOGGER, "queryByDistanceAndSort总耗时:" + (System.currentTimeMillis() - nowTime)); } catch (Exception e) { LoggerUtil.error(LOGGER, "queryByDistanceAndSorterror", e.getMessage()); } return personList; } }
TestController.java代码如下:
@RestController @RequestMapping("/sb") public class TestController { private static final Logger LOGGER = LoggerFactory.getLogger(TestController.class); @Reference(group = "sbTest", registry = "sb", version = "0.0.1") private TestService testService; @RequestMapping(value = "/test") public @ResponseBody ResultMessage count(@RequestBody Test test) { ResultMessage result = new ResultMessage(true, "查询成功"); try { result.setContent(testService.count()); } catch (Exception e) { result.change(false, "查询失败" + e.getMessage()); } return result; } @RequestMapping(value = "/addEs") public ResultMessage addEs() { ResultMessage result = new ResultMessage(true, "创建索引成功"); try { Test test = new Test(); Random random = new Random(); test.setId(String.valueOf(random.nextInt())); test.setName("名字" + random.nextInt()); double max = 0.00001; double min = 0.000001; double lat = 38.929986; double lon = 117.395645; DecimalFormat df = new DecimalFormat("######0.000000"); double s = random.nextDouble() % (max - min + 1) + max; String lons = df.format(s + lon); String lats = df.format(s + lat); Double dlon = Double.valueOf(lons); Double dlat = Double.valueOf(lats); test.setAddress(dlat + "," + dlon); testService.addEs(test); } catch (Exception e) { result.change(false, "创建索引失败" + e.getMessage()); } return result; } @RequestMapping(value = "/addEsBulk") public ResultMessage addEsBulk() { ResultMessage result = new ResultMessage(true, "创建索引成功"); try { double lat = 38.929986; double lon = 117.395645; List<Test> testList = new ArrayList<>(10000); for (int i = 0; i < 10000; i++) { double max = 0.00001; double min = 0.000001; Random random = new Random(); double s = random.nextDouble() % (max - min + 1) + max; DecimalFormat df = new DecimalFormat("######0.000000"); String lons = df.format(s + lon); String lats = df.format(s + lat); Double dlon = Double.valueOf(lons); Double dlat = Double.valueOf(lats); Test test = new Test(); test.setId(String.valueOf(i)); test.setName("名字" + i); test.setAddress(dlat + "," + dlon); testList.add(test); } testService.addEsBulk(testList); } catch (Exception e) { result.change(false, "创建索引失败" + e.getMessage()); } return result; } @RequestMapping(value = "/searchEsByName") public ResultMessage searchEs(@RequestBody Test test) { ResultMessage result = new ResultMessage(true, "根据name检索索引成功"); try { List<Test> tests = testService.searchEs(test.getName()); result.setContent(JSON.toJSONString(tests)); } catch (Exception e) { result.change(false, "根据name检索索引失败" + e.getMessage()); } return result; } @RequestMapping(value = "/findAllEs") public ResultMessage findAll(@RequestBody Test test) { ResultMessage result = new ResultMessage(true, "检索全部索引查询成功"); try { List<Test> all = testService.findAllEs(); result.setContent(JSON.toJSONString(all)); } catch (Exception e) { result.change(false, "检索全部索引查询失败" + e.getMessage()); } return result; } @RequestMapping(value = "/deleteEsById") public ResultMessage deleteEs(@RequestBody Test test) { ResultMessage result = new ResultMessage(true, "创建索引成功"); try { testService.deleteEs(test); } catch (Exception e) { result.change(false, "创建索引失败" + e.getMessage()); } return result; } @RequestMapping(value = "/deleteAll") public ResultMessage deleteAll() { ResultMessage result = new ResultMessage(true, "创建索引成功"); try { testService.deleteALl(); } catch (Exception e) { result.change(false, "创建索引失败" + e.getMessage()); } return result; } @RequestMapping(value = "/queryByDistanceAndSort") public ResultMessage queryByDistanceAndSort() { ResultMessage result = new ResultMessage(true, "创建索引成功"); try { List<Test> all = testService.queryByDistanceAndSort(); result.setContent(JSON.toJSONString(all)); } catch (Exception e) { result.change(false, "创建索引失败" + e.getMessage()); } return result; } }
(6)启动ElasticSearch6.6.1和zookeeper服务,然后启动微服务,最后启动web服务,使用idea工具RestFul Web Service测试,如下图:
第一张图是创建索引结果:
第二张图和第三张图是根据名称搜索索引结果:
第四张图是 搜索100公里范围内 + 按照距离排序最近的 + 分页前50条 查询结果:
(7)此方案搭建框架踩过的坑及总结
第一种错误:
Description: The bean 'testRepository', defined in null, could not be registered. A bean with that name has already been defined in null and overriding is disabled. Action: Consider renaming one of the beans or enabling overriding by setting spring.main.allow-bean-definition-overriding=true
如果出现上面错误,则需要在application-dev.properties和application.yml中分别配置如下(是否允许重复bean定义注册)
application-dev.properties中如下:
spring.main.allow-bean-definition-overriding=true
application.yml中如下:
spring: main: allow-bean-definition-overriding: true
第二种错误:
nested exception is java.lang.IllegalStateException: availableProcessors is already set to [4], rejecting [4]
如果出现上面错误,说明是netty冲突的问题
则需要在微微服务或web启动类中(注意是新增在第一行)增加如下配置
System.setProperty("es.set.netty.runtime.available.processors","false");
第三种错误:
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
如果出现上面错误,说明是必须要升级mysql-connector-java版本,建议升级版本到8.0.12版本,如下代码:
pom.xml中
<dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>8.0.12</version> </dependency>
application-dev.properties中两个地方需要变化:
spring.datasource.url=jdbc:mysql://127.0.0.1:3306/pms?serverTimezone=UTC&useSSL=false spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
第四种错误:如果涉及到经纬度按距离排序时,实体类表示距离的字段,必须增加注解和类型标识,如下代码:
@GeoPointField private GeoPoint location;
- 【SSH网上商城项目实战03】使用EasyUI搭建后台页面框架
- 从框架到完整项目搭建,实战项目《约个球》(5)-为我们自己的框架导入网络的开源框架
- golang实战使用gin+xorm搭建go语言web框架restgo详解5.2 跳转和重定向
- 从框架到完整项目搭建,实战项目《约个球》(6)- 利用SlidingMenu和Fragment构建我们的项目结构
- 【实战\聚焦Python分布式爬虫必学框架Scrapy 打造搜索引擎项目笔记】第2章 windows下搭建开发环境
- Spring+SpringMvc+Mybatis框架集成搭建教程一(项目创建)
- 架构实战项目心得(七):使用SpringBoot+Dubbo+Mybatisplus+Oracle搭建后台项目框架(一)
- 基于vue-cli网上商城项目实战开发——搭建一个完整的SPA项目开发框架(一)
- java极速mvc框架jfinal项目框架搭建过程(图文详解)
- golang实战使用gin+xorm搭建go语言web框架restgo详解5.3 资源控制器
- vue2.x+webpack快速搭建前端项目框架详解
- SpringBoot+Maven项目实战(2):集成SpringBoot
- golang实战使用gin+xorm搭建go语言web框架restgo详解5.4 控制器参数绑定
- 实战 :Spring MVC + 注解 +SqlServer 框架搭建及详解
- 【Activiti】从入门到放弃——项目实战之框架集成
- golang实战使用gin+xorm搭建go语言web框架restgo详解5.5 控制器模型绑定
- ASP.NET MVC企业级项目框架搭建实战
- Spring+SpringMvc+Mybatis框架集成搭建教程四(项目部署及测试)
- layUI前端框架使用详解_layUI前端框架项目实战(完整)
- 从框架到完整项目搭建,实战项目《约个球》(4)-Intellij IDEA 导入library和jar文件