Springboot with Impala and Kudu
2018-02-23 10:12
821 查看
转自
Run it using
Once you figure out your vm ip address, we will copy over the dept.txt file we created earlier with the python script and put it into hdfs.
ssh username:demo password:demo
In the project search for src/main/java/config/DatabaseConfig.java and set the datasource url to your kudu/impala VM url
hit apply/save.
You are all set. Hit run.
To see it in action go to your browser and visit url: http://localhost:8080/department?companyId=Company1
How to run this app
Please follow the instructions below to get kudu running locally with java and impala driver.Download the kudu quickstart VM
Kudu Quick Start VMCreate the department file to load
You can use this python script to create the department file to load: name it generate.py#!/usr/local/bin/python3 default_path = '/Users/ryang1/Desktop' def generateDepartmentMasterFile(): """ Company Id | Department Code | Department Description """ file = open(default_path+'/dept.txt','a') for x in range (1,501): file.write('Company{0}|{0}|Dept{0} Description\n'.format(str(x))) generateDepartmentMasterFile()
Run it using
$ python3 generate.py
Load the department file
Start your kudu quickstart VM. Once it has started run this command to find the vm ip. Should start with 192.168.*$ ip addr show [demo@quickstart ~]$ ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 08:00:27:8a:40:9a brd ff:ff:ff:ff:ff:ff inet 192.168.57.100/24 brd 192.168.57.255 scope global eth0 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 08:00:27:16:94:b5 brd ff:ff:ff:ff:ff:ff inet 10.0.3.15/24 brd 10.0.3.255 scope global eth1
Once you figure out your vm ip address, we will copy over the dept.txt file we created earlier with the python script and put it into hdfs.
ssh username:demo password:demo
$ scp dept.txt demo@192.168.57.100:~/ $ ssh demo@192.168.57.100 $ hdfs dfs -put -f dept.txt /data/dept
Create the department table in kudu/impala
CREATE EXTERNAL TABLE department_raw ( company_id string, department_code int, department_description string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LOCATION '/data/dept/'; CREATE TABLE department DISTRIBUTE BY HASH (company_id) INTO 16 BUCKETS TBLPROPERTIES( 'storage_handler' = 'com.cloudera.kudu.hive.KuduStorageHandler', 'kudu.table_name' = 'department', 'kudu.master_addresses' = '127.0.0.1', 'kudu.key_columns' = 'company_id' ) AS SELECT * FROM department_raw; # Set the database configuration
In the project search for src/main/java/config/DatabaseConfig.java and set the datasource url to your kudu/impala VM url
@Bean(name="dataSource") public DataSource dataSource() { DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create(); //CHANGE ME! dataSourceBuilder.url("jdbc:impala://192.168.57.100:21050"); dataSourceBuilder.username("demo"); dataSourceBuilder.password("demo"); dataSourceBuilder.driverClassName("com.cloudera.impala.jdbc4.Driver"); return dataSourceBuilder.build(); }
Setting up intellij
In order to run the spring boot app you will need to set the run configuration Run -> Edit Configurations Press the + icon create a new maven entry name it to springboot-run change the command line input field tospring-boot:run
hit apply/save.
You are all set. Hit run.
To see it in action go to your browser and visit url: http://localhost:8080/department?companyId=Company1
相关文章推荐
- Build a Reactive application with Angular 5 and Spring Boot 2.0
- Auto-Publishing and Monitoring APIs With Spring Boot--转
- Build a Basic CRUD App with Angular 5.0 and Spring Boot 2.0
- Creating a REST API with Spring Boot and MongoDB
- Configure a Spring JMS application with Spring Boot and annotation support
- Distributed transactions with multiple databases, Spring Boot, Spring Data JPA and Atomikos
- Manage Spring Boot Logs with Elasticsearch, Logstash and Kibana
- Microservices Reference Architecture - with Spring Boot, Spring Cloud and Netflix OSS--转
- A Simple Web App with Spring Boot, Spring Security and Stormpath – in 15 Minutes
- Distributed transactions with multiple databases, Spring Boot, Spring Data JPA and Atomikos
- Building Microservices with Spring Boot and Apache Thrift. Part 1 with servlet
- A Microservice Architecture with Spring Boot and Spring Cloud(二)
- Building Microservices with Spring Boot and Apache Thrift. Part 2. Swifty services
- Spring boot @Qualifier doesn't work with datasources
- SpringBoot UnsatisfiedDependencyException: Error creating bean with name
- Setting Up Swagger 2 with a Spring Boot REST API
- Key Components and Internals of Spring Boot Framework--转
- spring-mvc-and-ajax-with-json
- Spring Boot 项目构建 之 使用 Spring Boot 构建应用(Building an Application with Spring Boot)
- 两张表(多对多关联)操作 with Spring Boot, Spring Data JPA, H2