当前位置： 首页 > news >正文

Spring Boot集成starrocks快速入门Demo

news 来源：原创 2024/9/20 5:58:58

1.什么是starrocks？

StarRocks 是新一代极速全场景 MPP (Massively Parallel Processing) 数据库。StarRocks 的愿景是能够让用户的数据分析变得更加简单和敏捷。用户无需经过复杂的预处理，就可以用 StarRocks 来支持多种数据分析场景的极速分析。 StarRocks 架构简洁，采用了全面向量化引擎，并配备全新设计的 CBO (Cost Based Optimizer) 优化器，查询速度（尤其是多表关联查询）远超同类产品。 StarRocks 能很好地支持实时数据分析，并能实现对实时更新数据的高效查询。StarRocks 还支持现代化物化视图，进一步加速查询。 使用 StarRocks，用户可以灵活构建包括大宽表、星型模型、雪花模型在内的各类模型。 StarRocks 兼容 MySQL 协议，支持标准 SQL 语法，易于对接使用，全系统无外部依赖，高可用，易于运维管理。StarRocks 还兼容多种主流 BI 产品，包括 Tableau、Power BI、FineBI 和 Smartbi。 StarRocks 是 Linux 基金会项目，采用 Apache 2.0 许可证，可在 StarRocks GitHub 存储库中找到（请参阅 StarRocks 许可证）。StarRocks（i）链接或调用第三方软件库中的函数，其许可证可在 licenses-binary 文件夹中找到；和（ii）包含第三方软件代码，其许可证可在 licenses 文件夹中找到。

适用场景​

StarRocks 可以满足企业级用户的多种分析需求，包括 OLAP (Online Analytical Processing) 多维分析、定制报表、实时数据分析和 Ad-hoc 数据分析等。

OLAP 多维分析​

利用 StarRocks 的 MPP 框架和向量化执行引擎，用户可以灵活的选择雪花模型，星型模型，宽表模型或者预聚合模型。适用于灵活配置的多维分析报表，业务场景包括：

  • 用户行为分析
  • 用户画像、标签分析、圈人
  • 高维业务指标报表
  • 自助式报表平台
  • 业务问题探查分析
  • 跨主题业务分析
  • 财务报表
  • 系统监控分析
实时数据仓库​

StarRocks 设计和实现了主键表，能够实时更新数据并极速查询，可以秒级同步 TP (Transaction Processing) 数据库的变化，构建实时数仓，业务场景包括：

  • 电商大促数据分析
  • 物流行业的运单分析
  • 金融行业绩效分析、指标计算
  • 直播质量分析
  • 广告投放分析
  • 管理驾驶舱
  • 探针分析APM（Application Performance Management）
高并发查询​

StarRocks 通过良好的数据分布特性，灵活的索引以及物化视图等特性，可以解决面向用户侧的分析场景，业务场景包括：

  • 广告主报表分析
  • 零售行业渠道人员分析
  • SaaS 行业面向用户分析报表
  • Dashboard 多页面分析
统一分析​
  • 通过使用一套系统解决多维分析、高并发查询、预计算、实时分析查询等场景，降低系统复杂度和多技术栈开发与维护成本。
  • 使用 StarRocks 统一管理数据湖和数据仓库，将高并发和实时性要求很高的业务放在 StarRocks 中分析，也可以使用 External Catalog 和外部表进行数据湖上的分析。

存算一体架构

本地存储为实时查询提供了更低的查询延迟。 作为典型的大规模并行处理 (MPP) 数据库，StarRocks 支持存算一体架构。在这种架构中，BE 负责数据存储和计算。直接访问 BE 本地数据允许本地计算，避免了数据传输和复制，从而提供超快的查询和分析性能。该架构支持多副本数据存储，增强了集群处理高并发查询的能力并确保数据可靠性。非常适合追求最佳查询性能的场景。  

shared-data-1fac1b5ab7d46bf34f67c93ecc8e6c28

2.环境搭建

采用docker搭建最简单的测试环境

docker run -p 9030:9030 -p 8030:8030 -p 8040:8040 -itd --name quickstart starrocks/allin1-ubuntu

333

3.代码工程

实验目的

  1. 测试mysql创建 修改 插入删除数据
  2. 用streamload导入数据

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><parent><artifactId>springboot-demo</artifactId><groupId>com.et</groupId><version>1.0-SNAPSHOT</version></parent><modelVersion>4.0.0</modelVersion><artifactId>starrocks</artifactId><properties><maven.compiler.source>8</maven.compiler.source><maven.compiler.target>8</maven.compiler.target></properties><dependencies><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-autoconfigure</artifactId></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-test</artifactId><scope>test</scope></dependency><dependency><groupId>mysql</groupId><artifactId>mysql-connector-java</artifactId><version>5.1.48</version></dependency><dependency><groupId>org.apache.httpcomponents</groupId><artifactId>httpclient</artifactId><version>4.5.3</version></dependency></dependencies></project>

mysql连接

/**
Copyright (c) 2021 Beijing Dingshi Zongheng Technology Co., Ltd. All rights reserved.Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing,software distributed under the License is distributed on an"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANYKIND, either express or implied.  See the License for thespecific language governing permissions and limitationsunder the License.
**/package com.et.starrocks.mysql;import java.sql.*;public class MysqlClient {public static void main(String[] args) {String host = "172.30.17.1";//query_port in fe.confString port = "9030";String user = "root";//password is empty by defaultString password = "";//connect to starrocksConnection conn = null;try {conn = getConn(host, port, user, password, "");} catch (Exception e) {System.out.println("connect to starrocks failed");e.printStackTrace();return;}System.out.println("connect to starrocks successfully");//create statementStatement stmt = null;try {stmt = conn.createStatement();} catch (SQLException e) {System.out.println("create statement failed");e.printStackTrace();closeConn(conn);return;}System.out.println("create statement successfully");//create databasetry {stmt.execute("CREATE DATABASE IF NOT EXISTS db_test");} catch (SQLException e) {System.out.println("create database failed");e.printStackTrace();closeStmt(stmt);closeConn(conn);return;}System.out.println("create database successfully");//set db contexttry {stmt.execute("USE db_test");} catch (SQLException e) {System.out.println("set db context failed");e.printStackTrace();closeStmt(stmt);closeConn(conn);return;}System.out.println("set db context successfully");//create tabletry {stmt.execute("CREATE TABLE IF NOT EXISTS table_test(siteid INT, citycode SMALLINT, pv BIGINT SUM) " +"AGGREGATE KEY(siteid, citycode) " +"DISTRIBUTED BY HASH(siteid) BUCKETS 10 " +"PROPERTIES(\"replication_num\" = \"1\")");} catch (Exception e) {System.out.println("create table failed");e.printStackTrace();closeStmt(stmt);closeConn(conn);return;}System.out.println("create table successfully");//insert datatry {stmt.execute("INSERT INTO table_test values(1, 2, 3), (4, 5, 6), (1, 2, 4)");} catch (Exception e) {System.out.println("insert data failed");e.printStackTrace();closeStmt(stmt);closeConn(conn);return;}System.out.println("insert data successfully");//query datatry {ResultSet result = stmt.executeQuery("SELECT * FROM table_test");System.out.println("data queried is :");while (result.next()) {int siteid = result.getInt("siteid");int citycode = result.getInt("citycode");int pv = result.getInt("pv");System.out.println("\t" + siteid + "\t" + citycode + "\t" + pv);}} catch (Exception e) {System.out.println("query data failed");e.printStackTrace();closeStmt(stmt);closeConn(conn);return;}//drop database/*   try {stmt.execute("DROP DATABASE IF EXISTS db_test");} catch (Exception e) {System.out.println("drop database failed");e.printStackTrace();closeStmt(stmt);closeConn(conn);return;}*/System.out.println("drop database successfully");closeStmt(stmt);closeConn(conn);}public static Connection getConn(String host, String port, String user, String password, String database) throws Exception {Class.forName("com.mysql.jdbc.Driver");String url = "jdbc:mysql://" + host + ":" + port + "/" + database + "?user=" + user + "&password=" + password;return DriverManager.getConnection(url);}public static void closeConn(Connection conn) {try {conn.close();System.out.println("conn closed");} catch (Exception e) {System.out.println("close conn failed");e.printStackTrace();}}public static void closeStmt(Statement stmt) {try {stmt.close();System.out.println("stmt closed");} catch (Exception e) {System.out.println("close stmt failed");e.printStackTrace();}}
}

streamload方式

package com.et.starrocks.streamload;// Copyright (c) 2021 Beijing Dingshi Zongheng Technology Co., Ltd. All rights reserved.
//
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.import org.apache.commons.codec.binary.Base64;
import org.apache.http.HttpHeaders;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultRedirectStrategy;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;import java.io.IOException;
import java.nio.charset.StandardCharsets;
/*** This class is a java demo for starrocks stream load** The pom.xml dependency:**         <dependency>*             <groupId>org.apache.httpcomponents</groupId>*             <artifactId>httpclient</artifactId>*             <version>4.5.3</version>*         </dependency>** How to use:** 1 create a table in starrocks with any mysql client** CREATE TABLE `stream_test` (*   `id` bigint(20) COMMENT "",*   `id2` bigint(20) COMMENT "",*   `username` varchar(32) COMMENT ""* ) ENGINE=OLAP* DUPLICATE KEY(`id`)* DISTRIBUTED BY HASH(`id`) BUCKETS 20;*** 2 change the StarRocks cluster, db, user config in this class** 3 run this class, you should see the following output:** {*     "TxnId": 27,*     "Label": "39c25a5c-7000-496e-a98e-348a264c81de",*     "Status": "Success",*     "Message": "OK",*     "NumberTotalRows": 10,*     "NumberLoadedRows": 10,*     "NumberFilteredRows": 0,*     "NumberUnselectedRows": 0,*     "LoadBytes": 50,*     "LoadTimeMs": 151* }** Attention:** 1 wrong dependency version(such as 4.4) of httpclient may cause shaded.org.apache.http.ProtocolException*   Caused by: shaded.org.apache.http.ProtocolException: Content-Length header already present*     at shaded.org.apache.http.protocol.RequestContent.process(RequestContent.java:96)*     at shaded.org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)*     at shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182)*     at shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)*     at shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)*     at shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)**2 run this class more than once, the status code for http response is still ok, and you will see*  the following output:** {*     "TxnId": -1,*     "Label": "39c25a5c-7000-496e-a98e-348a264c81de",*     "Status": "Label Already Exists",*     "ExistingJobStatus": "FINISHED",*     "Message": "Label [39c25a5c-7000-496e-a98e-348a264c81de"] has already been used.",*     "NumberTotalRows": 0,*     "NumberLoadedRows": 0,*     "NumberFilteredRows": 0,*     "NumberUnselectedRows": 0,*     "LoadBytes": 0,*     "LoadTimeMs": 0* }* 3 when the response statusCode is 200, that doesn't mean your stream load is ok, there may be still*   some stream problem unless you see the output with 'ok' message*/
public class StarRocksStreamLoad {private final static String STARROCKS_HOST = "127.0.0.1";private final static String STARROCKS_DB = "db_test";private final static String STARROCKS_TABLE = "stream_test";private final static String STARROCKS_USER = "root";private final static String STARROCKS_PASSWORD = "";private final static int STARROCKS_HTTP_PORT = 8040;private void sendData(String content) throws Exception {final String loadUrl = String.format("http://%s:%s/api/%s/%s/_stream_load",STARROCKS_HOST,STARROCKS_HTTP_PORT,STARROCKS_DB,STARROCKS_TABLE);final HttpClientBuilder httpClientBuilder = HttpClients.custom().setRedirectStrategy(new DefaultRedirectStrategy() {@Overrideprotected boolean isRedirectable(String method) {return true;}});try (CloseableHttpClient client = httpClientBuilder.build()) {HttpPut put = new HttpPut(loadUrl);StringEntity entity = new StringEntity(content, "UTF-8");put.setHeader(HttpHeaders.EXPECT, "100-continue");put.setHeader(HttpHeaders.AUTHORIZATION, basicAuthHeader(STARROCKS_USER, STARROCKS_PASSWORD));// the label header is optional, not necessary// use label header can ensure at most once semanticsput.setHeader("label", "39c25a5c-7000-496e-a98e-348a264c81de1");put.setEntity(entity);try (CloseableHttpResponse response = client.execute(put)) {String loadResult = "";if (response.getEntity() != null) {loadResult = EntityUtils.toString(response.getEntity());}final int statusCode = response.getStatusLine().getStatusCode();// statusCode 200 just indicates that starrocks be service is ok, not stream load// you should see the output content to find whether stream load is successif (statusCode != 200) {throw new IOException(String.format("Stream load failed, statusCode=%s load result=%s", statusCode, loadResult));}System.out.println(loadResult);}}}private String basicAuthHeader(String username, String password) {final String tobeEncode = username + ":" + password;byte[] encoded = Base64.encodeBase64(tobeEncode.getBytes(StandardCharsets.UTF_8));return "Basic " + new String(encoded);}public static void main(String[] args) throws Exception {int id1 = 1;int id2 = 10;String id3 = "Simon";int rowNumber = 10;String oneRow = id1 + "\t" + id2 + "\t" + id3 + "\n";StringBuilder stringBuilder = new StringBuilder();for (int i = 0; i < rowNumber; i++) {stringBuilder.append(oneRow);}stringBuilder.deleteCharAt(stringBuilder.length() - 1);String loadData = stringBuilder.toString();StarRocksStreamLoad starrocksStreamLoad = new StarRocksStreamLoad();starrocksStreamLoad.sendData(loadData);}
}

以上只是一些关键代码，所有代码请参见下面代码仓库

代码仓库

  • https://github.com/Harries/springboot-demo

4.测试

MysqlClient

启动main方法，可以看到执行成功

connect to starrocks successfully
create statement successfully
create database successfully
set db context successfully
create table successfully
insert data successfully
data queried is :1 2 74 5 6
drop database successfully
stmt closed
conn closed

StarRocksStreamLoad

启动main方法，可以看到插入成功

20:51:47.521 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://127.0.0.1:8040] can be kept alive indefinitely
20:51:47.521 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://127.0.0.1:8040][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]
{"TxnId": 2,"Label": "39c25a5c-7000-496e-a98e-348a264c81de1","Status": "Success","Message": "OK","NumberTotalRows": 10,"NumberLoadedRows": 10,"NumberFilteredRows": 0,"NumberUnselectedRows": 0,"LoadBytes": 109,"LoadTimeMs": 975,"BeginTxnTimeMs": 261,"StreamLoadPlanTimeMs": 342,"ReadDataTimeMs": 0,"WriteDataTimeMs": 106,"CommitAndPublishTimeMs": 259
}

5.引用

  • Architecture | StarRocks
  • Spring Boot集成starrocks快速入门Demo | Harries Blog™

相关文章：

  • 北京网站建设多少钱？
  • 辽宁网页制作哪家好_网站建设
  • 高端品牌网站建设_汉中网站制作
  • laravel为Model设置全局作用域
  • Unity Apple Vision Pro 开发（四）：体积相机 Volume Camera
  • golang性能调试工具net/http/pprof
  • 时序数据库如何选型？详细指标总结！
  • ubuntu如何彻底卸载android studio？
  • 掌握Python中的文件序列化：Json和Pickle模块解析
  • 笔记：Few-Shot Learning小样本分类问题 + 孪生网络 + 预训练与微调
  • Python面试整理-字典和集合的操作
  • Eureka——Spring Cloud中的服务注册与发现组件
  • python运维实战-ssh工具
  • 如何建设和维护数据仓库：深入指南
  • 开源的语音合成工具_ChatTTS_用法及资源
  • 农场驿站平台小程序的设计
  • 概率论原理精解【4】
  • 微信小程序数组绑定使用案例（一）
  • #Java异常处理
  • ES学习笔记(12)--Symbol
  • iBatis和MyBatis在使用ResultMap对应关系时的区别
  • node 版本过低
  • oschina
  • React-flux杂记
  • 关于使用markdown的方法（引自CSDN教程）
  • 海量大数据大屏分析展示一步到位：DataWorks数据服务+MaxCompute Lightning对接DataV最佳实践...
  • 解析带emoji和链接的聊天系统消息
  • 力扣(LeetCode)56
  • 码农张的Bug人生 - 初来乍到
  • 三栏布局总结
  • 消息队列系列二（IOT中消息队列的应用）
  • 移动端解决方案学习记录
  • $refs 、$nextTic、动态组件、name的使用
  • （13）DroneCAN 适配器节点（一）
  • （3）(3.5) 遥测无线电区域条例
  • (Spark3.2.0)Spark SQL 初探： 使用大数据分析2000万KF数据
  • （定时器/计数器）中断系统（详解与使用）
  • （附源码）springboot 智能停车场系统 毕业设计065415
  • (附源码)计算机毕业设计ssm高校《大学语文》课程作业在线管理系统
  • （利用IDEA+Maven）定制属于自己的jar包
  • (三分钟了解debug)SLAM研究方向-Debug总结
  • (太强大了) - Linux 性能监控、测试、优化工具
  • （一）VirtualBox安装增强功能
  • (转)使用VMware vSphere标准交换机设置网络连接
  • (转)自己动手搭建Nginx+memcache+xdebug+php运行环境绿色版 For windows版
  • .NET 5.0正式发布，有什么功能特性(翻译)
  • .net mvc 获取url中controller和action
  • .Net语言中的StringBuilder：入门到精通
  • @Validated和@Valid校验参数区别
  • @zabbix数据库历史与趋势数据占用优化（mysql存储查询）
  • @基于大模型的旅游路线推荐方案
  • [ A*实现 ] C++，矩阵地图
  • [012-1].第12节：Mysql的配置文件的使用
  • [1]-基于图搜索的路径规划基础
  • [20160902]rm -rf的惨案.txt
  • [ACL2022] Text Smoothing: 一种在文本分类任务上的数据增强方法
  • [C#7] 1.Tuples（元组）
  • [C++]C++类基本语法