编译aws并访问minio
Aws
amazon (S3) 是一个公开的服务，Web 应用程序开发人员可以使用它存储数字资产，包括图片、视频、音乐和文档。 S3 提供一个 RESTful API 以编程方式实现与该服务的交互.
MinIO是兼容AWS SDK,所以可以通过aws访问minio文件系统。
指导文档：https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/getting-started.html
Aws安装
Windows系统
vcpkg install aws-sdk-cpp 命令行在线安装
需要先安装vcpkg，参看https://blog.csdn.net/baidu_16370559/article/details/140697821?spm=1001.2014.3001.5501
在powershell输入下面命令
vcpkg search aws-sdk-cpp
vcpkg install aws-sdk-cpp[s3]:x64-windows
报错，不能通过
源码安装：
参看https://docs.aws.amazon.com/zh_cn/sdk-for-cpp/v1/developer-guide/setup-windows.html
前提要求：
Install CMake (minimum version 3.13) ，可以在powershell 输入cmake --version 查看版本，这里cmake 的版本是
- 源码下载：GitHub - aws/aws-sdk-cpp: AWS SDK for C++ 有2种办法
- 直接从git下载zip文件
这里下载的版本是aws-sdk-cpp-1.11.370.zip. ctr/aws-crt-cpp目录为空，需要手动从git下载.
使用的aws-crt-cpp版本是v0.27.4,此时aws-crt-cpp目录的ctr目录下的各个文件夹也是空的。需要手动下载ctr目录下各文件夹的源码
- 在powershell通过git下装
前提条件是：安装git
在powershell 输入 git clone --recurse-submodules GitHub - aws/aws-sdk-cpp: AWS SDK for C++ 这样通过git下载。这种办法只能下载main分支的代码。
这种办法不一定会把所有的文件都下载，也可能需要手动下载依赖库。
- mkdir sdk_build cd sdk_build
- Generate build files
cmake ..\aws-sdk-cpp-1.11.370 -DCMAKE_BUILD_TYPE=Release -DBUILD_ONLY="s3" -DCMAKE_PREFIX_PATH="E:\code\aws\aws\aws-sdk-cpp-1.11.370\install" -DENABLE_TESTING=OFF
- cmake --build --config=Debug
- cmake --install --config=Debug
注意：步骤3过程中，使用powershell编译出错.改成cmake-gui进行编译。
ALL_BUILD右键，生成
单击INSTALL,右键，仅用于项目，仅生成INSTALL
Windows系统使用aws
VS2019
直接使用aws,编译测试demo会报错。
解决方法
官方文档有说到，要添加USE_IMPORT_EXPORT和
USE_WINDOWS_DLL_SEMANTICS两个宏，所以这个问题是符号导出的问题
QT
解决办法：
在pro文件加上预定义
DEFINES +=USE_IMPORT_EXPORT
DEFINES +=USE_WINDOWS_DLL_SEMANTICS
Linux系统
vcpkg install aws-sdk-cpp 命令行在线安装
* aws-c-auth:x64-linux@0.7.22
* aws-c-cal:x64-linux@0.6.15
* aws-c-common:x64-linux@0.9.21
* aws-c-compression:x64-linux@0.2.18
* aws-c-event-stream:x64-linux@0.4.2
* aws-c-http:x64-linux@0.8.2
* aws-c-io:x64-linux@0.14.9
* aws-c-mqtt:x64-linux@0.10.4
* aws-c-s3:x64-linux@0.5.10
* aws-c-sdkutils:x64-linux@0.1.16
* aws-checksums:x64-linux@0.1.18
* aws-crt-cpp:x64-linux@0.26.12
aws-sdk-cpp[core,dynamodb,kinesis,s3]:x64-linux@1.11.352
* s2n:x64-linux@1.4.16
没通过，应该因为需要依赖太多第三方库，直接在线安装很多时候下载不成功
源码安装
下载地址：GitHub - aws/aws-sdk-cpp: AWS SDK for C++
参考：Build the AWS SDK for C++ on Linux/macOS - AWS SDK for C++
前提：
GNU Compiler Collection (GCC) 4.9 or later, or
Install CMake (minimum version 3.13
额外的要求：
sudo apt-get install libcurl4-openssl-dev libssl-dev uuid-dev zlib1g-dev libpulse-dev
安装步骤：
- 下载源码
源码地址：GitHub - aws/aws-sdk-cpp: AWS SDK for C++
有2种办法
- 直接从git下载zip文件
这里下载的版本是aws-sdk-cpp-1.11.370.zip. 如果在windows系统已经下载好源码，可以直接拿来用。aws-crt-cpp版本是v0.27.4。此时aws-crt-cpp目录的ctr目录下的各个文件夹也是空的。需要手动下载ctr目录下各文件夹的源码
- 在shell 通过git下装
前提条件是：安装git
git clone --recurse-submodules GitHub - aws/aws-sdk-cpp: AWS SDK for C++
注意事项：ctr/aws-crt-cpp目录下的很多文件可能拉不下来，需要手动从git下载.
- mkdir sdk_build cd sdk_build
- Generate build files
这里我使用的是cmake ../aws-sdk-cpp -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/root/software/aws/aws/install/ -DBUILD_ONLY="s3" -DENABLE_TESTING=OFF
- make -j4
- make install
注意事项：
我这里A.直接从git下载zip文件，编译通过后。调用aws库，会出现libaws-c-s3.so库缺少
libcrypto.so.3。因为我的电脑openssl的版本只有libcrypto.so.1.版本太低了。需要升版libcrypto。
升版方法参考：编译openssl文章。具体操作是从Downloads | Library下载源码编译安装，然后设置软连接，同时把openssl的lib库加载环境变量。
客户端使用aws
要求
- 最开始必须调用Aws::InitAPI
- 在程序关闭前必须调用Aws::ShutdownAPI
- 其他的aws的api在Aws::InitAPI和Aws::ShutdownAPI之间调用
QT通过aws访问minio
pro配置
INCLUDEPATH += $$PWD/install/include
DEPENDPATH += $$PWD/install/include
DESTDIR=$$PWD/bin
LIBS += -L$$PWD/install/lib/ -laws-c-auth -laws-c-cal\
-laws-c-common -laws-c-compression -laws-c-event-stream -laws-checksums -laws-c-http -laws-c-io\
-laws-c-mqtt -laws-cpp-sdk-core -laws-crt-cpp -laws-c-s3 -laws-c-sdkutils -ls2n -laws-cpp-sdk-s3
代码编写
参看：https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/examples-s3-objects.html
#ifndef MINIOSERVICE_H
#define MINIOSERVICE_H
#include <QMutex>
#include <QString>
#include <QStringList>
#include "aws/s3/S3Client.h"
#include "aws/core/Aws.h"
#include "aws/core/auth/AWSCredentialsProvider.h"
#include "aws/s3/model/PutObjectRequest.h"
#include "aws/s3/model/GetObjectRequest.h"
#include "aws/s3/model/CreateBucketRequest.h"
#include "aws/s3/model/DeleteObjectRequest.h"
#include "aws/s3/model/DeleteBucketRequest.h"
#include "aws/s3/model/ListObjectsRequest.h"
using namespace std;
using namespace Aws::S3;
using namespace Aws::S3::Model;
struct RetStatus
{
int nRetCode = 0;
QString msg ="成功";
};
class MinioService
{
public:
MinioService();
~MinioService();
public:
//单例
static MinioService * GetInstance();
//释放
static void DeleteInstance();
/**
* @brief LoginMinio 登录minio文件系统
* @return 错误码
*/
RetStatus LoginMinio();
RetStatus CreateBucket(QString Bucketname);
RetStatus GetBuckets(QStringList& bucketsname);
RetStatus GetBucketFile(QString Bucketname,QStringList& fileList);
/**
* @brief DownLoadFile 从minio文件系统下载文件
* @param strBucket 桶的名称
* @param strObject 需下装的内容
* @param strSavePath 下载到本地的路径
* @return 错误码
*/
RetStatus DownLoadFile(QString strBucket,QString strObject,QString strSavePath);
/**
* @brief UpLoadFile 把本地文件上传到minio文件系统
* @param strBucket 桶的名称
* @param strObject 存放到minio的文件路径
* @param strFilePath 本地文件路径
* @return
*/
RetStatus UpLoadFile(QString strBucket,QString strObject,QString strFilePath);
RetStatus DeleteFile(QString strBucket,QString strObject);
// // 删除Bucket，一定要先删除文件，再删除Bucket
RetStatus DeleteBucket(QString strBucket);
private:
static MinioService *m_pInstance;
static QMutex m_mutex;
S3Client* m_client;
Aws::SDKOptions m_options;
};
#endif // MINIOSERVICE_H
#include "minioservice.h"
#include <QFile>
MinioService *MinioService::m_pInstance = nullptr;
QMutex MinioService::m_mutex;
MinioService::MinioService()
{
m_client = nullptr;
m_options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Debug;
Aws::InitAPI(m_options);
Aws::Client::ClientConfiguration cfg;
cfg.endpointOverride = "127.0.0.1:3003";
cfg.scheme = Aws::Http::Scheme::HTTP;
cfg.verifySSL = false;
Aws::Auth::AWSCredentials cred("minioadmin", "minioadmin");
m_client = new S3Client(cred, cfg, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Always, false,
Aws::S3::US_EAST_1_REGIONAL_ENDPOINT_OPTION::NOT_SET);
}
MinioService::~MinioService()
{
Aws::ShutdownAPI(m_options);
if(m_client)
{
delete m_client;
m_client = nullptr;
}
}
MinioService *MinioService::GetInstance()
{
if(nullptr == m_pInstance)
{
m_mutex.lock();
if( nullptr == m_pInstance)
{
m_pInstance = new MinioService();
}
m_mutex.unlock();
}
return m_pInstance;
}
void MinioService::DeleteInstance()
{
m_mutex.lock();
if(m_pInstance)
{
delete m_pInstance;
m_pInstance = nullptr;
}
m_mutex.unlock();
}
RetStatus MinioService::LoginMinio()
{
RetStatus Reture;
if(m_client == nullptr)
{
Reture.nRetCode = -1;
Reture.msg = "创建连接客户端不成功。";
return Reture;
}
auto response = m_client->ListBuckets();
if (response.IsSuccess())
{
Reture.nRetCode = 0;
Reture.msg = QString("登录minio文件系统成功.");
return Reture;
}
else
{
Reture.nRetCode = 1;
Reture.msg = "登录失败";
}
return Reture;
}
RetStatus MinioService::CreateBucket(QString Bucketname)
{
RetStatus Reture;
if(m_client == nullptr)
{
Reture.nRetCode = -1;
Reture.msg = "创建连接客户端不成功。";
return Reture;
}
Aws::S3::Model::CreateBucketRequest create_bucket_request;
create_bucket_request.SetBucket(Bucketname.toStdString());
create_bucket_request.SetCreateBucketConfiguration({});
auto create_bucket_outcome = m_client->CreateBucket(create_bucket_request);
if (create_bucket_outcome.IsSuccess())
{
Reture.nRetCode = 0;
Reture.msg = "创建Bucket成功。";
return Reture;
}
else
{
string strErr = create_bucket_outcome.GetError().GetMessage();
Reture.nRetCode = 2;
Reture.msg = "创建Bucket失败：" + QString::fromStdString(strErr) ;
}
return Reture;
}
RetStatus MinioService::GetBuckets(QStringList& bucketsname)
{
bucketsname.clear();
RetStatus Reture;
if(m_client == nullptr)
{
Reture.nRetCode = -1;
Reture.msg = "创建连接客户端不成功。";
return Reture;
}
auto response = m_client->ListBuckets();
if (response.IsSuccess())
{
auto buckets = response.GetResult().GetBuckets();
for(auto bucket :buckets)
{
bucketsname.append(QString::fromStdString(bucket.GetName()));
}
Reture.nRetCode = 0;
Reture.msg = "获取buckets成功。";
}
else
{
Reture.nRetCode = 3;
string strErr = response.GetError().GetMessage();
Reture.msg = "获取buckets不成功:" + QString::fromStdString(strErr) ;
}
return Reture;
}
RetStatus MinioService::GetBucketFile(QString Bucketname, QStringList &fileList)
{
fileList.clear();
RetStatus Reture;
if(m_client == nullptr)
{
Reture.nRetCode = -1;
Reture.msg = "创建连接客户端不成功。";
return Reture;
}
Aws::S3::Model::Bucket temps{};
std::string stringK = Bucketname.toStdString();
temps.SetName(stringK);
/// 查找MINIO相关数据
Aws::S3::Model::ListObjectsRequest objects_request;
objects_request.WithBucket(temps.GetName()); /// 设置桶路径
auto list_objects_outcome = m_client->ListObjects(objects_request); /// 判断存储桶及相关连接是否有效
if (list_objects_outcome.IsSuccess())
{
Aws::Vector<Aws::S3::Model::Object> object_list = list_objects_outcome.GetResult().GetContents();
for (auto s3_object : object_list)
{
QString filename =QString::fromStdString(s3_object.GetKey());
fileList.append(filename);
}
Reture.nRetCode = 0;
Reture.msg = "获取文件列表成功。";
}
else
{
Reture.nRetCode = 8;
string strErr = list_objects_outcome.GetError().GetMessage();
Reture.msg = "获取文件列表失败:" + QString::fromStdString(strErr) ;
}
return Reture;
}
RetStatus MinioService::DownLoadFile(QString strBucket, QString strObject, QString strSavePath)
{
RetStatus Reture;
if(m_client == nullptr)
{
Reture.nRetCode = -1;
Reture.msg = "创建连接客户端不成功。";
return Reture;
}
Aws::S3::Model::GetObjectRequest object_request;
object_request.WithBucket(strBucket.toStdString().c_str()).WithKey(strObject.toStdString().c_str());
auto get_object_outcome = m_client->GetObject(object_request);
if (get_object_outcome.IsSuccess())
{
Aws::IOStream &local_file = get_object_outcome.GetResult().GetBody();
std::istreambuf_iterator<char> begin(local_file);
std::istreambuf_iterator<char> end;
std::string buffer(begin, end); // 读取整个文件内容
QFile file_(strSavePath);
if(file_.open(QIODevice::ReadWrite | QIODevice::Truncate))//以截取方式打开文件,文件原有的内容全部被删除
{
file_.write(QByteArray::fromStdString(buffer));
file_.close();
Reture.nRetCode = 0;
Reture.msg = "下载文件成功。";
}
else
{
Reture.nRetCode = 4;
Reture.msg = "下载文件失败:创建本地保存文件失败";
}
}
else
{
string strErr = get_object_outcome.GetError().GetMessage() ;
Reture.nRetCode = 5;
Reture.msg = "下载文件失败:" + QString::fromStdString(strErr);
}
return Reture;
}
RetStatus MinioService::UpLoadFile(QString strBucket, QString strObject, QString strFilePath)
{
RetStatus Reture;
if(m_client == nullptr)
{
Reture.nRetCode = -1;
Reture.msg = "创建连接客户端不成功。";
return Reture;
}
QFile file(strFilePath);
bool bOpen = file.open(QIODevice::ReadOnly);
Aws::S3::Model::PutObjectRequest putObjectRequest;
putObjectRequest.WithBucket(strBucket.toStdString().c_str()).WithKey(strObject.toStdString().c_str());
std::shared_ptr<Aws::IOStream> input_data = Aws::MakeShared<Aws::StringStream>("PutObjectInputStream");
*input_data << file.readAll().toStdString();
file.close();
putObjectRequest.SetBody(input_data);
auto putObjectResult = m_client->PutObject(putObjectRequest);
if (putObjectResult.IsSuccess())
{
Reture.nRetCode = 0;
Reture.msg = "上传文件成功";
}
else
{
string strErr = putObjectResult.GetError().GetMessage() ;
Reture.nRetCode = 6;
Reture.msg = "上传文件失败:" + QString::fromStdString(strErr);
}
return Reture;
}
RetStatus MinioService::DeleteFile(QString strBucket, QString strObject)
{
RetStatus Reture;
if(m_client == nullptr)
{
Reture.nRetCode = -1;
Reture.msg = "创建连接客户端不成功。";
return Reture;
}
Aws::S3::Model::DeleteObjectRequest delete_object_request;
delete_object_request.WithBucket(strBucket.toStdString().c_str()).WithKey(strObject.toStdString().c_str());
auto delete_object_outcome = m_client->DeleteObject(delete_object_request);
if (delete_object_outcome.IsSuccess())
{
Reture.nRetCode = 0;
Reture.msg = "删除文件成功";
}
else
{
string strErr = delete_object_outcome.GetError().GetMessage() ;
Reture.nRetCode = 7;
Reture.msg = "删除文件失败:" + QString::fromStdString(strErr);
}
return Reture;
}
RetStatus MinioService::DeleteBucket(QString strBucket)
{
RetStatus Reture;
if(m_client == nullptr)
{
Reture.nRetCode = -1;
Reture.msg = "创建连接客户端不成功。";
return Reture;
}
Aws::S3::Model::DeleteBucketRequest delete_bucket_request;
delete_bucket_request.SetBucket(strBucket.toStdString().c_str());
auto delete_bucket_outcome = m_client->DeleteBucket(delete_bucket_request);
if (delete_bucket_outcome.IsSuccess())
{
Reture.nRetCode = 0;
Reture.msg = "删除bucket成功";
}
else
{
string strErr = delete_bucket_outcome.GetError().GetMessage() ;
Reture.nRetCode = 7;
Reture.msg = "删除文件失败:" + QString::fromStdString(strErr);
}
return Reture;
}