编译aws并访问minio

小飞侠hello2024-08-18 22:22

Aws

amazon (S3) 是一个公开的服务，Web 应用程序开发人员可以使用它存储数字资产，包括图片、视频、音乐和文档。 S3 提供一个 RESTful API 以编程方式实现与该服务的交互.

MinIO是兼容AWS SDK,所以可以通过aws访问minio文件系统。

指导文档：https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/getting-started.html

Aws安装
Windows系统
vcpkg install aws-sdk-cpp 命令行在线安装

需要先安装vcpkg，参看https://blog.csdn.net/baidu_16370559/article/details/140697821?spm=1001.2014.3001.5501

在powershell输入下面命令

vcpkg search aws-sdk-cpp

vcpkg install aws-sdk-cpp[s3]:x64-windows

报错，不能通过

源码安装：

参看https://docs.aws.amazon.com/zh_cn/sdk-for-cpp/v1/developer-guide/setup-windows.html

前提要求：

Install CMake (minimum version 3.13) ，可以在powershell 输入cmake --version 查看版本，这里cmake 的版本是

  1. 源码下载：GitHub - aws/aws-sdk-cpp: AWS SDK for C++有2种办法
  1. 直接从git下载zip文件

这里下载的版本是aws-sdk-cpp-1.11.370.zip. ctr/aws-crt-cpp目录为空，需要手动从git下载.

使用的aws-crt-cpp版本是v0.27.4,此时aws-crt-cpp目录的ctr目录下的各个文件夹也是空的。需要手动下载ctr目录下各文件夹的源码

  1. 在powershell通过git下装

前提条件是：安装git

在powershell 输入 git clone --recurse-submodules GitHub - aws/aws-sdk-cpp: AWS SDK for C++ 这样通过git下载。这种办法只能下载main分支的代码。

这种办法不一定会把所有的文件都下载，也可能需要手动下载依赖库。

  1. mkdir sdk_build cd sdk_build
  2. Generate build files

cmake ..\aws-sdk-cpp-1.11.370 -DCMAKE_BUILD_TYPE=Release -DBUILD_ONLY="s3" -DCMAKE_PREFIX_PATH="E:\code\aws\aws\aws-sdk-cpp-1.11.370\install" -DENABLE_TESTING=OFF

  1. cmake --build --config=Debug
  2. cmake --install --config=Debug

注意：步骤3过程中，使用powershell编译出错.改成cmake-gui进行编译。

ALL_BUILD右键，生成

单击INSTALL,右键，仅用于项目，仅生成INSTALL

Windows系统使用aws

VS2019

直接使用aws,编译测试demo会报错。

解决方法

官方文档有说到，要添加USE_IMPORT_EXPORT和

USE_WINDOWS_DLL_SEMANTICS两个宏，所以这个问题是符号导出的问题

QT

解决办法：

在pro文件加上预定义

DEFINES +=USE_IMPORT_EXPORT

DEFINES +=USE_WINDOWS_DLL_SEMANTICS

Linux系统
vcpkg install aws-sdk-cpp 命令行在线安装

* aws-c-auth:x64-linux@0.7.22

* aws-c-cal:x64-linux@0.6.15

* aws-c-common:x64-linux@0.9.21

* aws-c-compression:x64-linux@0.2.18

* aws-c-event-stream:x64-linux@0.4.2

* aws-c-http:x64-linux@0.8.2

* aws-c-io:x64-linux@0.14.9

* aws-c-mqtt:x64-linux@0.10.4

* aws-c-s3:x64-linux@0.5.10

* aws-c-sdkutils:x64-linux@0.1.16

* aws-checksums:x64-linux@0.1.18

* aws-crt-cpp:x64-linux@0.26.12

aws-sdk-cpp[core,dynamodb,kinesis,s3]:x64-linux@1.11.352

* s2n:x64-linux@1.4.16

没通过，应该因为需要依赖太多第三方库，直接在线安装很多时候下载不成功

源码安装

下载地址： GitHub - aws/aws-sdk-cpp: AWS SDK for C++

参考： Build the AWS SDK for C++ on Linux/macOS - AWS SDK for C++

前提：

GNU Compiler Collection (GCC) 4.9 or later, or

Install CMake (minimum version 3.13

额外的要求：

sudo apt-get install libcurl4-openssl-dev libssl-dev uuid-dev zlib1g-dev libpulse-dev

安装步骤：

  1. 下载源码

源码地址：GitHub - aws/aws-sdk-cpp: AWS SDK for C++

有2种办法

  1. 直接从git下载zip文件

这里下载的版本是aws-sdk-cpp-1.11.370.zip. 如果在windows系统已经下载好源码，可以直接拿来用。aws-crt-cpp版本是v0.27.4。此时aws-crt-cpp目录的ctr目录下的各个文件夹也是空的。需要手动下载ctr目录下各文件夹的源码

  1. 在shell 通过git下装

前提条件是：安装git

git clone --recurse-submodules GitHub - aws/aws-sdk-cpp: AWS SDK for C++

注意事项：ctr/aws-crt-cpp目录下的很多文件可能拉不下来，需要手动从git下载.

  1. mkdir sdk_build cd sdk_build
  2. Generate build files

这里我使用的是cmake ../aws-sdk-cpp -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/root/software/aws/aws/install/ -DBUILD_ONLY="s3" -DENABLE_TESTING=OFF

  1. make -j4
  2. make install

注意事项：

我这里A.直接从git下载zip文件，编译通过后。调用aws库，会出现libaws-c-s3.so库缺少

libcrypto.so.3。因为我的电脑openssl的版本只有libcrypto.so.1.版本太低了。需要升版libcrypto。

升版方法参考：编译openssl文章。具体操作是从Downloads | Library下载源码编译安装，然后设置软连接，同时把openssl的lib库加载环境变量。

客户端使用aws

要求

  1. 最开始必须调用Aws::InitAPI
  2. 在程序关闭前必须调用Aws::ShutdownAPI
  3. 其他的aws的api在Aws::InitAPI和Aws::ShutdownAPI之间调用
QT通过aws访问minio
pro配置

INCLUDEPATH += $$PWD/install/include

DEPENDPATH += $$PWD/install/include

DESTDIR=$$PWD/bin

LIBS += -L$$PWD/install/lib/ -laws-c-auth -laws-c-cal\

-laws-c-common -laws-c-compression -laws-c-event-stream -laws-checksums -laws-c-http -laws-c-io\

-laws-c-mqtt -laws-cpp-sdk-core -laws-crt-cpp -laws-c-s3 -laws-c-sdkutils -ls2n -laws-cpp-sdk-s3

代码编写

参看：https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/examples-s3-objects.html

#ifndef MINIOSERVICE_H

#define MINIOSERVICE_H

#include <QMutex>

#include <QString>

#include <QStringList>

#include "aws/s3/S3Client.h"

#include "aws/core/Aws.h"

#include "aws/core/auth/AWSCredentialsProvider.h"

#include "aws/s3/model/PutObjectRequest.h"

#include "aws/s3/model/GetObjectRequest.h"

#include "aws/s3/model/CreateBucketRequest.h"

#include "aws/s3/model/DeleteObjectRequest.h"

#include "aws/s3/model/DeleteBucketRequest.h"

#include "aws/s3/model/ListObjectsRequest.h"

using namespace std;

using namespace Aws::S3;

using namespace Aws::S3::Model;

struct RetStatus

{

int nRetCode = 0;

QString msg ="成功";

};

class MinioService

{

public:

MinioService();

~MinioService();

public:

//单例

static MinioService * GetInstance();

//释放

static void DeleteInstance();

/**

* @brief LoginMinio 登录minio文件系统

* @return 错误码

*/

RetStatus LoginMinio();

RetStatus CreateBucket(QString Bucketname);

RetStatus GetBuckets(QStringList& bucketsname);

RetStatus GetBucketFile(QString Bucketname,QStringList& fileList);

/**

* @brief DownLoadFile 从minio文件系统下载文件

* @param strBucket 桶的名称

* @param strObject 需下装的内容

* @param strSavePath 下载到本地的路径

* @return 错误码

*/

RetStatus DownLoadFile(QString strBucket,QString strObject,QString strSavePath);

/**

* @brief UpLoadFile 把本地文件上传到minio文件系统

* @param strBucket 桶的名称

* @param strObject 存放到minio的文件路径

* @param strFilePath 本地文件路径

* @return

*/

RetStatus UpLoadFile(QString strBucket,QString strObject,QString strFilePath);

RetStatus DeleteFile(QString strBucket,QString strObject);

// // 删除Bucket，一定要先删除文件，再删除Bucket

RetStatus DeleteBucket(QString strBucket);

private:

static MinioService *m_pInstance;

static QMutex m_mutex;

S3Client* m_client;

Aws::SDKOptions m_options;

};

#endif // MINIOSERVICE_H

#include "minioservice.h"

#include <QFile>

MinioService *MinioService::m_pInstance = nullptr;

QMutex MinioService::m_mutex;

MinioService::MinioService()

{

m_client = nullptr;

m_options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Debug;

Aws::InitAPI(m_options);

Aws::Client::ClientConfiguration cfg;

cfg.endpointOverride = "127.0.0.1:3003";

cfg.scheme = Aws::Http::Scheme::HTTP;

cfg.verifySSL = false;

Aws::Auth::AWSCredentials cred("minioadmin", "minioadmin");

m_client = new S3Client(cred, cfg, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Always, false,

Aws::S3::US_EAST_1_REGIONAL_ENDPOINT_OPTION::NOT_SET);

}

MinioService::~MinioService()

{

Aws::ShutdownAPI(m_options);

if(m_client)

{

delete m_client;

m_client = nullptr;

}

}

MinioService *MinioService::GetInstance()

{

if(nullptr == m_pInstance)

{

m_mutex.lock();

if( nullptr == m_pInstance)

{

m_pInstance = new MinioService();

}

m_mutex.unlock();

}

return m_pInstance;

}

void MinioService::DeleteInstance()

{

m_mutex.lock();

if(m_pInstance)

{

delete m_pInstance;

m_pInstance = nullptr;

}

m_mutex.unlock();

}

RetStatus MinioService::LoginMinio()

{

RetStatus Reture;

if(m_client == nullptr)

{

Reture.nRetCode = -1;

Reture.msg = "创建连接客户端不成功。";

return Reture;

}

auto response = m_client->ListBuckets();

if (response.IsSuccess())

{

Reture.nRetCode = 0;

Reture.msg = QString("登录minio文件系统成功.");

return Reture;

}

else

{

Reture.nRetCode = 1;

Reture.msg = "登录失败";

}

return Reture;

}

RetStatus MinioService::CreateBucket(QString Bucketname)

{

RetStatus Reture;

if(m_client == nullptr)

{

Reture.nRetCode = -1;

Reture.msg = "创建连接客户端不成功。";

return Reture;

}

Aws::S3::Model::CreateBucketRequest create_bucket_request;

create_bucket_request.SetBucket(Bucketname.toStdString());

create_bucket_request.SetCreateBucketConfiguration({});

auto create_bucket_outcome = m_client->CreateBucket(create_bucket_request);

if (create_bucket_outcome.IsSuccess())

{

Reture.nRetCode = 0;

Reture.msg = "创建Bucket成功。";

return Reture;

}

else

{

string strErr = create_bucket_outcome.GetError().GetMessage();

Reture.nRetCode = 2;

Reture.msg = "创建Bucket失败：" + QString::fromStdString(strErr) ;

}

return Reture;

}

RetStatus MinioService::GetBuckets(QStringList& bucketsname)

{

bucketsname.clear();

RetStatus Reture;

if(m_client == nullptr)

{

Reture.nRetCode = -1;

Reture.msg = "创建连接客户端不成功。";

return Reture;

}

auto response = m_client->ListBuckets();

if (response.IsSuccess())

{

auto buckets = response.GetResult().GetBuckets();

for(auto bucket :buckets)

{

bucketsname.append(QString::fromStdString(bucket.GetName()));

}

Reture.nRetCode = 0;

Reture.msg = "获取buckets成功。";

}

else

{

Reture.nRetCode = 3;

string strErr = response.GetError().GetMessage();

Reture.msg = "获取buckets不成功:" + QString::fromStdString(strErr) ;

}

return Reture;

}

RetStatus MinioService::GetBucketFile(QString Bucketname, QStringList &fileList)

{

fileList.clear();

RetStatus Reture;

if(m_client == nullptr)

{

Reture.nRetCode = -1;

Reture.msg = "创建连接客户端不成功。";

return Reture;

}

Aws::S3::Model::Bucket temps{};

std::string stringK = Bucketname.toStdString();

temps.SetName(stringK);

/// 查找MINIO相关数据

Aws::S3::Model::ListObjectsRequest objects_request;

objects_request.WithBucket(temps.GetName()); /// 设置桶路径

auto list_objects_outcome = m_client->ListObjects(objects_request); /// 判断存储桶及相关连接是否有效

if (list_objects_outcome.IsSuccess())

{

Aws::Vector<Aws::S3::Model::Object> object_list = list_objects_outcome.GetResult().GetContents();

for (auto s3_object : object_list)

{

QString filename =QString::fromStdString(s3_object.GetKey());

fileList.append(filename);

}

Reture.nRetCode = 0;

Reture.msg = "获取文件列表成功。";

}

else

{

Reture.nRetCode = 8;

string strErr = list_objects_outcome.GetError().GetMessage();

Reture.msg = "获取文件列表失败:" + QString::fromStdString(strErr) ;

}

return Reture;

}

RetStatus MinioService::DownLoadFile(QString strBucket, QString strObject, QString strSavePath)

{

RetStatus Reture;

if(m_client == nullptr)

{

Reture.nRetCode = -1;

Reture.msg = "创建连接客户端不成功。";

return Reture;

}

Aws::S3::Model::GetObjectRequest object_request;

object_request.WithBucket(strBucket.toStdString().c_str()).WithKey(strObject.toStdString().c_str());

auto get_object_outcome = m_client->GetObject(object_request);

if (get_object_outcome.IsSuccess())

{

Aws::IOStream &local_file = get_object_outcome.GetResult().GetBody();

std::istreambuf_iterator<char> begin(local_file);

std::istreambuf_iterator<char> end;

std::string buffer(begin, end); // 读取整个文件内容

QFile file_(strSavePath);

if(file_.open(QIODevice::ReadWrite | QIODevice::Truncate))//以截取方式打开文件,文件原有的内容全部被删除

{

file_.write(QByteArray::fromStdString(buffer));

file_.close();

Reture.nRetCode = 0;

Reture.msg = "下载文件成功。";

}

else

{

Reture.nRetCode = 4;

Reture.msg = "下载文件失败:创建本地保存文件失败";

}

}

else

{

string strErr = get_object_outcome.GetError().GetMessage() ;

Reture.nRetCode = 5;

Reture.msg = "下载文件失败:" + QString::fromStdString(strErr);

}

return Reture;

}

RetStatus MinioService::UpLoadFile(QString strBucket, QString strObject, QString strFilePath)

{

RetStatus Reture;

if(m_client == nullptr)

{

Reture.nRetCode = -1;

Reture.msg = "创建连接客户端不成功。";

return Reture;

}

QFile file(strFilePath);

bool bOpen = file.open(QIODevice::ReadOnly);

Aws::S3::Model::PutObjectRequest putObjectRequest;

putObjectRequest.WithBucket(strBucket.toStdString().c_str()).WithKey(strObject.toStdString().c_str());

std::shared_ptr<Aws::IOStream> input_data = Aws::MakeShared<Aws::StringStream>("PutObjectInputStream");

*input_data << file.readAll().toStdString();

file.close();

putObjectRequest.SetBody(input_data);

auto putObjectResult = m_client->PutObject(putObjectRequest);

if (putObjectResult.IsSuccess())

{

Reture.nRetCode = 0;

Reture.msg = "上传文件成功";

}

else

{

string strErr = putObjectResult.GetError().GetMessage() ;

Reture.nRetCode = 6;

Reture.msg = "上传文件失败:" + QString::fromStdString(strErr);

}

return Reture;

}

RetStatus MinioService::DeleteFile(QString strBucket, QString strObject)

{

RetStatus Reture;

if(m_client == nullptr)

{

Reture.nRetCode = -1;

Reture.msg = "创建连接客户端不成功。";

return Reture;

}

Aws::S3::Model::DeleteObjectRequest delete_object_request;

delete_object_request.WithBucket(strBucket.toStdString().c_str()).WithKey(strObject.toStdString().c_str());

auto delete_object_outcome = m_client->DeleteObject(delete_object_request);

if (delete_object_outcome.IsSuccess())

{

Reture.nRetCode = 0;

Reture.msg = "删除文件成功";

}

else

{

string strErr = delete_object_outcome.GetError().GetMessage() ;

Reture.nRetCode = 7;

Reture.msg = "删除文件失败:" + QString::fromStdString(strErr);

}

return Reture;

}

RetStatus MinioService::DeleteBucket(QString strBucket)

{

RetStatus Reture;

if(m_client == nullptr)

{

Reture.nRetCode = -1;

Reture.msg = "创建连接客户端不成功。";

return Reture;

}

Aws::S3::Model::DeleteBucketRequest delete_bucket_request;

delete_bucket_request.SetBucket(strBucket.toStdString().c_str());

auto delete_bucket_outcome = m_client->DeleteBucket(delete_bucket_request);

if (delete_bucket_outcome.IsSuccess())

{

Reture.nRetCode = 0;

Reture.msg = "删除bucket成功";

}

else

{

string strErr = delete_bucket_outcome.GetError().GetMessage() ;

Reture.nRetCode = 7;

Reture.msg = "删除文件失败:" + QString::fromStdString(strErr);

}

return Reture;

}

相关推荐
BeyondESH35 分钟前
Linux线程同步—竞态条件和互斥锁(C语言)
linux·服务器·c++
豆浩宇1 小时前
Halcon OCR检测 免训练版
c++·人工智能·opencv·算法·计算机视觉·ocr
WG_171 小时前
C++多态
开发语言·c++·面试
Charles Ray2 小时前
C++学习笔记 —— 内存分配 new
c++·笔记·学习
重生之我在20年代敲代码2 小时前
strncpy函数的使用和模拟实现
c语言·开发语言·c++·经验分享·笔记
迷迭所归处8 小时前
C++ —— 关于vector
开发语言·c++·算法
CV工程师小林9 小时前
【算法】BFS 系列之边权为 1 的最短路问题
数据结构·c++·算法·leetcode·宽度优先
white__ice9 小时前
2024.9.19
c++
天玑y9 小时前
算法设计与分析（背包问题
c++·经验分享·笔记·学习·算法·leetcode·蓝桥杯
锦亦之223310 小时前
QT+OSG+OSG-earth如何在窗口显示一个地球
开发语言·qt
热门推荐
01RAG 实践- Ollama+RagFlow 部署本地知识库02组基轨迹建模 GBTM的介绍与实现（Stata 或 R）032024年高教社杯数学建模国赛C题超详细解题思路分析04Coze扣子平台完整体验和实践（附国内和国际版对比）05苍穹外卖面试总结06【2024数模国赛赛题思路公开】国赛B题思路丨附可运行代码丨无偿自提07yolov8实战第五天——yolov8+ffmpg实时视频流检测并进行实时推流——（推流，保姆教学）08CCF-CSP认证考试 202406-3 文本分词 100分题解09【2024高教社杯全国大学生数学建模竞赛】B题 生产过程中的决策问题——解题思路 代码 论文10浏览器插件——ModHeader 的分享和学习