开源的不需要写代码的爬虫maxun

转自github热门项目

GitHub - getmaxun/maxun: ?? Open-source no-code web data extraction platform. Turn websites to APIs and spreadsheets with no-code robots in minutes! In Beta,更多详细信息见github。本文主要是讲一下自己部署遇到的一些小问题。可以直接看最后一节,获得博主专属个人经验。

Maxun Open-Source No-Code Web Data Extraction Platform

Local Setup

Docker Compose
复制代码
git clone https://github.com/getmaxun/maxun
docker-compose up -d --build
Without Docker
  1. Ensure you have Node.js, PostgreSQL, MinIO and Redis installed on your system.

  2. Run the commands below

    git clone https://github.com/getmaxun/maxun

    change directory to the project root

    cd maxun

    install dependencies

    npm install

    change directory to maxun-core to install dependencies

    cd maxun-core

    npm install

    start frontend and backend together

    npm run start

You can access the frontend at http://localhost:5173/ and backend at http://localhost:8080/

Environment Variables

  1. Create a file named .env in the root folder of the project
  2. Example env file can be viewed here.

Variable

Mandatory

Description

If Not Set

BACKEND_URL

Yes

URL to run backend on.

Backend won't start. If not sure, set to http://localhost:8080

VITE_BACKEND_URL

Yes

URL to run backend on.

Backend won't start. If not sure, set to http://localhost:8080

JWT_SECRET

Yes

Secret key used to sign and verify JSON Web Tokens (JWTs) for authentication.

JWT authentication will not work.

DB_NAME

Yes

Name of the Postgres database to connect to.

Database connection will fail.

DB_USER

Yes

Username for Postgres database authentication.

Database connection will fail.

DB_PASSWORD

Yes

Password for Postgres database authentication.

Database connection will fail.

DB_HOST

Yes

Host address where the Postgres database server is running.

Database connection will fail.

DB_PORT

Yes

Port number used to connect to the Postgres database server.

Database connection will fail.

ENCRYPTION_KEY

Yes

Key used for encrypting sensitive data (proxies, passwords).

Encryption functionality will not work.

MINIO_ENDPOINT

Yes

Endpoint URL for MinIO, to store Robot Run Screenshots.

Connection to MinIO storage will fail.

MINIO_PORT

Yes

Port number for MinIO service.

Connection to MinIO storage will fail.

MINIO_ACCESS_KEY

Yes

Access key for authenticating with MinIO.

MinIO authentication will fail.

GOOGLE_CLIENT_ID

No

Client ID for Google OAuth, used for Google Sheet integration authentication.

Google login will not work.

GOOGLE_CLIENT_SECRET

No

Client Secret for Google OAuth.

Google login will not work.

GOOGLE_REDIRECT_URI

No

Redirect URI for handling Google OAuth responses.

Google login will not work.

REDIS_HOST

Yes

Host address of the Redis server, used by BullMQ for scheduling robots.

Redis connection will fail.

REDIS_PORT

Yes

Port number for the Redis server.

Redis connection will fail.

MAXUN_TELEMETRY

No

Disables telemetry to stop sending anonymous usage data. Keeping it enabled helps us understand how the product is used and assess the impact of any new changes. Please keep it enabled.

Telemetry data will not be collected.

How Does It Work

Maxun lets you create custom robots which emulate user actions and extract data. A robot can perform any of the actions: Capture List, Capture Text or Capture Screenshot. Once a robot is created, it will keep extracting data for you without manual intervention

1. Robot Actions

  1. Capture List: Useful to extract structured and bulk items from the website. Example: Scrape products from Amazon etc.
  2. Capture Text: Useful to extract individual text content from the website.
  3. Capture Screenshot: Get fullpage or visible section screenshots of the website.

2. BYOP

BYOP (Bring Your Own Proxy) lets you connect external proxies to bypass anti-bot protection. Currently, the proxies are per user. Soon you'll be able to configure proxy per robot.

Features

  • Extract Data With No-Code
  • Handle Pagination & Scrolling
  • Run Robots On A Specific Schedule
  • Turn Websites to APIs
  • Turn Websites to Spreadsheets
  • Adapt To Website Layout Changes (coming soon)
  • Extract Behind Login, With Two-Factor Authentication Support (coming soon)
  • Integrations (currently Google Sheet)
  • +++ A lot of amazing things soon!

我遇到的问题

我是mac电脑,首先去docker官网下载了docker.desktop。然后我进入了docker.desktop里面执行

git clone GitHub - getmaxun/maxun: ?? Open-source no-code web data extraction platform. Turn websites to APIs and spreadsheets with no-code robots in minutes! In Beta和docker-compose up -d --build。这个时候maxun的代码是下载下来了,会报错,报错找不到env文件,所以我们需要自己去maxun代码的目录下去新建一个.env文件,里面的内容直接复制Example即可。

继续docker-compose up -d --build,依然会报错下载各种镜像网络不通,docker环境没有安全上网,然后我直接使用mac电脑的终端运行,因为电脑有安全上网就直接把所有镜像下载下来,并且直接构建了应用,可以直接使用了。

相关推荐
一只爱撸猫的程序猿10 小时前
五周、两个开源仓库、九次拦截之后:AI 时代程序员缺的到底是什么
人工智能·后端·开源
ClouGence14 小时前
DBeaver太重,Navicat付费?试试这4款好用的开源数据库管理工具!
数据库·开源
分布式存储与RustFS15 小时前
RustFS 分片上传怎么配:大文件断点续传的最小可验证流程
运维·云原生·开源·对象存储·分布式存储·s3·性能基准
孟健16 小时前
用 HyperFrames 解决前端视频生成难题:国产 Flash 模型实测 3 条 1080p 视频
前端·人工智能·开源
wuyk55516 小时前
Spider-第14讲-表单提交与CSRF
前端·爬虫·python·elasticsearch·csrf
分布式存储与RustFS18 小时前
Rivet 的 Agent 状态存进自建 RustFS:主桶配置与上线边界
运维·云原生·开源·对象存储·分布式存储·s3·性能基准
量子客18 小时前
IBM、AMD正式开源「量子纠错解码评估框架」,突破多GPU调度,量子解码吞吐量提升达7.5 倍
开源·量子计算
feasibility.20 小时前
399美元桌面机器人Doly:可爱电子宠物+树莓派开源平台,但不能像ChatGPT那样聊天
机器人·开源·嵌入式·树莓派·diy·桌宠
翼帆21 小时前
我用 AI 复刻了 CodeTank,重温代码坦克
前端·javascript·开源·ai编程·游戏开发·codetank
狂师1 天前
牛逼!谷歌亲自下场开源的AI自动化神器!
人工智能·开源·测试