Overview

Full name
docker.io/crawlabteam/crawlab-pro
Registry
docker.io
Namespace
crawlabteam
Repository
crawlab-pro
Cached tags
5
Last synced
07/06/2026, 10:25 AM

Introduction

Crawlab Pro 分布式爬虫管理平台镜像,提供任务调度、节点管理与可视化监控,适合企业级数据采集私有化部署。

Details

crawlabteam/crawlab-pro 运行 Crawlab 主控与 Web UI,Worker 节点执行 Scrapy/Colly 等爬虫任务。典型用于替代 Scrapyd 的可视化方案,与 Airflow 相比更聚焦爬虫领域。定位为 Web + 调度平台,需 MongoDB/Redis 与任务 artifact 存储;K8s 可 master/worker 分离 Deployment。

快速启动

docker run -d --name crawlab-pro \
  -p 8080:8080 \
  -e CRAWLAB_MONGO_HOST=mongo \
  docker.io/crawlabteam/crawlab-pro:latest

推荐实践

docker run -d --name crawlab-pro \
  --restart unless-stopped \
  -p 8080:8080 \
  -e CRAWLAB_MONGO_HOST=mongo.internal \
  -e CRAWLAB_REDIS_HOST=redis.internal \
  -v /data/crawlab:/root/.crawlab \
  docker.io/crawlabteam/crawlab-pro:0.6.0

核心参数说明

  • -p 8080:8080 — Crawlab Web 与 API 端口
  • -e CRAWLAB_MONGO_HOST — MongoDB 任务元数据
  • -e CRAWLAB_REDIS_HOST — 队列与缓存
  • -v .../.crawlab:... — 配置与日志
  • 固定 tag — Pro 功能与 API pin

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: crawlab-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: crawlab-master
  template:
    metadata:
      labels:
        app: crawlab-master
    spec:
      containers:
        - name: crawlab
          image: docker.io/crawlabteam/crawlab-pro:0.6.0
          ports:
            - containerPort: 8080
          envFrom:
            - secretRef:
                name: crawlab-secrets
---
apiVersion: v1
kind: Service
metadata:
  name: crawlab
spec:
  selector:
    app: crawlab-master
  ports:
    - port: 8080
      targetPort: 8080

Worker 独立 Deployment 水平扩展;遵守目标站点 robots 与合规。

Latest tags (20)

View all 5 tags →
TagArchPushed AtSizeDigestSyncWishAction
docker.io/crawlabteam/crawlab-pro
:edge
linux/amd64
2026/07/06 18:04984.8 MBsha256:3c1df646e3f3
docker.io/crawlabteam/crawlab-pro
:latest
linux/amd64
2026/07/06 18:04984.8 MBsha256:3c1df646e3f3
docker.io/crawlabteam/crawlab-pro
:stable
linux/amd64
2026/07/06 16:40984.8 MBsha256:82d4d9b86d6c
docker.io/crawlabteam/crawlab-pro
:0.7.3
linux/amd64
2026/07/06 16:40984.8 MBsha256:82d4d9b86d6c
docker.io/crawlabteam/crawlab-pro
:client-feedback-2026-07-02
linux/amd64
2026/07/02 21:33984.8 MBsha256:a293322f70d6
docker.io/crawlabteam/crawlab-pro
:0.7.2
linux/amd64
2026/07/02 13:46985.8 MBsha256:e7b86d719d77
docker.io/crawlabteam/crawlab-pro
:develop
linux/amd64
2026/07/02 13:41985.8 MBsha256:ba8428435cce
docker.io/crawlabteam/crawlab-pro
:test
linux/amd64
2026/07/02 13:31985.8 MBsha256:3d31bf7546d5
docker.io/crawlabteam/crawlab-pro
:0.7.1
linux/amd64
2026/06/10 15:45992.5 MBsha256:4f84fe1612a3
docker.io/crawlabteam/crawlab-pro
:0.7.0
linux/amd64
2026/06/08 16:55991.6 MBsha256:0cd1cf25c551