Compare commits
246 Commits
b352eea21d
...
origin-syn
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
080625567b | ||
|
|
266c0f17c1 | ||
|
|
225d13fb6e | ||
|
|
2ed1250604 | ||
|
|
ca4a2cd07a | ||
|
|
73ac9187a6 | ||
|
|
7503e3fa8b | ||
|
|
532438faba | ||
|
|
a1376b51b0 | ||
|
|
1e087c1aae | ||
|
|
2e2de02476 | ||
|
|
aeb4e1710c | ||
|
|
b4b80cb572 | ||
|
|
70e902e7ea | ||
|
|
14cebf22c3 | ||
|
|
73d25fcbcb | ||
|
|
01ecde45c5 | ||
|
|
a2ebfc0b81 | ||
|
|
1dc2cac19a | ||
|
|
14db1e0cb1 | ||
|
|
01f4dbb83a | ||
|
|
0d63d6d75a | ||
|
|
fe692fcd1d | ||
|
|
79b177b345 | ||
|
|
18b0c2d211 | ||
|
|
e4e01c1686 | ||
|
|
648e7d2f14 | ||
|
|
0d78d63437 | ||
|
|
289b1c67ce | ||
|
|
5c990e651e | ||
|
|
52095eb992 | ||
|
|
a4e4745921 | ||
|
|
a2d0bd3c1c | ||
|
|
b9275bc174 | ||
|
|
683a3934fc | ||
|
|
e5d0b2d9ab | ||
|
|
b0514c3e9a | ||
|
|
d618a5abf0 | ||
|
|
8be12ca9bc | ||
|
|
5ca46b0a57 | ||
|
|
afeeb8327c | ||
|
|
a7d8f8be6c | ||
|
|
4d2da94692 | ||
|
|
8cd1a4e2b1 | ||
|
|
b746812e23 | ||
|
|
3fbbf87c10 | ||
|
|
b0e97f21a4 | ||
|
|
a00ff1804c | ||
|
|
c3810fdb7b | ||
|
|
203937335d | ||
|
|
b16c63225f | ||
|
|
710b953120 | ||
|
|
45a45dc0fe | ||
|
|
3ca569ed1e | ||
|
|
c315bb350c | ||
|
|
111f30d150 | ||
|
|
6d398b66bc | ||
|
|
f46c231323 | ||
|
|
934919c699 | ||
|
|
12d84a22e4 | ||
|
|
9d4810eace | ||
|
|
dc457aff9e | ||
|
|
3ce66a59ee | ||
|
|
578847d596 | ||
|
|
147b324658 | ||
|
|
6416c14f20 | ||
|
|
e47c87f3a3 | ||
|
|
add6e6aeb5 | ||
|
|
1ecce13814 | ||
|
|
7968cef8c3 | ||
|
|
7c2b154c20 | ||
|
|
34148b5d8a | ||
|
|
7ba916f00c | ||
|
|
79e408333d | ||
|
|
af4f4a9ed4 | ||
|
|
74a7cd22b0 | ||
|
|
aece8123c3 | ||
|
|
e1cca219b8 | ||
|
|
70658041a5 | ||
|
|
4419c5bacd | ||
|
|
668226a99d | ||
|
|
4eec0dd5a4 | ||
|
|
b315d11c28 | ||
|
|
8e60f50616 | ||
|
|
72faf0433a | ||
|
|
f85e3be40e | ||
|
|
3616d64735 | ||
|
|
7f9116f12a | ||
|
|
01bae23df5 | ||
|
|
11f0d2c745 | ||
|
|
b073bbe97d | ||
|
|
955a6439a0 | ||
|
|
c6ecfb5b77 | ||
|
|
172917ac41 | ||
|
|
ad2bfcd2f5 | ||
|
|
a5f6898d06 | ||
|
|
83bd5839d0 | ||
|
|
83bc90ecf2 | ||
|
|
6ea3a960e7 | ||
|
|
cbfc29e953 | ||
|
|
fd2013b448 | ||
|
|
b66465aa6a | ||
|
|
a684fde472 | ||
|
|
fbd1a4dbea | ||
|
|
824fa7b76c | ||
|
|
be8a8492a9 | ||
|
|
b429721070 | ||
|
|
73d1dff21d | ||
|
|
d7d1021ec3 | ||
|
|
735234f7bc | ||
|
|
e09ce6d13b | ||
|
|
46e55fa623 | ||
|
|
a92d1af4f2 | ||
|
|
1be28f10e9 | ||
|
|
9c16c9c589 | ||
|
|
8711b205cf | ||
|
|
e316adca1f | ||
|
|
0457fbfac0 | ||
|
|
58df800685 | ||
|
|
7c198f7d6f | ||
|
|
ec4d1b2567 | ||
|
|
4ff4d8cfdc | ||
|
|
b96b136fa3 | ||
|
|
02ddfc7a76 | ||
|
|
8defef904f | ||
|
|
13c3a04a48 | ||
|
|
60a6494260 | ||
|
|
995e2ee65a | ||
|
|
7f83bceba1 | ||
|
|
1eb622d33f | ||
|
|
5d8ee8fe79 | ||
|
|
6f1b8e46f5 | ||
|
|
f9b2d2da25 | ||
|
|
26a920a1cc | ||
|
|
e695fd5184 | ||
|
|
0caf62c3d2 | ||
|
|
326f78705d | ||
|
|
afa8e44909 | ||
|
|
72fdb1841c | ||
|
|
34bc980eea | ||
|
|
2dab555ca5 | ||
|
|
d537924aa9 | ||
|
|
dc6ecc1edf | ||
|
|
f6e83352d7 | ||
|
|
46f46039ea | ||
|
|
8c4a6cd12a | ||
|
|
236b73755c | ||
|
|
b04b05e4da | ||
|
|
d691e53886 | ||
|
|
dabb278170 | ||
|
|
7b12cebb45 | ||
|
|
9d75d673f8 | ||
|
|
8d97f5318c | ||
|
|
110f407ae2 | ||
|
|
8db9f3a408 | ||
|
|
2ae720ae09 | ||
|
|
f87c37528b | ||
|
|
71b09a89de | ||
|
|
d5764f3b50 | ||
|
|
4e5ed76cb6 | ||
|
|
ecc8b6ce6c | ||
| 9ad6e037dc | |||
|
|
4a58b366b6 | ||
|
|
bbc833f2c9 | ||
|
|
325c2f7c82 | ||
|
|
728eea4b83 | ||
|
|
e53eaf738e | ||
|
|
ad2bb0a6ea | ||
|
|
7589b4e418 | ||
|
|
b400b494ad | ||
|
|
9a59dac6d0 | ||
|
|
03a85d5523 | ||
|
|
33186911f3 | ||
|
|
144d03965e | ||
|
|
e1cd176ab2 | ||
|
|
2e30821e0f | ||
|
|
7151f6b5e0 | ||
|
|
3149d80456 | ||
|
|
62315cee6c | ||
|
|
2f67b376ee | ||
|
|
489e7e8269 | ||
|
|
1b02160b20 | ||
|
|
d8d5b56c13 | ||
|
|
486e690204 | ||
|
|
380dd9f206 | ||
|
|
fba85ca365 | ||
|
|
99236c9feb | ||
|
|
0cad577d7c | ||
|
|
274d294b2e | ||
|
|
f63ae84853 | ||
|
|
4b50542779 | ||
|
|
20bd41008c | ||
|
|
8b490d3dd4 | ||
|
|
99e94a5e57 | ||
|
|
949a5b42ce | ||
|
|
1fdef3c67f | ||
|
|
a0624221de | ||
|
|
63043c56d4 | ||
|
|
6ed033fff2 | ||
|
|
785d7563f9 | ||
|
|
b6dd1d7931 | ||
|
|
bf802c1019 | ||
|
|
d3dd4898a4 | ||
|
|
1c1acbbe7d | ||
|
|
eeeed6d7b1 | ||
|
|
9cbede6e09 | ||
|
|
b8f88ca06e | ||
|
|
23e0705a99 | ||
|
|
a07b9d7fb2 | ||
|
|
efbff69a91 | ||
|
|
e962990ca0 | ||
|
|
e4a5f5acc0 | ||
|
|
5c7f1187c0 | ||
|
|
0aad7b3faf | ||
|
|
28d410f38f | ||
|
|
fe8c6b94ae | ||
|
|
e09f6d4a80 | ||
|
|
085e8f2f84 | ||
|
|
2952f3bc95 | ||
|
|
5dd7a92cf5 | ||
|
|
ea212c8931 | ||
|
|
14269d9513 | ||
|
|
8106a8ab7e | ||
|
|
bbca159684 | ||
|
|
6434ec3e00 | ||
|
|
8cfaab49fb | ||
|
|
2cd0fcff93 | ||
|
|
7ffe5279a3 | ||
|
|
12318f8770 | ||
|
|
e8bec339a7 | ||
|
|
e9684a81aa | ||
|
|
903c591181 | ||
|
|
923dfd2495 | ||
|
|
0d83d56a05 | ||
|
|
ef16abfbb9 | ||
|
|
cfbd5d6e84 | ||
|
|
497d65d41d | ||
|
|
c2e45128ee | ||
|
|
c80e1889ef | ||
|
|
7f9a6bd9cb | ||
|
|
32f4705491 | ||
|
|
bb06bc8f5a | ||
|
|
d39d310544 | ||
|
|
7efb86ed3a | ||
|
|
6791869e0c | ||
|
|
8a64a6cd9b |
4
.gitignore
vendored
4
.gitignore
vendored
@@ -19,10 +19,10 @@ backend-java/.idea/
|
||||
backend-java/*.iml
|
||||
|
||||
# ===== backend / app =====
|
||||
__pycache__/
|
||||
backend/tmp/
|
||||
backend/__pycache__
|
||||
app/__pycache__
|
||||
app/assets/
|
||||
app/new_web_source
|
||||
app/user_data/
|
||||
|
||||
@@ -43,6 +43,7 @@ MANIFEST
|
||||
.installed.cfg
|
||||
|
||||
# Build / packaging
|
||||
app.zip
|
||||
build/
|
||||
dist/
|
||||
develop-eggs/
|
||||
@@ -108,3 +109,4 @@ xlsx/
|
||||
OPS_REDIS_MYSQL_OPTIMIZATION_NOTES.md
|
||||
架构.md
|
||||
*ts.%
|
||||
.omc
|
||||
|
||||
7818
2026_05_29.log
Normal file
7818
2026_05_29.log
Normal file
File diff suppressed because one or more lines are too long
409
README.md
409
README.md
@@ -1,409 +0,0 @@
|
||||
# crawler-plugin
|
||||
|
||||
## 这个仓库是什么
|
||||
|
||||
这个仓库不是一个“单体项目”,而是一个混合工作目录。当前更像是以下三层共同组成的一套系统:
|
||||
|
||||
- `app/`:Python 桌面宿主、Flask 页面承载层、`pywebview` 本地桥接层、自动化任务入口
|
||||
- `backend-java/`:Java 业务后端,负责任务模型、文件处理、进度缓存、结果组装、落库、下载
|
||||
- `frontend-vue/`:Vue 多页面前端,负责工具页交互和任务发起
|
||||
|
||||
如果你是第一次接手这个仓库,不要先把它理解成:
|
||||
|
||||
- 纯 Java 后端项目
|
||||
- 纯 Python 自动化项目
|
||||
- 纯前后端分离 Web 项目
|
||||
|
||||
它当前的真实形态更接近:
|
||||
|
||||
`Vue 页面 -> Python 桌面桥接/Flask -> Java 任务系统 -> Redis / DB / OSS`
|
||||
|
||||
同时,Python 自动化还会再去驱动本地紫鸟客户端和浏览器。
|
||||
|
||||
## 先看哪里
|
||||
|
||||
如果你的目标是理解当前主线,请按这个顺序读:
|
||||
|
||||
1. `app/`
|
||||
2. `backend-java/`
|
||||
3. `frontend-vue/`
|
||||
|
||||
原因很简单:
|
||||
|
||||
- `app/` 决定了桌面端怎么承载页面、怎么暴露本地能力、怎么把任务推给 Python 自动化
|
||||
- `backend-java/` 决定了任务怎么创建、状态怎么缓存、结果怎么回传、文件怎么生成和下载
|
||||
- `frontend-vue/` 决定了用户是怎么触发这些链路的
|
||||
|
||||
以下目录不要默认当成当前主线:
|
||||
|
||||
- `source_code/`
|
||||
- `backend/`
|
||||
|
||||
它们和 `app/` 有明显重叠,更像历史版本、迁移残留或中间态副本。阅读时要先带着“可能不是当前生效版本”的假设。
|
||||
|
||||
## 主线架构怎么分工
|
||||
|
||||
### Python 层:桌面壳 + 本地桥接 + 自动化执行
|
||||
|
||||
`app/` 不是单纯的业务后端。它更像一个桌面宿主,负责三件事:
|
||||
|
||||
- 承载 Flask 页面和登录态
|
||||
- 通过 `pywebview` 暴露本地文件选择、保存文件、上传文件、任务入队等能力
|
||||
- 启动和调度 Python 自动化任务
|
||||
|
||||
这一层解决的是“本地能力”和“桌面壳”的问题,不是最终任务结果的权威存储。
|
||||
|
||||
### Java 层:任务系统 + 结果系统
|
||||
|
||||
`backend-java/` 是当前业务核心后端,负责:
|
||||
|
||||
- 文件上传与临时文件管理
|
||||
- 任务创建、历史查询、批量轮询
|
||||
- Redis 进度缓存
|
||||
- 分片结果接收与聚合
|
||||
- 结果文件生成
|
||||
- OSS 上传
|
||||
- 下载接口输出
|
||||
|
||||
从模块名看,Java 端已经承接了大部分工具型任务,例如:
|
||||
|
||||
- `brand`
|
||||
- `dedupe`
|
||||
- `split`
|
||||
- `convert`
|
||||
- `deletebrand`
|
||||
- `productrisk`
|
||||
- `shopmatch`
|
||||
- `pricetrack`
|
||||
|
||||
### Vue 层:多页面工具前端
|
||||
|
||||
`frontend-vue/` 不是单页应用,而是多入口页面工程。它的作用主要是:
|
||||
|
||||
- 让用户选择文件或参数
|
||||
- 调用 Java API 创建任务或查询任务
|
||||
- 调用 `pywebview` 桥接拿本地能力
|
||||
- 在任务执行过程中轮询状态和下载结果
|
||||
|
||||
它构建后的产物输出到 `new_web_source/`,再由 Python/Flask 暴露出来供桌面端使用。
|
||||
|
||||
## 如何理解目录
|
||||
|
||||
### 当前最值得关注的目录
|
||||
|
||||
#### `app/`
|
||||
|
||||
这是当前阅读优先级最高的 Python 目录,重点看这些角色:
|
||||
|
||||
- `main.py`:创建 `pywebview` 窗口,暴露本地桥接 API
|
||||
- `app.py`:Flask 应用入口
|
||||
- `blueprints/`:页面、登录、品牌、通信等 HTTP 层
|
||||
- `amazon/`:删除品牌、商品风险、匹配、跟价等自动化任务实现
|
||||
|
||||
#### `backend-java/`
|
||||
|
||||
这是任务系统核心,重点关注:
|
||||
|
||||
- controller:对外 API
|
||||
- service:任务创建、进度、结果回传、组装
|
||||
- model / mapper:任务和结果模型
|
||||
- `application.yml`:后端依赖配置
|
||||
|
||||
#### `frontend-vue/`
|
||||
|
||||
这是当前工具页前端源码,重点关注:
|
||||
|
||||
- `src/shared/bridges/pywebview.ts`:前端如何调 Python 本地桥接
|
||||
- `src/shared/api/java-modules.ts`:前端如何调 Java API
|
||||
- `src/pages/brand/components/`:各个具体工具页
|
||||
|
||||
### 不要误判成源码主线的目录
|
||||
|
||||
#### `new_web_source/`
|
||||
|
||||
更像前端构建产物目录,不是首选手改源码位置。
|
||||
|
||||
#### `source_code/`
|
||||
|
||||
和 `app/` 结构高度相似,但从当前仓库关系看,更像旧线或中间迁移副本。
|
||||
|
||||
#### `backend/`
|
||||
|
||||
另一套 Python 后端实现,和当前主线职责不完全一致,更像旧后台线。
|
||||
|
||||
## 用“patrol-delete”看懂最近新增的自动化链路
|
||||
|
||||
从最近的提交看,`patrol-delete` 是当前最新落地的一条自动化模块。它同样同时经过:
|
||||
|
||||
- 前端
|
||||
- Python 桌面桥接
|
||||
- Python 自动化执行
|
||||
- Java 任务系统
|
||||
|
||||
但它和“删除品牌”有一个关键差别:
|
||||
|
||||
- `delete-brand` 更偏“本地 Excel 文件驱动”
|
||||
- `patrol-delete` 更偏“店铺输入 + 任务模板 + 串行自动化执行”
|
||||
|
||||
如果你想快速理解当前新增自动化功能的开发方式,优先看这条线更合适。
|
||||
|
||||
下面按职责来拆。
|
||||
|
||||
### 1. 用户先在前端输入店铺,并加入备选区
|
||||
|
||||
用户打开 `patrol-delete` 页面后,不是先选本地文件,而是:
|
||||
|
||||
- 输入店铺名
|
||||
- 将店铺加入备选区
|
||||
- 勾选要处理的店铺
|
||||
- 点击“匹配店铺”
|
||||
|
||||
这一阶段前端的核心职责是收集“要处理哪个店铺”。
|
||||
|
||||
### 2. 前端先让 Java 做店铺匹配
|
||||
|
||||
前端不会直接把“店铺名”交给 Python 执行,而是先调用 Java 的匹配接口。
|
||||
|
||||
Java 在这一阶段负责:
|
||||
|
||||
1. 根据店铺名查索引
|
||||
2. 判断是否能命中店铺
|
||||
3. 返回 `shopId`、平台、公司、匹配状态等信息
|
||||
|
||||
这里要注意:
|
||||
|
||||
- 这是 `frontend -> Java`
|
||||
- 目的是先把“可执行店铺”筛出来
|
||||
- Python 还没有开始实际自动化
|
||||
|
||||
### 3. 前端要求 Java 创建“patrol-delete 任务”
|
||||
|
||||
匹配通过后,前端调用 Java 的建任务接口。
|
||||
|
||||
Java 在这一阶段做的事情也不是“立刻去删商品”,而是:
|
||||
|
||||
1. 接收店铺与模板结构
|
||||
2. 创建 `taskId`
|
||||
3. 创建结果记录
|
||||
4. 返回当前任务项的初始数据给前端
|
||||
|
||||
这一阶段可以理解成:
|
||||
|
||||
- Java 负责建模和建任务
|
||||
- Python 还没有开始自动化执行
|
||||
|
||||
### 4. 前端把“可执行项”推入 Python 本地队列
|
||||
|
||||
`patrol-delete` 不是由 Java 主动调用 Python,而是前端把任务 payload 推给 Python。
|
||||
|
||||
这一步通过 `pywebview` 暴露的 `enqueue_json()` 完成,前端会构造一个类似下面的 payload:
|
||||
|
||||
- `type: patrol-delete-run`
|
||||
- `taskId`
|
||||
- `user_id`
|
||||
- `items`
|
||||
- `template_rows`
|
||||
- `country_sections`
|
||||
- `cart_ratios`
|
||||
|
||||
然后交给 Python。
|
||||
|
||||
重要区分:
|
||||
|
||||
- 这一步不是 Java 队列
|
||||
- 这是 Python 本地进程内队列
|
||||
- 当前仓库里对应的是 `JSON_TASK_QUEUE`
|
||||
|
||||
### 5. Python 的任务监听器开始消费队列
|
||||
|
||||
Python 侧有一个任务监听器持续盯着 `JSON_TASK_QUEUE`。
|
||||
|
||||
按 `patrol-delete` 这条链路的设计,Python 拿到任务后会按店铺串行执行,再在店铺内部做页面自动化。
|
||||
|
||||
这一层的职责重点是:
|
||||
|
||||
1. 根据 `taskId` 和店铺信息定位当前任务
|
||||
2. 串行执行当前店铺,避免多个店铺同时抢本地浏览器环境
|
||||
3. 按模板结构产出各国家的删除结果与购物车比例
|
||||
4. 在执行过程中持续回传 Java
|
||||
|
||||
同时,Python 会在本地维护当前任务状态,例如:
|
||||
|
||||
- 当前店铺
|
||||
- 当前步骤
|
||||
- 已完成/失败数量
|
||||
- 是否需要继续下一个店铺
|
||||
|
||||
这些状态主要服务于 Python 本地执行期,不是前端最终查询任务状态的权威来源。
|
||||
|
||||
### 6. Python 自动化驱动紫鸟客户端和浏览器
|
||||
|
||||
`patrol-delete` 的“真正执行动作”也发生在这里。
|
||||
|
||||
Python 不会直接操作普通浏览器,而是先和紫鸟客户端通信,再接管浏览器调试口。大致过程是:
|
||||
|
||||
1. 连接或启动本机紫鸟客户端
|
||||
2. 打开指定店铺
|
||||
3. 进入需要巡店/删除的目标页面
|
||||
4. 根据模板中的国家块、状态和数量条件读取页面数据
|
||||
5. 找到符合删除条件的商品
|
||||
6. 执行删除并整理结果
|
||||
|
||||
所以这一段实际上又分两层:
|
||||
|
||||
- `Python -> 紫鸟客户端`:本地 HTTP IPC
|
||||
- `Python -> 浏览器`:通过调试口/自动化驱动执行页面操作
|
||||
|
||||
### 7. Python 一边执行,一边把结果分片回传给 Java
|
||||
|
||||
这条链路的关键点在这里:
|
||||
|
||||
- Python 不是等整个任务做完再一次性提交结果
|
||||
- 而是每处理完一部分,就立刻回传给 Java
|
||||
|
||||
`patrol-delete` 这里是按“店铺结果分片”回传,更强调:
|
||||
|
||||
- `shopName`
|
||||
- `submissionId`
|
||||
- `chunkIndex`
|
||||
- `chunkTotal`
|
||||
- `shopDone`
|
||||
- `countrySections`
|
||||
- `cartRatios`
|
||||
- `error`
|
||||
|
||||
回传方式:
|
||||
|
||||
- `Python -> Java`
|
||||
- `Content-Type: application/json`
|
||||
- 接口:`/api/patrol-delete/tasks/{taskId}/result`
|
||||
|
||||
回传的数据里会包含:
|
||||
|
||||
- 当前店铺标识
|
||||
- `chunkIndex`
|
||||
- `chunkTotal`
|
||||
- 该店铺当前片段的国家结果
|
||||
- 购物车比例
|
||||
- 当前片段是否已结束
|
||||
- 失败时的错误信息
|
||||
|
||||
因此,`patrol-delete` 这条线里更核心的是一类传输:
|
||||
|
||||
1. 执行结果回传:`application/json`
|
||||
|
||||
它不依赖“先上传本地 Excel 文件”这个前置动作。
|
||||
|
||||
### 8. Java 负责接收分片、缓存进度、判断是否可以完结
|
||||
|
||||
Java 收到 Python 的结果分片后,不会简单地“收一条写一条最终结果”,而是会先做任务系统层面的处理:
|
||||
|
||||
1. 校验 `taskId`
|
||||
2. 按 `taskId + shopName` 聚合店铺结果
|
||||
3. 过滤重复分片
|
||||
4. 将结果分片缓存起来
|
||||
5. 更新实时进度
|
||||
6. 判断某个店铺是否已 `shopDone`
|
||||
7. 判断整个任务是否达到 finalize 条件
|
||||
|
||||
这一步说明 Java 才是“任务状态和最终结果”的权威系统。
|
||||
|
||||
### 9. Java 在分片收齐后生成最终结果
|
||||
|
||||
当 Java 发现所有需要的结果分片都已收齐时,会执行最终组装:
|
||||
|
||||
1. 合并分片
|
||||
2. 重建最终结果数据
|
||||
3. 生成结果 Excel
|
||||
4. 上传 OSS
|
||||
5. 更新任务状态
|
||||
6. 写入结果记录
|
||||
|
||||
如果这一步成功,任务会变成 `SUCCESS`;否则会进入 `FAILED`。
|
||||
|
||||
### 10. 前端轮询 Java,不轮询 Python
|
||||
|
||||
前端展示任务状态时,查询对象是 Java,而不是 Python 本地队列。
|
||||
|
||||
前端主要关心的是:
|
||||
|
||||
- 任务详情
|
||||
- 批量进度
|
||||
- 下载地址
|
||||
|
||||
也就是说:
|
||||
|
||||
- Python 负责执行
|
||||
- Java 负责对前端提供任务状态与结果
|
||||
|
||||
### 11. 前端最终通过 Java 下载结果,再交给 Python 保存到本地
|
||||
|
||||
任务成功后,前端拿到 Java 下载地址,然后再通过 `pywebview` 调 Python 的保存能力,把文件落回用户本地磁盘。
|
||||
|
||||
所以最后一步仍然是混合协作:
|
||||
|
||||
- 下载来源是 Java
|
||||
- 本地保存能力来自 Python 桌面桥接
|
||||
|
||||
## 这条例子可以类比到哪些模块
|
||||
|
||||
`patrol-delete` 不是孤例,它更像当前仓库最近新增自动化模块的代表模式。
|
||||
|
||||
同类模式至少还能看到这些任务类型:
|
||||
|
||||
- `patrol-delete-run`
|
||||
- `delete-brand-run`
|
||||
- `product-risk-resolve-run`
|
||||
- `shop-match-run`
|
||||
- `price-track`
|
||||
|
||||
它们的共性通常是:
|
||||
|
||||
1. 前端先建任务
|
||||
2. Python 负责本地自动化执行
|
||||
3. Java 负责任务状态、结果缓存和最终输出
|
||||
|
||||
其中:
|
||||
|
||||
- `patrol-delete` 更适合理解“店铺驱动 + 模板驱动”的新链路
|
||||
- `delete-brand` 更适合理解“本地文件上传 + 文件解析驱动”的旧链路
|
||||
|
||||
理解这两条线之后,再看商品风险、店铺匹配、跟价等模块会容易很多。
|
||||
|
||||
## 接手时最容易踩的坑
|
||||
|
||||
### 不要把 `enqueue_json()` 当成 Java 队列
|
||||
|
||||
它是前端调 Python 的本地桥接,进入的是 Python 进程内队列,不是 Java 消息系统。
|
||||
|
||||
### 不要把 `new_web_source/` 当成首选源码目录
|
||||
|
||||
它更像构建产物输出,首选还是看 `frontend-vue/`。
|
||||
|
||||
### 不要默认 `source_code/` 和 `backend/` 仍是当前主线
|
||||
|
||||
这两个目录有参考价值,但从当前结构看,不应先于 `app/`、`backend-java/`、`frontend-vue/` 阅读。
|
||||
|
||||
### 不要把 Python 当成最终任务状态来源
|
||||
|
||||
Python 负责执行期状态;前端面向用户看到的任务状态、历史、下载结果,当前主权在 Java。
|
||||
|
||||
## 当前文档的边界
|
||||
|
||||
这份 README 基于当前仓库的静态分析整理,有几个边界需要明确:
|
||||
|
||||
- 没有运行仓库程序做启动验证
|
||||
- 没有确认最终发布时到底启用哪一个入口
|
||||
- 对 `source_code/`、`backend/` 的判断是“更像旧线/残留”,不是运行态绝对结论
|
||||
|
||||
所以这份文档的目标不是提供启动手册,而是帮助接手开发者先建立正确的系统地图。
|
||||
|
||||
## 一句话结论
|
||||
|
||||
理解这个仓库最有效的方式不是按语言分开看,而是按职责看:
|
||||
|
||||
- Python 负责桌面壳、本地桥接、自动化执行
|
||||
- Java 负责任务系统、结果系统、下载系统
|
||||
- Vue 负责工具页交互和任务发起
|
||||
|
||||
其中,“删除品牌自动化”更适合帮助你理解旧的文件驱动链路;当前最新落地的 `patrol-delete`,则更适合当作你理解新增自动化功能的第一条例子。
|
||||
15
app/.env
15
app/.env
@@ -1,8 +1,9 @@
|
||||
base_url=http://47.110.241.161:15124
|
||||
workflow_id=7608812635877900322
|
||||
mysql_host=8.136.19.173
|
||||
mysql_host=47.110.241.161
|
||||
mysql_user=aiimage
|
||||
|
||||
proxy_url=https://api.jikip.com/ip-get?num=1&minute=1&format=json&area=all&protocol=1&mode=2&key=t24g6gi44ubufd8
|
||||
proxy_url=https://api.jikip.com/ip-get?num=1&minute=3&format=json&area=all&protocol=1&mode=2&key=t24g6gi44ubufd8
|
||||
proxy_mode=2
|
||||
|
||||
zn_company=rongchuang123
|
||||
@@ -10,9 +11,13 @@ zn_username=%E8%87%AA%E5%8A%A8%E5%8C%96_Robot
|
||||
|
||||
client_name=ShuFuAI
|
||||
|
||||
# java_api_base=http://api.aishufu.top:18080/
|
||||
java_api_base=http://api.aishufu.top:18080/
|
||||
# java_api_base=http://api.aishufu.top:18080/
|
||||
|
||||
# 与 Java 后端共享的 JWT 签名密钥,必须与 backend-java 的 AIIMAGE_JWT_SECRET 完全一致
|
||||
AIIMAGE_JWT_SECRET=please-change-this-secret-please-rotate-at-least-32-bytes
|
||||
|
||||
|
||||
# java_api_base=http://47.111.163.154:18080
|
||||
# java_api_base=http://127.0.0.1:18080
|
||||
java_api_base=http://8.136.19.173:18080
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
base_url=http://8.136.19.173:15124
|
||||
base_url=http://47.111.163.154:15124
|
||||
workflow_id=7608812635877900322
|
||||
mysql_host=8.136.19.173
|
||||
mysql_host=47.111.163.154
|
||||
mysql_user=aiimage
|
||||
|
||||
proxy_url=https://api.jikip.com/ip-get?num=1&minute=1&format=json&area=all&protocol=1&mode=2&key=t24g6gi44ubufd8
|
||||
@@ -9,7 +9,7 @@ proxy_mode=2
|
||||
client_name=ShuFuAI
|
||||
|
||||
|
||||
java_api_base=http://127.0.0.1:18080
|
||||
# java_api_base=http://8.136.19.173:18080
|
||||
java_api_base=http://api.aishufu.top:18080/
|
||||
# java_api_base=http://api.aishufu.top:18080/
|
||||
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
workflow_id=7608812635877900322
|
||||
mysql_host=8.136.19.173
|
||||
mysql_host=47.111.163.154
|
||||
mysql_user=aiimage
|
||||
|
||||
proxy_url=https://api.jikip.com/ip-get?num=1&minute=1&format=json&area=all&protocol=1&mode=2&key=t24g6gi44ubufd8
|
||||
@@ -11,8 +11,13 @@ zn_username=%E8%87%AA%E5%8A%A8%E5%8C%96_Robot
|
||||
client_name=ShuFuAI
|
||||
|
||||
|
||||
java_api_base=http://47.111.163.154:18080
|
||||
# java_api_base=http://127.0.0.1:18080
|
||||
# java_api_base=http://8.136.19.173:18080
|
||||
java_api_base=http://api.aishufu.top:18080/
|
||||
# java_api_base=http://api.aishufu.top:18080/
|
||||
# java_api_base=http://api.aishufu.top:18080/
|
||||
|
||||
# 与 Java 后端共享的 JWT 签名密钥,必须与 backend-java 的 AIIMAGE_JWT_SECRET 完全一致
|
||||
AIIMAGE_JWT_SECRET=please-change-this-secret-please-rotate-at-least-32-bytes
|
||||
# JWT cookie 名称,默认 aiimage_token;改动需与 Java 端 aiimage.auth.cookie-name 保持一致
|
||||
# AIIMAGE_AUTH_COOKIE_NAME=aiimage_token
|
||||
|
||||
|
||||
|
||||
@@ -1,13 +0,0 @@
|
||||
# Project-local RTK filters — commit this file with your repo.
|
||||
# Filters here override user-global and built-in filters.
|
||||
# Docs: https://github.com/rtk-ai/rtk#custom-filters
|
||||
schema_version = 1
|
||||
|
||||
# Example: suppress build noise from a custom tool
|
||||
# [filters.my-tool]
|
||||
# description = "Compact my-tool output"
|
||||
# match_command = "^my-tool\\s+build"
|
||||
# strip_ansi = true
|
||||
# strip_lines_matching = ["^\\s*$", "^Downloading", "^Installing"]
|
||||
# max_lines = 30
|
||||
# on_empty = "my-tool: ok"
|
||||
847
app/amazon/amazon_base.py
Normal file
847
app/amazon/amazon_base.py
Normal file
@@ -0,0 +1,847 @@
|
||||
import traceback
|
||||
import winreg
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
import requests
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from datetime import datetime
|
||||
|
||||
from DrissionPage import Chromium,ChromiumOptions
|
||||
from DrissionPage.common import By
|
||||
from DrissionPage._pages.chromium_tab import ChromiumTab
|
||||
from typing import Literal,Dict,Any,TypeVar,Type
|
||||
|
||||
|
||||
from amazon.tool import get_shop_info,show_notification
|
||||
from config import runing_shop
|
||||
|
||||
# 定义类型变量,用于泛型类型注解
|
||||
T = TypeVar('T', bound='AmamzonBase')
|
||||
|
||||
|
||||
def kill_process(version: Literal["v5", "v6"]):
|
||||
"""杀紫鸟客户端进程(独立函数版本)"""
|
||||
driver = ZiniaoDriver({})
|
||||
driver.kill_process(version)
|
||||
|
||||
|
||||
class ZiniaoDriver:
|
||||
"""紫鸟浏览器自动化驱动类"""
|
||||
|
||||
def __init__(self, user_info: dict, socket_port: int = 19890):
|
||||
"""
|
||||
初始化紫鸟浏览器驱动
|
||||
|
||||
Args:
|
||||
user_info: 用户信息字典,包含 company, username, password
|
||||
socket_port: 客户端通信端口,默认 19890
|
||||
"""
|
||||
self.user_info = user_info
|
||||
self.socket_port = socket_port
|
||||
self.client_path = None
|
||||
self.browser = None
|
||||
self.tab: ChromiumTab = None
|
||||
self.store_id = None
|
||||
|
||||
self.url = None #用于记录当前链接,实现断点续传
|
||||
|
||||
|
||||
|
||||
def get_zinaio_exe(self, protocol_name: str = "superbrowser"):
|
||||
"""
|
||||
获取紫鸟安装目录
|
||||
|
||||
Args:
|
||||
protocol_name: 协议名称,默认 "superbrowser"
|
||||
|
||||
Returns:
|
||||
exe_path: 紫鸟浏览器可执行文件路径
|
||||
"""
|
||||
try:
|
||||
key_path = rf"SOFTWARE\Classes\{protocol_name}\shell\open\command"
|
||||
key = winreg.OpenKey(winreg.HKEY_CURRENT_USER, key_path)
|
||||
command, _ = winreg.QueryValueEx(key, "")
|
||||
winreg.CloseKey(key)
|
||||
if isinstance(command, str):
|
||||
sub = "ziniao.exe"
|
||||
exe_path = command[0: command.find(sub) + len(sub) + 1]
|
||||
else:
|
||||
exe_path = command[0]
|
||||
return exe_path
|
||||
except FileNotFoundError:
|
||||
try:
|
||||
key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, key_path)
|
||||
command, _ = winreg.QueryValueEx(key, "")
|
||||
winreg.CloseKey(key)
|
||||
if isinstance(command, str):
|
||||
sub = "ziniao.exe"
|
||||
exe_path = command[0: command.find(sub) + len(sub) + 1]
|
||||
else:
|
||||
exe_path = command[0]
|
||||
return exe_path
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
|
||||
def update_core(self):
|
||||
"""
|
||||
下载所有内核,打开店铺前调用,需客户端版本5.285.7以上
|
||||
因为http有超时时间,所以这个action适合循环调用,直到返回成功
|
||||
"""
|
||||
data = {
|
||||
"action": "updateCore",
|
||||
"requestId": str(uuid.uuid4()),
|
||||
}
|
||||
data.update(self.user_info)
|
||||
while True:
|
||||
url = f'http://127.0.0.1:{self.socket_port}'
|
||||
response = requests.post(url, json.dumps(data).encode('utf-8'), timeout=120)
|
||||
result = response.json()
|
||||
print(result)
|
||||
if result is None:
|
||||
print("等待客户端启动...")
|
||||
time.sleep(2)
|
||||
continue
|
||||
if result.get("statusCode") is None or result.get("statusCode") == -10003:
|
||||
print("当前版本不支持此接口,请升级客户端")
|
||||
return
|
||||
elif result.get("statusCode") == 0:
|
||||
print("更新内核完成")
|
||||
return
|
||||
else:
|
||||
print(f"等待更新内核: {json.dumps(result)}")
|
||||
time.sleep(2)
|
||||
|
||||
def kill_process(self, version: Literal["v5", "v6"]):
|
||||
"""
|
||||
杀紫鸟客户端进程
|
||||
|
||||
Args:
|
||||
version: 客户端版本
|
||||
"""
|
||||
if version == "v5":
|
||||
process_name = 'SuperBrowser.exe'
|
||||
os.system('taskkill /f /t /im ' + "starter.exe")
|
||||
else:
|
||||
process_name = 'ziniao.exe'
|
||||
os.system('taskkill /f /t /im ' + process_name)
|
||||
time.sleep(3)
|
||||
|
||||
def get_browser_list(self) -> list:
|
||||
"""
|
||||
获取浏览器列表
|
||||
|
||||
Returns:
|
||||
list: 浏览器列表
|
||||
"""
|
||||
request_id = str(uuid.uuid4())
|
||||
data = {
|
||||
"action": "getBrowserList",
|
||||
"requestId": request_id
|
||||
}
|
||||
data.update(self.user_info)
|
||||
|
||||
url = f'http://127.0.0.1:{self.socket_port}'
|
||||
response = requests.post(url, json.dumps(data).encode('utf-8'), timeout=120)
|
||||
r = response.json()
|
||||
if str(r.get("statusCode")) == "0":
|
||||
print(r)
|
||||
return r.get("browserList")
|
||||
elif str(r.get("statusCode")) == "-10003":
|
||||
print(f"【get_browser_list】登录失败 {json.dumps(r, ensure_ascii=False)}")
|
||||
# exit()
|
||||
else:
|
||||
print(f"【get_browser_list】失败 {json.dumps(r, ensure_ascii=False)} ")
|
||||
# exit()
|
||||
|
||||
def open_store(self, store_info, isWebDriverReadOnlyMode=0, isprivacy=0,
|
||||
isHeadless=0, cookieTypeSave=0, jsInfo=""):
|
||||
"""
|
||||
打开店铺
|
||||
|
||||
Args:
|
||||
store_info: 店铺信息(browserId 或 browserOauth)
|
||||
isWebDriverReadOnlyMode: 是否只读模式,默认 0
|
||||
isprivacy: 隐私模式,默认 0
|
||||
isHeadless: 无头模式,默认 0
|
||||
cookieTypeSave: cookie保存类型,默认 0
|
||||
jsInfo: 注入的JS信息,默认 ""
|
||||
|
||||
Returns:
|
||||
dict: 返回结果
|
||||
"""
|
||||
request_id = str(uuid.uuid4())
|
||||
data = {
|
||||
"action": "startBrowser",
|
||||
"isWaitPluginUpdate": 0,
|
||||
"isHeadless": isHeadless,
|
||||
"requestId": request_id,
|
||||
"isWebDriverReadOnlyMode": isWebDriverReadOnlyMode,
|
||||
"cookieTypeLoad": 0,
|
||||
"cookieTypeSave": cookieTypeSave,
|
||||
"runMode": "1",
|
||||
"isLoadUserPlugin": True,
|
||||
"pluginIdType": 1,
|
||||
"privacyMode": isprivacy
|
||||
}
|
||||
data.update(self.user_info)
|
||||
|
||||
if store_info.isdigit():
|
||||
data["browserId"] = store_info
|
||||
else:
|
||||
data["browserOauth"] = store_info
|
||||
|
||||
if len(str(jsInfo)) > 2:
|
||||
data["injectJsInfo"] = json.dumps(jsInfo)
|
||||
|
||||
url = f'http://127.0.0.1:{self.socket_port}'
|
||||
response = requests.post(url, json.dumps(data).encode('utf-8'), timeout=120)
|
||||
r = response.json()
|
||||
if str(r.get("statusCode")) == "0":
|
||||
return r
|
||||
elif str(r.get("statusCode")) == "-10003":
|
||||
raise RuntimeError(f"【open_store】登录失败 {json.dumps(r, ensure_ascii=False)}")
|
||||
# exit()
|
||||
else:
|
||||
raise RuntimeError(f"【open_store】失败 {json.dumps(r, ensure_ascii=False)} ")
|
||||
# exit()
|
||||
|
||||
def get_browser(self, port) -> Chromium:
|
||||
"""
|
||||
获取浏览器实例
|
||||
|
||||
Args:
|
||||
port: 调试端口
|
||||
|
||||
Returns:
|
||||
Chromium: DrissionPage浏览器实例
|
||||
"""
|
||||
co = ChromiumOptions()
|
||||
# 设置不加载图片、静音
|
||||
co.set_local_port(port)
|
||||
co.no_imgs(True).mute(True)
|
||||
browser = Chromium(co)
|
||||
return browser
|
||||
|
||||
def start_client(self):
|
||||
"""启动紫鸟客户端"""
|
||||
|
||||
# 检查端口是否已经启动
|
||||
try:
|
||||
url = f'http://127.0.0.1:{self.socket_port}'
|
||||
response = requests.get(url, timeout=2)
|
||||
print(f"端口 {self.socket_port} 已经启动,跳过启动客户端操作")
|
||||
return
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
# 端口未启动,继续执行启动操作
|
||||
print(f"端口 {self.socket_port} 未启动,开始启动客户端")
|
||||
|
||||
self.kill_process('v6')
|
||||
time.sleep(5)
|
||||
self.client_path = self.get_zinaio_exe("superbrowserv6").strip('"')
|
||||
print(self.client_path)
|
||||
cmd = [self.client_path, '--run_type=web_driver', '--ipc_type=http',
|
||||
'--port=' + str(self.socket_port)]
|
||||
print(" ".join(cmd))
|
||||
|
||||
# 最大重试次数
|
||||
max_retries = 3
|
||||
|
||||
for retry_count in range(max_retries):
|
||||
print(f"第 {retry_count + 1} 次尝试启动客户端...")
|
||||
|
||||
# 启动进程
|
||||
subprocess.Popen(cmd)
|
||||
|
||||
# 循环检测10秒,每0.5秒检测一次
|
||||
start_check_time = time.time()
|
||||
client_started = False
|
||||
|
||||
while time.time() - start_check_time < 10:
|
||||
try:
|
||||
response = requests.get(url, timeout=2)
|
||||
print(f"客户端启动成功!(第 {retry_count + 1} 次尝试)")
|
||||
client_started = True
|
||||
break
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
# 端口还未启动,继续等待
|
||||
time.sleep(0.5)
|
||||
|
||||
if client_started:
|
||||
time.sleep(5) # 等待客户端完全启动
|
||||
# 更新内核
|
||||
self.update_core()
|
||||
return
|
||||
else:
|
||||
print(f"第 {retry_count + 1} 次尝试启动失败,10秒内未检测到客户端启动")
|
||||
|
||||
# 超过最大重试次数,抛出异常
|
||||
raise RuntimeError(f"客户端启动失败:重试 {max_retries} 次后仍未成功启动")
|
||||
|
||||
def open_shop(self, shop_name: str):
|
||||
"""
|
||||
打开指定店铺
|
||||
|
||||
Args:
|
||||
shop_name: 店铺名称
|
||||
|
||||
Returns:
|
||||
Chromium or str: 成功返回浏览器实例,失败返回错误信息
|
||||
"""
|
||||
print("=============打开指定店铺================")
|
||||
# 启动客户端
|
||||
self.start_client()
|
||||
|
||||
# 获取店铺列表
|
||||
shop_ls = self.get_browser_list()
|
||||
self.store_id = None
|
||||
print(shop_ls)
|
||||
for shop in shop_ls:
|
||||
if shop.get("browserName") == shop_name:
|
||||
self.store_id = shop.get('browserOauth')
|
||||
break
|
||||
|
||||
if not self.store_id:
|
||||
print("店铺不存在")
|
||||
return "店铺不存在"
|
||||
|
||||
# 打开店铺
|
||||
ret_json = self.open_store(self.store_id)
|
||||
print(ret_json)
|
||||
self.store_id = ret_json.get("browserOauth")
|
||||
if self.store_id is None:
|
||||
self.store_id = ret_json.get("browserId")
|
||||
|
||||
# 获取drissionpage浏览器会话
|
||||
self.browser = self.get_browser(ret_json.get('debuggingPort'))
|
||||
|
||||
ip_check_url = ret_json.get("ipDetectionPage")
|
||||
if not ip_check_url:
|
||||
print("ip检测页地址为空,请升级紫鸟浏览器到最新版")
|
||||
print(f"=====关闭店铺:{shop_name}=====")
|
||||
self.close_store(self.store_id)
|
||||
# exit()
|
||||
raise RuntimeError("没有IP检测地址,为了店铺安全不打开店铺")
|
||||
ip_usable = self.open_ip_check(self.browser, ip_check_url)
|
||||
if ip_usable:
|
||||
print("ip检测通过,打开店铺平台主页")
|
||||
self.open_launcher_page(ret_json.get("launcherPage"), self.browser)
|
||||
else:
|
||||
print("IP检测不通过")
|
||||
raise RuntimeError("IP检测不通过,可能是因为网络环境变化导致的,为了店铺安全不打开店铺")
|
||||
return self.browser
|
||||
|
||||
def close_store(self, browser_oauth=None):
|
||||
"""
|
||||
关闭店铺
|
||||
|
||||
Args:
|
||||
browser_oauth: 店铺OAuth标识,如果不提供则使用当前打开的店铺
|
||||
|
||||
Returns:
|
||||
dict: 返回结果
|
||||
"""
|
||||
if browser_oauth is None:
|
||||
browser_oauth = self.store_id
|
||||
|
||||
request_id = str(uuid.uuid4())
|
||||
data = {
|
||||
"action": "stopBrowser",
|
||||
"requestId": request_id,
|
||||
"duplicate": 0,
|
||||
"browserOauth": browser_oauth
|
||||
}
|
||||
data.update(self.user_info)
|
||||
|
||||
url = f'http://127.0.0.1:{self.socket_port}'
|
||||
response = requests.post(url, json.dumps(data).encode('utf-8'), timeout=120)
|
||||
r = response.json()
|
||||
if str(r.get("statusCode")) == "0":
|
||||
return r
|
||||
elif str(r.get("statusCode")) == "-10003":
|
||||
raise RuntimeError(f"【close_store】登录失败 {json.dumps(r, ensure_ascii=False)}")
|
||||
# exit()
|
||||
else:
|
||||
raise RuntimeError(f"【close_store】失败: {json.dumps(r, ensure_ascii=False)} ")
|
||||
# exit()
|
||||
|
||||
def open_launcher_page(self, launcher_page: str, browser: Chromium = None):
|
||||
"""
|
||||
打开启动页面
|
||||
|
||||
Args:
|
||||
launcher_page: 要打开的页面URL
|
||||
browser: 浏览器实例,如果不提供则使用当前浏览器实例
|
||||
"""
|
||||
if browser is None:
|
||||
browser = self.browser
|
||||
|
||||
tab = browser.new_tab(url=launcher_page)
|
||||
self.tab = tab
|
||||
return tab
|
||||
|
||||
def open_ip_check(self, browser: Chromium, ip_check_url: str):
|
||||
"""
|
||||
打开ip检测页检测ip是否正常
|
||||
:param browser: drissionpage浏览器会话
|
||||
:param ip_check_url ip检测页地址
|
||||
:return 检测结果
|
||||
"""
|
||||
try:
|
||||
tab = browser.latest_tab
|
||||
tab.get(ip_check_url)
|
||||
success_button = tab.ele((By.XPATH, '//button[contains(@class, "styles_btn--success")]'),
|
||||
timeout=60) # 等待查找元素60秒
|
||||
if success_button:
|
||||
print("ip检测成功")
|
||||
return True
|
||||
else:
|
||||
print("ip检测超时")
|
||||
return False
|
||||
except Exception as e:
|
||||
print("ip检测异常:" + traceback.format_exc())
|
||||
return False
|
||||
|
||||
|
||||
class AmamzonBase(ZiniaoDriver):
|
||||
"""亚马逊操作基类,包含一些通用方法"""
|
||||
mark_name = "操作基类"
|
||||
|
||||
def log(self, message: str, level: str = "INFO"):
|
||||
"""日志输出
|
||||
Args:
|
||||
message: 日志消息
|
||||
level: 日志级别
|
||||
"""
|
||||
try:
|
||||
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
# if level == "ERROR":
|
||||
# show_notification(message, "error")
|
||||
print(f"[{timestamp}] [{self.mark_name}] [{level}] {message}")
|
||||
except Exception as e:
|
||||
print(f"输出出错,{e}")
|
||||
|
||||
|
||||
def SwitchingCountries(self, country_name: str):
|
||||
"""
|
||||
切换国家
|
||||
操作:
|
||||
1、//div[@class="dropdown-account-switcher-header-label"]/span[last()] 获取此元素文本,判断当前国家,如果与目标国家相同则不操作,否则执行下一步
|
||||
2、点击 //div[@class="dropdown-account-switcher-header-label"] 打开下拉框
|
||||
3、点击 //div[@class="dropdown-account-switcher-list-item"] 第一个展开国家列表
|
||||
4、点击 //div[@class="dropdown-account-switcher-list-item dropdown-account-switcher-list-item-indented" and @title="国家名"] 切换到目标国家
|
||||
5、等待页面加载完成,判断国家是否切换成功,成功则返回True,否则返回False
|
||||
|
||||
Args:
|
||||
country_name: 目标国家名称
|
||||
|
||||
Returns:
|
||||
bool: 切换成功返回True,失败返回False
|
||||
"""
|
||||
try:
|
||||
print("=============切换国家================")
|
||||
if self.browser is None:
|
||||
print("浏览器实例不存在,请先打开店铺")
|
||||
return False
|
||||
|
||||
# 获取当前标签页
|
||||
tab = self.tab
|
||||
tab.wait.doc_loaded(timeout=120, raise_err=False)
|
||||
|
||||
# 步骤1:获取当前国家名称
|
||||
self.log(f"正在检查当前国家...")
|
||||
current_country_ele = tab.ele('xpath://div[@class="dropdown-account-switcher-header-label"]/span[last()]',
|
||||
timeout=20)
|
||||
if current_country_ele:
|
||||
current_country = current_country_ele.text.strip()
|
||||
self.log(f"当前国家:{current_country}")
|
||||
|
||||
# 判断是否与目标国家相同
|
||||
if current_country == country_name:
|
||||
self.log(f"当前已经是目标国家 {country_name},无需切换")
|
||||
return True
|
||||
else:
|
||||
self.log("无法获取当前国家信息")
|
||||
return False
|
||||
|
||||
# 步骤2:点击打开下拉框
|
||||
self.log(f"正在打开国家切换下拉框...")
|
||||
dropdown_header = tab.ele('xpath://div[@class="dropdown-account-switcher-header-label"]', timeout=10)
|
||||
if not dropdown_header:
|
||||
self.log("找不到国家切换下拉框")
|
||||
return False
|
||||
dropdown_header.click()
|
||||
time.sleep(1) # 等待下拉框展开
|
||||
|
||||
# 步骤3:点击第一个展开国家列表
|
||||
self.log(f"正在展开国家列表...")
|
||||
first_item = tab.ele('xpath://div[@class="dropdown-account-switcher-list-item"]', timeout=10)
|
||||
if not first_item:
|
||||
self.log("找不到国家列表项")
|
||||
return False
|
||||
first_item.click()
|
||||
time.sleep(1) # 等待国家列表展开
|
||||
|
||||
# 步骤4:点击目标国家
|
||||
self.log(f"正在切换到国家:{country_name}")
|
||||
target_country_xpath = f'//div[@class="dropdown-account-switcher-list-item dropdown-account-switcher-list-item-indented" and @title="{country_name}"]'
|
||||
target_country = tab.ele(f'xpath:{target_country_xpath}', timeout=10)
|
||||
if not target_country:
|
||||
self.log(f"找不到目标国家:{country_name}")
|
||||
return False
|
||||
target_country.click()
|
||||
|
||||
# 步骤5:等待页面加载完成并验证切换结果
|
||||
self.log(f"等待页面加载...")
|
||||
# time.sleep(3) # 等待页面加载
|
||||
self.tab.wait.doc_loaded()
|
||||
|
||||
# 再次检查当前国家
|
||||
new_country_ele = tab.ele('xpath://div[@class="dropdown-account-switcher-header-label"]/span[last()]',
|
||||
timeout=10)
|
||||
if new_country_ele:
|
||||
new_country = new_country_ele.text.strip()
|
||||
if new_country == country_name:
|
||||
self.log(f"国家切换成功:{new_country}")
|
||||
return True
|
||||
else:
|
||||
self.log(f"国家切换失败,当前国家:{new_country},目标国家:{country_name}")
|
||||
return False
|
||||
else:
|
||||
self.log("无法验证切换结果")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"切换国家时发生异常:{traceback.format_exc()}")
|
||||
return False
|
||||
|
||||
def need_login(self):
|
||||
"""
|
||||
判断是否需要登录,部分国家可能需要登录后才能切换国家
|
||||
处理流程:
|
||||
|
||||
"""
|
||||
time.sleep(3) # 等待页面可能的登录元素加载,避免跳转等等
|
||||
self.tab.wait.doc_loaded(timeout=30, raise_err=False)
|
||||
need_login_ele = self.tab.eles('xpath://h1[@class="a-spacing-small"]|//span[contains(text(),"登录")]', timeout=5)
|
||||
if len(need_login_ele) > 0:
|
||||
self.log("检测到需要登录元素")
|
||||
return True
|
||||
return False
|
||||
|
||||
def login(self, password, username=""):
|
||||
try:
|
||||
self.tab.wait.doc_loaded(timeout=30, raise_err=False)
|
||||
for _ in range(4):
|
||||
switch_account = self.tab.eles(
|
||||
'xpath://div[@data-test-id="switchableAccounts"]//div[@class="a-fixed-left-grid"]', timeout=5)
|
||||
if len(switch_account) > 0:
|
||||
switch_account[0].click()
|
||||
time.sleep(1)
|
||||
self.tab.wait.doc_loaded(timeout=30, raise_err=False)
|
||||
|
||||
pwd_input = self.tab.eles('xpath://input[@type="password"]', timeout=5)
|
||||
if len(pwd_input) > 0:
|
||||
pwd_input[0].input(password, clear=True)
|
||||
submit_btn = self.tab.eles('xpath://input[@id="signInSubmit"]', timeout=5)
|
||||
if len(submit_btn) > 0:
|
||||
submit_btn[0].click()
|
||||
self.tab.wait.doc_loaded(timeout=30, raise_err=False)
|
||||
|
||||
send_code = self.tab.eles('xpath://span[@id="auth-send-code" and contains(string(.),"发送一次性密码")]',
|
||||
timeout=5)
|
||||
self.log(f"发送一次性密码:{len(send_code)}")
|
||||
if len(send_code) > 0:
|
||||
self.log("检测到 发送一次性密码")
|
||||
send_code[0].click()
|
||||
time.sleep(1)
|
||||
self.tab.wait.doc_loaded(timeout=30, raise_err=False)
|
||||
else:
|
||||
for j in range(5):
|
||||
opt_code_input = self.tab.eles('xpath://input[@name="otpCode"]', timeout=30)
|
||||
self.log(f"验证码输入框:{len(opt_code_input)}")
|
||||
if len(opt_code_input) > 0:
|
||||
opt_code_input[0].wait.displayed(timeout=10, raise_err=False)
|
||||
for _ in range(30):
|
||||
if opt_code_input[0].value is not None and opt_code_input[0].value.strip() != "":
|
||||
print("检测到验证码输入完成")
|
||||
submit_btn = self.tab.eles('xpath://input[@id="auth-signin-button"]', timeout=10)
|
||||
if len(submit_btn) > 0:
|
||||
submit_btn[0].click()
|
||||
self.tab.wait.doc_loaded(timeout=20, raise_err=False)
|
||||
# return True
|
||||
error_mes = self.tab.eles('xpath://div[@id="auth-error-message-box"]',
|
||||
timeout=10)
|
||||
if len(error_mes) > 0:
|
||||
self.log("验证码输入错误")
|
||||
self.log(f"验证码输入错误提示:{error_mes[0].text}")
|
||||
self.tab.refresh()
|
||||
else:
|
||||
self.log("登入成功")
|
||||
return True
|
||||
|
||||
time.sleep(1)
|
||||
|
||||
else:
|
||||
break
|
||||
|
||||
submit_btn = self.tab.eles('xpath://input[@id="auth-signin-button"]', timeout=10)
|
||||
if len(submit_btn) > 0:
|
||||
submit_btn[0].click()
|
||||
except Exception as e:
|
||||
self.log(f"登录过程中发生异常:{traceback.format_exc()}")
|
||||
return False
|
||||
|
||||
def SwitchPage(self):
|
||||
"""
|
||||
切换至 管理所有库存页面
|
||||
1、等待 //navigation-favorites-bar[@class="hydrated"] 出现
|
||||
"""
|
||||
navigation = self.tab.ele('xpath://navigation-favorites-bar[@class="hydrated"]')
|
||||
navigation.wait.displayed(raise_err=False)
|
||||
page_btn = navigation.sr('xpath://internal-fav-bar-links[@data-internal="navigation"]').sr(
|
||||
'xpath://a[@data-page-id="ezdpc-gui-inventory-mons"]')
|
||||
page_btn.wait.displayed(raise_err=False)
|
||||
page_btn.click(timeout=5)
|
||||
|
||||
self.tab.wait.doc_loaded()
|
||||
# 等待搜索框出现
|
||||
search_region = self.tab.ele('xpath://div[@id="searchBoxContainer"]//kat-input-group')
|
||||
search_region.wait.displayed(raise_err=False, timeout=60)
|
||||
|
||||
def search(self,filter_type="ApprovalRequired"):
|
||||
sku_ls = []
|
||||
|
||||
load_ele = self.tab.eles("xpath://div[contains(@class,'Loader-module__loader')]",timeout=5)
|
||||
if len(load_ele) > 0:
|
||||
load_ele[0].wait.deleted(timeout=3, raise_err=False)
|
||||
time.sleep(0.5)
|
||||
|
||||
drop_down = self.tab.ele('xpath://div[contains(@class,"VolusListingStatusDropDown-module__verticalContainer")]//kat-dropdown')
|
||||
drop_down.wait.displayed(raise_err=False)
|
||||
drop_down.wait.enabled(raise_err=False)
|
||||
time.sleep(0.6)
|
||||
drop_down.click()
|
||||
# //kat-option[@value="SearchSuppressed"]
|
||||
xp = f'xpath://kat-option[@value="{filter_type}"]'
|
||||
self.log(f"正在寻找筛选条件 {filter_type},xpath: {xp}")
|
||||
approval_required = self.tab.eles(xp,timeout=5)
|
||||
if len(approval_required) == 0:
|
||||
self.log(f"没有需要{filter_type}选项】没有需要{filter_type}的商品了")
|
||||
return sku_ls # "没有需要审批的商品了"
|
||||
else:
|
||||
approval_required = approval_required[0]
|
||||
approval_required.wait.displayed(raise_err=False)
|
||||
approval_required.click()
|
||||
|
||||
approval_required_text = approval_required.text
|
||||
self.log(f"已选择筛选条件: {approval_required_text}")
|
||||
|
||||
count = re.findall(r'\d+', approval_required_text)
|
||||
if count:
|
||||
count = int(count[0])
|
||||
self.log(f"待审批的商品数量: {count}")
|
||||
if count <= 0:
|
||||
self.log(f"没有需要{filter_type}的商品了")
|
||||
return sku_ls #"没有需要审批的商品了"
|
||||
for _ in range(3):
|
||||
# 等待加载完成
|
||||
load_ele = self.tab.eles("xpath://div[contains(@class,'Loader-module__loader')]")
|
||||
if len(load_ele) > 0:
|
||||
load_ele[0].wait.deleted(timeout=3, raise_err=False)
|
||||
time.sleep(0.5)
|
||||
|
||||
sku_ls = self.tab.eles("xpath://div[@data-sku]",timeout=3)
|
||||
if len(sku_ls) > 0:
|
||||
break
|
||||
approval_required.click()
|
||||
return sku_ls
|
||||
|
||||
|
||||
class TaskBase:
|
||||
task_name = "任务基类"
|
||||
|
||||
country_info = {
|
||||
"DE": "德国",
|
||||
"FR": "法国",
|
||||
"ES": "西班牙",
|
||||
"IT": "意大利",
|
||||
"UK": "英国"
|
||||
}
|
||||
|
||||
def __init__(self, user_info: dict = None):
|
||||
"""初始化审批任务处理器
|
||||
|
||||
Args:
|
||||
user_info: 用户信息字典,包含 company, username, password
|
||||
"""
|
||||
self.user_info = user_info or {}
|
||||
self.running = True
|
||||
|
||||
|
||||
def log(self, message: str, level: str = "INFO"):
|
||||
"""日志输出
|
||||
Args:
|
||||
message: 日志消息
|
||||
level: 日志级别
|
||||
"""
|
||||
try:
|
||||
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
# if level == "ERROR":
|
||||
# show_notification(message, "error")
|
||||
print(f"[{timestamp}] [{self.task_name}] [{level}] {message}")
|
||||
except Exception as e:
|
||||
print(f"输出出错,{e}")
|
||||
|
||||
def process_task(self, task_data: dict):
|
||||
pass
|
||||
|
||||
|
||||
def open_shop(self, cls: Type[T], max_retries: int, company_name: str, shop_name: str, iskill: bool = False) -> T:
|
||||
error_info = ""
|
||||
driver = None
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
self.log(f"尝试打开店铺 {shop_name} (第 {retry + 1}/{max_retries} 次)")
|
||||
|
||||
if iskill:
|
||||
self.log("重试前先杀掉浏览器进程...")
|
||||
kill_process("v6")
|
||||
kill_process("v5")
|
||||
time.sleep(2)
|
||||
|
||||
# 组装用户信息并创建驱动
|
||||
user_info = {
|
||||
**self.user_info,
|
||||
"company": company_name
|
||||
}
|
||||
driver = cls(user_info)
|
||||
browser = driver.open_shop(shop_name)
|
||||
|
||||
if browser and browser != "店铺不存在":
|
||||
self.log(f"成功打开店铺 {shop_name}")
|
||||
else:
|
||||
self.log(f"打开店铺失败: {browser}", "WARNING")
|
||||
driver = None
|
||||
continue
|
||||
|
||||
# 判断是否需要登录
|
||||
need_login = driver.need_login()
|
||||
print("【是否需要登录】:", need_login)
|
||||
if need_login:
|
||||
self.log(f"店铺 {shop_name} 需要登录,正在登录...")
|
||||
# 获取店铺凭证
|
||||
response = get_shop_info(shop_name)
|
||||
print("【获取店铺凭证返回】:", response.text)
|
||||
shop_data = response.json()
|
||||
if not shop_data:
|
||||
mes = f"获取店铺凭证失败,响应数据: {shop_data.get('message', '未知错误')}"
|
||||
self.log(mes, "ERROR")
|
||||
# show_notification(mes, "ERROR")
|
||||
continue
|
||||
|
||||
password = shop_data["data"]["password"]
|
||||
|
||||
login_success = driver.login(password)
|
||||
if login_success:
|
||||
self.log(f"店铺 {shop_name} 登录成功,正在重新打开店铺...")
|
||||
browser = driver.open_shop(shop_name)
|
||||
if browser and browser != "店铺不存在":
|
||||
self.log(f"成功打开店铺 {shop_name} 登录后")
|
||||
break
|
||||
else:
|
||||
self.log(f"登录后打开店铺失败: {browser}", "WARNING")
|
||||
driver = None
|
||||
else:
|
||||
self.log(f"店铺 {shop_name} 登录失败", "WARNING")
|
||||
driver = None
|
||||
else:
|
||||
break
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"打开店铺异常: {traceback.format_exc()}", "INFO")
|
||||
driver = None
|
||||
error_info = str(e)
|
||||
time.sleep(10)
|
||||
|
||||
# 如果还有重试机会,等待后继续
|
||||
if retry < max_retries - 1:
|
||||
time.sleep(3)
|
||||
|
||||
# 检查是否成功打开
|
||||
if not driver or not browser or browser == "店铺不存在":
|
||||
error_msg = f"店铺 {shop_name} 打开失败,已重试 {max_retries} 次,跳过该店铺,{error_info}"
|
||||
self.log(error_msg, "ERROR")
|
||||
# 从执行列表中移除
|
||||
if shop_name in runing_shop:
|
||||
del runing_shop[shop_name]
|
||||
return driver
|
||||
return driver
|
||||
|
||||
|
||||
def process_shop(self, shop_data: Dict[str, Any], task_id: int):
|
||||
pass
|
||||
|
||||
def action_init(self, driver, country_name, shop_name, risk_listing_filter=None):
|
||||
"""切换到指定国际 -> 页面 -》 筛选好"""
|
||||
max_retries = 5
|
||||
switch_success = False
|
||||
switch_success_pg = False
|
||||
search_success = False
|
||||
sku_ls = []
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
self.log(f"尝试切换到国家 {country_name} (第 {retry + 1}/{max_retries} 次)")
|
||||
if retry > 2:
|
||||
# 刷新不行就重新打开店铺
|
||||
self.log("重试前重新打开店铺...")
|
||||
try:
|
||||
driver.close_store()
|
||||
time.sleep(3)
|
||||
driver.open_shop(shop_name)
|
||||
except Exception as e:
|
||||
self.log(f"关闭重新打开店铺: {str(e)}", "WARNING")
|
||||
# 如果不是第一次尝试,先刷新页面
|
||||
if retry > 0:
|
||||
self.log("重试前刷新页面...")
|
||||
try:
|
||||
driver.tab.refresh()
|
||||
time.sleep(3)
|
||||
except Exception as e:
|
||||
self.log(f"刷新页面失败: {str(e)}", "WARNING")
|
||||
|
||||
switch_success = driver.SwitchingCountries(country_name)
|
||||
if switch_success:
|
||||
self.log(f"成功切换到国家 {country_name}")
|
||||
|
||||
# 切换到库存管理页面
|
||||
driver.SwitchPage()
|
||||
self.log(f"已切换到库存管理页面")
|
||||
time.sleep(3)
|
||||
switch_success_pg = True
|
||||
|
||||
if risk_listing_filter is None:
|
||||
self.log(f"risk_listing_filter 为空,不筛选, {risk_listing_filter}")
|
||||
search_success = True
|
||||
break
|
||||
|
||||
sku_ls = driver.search(filter_type=risk_listing_filter)
|
||||
search_success = True
|
||||
self.log(f"国家 {country_name} 搜索出 {len(sku_ls)} 商品,开始处理...")
|
||||
break
|
||||
else:
|
||||
self.log(f"切换到国家 {country_name} 失败", "WARNING")
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"切换国家 {country_name} 异常: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
|
||||
# 如果还有重试机会,等待后继续
|
||||
if retry < max_retries - 1:
|
||||
time.sleep(2)
|
||||
|
||||
return switch_success, switch_success_pg, search_success, sku_ls
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,42 +1,17 @@
|
||||
import json
|
||||
import sys
|
||||
import os
|
||||
import io
|
||||
# sys.stdout.reconfigure(encoding='utf-8')
|
||||
|
||||
import time
|
||||
import re
|
||||
import traceback
|
||||
from datetime import datetime
|
||||
from DrissionPage import Chromium, ChromiumOptions
|
||||
import requests
|
||||
|
||||
from amazon.del_brand import AmamzonBase, kill_process
|
||||
from amazon.tool import show_notification, get_shop_info, remove_special_characters, split_currency_values
|
||||
from amazon.amazon_base import AmamzonBase, kill_process,TaskBase
|
||||
from amazon.tool import show_notification
|
||||
|
||||
from config import runing_task, runing_shop, base_dir, DELETE_BRAND_API_BASE
|
||||
from config import runing_task, runing_shop, DELETE_BRAND_API_BASE
|
||||
|
||||
|
||||
class AmzonePStatus(AmamzonBase):
|
||||
mark_name = "状态查询"
|
||||
|
||||
def SwitchPage(self):
|
||||
"""
|
||||
切换至 管理所有库存页面
|
||||
1、等待 //navigation-favorites-bar[@class="hydrated"] 出现
|
||||
"""
|
||||
navigation = self.tab.ele('xpath://navigation-favorites-bar[@class="hydrated"]')
|
||||
navigation.wait.displayed(raise_err=False)
|
||||
page_btn = navigation.sr('xpath://internal-fav-bar-links[@data-internal="navigation"]').sr(
|
||||
'xpath://a[@data-page-id="ezdpc-gui-inventory-mons"]')
|
||||
page_btn.wait.displayed(raise_err=False)
|
||||
page_btn.click(timeout=5)
|
||||
|
||||
self.tab.wait.doc_loaded()
|
||||
# 等待搜索框出现
|
||||
search_region = self.tab.ele('xpath://div[@id="searchBoxContainer"]//kat-input-group')
|
||||
search_region.wait.displayed(raise_err=False, timeout=60)
|
||||
|
||||
def search_asin(self, asin):
|
||||
search_region = self.tab.ele('xpath://div[@id="searchBoxContainer"]//kat-input-group')
|
||||
search_region.wait.displayed(raise_err=False)
|
||||
@@ -81,14 +56,8 @@ class AmzonePStatus(AmamzonBase):
|
||||
self.browser.close_tabs(close_tab)
|
||||
|
||||
def run_page_action(self, appoint_asin: str = None):
|
||||
"""
|
||||
|
||||
miniprice_info :
|
||||
{
|
||||
"B0DSJGJBWV" : "19.81" # asin : 最低价
|
||||
}
|
||||
"""
|
||||
print(f"【{self.mark_name}】,开始执行")
|
||||
self.log(f"==================={self.mark_name}=======================")
|
||||
self.log("开始执行...")
|
||||
num = 0
|
||||
retry_num = 0
|
||||
already_asin = set()
|
||||
@@ -174,29 +143,8 @@ class AmzonePStatus(AmamzonBase):
|
||||
self.tab.wait.doc_loaded(raise_err=False, timeout=120)
|
||||
|
||||
|
||||
class StatusTask:
|
||||
country_info = {
|
||||
"DE": "德国",
|
||||
"FR": "法国",
|
||||
"ES": "西班牙",
|
||||
"IT": "意大利",
|
||||
"UK": "英国"
|
||||
}
|
||||
|
||||
def __init__(self, user_info: dict = None):
|
||||
"""初始化审批任务处理器
|
||||
|
||||
Args:
|
||||
user_info: 用户信息字典,包含 company, username, password
|
||||
"""
|
||||
self.user_info = user_info or {}
|
||||
self.running = True
|
||||
|
||||
def log(self, message: str, level: str = "INFO"):
|
||||
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
if level == "ERROR":
|
||||
show_notification(message, "error")
|
||||
print(f"[{timestamp}] [PriceTask] [{level}] {message}")
|
||||
class StatusTask(TaskBase):
|
||||
task_name = "状态查询-TASK"
|
||||
|
||||
def process_task(self, task_data: dict):
|
||||
"""处理审批任务主入口
|
||||
@@ -218,21 +166,12 @@ class StatusTask:
|
||||
# 用于测试
|
||||
limit = data.get("limit", None)
|
||||
|
||||
if not task_id:
|
||||
self.log("任务ID为空,跳过", "WARNING")
|
||||
return
|
||||
|
||||
# if not items:
|
||||
# self.log("店铺列表为空,跳过", "WARNING")
|
||||
# return
|
||||
|
||||
if not queryAsins:
|
||||
self.log("queryAsins 列表为空,跳过", "WARNING")
|
||||
if not task_id or not queryAsins:
|
||||
self.log("任务ID / queryAsins列表 为空,跳过", "WARNING")
|
||||
return
|
||||
|
||||
self.log(f"开始处理审批任务 {task_id},共 1 个店铺,{len(queryAsins)} 个国家")
|
||||
|
||||
from config import runing_task
|
||||
runing_task[task_id] = {
|
||||
"status": "running",
|
||||
"start_time": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
|
||||
@@ -265,7 +204,6 @@ class StatusTask:
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["processed_shops"] += 1
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"处理店铺 {shop_name} 失败: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
|
||||
@@ -279,96 +217,12 @@ class StatusTask:
|
||||
self.log(f"任务 {task_id} 处理完成!")
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"任务处理失败: {traceback.format_exc()}", "ERROR")
|
||||
if task_id:
|
||||
from config import runing_task
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["status"] = "failed"
|
||||
runing_task[task_id]["error"] = str(e)
|
||||
|
||||
def open_shop(self, max_retries, company_name, shop_name, iskill=False):
|
||||
error_info = ""
|
||||
driver = None
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
self.log(f"尝试打开店铺 {shop_name} (第 {retry + 1}/{max_retries} 次)")
|
||||
|
||||
if iskill:
|
||||
self.log("重试前先杀掉浏览器进程...")
|
||||
kill_process("v6")
|
||||
kill_process("v5")
|
||||
time.sleep(2)
|
||||
|
||||
# 组装用户信息并创建驱动
|
||||
user_info = {
|
||||
**self.user_info,
|
||||
"company": company_name
|
||||
}
|
||||
driver = AmzonePStatus(user_info)
|
||||
browser = driver.open_shop(shop_name)
|
||||
|
||||
if browser and browser != "店铺不存在":
|
||||
self.log(f"成功打开店铺 {shop_name}")
|
||||
else:
|
||||
self.log(f"打开店铺失败: {browser}", "WARNING")
|
||||
driver = None
|
||||
continue
|
||||
|
||||
# 判断是否需要登录
|
||||
need_login = driver.need_login()
|
||||
print("【是否需要登录】:", need_login)
|
||||
if need_login:
|
||||
self.log(f"店铺 {shop_name} 需要登录,正在登录...")
|
||||
# 获取店铺凭证
|
||||
response = get_shop_info(shop_name)
|
||||
print("【获取店铺凭证返回】:", response.text)
|
||||
shop_data = response.json()
|
||||
if not shop_data:
|
||||
mes = f"获取店铺凭证失败,响应数据: {shop_data.get('message', '未知错误')}"
|
||||
self.log(mes, "ERROR")
|
||||
show_notification(mes, "ERROR")
|
||||
continue
|
||||
|
||||
password = shop_data["data"]["password"]
|
||||
|
||||
login_success = driver.login(password)
|
||||
if login_success:
|
||||
self.log(f"店铺 {shop_name} 登录成功,正在重新打开店铺...")
|
||||
browser = driver.open_shop(shop_name)
|
||||
if browser and browser != "店铺不存在":
|
||||
self.log(f"成功打开店铺 {shop_name} 登录后")
|
||||
break
|
||||
else:
|
||||
self.log(f"登录后打开店铺失败: {browser}", "WARNING")
|
||||
driver = None
|
||||
else:
|
||||
self.log(f"店铺 {shop_name} 登录失败", "WARNING")
|
||||
driver = None
|
||||
else:
|
||||
break
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"打开店铺异常: {traceback.format_exc()}", "INFO")
|
||||
driver = None
|
||||
error_info = str(e)
|
||||
time.sleep(10)
|
||||
|
||||
# 如果还有重试机会,等待后继续
|
||||
if retry < max_retries - 1:
|
||||
time.sleep(3)
|
||||
|
||||
# 检查是否成功打开
|
||||
if not driver or not browser or browser == "店铺不存在":
|
||||
error_msg = f"店铺 {shop_name} 打开失败,已重试 {max_retries} 次,跳过该店铺,{error_info}"
|
||||
self.log(error_msg, "ERROR")
|
||||
# 从执行列表中移除
|
||||
if shop_name in runing_shop:
|
||||
del runing_shop[shop_name]
|
||||
return driver
|
||||
return driver
|
||||
|
||||
def process_shop(self, shop_item: dict, country_codes: list, task_id: int,
|
||||
user_id=None, stage_index=None, final_stage: bool = True, limit: str = None):
|
||||
"""处理单个店铺
|
||||
@@ -382,7 +236,6 @@ class StatusTask:
|
||||
shop_name = shop_item.get("shopName", "未知店铺")
|
||||
company_name = shop_item.get("companyName", "")
|
||||
|
||||
|
||||
if not company_name:
|
||||
self.log(f"店铺 {shop_name} 的公司名称为空,跳过", "WARNING")
|
||||
return
|
||||
@@ -403,7 +256,7 @@ class StatusTask:
|
||||
country_code = value.get("country")
|
||||
all_asin = value.get("asins")
|
||||
# 打开店铺
|
||||
driver = self.open_shop(max_retries=max_retries, company_name=company_name,
|
||||
driver = self.open_shop(cls=AmzonePStatus,max_retries=max_retries, company_name=company_name,
|
||||
shop_name=shop_name, iskill=iskill)
|
||||
if driver is None:
|
||||
self.log(f"任务 {task_id} 启动店铺失败,结束任务", "ERROR")
|
||||
@@ -419,7 +272,6 @@ class StatusTask:
|
||||
try:
|
||||
self.process_country(driver, country_code, task_id, shop_name,all_asin=all_asin )
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"处理国家 {country_code} 失败: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
if "与页面的连接已断开" in str(e):
|
||||
@@ -450,8 +302,7 @@ class StatusTask:
|
||||
self.log(f"店铺 {shop_name} 已从执行列表中移除")
|
||||
|
||||
def process_country(self, driver: AmzonePStatus, country_code: str, task_id: int, shop_name: str,
|
||||
all_asin:list,
|
||||
limit: str = None):
|
||||
all_asin:list,limit: str = None):
|
||||
"""处理单个国家的审批任务
|
||||
|
||||
Args:
|
||||
@@ -505,7 +356,6 @@ class StatusTask:
|
||||
self.log(f"切换到国家 {country_name} 失败", "WARNING")
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"切换国家 {country_name} 异常: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
|
||||
@@ -527,7 +377,6 @@ class StatusTask:
|
||||
driver.SwitchPage()
|
||||
self.log(f"已切换到库存管理页面")
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"切换页面失败: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
|
||||
@@ -537,8 +386,6 @@ class StatusTask:
|
||||
self.log(f"切换页面失败重试退出", "ERROR")
|
||||
return
|
||||
|
||||
|
||||
|
||||
# 处理所有需要审批的商品(通过yield获取结果)
|
||||
|
||||
# 指定 asin
|
||||
@@ -679,6 +526,7 @@ class StatusTask:
|
||||
raise RuntimeError("已达到最大重试次数,结果回传最终失败")
|
||||
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
# 使用示例
|
||||
user_info = {
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
@@ -20,7 +21,7 @@ except ImportError:
|
||||
STATUS_OK = "0"
|
||||
STATUS_LOGIN_FAILED = "-10003"
|
||||
|
||||
DEFAULT_SOCKET_PORT = 19890
|
||||
DEFAULT_SOCKET_PORT = 20000
|
||||
CLIENT_API_TIMEOUT = 120
|
||||
PORT_CHECK_TIMEOUT = 2
|
||||
UPDATE_CORE_RETRY_DELAY = 2
|
||||
@@ -30,6 +31,7 @@ CLIENT_READY_TIMEOUT = 10
|
||||
CLIENT_READY_INTERVAL = 0.5
|
||||
CLIENT_POST_START_DELAY = 5
|
||||
PROCESS_KILL_DELAY = 3
|
||||
ZINIAO_WEBDRIVER_LOG_TAIL_LINES = 80
|
||||
|
||||
DOC_LOAD_TIMEOUT = 30
|
||||
COUNTRY_INITIAL_LOAD_TIMEOUT = 120
|
||||
@@ -152,6 +154,49 @@ class ZiniaoDriver:
|
||||
time.sleep(CLIENT_READY_INTERVAL)
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def _redact_ziniao_log_line(line: str) -> str:
|
||||
return re.sub(r'("password"\s*:\s*)"[^"]*"', r'\1"***"', line)
|
||||
|
||||
@staticmethod
|
||||
def _ziniao_webdriver_log_path() -> Optional[str]:
|
||||
appdata = os.getenv("APPDATA")
|
||||
if not appdata:
|
||||
return None
|
||||
filename = f"webdriver.{time.strftime('%Y%m%d')}.log"
|
||||
return os.path.join(
|
||||
appdata,
|
||||
"ziniaobrowser",
|
||||
"instances",
|
||||
"userdata1",
|
||||
"logs",
|
||||
"client",
|
||||
filename,
|
||||
)
|
||||
|
||||
def _log_ziniao_webdriver_tail(self) -> None:
|
||||
log_path = self._ziniao_webdriver_log_path()
|
||||
if not log_path:
|
||||
logger.warning("无法读取紫鸟 webdriver 日志: APPDATA 环境变量为空")
|
||||
return
|
||||
if not os.path.exists(log_path):
|
||||
logger.warning("紫鸟 webdriver 日志不存在:{}", log_path)
|
||||
return
|
||||
|
||||
try:
|
||||
with open(log_path, "r", encoding="utf-8", errors="replace") as log_file:
|
||||
lines = log_file.readlines()
|
||||
except OSError as exc:
|
||||
logger.warning("读取紫鸟 webdriver 日志失败:{} | {}", log_path, exc)
|
||||
return
|
||||
|
||||
tail_lines = lines[-ZINIAO_WEBDRIVER_LOG_TAIL_LINES:]
|
||||
tail_text = "".join(self._redact_ziniao_log_line(line) for line in tail_lines).strip()
|
||||
if tail_text:
|
||||
logger.error("紫鸟 webdriver 日志尾部({}):\n{}", log_path, tail_text)
|
||||
else:
|
||||
logger.warning("紫鸟 webdriver 日志为空:{}", log_path)
|
||||
|
||||
def get_zinaio_exe(self, protocol_name: str = "superbrowser"):
|
||||
"""从 Windows 注册表读取紫鸟客户端可执行文件路径.
|
||||
|
||||
@@ -402,7 +447,7 @@ class ZiniaoDriver:
|
||||
|
||||
for retry_count in range(CLIENT_START_RETRIES):
|
||||
logger.info("第 {}/{} 次尝试启动紫鸟客户端", retry_count + 1, CLIENT_START_RETRIES)
|
||||
subprocess.Popen(cmd)
|
||||
process = subprocess.Popen(cmd)
|
||||
|
||||
if self._wait_until_client_ready(CLIENT_READY_TIMEOUT):
|
||||
logger.info("紫鸟客户端启动成功,第 {} 次尝试", retry_count + 1)
|
||||
@@ -411,6 +456,9 @@ class ZiniaoDriver:
|
||||
return
|
||||
|
||||
logger.warning("第 {} 次尝试启动失败,10秒内未检测到客户端启动", retry_count + 1)
|
||||
if process.poll() is not None:
|
||||
logger.warning("紫鸟客户端进程已退出,returncode={}", process.returncode)
|
||||
self._log_ziniao_webdriver_tail()
|
||||
|
||||
logger.error("紫鸟客户端启动失败,已重试 {} 次", CLIENT_START_RETRIES)
|
||||
raise RuntimeError(f"客户端启动失败:重试 {CLIENT_START_RETRIES} 次后仍未成功启动")
|
||||
|
||||
206
app/amazon/chrome_base.py
Normal file
206
app/amazon/chrome_base.py
Normal file
@@ -0,0 +1,206 @@
|
||||
import json
|
||||
import sys
|
||||
import os
|
||||
import io
|
||||
|
||||
import time
|
||||
import traceback
|
||||
from datetime import datetime
|
||||
from DrissionPage import Chromium, ChromiumOptions
|
||||
from collections import defaultdict
|
||||
|
||||
from config import base_dir,debug
|
||||
from amazon.tool import get_shop_info,show_notification
|
||||
|
||||
|
||||
|
||||
class ChromeAmzoneBase:
|
||||
mark_name = "谷歌浏览器基类"
|
||||
|
||||
country_info = {
|
||||
"英国": {
|
||||
"url": "https://www.amazon.co.uk/dp/B0CJ8SNXXV",
|
||||
"zip_code": "SW1A 1AA",
|
||||
"mark": "SW1A 1"
|
||||
},
|
||||
"德国": {
|
||||
"url": "https://www.amazon.de/dp/B0CC8CW9G2?th=1",
|
||||
"zip_code": "10115"
|
||||
},
|
||||
"法国": {
|
||||
"url": "https://www.amazon.fr/dp/B0FRG1MJ8H?th=1",
|
||||
"zip_code": "75001"
|
||||
},
|
||||
"西班牙": {
|
||||
"url": "https://www.amazon.es/dp/B08ZXVNYNN",
|
||||
"zip_code": "28001"
|
||||
},
|
||||
"意大利": {
|
||||
"url": "https://www.amazon.it/dp/B0D1P17T2Q",
|
||||
"zip_code": "20121"
|
||||
}
|
||||
}
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
杀死当前谷歌浏览器进程,并使用 drissionpage 启动谷歌浏览器,使用系统安装的浏览器默认用户文件夹
|
||||
"""
|
||||
print("正在关闭现有的chromium浏览器进程...")
|
||||
if not debug:
|
||||
os.system('taskkill /f /t /im chrome.exe')
|
||||
time.sleep(2)
|
||||
|
||||
print("正在启动chromium浏览器...")
|
||||
# 配置浏览器选项
|
||||
co = ChromiumOptions()
|
||||
co.no_imgs(True)
|
||||
user_data_path = os.path.join(base_dir, "user_data", "chrome_data")
|
||||
if not os.path.exists(user_data_path):
|
||||
os.makedirs(user_data_path, exist_ok=True)
|
||||
co.set_user_data_path(user_data_path)
|
||||
co.set_local_port(port=19897)
|
||||
self.browser = Chromium(co)
|
||||
self.tab = self.browser.latest_tab
|
||||
print("Chrome浏览器启动成功")
|
||||
|
||||
def log(self, message: str, level: str = "INFO"):
|
||||
"""日志输出
|
||||
Args:
|
||||
message: 日志消息
|
||||
level: 日志级别
|
||||
"""
|
||||
try:
|
||||
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
# if level == "ERROR":
|
||||
# show_notification(message, "error")
|
||||
print(f"[{timestamp}] [{self.mark_name}] [{level}] {message}")
|
||||
except Exception as e:
|
||||
print(f"输出出错,{e}")
|
||||
|
||||
def close_init_popup(self):
|
||||
"""
|
||||
关闭所有的初始化弹窗
|
||||
"""
|
||||
self.tab.wait.doc_loaded(timeout=60, raise_err=True)
|
||||
continue_shipping = self.tab.eles('xpath://button[@alt="Continue shopping"]',timeout=3)
|
||||
if len(continue_shipping) > 0:
|
||||
continue_shipping[0].click()
|
||||
time.sleep(0.5)
|
||||
self.tab.wait.doc_loaded(timeout=60, raise_err=True)
|
||||
footbar = self.tab.eles('xpath://footer[@class="el-dialog__footer"]', timeout=5)
|
||||
if len(footbar) > 0:
|
||||
do_not_remind = footbar[0].eles('xpath:.//input[@class="el-checkbox__original"]')
|
||||
if len(do_not_remind) > 0:
|
||||
do_not_remind[0].check()
|
||||
resume_immediately = footbar[0].eles('xpath:.//button')
|
||||
if len(resume_immediately) > 0:
|
||||
resume_immediately[0].click()
|
||||
|
||||
self.tab.wait.doc_loaded(timeout=60, raise_err=True)
|
||||
accept_btn = self.tab.eles('xpath://input[@id="sp-cc-accept"]', timeout=5)
|
||||
if len(accept_btn) > 0:
|
||||
accept_btn[0].click()
|
||||
|
||||
def _set_zip_code(self, zip_code, mark=None):
|
||||
"""
|
||||
设置邮编
|
||||
|
||||
Args:
|
||||
zip_code: 目标邮编
|
||||
"""
|
||||
try:
|
||||
# 检查当前邮编
|
||||
zip_display = self.tab.ele('xpath://div[@id="glow-ingress-block"]', timeout=10)
|
||||
|
||||
if zip_display:
|
||||
current_text = zip_display.text
|
||||
# print(f"当前地址信息: {current_text}")
|
||||
# 先检查标识
|
||||
if mark is not None and mark in current_text:
|
||||
self.log(f"邮编检测到标识: {mark},无需修改")
|
||||
return True
|
||||
# 检查是否已经包含目标邮编
|
||||
if zip_code in current_text:
|
||||
self.log(f"邮编已经设置为: {zip_code},无需修改")
|
||||
return True
|
||||
|
||||
# 需要设置邮编
|
||||
self.log(f"正在设置邮编为: {zip_code}")
|
||||
|
||||
# 点击地址选择按钮
|
||||
location_link = self.tab.ele('xpath://a[@id="nav-global-location-popover-link"]', timeout=10)
|
||||
if not location_link:
|
||||
self.log("找不到地址设置按钮")
|
||||
return False
|
||||
|
||||
location_link.click()
|
||||
time.sleep(1)
|
||||
|
||||
# 等待邮编输入框出现
|
||||
zip_input = self.tab.ele('xpath://input[@id="GLUXZipUpdateInput"]', timeout=10)
|
||||
if not zip_input:
|
||||
print("找不到邮编输入框")
|
||||
return False
|
||||
|
||||
# 输入邮编
|
||||
zip_input.input(zip_code, clear=True)
|
||||
time.sleep(0.5)
|
||||
|
||||
# 点击提交按钮
|
||||
submit_btn = self.tab.ele('xpath://input[@aria-labelledby="GLUXZipUpdate-announce"]', timeout=10)
|
||||
if not submit_btn:
|
||||
print("找不到提交按钮")
|
||||
return False
|
||||
|
||||
submit_btn.click()
|
||||
|
||||
continue_btn = self.tab.eles('xpath://div[@class="a-popover-footer"]//input[@id="GLUXConfirmClose"]',
|
||||
timeout=10)
|
||||
if len(continue_btn) > 0:
|
||||
continue_btn[0].click()
|
||||
|
||||
# 等待提交按钮消失(表示请求已发送)
|
||||
print("等待邮编更新...")
|
||||
time.sleep(2)
|
||||
|
||||
# 等待页面加载完成
|
||||
self.tab.wait.doc_loaded(timeout=30, raise_err=False)
|
||||
time.sleep(2)
|
||||
|
||||
# 验证邮编是否设置成功
|
||||
zip_display_after = self.tab.ele('xpath://div[@id="glow-ingress-block"]', timeout=10)
|
||||
if zip_display_after:
|
||||
updated_text = zip_display_after.text
|
||||
# print(f"更新后的地址信息: {updated_text}")
|
||||
|
||||
if zip_code in updated_text:
|
||||
print(f"邮编设置成功: {zip_code}")
|
||||
return True
|
||||
else:
|
||||
# print(f"邮编设置可能失败,当前显示: {updated_text}")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"设置邮编时出错: {traceback.format_exc()}")
|
||||
return False
|
||||
|
||||
def close(self):
|
||||
"""关闭浏览器"""
|
||||
try:
|
||||
if self.browser:
|
||||
self.browser.quit()
|
||||
print("浏览器已关闭")
|
||||
except Exception as e:
|
||||
print(f"关闭浏览器时出错: {str(e)}")
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,88 +1,23 @@
|
||||
import json
|
||||
import sys
|
||||
import os
|
||||
import io
|
||||
# sys.stdout.reconfigure(encoding='utf-8')
|
||||
|
||||
import time
|
||||
import re
|
||||
import traceback
|
||||
from datetime import datetime
|
||||
from DrissionPage import Chromium, ChromiumOptions
|
||||
from collections import defaultdict
|
||||
import requests
|
||||
|
||||
from amazon.del_brand import AmamzonBase, kill_process
|
||||
from amazon.tool import show_notification,get_shop_info,remove_special_characters,split_currency_values
|
||||
|
||||
from config import runing_task, runing_shop,base_dir,DELETE_BRAND_API_BASE
|
||||
from amazon.amazon_base import TaskBase
|
||||
from amazon.chrome_base import ChromeAmzoneBase
|
||||
|
||||
|
||||
class ChromeAmzone:
|
||||
from config import runing_task,runing_shop,base_dir,DELETE_BRAND_API_BASE
|
||||
|
||||
|
||||
class ChromeAmzone(ChromeAmzoneBase):
|
||||
mark_name = "亚马逊详情采集"
|
||||
country_info = {
|
||||
"英国": {
|
||||
"url": "https://www.amazon.co.uk/dp/B0CJ8SNXXV",
|
||||
"zip_code": "SW1A 1AA",
|
||||
"mark": "SW1A 1"
|
||||
},
|
||||
"德国": {
|
||||
"url": "https://www.amazon.de/dp/B0CC8CW9G2?th=1",
|
||||
"zip_code": "10115"
|
||||
},
|
||||
"法国": {
|
||||
"url": "https://www.amazon.fr/dp/B0FRG1MJ8H?th=1",
|
||||
"zip_code": "75001"
|
||||
},
|
||||
"西班牙": {
|
||||
"url": "https://www.amazon.es/dp/B08ZXVNYNN",
|
||||
"zip_code": "28001"
|
||||
},
|
||||
"意大利": {
|
||||
"url": "https://www.amazon.it/dp/B0D1P17T2Q",
|
||||
"zip_code": "20121"
|
||||
}
|
||||
}
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
杀死当前谷歌浏览器进程,并使用 drissionpage 启动谷歌浏览器,使用系统安装的浏览器默认用户文件夹
|
||||
"""
|
||||
# 杀死现有的Chrome进程
|
||||
print("正在关闭现有的chromium浏览器进程...")
|
||||
os.system('taskkill /f /t /im chrome.exe')
|
||||
time.sleep(2)
|
||||
|
||||
print("正在启动chromium浏览器...")
|
||||
# 配置浏览器选项
|
||||
co = ChromiumOptions()
|
||||
user_data_path = os.path.join(base_dir, "user_data", "chrome_data")
|
||||
if not os.path.exists(user_data_path):
|
||||
os.makedirs(user_data_path, exist_ok=True)
|
||||
co.set_user_data_path(user_data_path)
|
||||
co.set_local_port(port=19897)
|
||||
self.browser = Chromium(co)
|
||||
self.tab = self.browser.latest_tab
|
||||
print("Chrome浏览器启动成功")
|
||||
|
||||
def close_init_popup(self):
|
||||
"""
|
||||
关闭所有的初始化弹窗
|
||||
"""
|
||||
self.tab.wait.doc_loaded(timeout=60, raise_err=True)
|
||||
footbar = self.tab.eles('xpath://footer[@class="el-dialog__footer"]', timeout=5)
|
||||
if len(footbar) > 0:
|
||||
do_not_remind = footbar[0].eles('xpath:.//input[@class="el-checkbox__original"]')
|
||||
if len(do_not_remind) > 0:
|
||||
do_not_remind[0].check()
|
||||
resume_immediately = footbar[0].eles('xpath:.//button')
|
||||
if len(resume_immediately) > 0:
|
||||
resume_immediately[0].click()
|
||||
|
||||
self.tab.wait.doc_loaded(timeout=60, raise_err=True)
|
||||
accept_btn = self.tab.eles('xpath://input[@id="sp-cc-accept"]', timeout=5)
|
||||
if len(accept_btn) > 0:
|
||||
accept_btn[0].click()
|
||||
|
||||
def run(self, country, asin):
|
||||
"""
|
||||
@@ -99,7 +34,7 @@ class ChromeAmzone:
|
||||
# 验证国家是否支持
|
||||
if country not in self.country_info:
|
||||
error_msg = f"不支持的国家: {country},支持的国家有: {list(self.country_info.keys())}"
|
||||
print(error_msg)
|
||||
self.log(error_msg)
|
||||
show_notification(error_msg, "error")
|
||||
return None
|
||||
|
||||
@@ -114,7 +49,7 @@ class ChromeAmzone:
|
||||
domain = base_url.split("/dp/")[0]
|
||||
# 拼接新的URL
|
||||
product_url = f"{domain}/dp/{asin}"
|
||||
print(f"正在访问: {product_url}")
|
||||
self.log(f"正在访问: {product_url}")
|
||||
|
||||
# 打开链接
|
||||
self.tab.get(product_url)
|
||||
@@ -123,13 +58,13 @@ class ChromeAmzone:
|
||||
|
||||
self.close_init_popup()
|
||||
# 2. 切换国家/设置邮编
|
||||
print(f"正在检查并设置邮编: {zip_code},标识: {mark}")
|
||||
self.log(f"正在检查并设置邮编: {zip_code},标识: {mark}")
|
||||
self._set_zip_code(zip_code, mark)
|
||||
|
||||
# self.tab.wait.doc_loaded(timeout=5, raise_err=False)
|
||||
|
||||
# 3. 抓取数据
|
||||
print("正在抓取商品数据...")
|
||||
self.log("正在抓取商品数据...")
|
||||
data = self._scrape_data()
|
||||
|
||||
# 添加基本信息
|
||||
@@ -138,100 +73,16 @@ class ChromeAmzone:
|
||||
data['url'] = product_url
|
||||
data['timestamp'] = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
print(f"数据抓取完成: {json.dumps(data)}")
|
||||
self.log(f"数据抓取完成: {json.dumps(data)}")
|
||||
return data
|
||||
|
||||
except Exception as e:
|
||||
error_msg = f"运行出错: {traceback.format_exc()}"
|
||||
print(error_msg)
|
||||
show_notification(f"采集失败: {str(e)}", "error")
|
||||
self.log(error_msg)
|
||||
# show_notification(f"采集失败: {str(e)}", "error")
|
||||
raise RuntimeError(error_msg)
|
||||
return {}
|
||||
|
||||
def _set_zip_code(self, zip_code, mark=None):
|
||||
"""
|
||||
设置邮编
|
||||
|
||||
Args:
|
||||
zip_code: 目标邮编
|
||||
"""
|
||||
try:
|
||||
# 检查当前邮编
|
||||
zip_display = self.tab.ele('xpath://div[@id="glow-ingress-block"]', timeout=10)
|
||||
|
||||
if zip_display:
|
||||
current_text = zip_display.text
|
||||
# print(f"当前地址信息: {current_text}")
|
||||
# 先检查标识
|
||||
if mark is not None and mark in current_text:
|
||||
print(f"邮编检测到标识: {mark},无需修改")
|
||||
return True
|
||||
# 检查是否已经包含目标邮编
|
||||
if zip_code in current_text:
|
||||
print(f"邮编已经设置为: {zip_code},无需修改")
|
||||
return True
|
||||
|
||||
# 需要设置邮编
|
||||
print(f"正在设置邮编为: {zip_code}")
|
||||
|
||||
# 点击地址选择按钮
|
||||
location_link = self.tab.ele('xpath://a[@id="nav-global-location-popover-link"]', timeout=10)
|
||||
if not location_link:
|
||||
print("找不到地址设置按钮")
|
||||
return False
|
||||
|
||||
location_link.click()
|
||||
time.sleep(1)
|
||||
|
||||
# 等待邮编输入框出现
|
||||
zip_input = self.tab.ele('xpath://input[@id="GLUXZipUpdateInput"]', timeout=10)
|
||||
if not zip_input:
|
||||
print("找不到邮编输入框")
|
||||
return False
|
||||
|
||||
# 输入邮编
|
||||
zip_input.input(zip_code, clear=True)
|
||||
time.sleep(0.5)
|
||||
|
||||
# 点击提交按钮
|
||||
submit_btn = self.tab.ele('xpath://input[@aria-labelledby="GLUXZipUpdate-announce"]', timeout=10)
|
||||
if not submit_btn:
|
||||
print("找不到提交按钮")
|
||||
return False
|
||||
|
||||
submit_btn.click()
|
||||
|
||||
continue_btn = self.tab.eles('xpath://div[@class="a-popover-footer"]//input[@id="GLUXConfirmClose"]',
|
||||
timeout=10)
|
||||
if len(continue_btn) > 0:
|
||||
continue_btn[0].click()
|
||||
|
||||
# 等待提交按钮消失(表示请求已发送)
|
||||
print("等待邮编更新...")
|
||||
time.sleep(2)
|
||||
|
||||
# 等待页面加载完成
|
||||
self.tab.wait.doc_loaded(timeout=30, raise_err=False)
|
||||
time.sleep(2)
|
||||
|
||||
# 验证邮编是否设置成功
|
||||
zip_display_after = self.tab.ele('xpath://div[@id="glow-ingress-block"]', timeout=10)
|
||||
if zip_display_after:
|
||||
updated_text = zip_display_after.text
|
||||
# print(f"更新后的地址信息: {updated_text}")
|
||||
|
||||
if zip_code in updated_text:
|
||||
print(f"邮编设置成功: {zip_code}")
|
||||
return True
|
||||
else:
|
||||
# print(f"邮编设置可能失败,当前显示: {updated_text}")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"设置邮编时出错: {traceback.format_exc()}")
|
||||
return False
|
||||
|
||||
def _scrape_data(self):
|
||||
"""
|
||||
抓取商品数据
|
||||
@@ -241,7 +92,8 @@ class ChromeAmzone:
|
||||
"""
|
||||
data = {
|
||||
'image_url': "",
|
||||
'title': ""
|
||||
'title': "",
|
||||
"sku" : ""
|
||||
}
|
||||
|
||||
try:
|
||||
@@ -250,42 +102,51 @@ class ChromeAmzone:
|
||||
title_ele = self.tab.ele('xpath://h1[@id="title"]',timeout=30)
|
||||
title = title_ele.text
|
||||
data["title"] = title
|
||||
sku_ele_ls = self.tab.eles('xpath://ul[@class="a-unordered-list a-vertical a-spacing-mini"]', timeout=20)
|
||||
if len(sku_ele_ls) > 0:
|
||||
data["sku"] = sku_ele_ls[0].text
|
||||
imge_ele = self.tab.ele('xpath://div[@id="imgTagWrapperId"]//img',timeout=20)
|
||||
image_url = imge_ele.attr("src")
|
||||
|
||||
image_url = ""
|
||||
min_image_url = ""
|
||||
data_a_dynamic_image = imge_ele.attr("data-a-dynamic-image")
|
||||
if data_a_dynamic_image:
|
||||
dynamic_image_json = json.loads(data_a_dynamic_image)
|
||||
self.log(f"图片信息:{dynamic_image_json}")
|
||||
max_area = 0
|
||||
min_area = 0
|
||||
|
||||
for url, (width, height) in dynamic_image_json.items():
|
||||
area = width * height
|
||||
if area > max_area:
|
||||
max_area = area
|
||||
image_url = url
|
||||
if min_area == 0:
|
||||
min_area = area
|
||||
min_image_url = url
|
||||
if area < min_area:
|
||||
min_area = area
|
||||
min_image_url = url
|
||||
|
||||
if not image_url:
|
||||
image_url = imge_ele.attr("src")
|
||||
|
||||
if not min_image_url:
|
||||
min_image_url = imge_ele.attr("src")
|
||||
|
||||
data["image_url"] = image_url
|
||||
data["min_image_url"] = min_image_url
|
||||
|
||||
return data
|
||||
|
||||
except Exception as e:
|
||||
print(f"抓取数据时出错: {traceback.format_exc()}")
|
||||
self.log(f"抓取数据时出错: {traceback.format_exc()}")
|
||||
return data
|
||||
|
||||
def close(self):
|
||||
"""关闭浏览器"""
|
||||
try:
|
||||
if self.browser:
|
||||
self.browser.quit()
|
||||
print("浏览器已关闭")
|
||||
except Exception as e:
|
||||
print(f"关闭浏览器时出错: {str(e)}")
|
||||
|
||||
|
||||
class SpiderTask:
|
||||
mark_name = "亚马逊采集"
|
||||
|
||||
def __init__(self, user_info: dict = None):
|
||||
"""初始化审批任务处理器
|
||||
|
||||
Args:
|
||||
user_info: 用户信息字典,包含 company, username, password
|
||||
"""
|
||||
self.user_info = user_info or {}
|
||||
self.running = True
|
||||
|
||||
def log(self, message: str, level: str = "INFO"):
|
||||
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
if level == "ERROR":
|
||||
show_notification(message, "error")
|
||||
print(f"[{timestamp}] [PriceTask] [{level}] {message}")
|
||||
class SpiderTask(TaskBase):
|
||||
task_name = "亚马逊采集"
|
||||
|
||||
@staticmethod
|
||||
def group_by_id_prefix(data):
|
||||
@@ -307,6 +168,31 @@ class SpiderTask:
|
||||
|
||||
return list(grouped.values())
|
||||
|
||||
@staticmethod
|
||||
def normalize_groups(data):
|
||||
groups = data.get("groups")
|
||||
if isinstance(groups, list) and len(groups) > 0:
|
||||
return groups
|
||||
|
||||
rows = data.get("rows") or data.get("items") or []
|
||||
if not isinstance(rows, list) or len(rows) == 0:
|
||||
return []
|
||||
|
||||
normalized = []
|
||||
for items in SpiderTask.group_by_id_prefix(rows):
|
||||
first = items[0] if items else {}
|
||||
item_id = str(first.get("id") or first.get("displayId") or "")
|
||||
base_id = item_id.split("_")[0] if item_id else ""
|
||||
normalized.append({
|
||||
"sourceFileKey": first.get("sourceFileKey", ""),
|
||||
"sourceFilename": first.get("sourceFilename", ""),
|
||||
"groupKey": first.get("groupKey") or base_id,
|
||||
"baseId": first.get("baseId") or base_id,
|
||||
"displayId": first.get("displayId") or item_id,
|
||||
"items": items,
|
||||
})
|
||||
return normalized
|
||||
|
||||
def process_task(self, task_data: dict):
|
||||
"""处理审批任务主入口
|
||||
|
||||
@@ -316,7 +202,7 @@ class SpiderTask:
|
||||
try:
|
||||
data = task_data.get("data", {})
|
||||
task_id = data.get("taskId")
|
||||
groups = data.get("groups")
|
||||
groups = self.normalize_groups(data)
|
||||
|
||||
# 用于测试
|
||||
limit = data.get("limit", None)
|
||||
@@ -328,7 +214,10 @@ class SpiderTask:
|
||||
|
||||
self.log(f"开始处理爬取任务 {task_id},{len(groups)} 个任务")
|
||||
|
||||
from config import runing_task
|
||||
if not groups:
|
||||
self.log("appearance-patent groups/rows is empty, skip", "WARNING")
|
||||
return
|
||||
|
||||
runing_task[task_id] = {
|
||||
"status": "running",
|
||||
"start_time": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
|
||||
@@ -357,6 +246,7 @@ class SpiderTask:
|
||||
result = []
|
||||
for gp_index,gp in enumerate(groups):
|
||||
items = gp.get("items", [])
|
||||
return_data = {}
|
||||
|
||||
for index,value in enumerate(items):
|
||||
|
||||
@@ -366,15 +256,16 @@ class SpiderTask:
|
||||
}
|
||||
asin = value.get("asin")
|
||||
country = value.get("country")
|
||||
return_data = None
|
||||
for _ in range(max_retry):
|
||||
try:
|
||||
return_data = chrome.run(country, asin)
|
||||
return_data = chrome.run(country, asin) or {}
|
||||
self.log(f"抓取结果->{return_data}")
|
||||
break
|
||||
except Exception as e:
|
||||
if "与页面的连接已断开" in str(e):
|
||||
chrome = ChromeAmzone()
|
||||
if not isinstance(return_data, dict):
|
||||
return_data = {}
|
||||
if return_data.get("image_url"):
|
||||
break
|
||||
|
||||
@@ -389,6 +280,7 @@ class SpiderTask:
|
||||
"country": i.get("country"),
|
||||
"url": return_data.get("image_url"),
|
||||
"title": return_data.get("title"),
|
||||
"sku": return_data.get("sku")
|
||||
}
|
||||
for i in items
|
||||
]
|
||||
@@ -406,7 +298,7 @@ class SpiderTask:
|
||||
result.append(res)
|
||||
print("================")
|
||||
is_done = gp_index == len(groups)-1
|
||||
if len(result) > 20 or is_done:
|
||||
if len(result) > 10 or is_done:
|
||||
self.post_result(task_id=task_id,chunkIndex=gp_index+1,chunkTotal=len(groups),
|
||||
asin=asin,item_data=result,is_done=is_done)
|
||||
result = []
|
||||
@@ -416,19 +308,16 @@ class SpiderTask:
|
||||
is_done = True
|
||||
self.post_result(task_id=task_id, chunkIndex=len(groups), chunkTotal=len(groups),
|
||||
asin=asin, item_data=result, is_done=is_done)
|
||||
|
||||
try:
|
||||
chrome.close()
|
||||
except Exception as e:
|
||||
print("退出浏览器出错",e)
|
||||
# 更新已处理店铺数
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["processed_shops"] += 1
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"处理店铺 {task_id} 失败: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
|
||||
try:
|
||||
chrome.close()
|
||||
except Exception as e:
|
||||
print("退出浏览器出错", e)
|
||||
# 更新任务状态
|
||||
if task_id in runing_task:
|
||||
if runing_task[task_id].get("stop_requested", False):
|
||||
@@ -439,10 +328,8 @@ class SpiderTask:
|
||||
self.log(f"任务 {task_id} 处理完成!")
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"任务处理失败: {traceback.format_exc()}", "ERROR")
|
||||
if task_id:
|
||||
from config import runing_task
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["status"] = "failed"
|
||||
runing_task[task_id]["error"] = str(e)
|
||||
@@ -456,7 +343,7 @@ class SpiderTask:
|
||||
url = f"{DELETE_BRAND_API_BASE}/api/appearance-patent/tasks/{task_id}/result"
|
||||
|
||||
payload ={
|
||||
"submissionId": f"{int(time.time())}",
|
||||
"submissionId": f"{task_id}",
|
||||
"chunkIndex": chunkIndex,
|
||||
"chunkTotal": chunkTotal,
|
||||
"error": error,
|
||||
@@ -497,7 +384,7 @@ class SpiderTask:
|
||||
time.sleep(2)
|
||||
|
||||
self.log(f"已达到最大重试次数,结果回传最终失败", "ERROR")
|
||||
raise RuntimeError("已达到最大重试次数,结果回传最终失败")
|
||||
# raise RuntimeError("已达到最大重试次数,结果回传最终失败")
|
||||
|
||||
if __name__ == '__main__':
|
||||
# spide = SpiderTask()
|
||||
@@ -8381,4 +8268,4 @@ if __name__ == '__main__':
|
||||
]
|
||||
# spide.process_task(task_data)
|
||||
res = SpiderTask.group_by_id_prefix(task_data)
|
||||
print(res)
|
||||
print(res)
|
||||
|
||||
8384
app/amazon/detail_spider_备份.py
Normal file
8384
app/amazon/detail_spider_备份.py
Normal file
File diff suppressed because it is too large
Load Diff
@@ -1,20 +1,19 @@
|
||||
import time
|
||||
import traceback
|
||||
import requests
|
||||
import threading
|
||||
from datetime import datetime
|
||||
from typing import Dict, Any, List
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from config import JSON_TASK_QUEUE, runing_task, runing_shop, DELETE_BRAND_API_BASE, ZN_COMPANY, ZN_USERNAME, ZN_PASSWORD
|
||||
from amazon.del_brand import AmazoneDriver, kill_process
|
||||
from amazon.del_brand import AmazoneDriver, DelbramdTask
|
||||
from amazon.approve import ApproveTask
|
||||
from amazon.match_action import MatchTak
|
||||
from amazon.price_match import PriceTask
|
||||
from amazon.asin_status import StatusTask
|
||||
from amazon.patrol_delete import PatrolDeleteTask
|
||||
from amazon.detail_spider import SpiderTask
|
||||
|
||||
|
||||
from amazon.tool import get_shop_info,show_notification
|
||||
from amazon.similar_asin import SimilarAsinTask
|
||||
from amazon.amazon_base import kill_process
|
||||
|
||||
|
||||
class TaskMonitor:
|
||||
@@ -32,6 +31,9 @@ class TaskMonitor:
|
||||
self.chunk_index = 1 # 当前处理的分块索引
|
||||
self.max_workers = 5 # 线程池最大线程数
|
||||
self.executor = None # 线程池执行器
|
||||
self.serial_task_locks = {
|
||||
"product-risk-resolve-run": threading.Lock(),
|
||||
}
|
||||
|
||||
# 在提交新任务前杀掉旧进程(确保环境干净)
|
||||
kill_process("v6")
|
||||
@@ -56,30 +58,25 @@ class TaskMonitor:
|
||||
futures = [] # 保存所有提交的任务Future对象
|
||||
|
||||
task_type_info = {
|
||||
"delete-brand-run" : "删除品牌",
|
||||
"product-risk-resolve-run" : "产品风险审批",
|
||||
"shop-match-run" : "匹配价格",
|
||||
"price-track-run" : "跟价",
|
||||
"query-asin-run" : "状态查询",
|
||||
"patrol-delete-run" : "巡店删除",
|
||||
"appearance-patent-run" : "亚马逊采集",
|
||||
"similar-asin-run" : "相似ASIN"
|
||||
}
|
||||
try:
|
||||
while self.running:
|
||||
try:
|
||||
# 使用较长超时时间等待任务,减少空等待异常
|
||||
# 超时后继续循环检查self.running状态,避免卡死
|
||||
task_data = JSON_TASK_QUEUE.get(block=True, timeout=30)
|
||||
|
||||
# 检查任务类型
|
||||
task_type = task_data.get("type", "")
|
||||
|
||||
if task_type == "delete-brand-run":
|
||||
# 提交删除品牌任务到线程池
|
||||
self.log(f"接收到【删除品牌】任务,提交到线程池处理...")
|
||||
future = self.executor.submit(self._process_task_wrapper, task_data)
|
||||
futures.append(future)
|
||||
|
||||
elif task_type in task_type_info:
|
||||
|
||||
if task_type in task_type_info:
|
||||
# 提交产品风险审批任务到线程池
|
||||
self.log(f"接收到任务,提交到线程池处理...")
|
||||
future = self.executor.submit(self._process_approve_task_wrapper, task_data, task_type_info[task_type])
|
||||
@@ -112,13 +109,23 @@ class TaskMonitor:
|
||||
task_data: 任务数据
|
||||
"""
|
||||
try:
|
||||
self.log(f"线程 {id(task_data)} 开始处理删除品牌任务...")
|
||||
self.log(f"线程 {id(task_data)} 开始处理任务...")
|
||||
self.process_task(task_data)
|
||||
self.log(f"线程 {id(task_data)} 删除品牌任务处理完成")
|
||||
self.log(f"线程 {id(task_data)} 任务处理完成")
|
||||
except Exception as e:
|
||||
self.log(f"线程 {id(task_data)} 删除品牌任务处理异常: {traceback.format_exc()}", "ERROR")
|
||||
|
||||
def _process_approve_task_wrapper(self, task_data: Dict[str, Any],TASK_TYPE:str):
|
||||
task_type = task_data.get("type", "")
|
||||
task_lock = self.serial_task_locks.get(task_type)
|
||||
if task_lock is not None:
|
||||
self.log(f"线程 {id(task_data)} 等待 {task_type} 串行锁...")
|
||||
with task_lock:
|
||||
self.log(f"线程 {id(task_data)} 获取 {task_type} 串行锁,开始处理...")
|
||||
return self._process_approve_task_unlocked(task_data, TASK_TYPE)
|
||||
return self._process_approve_task_unlocked(task_data, TASK_TYPE)
|
||||
|
||||
def _process_approve_task_unlocked(self, task_data: Dict[str, Any],TASK_TYPE:str):
|
||||
"""审批任务处理包装器(用于线程池调用)
|
||||
|
||||
Args:
|
||||
@@ -126,552 +133,24 @@ class TaskMonitor:
|
||||
"""
|
||||
try:
|
||||
TASK_INFO = {
|
||||
"删除品牌": DelbramdTask ,
|
||||
"产品风险审批" : ApproveTask,
|
||||
"匹配价格" : MatchTak,
|
||||
"跟价" : PriceTask,
|
||||
"状态查询" : StatusTask,
|
||||
"巡店删除" : PatrolDeleteTask,
|
||||
"亚马逊采集" : SpiderTask,
|
||||
"相似ASIN" : SimilarAsinTask
|
||||
}
|
||||
self.log(f"线程 {id(task_data)} 开始处理产品风险审批任务...")
|
||||
# 创建ApproveTask实例并处理任务
|
||||
TASK_CLS = TASK_INFO[TASK_TYPE] # 根据任务类型选择处理类,默认为ApproveTask
|
||||
approve_task = TASK_CLS(user_info=self.user_info)
|
||||
approve_task.process_task(task_data)
|
||||
self.log(f"线程 {id(task_data)} 产品风险审批任务处理完成")
|
||||
self.log(f"线程 {id(task_data)} {TASK_TYPE} 任务处理完成")
|
||||
except Exception as e:
|
||||
self.log(f"线程 {id(task_data)} 产品风险审批任务处理异常: {traceback.format_exc()}", "ERROR")
|
||||
self.log(f"线程 {id(task_data)} {TASK_TYPE} 任务处理异常: {traceback.format_exc()}", "ERROR")
|
||||
|
||||
def process_task(self, task_data: Dict[str, Any]):
|
||||
"""处理单个任务
|
||||
|
||||
Args:
|
||||
task_data: 任务数据
|
||||
"""
|
||||
try:
|
||||
data = task_data.get("data", {})
|
||||
task_id = data.get("taskId")
|
||||
items = data.get("items", [])
|
||||
|
||||
if not task_id:
|
||||
self.log("任务ID为空,跳过", "WARNING")
|
||||
return
|
||||
|
||||
# 初始化任务状态
|
||||
self.update_task_status(
|
||||
task_id,
|
||||
status="running",
|
||||
start_time=datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
|
||||
total_shops=len(items),
|
||||
processed_shops=0,
|
||||
total_asins=sum(shop.get("totalRows", 0) for shop in items),
|
||||
processed_asins=0,
|
||||
success_count=0,
|
||||
failed_count=0,
|
||||
stop_requested=False # 暂停请求标志
|
||||
)
|
||||
|
||||
self.log(f"开始处理任务 {task_id},共 {len(items)} 个店铺")
|
||||
|
||||
# 遍历处理每个店铺
|
||||
for idx, shop_data in enumerate(items, 1):
|
||||
shop_name = shop_data.get("shopName", "未知店铺")
|
||||
self.log(f"[{idx}/{len(items)}] 开始处理店铺: {shop_name}")
|
||||
|
||||
try:
|
||||
self.process_shop(shop_data, task_id)
|
||||
# 更新已处理店铺数
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["processed_shops"] += 1
|
||||
except Exception as e:
|
||||
self.log(f"处理店铺 {shop_name} 失败: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
|
||||
# 检查任务最终状态
|
||||
if task_id in runing_task and runing_task[task_id].get("stop_requested", False):
|
||||
self.update_task_status(task_id, status="stopped")
|
||||
self.log(f"任务 {task_id} 已被暂停!")
|
||||
else:
|
||||
self.update_task_status(task_id, status="completed")
|
||||
self.log(f"任务 {task_id} 处理完成!")
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"任务处理失败: {traceback.format_exc()}", "ERROR")
|
||||
if task_id:
|
||||
self.update_task_status(task_id, status="failed", error=str(e))
|
||||
|
||||
def process_shop(self, shop_data: Dict[str, Any], task_id: int):
|
||||
"""处理单个店铺(包含重试逻辑)
|
||||
|
||||
Args:
|
||||
shop_data: 店铺数据
|
||||
task_id: 任务ID
|
||||
"""
|
||||
shop_name = shop_data.get("shopName", "未知店铺")
|
||||
result_id = shop_data.get("resultId")
|
||||
countries = shop_data.get("countries", [])
|
||||
company_name = shop_data.get("companyName", "未知公司")
|
||||
|
||||
if not countries:
|
||||
self.log(f"店铺 {shop_name} 没有国家数据,跳过", "WARNING")
|
||||
return
|
||||
|
||||
# 更新当前处理的店铺
|
||||
self.update_task_status(task_id, current_shop=shop_name)
|
||||
|
||||
# 将店铺添加到正在执行中的店铺列表
|
||||
start_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
runing_shop[shop_name] = start_time
|
||||
self.log(f"账号:{company_name},店铺 {shop_name} 已添加到执行列表,开始时间: {start_time}")
|
||||
|
||||
# 店铺打开重试最多3次
|
||||
driver = None
|
||||
browser = None
|
||||
max_retries = 3
|
||||
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
self.log(f"尝试打开店铺 {shop_name} (第 {retry + 1}/{max_retries} 次)")
|
||||
|
||||
# 如果不是第一次尝试,先杀进程
|
||||
# if retry > 0:
|
||||
# self.log("重试前先杀掉浏览器进程...")
|
||||
# kill_process("v6")
|
||||
# kill_process("v5")
|
||||
# time.sleep(2)
|
||||
|
||||
# 创建驱动并打开店铺
|
||||
user_info = {
|
||||
**self.user_info,
|
||||
"company": company_name
|
||||
}
|
||||
driver = AmazoneDriver(user_info)
|
||||
browser = driver.open_shop(shop_name)
|
||||
|
||||
if browser and browser != "店铺不存在":
|
||||
self.log(f"成功打开店铺 {shop_name}")
|
||||
break
|
||||
else:
|
||||
self.log(f"打开店铺失败: {browser}", "WARNING")
|
||||
driver = None
|
||||
|
||||
# 判断是否需要登录
|
||||
driver.tab.wait.doc_loaded(timeout=30, raise_err=False) # 等待页面加载,避免过早判断登录状态
|
||||
need_login = driver.need_login()
|
||||
print("【是否需要登录】:",need_login)
|
||||
if need_login:
|
||||
self.log(f"店铺 {shop_name} 需要登录,正在登录...")
|
||||
# 获取店铺凭证
|
||||
response = get_shop_info(shop_name)
|
||||
print("【获取店铺凭证返回】:",response.text)
|
||||
shop_data = response.json()
|
||||
if not shop_data:
|
||||
mes = f"获取店铺凭证失败,响应数据: {shop_data.get('message', '未知错误')}"
|
||||
self.log(mes, "ERROR")
|
||||
show_notification(mes, "ERROR")
|
||||
continue
|
||||
|
||||
password = shop_data["data"]["password"]
|
||||
|
||||
login_success = driver.login(password)
|
||||
if login_success:
|
||||
self.log(f"店铺 {shop_name} 登录成功,正在重新打开店铺...")
|
||||
browser = driver.open_shop(shop_name)
|
||||
if browser and browser != "店铺不存在":
|
||||
self.log(f"成功打开店铺 {shop_name} 登录后")
|
||||
break
|
||||
else:
|
||||
self.log(f"登录后打开店铺失败: {browser}", "WARNING")
|
||||
driver = None
|
||||
else:
|
||||
self.log(f"店铺 {shop_name} 登录失败", "WARNING")
|
||||
driver = None
|
||||
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"打开店铺异常: {str(e)}", "ERROR")
|
||||
driver = None
|
||||
|
||||
# 如果还有重试机会,等待后继续
|
||||
if retry < max_retries - 1:
|
||||
time.sleep(3)
|
||||
|
||||
# 检查是否成功打开
|
||||
if not driver or not browser or browser == "店铺不存在":
|
||||
self.log(f"店铺 {shop_name} 打开失败,已重试 {max_retries} 次,跳过该店铺", "ERROR")
|
||||
return
|
||||
|
||||
try:
|
||||
# 处理每个国家
|
||||
chunk_index =1
|
||||
for country_data in countries:
|
||||
# 检查是否收到暂停请求
|
||||
if task_id in runing_task and runing_task[task_id].get("stop_requested", False):
|
||||
self.log(f"检测到任务 {task_id} 的暂停请求,停止处理国家", "WARNING")
|
||||
break # 跳出循环,进入finally关闭店铺
|
||||
|
||||
try:
|
||||
chunk_index = self.process_country(driver, country_data, task_id, result_id, shop_data,chunk_index)
|
||||
except Exception as e:
|
||||
country_name = country_data.get("country", "未知")
|
||||
self.log(f"处理国家 {country_name} 失败: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
|
||||
finally:
|
||||
# 关闭店铺
|
||||
try:
|
||||
if driver:
|
||||
self.log(f"关闭店铺 {shop_name}")
|
||||
driver.close_store()
|
||||
time.sleep(2)
|
||||
except Exception as e:
|
||||
self.log(f"关闭店铺失败: {str(e)}", "WARNING")
|
||||
|
||||
# 从正在执行中的店铺列表中移除
|
||||
if shop_name in runing_shop:
|
||||
del runing_shop[shop_name]
|
||||
self.log(f"店铺 {shop_name} 已从执行列表中移除")
|
||||
|
||||
def process_country(self, driver: AmazoneDriver, country_data: Dict[str, Any],
|
||||
task_id: int, result_id: int, shop_data: Dict[str, Any],chunk_index:int):
|
||||
"""处理单个国家的所有ASIN
|
||||
|
||||
Args:
|
||||
driver: 亚马逊驱动实例
|
||||
country_data: 国家数据
|
||||
task_id: 任务ID
|
||||
result_id: 结果ID
|
||||
shop_data: 店铺数据
|
||||
"""
|
||||
country = country_data.get("country", "未知")
|
||||
items = country_data.get("items", [])
|
||||
|
||||
self.log(f"开始处理国家: {country},共 {len(items)} 个ASIN")
|
||||
self.update_task_status(task_id, current_country=country)
|
||||
|
||||
# 切换国家,最多重试3次
|
||||
max_retries = 3
|
||||
switch_success = False
|
||||
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
self.log(f"尝试切换到国家 {country} (第 {retry + 1}/{max_retries} 次)")
|
||||
|
||||
# 如果不是第一次尝试,先刷新页面
|
||||
if retry > 0:
|
||||
self.log("重试前刷新页面...")
|
||||
try:
|
||||
driver.tab.refresh()
|
||||
time.sleep(3)
|
||||
except Exception as e:
|
||||
self.log(f"刷新页面失败: {str(e)}", "WARNING")
|
||||
|
||||
switch_success = driver.SwitchingCountries(country)
|
||||
if switch_success:
|
||||
self.log(f"成功切换到国家 {country}")
|
||||
break
|
||||
else:
|
||||
self.log(f"切换到国家 {country} 失败", "WARNING")
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"切换国家 {country} 异常: {str(e)}", "ERROR")
|
||||
|
||||
# 如果还有重试机会,等待后继续
|
||||
if retry < max_retries - 1:
|
||||
time.sleep(2)
|
||||
|
||||
# 如果切换失败,回传该国家所有ASIN为失败状态
|
||||
if not switch_success:
|
||||
self.log(f"切换到国家 {country} 失败,已重试 {max_retries} 次,将所有ASIN标记为失败", "ERROR")
|
||||
chunk_index = self._report_all_asins_failed(country, items, task_id, shop_data,chunk_index)
|
||||
return chunk_index
|
||||
|
||||
# 切换到库存管理页面
|
||||
try:
|
||||
driver.SwitchPage()
|
||||
self.log(f"已切换到库存管理页面")
|
||||
except Exception as e:
|
||||
self.log(f"切换页面失败: {str(e)}", "ERROR")
|
||||
# 切换页面失败也回传所有ASIN为失败
|
||||
chunk_index = self._report_all_asins_failed(country, items, task_id, shop_data,chunk_index)
|
||||
return chunk_index
|
||||
|
||||
# 处理每个ASIN
|
||||
file_key = shop_data.get("fileKey", "")
|
||||
source_filename = shop_data.get("sourceFilename", "")
|
||||
total_rows = shop_data.get("totalRows", 0)
|
||||
|
||||
for idx, asin_item in enumerate(items, 1):
|
||||
# 检查是否收到暂停请求
|
||||
if task_id in runing_task and runing_task[task_id].get("stop_requested", False):
|
||||
self.log(f"检测到任务 {task_id} 的暂停请求,停止处理", "WARNING")
|
||||
return # 返回到上层,会触发店铺关闭
|
||||
|
||||
try:
|
||||
self.log(f"[{idx}/{len(items)}] 处理ASIN: {asin_item.get('asin', '')}")
|
||||
self.process_asin(driver, asin_item, country, task_id, result_id,
|
||||
file_key, source_filename, total_rows,chunk_index)
|
||||
except Exception as e:
|
||||
asin = asin_item.get("asin", "未知")
|
||||
self.log(f"处理ASIN {asin} 失败: {str(e)}", "ERROR")
|
||||
# 继续处理下一个ASIN
|
||||
chunk_index += 1
|
||||
return chunk_index
|
||||
|
||||
|
||||
def process_asin(self, driver: AmazoneDriver, asin_item: Dict[str, Any],
|
||||
country: str, task_id: int, result_id: int, file_key: str,
|
||||
source_filename: str, total_rows: int,chunk_index:int):
|
||||
"""处理单个ASIN并回传结果
|
||||
|
||||
Args:
|
||||
driver: 亚马逊驱动实例
|
||||
asin_item: ASIN数据项
|
||||
country: 国家名称
|
||||
task_id: 任务ID
|
||||
result_id: 结果ID
|
||||
file_key: 文件KEY
|
||||
source_filename: 源文件名
|
||||
total_rows: 总行数
|
||||
chunk_index: 当前处理索引
|
||||
"""
|
||||
asin = asin_item.get("asin", "")
|
||||
row_index = asin_item.get("rowIndex", 0)
|
||||
|
||||
# 更新当前处理的ASIN
|
||||
self.update_task_status(task_id, current_asin=asin)
|
||||
|
||||
status = "失败"
|
||||
max_retries = 3 # 最多重试3次
|
||||
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
self.log(f"处理ASIN {asin} (第 {retry + 1}/{max_retries} 次)")
|
||||
|
||||
# 如果不是第一次尝试,先刷新页面
|
||||
if retry > 0:
|
||||
self.log("重试前刷新页面...")
|
||||
try:
|
||||
driver.tab.refresh()
|
||||
time.sleep(3)
|
||||
except Exception as e:
|
||||
self.log(f"刷新页面失败: {str(e)}", "WARNING")
|
||||
|
||||
# 搜索ASIN
|
||||
sku_ls = driver.search(asin=asin)
|
||||
self.log(f"搜索到 {len(sku_ls)} 个SKU")
|
||||
|
||||
if len(sku_ls) == 0:
|
||||
status = "查询不到"
|
||||
self.log(f"ASIN {asin} 未找到商品", "WARNING")
|
||||
break # 查询不到商品,无需重试
|
||||
else:
|
||||
# 删除所有找到的SKU,如果任何一个失败则重新开始整个流程
|
||||
success_count = 0
|
||||
all_success = True # 标记是否所有SKU都删除成功
|
||||
total_sku_count = len(sku_ls)
|
||||
|
||||
for sku in sku_ls:
|
||||
try:
|
||||
suc = driver.del_action(sku)
|
||||
if suc:
|
||||
success_count += 1
|
||||
self.log(f"SKU 删除成功 ({success_count}/{total_sku_count})")
|
||||
else:
|
||||
self.log(f"SKU 删除失败", "WARNING")
|
||||
all_success = False
|
||||
break # 任何一个失败,退出循环,准备重试整个流程
|
||||
except Exception as e:
|
||||
self.log(f"删除SKU异常: {str(e)}", "ERROR")
|
||||
all_success = False
|
||||
break # 发生异常,退出循环,准备重试整个流程
|
||||
|
||||
# 判断删除结果
|
||||
if all_success and success_count == total_sku_count:
|
||||
status = "成功"
|
||||
self.log(f"ASIN {asin} 所有SKU删除成功 ({success_count}/{total_sku_count})")
|
||||
# 更新成功计数
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["success_count"] += 1
|
||||
break # 全部成功,跳出重试循环
|
||||
else:
|
||||
# 有失败的SKU
|
||||
if retry < max_retries - 1:
|
||||
self.log(f"有SKU删除失败,准备重试整个流程... ({retry + 1}/{max_retries})")
|
||||
time.sleep(2)
|
||||
continue # 继续下一次重试
|
||||
else:
|
||||
# 所有重试都用完了
|
||||
if success_count > 0:
|
||||
status = "部分成功"
|
||||
self.log(f"ASIN {asin} 部分SKU删除成功 ({success_count}/{total_sku_count})", "WARNING")
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["success_count"] += 1
|
||||
else:
|
||||
status = "失败"
|
||||
self.log(f"ASIN {asin} 所有SKU删除失败", "ERROR")
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["failed_count"] += 1
|
||||
|
||||
except Exception as e:
|
||||
status = "删除异常"
|
||||
self.log(f"处理ASIN {asin} 异常: {str(e)}", "ERROR")
|
||||
|
||||
# 如果还有重试机会,继续重试
|
||||
if retry < max_retries - 1:
|
||||
self.log(f"发生异常,准备重试... ({retry + 1}/{max_retries})")
|
||||
time.sleep(2)
|
||||
else:
|
||||
# 所有重试都失败了,更新失败计数
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["failed_count"] += 1
|
||||
|
||||
# 更新已处理ASIN计数
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["processed_asins"] += 1
|
||||
|
||||
# 回传结果到API
|
||||
try:
|
||||
payload = {
|
||||
"submissionId": "",
|
||||
"files": [{
|
||||
"fileKey": file_key,
|
||||
"sourceFilename": source_filename,
|
||||
"chunkIndex": chunk_index,
|
||||
"chunkTotal": total_rows,
|
||||
"processedRows": chunk_index,
|
||||
"totalRows": total_rows,
|
||||
"currentCountry": country,
|
||||
"currentAsin": asin,
|
||||
"countries": [{
|
||||
"country": country,
|
||||
"items": [{
|
||||
"asin": asin,
|
||||
"status": status
|
||||
}]
|
||||
}]
|
||||
}]
|
||||
}
|
||||
|
||||
self.post_result(task_id, payload)
|
||||
self.log(f"ASIN {asin} 结果已回传,状态: {status}")
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"回传结果失败: {str(e)}", "ERROR")
|
||||
|
||||
def _report_all_asins_failed(self, country: str, items: List[Dict[str, Any]],
|
||||
task_id: int, shop_data: Dict[str, Any],chunk_index:int):
|
||||
"""将国家下所有ASIN标记为失败并回传
|
||||
|
||||
Args:
|
||||
country: 国家名称
|
||||
items: ASIN列表
|
||||
task_id: 任务ID
|
||||
shop_data: 店铺数据
|
||||
"""
|
||||
self.log(f"开始回传国家 {country} 下的 {len(items)} 个ASIN失败状态")
|
||||
|
||||
file_key = shop_data.get("fileKey", "")
|
||||
source_filename = shop_data.get("sourceFilename", "")
|
||||
total_rows = shop_data.get("totalRows", 0)
|
||||
|
||||
for asin_item in items:
|
||||
asin = asin_item.get("asin", "")
|
||||
|
||||
try:
|
||||
# 更新失败计数
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["failed_count"] += 1
|
||||
runing_task[task_id]["processed_asins"] += 1
|
||||
|
||||
# 回传失败状态
|
||||
payload = {
|
||||
"submissionId": "",
|
||||
"files": [{
|
||||
"fileKey": file_key,
|
||||
"sourceFilename": source_filename,
|
||||
"chunkIndex": chunk_index,
|
||||
"chunkTotal": total_rows,
|
||||
"processedRows": chunk_index,
|
||||
"totalRows": total_rows,
|
||||
"currentCountry": country,
|
||||
"currentAsin": asin,
|
||||
"countries": [{
|
||||
"country": country,
|
||||
"items": [{
|
||||
"asin": asin,
|
||||
"status": "失败"
|
||||
}]
|
||||
}]
|
||||
}]
|
||||
}
|
||||
|
||||
self.post_result(task_id, payload)
|
||||
chunk_index += 1
|
||||
self.log(f"ASIN {asin} 失败状态已回传")
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"回传ASIN {asin} 失败状态时出错: {str(e)}", "ERROR")
|
||||
|
||||
return chunk_index
|
||||
|
||||
def post_result(self, task_id: int, payload: Dict[str, Any]):
|
||||
"""回传结果到API(带重试机制)
|
||||
|
||||
Args:
|
||||
task_id: 任务ID
|
||||
payload: 结果数据
|
||||
"""
|
||||
url = f"{DELETE_BRAND_API_BASE}/api/delete-brand/tasks/{task_id}/result"
|
||||
max_retries = 3 # 最多重试3次
|
||||
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
self.log(f"尝试回传结果 (第 {retry + 1}/{max_retries} 次)")
|
||||
|
||||
response = requests.post(
|
||||
url,
|
||||
json=payload,
|
||||
headers={"Content-Type": "application/json"},
|
||||
timeout=30,
|
||||
verify=False # 忽略SSL证书验证
|
||||
)
|
||||
print("【结果提交】:",payload)
|
||||
print("【结果提交返回】:",response.text)
|
||||
|
||||
if response.status_code == 200:
|
||||
self.log(f"结果回传成功: {url}")
|
||||
return # 成功后直接返回,不再重试
|
||||
else:
|
||||
self.log(f"结果回传失败,状态码: {response.status_code}", "WARNING")
|
||||
# 如果还有重试机会,继续重试
|
||||
if retry < max_retries - 1:
|
||||
self.log(f"准备重试... ({retry + 1}/{max_retries})")
|
||||
time.sleep(2) # 等待2秒后重试
|
||||
else:
|
||||
self.log(f"已达到最大重试次数,结果回传最终失败", "ERROR")
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"调用API异常: {str(e)}", "ERROR")
|
||||
# 如果还有重试机会,继续重试
|
||||
if retry < max_retries - 1:
|
||||
self.log(f"发生异常,准备重试... ({retry + 1}/{max_retries})")
|
||||
time.sleep(2) # 等待2秒后重试
|
||||
else:
|
||||
self.log(f"已达到最大重试次数,结果回传最终失败", "ERROR")
|
||||
|
||||
def update_task_status(self, task_id: int, **kwargs):
|
||||
"""更新任务状态
|
||||
|
||||
Args:
|
||||
task_id: 任务ID
|
||||
**kwargs: 要更新的字段
|
||||
"""
|
||||
if task_id not in runing_task:
|
||||
runing_task[task_id] = {}
|
||||
|
||||
runing_task[task_id].update(kwargs)
|
||||
|
||||
def stop(self):
|
||||
"""停止监控"""
|
||||
self.running = False
|
||||
|
||||
@@ -1,82 +1,24 @@
|
||||
import time
|
||||
import re
|
||||
import traceback
|
||||
from DrissionPage._pages.chromium_tab import ChromiumTab
|
||||
|
||||
from config import runing_task, runing_shop
|
||||
from config import runing_task, runing_shop,DELETE_BRAND_API_BASE
|
||||
from datetime import datetime
|
||||
from amazon.del_brand import AmamzonBase, kill_process
|
||||
|
||||
from amazon.tool import show_notification,get_shop_info
|
||||
from amazon.amazon_base import AmamzonBase, kill_process,TaskBase
|
||||
from amazon.tool import show_notification
|
||||
import requests
|
||||
|
||||
|
||||
class AmzoneMatchAction(AmamzonBase):
|
||||
mark_name = "匹配价格"
|
||||
|
||||
def SwitchPage(self):
|
||||
"""
|
||||
切换至 管理所有库存页面
|
||||
1、等待 //navigation-favorites-bar[@class="hydrated"] 出现
|
||||
"""
|
||||
navigation = self.tab.ele('xpath://navigation-favorites-bar[@class="hydrated"]')
|
||||
navigation.wait.displayed(raise_err=False)
|
||||
page_btn = navigation.sr('xpath://internal-fav-bar-links[@data-internal="navigation"]').sr(
|
||||
'xpath://a[@data-page-id="ezdpc-gui-inventory-mons"]')
|
||||
page_btn.wait.displayed(raise_err=False)
|
||||
page_btn.click(timeout=5)
|
||||
def __init__(self, user_info: dict, socket_port: int = 19890):
|
||||
super().__init__(user_info,socket_port)
|
||||
self.already_asin = set()
|
||||
|
||||
self.tab.wait.doc_loaded()
|
||||
# 等待搜索框出现
|
||||
search_region = self.tab.ele('xpath://div[@id="searchBoxContainer"]//kat-input-group')
|
||||
search_region.wait.displayed(raise_err=False)
|
||||
|
||||
def search(self,filter_type="ApprovalRequired"):
|
||||
sku_ls = []
|
||||
|
||||
load_ele = self.tab.eles("xpath://div[contains(@class,'Loader-module__loader')]",timeout=5)
|
||||
if len(load_ele) > 0:
|
||||
load_ele[0].wait.deleted(timeout=3, raise_err=False)
|
||||
time.sleep(0.5)
|
||||
|
||||
drop_down = self.tab.ele('xpath://div[contains(@class,"VolusListingStatusDropDown-module__verticalContainer")]//kat-dropdown')
|
||||
drop_down.wait.displayed(raise_err=False)
|
||||
drop_down.wait.enabled(raise_err=False)
|
||||
time.sleep(0.6)
|
||||
drop_down.click()
|
||||
# //kat-option[@value="SearchSuppressed"]
|
||||
xp = f'xpath://kat-option[@value="{filter_type}"]'
|
||||
print(f"【{self.mark_name}】正在寻找筛选条件 {filter_type},xpath: {xp}")
|
||||
approval_required = self.tab.eles(xp,timeout=5)
|
||||
if len(approval_required) == 0:
|
||||
print(f"【{self.mark_name}】没有需要{filter_type}选项】没有需要{filter_type}的商品了")
|
||||
return sku_ls # "没有需要审批的商品了"
|
||||
else:
|
||||
approval_required = approval_required[0]
|
||||
approval_required.wait.displayed(raise_err=False)
|
||||
approval_required.click()
|
||||
|
||||
approval_required_text = approval_required.text
|
||||
print(f"【{self.mark_name}】已选择筛选条件: {approval_required_text}")
|
||||
|
||||
count = re.findall(r'\d+', approval_required_text)
|
||||
if count:
|
||||
count = int(count[0])
|
||||
print(f"【{self.mark_name}】待审批的商品数量: {count}")
|
||||
if count <= 0:
|
||||
print(f"【{self.mark_name}】没有需要{filter_type}的商品了")
|
||||
return sku_ls #"没有需要审批的商品了"
|
||||
for _ in range(3):
|
||||
# 等待加载完成
|
||||
load_ele = self.tab.eles("xpath://div[contains(@class,'Loader-module__loader')]")
|
||||
if len(load_ele) > 0:
|
||||
load_ele[0].wait.deleted(timeout=3, raise_err=False)
|
||||
time.sleep(0.5)
|
||||
|
||||
sku_ls = self.tab.eles("xpath://div[@data-sku]",timeout=3)
|
||||
if len(sku_ls) > 0:
|
||||
break
|
||||
approval_required.click()
|
||||
return sku_ls
|
||||
def reset_already_asin(self):
|
||||
self.already_asin = set()
|
||||
self.log("already_asin 已重置")
|
||||
|
||||
def wait_loaded(self):
|
||||
# 等待加载完成
|
||||
@@ -86,16 +28,19 @@ class AmzoneMatchAction(AmamzonBase):
|
||||
timeout=3)
|
||||
load_ele.wait.deleted(timeout=5, raise_err=False)
|
||||
except Exception as e:
|
||||
print(f"【{self.mark_name}】等待加载中消失出错", e)
|
||||
|
||||
def run_page_action(self):
|
||||
print(f"【{self.mark_name}】开始执行")
|
||||
print(f"【{self.mark_name}】等待加载中消失出错", e)
|
||||
|
||||
def run_page_action(self,skip_asin=[]):
|
||||
self.log(f"==================={self.mark_name}=======================")
|
||||
self.log("开始执行...")
|
||||
num = 0
|
||||
retry_num = 0
|
||||
already_asin = set()
|
||||
get_page_faild = 0
|
||||
|
||||
while retry_num < 3: # 最多重试3次
|
||||
max_retry_num = 5
|
||||
total_page = 0
|
||||
current_page = 0
|
||||
|
||||
while retry_num < max_retry_num:
|
||||
# if num > 3: #测试
|
||||
# return
|
||||
# 等待加载完成
|
||||
@@ -103,6 +48,7 @@ class AmzoneMatchAction(AmamzonBase):
|
||||
load_ele = self.tab.eles("xpath://div[contains(@class,'Loader-module__loader')]",timeout=5)
|
||||
if len(load_ele) > 0:
|
||||
load_ele[0].wait.deleted(timeout=3, raise_err=False)
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
# 获取当前页码
|
||||
@@ -110,39 +56,59 @@ class AmzoneMatchAction(AmamzonBase):
|
||||
page_pamel = self.tab.eles('xpath://kat-pagination',timeout=5)
|
||||
if len(page_pamel) > 0:
|
||||
current_page = page_pamel[0].sr('xpath:.//ul[@class="pages"]//li[@aria-current="true"]').text
|
||||
|
||||
current_page = int(current_page.strip())
|
||||
# 测试
|
||||
# if current_page > 1:
|
||||
# break
|
||||
# 总页数
|
||||
total_page = page_pamel[0].sr.eles('xpath:.//ul[@class="pages"]//span[@class="page__inner"][last()]')
|
||||
if len(total_page) > 0:
|
||||
total_page = total_page[-1].text
|
||||
else:
|
||||
total_page = 0
|
||||
print(f"【{self.mark_name}】当前页码: {current_page} / 总页数: {total_page}")
|
||||
get_page_faild = 0
|
||||
total_page = int(total_page[-1].text.strip())
|
||||
self.log(f"【{self.mark_name}】当前页码: {current_page} / 总页数: {total_page}")
|
||||
except Exception as e:
|
||||
print(f"【{self.mark_name}】获取页码失败", e)
|
||||
get_page_faild+= 1
|
||||
if get_page_faild > 2:
|
||||
show_notification(f"【{self.mark_name}】获取页码失败超3次停止任务!")
|
||||
break
|
||||
self.log(f"【{self.mark_name}】获取页码失败", e)
|
||||
|
||||
# 保存当前URL,用于失败重试时访问
|
||||
try:
|
||||
self.url = self.tab.url
|
||||
self.log(f"获取当前的链接:{self.url}")
|
||||
except Exception as e:
|
||||
self.url = None
|
||||
self.log(f"获取当前的链接失败:{e}")
|
||||
|
||||
|
||||
sku_ls = self.tab.eles("xpath://div[@data-sku]",timeout=10)
|
||||
print(f"【{self.mark_name}】获取到 {len(sku_ls)}")
|
||||
self.log(f"【{self.mark_name}】获取到 {len(sku_ls)}")
|
||||
if len(sku_ls) == 0:
|
||||
retry_num += 1
|
||||
self.log(f"没有获取到SKU列表,重试{retry_num}/{max_retry_num}")
|
||||
self.tab.refresh()
|
||||
self.tab.wait.doc_loaded(raise_err=False, timeout=120)
|
||||
if retry_num == max_retry_num:
|
||||
raise RuntimeError(f"与页面的连接已断开,没有获取到SKU列表,重试{retry_num}/{max_retry_num}")
|
||||
continue
|
||||
|
||||
# for sku_ele in sku_ls[0:2]:
|
||||
for sku_ele in sku_ls:
|
||||
# solve_problem = sku_ele.eles('xpath:.//kat-link[@label="解决商品信息问题"]')
|
||||
|
||||
|
||||
asin = sku_ele.ele('xpath:.//div[contains(@class,"JanusSplitBox-module__container")]//div[contains(@class,"JanusSplitBox-module__panel--") and contains(string(.),"ASIN")]/..//div[last()]',timeout=3).text
|
||||
print(f"【{self.mark_name}】ASIN {asin} 找到....")
|
||||
if asin in already_asin:
|
||||
print(f"【{self.mark_name}】{asin} 已经处理过了,跳过")
|
||||
self.log(f"ASIN {asin} 找到....")
|
||||
if asin in skip_asin:
|
||||
self.log(f"ASIN {asin} 跳过....")
|
||||
yield (asin,"有最低价跳过")
|
||||
self.already_asin.add(asin)
|
||||
continue
|
||||
|
||||
if asin in self.already_asin:
|
||||
self.log(f"【{self.mark_name}】{asin} 已经处理过了,跳过")
|
||||
continue
|
||||
|
||||
price_match = sku_ele.eles('xpath:.//div[@data-test-id="FeaturedOfferPrice"]//a[text()="匹配"]',timeout=2)
|
||||
if len(price_match) == 0:
|
||||
print(f"【{self.mark_name}】{asin},没有推荐价格匹配按钮")
|
||||
self.log(f"【{self.mark_name}】{asin},没有推荐价格匹配按钮")
|
||||
yield (asin,"无需处理")
|
||||
already_asin.add(asin)
|
||||
self.already_asin.add(asin)
|
||||
continue
|
||||
try:
|
||||
price_match[0].click()
|
||||
@@ -154,14 +120,14 @@ class AmzoneMatchAction(AmamzonBase):
|
||||
save_all_btn.wait.deleted(timeout=10, raise_err=False)
|
||||
|
||||
yield (asin,"处理完成")
|
||||
already_asin.add(asin)
|
||||
self.already_asin.add(asin)
|
||||
except Exception as e:
|
||||
print(f"{asin} 处理失败", e)
|
||||
yield (asin,"处理失败")
|
||||
already_asin.add(asin)
|
||||
self.already_asin.add(asin)
|
||||
continue
|
||||
|
||||
|
||||
|
||||
# 判断是否存在需要翻页的情况
|
||||
page_pamel = self.tab.eles('xpath://kat-pagination',timeout=5)
|
||||
if len(page_pamel) == 0:
|
||||
@@ -172,56 +138,27 @@ class AmzoneMatchAction(AmamzonBase):
|
||||
break
|
||||
next_page_btn.click()
|
||||
num += 1
|
||||
print(f"【{self.mark_name}】【程序计算】正在翻页,已翻 {num} 页...")
|
||||
|
||||
already_asin = set()
|
||||
self.log(f"【{self.mark_name}】【程序计算】正在翻页,已翻 {num} 页...")
|
||||
|
||||
except Exception as e:
|
||||
print(f"【{self.mark_name}】处理匹配操作异常", e)
|
||||
self.log(f"【{self.mark_name}】处理匹配操作异常", e)
|
||||
traceback.print_exc()
|
||||
retry_num += 1
|
||||
self.tab.refresh()
|
||||
self.tab.wait.doc_loaded(raise_err=False,timeout=120)
|
||||
|
||||
|
||||
#检查页数是否相等,不相等则继续
|
||||
self.log(f"开始检查页数,当前页数 {current_page} / {total_page}")
|
||||
if total_page != 0 and current_page!= 0 and current_page < total_page:
|
||||
self.log(f"检查到页数还未完成,重启浏览器继续")
|
||||
raise RuntimeError(f"与页面的连接已断开,检查到页数还未完成,重启浏览器继续")
|
||||
|
||||
class MatchTak:
|
||||
|
||||
country_info = {
|
||||
"DE": "德国",
|
||||
"FR": "法国",
|
||||
"ES": "西班牙",
|
||||
"IT": "意大利",
|
||||
"UK": "英国"
|
||||
}
|
||||
|
||||
|
||||
def __init__(self, user_info: dict = None):
|
||||
"""初始化审批任务处理器
|
||||
|
||||
Args:
|
||||
user_info: 用户信息字典,包含 company, username, password
|
||||
"""
|
||||
self.user_info = user_info or {}
|
||||
self.running = True
|
||||
|
||||
def log(self, message: str, level: str = "INFO"):
|
||||
"""日志输出
|
||||
|
||||
Args:
|
||||
message: 日志消息
|
||||
level: 日志级别
|
||||
"""
|
||||
from datetime import datetime
|
||||
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
if level == "ERROR":
|
||||
show_notification(message, "error")
|
||||
print(f"[{timestamp}] [MatchTak] [{level}] {message}")
|
||||
|
||||
class MatchTak(TaskBase):
|
||||
task_name = "匹配价格-TASK"
|
||||
|
||||
def process_task(self, task_data: dict):
|
||||
"""处理审批任务主入口
|
||||
|
||||
Args:
|
||||
task_data: 任务数据
|
||||
"""
|
||||
@@ -238,21 +175,12 @@ class MatchTak:
|
||||
# 用于测试
|
||||
limit = data.get("limit",None)
|
||||
|
||||
if not task_id:
|
||||
self.log("任务ID为空,跳过", "WARNING")
|
||||
if not task_id or not items or not country_codes:
|
||||
self.log(f"任务ID为空{task_id}/店铺列表为空{items}/国家列表为空{country_codes},跳过", "WARNING")
|
||||
return
|
||||
|
||||
if not items:
|
||||
self.log("店铺列表为空,跳过", "WARNING")
|
||||
return
|
||||
|
||||
if not country_codes:
|
||||
self.log("国家列表为空,跳过", "WARNING")
|
||||
return
|
||||
|
||||
|
||||
self.log(f"开始处理审批任务 {task_id},共 {len(items)} 个店铺,{len(country_codes)} 个国家")
|
||||
|
||||
from config import runing_task
|
||||
runing_task[task_id] = {
|
||||
"status": "running",
|
||||
"start_time": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
|
||||
@@ -287,7 +215,6 @@ class MatchTak:
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["processed_shops"] += 1
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"处理店铺 {shop_name} 失败: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
|
||||
@@ -301,98 +228,13 @@ class MatchTak:
|
||||
self.log(f"任务 {task_id} 处理完成!")
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"任务处理失败: {traceback.format_exc()}", "ERROR")
|
||||
if task_id:
|
||||
from config import runing_task
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["status"] = "failed"
|
||||
runing_task[task_id]["error"] = str(e)
|
||||
|
||||
def open_shop(self,max_retries,company_name,shop_name,iskill=False):
|
||||
error_info = ""
|
||||
driver = None
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
self.log(f"尝试打开店铺 {shop_name} (第 {retry + 1}/{max_retries} 次)")
|
||||
|
||||
if iskill:
|
||||
self.log("重试前先杀掉浏览器进程...")
|
||||
kill_process("v6")
|
||||
kill_process("v5")
|
||||
time.sleep(2)
|
||||
|
||||
# 组装用户信息并创建驱动
|
||||
user_info = {
|
||||
**self.user_info,
|
||||
"company": company_name
|
||||
}
|
||||
driver = AmzoneMatchAction(user_info)
|
||||
browser = driver.open_shop(shop_name)
|
||||
|
||||
if browser and browser != "店铺不存在":
|
||||
self.log(f"成功打开店铺 {shop_name}")
|
||||
else:
|
||||
self.log(f"打开店铺失败: {browser}", "WARNING")
|
||||
driver = None
|
||||
continue
|
||||
|
||||
# 判断是否需要登录
|
||||
need_login = driver.need_login()
|
||||
print("【是否需要登录】:", need_login)
|
||||
if need_login:
|
||||
self.log(f"店铺 {shop_name} 需要登录,正在登录...")
|
||||
# 获取店铺凭证
|
||||
response = get_shop_info(shop_name)
|
||||
print("【获取店铺凭证返回】:", response.text)
|
||||
shop_data = response.json()
|
||||
if not shop_data:
|
||||
mes = f"获取店铺凭证失败,响应数据: {shop_data.get('message', '未知错误')}"
|
||||
self.log(mes, "ERROR")
|
||||
show_notification(mes, "ERROR")
|
||||
continue
|
||||
|
||||
password = shop_data["data"]["password"]
|
||||
|
||||
login_success = driver.login(password)
|
||||
if login_success:
|
||||
self.log(f"店铺 {shop_name} 登录成功,正在重新打开店铺...")
|
||||
browser = driver.open_shop(shop_name)
|
||||
if browser and browser != "店铺不存在":
|
||||
self.log(f"成功打开店铺 {shop_name} 登录后")
|
||||
break
|
||||
else:
|
||||
self.log(f"登录后打开店铺失败: {browser}", "WARNING")
|
||||
driver = None
|
||||
else:
|
||||
self.log(f"店铺 {shop_name} 登录失败", "WARNING")
|
||||
driver = None
|
||||
else:
|
||||
break
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"打开店铺异常: {traceback.format_exc()}", "INFO")
|
||||
driver = None
|
||||
error_info = str(e)
|
||||
time.sleep(10)
|
||||
|
||||
# 如果还有重试机会,等待后继续
|
||||
if retry < max_retries - 1:
|
||||
time.sleep(3)
|
||||
|
||||
# 检查是否成功打开
|
||||
if not driver or not browser or browser == "店铺不存在":
|
||||
error_msg = f"店铺 {shop_name} 打开失败,已重试 {max_retries} 次,跳过该店铺,{error_info}"
|
||||
self.log(error_msg, "ERROR")
|
||||
# 从执行列表中移除
|
||||
if shop_name in runing_shop:
|
||||
del runing_shop[shop_name]
|
||||
return driver
|
||||
return driver
|
||||
|
||||
# def process_shop(self, shop_item: dict, country_codes: list, task_id: int, risk_listing_filter: str):
|
||||
def process_shop(self, shop_item: dict, country_codes: list, task_id: int, risk_listing_filter: str,
|
||||
def process_shop(self, shop_item: dict, country_codes: list, task_id: int, risk_listing_filter: str,
|
||||
user_id=None, stage_index=None, final_stage: bool = True,limit:str=None):
|
||||
"""处理单个店铺
|
||||
|
||||
@@ -405,10 +247,12 @@ class MatchTak:
|
||||
shop_name = shop_item.get("shopName", "未知店铺")
|
||||
company_name = shop_item.get("companyName", "")
|
||||
|
||||
skip_asins_by_country = shop_item.get("skipAsinsByCountry",{})
|
||||
skipAsinDetailsByCountry = shop_item.get("skipAsinDetailsByCountry",{})
|
||||
|
||||
if not company_name:
|
||||
self.log(f"店铺 {shop_name} 的公司名称为空,跳过", "WARNING")
|
||||
return
|
||||
|
||||
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["current_shop"] = shop_name
|
||||
@@ -430,8 +274,13 @@ class MatchTak:
|
||||
self.log(f"检测到任务 {task_id} 的暂停请求,停止处理国家", "WARNING")
|
||||
break
|
||||
|
||||
skip_asin = skip_asins_by_country.get(country_code,[])
|
||||
if skipAsinDetailsByCountry :
|
||||
skipAsinDetails = {i.get("asin"):i.get("minimumPrice") for i in skipAsinDetailsByCountry.get(country_code)}
|
||||
else:
|
||||
skipAsinDetails = {}
|
||||
# 打开店铺
|
||||
driver = self.open_shop(max_retries=max_retries, company_name=company_name,
|
||||
driver = self.open_shop(cls=AmzoneMatchAction,max_retries=max_retries, company_name=company_name,
|
||||
shop_name=shop_name, iskill=iskill)
|
||||
if driver is None:
|
||||
self.log(f"任务 {task_id} 启动店铺失败,结束任务", "ERROR")
|
||||
@@ -439,18 +288,26 @@ class MatchTak:
|
||||
self.post_result(task_id, shop_name, country_code, "", "", is_done=True)
|
||||
return
|
||||
|
||||
try:
|
||||
self.process_country(driver, country_code, task_id, shop_name,risk_listing_filter,limit)
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"处理国家 {country_code} 失败: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
if "与页面的连接已断开" in str(e):
|
||||
iskill = True
|
||||
current_url = None
|
||||
max_retries = 200
|
||||
for _ in range(max_retries):
|
||||
try:
|
||||
self.process_country(driver, country_code, task_id, shop_name,risk_listing_filter,limit,
|
||||
target_url=current_url,skip_asin=skip_asin,skipAsinDetails=skipAsinDetails)
|
||||
driver.reset_already_asin()
|
||||
driver.close_store()
|
||||
break
|
||||
except Exception as e:
|
||||
self.log(f"处理国家 {country_code} 失败: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
if "与页面的连接已断开" in str(e):
|
||||
iskill = True
|
||||
current_url = driver.url
|
||||
|
||||
# 更新已处理国家数
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["processed_countries"] += 1
|
||||
|
||||
self.log(f"{task_id}任务处理完成")
|
||||
# 最后回传,标记完成
|
||||
try:
|
||||
@@ -467,19 +324,12 @@ class MatchTak:
|
||||
if shop_name in runing_shop:
|
||||
del runing_shop[shop_name]
|
||||
self.log(f"店铺 {shop_name} 已从执行列表中移除")
|
||||
|
||||
def process_country(self, driver: AmzoneMatchAction, country_code: str, task_id: int, shop_name: str, risk_listing_filter: str,limit:str=None):
|
||||
|
||||
def process_country(self, driver, country_code, task_id, shop_name, risk_listing_filter,limit=None,target_url=None,
|
||||
skip_asin=[],skipAsinDetails={}):
|
||||
"""处理单个国家的审批任务
|
||||
|
||||
Args:
|
||||
driver: AmzoneApprove驱动实例
|
||||
country_code: 国家代码(如 UK, DE, FR 等)
|
||||
task_id: 任务ID
|
||||
shop_name: 店铺名称
|
||||
risk_listing_filter: 风险商品筛选条件
|
||||
"""
|
||||
from config import runing_task
|
||||
|
||||
|
||||
# 转换国家代码为中文名称
|
||||
country_name = self.country_info.get(country_code, country_code)
|
||||
info_mes = f"开始处理国家: {country_name} ({country_code})"
|
||||
@@ -490,95 +340,35 @@ class MatchTak:
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["current_country"] = country_name
|
||||
|
||||
# 切换国家,最多重试3次
|
||||
max_retries = 3
|
||||
switch_success = False
|
||||
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
self.log(f"尝试切换到国家 {country_name} (第 {retry + 1}/{max_retries} 次)")
|
||||
if retry > 1:
|
||||
# 刷新不行就重新打开店铺
|
||||
self.log("重试前重新打开店铺...")
|
||||
try:
|
||||
driver.close_store()
|
||||
time.sleep(3)
|
||||
driver.open_shop(shop_name)
|
||||
except Exception as e:
|
||||
self.log(f"关闭重新打开店铺: {str(e)}", "WARNING")
|
||||
|
||||
# 如果不是第一次尝试,先刷新页面
|
||||
if retry > 0:
|
||||
self.log("重试前刷新页面...")
|
||||
try:
|
||||
driver.tab.refresh()
|
||||
time.sleep(3)
|
||||
except Exception as e:
|
||||
self.log(f"刷新页面失败: {str(e)}", "WARNING")
|
||||
|
||||
switch_success = driver.SwitchingCountries(country_name)
|
||||
if switch_success:
|
||||
self.log(f"成功切换到国家 {country_name}")
|
||||
break
|
||||
else:
|
||||
self.log(f"切换到国家 {country_name} 失败", "WARNING")
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"切换国家 {country_name} 异常: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
|
||||
# 如果还有重试机会,等待后继续
|
||||
if retry < max_retries - 1:
|
||||
time.sleep(2)
|
||||
|
||||
|
||||
max_retries = 5
|
||||
switch_success, switch_success_pg, search_success, sku_ls = self.action_init(driver, country_name, shop_name,
|
||||
risk_listing_filter)
|
||||
# 如果切换失败,直接返回
|
||||
if not switch_success:
|
||||
error_message = f"切换到国家 {country_name} 失败,已重试 {max_retries} 次,跳过该国家"
|
||||
if not switch_success or not switch_success_pg or not search_success:
|
||||
error_message = f"切换到国家({switch_success})/库存页面({switch_success_pg})/搜索筛选到指定选项({search_success}) {country_name} 失败,已重试 {max_retries} 次,跳过该国家"
|
||||
self.log(error_message, "ERROR")
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["processed_countries"] += 1
|
||||
return
|
||||
|
||||
# 切换到库存管理页面
|
||||
try:
|
||||
driver.SwitchPage()
|
||||
self.log(f"已切换到库存管理页面")
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self.log(f"切换页面失败: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["processed_countries"] += 1
|
||||
return
|
||||
|
||||
# 搜索需要审批的商品,最多重试3次
|
||||
sku_ls = []
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
self.log(f"尝试搜索匹配操作商品 (第 {retry + 1}/{max_retries} 次)")
|
||||
sku_ls = driver.search(filter_type=risk_listing_filter)
|
||||
break
|
||||
except Exception as e:
|
||||
self.log(f"搜索商品异常: {str(e)}", "ERROR")
|
||||
if retry < max_retries - 1:
|
||||
try:
|
||||
driver.tab.refresh()
|
||||
time.sleep(3)
|
||||
except Exception as refresh_error:
|
||||
self.log(f"刷新页面失败: {str(refresh_error)}", "WARNING")
|
||||
|
||||
# 如果没有需要审批的商品,直接返回
|
||||
|
||||
# # 如果没有需要审批的商品,直接返回
|
||||
if len(sku_ls) == 0:
|
||||
self.log(f"国家 {country_name} 没有搜索出的商品")
|
||||
self.log(f"国家 {country_name} 没有需要审批的商品")
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["processed_countries"] += 1
|
||||
return
|
||||
|
||||
|
||||
self.log(f"国家 {country_name} 搜索出 {len(sku_ls)} 商品,开始处理...")
|
||||
|
||||
|
||||
if target_url is not None: # 从失败的链接继续
|
||||
self.log(f"开始访问:{target_url}")
|
||||
driver.tab.get(url=target_url)
|
||||
driver.tab.wait.doc_loaded(timeout=60, raise_err=False)
|
||||
|
||||
result = []
|
||||
# 处理所有需要审批的商品(通过yield获取结果)
|
||||
for asin, status in driver.run_page_action():
|
||||
for asin, status in driver.run_page_action(skip_asin):
|
||||
# 检查是否收到暂停请求
|
||||
if task_id in runing_task and runing_task[task_id].get("stop_requested", False):
|
||||
self.log(f"检测到任务 {task_id} 的暂停请求,停止处理ASIN", "WARNING")
|
||||
@@ -590,22 +380,28 @@ class MatchTak:
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["current_asin"] = asin
|
||||
runing_task[task_id]["processed_asins"] += 1
|
||||
|
||||
runing_task[task_id]["failed_count"] += 1
|
||||
|
||||
# 回传结果到API
|
||||
try:
|
||||
self.post_result(task_id, shop_name, country_code, asin, status)
|
||||
except Exception as e:
|
||||
self.log(f"回传结果失败: {str(e)}", "ERROR")
|
||||
|
||||
minimumPrice = ""
|
||||
if "有最低价跳过" in status:
|
||||
minimumPrice = skipAsinDetails.get(asin)
|
||||
result.append({
|
||||
"asin": asin,
|
||||
"status": status,
|
||||
"minimumPrice": minimumPrice,
|
||||
"done": False
|
||||
})
|
||||
|
||||
if len(result) > 10:
|
||||
self.post_result_batch(task_id, shop_name, country_code,result)
|
||||
result = []
|
||||
|
||||
if len(result) > 0:
|
||||
self.post_result_batch(task_id, shop_name, country_code, result)
|
||||
|
||||
self.log(f"国家 {country_name} 处理完成")
|
||||
|
||||
def post_stage_finished(self, task_id: int, user_id, stage_index):
|
||||
import requests
|
||||
from config import DELETE_BRAND_API_BASE
|
||||
|
||||
if user_id in (None, "", 0):
|
||||
raise ValueError("user_id is required for stage completion callback")
|
||||
@@ -651,9 +447,7 @@ class MatchTak:
|
||||
asin: ASIN
|
||||
status: 处理状态
|
||||
"""
|
||||
import requests
|
||||
from config import DELETE_BRAND_API_BASE
|
||||
|
||||
|
||||
url = f"{DELETE_BRAND_API_BASE}/api/shop-match/tasks/{task_id}/result"
|
||||
|
||||
payload = {
|
||||
@@ -707,7 +501,66 @@ class MatchTak:
|
||||
|
||||
self.log(f"已达到最大重试次数,结果回传最终失败", "ERROR")
|
||||
raise RuntimeError("已达到最大重试次数,结果回传最终失败")
|
||||
|
||||
|
||||
def post_result_batch(self, task_id: int, shop_name: str, country_code: str, country_data:list,
|
||||
is_done: bool = False):
|
||||
"""回传处理结果到API
|
||||
|
||||
Args:
|
||||
task_id: 任务ID
|
||||
shop_name: 店铺名称
|
||||
country_code: 国家代码
|
||||
asin: ASIN
|
||||
status: 处理状态
|
||||
"""
|
||||
|
||||
url = f"{DELETE_BRAND_API_BASE}/api/shop-match/tasks/{task_id}/result"
|
||||
|
||||
payload = {
|
||||
"shops": [
|
||||
{
|
||||
"error": "",
|
||||
"countries": {
|
||||
country_code: country_data
|
||||
},
|
||||
"shopName": shop_name
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
max_retries = 3
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
print("================【匹配】=====================")
|
||||
self.log(f"尝试回传结果 (第 {retry + 1}/{max_retries} 次)")
|
||||
self.log(f"回传URL: {url}")
|
||||
self.log(f"回传数据: {payload}")
|
||||
response = requests.post(
|
||||
url,
|
||||
json=payload,
|
||||
headers={"Content-Type": "application/json"},
|
||||
timeout=30,
|
||||
verify=False
|
||||
)
|
||||
self.log(f"回传结果: {response.text}")
|
||||
data = response.json() if response.text else {}
|
||||
if response.status_code == 200 and isinstance(data, dict) and data.get("success"):
|
||||
self.log(f"结果回传成功")
|
||||
return
|
||||
else:
|
||||
self.log(f"结果回传失败,状态码: {response.status_code}", "WARNING")
|
||||
print("=====================================")
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"调用API异常: {str(e)}", "ERROR")
|
||||
print("=====================================")
|
||||
|
||||
# 如果还有重试机会,等待后继续
|
||||
if retry < max_retries - 1:
|
||||
time.sleep(2)
|
||||
|
||||
self.log(f"已达到最大重试次数,结果回传最终失败", "ERROR")
|
||||
raise RuntimeError("已达到最大重试次数,结果回传最终失败")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
@@ -0,0 +1,128 @@
|
||||
function textOf(node) {
|
||||
return ((node && (node.innerText || node.textContent || '')) || '').replace(/\s+/g, ' ').trim();
|
||||
}
|
||||
|
||||
function isVisible(el) {
|
||||
if (!el || !el.getBoundingClientRect) return false;
|
||||
const rect = el.getBoundingClientRect();
|
||||
const style = window.getComputedStyle(el);
|
||||
return rect.width > 0 && rect.height > 0 && style.display !== 'none' && style.visibility !== 'hidden';
|
||||
}
|
||||
|
||||
function clickElement(el) {
|
||||
el.scrollIntoView({ block: 'center', inline: 'nearest' });
|
||||
if (el.focus) el.focus();
|
||||
for (const eventName of ['pointerdown', 'mousedown', 'pointerup', 'mouseup']) {
|
||||
el.dispatchEvent(new MouseEvent(eventName, { bubbles: true, cancelable: true, view: window }));
|
||||
}
|
||||
el.click();
|
||||
}
|
||||
|
||||
function summarize(el) {
|
||||
return {
|
||||
tag: el.tagName,
|
||||
text: textOf(el).slice(0, 500),
|
||||
attrs: Array.from(el.attributes || []).reduce((acc, attr) => {
|
||||
acc[attr.name] = attr.value;
|
||||
return acc;
|
||||
}, {}),
|
||||
outerHTML: (el.outerHTML || '').slice(0, 1000),
|
||||
};
|
||||
}
|
||||
|
||||
const view = document.querySelector('[data-testid="complete-drafts-view"]');
|
||||
if (!view) return { ok: false, reason: 'complete drafts view not found' };
|
||||
|
||||
const selectedRows = Array.from(view.querySelectorAll('.ag-center-cols-container [role="row"]'))
|
||||
.filter((row) => isVisible(row) && (row.getAttribute('aria-selected') === 'true' || row.classList.contains('ag-row-selected')));
|
||||
if (!selectedRows.length) return { ok: false, reason: 'no selected complete draft rows' };
|
||||
|
||||
const bulkDropdown = view.querySelector('#bulk-dropdown');
|
||||
if (!bulkDropdown) {
|
||||
return {
|
||||
ok: false,
|
||||
reason: 'bulk dropdown not found',
|
||||
selectedCount: selectedRows.length,
|
||||
};
|
||||
}
|
||||
|
||||
const toggle = bulkDropdown.shadowRoot && bulkDropdown.shadowRoot.querySelector('button[part="dropdown-button-toggle-button"], button');
|
||||
clickElement(toggle || bulkDropdown);
|
||||
|
||||
function findBulkDeleteAction() {
|
||||
const all = [];
|
||||
const seen = new Set();
|
||||
function walk(root) {
|
||||
if (!root || seen.has(root)) return;
|
||||
seen.add(root);
|
||||
const walker = document.createTreeWalker(root, NodeFilter.SHOW_ELEMENT);
|
||||
let node = walker.currentNode;
|
||||
while (node) {
|
||||
if (node.nodeType === Node.ELEMENT_NODE && node.tagName) all.push(node);
|
||||
if (node.shadowRoot) walk(node.shadowRoot);
|
||||
if ((node.tagName || '').toLowerCase() === 'slot' && node.assignedElements) {
|
||||
for (const assigned of node.assignedElements({ flatten: true })) walk(assigned);
|
||||
}
|
||||
node = walker.nextNode();
|
||||
}
|
||||
}
|
||||
walk(document);
|
||||
const exactAction = all.find((el) => {
|
||||
const action = String(el.getAttribute && el.getAttribute('data-action') || '');
|
||||
const text = textOf(el);
|
||||
return isVisible(el) && action === 'DELETE_DRAFTS' && /删除商品信息草稿/.test(text);
|
||||
});
|
||||
if (exactAction) return exactAction;
|
||||
|
||||
return all.find((el) => {
|
||||
const text = textOf(el);
|
||||
const attrs = Array.from(el.attributes || []).map((attr) => `${attr.name}=${attr.value}`).join(' ');
|
||||
const haystack = `${text} ${attrs}`;
|
||||
if (!isVisible(el) || !/删除商品信息草稿/.test(haystack)) return false;
|
||||
if (/^(HTML|BODY|KAT-DATA-GRID)$/i.test(el.tagName || '') && text.length > 80) return false;
|
||||
if (/^DIV$/i.test(el.tagName || '') && text !== '删除商品信息草稿') return false;
|
||||
if ((el.tagName || '').toLowerCase() === 'kat-icon') return false;
|
||||
return /^(BUTTON|KAT-BUTTON|KAT-OPTION|KAT-DROPDOWN-OPTION|LI|A|SPAN|DIV)$/i.test(el.tagName || '');
|
||||
});
|
||||
}
|
||||
|
||||
const deleteAction = findBulkDeleteAction();
|
||||
|
||||
if (!deleteAction) {
|
||||
const candidates = [];
|
||||
const seen = new Set();
|
||||
function walkCandidates(root) {
|
||||
if (!root || seen.has(root)) return;
|
||||
seen.add(root);
|
||||
const walker = document.createTreeWalker(root, NodeFilter.SHOW_ELEMENT);
|
||||
let node = walker.currentNode;
|
||||
while (node) {
|
||||
if (node.nodeType === Node.ELEMENT_NODE && node.tagName && isVisible(node)) {
|
||||
const text = textOf(node);
|
||||
const attrs = Array.from(node.attributes || []).map((attr) => `${attr.name}=${attr.value}`).join(' ');
|
||||
if (/删除|草稿|选择组操作|menu|option/i.test(`${text} ${attrs}`)) candidates.push(summarize(node));
|
||||
}
|
||||
if (node.shadowRoot) walkCandidates(node.shadowRoot);
|
||||
node = walker.nextNode();
|
||||
}
|
||||
}
|
||||
walkCandidates(document);
|
||||
return {
|
||||
ok: false,
|
||||
reason: 'bulk delete draft action not found',
|
||||
menuOpened: true,
|
||||
selectedCount: selectedRows.length,
|
||||
bulkDropdown: summarize(bulkDropdown),
|
||||
candidates: candidates.slice(0, 50),
|
||||
};
|
||||
}
|
||||
|
||||
clickElement(deleteAction);
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
selectedCount: selectedRows.length,
|
||||
bulkDropdown: summarize(bulkDropdown),
|
||||
clicked: summarize(deleteAction),
|
||||
rows: selectedRows.slice(0, 10).map(summarize),
|
||||
};
|
||||
100
app/amazon/scripts/patrol_delete/complete_draft_select_all.js
Normal file
100
app/amazon/scripts/patrol_delete/complete_draft_select_all.js
Normal file
@@ -0,0 +1,100 @@
|
||||
function textOf(node) {
|
||||
return ((node && (node.innerText || node.textContent || '')) || '').replace(/\s+/g, ' ').trim();
|
||||
}
|
||||
|
||||
function isVisible(el) {
|
||||
if (!el || !el.getBoundingClientRect) return false;
|
||||
const rect = el.getBoundingClientRect();
|
||||
const style = window.getComputedStyle(el);
|
||||
return rect.width > 0 && rect.height > 0 && style.display !== 'none' && style.visibility !== 'hidden';
|
||||
}
|
||||
|
||||
function clickElement(el) {
|
||||
el.scrollIntoView({ block: 'center', inline: 'nearest' });
|
||||
if (el.focus) el.focus();
|
||||
for (const eventName of ['pointerdown', 'mousedown', 'pointerup', 'mouseup']) {
|
||||
el.dispatchEvent(new MouseEvent(eventName, { bubbles: true, cancelable: true, view: window }));
|
||||
}
|
||||
el.click();
|
||||
}
|
||||
|
||||
function summarizeRow(row) {
|
||||
return {
|
||||
rowId: row.getAttribute('row-id') || '',
|
||||
text: textOf(row).slice(0, 500),
|
||||
selected: row.getAttribute('aria-selected') === 'true' || row.classList.contains('ag-row-selected'),
|
||||
};
|
||||
}
|
||||
|
||||
const payload = arguments[0] || {};
|
||||
const view = document.querySelector('[data-testid="complete-drafts-view"]');
|
||||
if (!view) return { ok: false, reason: 'complete drafts view not found' };
|
||||
|
||||
const rows = Array.from(view.querySelectorAll('.ag-center-cols-container [role="row"]'))
|
||||
.filter((row) => isVisible(row));
|
||||
const rowCount = rows.length;
|
||||
const selectedBefore = rows.filter((row) => row.getAttribute('aria-selected') === 'true' || row.classList.contains('ag-row-selected')).length;
|
||||
const maxSelectCountRaw = Number(payload.maxSelectCount || 0);
|
||||
const maxSelectCount = Number.isFinite(maxSelectCountRaw) && maxSelectCountRaw > 0 ? Math.floor(maxSelectCountRaw) : 0;
|
||||
|
||||
if (payload.probeOnly) {
|
||||
return {
|
||||
ok: true,
|
||||
probeOnly: true,
|
||||
rowCount,
|
||||
selectedCount: selectedBefore,
|
||||
rows: rows.slice(0, 10).map(summarizeRow),
|
||||
};
|
||||
}
|
||||
|
||||
if (rowCount <= 0) {
|
||||
return { ok: false, reason: 'no complete draft rows', rowCount, selectedCount: selectedBefore };
|
||||
}
|
||||
|
||||
const headerCheckbox = view.querySelector('.ag-header-select-all input[type="checkbox"]');
|
||||
if (!headerCheckbox) {
|
||||
return { ok: false, reason: 'complete draft select-all checkbox not found', rowCount, selectedCount: selectedBefore };
|
||||
}
|
||||
|
||||
if (maxSelectCount > 0 && maxSelectCount < rowCount) {
|
||||
if (headerCheckbox.checked) {
|
||||
clickElement(headerCheckbox);
|
||||
}
|
||||
for (const row of rows) {
|
||||
if (row.getAttribute('aria-selected') === 'true' || row.classList.contains('ag-row-selected')) {
|
||||
const checkbox = row.querySelector('input[type="checkbox"]');
|
||||
if (checkbox && checkbox.checked) clickElement(checkbox);
|
||||
}
|
||||
}
|
||||
for (const row of rows.slice(0, maxSelectCount)) {
|
||||
const checkbox = row.querySelector('input[type="checkbox"]');
|
||||
if (checkbox && !checkbox.checked) {
|
||||
clickElement(checkbox);
|
||||
} else if (!checkbox) {
|
||||
clickElement(row);
|
||||
}
|
||||
}
|
||||
} else if (!headerCheckbox.checked || selectedBefore < rowCount) {
|
||||
clickElement(headerCheckbox);
|
||||
}
|
||||
|
||||
let selectedRows = Array.from(view.querySelectorAll('.ag-center-cols-container [role="row"]'))
|
||||
.filter((row) => isVisible(row) && (row.getAttribute('aria-selected') === 'true' || row.classList.contains('ag-row-selected')));
|
||||
|
||||
if (selectedRows.length <= 0) {
|
||||
const firstRowCheckbox = rows[0] && rows[0].querySelector('input[type="checkbox"]');
|
||||
if (firstRowCheckbox && !firstRowCheckbox.checked) {
|
||||
clickElement(firstRowCheckbox);
|
||||
}
|
||||
}
|
||||
|
||||
selectedRows = Array.from(view.querySelectorAll('.ag-center-cols-container [role="row"]'))
|
||||
.filter((row) => isVisible(row) && (row.getAttribute('aria-selected') === 'true' || row.classList.contains('ag-row-selected')));
|
||||
|
||||
return {
|
||||
ok: selectedRows.length > 0,
|
||||
rowCount,
|
||||
selectedCount: selectedRows.length,
|
||||
headerChecked: headerCheckbox.checked,
|
||||
rows: selectedRows.slice(0, 10).map(summarizeRow),
|
||||
};
|
||||
@@ -0,0 +1,59 @@
|
||||
function textOf(node) {
|
||||
return ((node && (node.innerText || node.textContent || '')) || '').replace(/\s+/g, ' ').trim();
|
||||
}
|
||||
|
||||
function isVisible(el) {
|
||||
if (!el || !el.getBoundingClientRect) return false;
|
||||
const rect = el.getBoundingClientRect();
|
||||
const style = window.getComputedStyle(el);
|
||||
return rect.width > 0 && rect.height > 0 && style.display !== 'none' && style.visibility !== 'hidden';
|
||||
}
|
||||
|
||||
function clickElement(el) {
|
||||
el.scrollIntoView({ block: 'center', inline: 'nearest' });
|
||||
if (el.focus) el.focus();
|
||||
for (const eventName of ['pointerdown', 'mousedown', 'pointerup', 'mouseup']) {
|
||||
el.dispatchEvent(new MouseEvent(eventName, { bubbles: true, cancelable: true, view: window }));
|
||||
}
|
||||
el.click();
|
||||
}
|
||||
|
||||
function summarize(el) {
|
||||
return {
|
||||
tag: el.tagName,
|
||||
text: textOf(el),
|
||||
className: el.className || '',
|
||||
outerHTML: (el.outerHTML || '').slice(0, 1000),
|
||||
};
|
||||
}
|
||||
|
||||
const targetTag = String((arguments[0] && arguments[0].tag) || '').replace(/\s+/g, ' ').trim();
|
||||
if (!targetTag) return { ok: false, reason: 'target tag empty' };
|
||||
|
||||
const view = document.querySelector('[data-testid="complete-drafts-view"]');
|
||||
if (!view) return { ok: false, reason: 'complete drafts view not found' };
|
||||
|
||||
const chips = Array.from(view.querySelectorAll('.JanusChip-module__janusChipContainer--gRjdp'));
|
||||
const chip = chips.find((el) => {
|
||||
const text = textOf(el);
|
||||
return text === targetTag || text.startsWith(`${targetTag} (`) || text.startsWith(`${targetTag}(`);
|
||||
});
|
||||
|
||||
if (!chip) {
|
||||
return {
|
||||
ok: false,
|
||||
reason: 'complete draft chip not found',
|
||||
tag: targetTag,
|
||||
chips: chips.map(summarize),
|
||||
};
|
||||
}
|
||||
|
||||
if (!/active/.test(String(chip.className || ''))) {
|
||||
clickElement(chip);
|
||||
}
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
tag: targetTag,
|
||||
clicked: summarize(chip),
|
||||
};
|
||||
941
app/amazon/similar_asin.py
Normal file
941
app/amazon/similar_asin.py
Normal file
@@ -0,0 +1,941 @@
|
||||
# encoding utf-8
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
import base64
|
||||
import time
|
||||
import re
|
||||
import traceback
|
||||
from datetime import datetime
|
||||
from collections import defaultdict
|
||||
import requests
|
||||
from urllib.parse import quote
|
||||
import os
|
||||
from curl_cffi import requests as requests_frp
|
||||
|
||||
from config import base_dir
|
||||
|
||||
|
||||
from amazon.tool import show_notification,get_shop_info,remove_special_characters,split_currency_values
|
||||
from amazon.amazon_base import TaskBase
|
||||
|
||||
from amazon.chrome_base import ChromeAmzoneBase
|
||||
|
||||
from config import runing_task, runing_shop,base_dir,DELETE_BRAND_API_BASE
|
||||
|
||||
try:
|
||||
from config import proxy_url as CONFIG_PROXY_URL, proxy_mode as CONFIG_PROXY_MODE
|
||||
except ImportError:
|
||||
CONFIG_PROXY_URL = None
|
||||
CONFIG_PROXY_MODE = 1
|
||||
|
||||
# Forbidden 后 3 分钟内统一使用代理:记录代理生效截止时间与当前代理
|
||||
_FORBIDDEN_PROXY_UNTIL_Similar = 0.0
|
||||
_FORBIDDEN_PROXY_DICT_Similar = None
|
||||
_FORBIDDEN_PROXY_MINUTES_Similar = 0.5
|
||||
|
||||
|
||||
class ChromeAmzone(ChromeAmzoneBase):
|
||||
mark_name = "亚马逊相似ASIN"
|
||||
|
||||
def wait_load_complete(self):
|
||||
try:
|
||||
for _ in range(3):
|
||||
content_area = self.tab.eles('xpath://div[@class="ap-sbi-aside__main"]', timeout=20)
|
||||
if len(content_area) > 0:
|
||||
pass
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"等待加载完成失败,{e},等待5秒")
|
||||
time.sleep(5)
|
||||
|
||||
def encode_custom(self,input_):
|
||||
if not input_ or not isinstance(input_, str):
|
||||
return ""
|
||||
|
||||
transformed = ""
|
||||
|
||||
# 第一个字符:字符编码 + 字符串长度
|
||||
transformed += chr((ord(input_[0]) + len(input_)) & 0xFFFF)
|
||||
|
||||
# 后续字符:当前字符编码 + 前一个字符编码
|
||||
for i in range(1, len(input_)):
|
||||
current_code = ord(input_[i])
|
||||
prev_code = ord(input_[i - 1])
|
||||
transformed += chr((current_code + prev_code) & 0xFFFF)
|
||||
|
||||
# 对应 JavaScript 的 encodeURIComponent
|
||||
encoded = quote(transformed, safe="-_.!~*'()")
|
||||
|
||||
# 对应 replace(/[!'()*]/g, ...)
|
||||
for ch in ["!", "'", "(", ")", "*"]:
|
||||
encoded = encoded.replace(ch, "%" + format(ord(ch), "x"))
|
||||
return encoded
|
||||
|
||||
def image_to_base64(self,image_source: str) -> str:
|
||||
"""
|
||||
将图片链接(URL 或本地文件路径)转换为 Base64 编码的字符串。
|
||||
"""
|
||||
if image_source.startswith(('http://', 'https://')):
|
||||
response = requests.get(image_source, timeout=10,headers={
|
||||
"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36 Edg/142.0.0.0 18444"
|
||||
})
|
||||
response.raise_for_status() # 非 2xx 状态码将抛出异常
|
||||
image_data = response.content
|
||||
else:
|
||||
path = Path(image_source)
|
||||
if not path.is_file():
|
||||
raise ValueError(f"本地文件不存在: {image_source}")
|
||||
with open(path, 'rb') as f:
|
||||
image_data = f.read()
|
||||
base64_str = base64.b64encode(image_data).decode('utf-8')
|
||||
return base64_str
|
||||
|
||||
def get_aliprice_data(self,page=1,title="",domain="",category="",imageBase64=""):
|
||||
"""
|
||||
"fishkeeper Quick Aquarium Siphon Pump Gravel Cleaner - 256GPH Adjustable Powerful Fish Tank Vacuum Gravel Cleaning Kit for Aquarium Water Changer, Sand Cleaner, Dirt Removal : Amazon.co.uk: Pet Supplies"
|
||||
标题 :
|
||||
|
||||
"""
|
||||
global _FORBIDDEN_PROXY_UNTIL_Similar, _FORBIDDEN_PROXY_DICT_Similar
|
||||
|
||||
headers = {
|
||||
"accept": "application/json, text/plain, */*",
|
||||
"accept-language": "zh-CN,zh;q=0.9,en;q=0.8",
|
||||
"browser": "chrome",
|
||||
"cache-control": "no-cache",
|
||||
"channel": "chrome_offline",
|
||||
"content-type": "application/json;charset=UTF-8",
|
||||
"ext-id": "10100",
|
||||
"ext_id": "10100",
|
||||
"origin": "chrome-extension://ephkdklmdkaakeleplfpahjphaokcllh",
|
||||
"platform": "1688",
|
||||
"pragma": "no-cache",
|
||||
"priority": "u=1, i",
|
||||
"sec-fetch-dest": "empty",
|
||||
"sec-fetch-mode": "cors",
|
||||
"sec-fetch-site": "none",
|
||||
"sec-fetch-storage-access": "active",
|
||||
"version": "3.7.4",
|
||||
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36"
|
||||
}
|
||||
cookie = {
|
||||
"e-info": "[{\"e-name\":\"1688\",\"adid\":\"100\",\"version\":\"3.7.4\",\"ext_id\":\"10100\"}]",
|
||||
"language": "chinese",
|
||||
"province_code": "Guangdong",
|
||||
"_ct": "czg",
|
||||
"_e_ct": "czg",
|
||||
"is_reto": "1",
|
||||
"crossborder": "1",
|
||||
"agent": "0",
|
||||
"first_view": "1",
|
||||
"currency": "USD",
|
||||
"is_coo": "1",
|
||||
"plugin": "chrome",
|
||||
"plugin_ext": "100,129",
|
||||
"m-info": "[{\"platform\":\"1688\",\"version\":\"3.7.4\",\"browser\":\"chrome\",\"m\":\"uni\",\"t\":1777902599578}]"
|
||||
}
|
||||
url = "https://api.aliprice.com/index.php/chrome/items/imageAnalysis"
|
||||
itemTitle = f"{title} : {domain}: {category}"
|
||||
|
||||
params = {"page": page, "size": 20, "website": "1688_lite2", "language": "zh-CN", "currency": "USD", "from": "",
|
||||
"itemTitle": itemTitle, "domain": domain }
|
||||
params['sign'] = self.encode_custom(json.dumps(params, ensure_ascii=False))
|
||||
data = {
|
||||
"imageBase64" : imageBase64
|
||||
}
|
||||
|
||||
proxies = None
|
||||
|
||||
# 若 3 分钟内曾出现 Forbidden,则直接使用当时保存的代理
|
||||
now = time.time()
|
||||
if now < _FORBIDDEN_PROXY_UNTIL_Similar and _FORBIDDEN_PROXY_DICT_Similar:
|
||||
proxies = _FORBIDDEN_PROXY_DICT_Similar
|
||||
print("处于 Forbidden 代理窗口内,直接使用代理", proxies)
|
||||
|
||||
# 发送第一次请求
|
||||
try:
|
||||
response = requests_frp.post(url, headers=headers, params=params, json=data, impersonate="chrome101",
|
||||
cookies=cookie, proxies=proxies,verify=False)
|
||||
response.encoding = "utf-8"
|
||||
|
||||
# 检查响应状态码和数据有效性
|
||||
need_retry = False
|
||||
if response.status_code != 200:
|
||||
print(f"请求失败,状态码: {response.status_code}")
|
||||
need_retry = True
|
||||
else:
|
||||
try:
|
||||
result_data = response.json()
|
||||
# 检查是否获取到有效数据
|
||||
if not result_data or not isinstance(result_data, dict):
|
||||
print("返回数据为空或格式不正确")
|
||||
need_retry = True
|
||||
elif "Forbidden" in response.text or "Too Many Requests" in response.text:
|
||||
print("返回Forbidden或Too Many Requests")
|
||||
need_retry = True
|
||||
except Exception as e:
|
||||
print(f"解析响应JSON失败: {e}")
|
||||
need_retry = True
|
||||
|
||||
# 如果需要重试且配置了代理URL
|
||||
# need_retry = True
|
||||
if need_retry and CONFIG_PROXY_URL and not proxies:
|
||||
try:
|
||||
proxy_resp = requests.get(CONFIG_PROXY_URL, timeout=10)
|
||||
print("代理请求结果->:", proxy_resp.text)
|
||||
|
||||
# 模式 2:账号密码代理,接口返回 JSON
|
||||
if CONFIG_PROXY_MODE == 2:
|
||||
resp_json = proxy_resp.json()
|
||||
proxy_list = resp_json.get("data", {}).get("list") or []
|
||||
first_item = proxy_list[0] if proxy_list else None
|
||||
if first_item:
|
||||
ip = first_item.get("ip")
|
||||
port = first_item.get("port")
|
||||
account = first_item.get("account")
|
||||
password = first_item.get("password")
|
||||
if ip and port and account and password:
|
||||
auth_proxy = f"{account}:{password}@{ip}:{port}"
|
||||
print("获取到账号密码代理:", auth_proxy)
|
||||
proxies = {
|
||||
"http": f"http://{auth_proxy}",
|
||||
"https": f"http://{auth_proxy}",
|
||||
}
|
||||
# 默认模式 1:普通 IP:port 文本
|
||||
if CONFIG_PROXY_MODE != 2:
|
||||
proxy_ip = (proxy_resp.text or "").strip()
|
||||
if proxy_ip:
|
||||
proxies = {
|
||||
"http": f"http://{proxy_ip}",
|
||||
"https": f"https://{proxy_ip}",
|
||||
}
|
||||
|
||||
if proxies:
|
||||
print("使用代理重试请求")
|
||||
response = requests_frp.post(url, headers=headers, params=params, json=data,
|
||||
impersonate="chrome101", cookies=cookie, proxies=proxies)
|
||||
response.encoding = "utf-8"
|
||||
print("代理重试结果状态码:", response.status_code)
|
||||
|
||||
# 记录 3 分钟内都使用该代理
|
||||
_FORBIDDEN_PROXY_UNTIL_Similar = now + _FORBIDDEN_PROXY_MINUTES_Similar * 60
|
||||
_FORBIDDEN_PROXY_DICT_Similar = proxies
|
||||
|
||||
except Exception as e:
|
||||
print("获取代理或重试失败:", e)
|
||||
|
||||
# 返回最终结果
|
||||
result_data = response.json()
|
||||
return result_data
|
||||
|
||||
except Exception as e:
|
||||
print(f"get_aliprice_data 请求异常: {e}")
|
||||
return {}
|
||||
|
||||
def _scrape_data(self):
|
||||
"""
|
||||
抓取商品数据
|
||||
|
||||
Returns:
|
||||
dict: 抓取到的数据
|
||||
"""
|
||||
data = {
|
||||
'image_url': "",
|
||||
'title': "",
|
||||
'category' : '',
|
||||
'sku' : '',
|
||||
'success' : True
|
||||
}
|
||||
|
||||
try:
|
||||
# 等待页面加载
|
||||
# time.sleep(3)
|
||||
title_ele = self.tab.ele('xpath://h1[@id="title"]',timeout=30)
|
||||
title = title_ele.text
|
||||
data["title"] = title
|
||||
imge_ele = self.tab.ele('xpath://div[@id="imgTagWrapperId"]//img',timeout=20)
|
||||
|
||||
image_url = ""
|
||||
min_image_url = ""
|
||||
data_a_dynamic_image = imge_ele.attr("data-a-dynamic-image")
|
||||
if data_a_dynamic_image:
|
||||
dynamic_image_json = json.loads(data_a_dynamic_image)
|
||||
self.log(f"图片信息:{dynamic_image_json}")
|
||||
max_area = 0
|
||||
min_area = 0
|
||||
|
||||
for url, (width, height) in dynamic_image_json.items():
|
||||
area = width * height
|
||||
if area > max_area:
|
||||
max_area = area
|
||||
image_url = url
|
||||
if min_area == 0:
|
||||
min_area = area
|
||||
min_image_url = url
|
||||
if area < min_area:
|
||||
min_area = area
|
||||
min_image_url = url
|
||||
|
||||
if not image_url:
|
||||
image_url = imge_ele.attr("src")
|
||||
|
||||
if not min_image_url:
|
||||
min_image_url = imge_ele.attr("src")
|
||||
|
||||
|
||||
data["image_url"] = image_url
|
||||
data["min_image_url"] = min_image_url
|
||||
|
||||
# sku
|
||||
sku_ele_ls = self.tab.eles('xpath://ul[@class="a-unordered-list a-vertical a-spacing-mini"]',timeout=20)
|
||||
if len(sku_ele_ls) > 0:
|
||||
data["sku"] = sku_ele_ls[0].text
|
||||
category = self.tab.eles('xpath://div[@id="wayfinding-breadcrumbs_feature_div"]//span[@class="a-list-item"]//a[@class="a-link-normal a-color-tertiary"]',timeout=20)
|
||||
if len(category) > 0:
|
||||
data["category"] = category[0].text
|
||||
return data
|
||||
|
||||
except Exception as e:
|
||||
print(f"抓取数据时出错: {traceback.format_exc()}")
|
||||
data["success"] = False
|
||||
return data
|
||||
|
||||
def run(self, country, asin,total_page=2):
|
||||
"""
|
||||
运行亚马逊详情采集任务
|
||||
Args:
|
||||
country: 国家名称(如:英国、德国、法国、西班牙、意大利)
|
||||
asin: 亚马逊商品ASIN码
|
||||
Returns:
|
||||
dict: 包含采集到的数据
|
||||
"""
|
||||
return_data = {}
|
||||
try:
|
||||
# 验证国家是否支持
|
||||
if country not in self.country_info:
|
||||
error_msg = f"不支持的国家: {country},支持的国家有: {list(self.country_info.keys())}"
|
||||
print(error_msg)
|
||||
# show_notification(error_msg, "error")
|
||||
return None
|
||||
|
||||
# 获取国家配置
|
||||
country_config = self.country_info[country]
|
||||
zip_code = country_config["zip_code"]
|
||||
mark = country_config.get("mark")
|
||||
|
||||
# 1. 根据国家和ASIN拼接链接
|
||||
base_url = country_config["url"]
|
||||
# 提取域名部分
|
||||
domain = base_url.split("/dp/")[0]
|
||||
# 拼接新的URL
|
||||
product_url = f"{domain}/dp/{asin}"
|
||||
self.log(f"正在访问: {product_url}")
|
||||
|
||||
# 打开链接
|
||||
self.tab.get(product_url)
|
||||
time.sleep(3) # 等待页面初步加载
|
||||
self.tab.wait.doc_loaded(timeout=30, raise_err=False)
|
||||
|
||||
self.close_init_popup()
|
||||
# 2. 切换国家/设置邮编
|
||||
self.log(f"正在检查并设置邮编: {zip_code},标识: {mark}")
|
||||
self._set_zip_code(zip_code, mark)
|
||||
|
||||
# self.tab.wait.doc_loaded(timeout=5, raise_err=False)
|
||||
|
||||
# 3. 抓取数据
|
||||
self.log("正在抓取商品数据...")
|
||||
data = self._scrape_data()
|
||||
|
||||
return_data.update(data)
|
||||
|
||||
# 判断是否采集标题出错
|
||||
new_size = "220,220"
|
||||
pattern = r"\._(?:[A-Z]+)?(\d+)_\."
|
||||
# replacement = f"._{new_size}_.jpg"
|
||||
# 使用正则替换
|
||||
# image_new_url = re.sub(pattern, lambda m: replacement, data["image_url"])
|
||||
image_new_url = data["min_image_url"]
|
||||
|
||||
print(image_new_url)
|
||||
|
||||
iamge_base64 = self.image_to_base64(image_new_url)
|
||||
|
||||
similar_data = []
|
||||
# 4、请求获取插件的数据
|
||||
if image_new_url:
|
||||
for page_num in range(total_page):
|
||||
resp_data = self.get_aliprice_data(
|
||||
page=page_num+1,
|
||||
title = data["title"],
|
||||
domain=domain,
|
||||
category=data["category"],
|
||||
imageBase64 = iamge_base64
|
||||
)
|
||||
self.log(f"Aliprice 扩展数据获取:{str(resp_data)[0:200]}")
|
||||
if len(resp_data.get("data",[])) > 0:
|
||||
for i in resp_data.get("data"):
|
||||
similar_data.append(i)
|
||||
|
||||
data["similar_data"] = similar_data
|
||||
|
||||
# 添加基本信息
|
||||
data['country'] = country
|
||||
data['asin'] = asin
|
||||
data['url'] = product_url
|
||||
data['timestamp'] = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
print(f"数据抓取完成: {json.dumps(return_data)}")
|
||||
return_data.update(data)
|
||||
|
||||
return return_data
|
||||
|
||||
except Exception as e:
|
||||
error_msg = f"运行出错: {traceback.format_exc()}"
|
||||
print(error_msg)
|
||||
# show_notification(f"采集失败: {str(e)}", "error")
|
||||
raise RuntimeError(f"{traceback.format_exc()}")
|
||||
return return_data
|
||||
|
||||
|
||||
class SimilarAsinTask(TaskBase):
|
||||
mark_name = "相似ASIN采集"
|
||||
|
||||
@staticmethod
|
||||
def group_by_id_prefix(data):
|
||||
"""
|
||||
根据每条数据的 id 字段进行分组:
|
||||
- 如果 id 为 "2_1",则取 "_" 前面的 "2" 作为分组 key
|
||||
- 如果 id 为 "1",则分组 key 就是 "1"
|
||||
- 相同 key 的数据归为同一组
|
||||
|
||||
:param data: 原始列表数据
|
||||
:return: 二维列表,按 id 前缀分组
|
||||
"""
|
||||
grouped = defaultdict(list)
|
||||
|
||||
for item in data:
|
||||
item_id = str(item.get("id", ""))
|
||||
group_key = item_id.split("_")[0]
|
||||
grouped[group_key].append(item)
|
||||
|
||||
return list(grouped.values())
|
||||
|
||||
@staticmethod
|
||||
def normalize_groups(data):
|
||||
groups = data.get("groups")
|
||||
if isinstance(groups, list) and len(groups) > 0:
|
||||
return groups
|
||||
|
||||
rows = data.get("rows") or data.get("items") or []
|
||||
if not isinstance(rows, list) or len(rows) == 0:
|
||||
return []
|
||||
|
||||
normalized = []
|
||||
for items in SimilarAsinTask.group_by_id_prefix(rows):
|
||||
first = items[0] if items else {}
|
||||
item_id = str(first.get("id") or first.get("displayId") or "")
|
||||
base_id = item_id.split("_")[0] if item_id else ""
|
||||
normalized.append({
|
||||
"sourceFileKey": first.get("sourceFileKey", ""),
|
||||
"sourceFilename": first.get("sourceFilename", ""),
|
||||
"groupKey": first.get("groupKey") or base_id,
|
||||
"baseId": first.get("baseId") or base_id,
|
||||
"displayId": first.get("displayId") or item_id,
|
||||
"items": items,
|
||||
})
|
||||
return normalized
|
||||
|
||||
def fetch_parsed_payload(self, task_id, user_id=1):
|
||||
url = f"{DELETE_BRAND_API_BASE}/api/similar-asin/tasks/{task_id}/parsed-payload"
|
||||
response = requests.get(url, params={"user_id": user_id}, timeout=300, verify=False)
|
||||
response.raise_for_status()
|
||||
payload = response.json() if response.text else {}
|
||||
if isinstance(payload, dict) and payload.get("success"):
|
||||
return payload.get("data") or {}
|
||||
raise RuntimeError(f"获取货源查询解析载荷失败: {payload}")
|
||||
|
||||
def _merge_parsed_payload(self, data, parsed_payload):
|
||||
payload_rows = parsed_payload.get("allItems") or parsed_payload.get("items") or []
|
||||
merged = {
|
||||
**data,
|
||||
"groups": parsed_payload.get("groups") or [],
|
||||
"rows": payload_rows,
|
||||
}
|
||||
if parsed_payload.get("aiPrompt"):
|
||||
merged["prompt"] = parsed_payload.get("aiPrompt")
|
||||
if parsed_payload.get("apiKey") and not merged.get("api_key"):
|
||||
merged["api_key"] = parsed_payload.get("apiKey")
|
||||
return merged
|
||||
|
||||
def process_task(self, task_data: dict):
|
||||
"""处理审批任务主入口
|
||||
|
||||
Args:
|
||||
task_data: 任务数据
|
||||
"""
|
||||
try:
|
||||
data = task_data.get("data", {})
|
||||
task_id = data.get("taskId")
|
||||
parsed_payload = self.fetch_parsed_payload(task_id, data.get("user_id") or data.get("userId") or 1)
|
||||
data = self._merge_parsed_payload(data, parsed_payload)
|
||||
groups = self.normalize_groups(data)
|
||||
print(groups)
|
||||
|
||||
if not task_id:
|
||||
self.log("任务ID为空,跳过", "WARNING")
|
||||
return
|
||||
|
||||
self.log(f"开始处理爬取任务 {task_id},{len(groups)} 个任务")
|
||||
|
||||
if not groups:
|
||||
self.log("appearance-patent groups/rows is empty, skip", "WARNING")
|
||||
return
|
||||
|
||||
runing_task[task_id] = {
|
||||
"status": "running",
|
||||
"start_time": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"total_shops": 1,
|
||||
"processed_shops": 0,
|
||||
"total_countries": len(groups) ,
|
||||
"processed_countries": 0,
|
||||
"total_asins": 0,
|
||||
"processed_asins": 0,
|
||||
"success_count": 0,
|
||||
"failed_count": 0,
|
||||
"stop_requested": False
|
||||
}
|
||||
|
||||
# 检查是否收到暂停请求
|
||||
if task_id in runing_task and runing_task[task_id].get("stop_requested", False):
|
||||
self.log(f"检测到任务 {task_id} 的暂停请求,停止处理", "WARNING")
|
||||
runing_task[task_id]["status"] = "stopped"
|
||||
return
|
||||
max_retry = 3
|
||||
show_notification(f"开始爬取数据", "info")
|
||||
chrome = ChromeAmzone()
|
||||
try:
|
||||
# 数据整理
|
||||
# new_items = self.group_by_id_prefix(items)
|
||||
result = []
|
||||
for gp_index,gp in enumerate(groups):
|
||||
items = gp.get("items", [])
|
||||
group_item = []
|
||||
for index,value in enumerate(items):
|
||||
print(value)
|
||||
return_data = {
|
||||
'image_url': "",
|
||||
'title': "",
|
||||
'category' : '',
|
||||
'sku' : '',
|
||||
'success' : False,
|
||||
'similar_data' : []
|
||||
}
|
||||
asin = value.get("asin")
|
||||
country = value.get("country")
|
||||
for _ in range(max_retry):
|
||||
try:
|
||||
return_data = chrome.run(country, asin,total_page=1) or {}
|
||||
self.log(f"抓取结果->{return_data}")
|
||||
break
|
||||
except Exception as e:
|
||||
# if "与页面的连接已断开" in str(e):
|
||||
chrome = ChromeAmzone()
|
||||
self.log(f"{asin}抓取数据报错,{e}")
|
||||
# if not isinstance(return_data, dict):
|
||||
# return_data = {}
|
||||
# if return_data.get("image_url"):
|
||||
# break
|
||||
|
||||
group_item.append({
|
||||
"sourceFileKey": value.get("sourceFileKey",""),
|
||||
"sourceFilename": value.get("sourceFilename",""),
|
||||
"rowToken": value.get("rowToken",""),
|
||||
"groupKey": value.get("groupKey",""),
|
||||
"id": value.get("values",{}).get("id",""),
|
||||
"asin": value.get("asin",""),
|
||||
"sku" : return_data.get("sku",""),
|
||||
"country": value.get("country",""),
|
||||
"url": return_data.get("image_url",""),
|
||||
"title": return_data.get("title",""),
|
||||
"done": False,
|
||||
"urls" : [i.get("ori_picture") for i in return_data.get("similar_data",[])[:16]]
|
||||
})
|
||||
# task_id: int, chunkIndex:int,chunkTotal: int, country_code: str, asin: str, status: dict,error:str="",
|
||||
# item_data:dict={},
|
||||
|
||||
|
||||
res = {
|
||||
"sourceFileKey": gp.get("sourceFileKey"),
|
||||
"sourceFilename": gp.get("sourceFilename"),
|
||||
"groupKey": gp.get("groupKey"),
|
||||
"baseId": gp.get("baseId"),
|
||||
"displayId": gp.get("displayId"),
|
||||
"items": group_item
|
||||
}
|
||||
|
||||
# print("================")
|
||||
result.append(res)
|
||||
# print("================")
|
||||
is_done = gp_index == len(groups)-1
|
||||
if len(result) > 5 or is_done:
|
||||
self.post_result(task_id=task_id,chunkIndex=gp_index+1,chunkTotal=len(groups),
|
||||
asin=asin,item_data=result,is_done=is_done)
|
||||
result = []
|
||||
|
||||
|
||||
if len(result) > 0:
|
||||
is_done = True
|
||||
self.post_result(task_id=task_id, chunkIndex=len(groups), chunkTotal=len(groups),
|
||||
asin=asin, item_data=result, is_done=is_done)
|
||||
|
||||
|
||||
# 更新已处理店铺数
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["processed_shops"] += 1
|
||||
except Exception as e:
|
||||
self.log(f"处理店铺 {task_id} 失败: {str(e)}", "ERROR")
|
||||
self.log(traceback.format_exc(), "ERROR")
|
||||
|
||||
try:
|
||||
chrome.close()
|
||||
except Exception as e:
|
||||
print("退出浏览器出错", e)
|
||||
|
||||
# 更新任务状态
|
||||
if task_id in runing_task:
|
||||
if runing_task[task_id].get("stop_requested", False):
|
||||
runing_task[task_id]["status"] = "stopped"
|
||||
self.log(f"任务 {task_id} 已被暂停!")
|
||||
else:
|
||||
runing_task[task_id]["status"] = "completed"
|
||||
|
||||
self.log(f"任务 {task_id} 处理完成!")
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"任务处理失败: {traceback.format_exc()}", "ERROR")
|
||||
if task_id:
|
||||
if task_id in runing_task:
|
||||
runing_task[task_id]["status"] = "failed"
|
||||
runing_task[task_id]["error"] = str(e)
|
||||
|
||||
def post_result(self, task_id: int, chunkIndex:int,chunkTotal: int, asin: str, error:str="",
|
||||
item_data:dict={},
|
||||
is_done: bool = False):
|
||||
"""回传处理结果到API
|
||||
"""
|
||||
|
||||
url = f"{DELETE_BRAND_API_BASE}/api/similar-asin/tasks/{task_id}/result"
|
||||
|
||||
payload ={
|
||||
"submissionId": f"{task_id}",
|
||||
"chunkIndex": chunkIndex,
|
||||
"chunkTotal": chunkTotal,
|
||||
"error": error,
|
||||
"groups": item_data,
|
||||
"done": is_done
|
||||
}
|
||||
|
||||
max_retries = 3
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
request_timeout = 300 if is_done else 30
|
||||
print("================【详情采集】=====================")
|
||||
self.log(f"尝试回传结果 (第 {retry + 1}/{max_retries} 次)")
|
||||
self.log(f"回传URL: {url}")
|
||||
self.log(f"回传数据: {payload}")
|
||||
response = requests.post(
|
||||
url,
|
||||
json=payload,
|
||||
headers={"Content-Type": "application/json"},
|
||||
timeout=request_timeout,
|
||||
verify=False
|
||||
)
|
||||
self.log(f"回传结果: {response.text}")
|
||||
data = response.json() if response.text else {}
|
||||
if response.status_code == 200 and isinstance(data, dict) and data.get("success"):
|
||||
self.log(f"结果回传成功: {asin} - {item_data}")
|
||||
return
|
||||
else:
|
||||
self.log(f"结果回传失败,状态码: {response.status_code}", "WARNING")
|
||||
print("=====================================")
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"调用API异常: {str(e)}", "ERROR")
|
||||
print("=====================================")
|
||||
|
||||
# 如果还有重试机会,等待后继续
|
||||
if retry < max_retries - 1:
|
||||
time.sleep(2)
|
||||
|
||||
self.log(f"已达到最大重试次数,结果回传最终失败", "ERROR")
|
||||
# raise RuntimeError("已达到最大重试次数,结果回传最终失败")
|
||||
|
||||
if __name__ == '__main__':
|
||||
spide = ChromeAmzone()
|
||||
|
||||
resp = spide.run(
|
||||
country="德国", asin="B0CJ8SNXXV", total_page=2
|
||||
)
|
||||
print(resp)
|
||||
# task_data = {
|
||||
# "type": "similar-asin-run",
|
||||
# "ts": 1778038688824,
|
||||
# "data": {
|
||||
# "taskId": 6850,
|
||||
# "api_key": "sk-EMcDFg36zCeWUtCbRzKUbrRZyNeC6M4KBhY6fAVcNP7GG4xI",
|
||||
# "groups": [
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "groupKey": "similar-asin:6850",
|
||||
# "baseId": "",
|
||||
# "displayId": "1",
|
||||
# "items": [
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::2",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::1@2",
|
||||
# "id": "1",
|
||||
# "asin": "B0D792ND9V",
|
||||
# "country": "英国",
|
||||
# "price": "12.29",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::3",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::2@3",
|
||||
# "id": "2_1",
|
||||
# "asin": "B0D14L34S7",
|
||||
# "country": "英国",
|
||||
# "price": "14.88",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::4",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::2@3",
|
||||
# "id": "2_2",
|
||||
# "asin": "B0D14N8FJY",
|
||||
# "country": "英国",
|
||||
# "price": "15.36",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::5",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::2@3",
|
||||
# "id": "2_3",
|
||||
# "asin": "B0D14NBL5D",
|
||||
# "country": "英国",
|
||||
# "price": "14.87",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::6",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::2@3",
|
||||
# "id": "2_4",
|
||||
# "asin": "B0D14M5X4J",
|
||||
# "country": "英国",
|
||||
# "price": "14.41",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::7",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::3@7",
|
||||
# "id": "3_1",
|
||||
# "asin": "B0CWQFZMGY",
|
||||
# "country": "英国",
|
||||
# "price": "17.05",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::8",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::3@7",
|
||||
# "id": "3_2",
|
||||
# "asin": "B0CX8D68L3",
|
||||
# "country": "英国",
|
||||
# "price": "16.46",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::9",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::4@9",
|
||||
# "id": "4_1",
|
||||
# "asin": "B0CX87XXWJ",
|
||||
# "country": "英国",
|
||||
# "price": "17.54",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::10",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::5@10",
|
||||
# "id": "5_1",
|
||||
# "asin": "B0CR3PMR8L",
|
||||
# "country": "英国",
|
||||
# "price": "16.74",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# }
|
||||
# ]
|
||||
# }
|
||||
# ],
|
||||
# "rows": [
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::2",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::1@2",
|
||||
# "id": "1",
|
||||
# "asin": "B0D792ND9V",
|
||||
# "country": "英国",
|
||||
# "price": "12.29",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::3",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::2@3",
|
||||
# "id": "2_1",
|
||||
# "asin": "B0D14L34S7",
|
||||
# "country": "英国",
|
||||
# "price": "14.88",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::4",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::2@3",
|
||||
# "id": "2_2",
|
||||
# "asin": "B0D14N8FJY",
|
||||
# "country": "英国",
|
||||
# "price": "15.36",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::5",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::2@3",
|
||||
# "id": "2_3",
|
||||
# "asin": "B0D14NBL5D",
|
||||
# "country": "英国",
|
||||
# "price": "14.87",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::6",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::2@3",
|
||||
# "id": "2_4",
|
||||
# "asin": "B0D14M5X4J",
|
||||
# "country": "英国",
|
||||
# "price": "14.41",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::7",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::3@7",
|
||||
# "id": "3_1",
|
||||
# "asin": "B0CWQFZMGY",
|
||||
# "country": "英国",
|
||||
# "price": "17.05",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::8",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::3@7",
|
||||
# "id": "3_2",
|
||||
# "asin": "B0CX8D68L3",
|
||||
# "country": "英国",
|
||||
# "price": "16.46",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::9",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::4@9",
|
||||
# "id": "4_1",
|
||||
# "asin": "B0CX87XXWJ",
|
||||
# "country": "英国",
|
||||
# "price": "17.54",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# },
|
||||
# {
|
||||
# "sourceFileKey": "cdfa437df62b40a0bde5682a05c1f6a3",
|
||||
# "sourceFilename": "17(1) - 副本1.xlsx",
|
||||
# "rowToken": "cdfa437df62b40a0bde5682a05c1f6a3::row::10",
|
||||
# "groupKey": "cdfa437df62b40a0bde5682a05c1f6a3::5@10",
|
||||
# "id": "5_1",
|
||||
# "asin": "B0CR3PMR8L",
|
||||
# "country": "英国",
|
||||
# "price": "16.74",
|
||||
# "url": "",
|
||||
# "title": "",
|
||||
# "target_urls": []
|
||||
# }
|
||||
# ]
|
||||
# }
|
||||
# }
|
||||
# sim_asin = SimilarAsinTask({})
|
||||
# sim_asin.process_task(task_data)
|
||||
|
||||
|
||||
|
||||
@@ -124,7 +124,6 @@ def remove_special_characters(text: str) -> str:
|
||||
return cleaned
|
||||
|
||||
|
||||
|
||||
def split_currency_values(currency_str: str) -> tuple[float, float]:
|
||||
"""
|
||||
将包含两个货币值的字符串拆分成两个浮点数。
|
||||
|
||||
60
app/app.py
60
app/app.py
@@ -3,8 +3,9 @@
|
||||
按功能拆分为蓝图:认证(auth)、主页面(main)、管理员(admin)、图片(image)、品牌(brand)
|
||||
"""
|
||||
import os
|
||||
import secrets
|
||||
from datetime import timedelta
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from logging.handlers import RotatingFileHandler
|
||||
|
||||
from flask import Flask
|
||||
from flask_cors import CORS
|
||||
@@ -16,6 +17,52 @@ from blueprints.admin import admin_bp
|
||||
from blueprints.image import image_bp
|
||||
from blueprints.brand import brand_bp
|
||||
from blueprints.communication import communication_bp
|
||||
|
||||
from config import cache_path, debug
|
||||
|
||||
|
||||
def setup_flask_logging(app):
|
||||
"""配置Flask访问日志,输出到独立的日志文件"""
|
||||
|
||||
api_cache_path = os.path.join(cache_path, "api_logs")
|
||||
os.makedirs(api_cache_path, exist_ok=True)
|
||||
|
||||
# 只在非debug模式下配置文件日志
|
||||
if not debug:
|
||||
# 获取当前日期作为日志文件名的一部分
|
||||
today = datetime.now().strftime("%Y_%m_%d")
|
||||
log_file = os.path.join(api_cache_path, f'api_{today}.log')
|
||||
|
||||
# 创建文件处理器,使用RotatingFileHandler避免日志文件过大
|
||||
file_handler = RotatingFileHandler(
|
||||
log_file,
|
||||
maxBytes=10 * 1024 * 1024, # 10MB
|
||||
backupCount=5,
|
||||
encoding='utf-8'
|
||||
)
|
||||
|
||||
# 设置日志格式
|
||||
formatter = logging.Formatter(
|
||||
'[%(asctime)s] %(levelname)s in %(module)s: %(message)s',
|
||||
datefmt='%Y-%m-%d %H:%M:%S'
|
||||
)
|
||||
file_handler.setFormatter(formatter)
|
||||
file_handler.setLevel(logging.INFO)
|
||||
|
||||
# 配置Flask应用日志
|
||||
app.logger.addHandler(file_handler)
|
||||
app.logger.setLevel(logging.INFO)
|
||||
|
||||
# 配置Werkzeug日志(HTTP访问日志)
|
||||
werkzeug_logger = logging.getLogger('werkzeug')
|
||||
werkzeug_logger.addHandler(file_handler)
|
||||
werkzeug_logger.setLevel(logging.INFO)
|
||||
|
||||
print(f"API 日志已配置,输出到: {log_file}")
|
||||
else:
|
||||
print("Debug模式下,API 日志输出到控制台")
|
||||
|
||||
|
||||
def create_app():
|
||||
app = Flask(__name__, template_folder=BASE_DIR, static_folder=BASE_DIR)
|
||||
frontend_origin = os.environ.get('FRONTEND_ORIGIN', '*')
|
||||
@@ -25,12 +72,6 @@ def create_app():
|
||||
supports_credentials=True,
|
||||
resources={r"/api/*": {"origins": cors_origins}, r"/login": {"origins": cors_origins}},
|
||||
)
|
||||
# 生产环境必须通过环境变量注入固定 SECRET_KEY;否则服务重启会使旧会话失效。
|
||||
app.secret_key = os.environ.get('SECRET_KEY', 'dev-secret-key-change-me')
|
||||
app.config['PERMANENT_SESSION_LIFETIME'] = timedelta(days=7)
|
||||
app.config['SESSION_COOKIE_HTTPONLY'] = True
|
||||
app.config['SESSION_COOKIE_SAMESITE'] = os.environ.get('SESSION_COOKIE_SAMESITE', 'Lax')
|
||||
app.config['SESSION_COOKIE_SECURE'] = os.environ.get('SESSION_COOKIE_SECURE', '0') == '1'
|
||||
|
||||
# 注册蓝图(不设 url_prefix,保持原有 URL 路径不变,前端无需改动)
|
||||
app.register_blueprint(auth_bp)
|
||||
@@ -39,6 +80,9 @@ def create_app():
|
||||
app.register_blueprint(image_bp)
|
||||
app.register_blueprint(brand_bp)
|
||||
app.register_blueprint(communication_bp)
|
||||
|
||||
# 配置Flask访问日志
|
||||
setup_flask_logging(app)
|
||||
|
||||
return app
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
"""
|
||||
公共模块:数据库连接、初始化、会话校验、装饰器、模板渲染
|
||||
公共模块:数据库连接、初始化、JWT 鉴权装饰器、模板渲染
|
||||
供各蓝图复用
|
||||
"""
|
||||
import os
|
||||
@@ -8,18 +8,19 @@ from datetime import timedelta
|
||||
from functools import wraps
|
||||
|
||||
import pymysql
|
||||
from flask import request, redirect, url_for, session, jsonify, render_template, render_template_string
|
||||
from flask import request, redirect, url_for, jsonify, render_template, render_template_string, g
|
||||
|
||||
from config import mysql_host, mysql_user, mysql_password, mysql_database
|
||||
from jwt_util import parse_token, get_token_from_request
|
||||
|
||||
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
STATIC_DIR = os.path.join(BASE_DIR, 'static')
|
||||
ASSETS_DIR = os.path.join(BASE_DIR, 'assets')
|
||||
WEB_SOURCE_DIR = os.path.join(BASE_DIR, 'web_source')
|
||||
TEMPLATE_FALLBACK_DIRS = (
|
||||
WEB_SOURCE_DIR,
|
||||
os.path.join(WEB_SOURCE_DIR, 'templates_backup'),
|
||||
)
|
||||
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
STATIC_DIR = os.path.join(BASE_DIR, 'static')
|
||||
ASSETS_DIR = os.path.join(BASE_DIR, 'assets')
|
||||
WEB_SOURCE_DIR = os.path.join(BASE_DIR, 'web_source')
|
||||
TEMPLATE_FALLBACK_DIRS = (
|
||||
WEB_SOURCE_DIR,
|
||||
os.path.join(WEB_SOURCE_DIR, 'templates_backup'),
|
||||
)
|
||||
|
||||
|
||||
def get_db():
|
||||
@@ -35,13 +36,13 @@ def get_db():
|
||||
|
||||
def _render_html(template_name: str, **context):
|
||||
"""读取 HTML 模板:若为加密文件则先解密,再渲染。未加密或解密失败时按明文渲染。"""
|
||||
path = next(
|
||||
(candidate for candidate in (os.path.join(base_path, template_name) for base_path in TEMPLATE_FALLBACK_DIRS)
|
||||
if os.path.isfile(candidate)),
|
||||
None,
|
||||
)
|
||||
if path is None:
|
||||
return render_template(template_name, **context)
|
||||
path = next(
|
||||
(candidate for candidate in (os.path.join(base_path, template_name) for base_path in TEMPLATE_FALLBACK_DIRS)
|
||||
if os.path.isfile(candidate)),
|
||||
None,
|
||||
)
|
||||
if path is None:
|
||||
return render_template(template_name, **context)
|
||||
with open(path, "rb") as f:
|
||||
raw = f.read()
|
||||
try:
|
||||
@@ -54,13 +55,13 @@ def _render_html(template_name: str, **context):
|
||||
|
||||
def _render_html_new(template_name: str, **context):
|
||||
"""读取 HTML 模板:若为加密文件则先解密,再渲染。未加密或解密失败时按明文渲染。"""
|
||||
path = next(
|
||||
(candidate for candidate in (os.path.join(base_path, template_name) for base_path in TEMPLATE_FALLBACK_DIRS)
|
||||
if os.path.isfile(candidate)),
|
||||
None,
|
||||
)
|
||||
if path is None:
|
||||
return render_template(template_name, **context)
|
||||
path = next(
|
||||
(candidate for candidate in (os.path.join(base_path, template_name) for base_path in TEMPLATE_FALLBACK_DIRS)
|
||||
if os.path.isfile(candidate)),
|
||||
None,
|
||||
)
|
||||
if path is None:
|
||||
return render_template(template_name, **context)
|
||||
with open(path, "rb") as f:
|
||||
raw = f.read()
|
||||
try:
|
||||
@@ -72,9 +73,36 @@ def _render_html_new(template_name: str, **context):
|
||||
|
||||
|
||||
|
||||
def _resolve_jwt_user():
|
||||
"""从 Authorization/cookie 解析 JWT,命中后写入 flask.g 缓存。无效返回 None。"""
|
||||
cached = getattr(g, "_aiimage_user", None)
|
||||
if cached is not None:
|
||||
return cached or None
|
||||
token = get_token_from_request()
|
||||
payload = parse_token(token) if token else None
|
||||
if not payload:
|
||||
g._aiimage_user = False
|
||||
return None
|
||||
g._aiimage_user = payload
|
||||
g.aiimage_user_id = payload.get("user_id")
|
||||
g.aiimage_username = payload.get("username") or ""
|
||||
g.aiimage_device_id = payload.get("device_id") or ""
|
||||
return payload
|
||||
|
||||
|
||||
def current_user_id():
|
||||
user = _resolve_jwt_user()
|
||||
return user.get("user_id") if user else None
|
||||
|
||||
|
||||
def current_username():
|
||||
user = _resolve_jwt_user()
|
||||
return user.get("username") if user else ""
|
||||
|
||||
|
||||
def _is_session_user_valid():
|
||||
"""校验 session 中的 user_id 是否在数据库中仍存在;不存在则清除 session 并返回 False"""
|
||||
uid = session.get('user_id')
|
||||
"""兼容旧调用:判断当前 JWT 用户是否仍存在于数据库。"""
|
||||
uid = current_user_id()
|
||||
if not uid:
|
||||
return False
|
||||
try:
|
||||
@@ -83,23 +111,22 @@ def _is_session_user_valid():
|
||||
cur.execute("SELECT id FROM users WHERE id = %s", (uid,))
|
||||
row = cur.fetchone()
|
||||
conn.close()
|
||||
if not row:
|
||||
session.clear()
|
||||
return False
|
||||
return True
|
||||
return bool(row)
|
||||
except Exception:
|
||||
session.clear()
|
||||
return False
|
||||
|
||||
|
||||
def _get_current_admin_role():
|
||||
"""获取当前登录用户的管理角色:super_admin / admin / None(非管理员)"""
|
||||
uid = current_user_id()
|
||||
if not uid:
|
||||
return None, None
|
||||
try:
|
||||
conn = get_db()
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"SELECT id, username, is_admin, role, created_by_id FROM users WHERE id = %s",
|
||||
(session['user_id'],)
|
||||
(uid,)
|
||||
)
|
||||
row = cur.fetchone()
|
||||
conn.close()
|
||||
@@ -110,13 +137,19 @@ def _get_current_admin_role():
|
||||
return None, None
|
||||
|
||||
|
||||
def _unauthorized_response():
|
||||
if request.headers.get('X-Requested-With') == 'XMLHttpRequest' \
|
||||
or request.path.startswith('/api/') \
|
||||
or 'application/json' in (request.headers.get('Accept') or ''):
|
||||
return jsonify({'success': False, 'error': '未登录'}), 401
|
||||
return redirect(url_for('auth.login'))
|
||||
|
||||
|
||||
def login_required(f):
|
||||
@wraps(f)
|
||||
def decorated(*args, **kwargs):
|
||||
if not session.get('user_id') or not _is_session_user_valid():
|
||||
if request.headers.get('X-Requested-With') == 'XMLHttpRequest':
|
||||
return jsonify({'success': False, 'error': '未登录'}), 401
|
||||
return redirect(url_for('auth.login'))
|
||||
if not current_user_id():
|
||||
return _unauthorized_response()
|
||||
return f(*args, **kwargs)
|
||||
return decorated
|
||||
|
||||
@@ -124,22 +157,24 @@ def login_required(f):
|
||||
def admin_required(f):
|
||||
@wraps(f)
|
||||
def decorated(*args, **kwargs):
|
||||
if not session.get('user_id'):
|
||||
if request.headers.get('X-Requested-With') == 'XMLHttpRequest':
|
||||
return jsonify({'success': False, 'error': '未登录'}), 401
|
||||
return redirect(url_for('auth.login'))
|
||||
uid = current_user_id()
|
||||
if not uid:
|
||||
return _unauthorized_response()
|
||||
try:
|
||||
conn = get_db()
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT is_admin, role FROM users WHERE id = %s", (session['user_id'],))
|
||||
cur.execute("SELECT is_admin, role FROM users WHERE id = %s", (uid,))
|
||||
row = cur.fetchone()
|
||||
conn.close()
|
||||
if not row or not row.get('is_admin'):
|
||||
if request.headers.get('X-Requested-With') == 'XMLHttpRequest':
|
||||
if request.headers.get('X-Requested-With') == 'XMLHttpRequest' \
|
||||
or request.path.startswith('/api/') \
|
||||
or 'application/json' in (request.headers.get('Accept') or ''):
|
||||
return jsonify({'success': False, 'error': '需要管理员权限'}), 403
|
||||
return redirect(url_for('main.home'))
|
||||
except Exception as e:
|
||||
if request.headers.get('X-Requested-With') == 'XMLHttpRequest':
|
||||
if request.headers.get('X-Requested-With') == 'XMLHttpRequest' \
|
||||
or request.path.startswith('/api/'):
|
||||
return jsonify({'success': False, 'error': str(e)}), 500
|
||||
return redirect(url_for('main.home'))
|
||||
return f(*args, **kwargs)
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
|
||||
.module-page[data-v-e32c69fa]{min-height:100vh;background:#1a1a1a}.main-content[data-v-e32c69fa]{display:flex;height:calc(100vh - 56px);min-height:calc(100vh - 56px)}.left-panel[data-v-e32c69fa]{width:400px;background:#1e1e1e;padding:20px;overflow-y:auto;border-right:1px solid #2a2a2a}.right-panel[data-v-e32c69fa]{flex:1;min-width:0;background:#1a1a1a;display:flex;flex-direction:column}.section-title[data-v-e32c69fa],.subsection-title[data-v-e32c69fa]{font-size:13px;color:#bbb;margin-bottom:10px}.upload-zone[data-v-e32c69fa]{border:1px dashed #3a3a3a;border-radius:10px;padding:18px;background:#252525;margin-bottom:18px}.hint[data-v-e32c69fa],.loading-msg[data-v-e32c69fa],.files[data-v-e32c69fa],.muted[data-v-e32c69fa]{color:#888;font-size:12px;line-height:1.5}.link[data-v-e32c69fa]{color:#6ea8fe;text-decoration:none}.link[data-v-e32c69fa]:hover{color:#9fc5ff}.btns[data-v-e32c69fa],.run-row[data-v-e32c69fa]{display:flex;gap:10px;flex-wrap:wrap}.opt-btn[data-v-e32c69fa],.btn-run[data-v-e32c69fa],.btn-delete[data-v-e32c69fa]{border:none;cursor:pointer;border-radius:7px}.opt-btn[data-v-e32c69fa]{padding:8px 14px;color:#ccc;background:#2a2a2a;border:1px solid #3a3a3a}.btn-run[data-v-e32c69fa]{padding:10px 18px;color:#fff;background:#3498db;font-weight:600}.btn-queue[data-v-e32c69fa]{background:#27ae60}.btn-run[data-v-e32c69fa]:disabled{opacity:.55;cursor:not-allowed}.selected-files[data-v-e32c69fa]{margin-top:14px;color:#999;font-size:12px;word-break:break-all}.selected-files span[data-v-e32c69fa]{display:block;margin:4px 0}.prompt-card[data-v-e32c69fa]{margin-bottom:18px}.prompt-input[data-v-e32c69fa]{width:100%;box-sizing:border-box;resize:vertical;min-height:180px;padding:10px 12px;border:1px solid #333;border-radius:8px;background:#202020;color:#d8d8d8;font-size:12px;line-height:1.6;outline:none}.prompt-input[data-v-e32c69fa]:focus{border-color:#3498db}.prompt-default-label[data-v-e32c69fa]{margin-top:10px;color:#8d8d8d;font-size:12px}.prompt-preview[data-v-e32c69fa]{margin-top:10px;padding:12px;border:1px solid #2a2a2a;border-radius:8px;background:#202020;color:#9ea7b3;font-size:12px;line-height:1.6;white-space:pre-wrap}.parse-card[data-v-e32c69fa],.queue-payload[data-v-e32c69fa]{margin-top:14px;padding:12px;border:1px solid #2a2a2a;border-radius:8px;background:#202020;color:#b8c1cc;font-size:12px}.queue-payload[data-v-e32c69fa]{max-height:220px;overflow:auto;color:#8fd3ff;white-space:pre-wrap}.panel-header[data-v-e32c69fa]{padding:16px 20px;border-bottom:1px solid #2a2a2a;font-size:15px;font-weight:600;color:#ddd}.task-list-wrap[data-v-e32c69fa]{flex:1;padding:16px 20px;overflow:auto}.clean-result-summary[data-v-e32c69fa]{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:12px;margin-bottom:16px}.summary-card[data-v-e32c69fa]{padding:14px 16px;border:1px solid #2a2a2a;border-radius:8px;background:#1e1e1e}.summary-card strong[data-v-e32c69fa]{display:block;margin-top:8px;color:#eaf4ff;font-size:22px}.summary-label[data-v-e32c69fa]{color:#8d8d8d;font-size:12px}.result-list-wrap[data-v-e32c69fa]{border:1px solid #2a2a2a;border-radius:8px;background:#1e1e1e;min-height:180px;margin:0 0 16px}.result-list-header[data-v-e32c69fa]{display:flex;justify-content:space-between;padding:12px 16px;border-bottom:1px solid #2a2a2a;color:#ddd;font-size:14px}.empty-tasks[data-v-e32c69fa]{color:#666;font-size:13px;padding:18px;text-align:center}.result-table[data-v-e32c69fa]{--el-table-bg-color: #222;--el-table-tr-bg-color: #222;--el-table-header-bg-color: #2a2a2a;--el-table-text-color: #ccc;--el-table-border-color: #333}.task-list[data-v-e32c69fa]{list-style:none;margin:0;padding:12px}.task-item[data-v-e32c69fa]{display:flex;align-items:flex-start;justify-content:space-between;gap:12px;padding:12px 14px;border:1px solid #2a2a2a;border-radius:8px;margin-bottom:8px;background:#222}.left[data-v-e32c69fa]{flex:1;min-width:0}.id[data-v-e32c69fa]{color:#e0e0e0;font-size:13px;font-weight:600}.task-right[data-v-e32c69fa]{display:flex;gap:8px;align-items:center;flex-wrap:wrap}.status[data-v-e32c69fa]{padding:4px 10px;border-radius:6px;font-size:12px}.status.success[data-v-e32c69fa]{background:#2ecc712e;color:#2ecc71}.status.failed[data-v-e32c69fa]{background:#e74c3c2e;color:#ff6b6b}.status.running[data-v-e32c69fa]{background:#3498db2e;color:#3498db}.btn-delete[data-v-e32c69fa]{padding:6px 10px;color:#ff8f8f;background:#e74c3c1f}@media(max-width:1100px){.main-content[data-v-e32c69fa]{flex-direction:column;height:auto}.left-panel[data-v-e32c69fa]{width:100%;border-right:none;border-bottom:1px solid #2a2a2a}.clean-result-summary[data-v-e32c69fa]{grid-template-columns:repeat(2,minmax(0,1fr))}}
|
||||
@@ -1 +0,0 @@
|
||||
.module-page[data-v-d71413c4]{min-height:100vh;background:#1a1a1a}.main-content[data-v-d71413c4]{display:flex;height:calc(100vh - 56px);min-height:calc(100vh - 56px)}.left-panel[data-v-d71413c4]{width:400px;background:#1e1e1e;padding:20px;overflow-y:auto;border-right:1px solid #2a2a2a}.right-panel[data-v-d71413c4]{flex:1;min-width:0;background:#1a1a1a;display:flex;flex-direction:column}.section-title[data-v-d71413c4],.subsection-title[data-v-d71413c4]{font-size:13px;color:#bbb;margin-bottom:10px}.upload-zone[data-v-d71413c4]{border:1px dashed #3a3a3a;border-radius:10px;padding:18px;background:#252525;margin-bottom:18px}.hint[data-v-d71413c4],.loading-msg[data-v-d71413c4],.files[data-v-d71413c4],.muted[data-v-d71413c4]{color:#888;font-size:12px;line-height:1.5}.link[data-v-d71413c4]{color:#6ea8fe;text-decoration:none}.link[data-v-d71413c4]:hover{color:#9fc5ff}.btns[data-v-d71413c4],.run-row[data-v-d71413c4]{display:flex;gap:10px;flex-wrap:wrap}.opt-btn[data-v-d71413c4],.btn-run[data-v-d71413c4],.btn-delete[data-v-d71413c4],.download[data-v-d71413c4]{border:none;cursor:pointer;border-radius:7px}.opt-btn[data-v-d71413c4]{padding:8px 14px;color:#ccc;background:#2a2a2a;border:1px solid #3a3a3a}.btn-run[data-v-d71413c4]{padding:10px 18px;color:#fff;background:#3498db;font-weight:600}.btn-queue[data-v-d71413c4]{background:#27ae60}.btn-run[data-v-d71413c4]:disabled{opacity:.55;cursor:not-allowed}.selected-files[data-v-d71413c4]{margin-top:14px;color:#999;font-size:12px;word-break:break-all}.selected-files span[data-v-d71413c4]{display:block;margin:4px 0}.prompt-card[data-v-d71413c4]{margin-bottom:18px}.prompt-input[data-v-d71413c4]{width:100%;box-sizing:border-box;resize:vertical;min-height:180px;padding:10px 12px;border:1px solid #333;border-radius:8px;background:#202020;color:#d8d8d8;font-size:12px;line-height:1.6;outline:none}.prompt-input[data-v-d71413c4]:focus{border-color:#3498db}.prompt-default-label[data-v-d71413c4]{margin-top:10px;color:#8d8d8d;font-size:12px}.prompt-preview[data-v-d71413c4]{margin-top:10px;padding:12px;border:1px solid #2a2a2a;border-radius:8px;background:#202020;color:#9ea7b3;font-size:12px;line-height:1.6;white-space:pre-wrap}.parse-card[data-v-d71413c4],.queue-payload[data-v-d71413c4]{margin-top:14px;padding:12px;border:1px solid #2a2a2a;border-radius:8px;background:#202020;color:#b8c1cc;font-size:12px}.queue-payload[data-v-d71413c4]{max-height:220px;overflow:auto;color:#8fd3ff;white-space:pre-wrap}.panel-header[data-v-d71413c4]{padding:16px 20px;border-bottom:1px solid #2a2a2a;font-size:15px;font-weight:600;color:#ddd}.task-list-wrap[data-v-d71413c4]{flex:1;padding:16px 20px;overflow:auto}.clean-result-summary[data-v-d71413c4]{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:12px;margin-bottom:16px}.summary-card[data-v-d71413c4]{padding:14px 16px;border:1px solid #2a2a2a;border-radius:8px;background:#1e1e1e}.summary-card strong[data-v-d71413c4]{display:block;margin-top:8px;color:#eaf4ff;font-size:22px}.summary-label[data-v-d71413c4]{color:#8d8d8d;font-size:12px}.result-list-wrap[data-v-d71413c4]{border:1px solid #2a2a2a;border-radius:8px;background:#1e1e1e;min-height:180px;margin:0 0 16px}.result-list-header[data-v-d71413c4]{display:flex;justify-content:space-between;padding:12px 16px;border-bottom:1px solid #2a2a2a;color:#ddd;font-size:14px}.empty-tasks[data-v-d71413c4]{color:#666;font-size:13px;padding:18px;text-align:center}.result-table[data-v-d71413c4]{--el-table-bg-color: #222;--el-table-tr-bg-color: #222;--el-table-header-bg-color: #2a2a2a;--el-table-text-color: #ccc;--el-table-border-color: #333}.task-list[data-v-d71413c4]{list-style:none;margin:0;padding:12px}.task-item[data-v-d71413c4]{display:flex;align-items:flex-start;justify-content:space-between;gap:12px;padding:12px 14px;border:1px solid #2a2a2a;border-radius:8px;margin-bottom:8px;background:#222}.left[data-v-d71413c4]{flex:1;min-width:0}.id[data-v-d71413c4]{color:#e0e0e0;font-size:13px;font-weight:600}.task-right[data-v-d71413c4]{display:flex;gap:8px;align-items:center;flex-wrap:wrap}.status[data-v-d71413c4]{padding:4px 10px;border-radius:6px;font-size:12px}.status.success[data-v-d71413c4]{background:#2ecc712e;color:#2ecc71}.status.failed[data-v-d71413c4]{background:#e74c3c2e;color:#ff6b6b}.status.running[data-v-d71413c4]{background:#3498db2e;color:#3498db}.result-hint[data-v-d71413c4]{margin-top:6px;color:#e0b96d}.download[data-v-d71413c4]{padding:6px 10px;color:#d6ecff;background:#3498db2e}.btn-delete[data-v-d71413c4]{padding:6px 10px;color:#ff8f8f;background:#e74c3c1f}@media(max-width:1100px){.main-content[data-v-d71413c4]{flex-direction:column;height:auto}.left-panel[data-v-d71413c4]{width:100%;border-right:none;border-bottom:1px solid #2a2a2a}.clean-result-summary[data-v-d71413c4]{grid-template-columns:repeat(2,minmax(0,1fr))}}
|
||||
1
app/assets/appearance-patent-BoTBnU7J.css
Normal file
1
app/assets/appearance-patent-BoTBnU7J.css
Normal file
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
|
||||
import{Q as r}from"./pywebview-V_UIajJm.js";const n="";function s(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{s as e};
|
||||
@@ -1 +0,0 @@
|
||||
import{aV as r}from"./pywebview-Bt854mYs.js";const n="";function a(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{a as e};
|
||||
@@ -1 +0,0 @@
|
||||
import{Q as r}from"./pywebview-DiP0HdY6.js";const n="";function s(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{s as e};
|
||||
@@ -1 +0,0 @@
|
||||
import{aZ as r}from"./pywebview-C66x_2Dh.js";const n="";function a(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{a as e};
|
||||
@@ -1 +0,0 @@
|
||||
import{Q as r}from"./pywebview-Cs1Kot1q.js";const n="";function s(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{s as e};
|
||||
@@ -1 +0,0 @@
|
||||
import{Q as r}from"./pywebview-BCPdlDdb.js";const n="";function s(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{s as e};
|
||||
@@ -1 +0,0 @@
|
||||
import{aO as r}from"./pywebview-DuyK2jB1.js";const n="";function a(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{a as e};
|
||||
1
app/assets/brand-COze15GJ.js
Normal file
1
app/assets/brand-COze15GJ.js
Normal file
@@ -0,0 +1 @@
|
||||
import{bN as r}from"./java-modules-B8c-YG5x.js";const n="";function s(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{s as e};
|
||||
@@ -1 +0,0 @@
|
||||
import{aO as r}from"./pywebview-9YBa--7x.js";const n="";function a(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{a as e};
|
||||
@@ -1 +0,0 @@
|
||||
import{aI as r}from"./pywebview-D4gpiFjY.js";const n="";function a(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{a as e};
|
||||
@@ -1 +0,0 @@
|
||||
import{aO as r}from"./pywebview-ClWy2SbE.js";const n="";function a(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{a as e};
|
||||
@@ -1 +0,0 @@
|
||||
import{Q as r}from"./pywebview-D808cNhy.js";const n="";function s(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{s as e};
|
||||
@@ -1 +0,0 @@
|
||||
import{Q as r}from"./pywebview-Cq_E2BnJ.js";const n="";function s(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{s as e};
|
||||
@@ -1 +0,0 @@
|
||||
import{Q as r}from"./pywebview-D7_PdvyM.js";const n="";function s(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{s as e};
|
||||
@@ -1 +0,0 @@
|
||||
import{aO as r}from"./pywebview-CeWJDVeG.js";const n="";function a(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{a as e};
|
||||
1
app/assets/categorized-timers-JPA-olTr.js
Normal file
1
app/assets/categorized-timers-JPA-olTr.js
Normal file
@@ -0,0 +1 @@
|
||||
const r=new Map;function c(n){let t=r.get(n);return t||(t=new Map,r.set(n,t)),t}function u(n,t){const e=r.get(n);e&&(e.delete(t),e.size||r.delete(n))}function l(n,t,e){const i=window.setTimeout(()=>{u(n,i),t()},e);return c(n).set(i,{id:i,kind:"timeout",category:n}),i}function a(n,t,e){const i=window.setInterval(t,e);return c(n).set(i,{id:i,kind:"interval",category:n}),i}function f(n,t){if(t==null)return;const i=r.get(n)?.get(t);i?.kind==="interval"?window.clearInterval(t):window.clearTimeout(t),i?.cancel?.(),u(n,t)}function s(n){const t=r.get(n);if(t){for(const e of t.values())e.kind==="interval"?window.clearInterval(e.id):window.clearTimeout(e.id),e.cancel?.();r.delete(n)}}function d(n,t){return new Promise(e=>{const i=window.setTimeout(()=>{u(n,i),e()},t);c(n).set(i,{id:i,kind:"timeout",category:n,cancel:e})})}function m(n){const t=`${n}:`;for(const e of Array.from(r.keys()))(e===n||e.startsWith(t))&&s(e)}function w(n){const t=e=>`${n}:${e}`;return{setTimeout(e,i,o){return l(t(e),i,o)},setInterval(e,i,o){return a(t(e),i,o)},clearTimer(e,i){f(t(e),i)},clearCategory(e){s(t(e))},clearScope(){m(n)},sleep(e,i){return d(t(e),i)}}}export{w as c};
|
||||
1
app/assets/collect-data-DPGAQszU.css
Normal file
1
app/assets/collect-data-DPGAQszU.css
Normal file
File diff suppressed because one or more lines are too long
1
app/assets/collect-data.js
Normal file
1
app/assets/collect-data.js
Normal file
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
1
app/assets/convert-6oxZNMye.css
Normal file
1
app/assets/convert-6oxZNMye.css
Normal file
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
1
app/assets/dedupe-DpfPYQDv.css
Normal file
1
app/assets/dedupe-DpfPYQDv.css
Normal file
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
1
app/assets/delete-brand-DZ3x5hOX.css
Normal file
1
app/assets/delete-brand-DZ3x5hOX.css
Normal file
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user