220 Commits

Author SHA1 Message Date
super
9835831415 更新地址 更新后端 2026-05-29 22:53:44 +08:00
super
080625567b 同步 Gitee 应用代码更新 2026-05-29 19:47:23 +08:00
super
266c0f17c1 完成这个地址替换 2026-05-29 19:35:55 +08:00
super
225d13fb6e 提交优化更新 2026-05-29 16:49:18 +08:00
super
2ed1250604 增加公共下载进度、增加接收SKU、密钥分别存放 2026-05-28 16:41:20 +08:00
super
ca4a2cd07a python端登录完成 2026-05-27 12:41:59 +08:00
super
73ac9187a6 提交货源流程优化 2026-05-25 22:16:50 +08:00
super
7503e3fa8b 提交货源图片和相似ASIN优化 2026-05-24 21:59:21 +08:00
super
532438faba 提交最新的优化结果 2026-05-24 12:42:53 +08:00
super
a1376b51b0 优化图片尺寸 2026-05-24 02:31:58 +08:00
super
1e087c1aae 提交登录修改 2026-05-23 18:32:45 +08:00
super
2e2de02476 Merge gitee/master into local master (sync from old repo)
# Conflicts:
#	app/web_source/admin.html
#	app/web_source/brand-旧.html
#	app/web_source/home.html
#	app/web_source/index.html
#	app/web_source/login.html
#	backend/web_source/login.html
#	web_source/templates_backup/login.html
2026-05-23 12:00:12 +08:00
super
aeb4e1710c chore: 从 gitee 旧仓库恢复资源和软件更新包 2026-05-23 11:56:36 +08:00
super
b4b80cb572 更新这个货源 2026-05-22 09:41:31 +08:00
super
70e902e7ea 更新这个货源 2026-05-22 09:41:31 +08:00
super
14cebf22c3 提交一些更改 2026-05-20 09:05:28 +08:00
super
73d25fcbcb 提交一些更改 2026-05-20 09:05:28 +08:00
super
01ecde45c5 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-05-15 16:06:15 +08:00
super
a2ebfc0b81 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-05-15 16:06:15 +08:00
super
1dc2cac19a 修改完善这个专利部分 2026-05-15 16:06:12 +08:00
super
14db1e0cb1 修改完善这个专利部分 2026-05-15 16:06:12 +08:00
铭坤
01f4dbb83a modified: amazon/similar_asin.py 2026-05-15 11:33:02 +08:00
铭坤
0d63d6d75a modified: amazon/similar_asin.py 2026-05-15 11:33:02 +08:00
铭坤
fe692fcd1d modified: amazon/chrome_base.py
modified:   amazon/detail_spider.py
	modified:   amazon/match_action.py
	modified:   amazon/similar_asin.py
	deleted:    amazon/user_data/chrome_data/BrowserMetrics-spare.pma
	deleted:    amazon/user_data/chrome_data/BrowserMetrics/BrowserMetrics-69FAEF07-558.pma
	deleted:    amazon/user_data/chrome_data/Crashpad/metadata
	deleted:    amazon/user_data/chrome_data/Crashpad/settings.dat
2026-05-13 16:51:29 +08:00
铭坤
79b177b345 modified: amazon/chrome_base.py
modified:   amazon/detail_spider.py
	modified:   amazon/match_action.py
	modified:   amazon/similar_asin.py
	deleted:    amazon/user_data/chrome_data/BrowserMetrics-spare.pma
	deleted:    amazon/user_data/chrome_data/BrowserMetrics/BrowserMetrics-69FAEF07-558.pma
	deleted:    amazon/user_data/chrome_data/Crashpad/metadata
	deleted:    amazon/user_data/chrome_data/Crashpad/settings.dat
2026-05-13 16:51:29 +08:00
super
18b0c2d211 软件前端更新 2026-05-12 23:25:12 +08:00
super
e4e01c1686 软件前端更新 2026-05-12 23:25:12 +08:00
super
648e7d2f14 后台管理接口修复 后端BUG修复 2026-05-12 21:03:42 +08:00
super
0d78d63437 后台管理接口修复 后端BUG修复 2026-05-12 21:03:42 +08:00
super
289b1c67ce 提交货源采集更新 2026-05-10 01:00:08 +08:00
super
5c990e651e 提交货源采集更新 2026-05-10 01:00:08 +08:00
super
52095eb992 更新处理相关内容 2026-05-09 00:20:52 +08:00
super
a4e4745921 更新处理相关内容 2026-05-09 00:20:52 +08:00
super
a2d0bd3c1c Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-05-07 16:22:42 +08:00
super
683a3934fc 提交打包 2026-05-07 16:21:35 +08:00
铭坤
b0514c3e9a modified: amazon/price_match.py
modified:   config.py
2026-05-07 11:14:15 +08:00
铭坤
8be12ca9bc new file: requirements.txt 2026-05-07 11:00:44 +08:00
super
afeeb8327c 更新 2026-05-06 23:00:57 +08:00
super
4d2da94692 同步更新 2026-05-06 23:00:39 +08:00
super
b746812e23 更新外观模块 2026-05-06 22:41:26 +08:00
铭坤
3fbbf87c10 modified: .env
modified:   amazon/approve.py
	modified:   amazon/chrome_base.py
	modified:   amazon/detail_spider.py
	modified:   amazon/main.py
	modified:   amazon/price_match.py
	modified:   amazon/similar_asin.py
	modified:   amazon/tool.py
	new file:   amazon/user_data/chrome_data/BrowserMetrics-spare.pma
	new file:   amazon/user_data/chrome_data/BrowserMetrics/BrowserMetrics-69FAEF07-558.pma
2026-05-06 22:30:44 +08:00
super
c3810fdb7b 提交更新 2026-05-06 00:01:49 +08:00
super
b16c63225f 处理一些合并冲突 2026-05-05 14:11:35 +08:00
铭坤
45a45dc0fe Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin
modified:   amazon/detail_spider.py
	modified:   amazon/main.py
2026-05-04 19:17:11 +08:00
铭坤
c315bb350c new file: amazon/amazon_base.py
modified:   amazon/approve.py
	modified:   amazon/asin_status.py
	new file:   amazon/chrome_base.py
	modified:   amazon/del_brand.py
	modified:   amazon/detail_spider.py
	new file:   "amazon/detail_spider_\345\244\207\344\273\275.py"
	modified:   amazon/main.py
	modified:   amazon/match_action.py
	modified:   amazon/price_match.py
	new file:   amazon/similar_asin.py
	modified:   app.py
	new file:   assets/appearance-patent-Br1dtmol.css
	new file:   assets/appearance-patent-D1DgeYhe.css
	new file:   assets/delete-brand-BfMLVSQU.css
	new file:   assets/patrol-delete-BAzbYtgc.css
	new file:   assets/price-track-Hozgt_em.css
	new file:   assets/product-risk-CbX7bwZi.css
	new file:   assets/query-asin-B4RsOiza.css
	new file:   assets/shop-match-B2HWAgBY.css
	modified:   main.py
2026-05-04 19:15:50 +08:00
super
6d398b66bc 异常记录检测修改 2026-05-04 19:04:55 +08:00
super
934919c699 稳定组成这个文件优化 2026-05-02 16:40:13 +08:00
super
12d84a22e4 更新 2026-05-02 09:31:14 +08:00
super
dc457aff9e Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-05-02 09:29:43 +08:00
super
578847d596 提交更新 2026-05-02 09:28:26 +08:00
koko
6416c14f20 chore: serve bundled assets from app directory 2026-05-02 02:12:14 +08:00
koko
add6e6aeb5 chore: remove stale web assets 2026-05-02 02:07:13 +08:00
koko
7968cef8c3 fix: update patrol delete progress handling 2026-05-02 02:03:49 +08:00
koko
34148b5d8a Merge patrol delete tests and remove pycache artifacts 2026-05-01 22:46:11 +08:00
koko
79e408333d Merge branch 'dev/koko'
# Conflicts:
#	app/amazon/patrol_delete.py
2026-05-01 22:41:09 +08:00
koko
74a7cd22b0 Fix patrol delete condition handling 2026-05-01 22:36:37 +08:00
super
e1cca219b8 完善多店铺开启多窗口、外观进度条等 2026-05-01 21:09:54 +08:00
super
4419c5bacd 更新处理这个外观部分 2026-05-01 00:09:14 +08:00
铭坤
4eec0dd5a4 modified: amazon/__pycache__/price_match.cpython-39.pyc
modified:   amazon/price_match.py
	modified:   assets/appearance-patent.js
	modified:   assets/delete-brand.js
	modified:   assets/patrol-delete.js
	modified:   assets/price-track.js
	modified:   assets/product-risk.js
	modified:   assets/query-asin.js
	modified:   assets/shop-match.js
	modified:   new_web_source/delete-brand.html
2026-04-30 21:57:37 +08:00
koko
8e60f50616 Merge origin master into local master 2026-04-30 21:46:00 +08:00
koko
f85e3be40e Harden patrol delete logging 2026-04-30 21:43:15 +08:00
koko
7f9116f12a Merge patrol delete changes into master 2026-04-30 21:40:35 +08:00
super
11f0d2c745 修复商品风险多店铺问题 2026-04-30 21:31:11 +08:00
super
955a6439a0 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-30 11:38:13 +08:00
super
172917ac41 优化后台业务 2026-04-30 11:36:58 +08:00
铭坤
a5f6898d06 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin
app/amazon/__pycache__/approve.cpython-39.pyc
	app/amazon/__pycache__/asin_status.cpython-39.pyc
	app/amazon/__pycache__/del_brand.cpython-39.pyc
	app/amazon/__pycache__/match_action.cpython-39.pyc
	app/amazon/__pycache__/price_match.cpython-39.pyc
2026-04-30 11:00:02 +08:00
铭坤
83bc90ecf2 modified: amazon/__pycache__/approve.cpython-39.pyc
modified:   amazon/__pycache__/asin_status.cpython-39.pyc
	modified:   amazon/__pycache__/del_brand.cpython-39.pyc
	modified:   amazon/__pycache__/match_action.cpython-39.pyc
	modified:   amazon/__pycache__/price_match.cpython-39.pyc
	modified:   amazon/approve.py
	modified:   amazon/asin_status.py
	modified:   amazon/del_brand.py
	modified:   amazon/match_action.py
	modified:   amazon/price_match.py
	modified:   main.py
2026-04-30 10:52:52 +08:00
koko
cbfc29e953 Remove ts script 2026-04-28 22:41:56 +08:00
koko
b66465aa6a chore: update env 2026-04-28 22:40:39 +08:00
koko
fbd1a4dbea Remove README from master 2026-04-28 22:39:38 +08:00
koko
be8a8492a9 Remove tracked .rtk config 2026-04-28 22:38:17 +08:00
koko
b352eea21d Stop tracking remaining Python bytecode on master
Master still tracked a small set of generated __pycache__ artifacts after merging dev/koko. The ignore rules already match dev/koko, so this commit only removes the remaining bytecode files from the Git index while leaving local files on disk.

Constraint: dev/koko .gitignore already contains recursive __pycache__ ignore coverage

Rejected: Delete local cache directories | repository tracking cleanup only requires --cached removal

Confidence: high

Scope-risk: narrow

Tested: git ls-files shows no remaining paths containing __pycache__/

Not-tested: Application runtime; generated bytecode tracking only
2026-04-28 22:33:27 +08:00
koko
51bf0aaa8c Merge dev/koko into master preserving master conflict behavior
Conflict resolution kept master-side changes for non-patrol-delete files as requested. No app/amazon/patrol_delete.py conflict was present. Existing dev/koko assets were retained so /assets resources required by new_web_source pages resolve in the Flask app.

Constraint: User requested authentication-requiring git operations be reported as commands only

Rejected: Rework non-conflicting dev/koko changes during merge | scope was conflict resolution, not feature review

Confidence: high

Scope-risk: moderate

Tested: uv run python -c Flask test client returned 200 for /new_web_source/patrol-delete.html, /assets/patrol-delete.js, /assets/pywebview-C66x_2Dh.js

Tested: uv run python -m py_compile amazon\\patrol_delete.py amazon\\detail_spider.py amazon\\price_match.py main.py

Tested: uv run --group dev pytest tests\\test_patrol_delete.py tests\\test_amazon_base.py

Not-tested: authenticated Git push/pull; Java backend build
2026-04-28 22:31:38 +08:00
koko
28a752265c fix: brand html 2026-04-28 22:14:41 +08:00
koko
1d9d138423 fix: update frontend 2026-04-28 21:42:30 +08:00
铭坤
8c5edc6ad6 new file: assets/_plugin-vue_export-helper-Dh4yMZWw.js
new file:   assets/_plugin-vue_export-helper-xvHHTGU_.css
	new file:   assets/appearance-patent-BNTrbLJh.css
	new file:   assets/appearance-patent-BPCrG75P.css
	new file:   assets/appearance-patent.js
	new file:   assets/brand-1nUVsWrm.js
	new file:   assets/brand-5-W0Wonr.js
	new file:   assets/brand-BATjYfEL.js
	new file:   assets/brand-BZije8D7.js
	new file:   assets/brand-C0Q5LZ0P.js
	new file:   assets/brand-CFZZtTv8.js
	new file:   assets/brand-CKgrwtni.js
	new file:   assets/brand-CRmRvyGD.js
	new file:   assets/brand-CXHemZwJ.js
	new file:   assets/brand-DBpLCgAF.js
	new file:   assets/brand-DRn3N4FA.js
	new file:   assets/brand-DZ5_PGnw.js
	new file:   assets/brand-DysFlQbZ.js
	new file:   assets/brand-UU-ckLq6.js
	new file:   assets/convert-1dkFPbKa.css
	new file:   assets/convert-6b0yq-M0.css
	new file:   assets/convert-7wWJ02Tw.css
	new file:   assets/convert-BKSNvX8i.css
	new file:   assets/convert-BxkpMHO9.css
	new file:   assets/convert-CQ22EBpm.css
	new file:   assets/convert-Cfr3mUjs.css
	new file:   assets/convert-D4Yv2oRy.css
	new file:   assets/convert-DwbQug2y.css
	new file:   assets/dedupe-B0eqpVpj.css
	new file:   assets/dedupe-BHE_BCAJ.css
	new file:   assets/dedupe-BpNHwt51.css
	new file:   assets/dedupe-CS18ls2z.css
	new file:   assets/dedupe-DNlVfFj-.css
	new file:   assets/dedupe-Day_nGxq.css
	new file:   assets/dedupe-DyzjKbsT.css
	new file:   assets/dedupe-DzFvTfoj.css
	new file:   assets/dedupe-rhGu1E8E.css
	new file:   assets/delete-brand-BNvh4ru4.css
	new file:   assets/delete-brand-BP3XWKAC.css
	new file:   assets/delete-brand-BRIDBaoM.css
	new file:   assets/delete-brand-BcZomrN3.css
	new file:   assets/delete-brand-BgPrpPsj.css
	new file:   assets/delete-brand-ByeMdpEk.css
	new file:   assets/delete-brand-C-Rj3NMH.css
	new file:   assets/delete-brand-CDZTH98q.css
	new file:   assets/delete-brand-CS44Bk-y.css
	new file:   assets/delete-brand-CWLpe7lu.css
	new file:   assets/delete-brand-CdxGWtvK.css
	new file:   assets/delete-brand-Chu8awlt.css
	new file:   assets/delete-brand-CndjrFFS.css
	new file:   assets/delete-brand-CpNTe9WM.css
	new file:   assets/delete-brand-D7t16Ukk.css
	new file:   assets/delete-brand-DE0hLEak.css
	new file:   assets/delete-brand-DLyg-H8J.css
	new file:   assets/delete-brand-DO6pdlC3.css
	new file:   assets/delete-brand-DXSeQxXc.css
	new file:   assets/delete-brand-DZ6zYt8o.css
	new file:   assets/delete-brand-DfRLfrtm.css
	new file:   assets/delete-brand-Dl7T0pmm.css
	new file:   assets/delete-brand-Dt_MS9as.css
	new file:   assets/delete-brand-DvND7vBL.css
	new file:   assets/delete-brand-_jGFWTAn.css
	new file:   assets/el-alert-Ct0RpqUD.css
	new file:   assets/el-input-CezJelDw.css
	new file:   assets/el-input-xaztPnzw.css
	new file:   assets/el-table-column-Cy4YJvw0.css
	new file:   assets/listingFilters-B6-M_FNM.js
	new file:   assets/listingFilters-BmlAzw7M.css
	new file:   assets/listingFilters-BpGOU_pJ.js
	new file:   assets/listingFilters-CK58rX4v.css
	new file:   assets/patrol-delete-BPu1AAPM.css
	new file:   assets/patrol-delete-BctopIU4.css
	new file:   assets/patrol-delete-CSekhzSS.css
	new file:   assets/patrol-delete-DIvFZUTh.css
	new file:   assets/patrol-delete.js
	new file:   assets/price-track-1SjcUfP4.css
	new file:   assets/price-track-85NbckiJ.css
	new file:   assets/price-track-B5FVfJb_.css
	new file:   assets/price-track-BO5PtL0W.css
	new file:   assets/price-track-BccJT4Qr.css
	new file:   assets/price-track-BlSD-0un.css
	new file:   assets/price-track-BlVVl2eV.css
	new file:   assets/price-track-BwgEuRxe.css
	new file:   assets/price-track-CWQD_A7D.css
	new file:   assets/price-track-DNTcdRH8.css
	new file:   assets/price-track-Dv4EZPhf.css
	new file:   assets/price-track-Ji5JAoeR.css
	new file:   assets/price-track-RLJUgMl_.css
	new file:   assets/price-track-z8RU_FlW.css
	new file:   assets/price-track.js
	new file:   assets/product-risk-1G_ztqNY.css
	new file:   assets/product-risk-A5bhfaTH.css
	new file:   assets/product-risk-BBMcuGAw.css
	new file:   assets/product-risk-C3Yn2wAL.css
	new file:   assets/product-risk-CG-1Iq4P.css
	new file:   assets/product-risk-COeM91v5.css
	new file:   assets/product-risk-ClWs7d57.css
	new file:   assets/product-risk-CtpNfape.css
	new file:   assets/product-risk-DBsBE_mg.css
	new file:   assets/product-risk-DRYaWmI2.css
	new file:   assets/product-risk-Db49F4J5.css
	new file:   assets/product-risk-DoO8ZHM0.css
	new file:   assets/product-risk-DpSMKBCA.css
	new file:   assets/product-risk-DqFPFWEv.css
	new file:   assets/product-risk-cfL3Ch5-.css
	new file:   assets/product-risk-eXMv239F.css
	new file:   assets/product-risk-un2cxu--.css
	new file:   assets/product-risk.js
	new file:   assets/pywebview-3PePjYSu.js
	new file:   assets/pywebview-8G_9KMXd.css
	new file:   assets/pywebview-9YBa--7x.js
	new file:   assets/pywebview-B6ZNXusn.css
	new file:   assets/pywebview-BCPdlDdb.js
	new file:   assets/pywebview-BLoeiUcF.css
	new file:   assets/pywebview-BRrok0pS.js
	new file:   assets/pywebview-BT0f6KST.css
	new file:   assets/pywebview-BbIq-QPS.css
	new file:   assets/pywebview-Bc09DixN.js
	new file:   assets/pywebview-Bl6M97uH.js
	new file:   assets/pywebview-BnpJyyGZ.js
	new file:   assets/pywebview-Bt854mYs.js
	new file:   assets/pywebview-C5RGB8QW.css
	new file:   assets/pywebview-C66x_2Dh.js
	new file:   assets/pywebview-C8Bik-Sc.js
	new file:   assets/pywebview-CGwF8TuM.js
	new file:   assets/pywebview-CJLylbJu.js
	new file:   assets/pywebview-CeWJDVeG.js
	new file:   assets/pywebview-Chz98E9c.css
	new file:   assets/pywebview-ClWy2SbE.js
	new file:   assets/pywebview-Cq_E2BnJ.js
	new file:   assets/pywebview-Cs1Kot1q.js
	new file:   assets/pywebview-D-PKWjUg.js
	new file:   assets/pywebview-D-mMH8F6.css
	new file:   assets/pywebview-D4gpiFjY.js
	new file:   assets/pywebview-D74jYk6P.css
	new file:   assets/pywebview-D7_PdvyM.js
	new file:   assets/pywebview-D808cNhy.js
	new file:   assets/pywebview-DVgP2-kW.js
	new file:   assets/pywebview-Dee3nQjE.js
	new file:   assets/pywebview-DiP0HdY6.js
	new file:   assets/pywebview-Dp5dN8OO.css
	new file:   assets/pywebview-DuyK2jB1.js
	new file:   assets/pywebview-IqgMfeBe.css
	new file:   assets/pywebview-MkjZQlBu.css
	new file:   assets/pywebview-Q4glvYu5.css
	new file:   assets/pywebview-Rm5Hhqpf.js
	new file:   assets/pywebview-V_UIajJm.js
	new file:   assets/pywebview-hj_EXwAP.css
	new file:   assets/pywebview-ij4pgMq8.css
	new file:   assets/pywebview-leZCedEt.js
	new file:   assets/query-asin-B0yMJ9Or.css
	new file:   assets/query-asin-B5e5m-vs.css
	new file:   assets/query-asin-Bv7Jtgqa.css
	new file:   assets/query-asin-DZwJfwHU.css
	new file:   assets/query-asin.js
	new file:   assets/shop-match-Banz7DWL.css
	new file:   assets/shop-match-BiZP8IGM.css
	new file:   assets/shop-match-BrrGkYV7.css
	new file:   assets/shop-match-CVFsXUvd.css
	new file:   assets/shop-match-CfGJEHz0.css
	new file:   assets/shop-match-CfGWwDtw.css
	new file:   assets/shop-match-D0DGETCs.css
	new file:   assets/shop-match-DFEKOpvV.css
	new file:   assets/shop-match-DNFA8ceR.css
	new file:   assets/shop-match-DhT1CvMP.css
	new file:   assets/shop-match-L0xHTaQ2.css
	new file:   assets/shop-match-MsxXMG_G.css
	new file:   assets/shop-match-g3eRKhTP.css
	new file:   assets/shop-match.js
	new file:   assets/split-BoBVrLdC.css
	new file:   assets/split-BsJfbBqn.css
	new file:   assets/split-CRUIYKS6.css
	new file:   assets/split-DAa_Usoh.css
	new file:   assets/split-DidKJ84l.css
	new file:   assets/split-RNZYJ-zZ.css
	new file:   assets/split-R_8V5xgh.css
	new file:   assets/split-_O6NHqDj.css
	new file:   assets/split-mwLUJT1K.css
	new file:   assets/zh-cn-BjWx61Lu.js
	new file:   assets/zh-cn-C0iS6aoJ.js
	new file:   assets/zh-cn-CuE9Zzgz.js
	new file:   assets/zh-cn-DyKmB71i.js
	new file:   assets/zh-cn-Z9PBnl-n.js
2026-04-28 21:36:49 +08:00
铭坤
ee6fb5d33a modified: amazon/__pycache__/detail_spider.cpython-39.pyc
modified:   amazon/__pycache__/price_match.cpython-39.pyc
	modified:   amazon/detail_spider.py
	modified:   amazon/price_match.py
	modified:   blueprints/__pycache__/admin.cpython-39.pyc
	modified:   blueprints/__pycache__/brand.cpython-39.pyc
	modified:   blueprints/__pycache__/main.cpython-39.pyc
	modified:   brand_spider/__pycache__/main.cpython-39.pyc
	modified:   main.py
	modified:   web_source/brand.html
2026-04-28 21:11:48 +08:00
koko
cd2b437be5 Stop tracking generated Python bytecode
Generated __pycache__ files were already ignored in some paths but remained tracked in the repository, which caused merge noise and binary conflicts. This removes the tracked bytecode from the index and adds an explicit recursive ignore rule so future Python runs do not reintroduce them.

Constraint: Existing repository history already tracked pycache files across app, backend, and source_code

Rejected: Delete local pycache directories from disk | only repository tracking needed to be removed

Confidence: high

Scope-risk: narrow

Directive: Do not force-add __pycache__ or *.pyc files unless there is a documented runtime packaging reason

Tested: git ls-files shows no remaining paths containing __pycache__/

Not-tested: Application runtime; change only affects tracked generated artifacts and ignore rules
2026-04-28 20:57:53 +08:00
koko
a504e7a13b Merge master into dev/koko after patrol delete fixes
The merge keeps dev/koko patrol-delete asset fallback behavior while accepting the latest master feature work. Conflicts were limited to ignore rules, local environment/config artifacts, tracked pycache binaries, and the Flask main blueprint.

Constraint: master and dev/koko both edited app/blueprints/main.py around brand and asset serving

Rejected: Drop dev/koko asset fallback logic | patrol delete and rebuilt Vite assets still need hashed asset resolution across app/new_web_source locations

Confidence: medium

Scope-risk: broad

Directive: app/.env remains a tracked local-config file in this repository; do not print or normalize secrets during conflict handling

Tested: uv run python -m py_compile blueprints\\main.py amazon\\base.py amazon\\main.py

Tested: uv run --group dev pytest tests\\test_amazon_base.py tests\\test_patrol_delete.py

Not-tested: Full backend-java/frontend-vue test suites
2026-04-28 20:55:27 +08:00
koko
325687d532 Stabilize patrol delete automation against delayed Amazon UI
The patrol-delete flow now waits for country and listing-status controls before acting, uses the dedicated PatrolDeleteTask entry point, and includes the browser helper scripts required by the bulk-delete path. Tests cover delayed dropdown readiness, payload rejection/retry behavior, complete-draft normalization, filtered bulk deletion, and asset fallback lookup.

Constraint: Amazon listing UI exposes status controls through dynamic KAT components and delayed option rendering

Rejected: Keep row-by-row delete flow | bulk selection is the current implemented path and is covered by the added helper scripts

Confidence: high

Scope-risk: moderate

Directive: Do not remove the patrol_delete JS helper files without checking app/amazon/patrol_delete.py script loading

Tested: uv run --group dev pytest tests\\test_amazon_base.py tests\\test_patrol_delete.py

Tested: uv run python -m py_compile amazon\\base.py amazon\\main.py blueprints\\main.py

Not-tested: Real Amazon Seller Central browser session
2026-04-28 20:46:50 +08:00
super
9760d1171c 提交跟新 2026-04-28 17:16:04 +08:00
super
d42ed57119 提交暂存 2026-04-28 16:57:05 +08:00
super
18208f691c 更新内容 2026-04-28 16:56:53 +08:00
super
c06a4d0e06 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-28 15:33:42 +08:00
super
f21934e55b 合并错误信息 2026-04-28 15:25:34 +08:00
铭坤
515135c120 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-28 15:23:55 +08:00
铭坤
42bdd59c71 modified: .env
modified:   amazon/__pycache__/approve.cpython-39.pyc
	modified:   amazon/__pycache__/detail_spider.cpython-39.pyc
	modified:   amazon/__pycache__/price_match.cpython-39.pyc
	modified:   amazon/approve.py
	modified:   amazon/detail_spider.py
	modified:   amazon/price_match.py
	modified:   assets/convert.js
	modified:   assets/dedupe.js
	modified:   assets/delete-brand.js
	modified:   assets/split.js
	modified:   blueprints/__pycache__/admin.cpython-39.pyc
	modified:   blueprints/__pycache__/brand.cpython-39.pyc
	modified:   blueprints/__pycache__/main.cpython-39.pyc
	modified:   blueprints/main.py
	modified:   brand_spider/__pycache__/main.cpython-39.pyc
	modified:   new_web_source/convert.html
	modified:   new_web_source/dedupe.html
	modified:   new_web_source/delete-brand.html
	modified:   new_web_source/split.html
	modified:   tool/__pycache__/devices.cpython-39.pyc
2026-04-28 15:21:57 +08:00
koko
8be85fd332 Expose patrol delete progress in logs
Patrol delete can spend a long time switching countries, reading listing states, and confirming deletes, so the flow now emits concise progress logs around each high-value step. The logs avoid full DOM dumps outside existing diagnostics and focus on task, shop, country, status, deletion counts, and callback boundaries.

Constraint: Existing patrol delete runs are browser-driven and need observable progress without changing deletion behavior
Rejected: Log every DOM probe | too noisy for routine task monitoring
Confidence: high
Scope-risk: narrow
Tested: uv run --group dev pytest tests\\test_patrol_delete.py
Tested: uv run --group dev ruff check amazon\\patrol_delete.py
Tested: uv run python -m py_compile amazon\\patrol_delete.py
Not-tested: Live Seller Central browser run
2026-04-28 01:23:58 +08:00
super
e4bf104ae2 架构更改完成 2026-04-27 18:45:56 +08:00
super
5105bf7049 增加rufts 2026-04-27 18:45:33 +08:00
koko
08997e9e20 Rename patrol delete tests
Keep the patrol deletion test module aligned with the renamed patrol_delete implementation and update test function names so future searches no longer point at the old product module name.

Constraint: This is a naming-only follow-up to the patrol_delete module rename

Confidence: high

Scope-risk: narrow

Tested: uv run pytest tests/test_patrol_delete.py
2026-04-27 17:59:46 +08:00
koko
9146d17625 Align patrol delete module naming
Rename the patrol deletion implementation and browser helper scripts away from the generic product name so the filesystem matches the queued patrol-delete task domain. Update imports, script loading, and tests to use the patrol_delete module path.

Constraint: Existing task class remains ProductTask to avoid broad API churn beyond the requested file and script naming

Rejected: Rename ProductTask class now | would expand the change into a larger API update across tests and task dispatch

Confidence: high

Scope-risk: narrow

Directive: Keep patrol-delete script paths under app/amazon/scripts/patrol_delete when adding new helper scripts

Tested: uv run pytest tests/test_product.py

Not-tested: Remote push before this commit due previous SSH publickey failure
2026-04-27 17:58:40 +08:00
koko
1aca3d6152 Bring Amazon collection tasks into dev/koko
Merge origin/master into dev/koko while preserving the patrol delete task added on dev/koko. The task dispatcher now recognizes both patrol deletion and appearance patent collection, and the master-side SpiderTask implementation is included.

Constraint: origin fetch and push over SSH are blocked in this environment by Gitee publickey authentication

Rejected: Overwrite dev/koko with master | would drop the patrol delete branch work

Confidence: medium

Scope-risk: moderate

Directive: Do not remove either patrol-delete-run or appearance-patent-run without checking queued task producers

Tested: python -m py_compile app/amazon/approve.py app/amazon/asin_status.py app/amazon/detail_spider.py app/amazon/main.py app/amazon/match_action.py app/amazon/price_match.py

Not-tested: Remote push, blocked by SSH publickey authentication
2026-04-27 17:36:30 +08:00
super
052bb31aea Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-27 09:20:04 +08:00
super
225b525ba1 后端架构更新 2026-04-27 09:18:10 +08:00
koko
e1543416e3 feat: patrol delete 2026-04-27 00:23:44 +08:00
铭坤
0b57aea5f6 modified: amazon/__pycache__/approve.cpython-39.pyc
modified:   amazon/__pycache__/asin_status.cpython-39.pyc
	new file:   amazon/__pycache__/detail_spider.cpython-39.pyc
	modified:   amazon/__pycache__/main.cpython-39.pyc
	modified:   amazon/__pycache__/match_action.cpython-39.pyc
	modified:   amazon/__pycache__/price_match.cpython-39.pyc
	modified:   amazon/approve.py
	modified:   amazon/asin_status.py
	new file:   amazon/detail_spider.py
	modified:   amazon/main.py
	modified:   amazon/match_action.py
	modified:   amazon/price_match.py
	modified:   assets/convert.js
	modified:   assets/dedupe.js
	modified:   assets/delete-brand.js
	modified:   assets/split.js
	modified:   new_web_source/convert.html
	modified:   new_web_source/dedupe.html
	modified:   new_web_source/delete-brand.html
	modified:   new_web_source/split.html
	modified:   web_source/brand.html
2026-04-27 00:21:37 +08:00
koko
a04c2f9a19 feat: update amazon driver 2026-04-25 17:59:12 +08:00
koko
c352c34501 feat: update amazon driver 2026-04-25 17:58:28 +08:00
koko
5c9671df44 feat: update driver BaseClass 2026-04-25 17:14:23 +08:00
koko
8a8d3c5cd9 Merge branch 'master' into dev/koko 2026-04-25 15:47:02 +08:00
铭坤
d272afae1c modified: amazon/__pycache__/approve.cpython-39.pyc
new file:   amazon/__pycache__/asin_status.cpython-39.pyc
	modified:   amazon/__pycache__/main.cpython-39.pyc
	modified:   amazon/__pycache__/match_action.cpython-39.pyc
	modified:   amazon/__pycache__/price_match.cpython-39.pyc
	modified:   amazon/approve.py
	modified:   amazon/asin_status.py
	modified:   amazon/main.py
	modified:   amazon/match_action.py
	modified:   amazon/price_match.py
	modified:   web_source/brand.html
	modified:   web_source/templates_backup/brand.html
2026-04-24 23:00:59 +08:00
super
03e697f5d3 后台查询Asin文档更新 2026-04-24 15:14:14 +08:00
super
c502afb588 完成查询asin开发 2026-04-23 20:48:57 +08:00
koko
3fada5d198 Merge branch 'master' into dev/koko 2026-04-23 11:49:50 +00:00
super
c0fdea6570 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-23 15:25:49 +08:00
super
0391cb223f 完成后端架构重构等 2026-04-23 15:25:41 +08:00
koko
b25111e4b4 Merge remote-tracking branch 'origin/master' into dev/koko 2026-04-23 03:16:15 +00:00
铭坤
2f2db4986e modified: app/amazon/__pycache__/approve.cpython-39.pyc
modified:   app/amazon/__pycache__/del_brand.cpython-39.pyc
	modified:   app/amazon/__pycache__/match_action.cpython-39.pyc
	modified:   app/amazon/__pycache__/price_match.cpython-39.pyc
	modified:   app/amazon/approve.py
	modified:   app/amazon/del_brand.py
	modified:   app/amazon/match_action.py
	modified:   app/amazon/price_match.py
	modified:   app/assets/convert.js
	modified:   app/assets/dedupe.js
	modified:   app/assets/delete-brand.js
	modified:   app/assets/split.js
	modified:   app/new_web_source/convert.html
	modified:   app/new_web_source/dedupe.html
	modified:   app/new_web_source/delete-brand.html
	modified:   app/new_web_source/split.html
2026-04-23 10:41:07 +08:00
koko
6f970b3783 Clarify the current automation example in onboarding docs
Update the README so new contributors start from the latest patrol-delete automation flow instead of treating delete-brand as the primary example. Keep the older delete-brand path as contrast for the file-driven workflow.

Constraint: Recent repository changes made patrol-delete the newest automation module, so the onboarding example needed to match current development reality
Rejected: Leave delete-brand as the primary example | would mislead readers about the latest automation entry point
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Revisit this section when a newer automation module replaces patrol-delete as the primary example
Tested: README diff review
Not-tested: lint, typecheck, unit/integration tests not run (documentation-only change)
2026-04-22 08:33:02 +00:00
铭坤
95d5c82474 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-22 15:58:46 +08:00
铭坤
87507708ce new file: admin.html
new file:   "brand - \345\211\257\346\234\254.html"
	new file:   "brand-\346\227\247.html"
	new file:   brand.html
	new file:   home.html
	new file:   index.html
	new file:   login.html
2026-04-22 15:57:56 +08:00
koko
c6ae7ca170 Document repo structure and ignore local OMX state
Add a root README that explains the active runtime layers and onboarding path, and keep local OMX session state out of version control so developer tooling does not interfere with pulls.

Constraint: Local OMX state is developer-specific and should not block branch sync or appear in shared history
Rejected: Keep onboarding notes untracked | easy to lose and hard to share with the team
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep the README aligned with the active app/backend-java/frontend-vue execution path; revisit if the runtime architecture changes
Tested: git diff review; git status after staging
Not-tested: lint, typecheck, unit/integration tests not run (docs and ignore rules only)
2026-04-22 07:16:04 +00:00
super
6894f9cc57 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-22 01:09:51 +08:00
super
524d8763ce 处理后台跳转错误 2026-04-22 01:09:47 +08:00
super
1e845a1510 处理后台管理系统、修复BUG、处理权限 2026-04-22 01:09:28 +08:00
铭坤
0341838d19 modified: amazon/__pycache__/approve.cpython-39.pyc
modified:   amazon/__pycache__/main.cpython-39.pyc
	modified:   amazon/__pycache__/match_action.cpython-39.pyc
	new file:   amazon/__pycache__/price_match.cpython-39.pyc
	modified:   amazon/__pycache__/tool.cpython-39.pyc
	modified:   amazon/approve.py
	new file:   amazon/asin_status.py
	modified:   amazon/main.py
	modified:   amazon/price_match.py
	new file:   "amazon/price_match_\346\227\247.py"
	modified:   amazon/tool.py
	modified:   assets/convert.js
	modified:   assets/dedupe.js
	modified:   assets/delete-brand.js
	modified:   assets/split.js
	modified:   new_web_source/convert.html
	modified:   new_web_source/dedupe.html
	modified:   new_web_source/delete-brand.html
	modified:   new_web_source/split.html
	deleted:    web_source/admin.html
	deleted:    "web_source/brand - \345\211\257\346\234\254.html"
	deleted:    "web_source/brand-\346\227\247.html"
	deleted:    web_source/brand.html
	deleted:    web_source/home.html
	deleted:    web_source/index.html
	deleted:    web_source/login.html
2026-04-22 00:51:41 +08:00
super
ea35273597 改造后端服务,处理后台管理新增组(部门概念) 2026-04-22 00:32:58 +08:00
super
72c8167472 更新跟价相关内容修改 2026-04-20 00:41:49 +08:00
super
9b1138c83e Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-19 15:47:37 +08:00
super
dc5e23892f 提交改造app权限 2026-04-19 15:47:20 +08:00
铭坤
4f8cbc3e38 new file: update/PyQt5/Qt/.keep_dir.txt
new file:   update/PyQt5/Qt5/.keep_dir.txt
	new file:   update/PyQt5/QtCore.pyd
	new file:   update/PyQt5/QtGui.pyd
	new file:   update/PyQt5/QtWidgets.pyd
	new file:   update/PyQt5/qt-plugins/iconengines/qsvgicon.dll
	new file:   update/PyQt5/qt-plugins/imageformats/qgif.dll
	new file:   update/PyQt5/qt-plugins/imageformats/qicns.dll
	new file:   update/PyQt5/qt-plugins/imageformats/qico.dll
	new file:   update/PyQt5/qt-plugins/imageformats/qjpeg.dll
	new file:   update/PyQt5/qt-plugins/imageformats/qsvg.dll
	new file:   update/PyQt5/qt-plugins/imageformats/qtga.dll
	new file:   update/PyQt5/qt-plugins/imageformats/qtiff.dll
	new file:   update/PyQt5/qt-plugins/imageformats/qwbmp.dll
	new file:   update/PyQt5/qt-plugins/imageformats/qwebp.dll
	new file:   update/PyQt5/qt-plugins/mediaservice/dsengine.dll
	new file:   update/PyQt5/qt-plugins/mediaservice/qtmedia_audioengine.dll
	new file:   update/PyQt5/qt-plugins/mediaservice/wmfengine.dll
	new file:   update/PyQt5/qt-plugins/platforms/qminimal.dll
	new file:   update/PyQt5/qt-plugins/platforms/qoffscreen.dll
	new file:   update/PyQt5/qt-plugins/platforms/qwebgl.dll
	new file:   update/PyQt5/qt-plugins/platforms/qwindows.dll
	new file:   update/PyQt5/qt-plugins/platformthemes/qxdgdesktopportal.dll
	new file:   update/PyQt5/qt-plugins/printsupport/windowsprintersupport.dll
	new file:   update/PyQt5/qt-plugins/styles/qwindowsvistastyle.dll
	new file:   update/PyQt5/sip.pyd
	new file:   update/_bz2.pyd
	new file:   update/_ctypes.pyd
	new file:   update/_decimal.pyd
	new file:   update/_elementtree.pyd
	new file:   update/_hashlib.pyd
	new file:   update/_hashlib.zip
	new file:   update/_lzma.pyd
	new file:   update/_socket.pyd
	new file:   update/libcrypto-1_1-x64.dll
	new file:   update/libeay32.dll
	new file:   update/libffi-7.dll
	new file:   update/msvcp140.dll
	new file:   update/msvcp140_1.dll
	new file:   update/psutil/_psutil_windows.pyd
	new file:   update/pyexpat.pyd
	new file:   update/python3.dll
	new file:   update/python39.dll
	new file:   update/qt5core.dll
	new file:   update/qt5dbus.dll
	new file:   update/qt5gui.dll
	new file:   update/qt5multimedia.dll
	new file:   update/qt5network.dll
	new file:   update/qt5printsupport.dll
	new file:   update/qt5qml.dll
	new file:   update/qt5qmlmodels.dll
	new file:   update/qt5quick.dll
	new file:   update/qt5svg.dll
	new file:   update/qt5websockets.dll
	new file:   update/qt5widgets.dll
	new file:   update/select.pyd
	new file:   update/ssleay32.dll
	new file:   update/unicodedata.pyd
	new file:   update/update.exe
	new file:   update/vcruntime140.dll
	new file:   update/vcruntime140_1.dll
2026-04-19 15:39:46 +08:00
铭坤
3ffbb2b004 modified: ali_oss.py
modified:   config.py
	modified:   main.py
	new file:   winsrc.bat
2026-04-19 15:38:09 +08:00
super
34ed15a9bd Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-19 15:06:13 +08:00
super
4b6295dd44 改造后台权限和APP权限 2026-04-19 15:06:09 +08:00
supernijia
01116d5607 删除文件 desktop 2026-04-18 16:23:13 +00:00
铭坤
07904a29aa modified: amazon/match_action.py
modified:   amazon/price_match.py
	modified:   config.py
2026-04-19 00:18:28 +08:00
铭坤
8d683a791d Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-19 00:18:02 +08:00
super
ef0e0df0ac 完成匹配、跟价、权限部分 2026-04-17 12:52:16 +08:00
铭坤
ac07416352 modified: amazon/__pycache__/approve.cpython-39.pyc
modified:   amazon/__pycache__/del_brand.cpython-39.pyc
	modified:   amazon/__pycache__/match_action.cpython-39.pyc
	modified:   amazon/approve.py
	modified:   amazon/del_brand.py
	modified:   amazon/match_action.py
	new file:   amazon/price_match.py
	modified:   main.py
2026-04-17 10:04:24 +08:00
super
025ca6d4fd 后台管理跳过asin、后台相关数据做数据隔离 2026-04-14 10:04:50 +08:00
951a353881 提交任务更新 2026-04-13 20:07:00 +08:00
super
eb04caccf1 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-13 13:02:45 +08:00
super
cd84def61d 提交工作更新 2026-04-13 13:02:41 +08:00
铭坤
d8098b0378 modified: app/amazon/__pycache__/del_brand.cpython-39.pyc
modified:   app/amazon/__pycache__/match_action.cpython-39.pyc
	modified:   app/amazon/del_brand.py
	modified:   app/amazon/match_action.py
	modified:   app/assets/delete-brand.js
	modified:   app/new_web_source/delete-brand.html
	new file:   "app/web_source/brand - \345\211\257\346\234\254.html"
2026-04-12 20:52:26 +08:00
super
62d30ec190 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-12 17:15:54 +08:00
super
d0d3c6ee67 完成菜单权限添加和软件菜单改造 2026-04-12 17:15:15 +08:00
铭坤
21b6b5b270 modified: app/amazon/__pycache__/approve.cpython-39.pyc
modified:   app/amazon/__pycache__/del_brand.cpython-39.pyc
	modified:   app/amazon/__pycache__/main.cpython-39.pyc
	modified:   app/amazon/__pycache__/match_action.cpython-39.pyc
	modified:   app/amazon/__pycache__/tool.cpython-39.pyc
	modified:   app/amazon/approve.py
	modified:   app/amazon/del_brand.py
	modified:   app/amazon/main.py
	modified:   app/amazon/match_action.py
	modified:   app/amazon/tool.py
	modified:   app/assets/delete-brand.js
	modified:   app/new_web_source/delete-brand.html
2026-04-11 23:00:55 +08:00
super
365050b890 提交bug修复 后台权限管理 2026-04-11 22:25:14 +08:00
super
73392a9b83 修复BUG,完善匹配店铺 2026-04-11 00:57:34 +08:00
super
c47c03fde4 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-10 19:35:08 +08:00
super
86fd9475e4 匹配回收,bug修复 2026-04-10 19:33:58 +08:00
铭坤
2d0d1d3461 modified: app/amazon/__pycache__/approve.cpython-39.pyc
modified:   app/amazon/__pycache__/del_brand.cpython-39.pyc
	modified:   app/amazon/__pycache__/main.cpython-39.pyc
	new file:   app/amazon/__pycache__/match_action.cpython-39.pyc
	new file:   app/amazon/__pycache__/tool.cpython-39.pyc
	new file:   "app/amazon/approve - \345\211\257\346\234\254.py"
	modified:   app/amazon/approve.py
	modified:   app/amazon/del_brand.py
	modified:   app/amazon/main.py
	new file:   app/amazon/match_action.py
	new file:   app/amazon/tool.py
	modified:   app/main.py
	modified:   app/web_source/brand.html
2026-04-10 16:51:52 +08:00
super
ccaec4a984 完善后台管理店铺搜索 2026-04-08 21:08:52 +08:00
super
169ba7edb6 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-08 16:14:30 +08:00
super
0327d1cc51 修复接口内容缺失 2026-04-08 16:14:26 +08:00
铭坤
e02ddab599 modified: amazon/__pycache__/approve.cpython-39.pyc
modified:   amazon/__pycache__/del_brand.cpython-39.pyc
	modified:   amazon/__pycache__/main.cpython-39.pyc
	modified:   amazon/approve.py
	new file:   amazon/backend_api.py
	modified:   amazon/del_brand.py
	modified:   amazon/main.py
	modified:   blueprints/main.py
	modified:   main.py
2026-04-08 16:03:48 +08:00
super
336338cff5 完善后台接口字段 2026-04-07 17:57:03 +08:00
super
acff31e652 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-07 14:50:09 +08:00
super
ad304a6880 完成店铺增删改查 2026-04-07 14:50:06 +08:00
铭坤
cb34788e53 modified: "web_source/brand-\346\227\247.html"
modified:   web_source/brand.html
	modified:   web_source/templates_backup/brand.html
2026-04-07 14:10:33 +08:00
铭坤
13b0ffb5d8 new file: amazon/__pycache__/approve.cpython-39.pyc
modified:   amazon/__pycache__/del_brand.cpython-39.pyc
	modified:   amazon/__pycache__/main.cpython-39.pyc
	new file:   amazon/approve.py
	modified:   amazon/del_brand.py
	modified:   amazon/main.py
	modified:   main.py
2026-04-07 14:09:42 +08:00
super
c9947c4ac8 更改下载方法名 2026-04-06 22:26:44 +08:00
super
afffb7aad2 更改下载类型处理 2026-04-06 16:58:58 +08:00
super
9720453ac4 更新商品风险处理 2026-04-05 22:04:21 +08:00
super
1ff6d5220e 新增删除增加公司名 2026-04-04 00:10:08 +08:00
super
46d91fd8ca 提交接口进度完善 2026-04-03 00:53:52 +08:00
super
a24db2f73c Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-02 23:29:19 +08:00
super
ca9537f862 测试多任务暂存 2026-04-02 23:29:15 +08:00
铭坤
94ae7e68ff modified: app/amazon/__pycache__/del_brand.cpython-39.pyc
modified:   app/amazon/__pycache__/main.cpython-39.pyc
	modified:   app/amazon/main.py
	modified:   app/main.py
2026-04-02 23:03:13 +08:00
super
94e1930538 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-02 20:37:38 +08:00
super
5bbe5a3077 修改多文件 2026-04-02 20:37:35 +08:00
铭坤
8cead01270 modified: app/amazon/del_brand.py
backend/blueprints/get_resource.py
2026-04-02 20:35:50 +08:00
铭坤
99fc14b9f9 modified: backend/app.py
backend/blueprints/get_resource.py
2026-04-02 20:35:12 +08:00
super
d3671206e1 完善后端轮询记忆紫鸟店铺逻辑 2026-04-02 12:26:48 +08:00
super
0a8202788b Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-02 01:15:54 +08:00
super
855f1affd9 紫鸟查询店铺更新 2026-04-02 01:15:50 +08:00
铭坤
5a294e5c85 new file: app/amazon/__pycache__/del_brand.cpython-39.pyc
new file:   app/amazon/__pycache__/main.cpython-39.pyc
	modified:   app/amazon/del_brand.py
	modified:   app/amazon/main.py
	modified:   app/blueprints/__pycache__/main.cpython-39.pyc
	modified:   app/blueprints/main.py
	modified:   app/config.py
	modified:   app/main.py
	new file:   "app/web_source/brand-\346\227\247.html"
	modified:   app/web_source/brand.html
2026-04-02 00:03:16 +08:00
super
df606a1087 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-04-01 21:20:04 +08:00
super
ece3552b89 更新部分内容 2026-04-01 21:20:01 +08:00
铭坤
4df3945131 modified: app/ali_oss.py
new file:   app/amazon/del_brand.py
	new file:   app/amazon/main.py
	modified:   app/blueprints/__pycache__/brand.cpython-39.pyc
	modified:   app/blueprints/__pycache__/communication.cpython-39.pyc
	modified:   app/blueprints/__pycache__/main.cpython-39.pyc
	modified:   app/blueprints/brand.py
	new file:   "app/blueprints/brand_\345\244\207\344\273\275.py"
	modified:   app/blueprints/communication.py
	modified:   app/blueprints/main.py
	modified:   app/config.py
	modified:   app/main.py
2026-04-01 11:53:12 +08:00
super
6eb0f33421 删除模块 取消打开紫鸟 2026-03-31 23:12:46 +08:00
super
25ce9f74b8 处理这个python打开紫鸟 2026-03-30 23:15:08 +08:00
super
0f1f471590 处理掉一些BUG 2026-03-30 22:47:19 +08:00
super
d9487b6885 品牌多文件chunk修复 2026-03-30 21:06:10 +08:00
super
9845ab8f2a 暂存 2026-03-30 20:54:05 +08:00
super
6fa130386a Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-03-30 20:48:02 +08:00
super
53df8b9971 暂存 2026-03-30 20:39:13 +08:00
铭坤
d6625762cf modified: app/.env
modified:   app/ali_oss.py
	modified:   app/app.py
	new file:   app/blueprints/__pycache__/communication.cpython-39.pyc
	modified:   app/blueprints/communication.py
	modified:   app/brand_spider/__pycache__/main.cpython-39.pyc
	modified:   app/config.py
	modified:   app/main.py
	modified:   app/static/bg.jpg
	modified:   app/tool/__pycache__/devices.cpython-39.pyc
	modified:   app/web_source/brand.html
	modified:   app/web_source/templates_backup/brand.html
2026-03-30 19:44:04 +08:00
super
cb3b9e9cfe 完善删除品牌asin 2026-03-29 19:03:24 +08:00
super
3d39236b13 完成删除品牌 2026-03-29 13:41:03 +08:00
super
9e57fa5762 完成品牌删除模块 2026-03-29 00:22:45 +08:00
super
9c6232bc85 完成前后端紫鸟部分开发 2026-03-28 15:42:18 +08:00
super
d3c7938627 完成前后端删除、紫鸟部分开发 2026-03-28 15:42:02 +08:00
super
06256946fa 提交暂存 2026-03-27 22:00:01 +08:00
super
2042d386e2 提交过滤 2026-03-27 21:58:43 +08:00
铭坤
7176754564 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-03-27 21:56:53 +08:00
铭坤
a20222192e modified: assets/convert.js
modified:   assets/dedupe.js
	modified:   assets/split.js
	modified:   blueprints/__pycache__/brand.cpython-311.pyc
	modified:   blueprints/__pycache__/brand.cpython-39.pyc
	new file:   blueprints/communication.py
	modified:   config.py
	modified:   main.py
	modified:   new_web_source/convert.html
	modified:   new_web_source/dedupe.html
	modified:   new_web_source/split.html
2026-03-27 21:56:19 +08:00
super
a5c2681e88 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-03-27 21:51:34 +08:00
super
5e2f750359 更新运行日志
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 21:51:23 +08:00
super
27602890ab 品牌 2026-03-27 21:48:13 +08:00
铭坤
dfe0d652ce new file: assets/_plugin-vue_export-helper-Dh4yMZWw.js
new file:   assets/_plugin-vue_export-helper-xvHHTGU_.css
	new file:   assets/convert-1dkFPbKa.css
	new file:   assets/convert-6b0yq-M0.css
	new file:   assets/convert-BxkpMHO9.css
	new file:   assets/convert-D4Yv2oRy.css
	new file:   assets/convert-DwbQug2y.css
	modified:   assets/convert.js
	new file:   assets/dedupe-B0eqpVpj.css
	new file:   assets/dedupe-BHE_BCAJ.css
	new file:   assets/dedupe-Day_nGxq.css
	new file:   assets/dedupe-DzFvTfoj.css
	new file:   assets/dedupe-rhGu1E8E.css
	modified:   assets/dedupe.js
	new file:   assets/el-alert-Ct0RpqUD.css
	new file:   assets/pywebview-B6ZNXusn.css
	new file:   assets/pywebview-BRrok0pS.js
	new file:   assets/pywebview-BnpJyyGZ.js
	new file:   assets/pywebview-CGwF8TuM.js
	new file:   assets/pywebview-Chz98E9c.css
	new file:   assets/pywebview-D-PKWjUg.js
	new file:   assets/pywebview-DVgP2-kW.js
	new file:   assets/pywebview-IqgMfeBe.css
	new file:   assets/pywebview-Rm5Hhqpf.js
	new file:   assets/pywebview-leZCedEt.js
	new file:   assets/split-BsJfbBqn.css
	new file:   assets/split-DAa_Usoh.css
	new file:   assets/split-DidKJ84l.css
	new file:   assets/split-R_8V5xgh.css
	new file:   assets/split-_O6NHqDj.css
	modified:   assets/split.js
	modified:   blueprints/__pycache__/brand.cpython-311.pyc
	modified:   blueprints/__pycache__/brand.cpython-39.pyc
	modified:   blueprints/brand.py
	modified:   config.py
	modified:   main.py
	modified:   new_web_source/convert.html
	modified:   new_web_source/dedupe.html
	modified:   new_web_source/split.html
2026-03-27 21:35:28 +08:00
super
24076adf98 品牌 2026-03-27 21:33:07 +08:00
super
15a78abc1b 删除品牌 2026-03-27 21:32:00 +08:00
super
7fcdf1a761 提交品牌删除更新 2026-03-26 17:39:20 +08:00
super
d0ca28ae2d 提交品牌删除更新 2026-03-26 17:39:01 +08:00
super
32a9494dd6 更新后端新增内容 2026-03-26 16:42:25 +08:00
铭坤
a5da06537e modified: blueprints/__pycache__/__init__.cpython-39.pyc
modified:   blueprints/__pycache__/admin.cpython-39.pyc
	modified:   blueprints/__pycache__/auth.cpython-39.pyc
	modified:   blueprints/__pycache__/brand.cpython-311.pyc
	modified:   blueprints/__pycache__/brand.cpython-39.pyc
	modified:   blueprints/brand.py
	new file:   brand_spider/__pycache__/main.cpython-311.pyc
	modified:   brand_spider/__pycache__/main.cpython-39.pyc
	new file:   brand_spider/__pycache__/web_dec.cpython-311.pyc
	modified:   brand_spider/main.py
	modified:   config.py
	modified:   web_source/admin.html
	modified:   web_source/brand.html
	modified:   web_source/home.html
	modified:   web_source/index.html
	modified:   web_source/login.html
	new file:   web_source/templates_backup/admin.html
	new file:   web_source/templates_backup/brand.html
	new file:   web_source/templates_backup/home.html
	new file:   web_source/templates_backup/index.html
	new file:   web_source/templates_backup/login.html
2026-03-26 16:39:39 +08:00
super
21c6f41c69 增加紫鸟相关接口 2026-03-25 21:33:49 +08:00
super
506eb0faef 完善打包部分的压缩包 2026-03-25 09:52:04 +08:00
super
485b4dd485 完成品牌部分改造 2026-03-24 22:22:37 +08:00
super
4f728107d1 增加用户访问数据总表 2026-03-22 21:27:41 +08:00
super
01e491e332 数据去重完善 2026-03-22 20:58:36 +08:00
super
ed91c2ae7d 总表更新 2026-03-22 17:51:34 +08:00
super
55ac81f0cc 修复tab切换问题 2026-03-22 13:39:32 +08:00
super
64e32523cd 完成后台管理开发 2026-03-22 13:33:37 +08:00
super
5b636c3dfd 提交所有内容 2026-03-22 12:45:58 +08:00
super
c5d62c71a3 暂替 2026-03-22 11:37:23 +08:00
super
fa384a6ec8 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-03-22 11:37:03 +08:00
super
24cf47e1e3 暂提更新 2026-03-22 11:35:45 +08:00
super
8e66926b9e 更新新增三个模块 2026-03-22 11:20:09 +08:00
铭坤
574b372f42 new file: app/assets/_plugin-vue_export-helper-Dh4yMZWw.js
new file:   app/assets/_plugin-vue_export-helper-xvHHTGU_.css
	new file:   app/assets/convert-1dkFPbKa.css
	new file:   app/assets/convert-BxkpMHO9.css
	new file:   app/assets/convert-DwbQug2y.css
	modified:   app/assets/convert.js
	new file:   app/assets/dedupe-B0eqpVpj.css
	new file:   app/assets/dedupe-DzFvTfoj.css
	new file:   app/assets/dedupe-rhGu1E8E.css
	modified:   app/assets/dedupe.js
	new file:   app/assets/el-alert-Ct0RpqUD.css
	new file:   app/assets/pywebview-B6ZNXusn.css
	new file:   app/assets/pywebview-BnpJyyGZ.js
	new file:   app/assets/pywebview-CGwF8TuM.js
	new file:   app/assets/pywebview-D-PKWjUg.js
	new file:   app/assets/pywebview-IqgMfeBe.css
	new file:   app/assets/split-DAa_Usoh.css
	new file:   app/assets/split-DidKJ84l.css
	new file:   app/assets/split-R_8V5xgh.css
	modified:   app/assets/split.js
	modified:   app/blueprints/__pycache__/main.cpython-39.pyc
	modified:   app/blueprints/main.py
	modified:   app/main.py
	modified:   app/new_web_source/convert.html
	modified:   app/new_web_source/dedupe.html
	modified:   app/new_web_source/split.html
2026-03-22 10:29:12 +08:00
super
74a5fb9ce1 增加log输出、跳转tab顺序 2026-03-21 20:34:49 +08:00
super
4b83bf76b4 环境搭建配置 2026-03-21 10:00:16 +08:00
铭坤
82ccd9a8ea Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-03-20 23:16:13 +08:00
铭坤
2dbeec7d6d new file: app/blueprints/__init__.py
new file:   app/blueprints/__pycache__/__init__.cpython-311.pyc
	new file:   app/blueprints/__pycache__/__init__.cpython-39.pyc
	new file:   app/blueprints/__pycache__/admin.cpython-311.pyc
	new file:   app/blueprints/__pycache__/admin.cpython-39.pyc
	new file:   app/blueprints/__pycache__/auth.cpython-311.pyc
	new file:   app/blueprints/__pycache__/auth.cpython-39.pyc
	new file:   app/blueprints/__pycache__/brand.cpython-311.pyc
	new file:   app/blueprints/__pycache__/brand.cpython-39.pyc
	new file:   app/blueprints/__pycache__/image.cpython-311.pyc
	new file:   app/blueprints/__pycache__/image.cpython-39.pyc
	new file:   app/blueprints/__pycache__/main.cpython-311.pyc
	new file:   app/blueprints/__pycache__/main.cpython-39.pyc
	new file:   app/blueprints/admin.py
	new file:   app/blueprints/auth.py
	new file:   app/blueprints/brand.py
	new file:   app/blueprints/image.py
	new file:   app/blueprints/main.py
	new file:   app/brand_spider/__pycache__/main.cpython-39.pyc
	new file:   app/brand_spider/__pycache__/web_dec.cpython-39.pyc
	new file:   app/brand_spider/main.py
	new file:   app/brand_spider/web_dec.py
	new file:   app/static/bg.jpg
	new file:   "app/static/\345\223\201\347\211\214\346\226\207\346\241\243\346\240\274\345\274\217_\346\250\241\346\235\277.xlsx"
	new file:   "app/static/\346\250\241\346\235\2772-\344\273\245\346\226\207\344\273\266\345\244\271\346\226\271\345\274\217\344\270\212\344\274\240.zip"
	new file:   app/tool/.device_id
	new file:   app/tool/__pycache__/devices.cpython-311.pyc
	new file:   app/tool/__pycache__/devices.cpython-39.pyc
	new file:   app/tool/devices.py
2026-03-20 23:04:19 +08:00
super
fb32f7bd35 新增接口文档 2026-03-20 19:31:02 +08:00
super
63d3e4047c 打包项目 2026-03-20 18:13:47 +08:00
super
7c6d9dd989 Merge branch 'master' of https://gitee.com/TeaCodeNice/crawler-plugin 2026-03-20 14:26:41 +08:00
铭坤
ce121cf0d0 new file: ali_oss.py
new file:   app.py
	new file:   app/ali_oss.py
	new file:   app/app.py
	new file:   app/app_common.py
	new file:   app/config.py
	new file:   app/coze.py
	new file:   app/generate_api.py
	new file:   app/html_crypto.py
	new file:   app/main.py
	new file:   app/web_source/admin.html
	new file:   app/web_source/brand.html
	new file:   app/web_source/home.html
	new file:   app/web_source/index.html
	new file:   app/web_source/login.html
	new file:   app_common.py
	new file:   blueprints/__init__.py
	new file:   blueprints/__pycache__/__init__.cpython-311.pyc
	new file:   blueprints/__pycache__/__init__.cpython-39.pyc
	new file:   blueprints/__pycache__/admin.cpython-311.pyc
	new file:   blueprints/__pycache__/admin.cpython-39.pyc
	new file:   blueprints/__pycache__/auth.cpython-311.pyc
	new file:   blueprints/__pycache__/auth.cpython-39.pyc
	new file:   blueprints/__pycache__/brand.cpython-311.pyc
	new file:   blueprints/__pycache__/brand.cpython-39.pyc
	new file:   blueprints/__pycache__/image.cpython-311.pyc
	new file:   blueprints/__pycache__/image.cpython-39.pyc
	new file:   blueprints/__pycache__/main.cpython-311.pyc
	new file:   blueprints/__pycache__/main.cpython-39.pyc
	new file:   blueprints/admin.py
	new file:   blueprints/auth.py
	new file:   blueprints/brand.py
	new file:   blueprints/image.py
	new file:   blueprints/main.py
	new file:   brand_spider/__pycache__/main.cpython-39.pyc
	new file:   brand_spider/__pycache__/web_dec.cpython-39.pyc
	new file:   brand_spider/main.py
	new file:   brand_spider/web_dec.py
	new file:   config.py
	new file:   coze.py
	new file:   generate_api.py
	new file:   html_crypto.py
	new file:   main.py
	new file:   static/bg.jpg
	new file:   "static/\345\223\201\347\211\214\346\226\207\346\241\243\346\240\274\345\274\217_\346\250\241\346\235\277.xlsx"
	new file:   "static/\346\250\241\346\235\2772-\344\273\245\346\226\207\344\273\266\345\244\271\346\226\271\345\274\217\344\270\212\344\274\240.zip"
	new file:   tool/.device_id
	new file:   tool/__pycache__/devices.cpython-311.pyc
	new file:   tool/__pycache__/devices.cpython-39.pyc
	new file:   tool/devices.py
2026-03-20 11:23:57 +08:00
346 changed files with 34107 additions and 7135 deletions

2
.gitignore vendored
View File

@@ -43,6 +43,7 @@ MANIFEST
.installed.cfg
# Build / packaging
app.zip
build/
dist/
develop-eggs/
@@ -108,3 +109,4 @@ xlsx/
OPS_REDIS_MYSQL_OPTIMIZATION_NOTES.md
架构.md
*ts.%
.omc

7818
2026_05_29.log Normal file

File diff suppressed because one or more lines are too long

View File

@@ -1,9 +1,9 @@
base_url=http://8.136.19.173:15124
base_url=http://47.110.241.161:15124
workflow_id=7608812635877900322
mysql_host=8.136.19.173
mysql_host=47.110.241.161
mysql_user=aiimage
proxy_url=https://api.jikip.com/ip-get?num=1&minute=1&format=json&area=all&protocol=1&mode=2&key=t24g6gi44ubufd8
proxy_url=https://api.jikip.com/ip-get?num=1&minute=3&format=json&area=all&protocol=1&mode=2&key=t24g6gi44ubufd8
proxy_mode=2
zn_company=rongchuang123
@@ -11,10 +11,13 @@ zn_username=%E8%87%AA%E5%8A%A8%E5%8C%96_Robot
client_name=ShuFuAI
# java_api_base=http://api.aishufu.top:18080/
# java_api_base=http://api.aishufu.top:18080/
java_api_base=http://127.0.0.1:18080/
# 与 Java 后端共享的 JWT 签名密钥,必须与 backend-java 的 AIIMAGE_JWT_SECRET 完全一致
AIIMAGE_JWT_SECRET=please-change-this-secret-please-rotate-at-least-32-bytes
# java_api_base=http://47.111.163.154:18080
# java_api_base=http://127.0.0.1:18080
# java_api_base=http://8.136.19.173:18080
java_api_base=http://121.196.149.225:18080

View File

@@ -1,6 +1,6 @@
base_url=http://8.136.19.173:15124
base_url=http://47.111.163.154:15124
workflow_id=7608812635877900322
mysql_host=8.136.19.173
mysql_host=47.111.163.154
mysql_user=aiimage
proxy_url=https://api.jikip.com/ip-get?num=1&minute=1&format=json&area=all&protocol=1&mode=2&key=t24g6gi44ubufd8
@@ -9,7 +9,7 @@ proxy_mode=2
client_name=ShuFuAI
java_api_base=http://127.0.0.1:18080
# java_api_base=http://8.136.19.173:18080
java_api_base=http://api.aishufu.top:18080/
# java_api_base=http://api.aishufu.top:18080/

View File

@@ -1,5 +1,5 @@
workflow_id=7608812635877900322
mysql_host=8.136.19.173
mysql_host=47.111.163.154
mysql_user=aiimage
proxy_url=https://api.jikip.com/ip-get?num=1&minute=1&format=json&area=all&protocol=1&mode=2&key=t24g6gi44ubufd8
@@ -11,8 +11,13 @@ zn_username=%E8%87%AA%E5%8A%A8%E5%8C%96_Robot
client_name=ShuFuAI
java_api_base=http://47.111.163.154:18080
# java_api_base=http://127.0.0.1:18080
# java_api_base=http://8.136.19.173:18080
java_api_base=http://api.aishufu.top:18080/
# java_api_base=http://api.aishufu.top:18080/
# java_api_base=http://api.aishufu.top:18080/
# 与 Java 后端共享的 JWT 签名密钥,必须与 backend-java 的 AIIMAGE_JWT_SECRET 完全一致
AIIMAGE_JWT_SECRET=please-change-this-secret-please-rotate-at-least-32-bytes
# JWT cookie 名称,默认 aiimage_token改动需与 Java 端 aiimage.auth.cookie-name 保持一致
# AIIMAGE_AUTH_COOKIE_NAME=aiimage_token

View File

@@ -417,8 +417,8 @@ class AmamzonBase(ZiniaoDriver):
"""
try:
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
if level == "ERROR":
show_notification(message, "error")
# if level == "ERROR":
# show_notification(message, "error")
print(f"[{timestamp}] [{self.mark_name}] [{level}] {message}")
except Exception as e:
print(f"输出出错,{e}")
@@ -689,8 +689,8 @@ class TaskBase:
"""
try:
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
if level == "ERROR":
show_notification(message, "error")
# if level == "ERROR":
# show_notification(message, "error")
print(f"[{timestamp}] [{self.task_name}] [{level}] {message}")
except Exception as e:
print(f"输出出错,{e}")
@@ -739,7 +739,7 @@ class TaskBase:
if not shop_data:
mes = f"获取店铺凭证失败,响应数据: {shop_data.get('message', '未知错误')}"
self.log(mes, "ERROR")
show_notification(mes, "ERROR")
# show_notification(mes, "ERROR")
continue
password = shop_data["data"]["password"]

View File

@@ -308,8 +308,9 @@ class AmzoneApprove(AmamzonBase):
self.log("开始执行...")
num = 0
retry_num = 0
max_retry_num = 5
total_page = 0
current_page = 0
while retry_num < max_retry_num: # 最多重试3次
# if num > 3: #测试
# return
@@ -325,13 +326,13 @@ class AmzoneApprove(AmamzonBase):
page_pamel = self.tab.eles('xpath://kat-pagination',timeout=5)
if len(page_pamel) > 0:
current_page = page_pamel[0].sr('xpath:.//ul[@class="pages"]//li[@aria-current="true"]').text
current_page = int(current_page.strip())
# 总页数
total_page = page_pamel[0].sr.eles('xpath:.//ul[@class="pages"]//span[@class="page__inner"][last()]')
if len(total_page) > 0:
total_page = total_page[-1].text
else:
total_page = 0
total_page = int(total_page[-1].text.strip())
self.log(f"当前页码: {current_page} / 总页数: {total_page}")
except Exception as e:
self.log(f"获取页码失败:{e}")
@@ -543,6 +544,14 @@ class AmzoneApprove(AmamzonBase):
self.tab.refresh()
self.tab.wait.doc_loaded(raise_err=False,timeout=120)
#检查页数是否相等,不相等则继续
self.log(f"开始检查页数,当前页数 {current_page} / {total_page}")
if total_page != 0 and current_page!= 0 and current_page < total_page:
self.log(f"检查到页数还未完成,重启浏览器继续")
raise RuntimeError(f"与页面的连接已断开,检查到页数还未完成,重启浏览器继续")
class ApproveTask(TaskBase):
"""审批任务处理类:负责处理产品风险审批任务"""
task_name = "产品风险审批-TASK"
@@ -674,12 +683,12 @@ class ApproveTask(TaskBase):
return
current_url = None
max_retries = 200 #只要没有完成,一直重试
for _ in range(max_retries):
try:
self.process_country(driver, country_code, task_id, shop_name,risk_listing_filter,target_url=current_url)
driver.reset_already_asin()
driver.close_store()
break
except Exception as e:
self.log(f"处理国家 {country_code} 失败: {str(e)}", "ERROR")
@@ -777,7 +786,7 @@ class ApproveTask(TaskBase):
}
result.append(country_data)
if len(result) > 20:
if len(result) > 10:
self.post_result_batch(task_id, shop_name, country_code, result)
result = []

View File

@@ -9,7 +9,7 @@ from datetime import datetime
from DrissionPage import Chromium, ChromiumOptions
from collections import defaultdict
from config import base_dir
from config import base_dir,debug
from amazon.tool import get_shop_info,show_notification
@@ -46,7 +46,8 @@ class ChromeAmzoneBase:
杀死当前谷歌浏览器进程,并使用 drissionpage 启动谷歌浏览器,使用系统安装的浏览器默认用户文件夹
"""
print("正在关闭现有的chromium浏览器进程...")
os.system('taskkill /f /t /im chrome.exe')
if not debug:
os.system('taskkill /f /t /im chrome.exe')
time.sleep(2)
print("正在启动chromium浏览器...")
@@ -70,8 +71,8 @@ class ChromeAmzoneBase:
"""
try:
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
if level == "ERROR":
show_notification(message, "error")
# if level == "ERROR":
# show_notification(message, "error")
print(f"[{timestamp}] [{self.mark_name}] [{level}] {message}")
except Exception as e:
print(f"输出出错,{e}")
@@ -116,20 +117,20 @@ class ChromeAmzoneBase:
# print(f"当前地址信息: {current_text}")
# 先检查标识
if mark is not None and mark in current_text:
print(f"邮编检测到标识: {mark},无需修改")
self.log(f"邮编检测到标识: {mark},无需修改")
return True
# 检查是否已经包含目标邮编
if zip_code in current_text:
print(f"邮编已经设置为: {zip_code},无需修改")
self.log(f"邮编已经设置为: {zip_code},无需修改")
return True
# 需要设置邮编
print(f"正在设置邮编为: {zip_code}")
self.log(f"正在设置邮编为: {zip_code}")
# 点击地址选择按钮
location_link = self.tab.ele('xpath://a[@id="nav-global-location-popover-link"]', timeout=10)
if not location_link:
print("找不到地址设置按钮")
self.log("找不到地址设置按钮")
return False
location_link.click()
@@ -192,4 +193,14 @@ class ChromeAmzoneBase:
self.browser.quit()
print("浏览器已关闭")
except Exception as e:
print(f"关闭浏览器时出错: {str(e)}")
print(f"关闭浏览器时出错: {str(e)}")

View File

@@ -13,7 +13,7 @@ from amazon.amazon_base import TaskBase
from amazon.chrome_base import ChromeAmzoneBase
from config import runing_task, runing_shop,base_dir,DELETE_BRAND_API_BASE
from config import runing_task,runing_shop,base_dir,DELETE_BRAND_API_BASE
class ChromeAmzone(ChromeAmzoneBase):
@@ -79,7 +79,8 @@ class ChromeAmzone(ChromeAmzoneBase):
except Exception as e:
error_msg = f"运行出错: {traceback.format_exc()}"
self.log(error_msg)
show_notification(f"采集失败: {str(e)}", "error")
# show_notification(f"采集失败: {str(e)}", "error")
raise RuntimeError(error_msg)
return {}
def _scrape_data(self):
@@ -91,7 +92,8 @@ class ChromeAmzone(ChromeAmzoneBase):
"""
data = {
'image_url': "",
'title': ""
'title': "",
"sku" : ""
}
try:
@@ -100,9 +102,41 @@ class ChromeAmzone(ChromeAmzoneBase):
title_ele = self.tab.ele('xpath://h1[@id="title"]',timeout=30)
title = title_ele.text
data["title"] = title
sku_ele_ls = self.tab.eles('xpath://ul[@class="a-unordered-list a-vertical a-spacing-mini"]', timeout=20)
if len(sku_ele_ls) > 0:
data["sku"] = sku_ele_ls[0].text
imge_ele = self.tab.ele('xpath://div[@id="imgTagWrapperId"]//img',timeout=20)
image_url = imge_ele.attr("src")
image_url = ""
min_image_url = ""
data_a_dynamic_image = imge_ele.attr("data-a-dynamic-image")
if data_a_dynamic_image:
dynamic_image_json = json.loads(data_a_dynamic_image)
self.log(f"图片信息:{dynamic_image_json}")
max_area = 0
min_area = 0
for url, (width, height) in dynamic_image_json.items():
area = width * height
if area > max_area:
max_area = area
image_url = url
if min_area == 0:
min_area = area
min_image_url = url
if area < min_area:
min_area = area
min_image_url = url
if not image_url:
image_url = imge_ele.attr("src")
if not min_image_url:
min_image_url = imge_ele.attr("src")
data["image_url"] = image_url
data["min_image_url"] = min_image_url
return data
except Exception as e:
@@ -246,6 +280,7 @@ class SpiderTask(TaskBase):
"country": i.get("country"),
"url": return_data.get("image_url"),
"title": return_data.get("title"),
"sku": return_data.get("sku")
}
for i in items
]
@@ -263,7 +298,7 @@ class SpiderTask(TaskBase):
result.append(res)
print("================")
is_done = gp_index == len(groups)-1
if len(result) > 20 or is_done:
if len(result) > 10 or is_done:
self.post_result(task_id=task_id,chunkIndex=gp_index+1,chunkTotal=len(groups),
asin=asin,item_data=result,is_done=is_done)
result = []
@@ -273,18 +308,16 @@ class SpiderTask(TaskBase):
is_done = True
self.post_result(task_id=task_id, chunkIndex=len(groups), chunkTotal=len(groups),
asin=asin, item_data=result, is_done=is_done)
try:
chrome.close()
except Exception as e:
print("退出浏览器出错",e)
# 更新已处理店铺数
if task_id in runing_task:
runing_task[task_id]["processed_shops"] += 1
except Exception as e:
self.log(f"处理店铺 {task_id} 失败: {str(e)}", "ERROR")
self.log(traceback.format_exc(), "ERROR")
try:
chrome.close()
except Exception as e:
print("退出浏览器出错", e)
# 更新任务状态
if task_id in runing_task:
if runing_task[task_id].get("stop_requested", False):
@@ -351,7 +384,7 @@ class SpiderTask(TaskBase):
time.sleep(2)
self.log(f"已达到最大重试次数,结果回传最终失败", "ERROR")
raise RuntimeError("已达到最大重试次数,结果回传最终失败")
# raise RuntimeError("已达到最大重试次数,结果回传最终失败")
if __name__ == '__main__':
# spide = SpiderTask()

View File

@@ -30,14 +30,17 @@ class AmzoneMatchAction(AmamzonBase):
except Exception as e:
print(f"{self.mark_name}】等待加载中消失出错", e)
def run_page_action(self):
def run_page_action(self,skip_asin=[]):
self.log(f"==================={self.mark_name}=======================")
self.log("开始执行...")
num = 0
retry_num = 0
get_page_faild = 0
while retry_num < 3: # 最多重试3次
max_retry_num = 5
total_page = 0
current_page = 0
while retry_num < max_retry_num:
# if num > 3: #测试
# return
# 等待加载完成
@@ -45,6 +48,7 @@ class AmzoneMatchAction(AmamzonBase):
load_ele = self.tab.eles("xpath://div[contains(@class,'Loader-module__loader')]",timeout=5)
if len(load_ele) > 0:
load_ele[0].wait.deleted(timeout=3, raise_err=False)
time.sleep(0.5)
# 获取当前页码
@@ -52,23 +56,17 @@ class AmzoneMatchAction(AmamzonBase):
page_pamel = self.tab.eles('xpath://kat-pagination',timeout=5)
if len(page_pamel) > 0:
current_page = page_pamel[0].sr('xpath:.//ul[@class="pages"]//li[@aria-current="true"]').text
current_page = int(current_page.strip())
# 测试
# if current_page > 1:
# break
# 总页数
total_page = page_pamel[0].sr.eles('xpath:.//ul[@class="pages"]//span[@class="page__inner"][last()]')
if len(total_page) > 0:
total_page = total_page[-1].text
else:
total_page = 0
total_page = int(total_page[-1].text.strip())
self.log(f"{self.mark_name}】当前页码: {current_page} / 总页数: {total_page}")
get_page_faild = 0
except Exception as e:
self.log(f"{self.mark_name}】获取页码失败", e)
get_page_faild+= 1
if get_page_faild > 2:
show_notification(f"{self.mark_name}】获取页码失败超3次停止任务")
break
# 保存当前URL用于失败重试时访问
try:
@@ -81,12 +79,27 @@ class AmzoneMatchAction(AmamzonBase):
sku_ls = self.tab.eles("xpath://div[@data-sku]",timeout=10)
self.log(f"{self.mark_name}】获取到 {len(sku_ls)}")
if len(sku_ls) == 0:
retry_num += 1
self.log(f"没有获取到SKU列表重试{retry_num}/{max_retry_num}")
self.tab.refresh()
self.tab.wait.doc_loaded(raise_err=False, timeout=120)
if retry_num == max_retry_num:
raise RuntimeError(f"与页面的连接已断开没有获取到SKU列表,重试{retry_num}/{max_retry_num}")
continue
# for sku_ele in sku_ls[0:2]:
for sku_ele in sku_ls:
# solve_problem = sku_ele.eles('xpath:.//kat-link[@label="解决商品信息问题"]')
asin = sku_ele.ele('xpath:.//div[contains(@class,"JanusSplitBox-module__container")]//div[contains(@class,"JanusSplitBox-module__panel--") and contains(string(.),"ASIN")]/..//div[last()]',timeout=3).text
self.log(f"{self.mark_name}ASIN {asin} 找到....")
self.log(f"ASIN {asin} 找到....")
if asin in skip_asin:
self.log(f"ASIN {asin} 跳过....")
yield (asin,"有最低价跳过")
self.already_asin.add(asin)
continue
if asin in self.already_asin:
self.log(f"{self.mark_name}{asin} 已经处理过了,跳过")
continue
@@ -134,6 +147,12 @@ class AmzoneMatchAction(AmamzonBase):
self.tab.refresh()
self.tab.wait.doc_loaded(raise_err=False,timeout=120)
#检查页数是否相等,不相等则继续
self.log(f"开始检查页数,当前页数 {current_page} / {total_page}")
if total_page != 0 and current_page!= 0 and current_page < total_page:
self.log(f"检查到页数还未完成,重启浏览器继续")
raise RuntimeError(f"与页面的连接已断开,检查到页数还未完成,重启浏览器继续")
class MatchTak(TaskBase):
task_name = "匹配价格-TASK"
@@ -228,6 +247,9 @@ class MatchTak(TaskBase):
shop_name = shop_item.get("shopName", "未知店铺")
company_name = shop_item.get("companyName", "")
skip_asins_by_country = shop_item.get("skipAsinsByCountry",{})
skipAsinDetailsByCountry = shop_item.get("skipAsinDetailsByCountry",{})
if not company_name:
self.log(f"店铺 {shop_name} 的公司名称为空,跳过", "WARNING")
return
@@ -252,6 +274,11 @@ class MatchTak(TaskBase):
self.log(f"检测到任务 {task_id} 的暂停请求,停止处理国家", "WARNING")
break
skip_asin = skip_asins_by_country.get(country_code,[])
if skipAsinDetailsByCountry :
skipAsinDetails = {i.get("asin"):i.get("minimumPrice") for i in skipAsinDetailsByCountry.get(country_code)}
else:
skipAsinDetails = {}
# 打开店铺
driver = self.open_shop(cls=AmzoneMatchAction,max_retries=max_retries, company_name=company_name,
shop_name=shop_name, iskill=iskill)
@@ -262,10 +289,11 @@ class MatchTak(TaskBase):
return
current_url = None
max_retries = 200
for _ in range(max_retries):
try:
self.process_country(driver, country_code, task_id, shop_name,risk_listing_filter,limit,
target_url=current_url)
target_url=current_url,skip_asin=skip_asin,skipAsinDetails=skipAsinDetails)
driver.reset_already_asin()
driver.close_store()
break
@@ -297,7 +325,8 @@ class MatchTak(TaskBase):
del runing_shop[shop_name]
self.log(f"店铺 {shop_name} 已从执行列表中移除")
def process_country(self, driver, country_code, task_id, shop_name, risk_listing_filter,limit=None,target_url=None):
def process_country(self, driver, country_code, task_id, shop_name, risk_listing_filter,limit=None,target_url=None,
skip_asin=[],skipAsinDetails={}):
"""处理单个国家的审批任务
"""
@@ -339,7 +368,7 @@ class MatchTak(TaskBase):
result = []
# 处理所有需要审批的商品通过yield获取结果
for asin, status in driver.run_page_action():
for asin, status in driver.run_page_action(skip_asin):
# 检查是否收到暂停请求
if task_id in runing_task and runing_task[task_id].get("stop_requested", False):
self.log(f"检测到任务 {task_id} 的暂停请求停止处理ASIN", "WARNING")
@@ -351,16 +380,19 @@ class MatchTak(TaskBase):
if task_id in runing_task:
runing_task[task_id]["current_asin"] = asin
runing_task[task_id]["processed_asins"] += 1
runing_task[task_id]["failed_count"] += 1
minimumPrice = ""
if "有最低价跳过" in status:
minimumPrice = skipAsinDetails.get(asin)
result.append({
"asin": asin,
"status": status,
"minimumPrice": minimumPrice,
"done": False
})
if len(result) > 20:
if len(result) > 10:
self.post_result_batch(task_id, shop_name, country_code,result)
result = []

File diff suppressed because one or more lines are too long

View File

@@ -10,9 +10,10 @@ from datetime import datetime
from collections import defaultdict
import requests
from urllib.parse import quote
import os
from curl_cffi import requests as requests_frp
from config import base_dir
from amazon.tool import show_notification,get_shop_info,remove_special_characters,split_currency_values
@@ -29,9 +30,9 @@ except ImportError:
CONFIG_PROXY_MODE = 1
# Forbidden 后 3 分钟内统一使用代理:记录代理生效截止时间与当前代理
_FORBIDDEN_PROXY_UNTIL = 0.0
_FORBIDDEN_PROXY_DICT = None
_FORBIDDEN_PROXY_MINUTES = 0.5
_FORBIDDEN_PROXY_UNTIL_Similar = 0.0
_FORBIDDEN_PROXY_DICT_Similar = None
_FORBIDDEN_PROXY_MINUTES_Similar = 0.5
class ChromeAmzone(ChromeAmzoneBase):
@@ -76,7 +77,9 @@ class ChromeAmzone(ChromeAmzoneBase):
将图片链接URL 或本地文件路径)转换为 Base64 编码的字符串。
"""
if image_source.startswith(('http://', 'https://')):
response = requests.get(image_source, timeout=10)
response = requests.get(image_source, timeout=10,headers={
"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36 Edg/142.0.0.0 18444"
})
response.raise_for_status() # 非 2xx 状态码将抛出异常
image_data = response.content
else:
@@ -92,10 +95,10 @@ class ChromeAmzone(ChromeAmzoneBase):
"""
"fishkeeper Quick Aquarium Siphon Pump Gravel Cleaner - 256GPH Adjustable Powerful Fish Tank Vacuum Gravel Cleaning Kit for Aquarium Water Changer, Sand Cleaner, Dirt Removal : Amazon.co.uk: Pet Supplies"
标题 :
"""
global _FORBIDDEN_PROXY_UNTIL, _FORBIDDEN_PROXY_DICT
global _FORBIDDEN_PROXY_UNTIL_Similar, _FORBIDDEN_PROXY_DICT_Similar
headers = {
"accept": "application/json, text/plain, */*",
"accept-language": "zh-CN,zh;q=0.9,en;q=0.8",
@@ -141,21 +144,21 @@ class ChromeAmzone(ChromeAmzoneBase):
data = {
"imageBase64" : imageBase64
}
proxies = None
# 若 3 分钟内曾出现 Forbidden则直接使用当时保存的代理
now = time.time()
if now < _FORBIDDEN_PROXY_UNTIL and _FORBIDDEN_PROXY_DICT:
proxies = _FORBIDDEN_PROXY_DICT
if now < _FORBIDDEN_PROXY_UNTIL_Similar and _FORBIDDEN_PROXY_DICT_Similar:
proxies = _FORBIDDEN_PROXY_DICT_Similar
print("处于 Forbidden 代理窗口内,直接使用代理", proxies)
# 发送第一次请求
try:
response = requests_frp.post(url, headers=headers, params=params, json=data, impersonate="chrome101",
cookies=cookie, proxies=proxies)
cookies=cookie, proxies=proxies,verify=False)
response.encoding = "utf-8"
# 检查响应状态码和数据有效性
need_retry = False
if response.status_code != 200:
@@ -174,13 +177,14 @@ class ChromeAmzone(ChromeAmzoneBase):
except Exception as e:
print(f"解析响应JSON失败: {e}")
need_retry = True
# 如果需要重试且配置了代理URL
# need_retry = True
if need_retry and CONFIG_PROXY_URL and not proxies:
try:
proxy_resp = requests.get(CONFIG_PROXY_URL, timeout=10)
print("代理请求结果->", proxy_resp.text)
# 模式 2账号密码代理接口返回 JSON
if CONFIG_PROXY_MODE == 2:
resp_json = proxy_resp.json()
@@ -204,19 +208,19 @@ class ChromeAmzone(ChromeAmzoneBase):
if proxy_ip:
proxies = {
"http": f"http://{proxy_ip}",
"https": f"http://{proxy_ip}",
"https": f"https://{proxy_ip}",
}
if proxies:
print("使用代理重试请求")
response = requests_frp.post(url, headers=headers, params=params, json=data,
response = requests_frp.post(url, headers=headers, params=params, json=data,
impersonate="chrome101", cookies=cookie, proxies=proxies)
response.encoding = "utf-8"
print("代理重试结果状态码:", response.status_code)
# 记录 3 分钟内都使用该代理
_FORBIDDEN_PROXY_UNTIL = now + _FORBIDDEN_PROXY_MINUTES * 60
_FORBIDDEN_PROXY_DICT = proxies
_FORBIDDEN_PROXY_UNTIL_Similar = now + _FORBIDDEN_PROXY_MINUTES_Similar * 60
_FORBIDDEN_PROXY_DICT_Similar = proxies
except Exception as e:
print("获取代理或重试失败:", e)
@@ -240,6 +244,7 @@ class ChromeAmzone(ChromeAmzoneBase):
'image_url': "",
'title': "",
'category' : '',
'sku' : '',
'success' : True
}
@@ -250,8 +255,42 @@ class ChromeAmzone(ChromeAmzoneBase):
title = title_ele.text
data["title"] = title
imge_ele = self.tab.ele('xpath://div[@id="imgTagWrapperId"]//img',timeout=20)
image_url = imge_ele.attr("src")
image_url = ""
min_image_url = ""
data_a_dynamic_image = imge_ele.attr("data-a-dynamic-image")
if data_a_dynamic_image:
dynamic_image_json = json.loads(data_a_dynamic_image)
self.log(f"图片信息:{dynamic_image_json}")
max_area = 0
min_area = 0
for url, (width, height) in dynamic_image_json.items():
area = width * height
if area > max_area:
max_area = area
image_url = url
if min_area == 0:
min_area = area
min_image_url = url
if area < min_area:
min_area = area
min_image_url = url
if not image_url:
image_url = imge_ele.attr("src")
if not min_image_url:
min_image_url = imge_ele.attr("src")
data["image_url"] = image_url
data["min_image_url"] = min_image_url
# sku
sku_ele_ls = self.tab.eles('xpath://ul[@class="a-unordered-list a-vertical a-spacing-mini"]',timeout=20)
if len(sku_ele_ls) > 0:
data["sku"] = sku_ele_ls[0].text
category = self.tab.eles('xpath://div[@id="wayfinding-breadcrumbs_feature_div"]//span[@class="a-list-item"]//a[@class="a-link-normal a-color-tertiary"]',timeout=20)
if len(category) > 0:
data["category"] = category[0].text
@@ -271,12 +310,13 @@ class ChromeAmzone(ChromeAmzoneBase):
Returns:
dict: 包含采集到的数据
"""
return_data = {}
try:
# 验证国家是否支持
if country not in self.country_info:
error_msg = f"不支持的国家: {country},支持的国家有: {list(self.country_info.keys())}"
print(error_msg)
show_notification(error_msg, "error")
# show_notification(error_msg, "error")
return None
# 获取国家配置
@@ -290,7 +330,7 @@ class ChromeAmzone(ChromeAmzoneBase):
domain = base_url.split("/dp/")[0]
# 拼接新的URL
product_url = f"{domain}/dp/{asin}"
print(f"正在访问: {product_url}")
self.log(f"正在访问: {product_url}")
# 打开链接
self.tab.get(product_url)
@@ -299,38 +339,44 @@ class ChromeAmzone(ChromeAmzoneBase):
self.close_init_popup()
# 2. 切换国家/设置邮编
print(f"正在检查并设置邮编: {zip_code},标识: {mark}")
self.log(f"正在检查并设置邮编: {zip_code},标识: {mark}")
self._set_zip_code(zip_code, mark)
# self.tab.wait.doc_loaded(timeout=5, raise_err=False)
# 3. 抓取数据
print("正在抓取商品数据...")
self.log("正在抓取商品数据...")
data = self._scrape_data()
return_data.update(data)
# 判断是否采集标题出错
new_size = "220,220"
pattern = r"\._(?:[A-Z]+)?(\d+)_\."
replacement = f"._{new_size}_.jpg"
# replacement = f"._{new_size}_.jpg"
# 使用正则替换
image_new_url = re.sub(pattern, lambda m: replacement, data["image_url"])
# image_new_url = re.sub(pattern, lambda m: replacement, data["image_url"])
image_new_url = data["min_image_url"]
print(image_new_url)
iamge_base64 = self.image_to_base64(image_new_url)
similar_data = []
# 4、请求获取插件的数据
for page_num in range(total_page):
resp_data = self.get_aliprice_data(
page=page_num,
title = data["title"],
domain=domain,
category=data["category"],
imageBase64 = iamge_base64
)
self.log(f"Aliprice 扩展数据获取:{resp_data}")
if len(resp_data.get("data",[])) > 0:
for i in resp_data.get("data"):
similar_data.append(i)
if image_new_url:
for page_num in range(total_page):
resp_data = self.get_aliprice_data(
page=page_num+1,
title = data["title"],
domain=domain,
category=data["category"],
imageBase64 = iamge_base64
)
self.log(f"Aliprice 扩展数据获取:{str(resp_data)[0:200]}")
if len(resp_data.get("data",[])) > 0:
for i in resp_data.get("data"):
similar_data.append(i)
data["similar_data"] = similar_data
@@ -340,14 +386,17 @@ class ChromeAmzone(ChromeAmzoneBase):
data['url'] = product_url
data['timestamp'] = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
print(f"数据抓取完成: {json.dumps(data)}")
return data
print(f"数据抓取完成: {json.dumps(return_data)}")
return_data.update(data)
return return_data
except Exception as e:
error_msg = f"运行出错: {traceback.format_exc()}"
print(error_msg)
show_notification(f"采集失败: {str(e)}", "error")
return {}
# show_notification(f"采集失败: {str(e)}", "error")
raise RuntimeError(f"{traceback.format_exc()}")
return return_data
class SimilarAsinTask(TaskBase):
@@ -407,22 +456,6 @@ class SimilarAsinTask(TaskBase):
return payload.get("data") or {}
raise RuntimeError(f"获取货源查询解析载荷失败: {payload}")
@staticmethod
def _count_payload_rows(data):
rows = data.get("rows") or data.get("items") or []
if isinstance(rows, list) and rows:
return len(rows)
groups = data.get("groups") or []
if not isinstance(groups, list):
return 0
total = 0
for group in groups:
if isinstance(group, dict) and isinstance(group.get("items"), list):
total += len(group.get("items"))
return total
def _merge_parsed_payload(self, data, parsed_payload):
payload_rows = parsed_payload.get("allItems") or parsed_payload.get("items") or []
merged = {
@@ -445,28 +478,15 @@ class SimilarAsinTask(TaskBase):
try:
data = task_data.get("data", {})
task_id = data.get("taskId")
if task_id:
queued_rows = self._count_payload_rows(data)
try:
parsed_payload = self.fetch_parsed_payload(task_id, data.get("user_id") or data.get("userId") or 1)
data = self._merge_parsed_payload(data, parsed_payload)
self.log(
f"similar asin task {task_id} loaded full parsed payload from Java, queuedRows={queued_rows}, fullRows={self._count_payload_rows(data)}"
)
except Exception as e:
if not queued_rows:
raise
self.log(
f"similar asin task {task_id} failed to load full parsed payload, fallback to queued rows={queued_rows}: {e}",
"WARNING"
)
parsed_payload = self.fetch_parsed_payload(task_id, data.get("user_id") or data.get("userId") or 1)
data = self._merge_parsed_payload(data, parsed_payload)
groups = self.normalize_groups(data)
print(groups)
if not task_id:
self.log("任务ID为空跳过", "WARNING")
return
self.log(f"开始处理爬取任务 {task_id}{len(groups)} 个任务")
if not groups:
@@ -501,48 +521,51 @@ class SimilarAsinTask(TaskBase):
result = []
for gp_index,gp in enumerate(groups):
items = gp.get("items", [])
return_data = {}
group_item = []
for index,value in enumerate(items):
print(value)
return_data = {
'image_url': "",
'title': ""
'title': "",
'category' : '',
'sku' : '',
'success' : False,
'similar_data' : []
}
asin = value.get("asin")
country = value.get("country")
return_data = None
for _ in range(max_retry):
try:
return_data = chrome.run(country, asin) or {}
return_data = chrome.run(country, asin,total_page=1) or {}
self.log(f"抓取结果->{return_data}")
break
except Exception as e:
if "与页面的连接已断开" in str(e):
chrome = ChromeAmzone()
if not isinstance(return_data, dict):
return_data = {}
if return_data.get("image_url"):
break
# if "与页面的连接已断开" in str(e):
chrome = ChromeAmzone()
self.log(f"{asin}抓取数据报错,{e}")
# if not isinstance(return_data, dict):
# return_data = {}
# if return_data.get("image_url"):
# break
group_item.append({
"sourceFileKey": value.get("sourceFileKey",""),
"sourceFilename": value.get("sourceFilename",""),
"rowToken": value.get("rowToken",""),
"groupKey": value.get("groupKey",""),
"id": value.get("values",{}).get("id",""),
"asin": value.get("asin",""),
"sku" : return_data.get("sku",""),
"country": value.get("country",""),
"url": return_data.get("image_url",""),
"title": return_data.get("title",""),
"done": False,
"urls" : [i.get("ori_picture") for i in return_data.get("similar_data",[])[:16]]
})
# task_id: int, chunkIndex:int,chunkTotal: int, country_code: str, asin: str, status: dict,error:str="",
# item_data:dict={},
group_item = [
{
"sourceFileKey": i.get("sourceFileKey"),
"sourceFilename": i.get("sourceFilename"),
"rowToken": i.get("rowToken"),
"groupKey": i.get("groupKey"),
"id": i.get("id"),
"asin": i.get("asin"),
"country": i.get("country"),
"url": return_data.get("image_url"),
"title": return_data.get("title"),
"done": False,
"urls" : [i.get("ori_picture") for i in return_data.get("similar_data")]
}
for i in items
]
# task_id: int, chunkIndex:int,chunkTotal: int, country_code: str, asin: str, status: dict,error:str="",
# item_data:dict={},
res = {
"sourceFileKey": gp.get("sourceFileKey"),
"sourceFilename": gp.get("sourceFilename"),
@@ -552,11 +575,11 @@ class SimilarAsinTask(TaskBase):
"items": group_item
}
print("================")
# print("================")
result.append(res)
print("================")
# print("================")
is_done = gp_index == len(groups)-1
if len(result) > 20 or is_done:
if len(result) > 5 or is_done:
self.post_result(task_id=task_id,chunkIndex=gp_index+1,chunkTotal=len(groups),
asin=asin,item_data=result,is_done=is_done)
result = []
@@ -567,10 +590,7 @@ class SimilarAsinTask(TaskBase):
self.post_result(task_id=task_id, chunkIndex=len(groups), chunkTotal=len(groups),
asin=asin, item_data=result, is_done=is_done)
try:
chrome.close()
except Exception as e:
print("退出浏览器出错",e)
# 更新已处理店铺数
if task_id in runing_task:
runing_task[task_id]["processed_shops"] += 1
@@ -578,6 +598,11 @@ class SimilarAsinTask(TaskBase):
self.log(f"处理店铺 {task_id} 失败: {str(e)}", "ERROR")
self.log(traceback.format_exc(), "ERROR")
try:
chrome.close()
except Exception as e:
print("退出浏览器出错", e)
# 更新任务状态
if task_id in runing_task:
if runing_task[task_id].get("stop_requested", False):
@@ -585,7 +610,8 @@ class SimilarAsinTask(TaskBase):
self.log(f"任务 {task_id} 已被暂停!")
else:
runing_task[task_id]["status"] = "completed"
self.log(f"任务 {task_id} 处理完成!")
self.log(f"任务 {task_id} 处理完成!")
except Exception as e:
self.log(f"任务处理失败: {traceback.format_exc()}", "ERROR")
@@ -644,13 +670,13 @@ class SimilarAsinTask(TaskBase):
time.sleep(2)
self.log(f"已达到最大重试次数,结果回传最终失败", "ERROR")
raise RuntimeError("已达到最大重试次数,结果回传最终失败")
# raise RuntimeError("已达到最大重试次数,结果回传最终失败")
if __name__ == '__main__':
spide = ChromeAmzone()
resp = spide.run(
country="", asin="B0F79LCB3X", total_page=2
country="", asin="B0CJ8SNXXV", total_page=2
)
print(resp)
# task_data = {

View File

@@ -3,9 +3,8 @@
按功能拆分为蓝图:认证(auth)、主页面(main)、管理员(admin)、图片(image)、品牌(brand)
"""
import os
import secrets
import logging
from datetime import timedelta, datetime
from datetime import datetime
from logging.handlers import RotatingFileHandler
from flask import Flask
@@ -73,12 +72,6 @@ def create_app():
supports_credentials=True,
resources={r"/api/*": {"origins": cors_origins}, r"/login": {"origins": cors_origins}},
)
# 生产环境必须通过环境变量注入固定 SECRET_KEY否则服务重启会使旧会话失效。
app.secret_key = os.environ.get('SECRET_KEY', 'dev-secret-key-change-me')
app.config['PERMANENT_SESSION_LIFETIME'] = timedelta(days=7)
app.config['SESSION_COOKIE_HTTPONLY'] = True
app.config['SESSION_COOKIE_SAMESITE'] = os.environ.get('SESSION_COOKIE_SAMESITE', 'Lax')
app.config['SESSION_COOKIE_SECURE'] = os.environ.get('SESSION_COOKIE_SECURE', '0') == '1'
# 注册蓝图(不设 url_prefix保持原有 URL 路径不变,前端无需改动)
app.register_blueprint(auth_bp)

View File

@@ -1,5 +1,5 @@
"""
公共模块:数据库连接、初始化、会话校验、装饰器、模板渲染
公共模块:数据库连接、初始化、JWT 鉴权装饰器、模板渲染
供各蓝图复用
"""
import os
@@ -8,18 +8,19 @@ from datetime import timedelta
from functools import wraps
import pymysql
from flask import request, redirect, url_for, session, jsonify, render_template, render_template_string
from flask import request, redirect, url_for, jsonify, render_template, render_template_string, g
from config import mysql_host, mysql_user, mysql_password, mysql_database
from jwt_util import parse_token, get_token_from_request
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
STATIC_DIR = os.path.join(BASE_DIR, 'static')
ASSETS_DIR = os.path.join(BASE_DIR, 'assets')
WEB_SOURCE_DIR = os.path.join(BASE_DIR, 'web_source')
TEMPLATE_FALLBACK_DIRS = (
WEB_SOURCE_DIR,
os.path.join(WEB_SOURCE_DIR, 'templates_backup'),
)
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
STATIC_DIR = os.path.join(BASE_DIR, 'static')
ASSETS_DIR = os.path.join(BASE_DIR, 'assets')
WEB_SOURCE_DIR = os.path.join(BASE_DIR, 'web_source')
TEMPLATE_FALLBACK_DIRS = (
WEB_SOURCE_DIR,
os.path.join(WEB_SOURCE_DIR, 'templates_backup'),
)
def get_db():
@@ -35,13 +36,13 @@ def get_db():
def _render_html(template_name: str, **context):
"""读取 HTML 模板:若为加密文件则先解密,再渲染。未加密或解密失败时按明文渲染。"""
path = next(
(candidate for candidate in (os.path.join(base_path, template_name) for base_path in TEMPLATE_FALLBACK_DIRS)
if os.path.isfile(candidate)),
None,
)
if path is None:
return render_template(template_name, **context)
path = next(
(candidate for candidate in (os.path.join(base_path, template_name) for base_path in TEMPLATE_FALLBACK_DIRS)
if os.path.isfile(candidate)),
None,
)
if path is None:
return render_template(template_name, **context)
with open(path, "rb") as f:
raw = f.read()
try:
@@ -54,13 +55,13 @@ def _render_html(template_name: str, **context):
def _render_html_new(template_name: str, **context):
"""读取 HTML 模板:若为加密文件则先解密,再渲染。未加密或解密失败时按明文渲染。"""
path = next(
(candidate for candidate in (os.path.join(base_path, template_name) for base_path in TEMPLATE_FALLBACK_DIRS)
if os.path.isfile(candidate)),
None,
)
if path is None:
return render_template(template_name, **context)
path = next(
(candidate for candidate in (os.path.join(base_path, template_name) for base_path in TEMPLATE_FALLBACK_DIRS)
if os.path.isfile(candidate)),
None,
)
if path is None:
return render_template(template_name, **context)
with open(path, "rb") as f:
raw = f.read()
try:
@@ -72,9 +73,36 @@ def _render_html_new(template_name: str, **context):
def _resolve_jwt_user():
"""从 Authorization/cookie 解析 JWT命中后写入 flask.g 缓存。无效返回 None。"""
cached = getattr(g, "_aiimage_user", None)
if cached is not None:
return cached or None
token = get_token_from_request()
payload = parse_token(token) if token else None
if not payload:
g._aiimage_user = False
return None
g._aiimage_user = payload
g.aiimage_user_id = payload.get("user_id")
g.aiimage_username = payload.get("username") or ""
g.aiimage_device_id = payload.get("device_id") or ""
return payload
def current_user_id():
user = _resolve_jwt_user()
return user.get("user_id") if user else None
def current_username():
user = _resolve_jwt_user()
return user.get("username") if user else ""
def _is_session_user_valid():
"""校验 session 中的 user_id 是否在数据库中仍存在;不存在则清除 session 并返回 False"""
uid = session.get('user_id')
"""兼容旧调用:判断当前 JWT 用户是否仍存在于数据库。"""
uid = current_user_id()
if not uid:
return False
try:
@@ -83,23 +111,22 @@ def _is_session_user_valid():
cur.execute("SELECT id FROM users WHERE id = %s", (uid,))
row = cur.fetchone()
conn.close()
if not row:
session.clear()
return False
return True
return bool(row)
except Exception:
session.clear()
return False
def _get_current_admin_role():
"""获取当前登录用户的管理角色super_admin / admin / None非管理员"""
uid = current_user_id()
if not uid:
return None, None
try:
conn = get_db()
with conn.cursor() as cur:
cur.execute(
"SELECT id, username, is_admin, role, created_by_id FROM users WHERE id = %s",
(session['user_id'],)
(uid,)
)
row = cur.fetchone()
conn.close()
@@ -110,13 +137,19 @@ def _get_current_admin_role():
return None, None
def _unauthorized_response():
if request.headers.get('X-Requested-With') == 'XMLHttpRequest' \
or request.path.startswith('/api/') \
or 'application/json' in (request.headers.get('Accept') or ''):
return jsonify({'success': False, 'error': '未登录'}), 401
return redirect(url_for('auth.login'))
def login_required(f):
@wraps(f)
def decorated(*args, **kwargs):
if not session.get('user_id') or not _is_session_user_valid():
if request.headers.get('X-Requested-With') == 'XMLHttpRequest':
return jsonify({'success': False, 'error': '未登录'}), 401
return redirect(url_for('auth.login'))
if not current_user_id():
return _unauthorized_response()
return f(*args, **kwargs)
return decorated
@@ -124,22 +157,24 @@ def login_required(f):
def admin_required(f):
@wraps(f)
def decorated(*args, **kwargs):
if not session.get('user_id'):
if request.headers.get('X-Requested-With') == 'XMLHttpRequest':
return jsonify({'success': False, 'error': '未登录'}), 401
return redirect(url_for('auth.login'))
uid = current_user_id()
if not uid:
return _unauthorized_response()
try:
conn = get_db()
with conn.cursor() as cur:
cur.execute("SELECT is_admin, role FROM users WHERE id = %s", (session['user_id'],))
cur.execute("SELECT is_admin, role FROM users WHERE id = %s", (uid,))
row = cur.fetchone()
conn.close()
if not row or not row.get('is_admin'):
if request.headers.get('X-Requested-With') == 'XMLHttpRequest':
if request.headers.get('X-Requested-With') == 'XMLHttpRequest' \
or request.path.startswith('/api/') \
or 'application/json' in (request.headers.get('Accept') or ''):
return jsonify({'success': False, 'error': '需要管理员权限'}), 403
return redirect(url_for('main.home'))
except Exception as e:
if request.headers.get('X-Requested-With') == 'XMLHttpRequest':
if request.headers.get('X-Requested-With') == 'XMLHttpRequest' \
or request.path.startswith('/api/'):
return jsonify({'success': False, 'error': str(e)}), 500
return redirect(url_for('main.home'))
return f(*args, **kwargs)

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1 @@
import{bN as r}from"./java-modules-B8c-YG5x.js";const n="";function s(e){return r(`${n}/api/brand/expand-folder-recursive`,{folder:e})}export{s as e};

View File

@@ -0,0 +1 @@
const r=new Map;function c(n){let t=r.get(n);return t||(t=new Map,r.set(n,t)),t}function u(n,t){const e=r.get(n);e&&(e.delete(t),e.size||r.delete(n))}function l(n,t,e){const i=window.setTimeout(()=>{u(n,i),t()},e);return c(n).set(i,{id:i,kind:"timeout",category:n}),i}function a(n,t,e){const i=window.setInterval(t,e);return c(n).set(i,{id:i,kind:"interval",category:n}),i}function f(n,t){if(t==null)return;const i=r.get(n)?.get(t);i?.kind==="interval"?window.clearInterval(t):window.clearTimeout(t),i?.cancel?.(),u(n,t)}function s(n){const t=r.get(n);if(t){for(const e of t.values())e.kind==="interval"?window.clearInterval(e.id):window.clearTimeout(e.id),e.cancel?.();r.delete(n)}}function d(n,t){return new Promise(e=>{const i=window.setTimeout(()=>{u(n,i),e()},t);c(n).set(i,{id:i,kind:"timeout",category:n,cancel:e})})}function m(n){const t=`${n}:`;for(const e of Array.from(r.keys()))(e===n||e.startsWith(t))&&s(e)}function w(n){const t=e=>`${n}:${e}`;return{setTimeout(e,i,o){return l(t(e),i,o)},setInterval(e,i,o){return a(t(e),i,o)},clearTimer(e,i){f(t(e),i)},clearCategory(e){s(t(e))},clearScope(){m(n)},sleep(e,i){return d(t(e),i)}}}export{w as c};

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

1
app/assets/convert.js Normal file

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

1
app/assets/dedupe.js Normal file

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1 @@
const e=[{value:"SearchSuppressed",label:"在搜索结果中禁止显示"},{value:"ApprovalRequired",label:"需要批准"},{value:"Active",label:"在售"},{value:"DetailPageRemoved",label:"详情页面已删除"}];export{e as L};

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

1
app/assets/query-asin.js Normal file

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

1
app/assets/shop-match.js Normal file

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

1
app/assets/split.js Normal file

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -1,379 +1,17 @@
"""
管理员蓝图:用户管理(列表/创建/更新/删除)、生成历史、管理页
"""
import json
import pymysql
from flask import Blueprint, request, jsonify
from werkzeug.security import generate_password_hash
from app_common import get_db, _render_html, _get_current_admin_role, admin_required, login_required
from flask import session
admin_bp = Blueprint('admin', __name__)
def _parse_json(val, default=None):
if val is None:
return default if default is not None else []
if isinstance(val, (list, dict)):
return val
try:
return json.loads(val)
except Exception:
return default if default is not None else []
@admin_bp.route('/admin')
@login_required
@admin_required
def admin_page():
return _render_html('admin.html')
@admin_bp.route('/api/admin/users')
@admin_required
def admin_list_users():
"""分页获取用户列表;支持用户名模糊搜索、指定管理员所属普通用户筛选"""
role, current_row = _get_current_admin_role()
if not role:
return jsonify({'success': False, 'error': '需要管理员权限'}), 403
page = max(1, int(request.args.get('page', 1)))
page_size = min(50, max(5, int(request.args.get('page_size', 15))))
offset = (page - 1) * page_size
search_username = (request.args.get('username') or request.args.get('search') or '').strip()
created_by_id_arg = request.args.get('created_by_id') or request.args.get('admin_id')
created_by_id = int(created_by_id_arg) if created_by_id_arg and str(created_by_id_arg).isdigit() else None
if role != 'super_admin':
created_by_id = None
try:
conn = get_db()
with conn.cursor() as cur:
if role == 'super_admin':
where_parts = ["1=1"]
params = []
if search_username:
where_parts.append("u.username LIKE %s")
params.append("%" + search_username + "%")
if created_by_id is not None:
where_parts.append("u.created_by_id = %s")
params.append(created_by_id)
where_sql = " AND ".join(where_parts)
cur.execute(
"""SELECT u.id, u.username, u.is_admin, u.role, u.created_at, u.created_by_id,
creator.username AS creator_username
FROM users u
LEFT JOIN users creator ON creator.id = u.created_by_id
WHERE """ + where_sql + """ ORDER BY u.id LIMIT %s OFFSET %s""",
tuple(params) + (page_size, offset),
)
rows = cur.fetchall()
cur.execute("SELECT COUNT(*) as total FROM users u WHERE " + where_sql, tuple(params))
total = cur.fetchone()['total']
cur.execute("SELECT id, username FROM users WHERE role = 'admin' ORDER BY id")
admins = [{'id': r['id'], 'username': r['username']} for r in cur.fetchall()]
else:
admin_id = current_row['id']
where_parts = ["(u.id = %s OR (u.role = 'normal' AND u.created_by_id = %s))"]
params = [admin_id, admin_id]
if search_username:
where_parts.append("u.username LIKE %s")
params.append("%" + search_username + "%")
where_sql = " AND ".join(where_parts)
cur.execute(
"""SELECT u.id, u.username, u.is_admin, u.role, u.created_at, u.created_by_id,
creator.username AS creator_username
FROM users u
LEFT JOIN users creator ON creator.id = u.created_by_id
WHERE """ + where_sql + """ ORDER BY u.id LIMIT %s OFFSET %s""",
tuple(params) + (page_size, offset),
)
rows = cur.fetchall()
cur.execute(
"SELECT COUNT(*) as total FROM users u WHERE " + where_sql,
tuple(params),
)
total = cur.fetchone()['total']
admins = []
items = [
{
'id': r['id'],
'username': r['username'],
'is_admin': bool(r.get('is_admin')),
'role': r.get('role') or 'normal',
'created_by_id': r.get('created_by_id'),
'creator_username': r.get('creator_username') or '',
'created_at': r['created_at'].strftime('%Y-%m-%d %H:%M') if r.get('created_at') else '',
}
for r in rows
]
conn.close()
payload = {
'success': True,
'items': items,
'total': total,
'page': page,
'page_size': page_size,
'current_user_role': role,
'admins': admins,
}
return jsonify(payload)
except Exception as e:
return jsonify({'success': False, 'error': str(e)})
@admin_bp.route('/api/admin/user', methods=['POST'])
@admin_required
def admin_create_user():
data = request.get_json() or {}
username = (data.get('username') or '').strip()
password = data.get('password') or ''
role, current_row = _get_current_admin_role()
if not role:
return jsonify({'success': False, 'error': '需要管理员权限'}), 403
want_role = (data.get('role') or 'normal').strip() or 'normal'
if want_role not in ('admin', 'normal'):
want_role = 'normal'
if role == 'admin' and want_role == 'admin':
return jsonify({'success': False, 'error': '仅超级管理员可创建管理员'})
if want_role == 'admin':
want_created_by = current_row['id']
elif role == 'super_admin':
want_created_by = data.get('created_by_id')
else:
want_created_by = current_row['id']
if not username or not password:
return jsonify({'success': False, 'error': '用户名和密码不能为空'})
if len(username) < 2:
return jsonify({'success': False, 'error': '用户名至少2个字符'})
if len(password) < 6:
return jsonify({'success': False, 'error': '密码至少6个字符'})
if want_role == 'normal' and role == 'super_admin' and want_created_by is None:
try:
conn = get_db()
with conn.cursor() as cur:
cur.execute("SELECT id FROM users WHERE role = 'admin' ORDER BY id LIMIT 1")
r = cur.fetchone()
conn.close()
want_created_by = r['id'] if r else current_row['id']
except Exception:
want_created_by = current_row['id']
if want_role == 'normal' and want_created_by is None:
want_created_by = current_row['id']
is_admin = 1 if want_role in ('super_admin', 'admin') else 0
pwd_hash = generate_password_hash(password, method='pbkdf2:sha256')
try:
conn = get_db()
with conn.cursor() as cur:
cur.execute(
"INSERT INTO users (username, password_hash, is_admin, role, created_by_id) VALUES (%s, %s, %s, %s, %s)",
(username, pwd_hash, is_admin, want_role, want_created_by),
)
conn.commit()
conn.close()
return jsonify({'success': True, 'msg': '用户创建成功'})
except pymysql.IntegrityError:
return jsonify({'success': False, 'error': '用户名已存在'})
except Exception as e:
return jsonify({'success': False, 'error': str(e)})
@admin_bp.route('/api/admin/user/<int:uid>', methods=['PUT'])
@admin_required
def admin_update_user(uid):
data = request.get_json() or {}
password = data.get('password')
want_role = (data.get('role') or '').strip() or data.get('role')
role, current_row = _get_current_admin_role()
if not role:
return jsonify({'success': False, 'error': '需要管理员权限'}), 403
if want_role is None and not password:
return jsonify({'success': False, 'error': '请提供要修改的内容'})
try:
conn = get_db()
with conn.cursor() as cur:
cur.execute("SELECT id, role, created_by_id FROM users WHERE id = %s", (uid,))
target = cur.fetchone()
if not target:
conn.close()
return jsonify({'success': False, 'error': '用户不存在'})
if role == 'admin':
if target['role'] != 'normal' or target.get('created_by_id') != current_row['id']:
conn.close()
return jsonify({'success': False, 'error': '只能编辑自己创建的普通用户'}), 403
want_role = None
else:
if target.get('role') == 'super_admin':
conn.close()
return jsonify({'success': False, 'error': '不能修改超级管理员'})
if want_role == 'super_admin':
return jsonify({'success': False, 'error': '不能将用户设为超级管理员'})
if want_role not in ('admin', 'normal', None, ''):
want_role = None
if password:
if len(password) < 6:
conn.close()
return jsonify({'success': False, 'error': '密码至少6个字符'})
pwd_hash = generate_password_hash(password, method='pbkdf2:sha256')
cur.execute("UPDATE users SET password_hash = %s WHERE id = %s", (pwd_hash, uid))
if want_role is not None and want_role != '':
is_admin = 1 if want_role == 'admin' else 0
cur.execute(
"UPDATE users SET is_admin = %s, role = %s WHERE id = %s",
(is_admin, want_role, uid),
)
conn.commit()
conn.close()
return jsonify({'success': True, 'msg': '更新成功'})
except Exception as e:
return jsonify({'success': False, 'error': str(e)})
@admin_bp.route('/api/admin/user/<int:uid>', methods=['DELETE'])
@admin_required
def admin_delete_user(uid):
from flask import session
if session.get('user_id') == uid:
return jsonify({'success': False, 'error': '不能删除当前登录账号'})
role, current_row = _get_current_admin_role()
if not role:
return jsonify({'success': False, 'error': '需要管理员权限'}), 403
try:
conn = get_db()
with conn.cursor() as cur:
cur.execute("SELECT id, role, created_by_id FROM users WHERE id = %s", (uid,))
target = cur.fetchone()
if not target:
conn.close()
return jsonify({'success': False, 'error': '用户不存在'})
if target.get('role') == 'super_admin':
conn.close()
return jsonify({'success': False, 'error': '不能删除超级管理员'})
if role == 'admin':
if target.get('role') != 'normal' or target.get('created_by_id') != current_row['id']:
conn.close()
return jsonify({'success': False, 'error': '只能删除自己创建的普通用户'}), 403
cur.execute("DELETE FROM users WHERE id = %s", (uid,))
affected = cur.rowcount
conn.commit()
conn.close()
if affected == 0:
return jsonify({'success': False, 'error': '用户不存在'})
return jsonify({'success': True, 'msg': '删除成功'})
except Exception as e:
return jsonify({'success': False, 'error': str(e)})
@admin_bp.route('/api/admin/user/<int:uid>/column-permissions')
@login_required
def admin_user_column_permissions(uid):
"""获取指定用户的栏目权限列表:当前用户只能查自己,管理员可查任意用户。超级管理员返回全部栏目。"""
current_uid = session.get('user_id')
menu_type = (request.args.get('menu_type') or '').strip().lower()
if current_uid != uid:
role, _ = _get_current_admin_role()
if not role:
return jsonify({'success': False, 'error': '无权查看该用户的栏目权限'}), 403
try:
conn = get_db()
with conn.cursor() as cur:
cur.execute("SELECT role FROM users WHERE id = %s", (uid,))
user_row = cur.fetchone()
valid_menu_type = menu_type if menu_type in ('app', 'admin') else ''
if user_row and (user_row.get('role') or '').strip() == 'super_admin':
sql = """
SELECT id, name, column_key, menu_type, route_path, sort_order, created_at
FROM columns
"""
params = []
if valid_menu_type:
sql += " WHERE menu_type = %s"
params.append(valid_menu_type)
sql += " ORDER BY sort_order ASC, id ASC"
cur.execute(sql, tuple(params))
rows = cur.fetchall()
else:
sql = """
SELECT c.id, c.name, c.column_key, c.menu_type, c.route_path, c.sort_order, c.created_at
FROM columns c
INNER JOIN user_column_permission ucp ON ucp.column_id = c.id
WHERE ucp.user_id = %s
"""
params = [uid]
if valid_menu_type:
sql += " AND c.menu_type = %s"
params.append(valid_menu_type)
sql += " ORDER BY c.sort_order ASC, c.id ASC"
cur.execute(sql, tuple(params))
rows = cur.fetchall()
conn.close()
items = [
{
'id': r['id'],
'name': r['name'],
'column_key': r['column_key'],
'menu_type': r.get('menu_type') or 'app',
'route_path': r.get('route_path') or '',
'sort_order': r.get('sort_order') or 0,
'created_at': r['created_at'].strftime('%Y-%m-%d %H:%M') if r.get('created_at') else '',
}
for r in rows
]
return jsonify({'success': True, 'items': items})
except Exception as e:
return jsonify({'success': False, 'error': str(e)})
@admin_bp.route('/api/admin/history')
@admin_required
def admin_history():
"""管理员分页获取所有生成记录,支持按用户和时间筛选"""
page = max(1, int(request.args.get('page', 1)))
page_size = min(50, max(10, int(request.args.get('page_size', 15))))
offset = (page - 1) * page_size
user_id = request.args.get('user_id', type=int)
time_start = (request.args.get('time_start') or '').strip()
time_end = (request.args.get('time_end') or '').strip()
conditions, params = [], []
if user_id:
conditions.append("h.user_id = %s")
params.append(user_id)
if time_start:
conditions.append("h.created_at >= %s")
params.append(time_start)
if time_end:
conditions.append("h.created_at <= %s")
params.append(time_end + ' 23:59:59' if len(time_end) <= 10 else time_end)
where_clause = " AND ".join(conditions) if conditions else "1=1"
params_count = params[:]
params.extend([page_size, offset])
try:
conn = get_db()
with conn.cursor() as cur:
cur.execute(
"""SELECT h.id, h.user_id, h.created_at, h.panel_type, h.original_urls, h.params, h.result_urls,
h.long_image_url, u.username
FROM image_history h
LEFT JOIN users u ON h.user_id = u.id
WHERE """ + where_clause + """ ORDER BY h.created_at DESC LIMIT %s OFFSET %s""",
params,
)
rows = cur.fetchall()
cur.execute("SELECT COUNT(*) as total FROM image_history h WHERE " + where_clause, params_count)
total = cur.fetchone()['total']
conn.close()
items = []
for r in rows:
items.append({
'id': r['id'],
'user_id': r['user_id'],
'username': r.get('username') or '-',
'created_at': r['created_at'].strftime('%Y-%m-%d %H:%M') if r['created_at'] else '',
'panel_type': r['panel_type'] or '',
'original_urls': _parse_json(r['original_urls'], []),
'params': _parse_json(r['params'], {}),
'result_urls': _parse_json(r['result_urls'], []),
'long_image_url': (r.get('long_image_url') or '').strip() or None,
})
return jsonify({'success': True, 'items': items, 'total': total, 'page': page, 'page_size': page_size})
except Exception as e:
return jsonify({'success': False, 'error': str(e)})
"""
管理员蓝图:仅保留 /admin 页面渲染。
用户管理、栏目权限、生成历史等接口已全部迁移至 Java 端。
"""
from flask import Blueprint
from app_common import _render_html, admin_required, login_required
from config import JAVA_API_BASE
admin_bp = Blueprint('admin', __name__)
@admin_bp.route('/admin')
@login_required
@admin_required
def admin_page():
return _render_html('admin.html', java_api_base=JAVA_API_BASE)

View File

@@ -1,107 +1,56 @@
"""
认证蓝图:登录、登出、登录状态校验
"""
from flask import Blueprint, request, redirect, url_for, session, jsonify
from werkzeug.security import check_password_hash
from app_common import (
get_db,
_render_html,
_is_session_user_valid,
login_required,
BASE_DIR,
)
from tool.devices import DeviceIDGenerator
auth_bp = Blueprint('auth', __name__)
@auth_bp.route('/login', methods=['GET', 'POST'])
def login():
if session.get('user_id') and _is_session_user_valid():
return redirect(url_for('main.home'))
if request.method == 'POST':
data = request.get_json() if request.is_json else request.form
username = (data.get('username') or '').strip()
password = data.get('password') or ''
if not username or not password:
if request.is_json:
return jsonify({'success': False, 'error': '请输入用户名和密码'})
return _render_html('login.html', error='请输入用户名和密码')
try:
conn = get_db()
with conn.cursor() as cur:
cur.execute(
"SELECT id, password_hash, machine, is_admin FROM users WHERE username = %s",
(username,)
)
row = cur.fetchone()
if row and check_password_hash(row['password_hash'], password):
current_machine = DeviceIDGenerator().get_device_id()
stored_machine = (row.get('machine') or '').strip()
if not stored_machine:
with conn.cursor() as cur:
cur.execute("UPDATE users SET machine = %s WHERE id = %s", (current_machine, row['id']))
conn.commit()
conn.close()
session.permanent = True
session['user_id'] = row['id']
session['username'] = username
if request.is_json:
return jsonify({'success': True, 'redirect': url_for('main.home')})
return redirect(url_for('main.home'))
print("验证设备",stored_machine)
print("当前设备",current_machine)
if stored_machine != current_machine and row.get("is_admin") != 1:
conn.close()
err_msg = '当前设备与首次登录设备不一致,请在原设备上登录'
if request.is_json:
return jsonify({'success': False, 'error': err_msg})
return _render_html('login.html', error=err_msg)
conn.close()
session.permanent = True
session['user_id'] = row['id']
session['username'] = username
if request.is_json:
return jsonify({'success': True, 'redirect': url_for('main.home')})
return redirect(url_for('main.home'))
conn.close()
except Exception as e:
if request.is_json:
return jsonify({'success': False, 'error': str(e)})
return _render_html('login.html', error='登录失败,请稍后重试')
if request.is_json:
return jsonify({'success': False, 'error': '用户名或密码错误'})
return _render_html('login.html', error='用户名或密码错误')
return _render_html('login.html')
@auth_bp.route('/api/auth/check')
@login_required
def api_auth_check():
"""校验登录状态,用于页面加载时判断是否已登录;同时校验机器码是否与首次登录设备一致"""
if not session.get('user_id'):
return jsonify({'logged_in': False})
try:
conn = get_db()
with conn.cursor() as cur:
cur.execute("SELECT machine, is_admin FROM users WHERE id = %s", (session['user_id'],))
row = cur.fetchone()
conn.close()
if not row:
return jsonify({'logged_in': False})
stored_machine = (row.get('machine') or '').strip()
if stored_machine:
current_machine = DeviceIDGenerator().get_device_id()
if stored_machine != current_machine and row.get("is_admin") != 1:
session.clear()
return jsonify({'logged_in': False, 'error': '当前设备与首次登录设备不一致'})
except Exception:
return jsonify({'logged_in': False})
return jsonify({'logged_in': True, 'redirect': url_for('main.home')})
@auth_bp.route('/logout')
def logout():
session.clear()
return redirect(url_for('auth.login'))
"""
认证蓝图:登录页(仅 GET 渲染模板)+ 登出(清 JWT cookie+ token 同步
登录表单提交已经直连 Java 后端 /loginPython 这边只负责:
1. 渲染登录页模板
2. 把 Java 签发的 JWT 从前端写到 Python 同源 cookie供后续页面跳转携带
3. 登出:清 cookie 跳回登录页
"""
import sys
from flask import Blueprint, redirect, url_for, make_response, request, jsonify
from app_common import _render_html, current_user_id
from config import JAVA_API_BASE
from jwt_util import COOKIE_NAME, parse_token_with_reason
auth_bp = Blueprint('auth', __name__)
@auth_bp.route('/login', methods=['GET'])
def login():
if current_user_id():
return redirect(url_for('main.home'))
return _render_html('login.html', java_api_base=JAVA_API_BASE)
@auth_bp.route('/api/auth/sync', methods=['POST'])
def api_auth_sync():
"""前端拿到 Java 返回的 JWT 后调用,把 token 写进 Python 同源 cookie。"""
data = request.get_json(silent=True) or {}
token = (data.get('token') or '').strip()
if not token:
return jsonify({'success': False, 'error': '缺少 token'}), 400
payload, reason = parse_token_with_reason(token)
if not payload:
# 把根因打到服务端日志,并回传给前端,便于现场排查
print(f"[auth] /api/auth/sync 校验失败: {reason}", file=sys.stderr)
return jsonify({'success': False, 'error': f'token 无效: {reason}'}), 401
resp = make_response(jsonify({'success': True}))
resp.set_cookie(
COOKIE_NAME,
token,
max_age=7 * 24 * 3600,
path='/',
httponly=True,
samesite='Lax',
)
return resp
@auth_bp.route('/logout')
def logout():
resp = make_response(redirect(url_for('auth.login')))
resp.delete_cookie(COOKIE_NAME, path='/')
return resp

View File

@@ -13,9 +13,9 @@ from queue import Queue, Empty
from concurrent.futures import ThreadPoolExecutor, as_completed
from urllib.parse import urlparse
from flask import Blueprint, request, jsonify, session, send_file, redirect, Response
from flask import Blueprint, request, jsonify, send_file, redirect, Response
from app_common import get_db, login_required, BASE_DIR
from app_common import get_db, login_required, current_user_id, BASE_DIR
from config import bucket_path,JAVA_API_BASE
from brand_spider.main import single_file_handle, TaskCancelledError
@@ -70,12 +70,12 @@ def _get_uid_from_request_headers():
def _resolve_user_id():
"""
优先使用请求头 uid没有请求头 uid 时回退到 session['user_id']
优先使用请求头 uid没有请求头 uid 时回退到 JWT 解析的当前用户
若二者同时存在但不一致,则视为无效请求并返回 None。
"""
req_uid = _get_uid_from_request_headers()
if req_uid is not None:
sess_uid = session.get('user_id')
sess_uid = current_user_id()
if sess_uid is not None:
try:
if int(sess_uid) != int(req_uid):
@@ -84,7 +84,7 @@ def _resolve_user_id():
if str(sess_uid) != str(req_uid):
return None
return req_uid
return session.get('user_id')
return current_user_id()
def _get_user_id_or_error():

View File

@@ -1,726 +0,0 @@
"""
品牌爬虫蓝图:展开文件夹、运行任务、任务列表/详情、下载结果
"""
import os
import json
import threading
import traceback
import zipfile
import io
import time
import requests
from queue import Queue, Empty
from concurrent.futures import ThreadPoolExecutor, as_completed
from urllib.parse import urlparse
from flask import Blueprint, request, jsonify, session, send_file, redirect, Response
from app_common import get_db, login_required, BASE_DIR
from config import bucket_path,JAVA_API_BASE
from brand_spider.main import single_file_handle, TaskCancelledError
brand_bp = Blueprint('brand', __name__)
BRAND_OUTPUT_DIR = os.path.join(BASE_DIR, 'brand_output')
OSS_PREFIX = "brand_results"
# 任务完成时推送给前端的 SSE 队列task_id -> [Queue, ...]
_task_event_queues = {}
_task_event_lock = threading.Lock()
# 任务行级进度(当前处理文件的行数/总行数),仅存内存,不落库:
# task_id -> {
# 'file_index': int, # 当前处理第几个文件
# 'file_total': int, # 总文件数
# 'file_path': str, # 当前文件路径
# 'current_line': int, # 当前行号(或当前品牌序号)
# 'total_lines': int, # 总行数(或总品牌数)
# }
_task_line_progress = {}
_task_line_progress_lock = threading.Lock()
_cancelled_brand_tasks = set()
_cancelled_brand_tasks_lock = threading.Lock()
def _mark_task_cancelled(task_id):
with _cancelled_brand_tasks_lock:
_cancelled_brand_tasks.add(task_id)
def _is_task_cancelled(task_id):
with _cancelled_brand_tasks_lock:
return task_id in _cancelled_brand_tasks
def _get_uid_from_request_headers():
"""
从请求头读取 uid。
前端会在 headers 里携带 uid用于确定本次请求的归属用户。
"""
uid = request.headers.get('uid')
if uid is None:
return None
uid = str(uid).strip()
if not uid:
return None
try:
return int(uid)
except Exception:
return None
def _resolve_user_id():
"""
优先使用请求头 uid没有请求头 uid 时回退到 session['user_id']。
若二者同时存在但不一致,则视为无效请求并返回 None。
"""
req_uid = _get_uid_from_request_headers()
if req_uid is not None:
sess_uid = session.get('user_id')
if sess_uid is not None:
try:
if int(sess_uid) != int(req_uid):
return None
except Exception:
if str(sess_uid) != str(req_uid):
return None
return req_uid
return session.get('user_id')
def _get_user_id_or_error():
user_id = _resolve_user_id()
if not user_id:
return None, (jsonify({'success': False, 'error': '未登录或缺少uid'}), 401)
try:
return int(user_id), None
except Exception:
return None, (jsonify({'success': False, 'error': 'uid无效'}), 400)
def _push_task_event(task_id, event):
"""向订阅了该任务的所有 SSE 连接推送事件,并移除该任务的队列列表。
同时清理内存中的行级进度,不通过数据库中转行级进度。"""
with _task_event_lock:
queues = _task_event_queues.pop(task_id, [])
for q in queues:
try:
q.put_nowait(event)
except Exception:
pass
# 任务进入终态时,顺便清理行级进度缓存
with _task_line_progress_lock:
_task_line_progress.pop(task_id, None)
status = (event or {}).get('status') if isinstance(event, dict) else None
if status in ('success', 'failed', 'cancelled'):
with _cancelled_brand_tasks_lock:
_cancelled_brand_tasks.discard(task_id)
def _expand_folder_xlsx(folder_path):
"""返回文件夹下所有 .xlsx 文件的绝对路径列表"""
if not folder_path or not os.path.isdir(folder_path):
return []
paths = []
for name in os.listdir(folder_path):
if name.endswith('.xlsx') or name.endswith('.XLSX'):
paths.append(os.path.normpath(os.path.join(folder_path, name)))
return paths
def _upload_local_xlsx_to_oss(local_path, user_id):
"""将本地 xlsx 文件上传到 OSS返回 (url, None) 或 (None, error_message)。"""
if not local_path or not os.path.isfile(local_path):
return None, "文件不存在"
try:
from ali_oss import upload_file as oss_upload_file
except ImportError:
return None, "OSS 模块未配置"
base = os.path.basename(local_path)
safe_base = "".join(c if c.isalnum() or c in '-_.' else '_' for c in base)
if not safe_base.endswith('.xlsx'):
safe_base = safe_base + '.xlsx'
key = f"{bucket_path}brand_input/{user_id}/{int(time.time() * 1000)}/{safe_base}"
try:
with open(local_path, "rb") as f:
content = f.read()
url = oss_upload_file(content, key)
return url, None
except Exception as e:
return None, str(e)
def _run_brand_single(taskid,brand_ls, fileUrl,strategy,totalLines,chunkTotal,chunkIndex):
""""""
invalidBrands = [] # 不符合品牌的数据
queryFailedBrands = [] # 查询失败的数据
keptRows = []
for brand in brand_ls:
if _is_task_cancelled(taskid):
raise TaskCancelledError("任务已取消")
faild_data,query_faild_data = single_file_handle(brand,strategy)
invalidBrands.extend(faild_data)
queryFailedBrands.extend(query_faild_data)
if len(faild_data) == 0 and len(query_faild_data) == 0:
keptRows.append(brand)
data = { "strategy": strategy ,
"files": [
{
"fileUrl": fileUrl,
"originalFilename": "",
"relativePath": "",
"mainSheetName": "",
"chunkIndex": chunkIndex,
"chunkTotal": chunkTotal,
"totalLines": totalLines,
"keptRows": keptRows,
"invalidBrands": invalidBrands,
"queryFailedBrands": queryFailedBrands
}
]
}
resp = requests.post(f"{JAVA_API_BASE}/api/brand/tasks/{taskid}/result",headers={"accept":"application/json"},
# json={ "strategy": strategy ,
# "files": [
# { "fileUrl": fileUrl,"originalFilename": "","relativePath": "",
# "mainSheetName": "","columns": [],"keptRows": [],
# "invalidBrands": invalidBrands,"queryFailedBrands": queryFailedBrands
# }]})
json=data)
# print(data)
# print(taskid,brand_ls, fileUrl)
print("提交结果",data,"\n-->",resp.text)
return True
def _background_brand_task(task_id,data):
"""
后台执行爬虫任务
参数示例:
data : {
"taskId": 348,
"strategy": "Terms",
"files": [
{
"fileIndex": 1,
"fileUrl": "ab8cce4878754bfd89b4cd3ed20e1395",
"originalFilename": "品牌样例.xlsx",
"relativePath": "店铺A/品牌样例.xlsx",
"sheetName": "Sheet1",
"columns": [
"品牌",
"ASIH",
"状态",
"时间"
],
"rows": [
{
"品牌": "YQAUTEC",
"ASIH": "B0F28NZ752",
"状态": "",
"时间": "",
"__rowIndex": 2
}
],
"uniqueBrands": [
"YQAUTEC"
]
}
]
}
"""
try:
file_ls = data.get("files")
strategy = data.get("strategy")
if not isinstance(file_ls, list):
file_ls = []
tasks = []
for file_data in file_ls:
rows = file_data.get("rows") or []
brand_ls = [i.get("品牌") for i in rows if i.get("品牌")]
fileUrl = file_data.get("fileUrl")
if not brand_ls:
continue
for i in range(0, len(brand_ls), 5):
chunk = brand_ls[i:i + 5]
tasks.append((chunk, fileUrl))
if tasks:
max_workers = min(8, len(tasks))
with ThreadPoolExecutor(max_workers=max_workers) as executor:
if _is_task_cancelled(task_id):
_push_task_event(task_id, {'status': 'cancelled'})
return
futures = [
executor.submit(_run_brand_single, task_id, chunk, file_url, strategy,len(brand_ls),len(tasks),chunkIndex+1)
for chunkIndex,(chunk, file_url) in enumerate(tasks)
]
for future in as_completed(futures):
if _is_task_cancelled(task_id):
for f in futures:
f.cancel()
_push_task_event(task_id, {'status': 'cancelled'})
return
future.result()
_push_task_event(task_id, {'status': 'success'})
except Exception as e:
print("执行出错",traceback.format_exc())
_push_task_event(task_id, {'status': 'failed', 'error_message': str(e)})
print("执行完成")
def _parse_json(val, default=None):
if val is None:
return default if default is not None else []
if isinstance(val, (list, dict)):
return val
try:
return json.loads(val)
except Exception:
return default if default is not None else []
def _is_url(s):
if not isinstance(s, str) or not s.strip():
return False
return s.strip().startswith('http://') or s.strip().startswith('https://')
def _normalize_result_paths(raw):
"""
将 result_paths 规范化为 (url_list, zip_url)。
支持旧格式 list 或新格式 dict {"urls": [...], "zip_url": "..."}。
"""
if not raw:
return [], None
if isinstance(raw, dict):
urls = raw.get('urls') or []
if not isinstance(urls, list):
urls = []
zip_url = raw.get('zip_url')
if zip_url and not isinstance(zip_url, str):
zip_url = None
return urls, zip_url
if isinstance(raw, list):
return raw, None
return [], None
@brand_bp.route('/api/brand/expand-folder', methods=['POST'])
@login_required
def api_brand_expand_folder():
"""展开文件夹,返回其下所有 .xlsx 文件路径列表"""
try:
_, err = _get_user_id_or_error()
if err:
return err
data = request.get_json() or {}
folder = (data.get('folder') or '').strip()
if not folder:
return jsonify({'success': False, 'error': '请提供 folder 路径'}), 400
paths = _expand_folder_xlsx(folder)
return jsonify({'success': True, 'paths': paths})
except Exception as e:
return jsonify({'success': False, 'error': str(e)}), 500
def _expand_folder_xlsx_recursive_items(folder_path):
"""返回文件夹下所有 .xlsx 文件的绝对路径和相对路径列表(递归)"""
if not folder_path or not os.path.isdir(folder_path):
return []
items = []
root = os.path.normpath(folder_path)
for current_root, _, filenames in os.walk(root):
for name in filenames:
if not (name.endswith('.xlsx') or name.endswith('.XLSX')):
continue
absolute_path = os.path.normpath(os.path.join(current_root, name))
relative_path = os.path.relpath(absolute_path, root).replace('\\', '/')
items.append({
'absolutePath': absolute_path,
'relativePath': relative_path,
})
items.sort(key=lambda item: item['relativePath'])
return items
@brand_bp.route('/api/brand/expand-folder-recursive', methods=['POST'])
@login_required
def api_brand_expand_folder_recursive():
"""递归展开文件夹,返回 .xlsx 文件绝对路径及相对路径列表"""
try:
_, err = _get_user_id_or_error()
if err:
return err
data = request.get_json() or {}
folder = (data.get('folder') or '').strip()
if not folder:
return jsonify({'success': False, 'error': '请提供 folder 路径'}), 400
items = _expand_folder_xlsx_recursive_items(folder)
return jsonify({'success': True, 'items': items})
except Exception as e:
return jsonify({'success': False, 'error': str(e)}), 500
@brand_bp.route('/api/brand/run', methods=['POST'])
# @login_required
def api_brand_run():
"""立即运行:创建任务并后台执行,立即返回 task_id前端可轮询进度与取消"""
try:
data = request.get_json() or {}
paths = data.get('paths') or []
# task_type: 1=立即执行2=添加任务;此接口默认 1
try:
task_type = int(data.get('task_type') or 1)
except Exception:
task_type = 1
if task_type not in (1, 2):
task_type = 1
strategy = (data.get('strategy') or 'Terms').strip()
if strategy not in ('Terms', 'Simple'):
strategy = 'Terms'
if not isinstance(paths, list):
paths = []
paths = [p.strip() for p in paths if p and isinstance(p, str) and os.path.isfile(p.strip())]
if not paths:
return jsonify({'success': False, 'error': '没有有效的 xlsx 文件路径'}), 400
# 先上传到 OSS获取链接再入库
user_id, err = _get_user_id_or_error()
if err:
return err
urls = []
for p in paths:
url, err = _upload_local_xlsx_to_oss(p, user_id)
if err:
return jsonify({'success': False, 'error': f'上传文件失败: {os.path.basename(p)} - {err}'}), 500
base_name = os.path.basename(p)
urls.append({"fileUrl": url,"originalFilename": base_name,"relativePath": p })
# 请求提交
resp = requests.post(f"{JAVA_API_BASE}/api/brand/tasks?userId={user_id}",headers={
"content-type":"application/json",
},data=json.dumps({ "files": urls, "strategy": strategy,"taskType": 1,"archiveName": ""}))
print({ "files": urls, "strategy": strategy,"taskType": 1,"archiveName": ""})
print(f"{JAVA_API_BASE}/api/brand/tasks?userId={user_id}")
resp_data = resp.json()
# print(f"{JAVA_API_BASE}/api/brand/tasks?userId={user_id} 返回:",resp_data,"状态:",resp.status_code)
if resp_data.get("success"):
task_id = resp_data.get("data").get("taskId")
data = resp_data.get("data")
threading.Thread(target=_background_brand_task, args=(task_id,data), daemon=True).start()
return jsonify({'success': True, 'task_id': task_id})
else:
print(resp_data)
return jsonify({'success': False, 'error': "请求接口失败"}), 500
except Exception as e:
traceback.print_exc()
return jsonify({'success': False, 'error': str(e)}), 500
@brand_bp.route('/api/brand/tasks', methods=['GET', 'POST'])
# @login_required
def api_brand_tasks():
"""GET: 获取当前用户的任务列表POST: 添加任务(后台执行)"""
if request.method == 'GET':
try:
user_id, err = _get_user_id_or_error()
if err:
return err
resp = requests.get(f"{JAVA_API_BASE}/api/brand/tasks", params={
"userId": user_id
})
resp_data = resp.json()
print("获取列表",resp.text)
if resp_data.get("success"):
items = resp_data.get("data").get("items")
return jsonify({'success': True, 'items': items})
return jsonify({'success': True, 'items': []})
except Exception as e:
traceback.print_exc()
print("获取列表失败",e)
return jsonify({'success': False, 'error': str(e)}), 500
# POST
try:
data = request.get_json() or {}
paths = data.get('paths') or []
# task_type: 1=立即执行2=添加任务;此接口默认 2
try:
task_type = int(data.get('task_type') or 2)
except Exception:
task_type = 2
if task_type not in (1, 2):
task_type = 2
strategy = (data.get('strategy') or 'Terms').strip()
if strategy not in ('Terms', 'Simple'):
strategy = 'Terms'
if not isinstance(paths, list):
paths = []
paths = [p.strip() for p in paths if p and isinstance(p, str)]
if not paths:
return jsonify({'success': False, 'error': '请提供至少一个文件路径或链接'}), 400
user_id, err = _get_user_id_or_error()
if err:
return err
urls = []
for p in paths:
url, err = _upload_local_xlsx_to_oss(p, user_id)
if err:
return jsonify({'success': False, 'error': f'上传文件失败: {os.path.basename(p)} - {err}'}), 500
base_name = os.path.basename(p)
urls.append({"fileUrl": url, "originalFilename": base_name, "relativePath": p})
params = {
"userId": user_id
}
# resp = requests.post(f"{JAVA_API_BASE}/api/brand/tasks?userId={user_id}",headers={
# "content-type":"application/json",
# },data=json.dumps({ "files": urls, "strategy": strategy,"taskType": 1,"archiveName": ""}))
req_data = {"files": urls, "strategy": strategy, "taskType": 2, "archiveName": ""}
resp = requests.post(f"{JAVA_API_BASE}/api/brand/tasks?userId={user_id}", headers={"content-type": "application/json"},
data=json.dumps(req_data))
print(req_data)
# print(f"{JAVA_API_BASE}/api/brand/tasks?userId={user_id},返回", resp.text)
resp_data = resp.json()
if resp_data.get("success"):
task_id = resp_data.get("data").get("taskId")
data = resp_data.get("data")
return jsonify({'success': True, 'task_id': task_id})
return jsonify({'success': False, 'task_id': "请求异常"})
except Exception as e:
return jsonify({'success': False, 'error': str(e)}), 500
@brand_bp.route('/api/brand/tasks/<int:task_id>')
# @login_required
def api_brand_task_detail(task_id):
"""获取单个任务详情(用于轮询状态)"""
try:
resp = requests.get(f"{JAVA_API_BASE}/api/brand/tasks/{task_id}")
resp_data = resp.json()
if resp_data.get("success"):
task = resp_data["data"]["task"]
return jsonify({
'success': True,
'task': task
})
except Exception as e:
return jsonify({'success': False, 'error': str(e)}), 500
@brand_bp.route('/api/brand/tasks/<int:task_id>/line-progress')
# @login_required
def api_brand_task_line_progress(task_id):
"""
获取任务的“当前文件行级进度”(当前行数/总行数)。
该信息仅存放在内存字典 _task_line_progress 中,不写入数据库。
"""
try:
resp = requests.get(f"{JAVA_API_BASE}/api/brand/tasks/{task_id}")
resp_data = resp.json()
if resp_data.get("success"):
if resp_data["data"]["line_progress"]["has_progress"]:
info = resp_data["data"]["line_progress"]["info"]
resp = {
'file_index': int(info.get('file_index') or 0),
'file_total': int(info.get('file_total') or 0),
'file_name': info.get("file_name",""),
'current_line': int(info.get('current_line') or 0),
'total_lines': int(info.get('total_lines') or 0),
}
return jsonify({'success': True, 'has_progress': True, 'info': resp})
raise RuntimeError("获取进度失败")
except Exception as e:
return jsonify({'success': False, 'error': str(e)}), 500
@brand_bp.route('/api/brand/tasks/<int:task_id>/events')
# @login_required
def api_brand_task_events(task_id):
"""SSE任务完成时后端主动推送事件前端监听后隐藏进度条与取消按钮"""
user_id, err = _get_user_id_or_error()
if err:
return err
def _task_status_event():
conn = get_db()
try:
with conn.cursor() as cur:
cur.execute(
"SELECT status, error_message FROM brand_crawl_tasks WHERE id = %s AND user_id = %s",
(task_id, user_id)
)
row = cur.fetchone()
finally:
conn.close()
if not row:
return None
st = (row.get('status') or '').lower()
if st in ('success', 'failed', 'cancelled'):
return {'status': st, 'error_message': (row.get('error_message') or '').strip() or None}
return None
# 若任务已处于终态,直接返回一条事件后结束
ev = _task_status_event()
if ev is not None:
def _one_shot():
yield "data: " + json.dumps(ev, ensure_ascii=False) + "\n\n"
return Response(
_one_shot(),
mimetype='text/event-stream',
headers={'Cache-Control': 'no-cache', 'X-Accel-Buffering': 'no'}
)
# 否则注册队列,等待后台任务完成时推送
q = Queue()
with _task_event_lock:
_task_event_queues.setdefault(task_id, []).append(q)
def _stream():
try:
while True:
try:
event = q.get(timeout=20)
yield "data: " + json.dumps(event, ensure_ascii=False) + "\n\n"
return
except Empty:
yield ": keepalive\n\n"
finally:
with _task_event_lock:
lst = _task_event_queues.get(task_id, [])
if q in lst:
lst.remove(q)
if not lst:
_task_event_queues.pop(task_id, None)
return Response(
_stream(),
mimetype='text/event-stream',
headers={'Cache-Control': 'no-cache', 'X-Accel-Buffering': 'no'}
)
@brand_bp.route('/api/brand/tasks/<int:task_id>/cancel', methods=['POST'])
# @login_required
def api_brand_task_cancel(task_id):
"""取消正在执行或等待中的任务pending/running 均可取消)"""
try:
user_id, err = _get_user_id_or_error()
if err:
return err
resp = requests.post(
f"{JAVA_API_BASE}/api/brand/tasks/{task_id}/cancel",
headers={"accept": "application/json"},
timeout=20
)
try:
resp_data = resp.json()
except Exception:
return jsonify({'success': False, 'error': '取消接口返回非 JSON'}), 500
if not resp_data.get('success'):
return jsonify({'success': False, 'error': '第三方取消任务失败'}), 400
_mark_task_cancelled(task_id)
_push_task_event(task_id, {'status': 'cancelled'})
return jsonify({'success': True})
except Exception as e:
return jsonify({'success': False, 'error': str(e)}), 500
@brand_bp.route('/api/brand/tasks/<int:task_id>', methods=['DELETE'])
@login_required
def api_brand_task_delete(task_id):
"""删除任务(仅限非 running 状态)"""
try:
user_id, err = _get_user_id_or_error()
if err:
return err
resp = requests.delete(
f"{JAVA_API_BASE}/api/brand/tasks/{task_id}",
headers={"accept": "application/json"},
timeout=20
)
try:
resp_data = resp.json()
except Exception:
return jsonify({'success': False, 'error': '取消接口返回非 JSON'}), 500
if not resp_data.get('success'):
return jsonify({'success': False, 'error': '第三方取消任务失败'}), 400
return jsonify({'success': True})
except Exception as e:
return jsonify({'success': False, 'error': str(e)}), 500
@brand_bp.route('/api/brand/download/<int:task_id>')
@login_required
def api_brand_download(task_id):
"""下载任务结果:优先使用 OSS zip_url 重定向;否则本地/单链接/多链接按原逻辑处理"""
try:
user_id, err = _get_user_id_or_error()
if err:
return err
conn = get_db()
with conn.cursor() as cur:
cur.execute(
"SELECT result_paths FROM brand_crawl_tasks WHERE id = %s AND user_id = %s",
(task_id, user_id)
)
row = cur.fetchone()
conn.close()
if not row or not row.get('result_paths'):
return jsonify({'success': False, 'error': '无结果可下载'}), 404
raw = row['result_paths']
if isinstance(raw, str):
raw = _parse_json(raw)
url_list, zip_url = _normalize_result_paths(raw)
# 新格式:有 zip_url 直接重定向到 OSS
if zip_url and _is_url(zip_url):
return redirect(zip_url, code=302)
# 兼容旧格式result_paths 可能为 list本地路径或 url
if not url_list and isinstance(raw, list):
url_list = raw
local_paths = [p for p in url_list if isinstance(p, str) and not _is_url(p) and os.path.isfile(p)]
url_paths = [p.strip() for p in url_list if isinstance(p, str) and _is_url(p)]
if len(url_paths) == 1 and not local_paths:
return redirect(url_paths[0], code=302)
if len(local_paths) == 1 and not url_paths:
return send_file(
local_paths[0],
as_attachment=True,
download_name=os.path.basename(local_paths[0])
)
if local_paths or url_paths:
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w', zipfile.ZIP_DEFLATED) as zf:
for p in local_paths:
zf.write(p, os.path.basename(p))
for url in url_paths:
try:
import requests as req
r = req.get(url, timeout=30, stream=True)
r.raise_for_status()
name = os.path.basename(urlparse(url).path) or ('file_%s' % (url_paths.index(url)))
if not name or name == 'file_%s' % url_paths.index(url):
name = 'download_%s' % url_paths.index(url)
zf.writestr(name, r.content)
except Exception:
pass
buf.seek(0)
return send_file(
buf,
mimetype='application/zip',
as_attachment=True,
download_name='brand_task_%s.zip' % task_id
)
return jsonify({'success': False, 'error': '结果文件不存在或链接不可用'}), 404
except Exception as e:
return jsonify({'success': False, 'error': str(e)}), 500

View File

@@ -13,10 +13,10 @@ import subprocess
import requests
from urllib.parse import urlparse, quote
from flask import Blueprint, request, jsonify, Response, session, send_file
from flask import Blueprint, request, jsonify, Response, send_file
from PIL import Image
from app_common import get_db, login_required, BASE_DIR
from app_common import get_db, login_required, current_user_id, BASE_DIR
from config import STITCH_WORKFLOW_ID,client_name
image_bp = Blueprint('image', __name__)
@@ -146,11 +146,12 @@ def api_generate():
except (TypeError, ValueError):
hid = None
conn = get_db()
_uid = current_user_id()
with conn.cursor() as cur:
if hid is not None and hid > 0:
cur.execute(
"SELECT result_urls FROM image_history WHERE id=%s AND user_id=%s",
(hid, session['user_id']),
(hid, _uid),
)
row = cur.fetchone()
existing_urls = []
@@ -182,7 +183,7 @@ def api_generate():
_json.dumps(_sanitize_params_for_history(params)),
_json.dumps(merged_result_urls),
hid,
session['user_id'],
_uid,
),
)
if cur.rowcount > 0:
@@ -192,7 +193,7 @@ def api_generate():
"""INSERT INTO image_history (user_id, panel_type, original_urls, params, result_urls, long_image_url)
VALUES (%s, %s, %s, %s, %s, %s)""",
(
session['user_id'],
_uid,
params.get('panel_type', ''),
_json.dumps(result.get('original_urls') or []),
_json.dumps(_sanitize_params_for_history(params)),
@@ -366,7 +367,7 @@ def api_history():
conn = get_db()
with conn.cursor() as cur:
where_user = "user_id = %s"
params_where = [session['user_id']]
params_where = [current_user_id()]
if panel_type:
where_user += " AND panel_type = %s"
params_where.append(panel_type)

View File

@@ -1,23 +1,24 @@
"""
主页面蓝图首页、home、图片工作台、品牌页、静态文件、Logo
"""
"""
主页面蓝图首页、home、图片工作台、品牌页、静态文件、Logo
"""
import os
from flask import Blueprint, send_file, render_template_string
from app_common import (
get_db,
_render_html,
_is_session_user_valid,
login_required,
admin_required,
STATIC_DIR,
BASE_DIR,
ASSETS_DIR
)
from flask import redirect, url_for, session
from flask import Flask, request, Response, stream_with_context
import requests
from app_common import (
get_db,
_render_html,
login_required,
admin_required,
current_user_id,
current_username,
STATIC_DIR,
BASE_DIR,
ASSETS_DIR
)
from flask import redirect, url_for
from flask import Flask, request, Response, stream_with_context
import requests
from config import base_url,version,JAVA_API_BASE
main_bp = Blueprint('main', __name__)
@@ -63,35 +64,36 @@ def _resolve_asset_filename(filename):
return matches[0]
return safe_name
@main_bp.route('/')
def index():
if session.get('user_id') and _is_session_user_valid():
return redirect(url_for('main.home'))
return redirect(url_for('auth.login'))
@main_bp.route('/home')
@login_required
def home():
try:
conn = get_db()
with conn.cursor() as cur:
cur.execute("SELECT username, is_admin FROM users WHERE id = %s", (session['user_id'],))
row = cur.fetchone()
conn.close()
return _render_html('home.html', username=row.get('username', ''), is_admin=bool(row.get('is_admin')), user_id=session.get('user_id'),baseUrl=base_url,version=version)
except Exception:
return _render_html('home.html', username=session.get('username', ''), is_admin=False, user_id=session.get('user_id'),baseUrl=base_url,version=version)
@main_bp.route('/image')
@login_required
def wb():
return _render_html('index.html')
@main_bp.route('/')
def index():
if current_user_id():
return redirect(url_for('main.home'))
return redirect(url_for('auth.login'))
@main_bp.route('/home')
@login_required
def home():
uid = current_user_id()
try:
conn = get_db()
with conn.cursor() as cur:
cur.execute("SELECT username, is_admin FROM users WHERE id = %s", (uid,))
row = cur.fetchone()
conn.close()
return _render_html('home.html', username=row.get('username', ''), is_admin=bool(row.get('is_admin')), user_id=uid, baseUrl=base_url, version=version, java_api_base=JAVA_API_BASE)
except Exception:
return _render_html('home.html', username=current_username(), is_admin=False, user_id=uid, baseUrl=base_url, version=version, java_api_base=JAVA_API_BASE)
@main_bp.route('/image')
@login_required
def wb():
return _render_html('index.html')
@main_bp.route('/brand')
@login_required
def brand_page():
@@ -112,41 +114,41 @@ def brand_page():
if raw[:7] == b'gAAAAAB':
raise
content = raw.decode('utf-8', errors='replace')
return render_template_string(content, user_id=session.get('user_id'))
return render_template_string(content, user_id=current_user_id(), java_api_base=JAVA_API_BASE)
except Exception:
continue
return _render_html('brand.html', user_id=session.get('user_id'))
@main_bp.route('/brand/legacy')
@login_required
def brand_page_legacy():
legacy_path = os.path.join(BASE_DIR, 'web_source', 'templates_backup', 'brand.html')
if not os.path.isfile(legacy_path):
return '', 404
with open(legacy_path, 'rb') as f:
content = f.read().decode('utf-8', errors='replace')
content = content.replace(
'<header class="top-bar">',
'<header class="top-bar" style="display:none !important;">',
1,
return _render_html('brand.html', user_id=current_user_id(), java_api_base=JAVA_API_BASE)
@main_bp.route('/brand/legacy')
@login_required
def brand_page_legacy():
legacy_path = os.path.join(BASE_DIR, 'web_source', 'templates_backup', 'brand.html')
if not os.path.isfile(legacy_path):
return '', 404
with open(legacy_path, 'rb') as f:
content = f.read().decode('utf-8', errors='replace')
content = content.replace(
'<header class="top-bar">',
'<header class="top-bar" style="display:none !important;">',
1,
)
content = content.replace('height: calc(100vh - 56px);', 'height: 100vh;', 1)
return render_template_string(content, user_id=session.get('user_id'))
@main_bp.route('/static/<path:filename>')
def serve_static(filename):
"""提供 static 目录及子目录下的静态文件访问。"""
filepath = os.path.normpath(os.path.join(STATIC_DIR, filename))
static_abs = os.path.abspath(STATIC_DIR)
file_abs = os.path.abspath(filepath)
if not file_abs.startswith(static_abs) or not os.path.isfile(file_abs):
return '', 404
return send_file(file_abs, as_attachment=False)
return render_template_string(content, user_id=current_user_id())
@main_bp.route('/static/<path:filename>')
def serve_static(filename):
"""提供 static 目录及子目录下的静态文件访问。"""
filepath = os.path.normpath(os.path.join(STATIC_DIR, filename))
static_abs = os.path.abspath(STATIC_DIR)
file_abs = os.path.abspath(filepath)
if not file_abs.startswith(static_abs) or not os.path.isfile(file_abs):
return '', 404
return send_file(file_abs, as_attachment=False)
@main_bp.route('/assets/<path:filename>')
def serve_assets(filename):
@@ -165,8 +167,8 @@ def serve_assets(filename):
if os.path.isfile(file_abs):
return send_file(file_abs, as_attachment=False)
return '', 404
@main_bp.route('/new_web_source/<path:filename>')
@login_required
def serve_new_web_source(filename):
@@ -179,7 +181,7 @@ def serve_new_web_source(filename):
if filename.lower().endswith('.html'):
with open(file_abs, 'rb') as f:
content = f.read().decode('utf-8', errors='replace')
raw_uid = session.get('user_id')
raw_uid = current_user_id()
uid_value = str(int(raw_uid)) if str(raw_uid or '').isdigit() else '""'
uid_script = (
"\n<script>"
@@ -195,22 +197,22 @@ def serve_new_web_source(filename):
@main_bp.route('/logo.jpg', methods=['GET'])
def get_logo_image():
return send_file(os.path.join(BASE_DIR, "logo.jpg"), mimetype='image/jpeg')
@main_bp.route('/newApi/<path:path>', methods=['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'OPTIONS'])
def proxy(path):
target_url = f"{JAVA_API_BASE}/{path}"
# 复制请求参数
params = request.args.to_dict()
# 处理请求头
headers = {}
for key, value in request.headers:
if key.lower() in ['host', 'content-length', 'connection']:
continue
headers[key] = value
return send_file(os.path.join(BASE_DIR, "logo.jpg"), mimetype='image/jpeg')
@main_bp.route('/newApi/<path:path>', methods=['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'OPTIONS'])
def proxy(path):
target_url = f"{JAVA_API_BASE}/{path}"
# 复制请求参数
params = request.args.to_dict()
# 处理请求头
headers = {}
for key, value in request.headers:
if key.lower() in ['host', 'content-length', 'connection']:
continue
headers[key] = value
ignore_url = [f"{JAVA_API_BASE}/api/delete-brand/tasks/batch",
f"{JAVA_API_BASE}/api/delete-brand/tasks/progress/batch",
f"{JAVA_API_BASE}/api/delete-brand/history",
@@ -229,36 +231,36 @@ def proxy(path):
print("=============================")
except Exception as e:
print("打印失败", e)
try:
# 使用流式请求
req = requests.request(
method=request.method,
url=target_url,
params=params,
headers=headers,
data=request.get_data() if request.get_data() else None,
cookies=request.cookies,
stream=True, # 启用流式传输
try:
# 使用流式请求
req = requests.request(
method=request.method,
url=target_url,
params=params,
headers=headers,
data=request.get_data() if request.get_data() else None,
cookies=request.cookies,
stream=True, # 启用流式传输
timeout=proxy_timeout
)
# 流式响应
def generate():
for chunk in req.iter_content(chunk_size=8192):
if chunk:
yield chunk
# 构建响应
response = Response(stream_with_context(generate()), status=req.status_code)
# 复制响应头
for key, value in req.headers.items():
if key.lower() not in ['content-encoding', 'content-length', 'transfer-encoding', 'connection']:
response.headers[key] = value
return response
)
# 流式响应
def generate():
for chunk in req.iter_content(chunk_size=8192):
if chunk:
yield chunk
# 构建响应
response = Response(stream_with_context(generate()), status=req.status_code)
# 复制响应头
for key, value in req.headers.items():
if key.lower() not in ['content-encoding', 'content-length', 'transfer-encoding', 'connection']:
response.headers[key] = value
return response
except requests.exceptions.Timeout:
print(f"后端服务超时target_url={target_url}, timeout={proxy_timeout}")
return Response("后端服务超时", status=504)

View File

@@ -22,7 +22,7 @@ proxy_url = os.getenv("proxy_url")
proxy_mode = int(os.getenv("proxy_mode",1))
client_name=os.getenv("client_name") + ".exe"
JAVA_API_BASE = os.getenv("java_api_base", "http://127.0.0.1:18080")
JAVA_API_BASE = os.getenv("java_api_base", "http://api.aishufu.top:18080/").rstrip("/")
# 紫鸟浏览器配置
from urllib.parse import unquote
@@ -32,7 +32,7 @@ ZN_USERNAME = unquote(ZN_USERNAME)
ZN_PASSWORD = os.getenv("zn_password", "#20zsg25")
# 删除品牌API配置
DELETE_BRAND_API_BASE = os.getenv("DELETE_BRAND_API_BASE", JAVA_API_BASE)
DELETE_BRAND_API_BASE = os.getenv("DELETE_BRAND_API_BASE", JAVA_API_BASE).rstrip("/")
cache_path = "./user_data"
@@ -50,7 +50,6 @@ file_url_pre = f"https://{bucket}.oss-cn-hangzhou.aliyuncs.com/"
os.environ['OSS_ACCESS_KEY_ID'] = accessKeyId
os.environ['OSS_ACCESS_KEY_SECRET'] = accessKeySecret
os.environ['SECRET_KEY'] = "ddffc7c1d02121d9554d7b080b2511b6"
debug = True

119
app/jwt_util.py Normal file
View File

@@ -0,0 +1,119 @@
"""
JWT 工具:解析 Java 端签发的 token、从请求中提取 token。
与 Java 端 JwtService 共用同一个 HS256 密钥AIIMAGE_JWT_SECRET
"""
import os
import sys
import time
from typing import Optional, Tuple
from flask import request
try:
import jwt as pyjwt
_PYJWT_IMPORT_ERROR = None
except ImportError as _e:
pyjwt = None
_PYJWT_IMPORT_ERROR = _e
# 启动时立刻给出醒目提示,避免上线后才发现一直 401
print(
"[auth] WARNING: PyJWT 未安装,所有依赖 JWT 的接口都会返回 401。"
" 请执行 `pip install PyJWT==2.10.1` 或 `pip install -r requirements.txt`。",
file=sys.stderr,
)
_DEFAULT_SECRET = "please-change-this-secret-please-rotate-at-least-32-bytes"
COOKIE_NAME = os.getenv("AIIMAGE_AUTH_COOKIE_NAME", "aiimage_token")
# 桌面端与 Java 服务器时钟可能漂移(笔记本休眠、用户手动改时间、跨时区等),
# 给 JWT exp/nbf 校验留出容差,避免 /api/auth/sync 因为时间不同步而 401。
# 默认 5 分钟,可通过 AIIMAGE_JWT_LEEWAY_SECONDS 覆盖。
try:
_JWT_LEEWAY_SECONDS = int(os.getenv("AIIMAGE_JWT_LEEWAY_SECONDS", "300"))
except ValueError:
_JWT_LEEWAY_SECONDS = 300
def _signing_key() -> bytes:
"""与 Java JwtService.signingKey 保持一致UTF-8 字节,不足 32 字节右侧补 0。"""
secret = os.getenv("AIIMAGE_JWT_SECRET", _DEFAULT_SECRET)
key_bytes = secret.encode("utf-8")
if len(key_bytes) < 32:
key_bytes = key_bytes + b"\x00" * (32 - len(key_bytes))
return key_bytes
def _describe_clock_skew(token: str) -> str:
"""解析 token 里的 iat/exp与本地时钟比较返回 ' iat=.. exp=.. now=.. skew=..s' 字符串。
仅供日志使用,不做安全决策;解析失败返回空串,不影响主流程。"""
if pyjwt is None:
return ""
try:
unverified = pyjwt.decode(token, options={"verify_signature": False, "verify_exp": False, "verify_iat": False, "verify_nbf": False})
except Exception:
return ""
iat = unverified.get("iat")
exp = unverified.get("exp")
now = int(time.time())
parts = [f"now={now}"]
if isinstance(iat, (int, float)):
parts.append(f"iat={int(iat)} skew_iat={now - int(iat)}s")
if isinstance(exp, (int, float)):
parts.append(f"exp={int(exp)} skew_exp={now - int(exp)}s")
return ", " + " ".join(parts)
def parse_token_with_reason(token: str) -> Tuple[Optional[dict], Optional[str]]:
"""解析 JWT返回 (payload, error_reason)。
payload 命中时 error_reason 为 None失败时 payload 为 Noneerror_reason 描述根因。"""
if not token:
return None, "token 为空"
if pyjwt is None:
return None, f"PyJWT 未安装({_PYJWT_IMPORT_ERROR}"
# 打印 token 头,便于发现 alg 不是 HS256 等情况(不验签,仅 base64 解码 header
try:
header = pyjwt.get_unverified_header(token)
except Exception as e:
return None, f"无法解析 token header: {type(e).__name__}: {e}"
alg = header.get("alg") or "?"
try:
payload = pyjwt.decode(
token,
_signing_key(),
algorithms=["HS256", "HS384", "HS512"],
leeway=_JWT_LEEWAY_SECONDS,
)
except Exception as e:
# 常见ExpiredSignatureError / InvalidSignatureError / DecodeError / InvalidAlgorithmError
# 把 token 内 exp/iat 与本地时钟一起打出来,定位"时间漂移"类失败更直接
skew_info = _describe_clock_skew(token)
return None, f"{type(e).__name__}: {e} (token alg={alg}{skew_info})"
sub = payload.get("sub")
try:
user_id = int(sub) if sub is not None else None
except (TypeError, ValueError):
return None, f"sub 字段非法: {sub!r}"
if user_id is None:
return None, "payload 缺少 sub 字段"
return {
"user_id": user_id,
"username": payload.get("username") or "",
"device_id": payload.get("deviceId") or "",
}, None
def parse_token(token: str) -> Optional[dict]:
"""解析 JWT返回 {user_id, username, device_id};失败返回 None。"""
payload, _ = parse_token_with_reason(token)
return payload
def get_token_from_request() -> str:
"""优先从 Authorization: Bearer 取,其次从 cookie 取。"""
auth = request.headers.get("Authorization", "")
if auth.startswith("Bearer "):
token = auth[len("Bearer "):].strip()
if token:
return token
return (request.cookies.get(COOKIE_NAME) or "").strip()

View File

@@ -4,6 +4,8 @@ import json
import sys
import shutil
import os
import uuid
import ctypes
os.makedirs(cache_path,exist_ok=True)
if not debug:
today = datetime.datetime.now().strftime("%Y_%m_%d")
@@ -20,6 +22,7 @@ import subprocess
from amazon.del_brand import kill_process
from app import run_app
from generate_api import generate
from tool.devices import DeviceIDGenerator
with open("version.txt","w",encoding="utf-8") as file:
@@ -185,17 +188,7 @@ class WindowAPI:
if not result:
return {'success': False, 'error': '用户取消'}
path = result[0] if isinstance(result, (list, tuple)) else result
try:
import requests
resp = requests.get(url.strip(), timeout=120, stream=True)
resp.raise_for_status()
with open(path, 'wb') as f:
for chunk in resp.iter_content(chunk_size=65536):
if chunk:
f.write(chunk)
return {'success': True, 'path': path}
except Exception as e:
return {'success': False, 'error': str(e)}
return self._download_url_to_path(url, path)
def save_file_from_url_new(self, url, default_filename='download.bin'):
"""弹窗选择保存位置,从 url 下载文件并保存。根据文件后缀动态设置保存类型。"""
@@ -229,18 +222,124 @@ class WindowAPI:
return {'success': False, 'error': '用户取消'}
path = result[0] if isinstance(result, (list, tuple)) else result
return self._download_url_to_path(url, path)
def save_file_from_url_with_progress(self, url, default_filename='download.bin', download_id=''):
"""弹窗选择保存位置,从 url 下载文件并通过前端事件上报下载进度。"""
print("调用到save_file_from_url_with_progress方法")
if not url or not url.strip():
return {'success': False, 'error': '下载地址为空'}
filename = str(default_filename or 'download.bin').strip() or 'download.bin'
ext = os.path.splitext(filename)[1].lower()
if ext == '.zip':
file_types = ('ZIP 压缩包 (*.zip)', '所有文件 (*.*)')
elif ext in ('.xlsx', '.xls'):
file_types = ('Excel 文件 (*.xlsx;*.xls)', '所有文件 (*.*)')
elif ext == '.csv':
file_types = ('CSV 文件 (*.csv)', '所有文件 (*.*)')
elif ext == '.txt':
file_types = ('文本文件 (*.txt)', '所有文件 (*.*)')
elif ext == '.json':
file_types = ('JSON 文件 (*.json)', '所有文件 (*.*)')
else:
file_types = ('所有文件 (*.*)',)
result = self._window.create_file_dialog(
webview.SAVE_DIALOG,
save_filename=filename,
file_types=file_types
)
if not result:
return {'success': False, 'error': '用户取消'}
path = result[0] if isinstance(result, (list, tuple)) else result
progress_id = str(download_id or uuid.uuid4())
return self._download_url_to_path(url, path, progress_id)
def _download_url_to_path(self, url, path, download_id=''):
"""先下载到临时文件,完整下载后再替换为最终文件,避免用户打开半成品。"""
final_path = os.path.abspath(path)
final_dir = os.path.dirname(final_path) or os.getcwd()
final_name = os.path.basename(final_path)
temp_path = os.path.join(final_dir, f".{final_name}.crdownload")
last_progress_emit = 0.0
last_progress_percent = -1
try:
import requests
os.makedirs(final_dir, exist_ok=True)
if os.path.exists(temp_path):
os.remove(temp_path)
resp = requests.get(url.strip(), timeout=120, stream=True)
resp.raise_for_status()
with open(path, 'wb') as f:
total = int(resp.headers.get('Content-Length') or 0)
downloaded = 0
self._emit_download_progress(download_id, 'running', final_path, downloaded, total)
with open(temp_path, 'wb') as f:
self._hide_file(temp_path)
for chunk in resp.iter_content(chunk_size=65536):
if chunk:
f.write(chunk)
return {'success': True, 'path': path}
downloaded += len(chunk)
percent = int(downloaded * 100 / total) if total else 0
now_time = time.time()
if percent != last_progress_percent or now_time - last_progress_emit >= 0.2:
self._emit_download_progress(download_id, 'running', final_path, downloaded, total)
last_progress_emit = now_time
last_progress_percent = percent
f.flush()
os.fsync(f.fileno())
self._show_file(temp_path)
os.replace(temp_path, final_path)
self._emit_download_progress(download_id, 'success', final_path, downloaded, total)
return {'success': True, 'path': final_path}
except Exception as e:
try:
if os.path.exists(temp_path):
os.remove(temp_path)
except Exception:
pass
self._emit_download_progress(download_id, 'failed', final_path, 0, 0, str(e))
return {'success': False, 'error': str(e)}
def _emit_download_progress(self, download_id, status, path, downloaded, total, error=''):
if not download_id:
return
payload = {
'id': download_id,
'status': status,
'path': path,
'downloaded': downloaded,
'total': total,
'percent': round(downloaded * 100 / total, 1) if total else 0,
'error': error,
}
script = (
"window.dispatchEvent(new CustomEvent('pywebview-download-progress', "
f"{{ detail: {json.dumps(payload, ensure_ascii=False)} }}));"
)
try:
self._window.evaluate_js(script)
except Exception:
pass
def _hide_file(self, path):
"""Windows 下隐藏临时下载文件,减少用户误打开半成品的机会。"""
if os.name != 'nt':
return
try:
ctypes.windll.kernel32.SetFileAttributesW(path, 0x02)
except Exception:
pass
def _show_file(self, path):
"""恢复普通文件属性,避免最终保存文件被隐藏。"""
if os.name != 'nt':
return
try:
ctypes.windll.kernel32.SetFileAttributesW(path, 0x80)
except Exception:
pass
def save_template_xlsx(self):
"""弹窗选择保存位置,将品牌文档格式模板 xlsx 保存到用户选择的位置。"""
template_name = '品牌文档格式_模板.xlsx'
@@ -347,6 +446,13 @@ class WindowAPI:
except Exception as e:
return {'success': False, 'error': str(e)}
def get_device_id(self):
"""暴露给 HTML 的设备指纹接口:复用桌面端硬件特征生成 64 位 SHA256 ID。"""
try:
return {'success': True, 'device_id': DeviceIDGenerator().get_device_id()}
except Exception as e:
return {'success': False, 'error': str(e)}
def get_detail_del(self,task_id):
"""获取正在执行的删除品牌任务详情"""
task_info = runing_task.get(task_id)
@@ -432,7 +538,7 @@ def main():
api = WindowAPI(window)
window.expose(api.close, api.minimize, api.maximize, api.toggle_maximize, generate_images, api.save_image, api.save_image_to_folder, api.select_folder, api.select_brand_xlsx_files, api.select_brand_folder,
api.save_file_from_url, api.save_template_xlsx,api.save_template_zip,api.upload_file_to_java,api.open_external_url,api.enqueue_json,api.get_detail_del,api.save_file_from_url_new)
api.save_file_from_url, api.save_template_xlsx,api.save_template_zip,api.upload_file_to_java,api.open_external_url,api.enqueue_json,api.get_detail_del,api.save_file_from_url_new,api.save_file_from_url_with_progress,api.get_device_id)
webview.start(
debug=True,
storage_path=cache_path,

View File

@@ -0,0 +1,16 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>格式转换 - 数富AI</title>
<script type="module" crossorigin src="/assets/convert.js"></script>
<link rel="modulepreload" crossorigin href="/assets/java-modules-B8c-YG5x.js">
<link rel="modulepreload" crossorigin href="/assets/brand-COze15GJ.js">
<link rel="stylesheet" crossorigin href="/assets/java-modules-l5anrOZ2.css">
<link rel="stylesheet" crossorigin href="/assets/convert-6oxZNMye.css">
</head>
<body>
<div id="app"></div>
</body>
</html>

View File

@@ -0,0 +1,16 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>数据去重 - 数富AI</title>
<script type="module" crossorigin src="/assets/dedupe.js"></script>
<link rel="modulepreload" crossorigin href="/assets/java-modules-B8c-YG5x.js">
<link rel="modulepreload" crossorigin href="/assets/brand-COze15GJ.js">
<link rel="stylesheet" crossorigin href="/assets/java-modules-l5anrOZ2.css">
<link rel="stylesheet" crossorigin href="/assets/dedupe-DpfPYQDv.css">
</head>
<body>
<div id="app"></div>
</body>
</html>

View File

@@ -0,0 +1,17 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>删除品牌 - 数富AI</title>
<script type="module" crossorigin src="/assets/delete-brand.js"></script>
<link rel="modulepreload" crossorigin href="/assets/java-modules-B8c-YG5x.js">
<link rel="modulepreload" crossorigin href="/assets/brand-COze15GJ.js">
<link rel="modulepreload" crossorigin href="/assets/categorized-timers-JPA-olTr.js">
<link rel="stylesheet" crossorigin href="/assets/java-modules-l5anrOZ2.css">
<link rel="stylesheet" crossorigin href="/assets/delete-brand-DZ3x5hOX.css">
</head>
<body>
<div id="app"></div>
</body>
</html>

View File

@@ -0,0 +1,16 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>数据拆分 - 数富AI</title>
<script type="module" crossorigin src="/assets/split.js"></script>
<link rel="modulepreload" crossorigin href="/assets/java-modules-B8c-YG5x.js">
<link rel="modulepreload" crossorigin href="/assets/brand-COze15GJ.js">
<link rel="stylesheet" crossorigin href="/assets/java-modules-l5anrOZ2.css">
<link rel="stylesheet" crossorigin href="/assets/split-CR1PtDQE.css">
</head>
<body>
<div id="app"></div>
</body>
</html>

View File

@@ -54,6 +54,7 @@ pyinstaller==6.19.0
pyinstaller-hooks-contrib==2026.3
PyMsgBox==2.0.1
PyMySQL==1.1.2
PyJWT==2.10.1
pyperclip==1.11.0
PyQt5==5.15.11
PyQt5-Qt5==5.15.2

View File

View File

BIN
app/update/PyQt5/QtCore.pyd Normal file

Binary file not shown.

BIN
app/update/PyQt5/QtGui.pyd Normal file

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

BIN
app/update/PyQt5/sip.pyd Normal file

Binary file not shown.

BIN
app/update/_bz2.pyd Normal file

Binary file not shown.

BIN
app/update/_ctypes.pyd Normal file

Binary file not shown.

BIN
app/update/_decimal.pyd Normal file

Binary file not shown.

BIN
app/update/_elementtree.pyd Normal file

Binary file not shown.

BIN
app/update/_hashlib.pyd Normal file

Binary file not shown.

BIN
app/update/_hashlib.zip Normal file

Binary file not shown.

BIN
app/update/_lzma.pyd Normal file

Binary file not shown.

BIN
app/update/_socket.pyd Normal file

Binary file not shown.

Binary file not shown.

BIN
app/update/libeay32.dll Normal file

Binary file not shown.

BIN
app/update/libffi-7.dll Normal file

Binary file not shown.

BIN
app/update/msvcp140.dll Normal file

Binary file not shown.

BIN
app/update/msvcp140_1.dll Normal file

Binary file not shown.

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More