Compare commits

..

74 Commits

Author SHA1 Message Date
huangzd1997 46be044121 fix(实例路由): 转发失败不再伪装成「任务不存活」
归属节点滚动重启时跨节点转发会连接失败,原来返回 ApiResponse.fail(40903) ——
HTTP 200 + data:null。客户端把「拿不到数据」解析成 alive=false,于是把健康的长
任务主动停掉(2026-09-18 任务 28616:跑到 66/253 被自杀,只跑到第 2/6 页)。

改为 503 + 空 body:新客户端按状态码判未知继续跑;老客户端因 body 不是 JSON、
resp.json() 抛异常也落到未知——两边都不会再把未知当成死。顺带让这类故障在
HTTP 指标里可见(原先记成 200,滚动重启期间丢了多少心跳监控完全看不到)。
2026-09-18 16:25:33 +08:00
huangzd1997 e0303f9cba feat(上架): 店铺互斥改为按(设备,店铺),同店可在不同客户端并行跑不同国家
背景:紫鸟浏览器会话是每台客户端一份,同一家店可以在两台机器上并行;
国家是会话内的可切换状态(SwitchingCountries,断线重连还会再切),
所以真正必须串行的只有"同一台设备上的同一家店"。原来的全局按店铺互斥
把跨机器的合法并行也挡了(2026-09-18 任务 28624 被 28616 误挡)。

- V130:biz_publish_file 增加 device_id 列
- AdminAuthSupport.currentDeviceId():只读 JWT 签名的 deviceId claim,
  不信任客户端可控的 X-Device-Id 请求头
- activateFile 互斥键改为 (device_id, shop_name);device 为空(旧客户端
  token 无该 claim / 内部令牌调用)时退回全局店铺互斥,保守不放宽;
  存量 RUNNING 行的 NULL device 视为"来源不明",同样保守拦到跑完为止
- 前端与客户端无需改动:页面 axios 对 /newApi 已带 Authorization Bearer
2026-09-18 16:08:08 +08:00
huangzd1997 9d39705c77 fix(采集数据): 品牌检测移出任务锁 + 失败任务仍产出可下载的部分结果
taskId 28599 事故根因:分片回传在任务锁内同步做品牌检测,上游 16890 卡死时
单次持锁 103.5 秒(costMs=103556),期间该任务的心跳与客户端重试全部撞
40902「任务正在处理」,客户端 5 次重试预算耗尽后中止了整个采集。

- submitResult:行归一化 / 去重过滤 / 品牌检测移到任务锁外,锁内只保留落库与终态
- BrandCheckClient:整批加 90 秒总耗时上限(超时按查询失败降级,首轮始终执行),
  单次回传时延从「无上界」收敛到约 90 秒 + 一次读超时
- 失败且已收到分片时照常组装部分结果;任务已 FAILED 则保留失败态与真实失败原因,
  只补结果文件(乐观写结果行 + 条件更新不匹配再补偿写回,避免崩溃时留下「成功但无下载」)
- productrisk / shopmatch 按 pricetrack 的 preserveFailure 样板补齐,含组装落地时
  不再把失败行「洗成成功」
- 前端:采集 / 跟价 / 产品风险 / 店铺匹配的下载按钮放宽到失败任务
2026-09-18 15:53:15 +08:00
huangzd1997 986df86e89 fix(任务恢复): 停止意图优先于自动续跑 + 未覆盖模块的中断可见
自查上一提交时发现两处遗漏:

- 用户点了「停止循环」而子任务恰好因客户端中断失败时,续派逻辑跑在停止检查之前,
  会把循环从停止意图里拉回 RUNNING 自己转起来。改为停止请求优先:直接 STOPPED。
- 模块不支持自动续跑的中断任务此前完全无痕迹(用户只看到失败)。
  这类模块(上架/改价/审批/商品管理采集/跟价非循环任务)的续跑载荷需要用户在页面上
  选的执行参数(ziniao_version 等),而这份选择只存在于派发那一刻的浏览器里、没落到
  request_json —— 自动重排队会用错参数,所以它们**不纳入**续跑(白名单的设计原则就是
  「执行参数全在服务端」)。现补一条计数查询把这类中断数量带进 stale-check summary
  的 resume(u=N),运维据此人工重跑;将来把参数回写落库后即可纳入白名单。

测试:PriceTrackLoopRunServiceTest 新增「已请求停止时中断失败不续派而是停止」,
TaskResumeServiceTest 新增「统计不支持续跑的中断任务数」。
2026-09-18 15:48:33 +08:00
huangzd1997 bd359411a9 feat(任务恢复): 客户端中断的任务自动续跑(保留失败记录 + 重新排队)
2026-09-18 任务 28587(跟价,uid 977 店铺「张美莺」)在客户端 09:51/09:57 被更新脚本
重启后中断,任务一直挂 RUNNING 到 stale 兜底;跟价循环还有第二重问题——子任务失败
即整个循环判 FAILED,一次客户端更新就让无限循环彻底停下。

契约:中断任务**保留失败记录**(用户看得到「因客户端重启中断」),同时由服务端自动
重新排队续跑,长任务不再因为一次更新整个白跑。

- V129:biz_file_task 加 resume_of_task_id / resume_attempt,
  biz_price_track_loop_run 加 resume_attempt(代数封顶用);
- 新增 TaskResumeService:扫描最近 30 分钟内因客户端中断而失败、未续跑过、代数未超限的
  任务,复制请求参数重新排队成 PENDING,交给已有兜底拉取通道(客户端每分钟 pull-pending)
  领走执行——不新造第二套派发机制。只对注册了 ClientTaskPullSpi 的模块生效
  (相似ASIN/采集/外观专利),上架/改价等写操作模块刻意排除,避免盲目重跑;
- 幂等:按 resume_of_task_id 反查,同一原任务不会重复排队;
- 挂在 stale-check 巡检线(每 2 分钟、有分布式锁),并用隔离壳包住异常:
  续跑失败绝不拖垮判死主流程(判死优先级更高);
- PriceTrackLoopRunService:子任务因「客户端异常中断」失败时不终止循环,
  清 active_task_id 后保持 RUNNING,客户端下次 dispatchNext 拿到同一店铺/同一轮,
  即原地续跑(页内已处理 ASIN 由服务端 skip_asins 去重,不会重复改价);
  连续重派超过 3 次才真正判失败,避免会话持续不可用时无限重开浏览器
  (28587 就是这样白烧了 5 小时)。子任务成功一轮后计数归零;
- application.yml:aiimage.task-resume.enabled 默认跟随 client-task-pull 开关
  (兜底拉取关着时续跑任务无人领取,只会积压被告死)。

测试:TaskResumeServiceTest 6 例(排队/幂等/开关/无实现/插入失败/无归属用户)、
PriceTrackLoopRunServiceTest 新增 4 例(中断续跑/达上限/非中断仍终止/成功归零),
连带更新 DeleteBrandStaleTaskServiceTest 与 TaskModuleCoverageTest 的构造参数。
2026-09-18 15:42:07 +08:00
huangzd1997 3634ea1d62 feat(撞款): 采集明细落库后自动触发重扫,重复检查无需等次日 00:00
- shopdatacrawl 新增 DuplicateCheckRefreshPort 端口,TaskService 在明细落库成功后请求重扫
- shopduplicatecheck 新增 DuplicateCheckRefreshScheduler:10s 合并窗口去抖、单飞、
  扫描锁忙自动重试(最多 6 次),失败不影响采集归档
- 复用现有 SCAN_LOCK,与定时扫描、手动「重新分析」互斥;新增 5 个调度器单测
2026-09-17 23:08:15 +08:00
huangzd1997 3ce0569c59 fix(提示语): 失败提示优先展示后端真实原因,修正 10+ 处误导文案
上架/删除品牌/去重/转换/拆分/撤回/查ASIN/产品风险/跟价/巡店/店铺匹配/视频工作台:
- 解析失败不再只报通用文案,逐条展示后端文件级 errorMessage(上架新增「店铺未找到」专属弹窗)
- 匹配失败不再一律「请检查店铺名」,按 matchStatus 区分 CONFLICT/PENDING
- 去重/转换/拆分全失败弹窗优先列各文件真实原因,取不到才回退猜测文案
- dispatch-guard 新增 fileErrors 选项与 collectDistinctErrors 导出
2026-09-17 23:02:15 +08:00
huangzd1997 d2d95f0b71 feat(上架): 激活时按店铺互斥,同一店铺不允许两个任务同时跑
2026-09-17 事故(28519/28520/28521 相继提交同一店铺「林洪武」):同一店铺被多个
任务并发打开,客户端 startBrowser 全部返回 -10000,三个任务一起失败。

- PublishTaskService.activateFile:激活前查是否已有其他任务在同一店铺 RUNNING,
  有则拒绝并带出占用中的任务号(激活是任务真正开跑的唯一入口,能最早拦住);
  校验与更新之间仍有极小竞态窗口,真正串行由客户端店铺锁保证,这层负责尽早提示;
- 前端 BrandPublishTab:文件派发失败的原因回显给用户。原来错误只写进日志、提示
  固定为"已记录并继续",用户会误以为是文件问题而反复重传,看不到真正原因;
- 新增 3 个后端单测:同店铺被占用则拒绝、无占用则放行、店铺名为空跳过校验。
2026-09-17 15:48:51 +08:00
huangzd1997 2a51006888 feat(前端防连点): 工具页提交按钮加全局冷却,并精简 4.0.28 更新日志
各工具页的「开始上架 / 启动任务 / 匹配店铺」按钮在提交逻辑跑完后立刻恢复可点,
手快连点会重复发起(2026-09-17 上架事故:同一店铺被接连提交三次,服务端并存多个
同店铺任务,客户端并发打开同一店铺时全部失败)。

- 新增 shared/utils/submit-guard.ts:冷却按按钮元素各自计算(WeakMap,点 A 不影响
  B),在捕获阶段拦截、抢在 Vue 的 @click 之前;冷却自首次有效点击起算,被拦的
  点击不会把冷却越拖越长;
- main.ts 全局安装 installSubmitGuard(),覆盖所有 .btn-run 主按钮,页面零改动;
- 13 个单测覆盖边界:不同按钮互不影响、被拦不延长冷却、非法时长回落默认等;
- client-changelog 每条精简到 40 字以内(超出面板显示不下,已有守护测试)。
2026-09-17 15:41:41 +08:00
huangzd1997 a89de129ea chore(客户端更新日志): 追加 4.0.28 条目
4.0.28 修复两个用户可感知的问题:
- 多个任务同时操作同一店铺导致「打开店铺失败」→ 同一店铺改排队执行;
- 紫鸟更新内核期间直接报「打开店铺失败」→ 改为等待更新完成并显示进度。
2026-09-17 15:32:47 +08:00
huangzd1997 ddefcbed56 feat(任务恢复): 外观专利接上「补传恢复」;合并冲突保留兜底对象并可诊断
28459 的两个遗留项:
1) 外观专利缺「补传恢复」入口——分片缺失致组装 job 重试耗尽后,客户端补传缺口
   也无法自动重跑,只能人工重置 job。TaskFileJobService 早有
   resetTerminalFailedForRecovery,但只被删除品牌模块接了。现按同一口径在分片
   提交成功后检查并恢复;best-effort,恢复失败不影响补传本身。
2) 合并 CAS 冲突不可诊断且会删掉唯一的兜底对象——原实现每次冲突都删掉刚写入的
   版本化对象,重试耗尽即抛异常、行仍指向旧指针;一旦旧对象也不在,该分片永久
   读不到(28459 的 chunk-462/473/484 正是这个形态)。现在冲突时读回行上当前
   哈希并写进异常与日志;终局失败保留最后一个对象,作为读路径「同槽位兄弟对象」
   兜底的恢复源。外观专利与相似ASIN 同一口径。

测试:新增 17 个用例(补传恢复 8 / 外观专利合并冲突 5 / 相似ASIN 合并冲突 4),
RED 均已确认;全量 mvn test 3099 个 0 失败。
2026-09-17 11:04:38 +08:00
huangzd1997 3137299bfe fix(临时载荷): 收口删除守卫 + chunk 读兜底,修「对象被误删→任务永久失败」
线上任务 28459(外观专利)三个分片的载荷对象被删、DB 行仍指向已删的确定性
key,组装读不到 → 整单 FAILED。数据其实还在同槽位的版本化对象里,是行指错了。

三条收口:
- 上传补偿删除加准入:只有末段带 UUID 的版本化 key(本次写入独占)才允许入队。
  确定性 key 会被重传复用,删它就删掉了行仍在引用的对象;而删除队列
  (deleteObjectFromRetry -> removeObject)本身不做引用反查,准入必须卡在入队处。
- 引用守卫阈值 >1 收紧为 >0:原来把「恰好 1 行引用」当作调用方自己那行而放行,
  但调用方无法证明归属。物理删除本就约定在 DB 行删除之后执行,正常路径引用数
  必为 0;宁可留孤儿(有保留期清理兜底),也不删可能仍被引用的对象。
- chunk 读兜底:指针对象确已缺失(NoSuchKey)时回退同槽位版本化对象 chunk-N-*。
  仅限 chunk 槽位 + 确为缺失两个条件,避免误配无关对象或掩盖真实故障。

测试:新增 31 个用例(补偿删除准入 / 引用守卫 / 兄弟对象兜底),均先确认 RED
再实现;另更新 3 个断言旧行为的既有用例。全量 mvn test 3082 个 0 失败。
2026-09-17 10:42:53 +08:00
huangzd1997 1fe3368c5a perf(rustfs): 重试基础退避 500ms→200ms
线上高频的重试诱因是 unexpected end of stream——那是**立即失败**(RustFS 重置连接后
OkHttp 读响应即报错),不是等超时,500ms 基本是白等;每天上千次累计十几分钟。
降到 200ms 保留退避语义(真遇服务端过载仍会让步),又不让用户多等。
2026-09-17 01:55:35 +08:00
huangzd1997 7aea3a0a50 fix(rustfs): 修 unexpected end of stream 根因——零长度请求体缺 Content-Length
抓包定位(90 秒内 515 次 HTTP 411 Length Required):OkHttp 对长度为 0 的请求体
不写 Content-Length,请求于是既无 Content-Length 也无 Transfer-Encoding——HTTP/1.1
不允许这样,RustFS 直接回 411;客户端读到不完整响应就报 unexpected end of stream
再重试。业务里上传空内容(content 为 null/空)是常态,所以每天上千次。

修复用 network interceptor 在协议层补 Content-Length: 0:普通 interceptor 里加的
头会被 BridgeInterceptor 按 body 长度覆盖,加了不生效。

实测:部署后同样 90 秒抓包,411 从 515 次降到 0,两节点「确定性错误不重试」归零。
此前排除的公网链路、keepAlive、连接复用三项确实都不是原因——抓包才是对的入口。
2026-09-17 01:36:55 +08:00
huangzd1997 188aedec84 fix(任务收尾): 打破「数据永久缺失 → 卡死恢复无限重建 job」的死循环
线上任务 28459(appearance_patent)卡在 RUNNING:chunk 462/473/484 的 payload
对象已不存在,组装失败 → stale recovery 每 30 秒重建一次 assemble job 再失败,
而恢复过程又会刷新任务心跳、任务永远判不出「卡死」,48 分钟空转近百轮。

- appearance-patent:读 chunk 时若 payload 对象已不存在(RustFS NoSuchKey),
  跳过该分片让任务按已有分片出部分结果,不再抛异常把组装永久拖死;
  similar-asin 在原有 typeMismatch 跳过分支旁补同款处理
- TaskFileJobService.hasExhaustedAssembleJob + stale recovery:已有「重试耗尽且
  已终态收尾」的 assemble job 时放弃恢复,交给 finalizeStaleTask 按失败收尾
  —— 用户看到明确失败,而不是无限 RUNNING

实测:部署后两节点 28459 相关日志、rustfs 确定性错误、stale recovery 重建全部归零。
2026-09-17 01:19:14 +08:00
huangzd1997 d5952945dd fix(rustfs): 确定性错误不再重试;回退两处经实测无效的连接池实验
- 新增 isNonRetryable:NoSuchKey / AccessDenied / 签名错误等确定性错误立即失败,
  不再白打两次请求(线上 read 一个已清理的 chunk 会连打 3 次并留下一条 ERROR)
- connectionPoolMaxIdle 0→5、keepAlive 30000→300000 回退:
  实测证明 unexpected end of stream 与连接复用无关——禁用复用后新容器起来的
  第一次请求照样中招。至此已排除公网链路、keepAlive 过长、连接复用三项;
  另用 mc 并发压 200 个小对象全部成功,说明 RustFS 服务端无问题。
  剩余方向指向 MinIO SDK(8.5.17,已是最新)/OkHttp 与 RustFS 的协议细节,
  需抓包对比 mc 与 Java 的请求才能定位。数据不丢(重试都能成功),影响是每次 +600ms。
2026-09-17 00:58:01 +08:00
huangzd1997 6e1689dfe5 docs(更新日志): 追加客户端 4.0.27 条目 2026-09-17 00:33:29 +08:00
huangzd1997 4f16a02658 fix(稳定性): 修任务锁超时/实例转发/扫描降噪,补 rustfs 结论日志
线上日志扫描发现的缺陷批量修复:
- TaskFileJobService.resetStuckRunningJobsDetailed 去掉跨行长事务:两节点
  @Scheduled 各取 200 行在同一事务里逐行 UPDATE,互等行锁导致 biz_task_file_job
  每天上百次 Lock wait timeout。改逐行独立提交(本就有 CAS 条件,幂等)
- TaskOwnerForwardService 连接类失败重试 3 次:两节点滚动重启窗口内的
  Connection refused 会直接丢掉用户提交的结果(读超时不重放,避免重复提交)
- AdminApiGuardFilter 把 401/被顶下线降为 debug(线上单节点一天近 3000 条噪声)
- NoResourceFoundException 单独处理:如实返回 404,不再刷 ERROR 堆栈、不再伪装
  200(每天 580 条,绝大多数是外部扫描 /.env、/credentials、aliyun.json 等)
- RustfsObjectStorageService 补「重试后成功 / 重试耗尽」结论日志:一天上千条
  retrying 却无结论,无法判断数据是否落盘
- 连接池 keep-alive 300s→30s,减少复用已被对端关闭的空闲连接
- SimilarAsin 提交结果遇到已删除任务时带 TASK_NOT_FOUND(40401),客户端据此
  停止每 30 秒一轮的无谓重试
- 修 SimilarAsinResultRowDto 的 GBK 乱码 JsonAlias("浠锋牸" → "价格")
2026-09-17 00:32:48 +08:00
huangzd1997 5e9a59b327 test(鉴权): 兜底过滤器用例改用不存在的样例豁免前缀
exemptPrefixes 是配置化操作出口(aiimage.security.admin-guard-exempt-prefixes,
默认空、生产未配置任何值),原用例拿已下线的 /api/admin/shop-credential-checks
当样例路径,容易被误读成「该端点已配好豁免」——那条路径是 2026-09-04 引入
豁免机制时随手取的,并非真的配过。换成显然不存在的 /api/admin/example-exempt,
并注明不要改成真实业务端点。
2026-09-17 00:13:27 +08:00
huangzd1997 ab09cff427 docs(更新日志): 追加客户端 4.0.26 条目 2026-09-16 22:55:31 +08:00
huangzd1997 47a9520a82 fix(跟价): 失败任务也产出并保留部分结果文件,不再只剩一条失败记录
会话掉线这类「跑到第 N 页才断」的场景,任务本就该是 FAILED,但已经跑出来的行
(哪些 ASIN 真实改价过)必须随任务一起交付,否则用户无从对账、也无法只重跑漏掉的。

- submitResult 的 error / success=false 分支改走 finalizeFailedShop:先按失败落库
  (success=0 + 原因),若合并后仍有可用行就排队组装部分结果文件;一行可用数据都没有时
  才退化成原来的纯失败(并清掉合并缓存)
- markResultFilePending / assembleShopResult 增加 preserveFailure 语义:失败行只挂文件,
  不把 success 洗成 1、也不清 errorMessage
- updateTaskStatusFromLatestRows:失败行正在组装文件时不让任务提前终态,
  否则前端一停轮询、下载按钮永不出现(失败任务的结果文件同样要能下载)

PriceTrackTaskServiceTest 新增 2 条契约测试(部分结果排队 + 组装完成后仍失败且可下载),
跟价模块测试 20/20 通过。
2026-09-16 20:33:31 +08:00
huangzd1997 7a6c3c3fa3 fix(菜单权限): 授权树按操作者置灰不可授予项 + 放行保留既有授权
普通管理员勾到无权授予的菜单后,ensureGrantable 抛 403,而 createUser/updateUser
都是 @Transactional,整笔回滚——线上表现就是「后台不能保存权限,也不能创建用户」
(2026-09-16 周丽娥账号创建用户与保存 uid=45 权限双双失败,库里查无新用户)。
根因是校验 2026-07-28 就加了,但授权树一直拿全量菜单,两边规则不一致。

- permission-menus 按操作者标记 grantable;授权树据此置灰(不隐藏:授权是整树替换,
  隐藏会把超管授予过、操作者自己没有的菜单当取消勾选删掉,与 09-13「权限自己没掉」同类)
- ensureGrantable 放行目标已持有的授权,只拦新增,不构成提权
- GlobalExceptionHandler 补业务异常日志:此前普通业务异常一行都不记,本次排查只能靠
  反推响应体字节数(nginx body_bytes_sent 含 chunked 开销)才定位到根因
2026-09-16 18:46:21 +08:00
huangzd1997 803b5d583d docs(更新日志): 追加客户端 4.0.25 条目
跟价「指定ASIN模式」搜索打到屏外的导航搜索框,客户端 4.0.25 已修(app_client 29c12f2)。
2026-09-16 17:15:06 +08:00
huangzd1997 1c52cd529b fix(去重): 历史任务弹层补下载/删除按钮(漏了 history-item-actions 插槽)
任务面板的 #item-actions 只作用于「当前任务」列表,历史结果全部只在
#history-item-actions 弹层里渲染。16 个工具页里唯独去重页只给了前者,
表现为用户反馈「历史任务里压根没下载按钮」(既无下载也无删除)。
接口数据(success/resultId/downloadUrl)完整、类型检查与单测均不报错,
只能靠点开历史弹层才能发现。
2026-09-16 17:05:11 +08:00
huangzd1997 ab28b168ec fix(去重): 主链接行位于其子链接行之后时未被丢弃 + 导出表头与数据行错列
线上任务 28422(文件 新数据变体完.xlsx)结果中 16 个主ID同时保留了主链接与
子链接,违反所选规则(keepIntegerIds=false / keepUnderscoreIds=true /
keepIntegerMainIdsWhenNoSubIds=true)。

根因:旧实现把整数主链接行暂存,依赖「后到的同主ID子链接行」把它丢弃。
当顺序为「子链接在前、主链接在后」时,子行到来时暂存区尚空,无人记录该主ID
已有子链接,主链接一路存活到 flush。改用 IdRuleRowPicker 先收集、收尾统一按
「该主ID是否出现过子链接行」判定,与源文件顺序无关;补 subRows/mainRows/
droppedMainRows 排查日志。

同时修一处潜在错列:表头按 selectedColumns 原顺序写、数据行按
orderedSelectedColumns(id/ASIN/国家/价格/品牌 提前)写,两者不同序时整体
错列;本任务所选列恰为导出优先级顺序故未暴露。

实测:用线上源文件重算,输出 7273 → 7257 行(正好少掉那 16 个主/子并存的主ID)。
2026-09-16 16:33:04 +08:00
huangzd1997 367b4b7553 chore(更新日志): 补 4.0.24 条目(跟价改版为行内输入+底部「全部保存」提交) 2026-09-16 16:03:42 +08:00
huangzd1997 25c7323c47 chore(更新日志): 补 4.0.23 条目(跟价适配亚马逊新版商品页) 2026-09-16 13:40:59 +08:00
huangzd1997 aea0e16279 feat(站内通知): 铃铛支持按类型大类筛选(系统异常/任务异常/配置异常/系统通知)
原来铃铛只能按关键字和日期筛,任务失败、密钥欠费、麦象异常全混在一起,
用户没法只看自己关心的那类。

后端 NotificationService 增加 scene→大类映射(未知 scene 归「系统通知」兜底),
列表接口新增 category 参数,落到 SQL 是 scene IN (...);「系统通知」作兜底类
还要纳入未登记的 scene,故表达为「= system 或 不在其它三类里」。列表项返回
category,前端不必各自维护一份 scene 映射。前后端两个通知接口同步加参数。

前端铃铛筛选区新增一行类型 chip(全部 + 四类),与关键字、日期一起参与
hasFilter 与重置;空分类不下发,避免后端查询落空。
2026-09-16 10:32:11 +08:00
huangzd1997 f566573fce fix(菜单权限): 后台侧边栏为部分授权用户补全祖先分组(二级菜单不再平铺)
只勾选分组内子页面、未勾选分组本身时,getUserColumnPermissions 只返回
直接授权+展开后代,分组容器节点缺失,AdminMenuTreeBuilder 把子页面当根节点,
前端侧边栏渲染成一列平铺;且子页面 sort_order 是全局值、跨分组穿插显示,
与超管的「分组+组内顺序」视图完全对不上(线上 31 个账号如此,含 uid=972 阿武)。

修复:getUserColumnPermissions 增加 includeAncestorGroups 重载(默认 false),
true 时沿 parentId 链把有可见后代的祖先并入集合——分组无页面路由,仅还原
展示层级,不构成授权扩展;只有 current-user/menus(后台侧边栏)启用。

刻意保持原语义的出口:/permission-users/{id}/column-permissions 与登录响应
(桌面端 tool-catalog「组键命中即整组放行」,补组键会误放行整组工具)、
dedupe/invalidasin 的精确 key 校验。

新增 3 个测试:祖先补全含多层链、树组装还原分组与组内顺序、默认出口不含祖先。
2026-09-16 09:58:33 +08:00
huangzd1997 8fcceb3226 fix(撞款扫描): 保留期 90→7 天,并单独兜住最新 SUCCESS 行
该表每行含整份聚合 payload(线上实测约 2.3MB/行,24 行占 55MB),
而读取侧只认最新一行(selectLatestFullRow/LightRow 都是 ORDER BY id DESC LIMIT 1),
历史行纯占磁盘。保留期降到 7 天,行数下限仍由 PROTECTED_ROWS=10 兜底。

同时补一条保护线:连续失败多日时,最新 SUCCESS 行会滑出「最新 N 行」保护窗口,
再按时间线被删掉,界面就空白了——现在取「第 N 新行」与「最新 SUCCESS 行」
两者更靠前的那个 id 作为保护线,确保它永不被删。
2026-09-16 09:25:30 +08:00
huangzd1997 ed0d6575c8 feat(前端): 追加客户端 4.0.22 更新日志条目
包体积 -38%、启动时清理本地垃圾、启动与连接优化三条面向用户的说明。
version 与 web_config 发布记录一致(4.0.22),否则更新面板按本机版本筛选匹配不到。
2026-09-16 09:09:05 +08:00
huangzd1997 f2ada02383 feat(保留期清理): 补齐 7 处只增不删的数据;修结果对象孤儿
审核发现一批表/元数据只增不删,且对象存储存在活跃泄漏:

- module-cleanup 漏收 task_result_item / task_result_payload 的 payload 指针,
  每晚删完行就在 json-server 桶留下孤儿对象;指针收集触顶从"截断后照样删整组"
  改为任务组二分拆分,拆到单任务仍触顶则整组不删并记 error(截断会让指针随行消失)
- 结果文件对象此前从不回收(只删 DB 行、对象留在桶里),现改为事务提交后
  按 file_result / task_file_job 的 result_file_url 逐个回收
- 新增 7 个保留期清理:device_log_file 元数据、price_track_loop_run、撞款扫描行、
  密钥用量日统计、紫鸟记忆过期行、PUBLISH 任务、BRAND 任务。后两者刻意不复用
  module-cleanup(BRAND 不写 biz_file_task;PUBLISH 单任务可达数万行),
  改为调各自既有的业务删除入口,保证结果对象与子表一起回收
- 未读通知补保留期(180 天,已读仍 90 天);ZiniaoMemoryStoreService.deleteExpired
  此前全库无调用方(实现了没接线),现已挂定时任务

全部沿用既有范式:@Scheduled + 分布式锁单实例 + 分批 + 单轮批次数上限 + 失败只记日志。
2026-09-16 01:59:28 +08:00
huangzd1997 5e5816cd74 fix(站内通知): 麦象异常扫描滞留检查改用 update_time 过滤(18960 console 新增 update_time_end)——原按 create_time 过滤 + desc(id) 分页只能看到最新批次,积压深处的老任务永远看不到(2026-09-15 滞留在队 9 小时未被告警指出的漏报根因) 2026-09-16 01:05:26 +08:00
huangzd1997 d189d94c3b fix(更新日志): 面板随指定版本下拉联动;本机已是最新时展示本机条目
- 「指定版本更新」下拉选中版本时日志切到该版本自身的条目(升级/回退统一语义),
  未收录显示"暂无更新说明"(区块不消失),选回空项回默认区间;
  列表加载后自动默认选中第一条不触发切换
- 本机版本==线上最新时回退显示本机版本条目(原来整块消失),灰度/回滚仍隐藏
- 精简 4.0.20/4.0.21 超长文案,修复既有红测试(更新说明限 40 字)
2026-09-16 01:02:13 +08:00
huangzd1997 79b5d40327 feat(设备日志/后台配置/密钥检测): 补齐已上线未提交的设备日志与app_config,并合入检测模型解耦
三部分均已部署到双节点(当前线上 JAR 009efc3e),本次补齐仓库状态,避免"已上线未提交"
在后续最小构建比对里被误判。

- 设备日志管理:modules/devicelog + DeviceLogOssProperties(主机B 独立 MinIO,仅内网)
  + V127 device_log_file/device_log_config + admin-vue 日志管理页;客户端/麦象按 offset
  增量上报(X-Internal-Token),查询仅超管
- 后台通用配置:modules/appconfig + V128 app_config 键值表;工作台「开店流程」密码改服务端
  校验(POST /api/kd-flow/verify),改密码只需 UPDATE 该行、客户端无需重新发版
- 密钥检测模型解耦:新增 aiimage.user-secret.check-model(env AIIMAGE_USER_SECRET_CHECK_MODEL),
  默认 doubao-seed-2-0-lite-260215 —— 同系列 mini 在中继分组下无可用渠道(503 model_not_found),
  实测 lite 可路由;UserSecretModule 去掉 LlmTarget 改 resolveLlmHost,探测日志带 model=
- 前端:密钥面板「检测配置密钥」按钮不再折行;client-changelog 补 4.0.19/4.0.20/4.0.21 条目

注:application.yml 与 PropertiesConfig 同时承载上述多个部分,故未按功能拆分提交
2026-09-15 23:53:05 +08:00
huangzd1997 228d481211 perf(全链路): 连接池/事务边界/轮询与 IO 效率收口
Java:
- LLM 180s 读超时不再被全局 call-timeout 静默截断成 90s(长思考请求被掐断→重试→付费网关二次计费)
- 代理 HttpClient 缓存改有界 LRU(jikip 每次提取新 IP,无界缓存持续泄漏 selector 线程与连接池)
- 12 个 service 的 Redis 任务锁移出 @Transactional(自旋最坏 10s 白占 DB 连接,池仅 30),远端对象删/传改 afterCommit
- 结果文件 Job 闸门拒绝时不再回退内联执行(改重新入队,避免把背压转嫁给 MQ 消费线程)
- imagevideo 每秒扫描加列投影、过期清理加 LIMIT;权限页整表查询改列投影(不再拉回密码哈希)
- 哈希改 HexFormat;补 5 处"不能改"的技术依据注释(批量插入会丢回填主键、流式丢模板与图片等)

前端:4 个工具页轮询改轻量端点(带 fallback);PriceTrack 快照节流写盘;候选店铺表分页;页面隐藏时停表

客户端:HTTP 连接池按出口复用(Session 仍每请求新建,保持无跨请求状态);品牌检测 WIPO 逐请求握手;
代理配置按 mtime 缓存;串行任务改专属池;异常降级为标签页重连;紫鸟启动改端口轮询;模板编译缓存;
日志上报连接与落盘收口;Flask 版本 API 改按请求复用连接
2026-09-15 23:01:59 +08:00
huangzd1997 d6f8368493 feat(站内通知): 麦象异常扫描上线——任务停滞/失败/队列积压/批量空结果推管理员
用户要求「maixiang异常也要发通知」,确认范围(卡住/失败/队列堆积/服务不可用)与
受众(只发管理员)后实现:
- MaixiangConsoleClient:18960 后台只读接口客户端(batch/tasks、tasks、queue/status、
  batch/detail),token 鉴权、失败只记日志。URL 必须用 URI 对象提交——字符串形态会被
  RestClient 二次编码(%20→%2520),end_time 参数实测报 Incorrect DATETIME value,
  检查会静默失效;已加回归单测 buildUrlEncodesTimeParameterExactlyOnce
- MaixiangAnomalyScanner 五类检查:①批量任务停滞(status 0/1 超时无更新,默认 60 分钟)
  ②单任务滞留(创建超 30 分钟未完成)③近 30 分钟失败达最小条数 ④批量任务空结果
  (worker 报错时批次仍被标"已完成",是唯一可抓的批量失败信号)⑤队列积压(pending≥300
  / processing≥100);去重按天/小时/任务,全部为管理员全局事件(subjectUserId=null)
- 接入 NotificationScanScheduler(分布式锁内同跑);scene=maixiang_anomaly;
  AIIMAGE_NOTIFICATION_MAIXIANG_* 环境变量可调,令牌留空=跳过
- 前端 NotificationScene 类型补 maixiang_anomaly(铃铛不按 scene 渲染,无运行时改动)
- 测试:客户端解析(真实抓包样例)+ 扫描器 14 例 + 手工联调探针 MaixiangLiveProbeTest
  (-Dmaixiang.live=true 开启,只读不写通知)

已部署双节点(.env 加 console token、JAR cd3e06c4)并线上验证:首扫推 2×15 条管理员通知
(3 个变体任务停滞 / 39 个跟价任务滞留),二次扫描落库=0 去重生效。
2026-09-15 12:20:15 +08:00
huangzd1997 b0f764b6b6 feat(前端): 追加客户端 4.0.18 更新日志条目 2026-09-15 11:33:53 +08:00
huangzd1997 1403fec5fe feat(前端): 追加客户端 4.0.17 更新日志条目
跟价购物车归属未知跳过改价、店铺名回落、失败原因如实带出三条面向用户的说明。
2026-09-15 01:27:57 +08:00
huangzd1997 07b4ebe983 feat(成本): 密钥检测防抖 + 巡检降频隔日 + 客户端 4.0.16 更新日志
- 用户密钥「检测」90 秒新鲜期:同值重复检测复用上次通过结果,
  代理提取不再因连点/手滑重复扣费(只缓存 passed,失败允许立即重试)
- 连通性巡检由每日降为隔日(cron 0 30 4 */2 * *,双实例锁不变)
- changelog 追加 4.0.16 条目
2026-09-14 19:43:22 +08:00
huangzd1997 0b2b9303d0 chore(前端): 追加客户端 4.0.15 更新日志条目 2026-09-14 18:10:48 +08:00
huangzd1997 05a2c479a5 fix(测试): CollectDataNoUploadStaleTest 的 updateById 重载歧义(补编译验证) 2026-09-14 18:00:07 +08:00
huangzd1997 52b55df7b2 feat(任务判死): 心跳正常但 180 分钟无结果上报的二次判死线(13 模块)+ 同期待发改动
判死线(治 28131 型「主线程卡死、心跳线程照发」):
- 判据改看 biz_task_scope_state.last_chunk_at(HTTP 心跳不刷新它);从未上报跳过不判
- 中央线覆盖 DELETE_BRAND/PRODUCT_RISK_RESOLVE/PRICE_TRACK/SHOP_MATCH/PATROL_DELETE/QUERY_ASIN/WITHDRAW
- 自带线接入 COLLECT_DATA/SIMILAR_ASIN/APPEARANCE_PATENT/SHOP_DATA_CRAWL/PUBLISH/BRAND
- 客户端心跳带处理位置 progressText,判死文案含最后位置;no-result-upload-timeout-minutes 默认 180(0 关闭)

同期带上另一工作流的待发改动:跟价换 IP 重试、品牌检测重试上限与 LLM 并发下调、
教程包后台管理页与 V126 迁移、admin-vue 教程记录页。
2026-09-14 17:56:20 +08:00
huangzd1997 b54f72d3f6 fix(品牌检测): 熔断中止落真实原因+部分结果可下载,熔断加持续时长门槛
- 熔断改为「连续失败持续 6 分钟未恢复」才中止:原 8 次即中止,5 线程一轮
  就能凑满,重跑机制没机会生效导致任务失败率过高(任务 2309 复盘)
- 期间每 30s 冷却重试,重跑轮次 10→30;限流窗口恢复后任务自动跑完
- 失败终态上报内部接口 /api/internal/brand/tasks/{id}/abort:落真实原因 +
  用已收分片部分组装结果(未检测品牌单独成 sheet),不再悬挂到心跳超时被
  判「前端长时间无响应」且已跑数据无法下载
- 组装前从分片重建聚合(缓存快照可能缺后加字段如 keptBrands)
- 前端:失败任务有结果文件即显示「下载结果」
2026-09-14 17:04:19 +08:00
huangzd1997 24c5a09c7f feat(任务派发): 客户端兜底拉取页面未推送成功的任务 + 修店铺匹配定时任务误杀
问题:任务派发链路的"推送"只存在于页面里(Java 解析落库 PENDING → activate →
pywebview 桥 enqueue_json 推本机队列)。只解析没点启动、推送前关页面、在纯浏览器
打开,任务都会停在 PENDING,2 小时后被 StaleTaskRepairService 标失败
(「任务长期未被领取,已自动失败」,09-11 生产清理过 263 条同画像)。

- 服务端新增 GET /api/tasks/pull-pending(TaskClientPullController,身份从 JWT 取,
  不接受 user_id 参数):只挑创建超 5 分钟仍 PENDING 的本用户任务,逐条条件更新认领
  (PENDING→RUNNING + 接管 owner_instance_id)——与页面 activate 同一谓词,天然互斥,
  不会重复执行;认领后组装不出载荷则标 FAILED,不留 RUNNING 孤儿
- payload 由各业务模块实现 ClientTaskPullSpi 组装(task 侧不 import 业务模块,同 G5):
  首批 SIMILAR_ASIN / COLLECT_DATA / APPEARANCE_PATENT——这三个 Python 消费端会自行
  回拉明细,故载荷极简、客户端零模块知识;开关 aiimage.client-task-pull.enabled 默认 false
- 防双执行:三处 activate 由「非终态即可」收紧为只认 PENDING,未命中抛
  「任务已在执行中(可能已由客户端自动接管),无需重复启动」(顺带堵住整行 updateById
  把认领写入的 owner 覆盖回去的竞态);两个前端页 activate 失败即提示并停止入队
- 客户端:amazon/main.py 新增 pending_task_pull_worker,启动点挂在 app_client/main.py
  的 start_task_monitor(独立入口的 worker 线上并不生效);开关 pending_pull_enabled 用
  getattr 读取,避免 test/ 下的旧 config 缺键导致整包导入失败
- 同批修:StaleTaskRepairService 的 SCHEDULED 分支改按 scheduled_at + 120min 判死
  (原按 updated_at 会必杀排期 >2h 的店铺匹配定时任务,而 activate 又被 scheduledAt-90s 挡住)
- 测试:TaskClientPullServiceTest / StaleTaskRepairServiceTest / CollectDataTaskPullSpiImplTest /
  CollectDataActivateGuardTest 共 20 例;客户端 pending_task_pull_worker 7 例并更新启动顺序契约测试
2026-09-14 16:23:34 +08:00
huangzd1997 8cab9d4bad fix(密钥配置): 保存沿用刚检测过的输入值结果,消除「三项检测通过却提示未检测」死循环
- 服务端:对「输入值(未保存)」的检测结果按 uid+模块+值指纹暂存 Redis(TTL 30min,
  Redis 异常降级为需重新检测,不阻断保存);保存同一个值时落库该结果
  (passed/failed/error 一并沿用),改过值或从未检测则维持未检测
- 前端:保存后清理本地「输入值(未保存)」绿字,展示统一走服务端快照,避免展示与门禁矛盾;
  门禁提示改列「模块名(掩码):未检测 / 检测失败:原因 / 未配置」,同名掩码也能分辨模块
- 测试:UserApiSecretServiceTest 补沿用 / 值不一致 / Redis 降级用例;
  新增前后端一致性守卫测试(保存后必须清检测结果、提示必须带模块名)
2026-09-14 13:49:25 +08:00
huangzd1997 5ea52e5291 perf(F5+): 行数据按需拉取(结果行版本信号)+ 修复 progress/light 恒判 missing
行数据按需拉取(审查 F5 后续):
- V125 给 biz_file_result 补 updated_at(DEFAULT/ON UPDATE 由数据库维护,
  实体标注 insertStrategy/updateStrategy=NEVER —— 否则 selectById→updateById 的
  写回会把旧值写回去、ON UPDATE 不触发,版本信号静默冻结)
- 装配器回传 rowsVersion=「最后变更时间毫秒#行数」,5 个品牌工具页版本未变即跳过
  带行明细的重型 batch;前端变更信号为 rowsVersion + status/fileStatus/fileReady 复合
  (任务收尾常见「行早写完、之后才置成功」,只看行版本会把界面卡在旧状态)

修复线上缺陷(同一功能验证时暴露):
- TaskProgressLightAssembler 列裁剪漏选 module_type 却用它做模块过滤 →
  getModuleType() 恒为 null → light 恒把任务判成 missing;第七批把 light 接进
  跟价/定时匹配/商品风险的轮询后,消费方会把运行中任务判为 FAILED
- 补选中列 + 守卫用例 taskQueryMustSelectModuleType(已反向验证:去掉修复即红)
- 前端 lightClaimsAllTasksMissing:整体性 missing 结论用重型端点复核后再采信

契约与文档:light 白名单补 rowsVersion(Java 契约测试 / spec 06 §2 / 12 个端点描述)
测试:mvn test 2901 全绿;前端 npm test 765 全绿
2026-09-14 12:08:25 +08:00
huangzd1997 9166656673 perf(C6): 去重总数据列表顺序翻页改 keyset(前后端契约一起改)
背景:列表页 `ORDER BY id DESC LIMIT offset,size`,带筛选且选择性低时每页都要
对命中集做一次 filesort;深翻页 offset 也白扫索引。

改法(保持跳页/回退/改每页的原有行为):
- 后端:page 接口新增可选 `last_id` 游标 —— 传了就 `id < lastId` + `LIMIT size`(无 offset),
  响应新增 `nextLastId`(本页最后一行 id);不传仍是原 OFFSET 分页
- 管理前端:只有"下一页"用游标(上一页响应带回),跳页/改每页/筛选清空游标走 OFFSET
- 测试:新增 2 个后端契约测试(keyset 无 offset + nextLastId;无游标保持 LIMIT 30,15)

验证:mvn test 2897 全绿;admin-frontend-vue vue-tsc 通过 + 1619 测试全绿;
已部署 JAR 2b9ab774c60c3f5d802c9a2cee549008(双节点 health=200)与 admin-vue-20260914-112055。
2026-09-14 11:21:40 +08:00
huangzd1997 1360a44e01 fix(web): 货源查询/外观专利入队不再把 api_key 当必填
密钥已服务端化,本机无明文属正常状态,Java 侧 readApiKey 会按 uid 从用户密钥表兜底;
列成必填会把未在本机手输密钥的用户全部挡在启动之前。
2026-09-14 11:21:09 +08:00
huangzd1997 86c05e71a2 fix(web): 补齐命令式调用的 EP 按需样式,修拦截弹窗贴到文档左上角
Element Plus 走按需引入,只有模板里用到的组件才会注入样式;ElMessage /
ElMessageBox 都是 import 后函数式调用,构建产物里一条 .el-message-box 规则都没有,
弹窗因此没有定位与遮罩,渲染成普通块元素压在页头文字上(ElMessage 提示条同样受影响)。
入口显式引入两个样式模块,并加测试钉住「显式 import 的 EP 组件必须有样式引入」。
2026-09-14 11:21:08 +08:00
huangzd1997 fae26aa460 style(version): 指定版本下拉只显示版本号,去掉发布日期 2026-09-14 11:08:07 +08:00
huangzd1997 8803e22f39 feat(version): 桌面端更新面板支持指定版本安装
- 新增公开接口 GET /api/version/list(最近 50 条,字段口径同管理端列表),
  桌面端下拉不再拼 OSS 地址(版本包被删后拼地址会 404)
- 更新面板(登录页 + 首页)加「指定版本更新」选择器:默认选中最新版,
  可选全部已发布版本(含回退),回退/同版给出明确文案与二次确认
- 客户端无需发版:do_update_app(file_url) 本就接受任意版本包直链
2026-09-14 11:05:30 +08:00
huangzd1997 375b89154b perf(C5): 外观专利服务解析改走流式解析器(最后一条活路径)
AppearancePatentTaskService.parseWorkbook 原为 POI 全量 DOM(用户源文件整表入堆);
改为复用已有的流式孪生实现 AppearancePatentExcelParser(EasyExcel SAX,语义一致且自带单测),
服务侧只保留原有业务处理(分组键/状态过滤/字段补齐/hydrate)。外观专利模块测试全绿。

至此:用户源文件的解析已无 DOM 路径(剩余 DOM 仅用于报表模板写入,样式/图片/公式必须 DOM)。
2026-09-14 10:58:59 +08:00
huangzd1997 6a90adc765 feat(A1/A3): 三个客户端零调用前缀立即无条件收紧
实测桌面端 Python 侧调用面:/api/image-video、/api/task-file-jobs、/api/ziniao 为 0 次调用
(只有带 JWT 的网页端在用),因此从 user-tool-guard-enabled 开关名单移入无条件守卫名单,
不必等客户端铺开即完成收紧;其余 14 个前缀的客户端调用面与令牌携带情况已逐一实测,
仍在开关后面(老客户端不带身份,提前打开会 401)。

新增 2 个守卫测试:开关关闭时这三组前缀对匿名同样 401、带用户令牌仍放行。
2026-09-14 10:46:05 +08:00
huangzd1997 bcf66dc1d7 fix(startup): 两个 GroupDeletionGuard 显式命名 + Bean 名唯一性守卫测试
事故:2026-09-14 边界收敛新增 invalidasin/dedupe 两个同名 GroupDeletionGuard,
Spring 默认用简单类名做 Bean 名 → 启动抛 ConflictingBeanDefinitionException,
主机 A 的 java-server 连续重启失败(health 不通),**单测全绿也发现不了**
(不起完整 Spring 上下文),部署后才知道。

- 两个守卫分别显式命名 @Service("invalidAsinGroupDeletionGuard") / ("dedupeGroupDeletionGuard")
- 新增 SpringComponentBeanNameUniquenessTest:扫描 main 源码,同简单名的组件必须显式命名,
  否则红测试(把这次事故固化成可回归的守卫)
- 已重新打包部署:JAR 1f1e82bda227f2b0768574f8fd7c32c9,双节点 health=200
2026-09-14 10:37:44 +08:00
huangzd1997 c55c4a140b feat(A1/A3+C8): 客户端令牌链路打通(按人鉴权就绪)+ 快照 JSON 写入节流
A1/A3 客户端令牌链路(服务端守卫已能按 JWT 鉴权,缺口在客户端不带身份)
- 前端:新增 user-token-bridge(纯逻辑,值变化才推送)+ user-token-sync(启动安装,
  pywebviewready/storage/60s 轮询补推),main.ts 接入;桥接口补 set_user_token
- 客户端:新增 crawler_core/user_token.py(持有 + 对自家 Java 端点注入
  Authorization: Bearer <jwt>,Session.request 包装,第三方域名不注入、已有头不覆盖、
  登出清空、SHUFUAI_DISABLE_USER_TOKEN_HOOK 可关);main.py 暴露桥方法并在启动安装钩子
- 服务端:守卫契约测试补 5 个用户态前缀用例(开关关闭放行 / 打开后匿名 401 /
  用户令牌通过 / 内部令牌仍放行 / /api/ziniao 第二层)
- 翻开关的前置条件(客户端铺开后置 aiimage.security.user-tool-guard-enabled=true)写入报告

C8 快照 JSON 写入节流
- 读端改以 biz_task_result_item 行为准、JSON 仅兜底(历史任务 JSON 仍是唯一副本时可用)
- 整档 JSON 改 30s 节流写,终态路径 force 立即写;新增 2 个契约测试钉住语义

验证:mvn test 2888 全绿;npm test 741 全绿;user_token 钩子自测(注入/归一/第三方跳过/不覆盖/登出)通过
2026-09-14 10:20:15 +08:00
huangzd1997 7643094f1d refactor(boundary): 跨模块循环依赖清零(8 对 → 0),task→业务 依赖归零
共享内核下沉(跨模块共享的"身份/组织/安全"类型进 common)
- AdminUserEntity/AdminUserMapper、ShopManageGroupEntity/ShopManageGroupMapper → common
- AdminAuthSupport(33 文件 17 模块引用)、JwtService、AuthProperties、DeviceSessionPolicy
  → common/security(原先放在 admin/auth 里,任何模块用一次就多一条跨模块边)

端口化(消费方声明接口、数据方实现)
- task/spi:ResultFileJobHandler 新增 resolveResultFileUrl 钩子(BRAND 特例从 Worker 收回);
  Worker 改为调用 handler.onSuccess(该钩子历史上从未被调用,withdraw 的收尾靠 Worker 里的
  WITHDRAW 特判硬编码——现两者都归位,task 侧不再 import withdraw/brand)
- admin/spi/UserSecretCleanupPort(删除用户级联清理密钥)、notification/spi/ProxyBalancePort
  (代理余额探测)、shopdatacrawl/spi/ManagedShopNamesPort(可管店铺名)
- shopkey/spi/GroupDeletionGuardPort:分组删除守卫改由各业务模块实现(invalidasin/dedupe 两个实现),
  shopkey 不再直连它们的 Mapper

棘轮与量化(2026-09-14 实测)
- ArchitectureBoundaryTest:task→业务 119 → **0**(基线钉死为 0,新增跨模块动作必须走 task/spi)
- 跨模块 import 行数 650 → 605;双向依赖对 8 → **0**(usersecret/permission/shopkey 三向环、
  task↔withdraw、task↔brand、admin↔permission、admin↔usersecret、dedup↔shopkey、
  invalidasin↔shopkey、notification↔usersecret、shopdatacrawl↔shopduplicatecheck 全部拆解)

测试同步:接口契约 8→9、品牌 Handler 钩子契约、worker 的 10 个测试构造实参、通知/管理测试端口化。
mvn test 2881 全绿。
2026-09-14 10:05:04 +08:00
huangzd1997 1b480f915f test(G4): 定时匹配/格式转换/数据拆分 补 27 个契约测试(三个模块此前均零测试文件)
- ShopMatchResolveService 15:候选越权与幂等、匹配去重保序、国家偏好默认顺序与坏 JSON 兜底
- ConvertTemplateService 8:内置模板禁删(软禁用)、导入命名/后缀补全、设为默认时清掉其它默认位
- SplitRunService 4:下载/删除历史必须属于本人且模块匹配(含无结果文件不可下载)

至此审查点名的 6 个零测试模块全部有测试文件。
2026-09-14 09:40:56 +08:00
huangzd1997 ac36c08460 test(G4): 取款/查询ASIN ResolveService 各补 13 个契约测试
取款:候选增删幂等与越权保护、按店铺名批量删的归一化去重、匹配去重保序。
查询ASIN:额外覆盖「国家 ASIN 清单」契约(国家顺序固定、空国家不出现、同国家去重),
该清单会整体推给 Python,格式错会直接导致采集错列。
2026-09-14 07:11:59 +08:00
huangzd1997 ef3a2c9bd6 docs(更新日志): 追加客户端 4.0.13 条目 2026-09-14 07:08:39 +08:00
huangzd1997 da0f10f1cc test(G4): 巡店删除 ResolveService 补 13 个契约测试(该模块此前零测试文件)
覆盖:用户校验、索引未命中/同名冲突拒绝、重复添加幂等、他人记录不可删(同文案防探测)、
匹配结果去重保序、count 空值兜底。
2026-09-14 07:05:04 +08:00
huangzd1997 9ba231dc4c feat(D2): 导入进度跨节点可见(NodeSharedStore:本地快路径 + Redis 真源)
- 新增 common/service/NodeSharedStore:本地 Map 快路径 + Redis 跨节点真源 + 写节流
  (默认 500ms,逐行刷新进度不会打爆 Redis)+ 本节点条目快照(维护用)+ TTL 兜底过期
- 接入 DedupeTotalDataService(8 个进度/归属/分组/完成时间映射)、QueryAsinService、
  SkipPriceAsinService(各 3 个):轮询落到另一节点不再报"任务不存在"
- 保留期清理改为遍历本节点快照(跨节点过期由 Redis TTL 兜底),不再依赖全量遍历
- 测试同步:去重服务测试的反射注入改用 NodeSharedStore(未注入 Redis 时等价纯本地)

mvn test 2815 全绿
2026-09-14 06:54:48 +08:00
huangzd1997 9a6b57db58 refactor+perf+fix: G5 模块边界 SPI 化、C5 流式解析、C6/C7 查询优化、D13 队列持久化、A3/A4/A7 鉴权
模块边界(G5 / G7)
- 新增 task/spi/TaskModuleHeartbeatSpi + 13 个模块实现:TaskHeartbeatService 不再 import 任何
  业务模块(原先注入 12 个 CacheService 并用 switch 分发);启动校验重复注册
- 新增 task/spi/BrandTaskHeartbeatSpi(品牌任务心跳/中断)、BrandTaskStaleRepairSpi(陈旧修复)、
  CollectDataItemCleanupSpi(历史清理):跨模块 Mapper 操作收回业务模块
- G7:10 个被跨模块借用的 productrisk VO 迁至 common/model/vo
- 架构棘轮收紧:TASK_TO_BUSINESS_BASELINE 119 → 6(实测)
- 新增 TaskModuleHeartbeatSpiCoverageTest(moduleType 覆盖与拼写)

性能与容量(C5/C6/C7/D13)
- C5 流式解析:SkipPriceAsinService(含两行表头语义)、AppearancePatentExcelParser、
  BrandTaskService、DeleteBrandRunService、LocalFileStorageService.getExcelInfo 改 ExcelStreamReader;
  行数上限改为迭代中生效
- C6 去重总数据列表:关键字改前缀匹配(命中 uk_data_value);V124 删除永不生效的 idx_country
- C7 撞款扫描:按店逐批取数(索引前缀),不再全表 GROUP BY + JOIN + 全量拉内存
- D13 待删对象本地日志 PendingDeleteJournal(启动回放 + 收敛重写),RustfsDeleteRetryService 与
  TransientPayloadDeleteOrchestrator 接入;异步删除失败对象写回日志
- V123 删除 biz_file_result 两个被复合索引覆盖的单列索引

安全(A3/A4/A7 + 守卫名单)
- A4 数字人版本写操作要求管理员;A7 视频密钥按登录身份(超管例外)
- A3 上传接口加危险扩展名黑名单(可配置)
- AdminApiGuardFilter 用户态名单补 /api/ziniao(controller 已 requireAdmin,此处为开关打开后的第二层)

容量(明细表保留期)
- 新增 ShopDataCrawlItemRetentionService:biz_shop_data_crawl_item 按保留期(默认 30 天)分批清理
  (该表此前无任何清理策略,是增长最快的表),job 锁 + 单轮批次上限
2026-09-14 06:46:27 +08:00
huangzd1997 67223f8950 perf(frontend): 工具页分页/历史截断/轻量轮询/共享纯逻辑(审查 F5/F6/F9/F12/G3)
- F9 新增 useTablePaging composable 并接入 7 个工具页"匹配结果"表(只切渲染窗口,
  不加选择语义;每页 100 条)+ 深色分页样式
- F6 历史任务抽屉渲染截断(默认 50 条 + "显示全部/收起"),全选口径改为当前可见条目
- F5 新增 task-progress-polling 适配器:等待任务终态的紧循环走 /tasks/progress/light,
  异常或空响应回退重型 batch;已接入跟价/定时匹配的 waitForTaskTerminal
- F12 背景图 bg.jpg 251KB → 166KB(1920 宽 + quality 80,image-set 仍优先 webp)
- G3 抽出共享纯逻辑 task-queue-state(开始时间表序列化校验 + 记录缺失错误判定),
  5 个工具页删除逐字重复实现
- 新增 4 个单测文件(分页切片/历史截断/轻量轮询归一/任务队列纯逻辑)
2026-09-14 06:23:46 +08:00
huangzd1997 e76714c32e fix(log): 陈旧巡检 summary 日志占位符与实参对齐
每段 5 个占位符(x={}/{})只传 4 个参数,导致 withdraw 之后取值整体错位、
末尾 elapsedMs/thread 被打成字面量;改为每段 4 个占位符。
2026-09-14 05:51:06 +08:00
huangzd1997 95dfb69a18 refactor(ziniao/shopkey)+fix(stale): 打破模块循环依赖 + 采集陈旧判死全局化
模块边界(消除 ziniao ↔ shopkey 真实循环依赖):
- 新增 ziniao/service/port/{ShopKeyCatalogPort,ManagedShopNamePort}:消费方声明契约
- shopkey 侧新增 ShopKeyCatalogAdapter(读 shop_key + 白名单状态回写)、
  ManagedShopNameAdapter(店铺名校验),实现上述端口
- ZiniaoApiKeyProvider 改经端口取数,不再 import shopkey 的 Mapper/Entity;
  ZiniaoShopSwitchService 改依赖 ManagedShopNamePort
- 结果:ziniao → shopkey 的 import 归零,依赖单向(shopkey → ziniao)

陈旧判死(G1 全局判死 + D9 条件更新,替代此前的 owner 过滤/旧实体覆盖):
- ShopDataCrawlTaskService.finalizeOwnedStaleTasks → finalizeStaleTasks:去掉 owner 过滤,
  并入 DeleteBrandStaleTaskService 的 stale-check 巡检线(job 锁保证单实例扫描)
- 判死前必须持有任务锁(非阻塞获取,锁被占本轮跳过),FAILED 写入改 status CAS,
  仅在确实由 RUNNING 翻转为 FAILED 时才删缓存与分片(原实现会用扫描期旧实体覆盖在途任务)
- tryFinalizeTask 增加 allowOwnerTakeover 重载:判死场景允许跨实例接管(P1-8 盲区)
- 测试同步:owner 契约用例改为全局判死口径;mock 的 CAS 需先渲染 SQL 片段
  (MyBatis-Plus 的 where 参数延迟填充)才读参数表;新增锁被占跳过的用例

mvn test 2796 全绿
2026-09-14 05:44:37 +08:00
huangzd1997 24ada70997 refactor: 结果下载直链解析去重(11 处 → 1 处)
新增 common/service/ResultDownloadResolver,7 个模块的 resolveResultDownloadUrl
改为委托调用(appearancepatent/queryasin/withdraw/pricetrack/productrisk/
patroldelete/shopmatch)。

顺带修掉两处隐患:
- 原实现用 userId.equals(entity.getUserId()),userId 为 null 时 NPE
  (新实现用 Objects.equals)
- shopmatch 原实现在 url 为空时抛"任务不存在",与语义不符,统一为"暂无可下载文件"

其余 4 处**刻意保留**,因为它们本就不是重复:
- shopdatacrawl:有 validateUserId + requireResultEntity + ensureResultOwner 前置校验
- deletebrand:返回 null 而非抛异常(调用方依赖该语义)
- similarasin:已分叉为返回 record ResultDownloadInfo
- brand:走的是另一套下载路径
文件名解析 resolveResultDownloadFilename 同样没收口:各模块兜底文件名策略确有差异

配套更新 3 个显式构造 Service 的测试(DelegationTest/HistoryBatchTest/
RollbackSemanticsContractTest)注入新依赖的 mock。mvn test 2795 个全绿。
2026-09-14 05:17:10 +08:00
huangzd1997 ccce03b4d4 fix: 第四批修复(结果读取并发化/分页钳制/JWT 密钥强校验/凭据按 id)
性能
- listResultSnapshots 对指针化载荷改并发读取(有界池 + 保序汇总):此前逐行同步对象存储读,
  500 行结果文件生成要多花数十秒。仅在确有指针行时才走池——内联 JSON 直接串行读,
  避免为本地读取引入调度抖动(性能基准测试容差会被影响)

安全/正确性
- 管理端 GET /{id}/credential 改为按路径 id 查询(此前忽略 id、改用 shop_name,
  会出现「路径声明的店铺」与「实际读取凭据的店铺」不一致);凭据 VO 构造抽公共方法
- server profile 下 JWT 密钥缺失即拒绝启动:此前静默回退到公开默认值(等同无防护,
  任何人可伪造 token),只留 warn 日志拦不住发布事故;本地/测试 profile 保持宽松

边界
- 分页/条数参数补齐上限钳制(文档早已声明"最大 N"但无校验):
  PriceTrack/ShopMatch 的 page_size → 2000;TaskFileJob 的 limit → 1000;
  AppearancePatent/Publish 的 limit → 100
2026-09-14 05:02:33 +08:00
huangzd1997 4daf235385 docs(更新日志): 追加客户端 4.0.12 条目 2026-09-14 04:36:35 +08:00
huangzd1997 6d46506726 fix: 全维度审查修复(安全/正确性/性能/稳定性/客户端/前端)
安全
- /api/ziniao/** 五个匿名接口加管理员鉴权(此前可匿名换取任意员工店铺登录令牌)
- 删除 Flask 遗留后门:默认密码建超管 + 每次启动写生产 users 表(服务端与客户端各一份)
- 进度/详情接口归属过滤:新增 TaskProgressOwnershipSupport,11 模块 progress/light 与
  /tasks/batch 接入,DTO 补 userId,前端 13 个查询封装补传(未传时后端不过滤,兼容旧端)
- 代理提取链接(含账密)不再明文入日志(新增 common/util/SecretMasking)
- 全局异常兜底不再回传原始异常信息;内部令牌比较改常量时间
- 登录加失败计数与锁定(10 次锁 15 分钟);品牌源文件下载加 SSRF 防护
- AdminApiGuardFilter 覆盖前缀从 2 扩到 15(开关默认 false,行为不变,为收紧做准备)
- 生产关闭 springdoc/knife4j(/doc.html 匿名可读全部接口定义)

正确性
- 40901/40902 拆分:锁竞争不再被伪装成 success=true(此前客户端停止重试、分片静默丢失)
- 假成功收敛:集采明细批量写失败改为抛出、去重 worker 异常标失败、4 个 worker 改判
  success 字段、publish 空 ASIN 行参与批次 flush、巡店删除全失败带 error 上报
- 客户端心跳 discard 移入 finally(7 模块,失败路径不再留僵尸 RUNNING 任务)
- 状态机条件更新:跟价停止循环、集采 activate/fail、imagevideo 归档回填、店铺匹配提交

性能
- 前端入口包 JS 1.05MB→204KB、CSS 355KB→10.7KB(Element Plus 改按需 + el-config-provider)
- 载荷引用计数按指针里的 taskId 收敛(原 JSON 列 IN 全表扫且逐行调用)
- 店铺明细多值批量 INSERT;快照 upsert 预载缓存;结果文件列改单条 UPDATE
- 新增迁移 V120(补 3 个缺失索引)/V121(删 4 个被覆盖的冗余索引)/V122(URL 前缀索引)

稳定性
- 新增 common/util/ThreadPools 有界线程池替换 5 处无界队列(防堆积 OOM)
- Redis 锁释放改 Lua 原子校验(原裸 delete 会误删他人已过期的锁)
- imagevideo 加死节点接管;锁续期失败重试;调度池 4→16;openStream 全部加超时
- 事务内远程对象删除移到提交后;启动恢复锁按实例命名

客户端
- 不再 taskkill /f /im chrome.exe(改为按调试端口精准回收,不杀用户自己的浏览器)
- 密码检测不再无条件杀紫鸟进程;品牌检测加全局互斥(代理池不再互相覆盖)
- base_dir 统一到 exe 目录(原被 os.getcwd() 覆盖,日志/缓存会分裂两个目录)
- 缓存加定时清理;图片下载加超时;mkstemp 句柄托管

测试
- 同步更新受影响的契约测试(构造器签名/条件更新/方法改名/新增接口方法等)
- 修复 FaultInjectionTest 等 3 处 mock 未 stub 流式 read 导致的读循环 OOM
- mvn test 2795 个测试全绿
2026-09-14 04:15:36 +08:00
huangzd1997 c448f49e30 fix(临时存储): RustFS 上传失败改为直接失败,不再静默回落 local 指针
多实例/容器化部署下 local 指针只有写入它的实例能读(跨节点读直接报错、容器重建即丢),
此前上传失败静默降级会把跨节点不可读的脏指针落库,故障延后到其它节点的合并/组装才爆。
现改为在写失败点抛 BusinessException(中文原因透传调用方),并新增
fallback-to-local-on-error 开关(默认 false)供单机部署回退旧行为;
跨实例读 local 指针的报错改中文并带 objectKey;超限回落策略不变。
2026-09-14 02:31:16 +08:00
huangzd1997 3f5a234c59 fix(测试): 修复 3 处既有红灯——快照行尾、缺失基准文档、失效的边界棘轮
这三处在本次重构开始前就是红的(已在裸 HEAD 上复现),一并修掉,使 backend-java
相关测试恢复全绿(550 测试 0 失败)。

1. SimilarAsinSnapshotTest:golden 文件在 core.autocrlf=true 的检出下是 CRLF,
   而渲染结果按 LF 拼接,断言逐字符比较只差换行符即失败。改为读入时统一行尾。
   (这是测试自身缺陷,不是解析行为变化——两边的可打印内容完全一致。)

2. TxDurationBenchmarkTest:依赖 docs/tx-duration-benchmark.md,而该文档从未提交过
   (git 历史中不存在),导致 2 个用例必然失败。补齐文档,按 mock 环境实测记录
   单次事务段基线、200 分片上界与总耗时上界,并注明该基线只用于相对劣化判定。

3. ArchitectureBoundaryTest:task→业务依赖棘轮冻结在 84(task-212 后的存量),
   此后 TaskHeartbeatService 跨模块心跳(13)、StaleTaskRepairService(4)、
   ModuleHistoryCleanupService(2)、TaskResultFileJobWorker(2)持续接入新模块,
   实测已达 110,棘轮长期失效(恒红=无人看)。对齐到 110 恢复告警,并写明
   「新增依赖请走 Handler SPI,不要直接上调」。
   注意:并行会话正在改 task 模块(含 StaleTaskRepairService),其改动落地后需重新实测。
2026-09-14 02:01:16 +08:00
huangzd1997 b05bba50fa refactor(similar-asin): 抽出 LlmPipelineSupport,Service 降至 1735 行(累计 -70.7%)
在上一提交(3455 行)基础上,把「Python 结果回传 → 分片落库 → LLM 检测」整条流水线
(57 个方法 / 1694 行)抽为 SimilarAsinPipelineSupport。等价搬移,未改行为。

采用依赖倒置消除循环依赖:support 包声明 SimilarAsinPipelineHost 接口
(finalizeTask / findOrCreateResultRecordForAssembly / readCategorySwitch /
finalizeExhaustedResultFileJob),由 SimilarAsinTaskService 实现;这几项保留在宿主
是因为它们属于编排与事务边界(handleResultFileJobFailure 带 @Transactional)。

同时把 5 个被两侧共用的内部 record 提为 support 包顶层类型
(SubmittedTaskMetadata / FinalizeTaskResult / SubmitContext /
 PersistSubmittedChunkResult / PreparedSubmittedChunk),3 个仅流水线内部使用的
record 内联进流水线类;LlmBatchContext 一并归位。

测试适配:4 个测试类里对 mergeLlmRowsIntoChunk / bufferLlmRowsOrMerge /
flushLlmBufferedResults 的反射改指向流水线实例;RollbackSemanticsContractTest
通过 pipelineSupport() 反射取得实例(跨包,保持封装)。

验证:干净工作区叠加本改动跑 similarasin 386 + task 引用方 166 测试,
结果与基线一致(仅既有失败),零新增失败。
2026-09-14 01:51:36 +08:00
huangzd1997 9d92fb4af2 refactor(similar-asin): SimilarAsinTaskService 拆分为 9 个 support 类(5914→3455 行)
从 5914 行的巨型 Service 中按内聚单元抽出 9 个 support 类 + 1 个顶层 record,
净减 2459 行(-41.6%)。全部为等价搬移(Javadoc 标注搬移来源),未改任何行为:

- SimilarAsinLimits            阈值/开关解析(parse 上限、chunk merge 上限、LLM batch/缓冲/flush)
- SimilarAsinPayloadSupport    解析载荷编解码与读取门面
- SimilarAsinPoisonTracker     毒行滑窗熔断状态(Service 从此无进程内可变状态)
- SimilarAsinChunkMergeSupport 行键族 + chunk 合并纯计算
- SimilarAsinResultTextSupport 结果行判定与用户可见文本渲染
- SimilarAsinChunkPayloadSupport chunk 载荷读取、读失败诊断、orphan 兜底
- SimilarAsinResultWorkbookAssembler 结果文件装配(xlsx/zip、POI、DISPIMG、并发)
- SimilarAsinTaskOwnershipSupport    实例归属判定与 per-task 分布式锁
- SimilarAsinTaskProgressSupport     文件构建进度、任务视图映射与计数
- LlmBatchContext              从内部 record 提为顶层,供归属与 LLM 流水线共用

顺带清理 7 处死代码(imageUrlCellValue、resolveResultDownloadUrl/Filename、
applyLlmToPersistedChunks、countCompletedLlmStates、hasPromptFields、
userFacingConclusion、@PreDestroy import)。

assembleExecutor 仍由 Service 持有,shutdownAssembleExecutor 语义不变(18 个测试未动)。
验证:在干净 HEAD worktree 上叠加本改动跑 similarasin 全量测试,结果与裸 HEAD 一致
(386 测试,仅 3 个 SimilarAsinSnapshotTest 既有失败),零新增失败。
2026-09-14 01:30:11 +08:00
550 changed files with 32976 additions and 7807 deletions
@@ -0,0 +1,79 @@
import { expect, test, type Page } from '@playwright/test'
// 教程管理页验收(module 记录与版本):列表新增「版本号」列 + 上传时间列,
// 默认按上传时间降序;版本号/上传时间表头可点击切换升降序;空版本历史行沉底。
// 依赖 scripts/mock-admin-server.mjs 的教程包夹具(4 条,其中 1 条无版本号、1 条 09-01 历史行)。
async function openTutorial(page: Page) {
await page.goto('/admin-vue/records/tutorial')
await expect(page.locator('.admin-topbar h1')).toHaveText('教程管理')
await expect(page.locator('.panel-box tbody tr').first()).toBeVisible()
}
const versionTexts = (page: Page) =>
page.locator('.panel-box tbody tr td:nth-child(3)').allTextContents()
const fileTexts = (page: Page) =>
page.locator('.panel-box tbody tr td:nth-child(2) .file-name').allTextContents()
test('test_tutorial_list_default_desc_by_upload_time', async ({ page }) => {
await openTutorial(page)
// 表头:版本号、上传时间(均带排序标记)
await expect(page.locator('.sort-version')).toHaveText(/版本号/)
await expect(page.locator('.sort-time')).toHaveText(/上传时间/)
// 默认排序状态:上传时间降序(▼),版本号未激活(▲▼)
await expect(page.locator('.sort-time .sort-mark')).toHaveText('▼')
await expect(page.locator('.sort-version .sort-mark')).toHaveText('▲▼')
// 默认按上传时间降序:09-14 → 09-12 → 09-10 → 09-01;空版本行沉底
expect(await versionTexts(page)).toEqual(['v3.10.0', 'v3.2.0', 'v3.1.0', '—'])
expect(await fileTexts(page)).toEqual([
'数富AI-教学客户端-v4.zip',
'数富AI-教学客户端-v3.zip',
'数富AI-教学客户端-v2.zip',
'数富AI-教学客户端.zip',
])
// 当前生效 = 最新上传(夹具里 09-14 那条),不随列表排序变化
await expect(page.locator('.tag-active')).toHaveCount(1)
await expect(page.locator('.tag-active').locator('xpath=..')).toHaveText(/数富AI-教学客户端-v4\.zip/)
})
test('test_tutorial_sort_toggle_by_version_and_time', async ({ page }) => {
await openTutorial(page)
// 点「版本号」:首次为降序(v3.10.0 按自然序大于 v3.2.0),空版本行仍在末尾
await page.locator('.sort-version').click()
await expect(page.locator('.sort-version .sort-mark')).toHaveText('▼')
expect(await versionTexts(page)).toEqual(['v3.10.0', 'v3.2.0', 'v3.1.0', '—'])
// 再点一次切升序
await page.locator('.sort-version').click()
await expect(page.locator('.sort-version .sort-mark')).toHaveText('▲')
expect(await versionTexts(page)).toEqual(['v3.1.0', 'v3.2.0', 'v3.10.0', '—'])
// 点回「上传时间」:切换字段时回到降序默认
await page.locator('.sort-time').click()
await expect(page.locator('.sort-time .sort-mark')).toHaveText('▼')
await expect(page.locator('.sort-version .sort-mark')).toHaveText('▲▼')
expect(await fileTexts(page)).toEqual([
'数富AI-教学客户端-v4.zip',
'数富AI-教学客户端-v3.zip',
'数富AI-教学客户端-v2.zip',
'数富AI-教学客户端.zip',
])
})
test('test_tutorial_upload_dialog_has_version_field', async ({ page }) => {
await openTutorial(page)
await page.getByRole('button', { name: '上传教程包' }).click()
const dialog = page.locator('.el-dialog')
await expect(dialog).toBeVisible()
await expect(dialog.locator('.el-form-item').first()).toContainText('版本号')
await expect(dialog.locator('input').first()).toHaveAttribute('maxlength', '64')
// 不选文件直接提交:提示仍以 zip 为必填(版本号可空)
await dialog.getByRole('button', { name: '上传教程包' }).click()
await expect(dialog.locator('.el-alert')).toContainText('请选择 zip 压缩包')
})
@@ -142,6 +142,15 @@ function json(res, payload, status = 200) {
res.end(body) res.end(body)
} }
/* ===================== 教程包夹具(教程管理页:版本号列 + 上传时间排序验收用) ===================== */
// 故意乱序给出,且含一条无版本号的历史行:页面默认应按上传时间降序、空版本行沉底。
const TUTORIAL_PACKAGES = [
{ id: 3, file_name: '数富AI-教学客户端-v3.zip', version: 'v3.2.0', object_key: 'tutorial/20260912090000-数富AI-教学客户端-v3.zip', file_size: 1048576, file_url: 'https://oss.aishufu.top/client/tutorial/t3.zip', created_at: '2026-09-12 09:00' },
{ id: 1, file_name: '数富AI-教学客户端.zip', version: '', object_key: 'tutorial/数富AI-教学客户端.zip', file_size: 0, file_url: 'https://oss.aishufu.top/client/tutorial/legacy.zip', created_at: '2026-09-01 08:00' },
{ id: 4, file_name: '数富AI-教学客户端-v4.zip', version: 'v3.10.0', object_key: 'tutorial/20260914103000-数富AI-教学客户端-v4.zip', file_size: 2097152, file_url: 'https://oss.aishufu.top/client/tutorial/t4.zip', created_at: '2026-09-14 10:30' },
{ id: 2, file_name: '数富AI-教学客户端-v2.zip', version: 'v3.1.0', object_key: 'tutorial/20260910120000-数富AI-教学客户端-v2.zip', file_size: 524288, file_url: 'https://oss.aishufu.top/client/tutorial/t2.zip', created_at: '2026-09-10 12:00' },
]
/* ===================== 站内通知夹具(铃铛面板:搜索/按天分组/分页验收用) ===================== */ /* ===================== 站内通知夹具(铃铛面板:搜索/按天分组/分页验收用) ===================== */
const NOTIFICATIONS = [ const NOTIFICATIONS = [
...[6, 5, 4, 3, 2, 1].map((seq) => notif(seq, '2026-09-13', 22, seq, seq <= 3)), ...[6, 5, 4, 3, 2, 1].map((seq) => notif(seq, '2026-09-13', 22, seq, seq <= 3)),
@@ -243,6 +252,11 @@ const server = createServer((req, res) => {
if (url === '/api/admin/notifications') { if (url === '/api/admin/notifications') {
return json(res, notificationPage(new URL(req.url || '/', 'http://127.0.0.1').searchParams)) return json(res, notificationPage(new URL(req.url || '/', 'http://127.0.0.1').searchParams))
} }
if (url === '/api/admin/tutorials') {
// 与 Java 侧同契约:created_at 降序返回(页面"当前生效"取第一条)。
const items = [...TUTORIAL_PACKAGES].sort((a, b) => (a.created_at < b.created_at ? 1 : -1))
return json(res, { success: true, data: { items } })
}
if (url.startsWith('/api/')) { if (url.startsWith('/api/')) {
return json(res, { success: true, data: { items: [], total: 0 } }) return json(res, { success: true, data: { items: [], total: 0 } })
} }
+135
View File
@@ -0,0 +1,135 @@
import { http } from './http'
import { unwrap } from './envelope'
/** 日志文件行(桌面客户端 / 麦象采集机上报)。 */
export interface DeviceLogFileRow {
id: number
source: string
deviceId: string
deviceName: string | null
username: string | null
uid: number | null
fileName: string
logDate: string
uploadedBytes: number
partCount: number
lastUploadAt: string | null
createdAt: string | null
}
export interface DeviceLogPage {
items: DeviceLogFileRow[]
total: number
page: number
pageSize: number
/** 云端日志保留天数(页面提示用)。 */
retentionDays: number
}
export interface DeviceLogContent {
fileId: number
fileName: string
content: string
totalBytes: number
shownBytes: number
truncated: boolean
}
export interface DeviceLogDevice {
source: string
deviceId: string
deviceName: string | null
lastUploadAt: string | null
}
export interface DeviceLogOverride {
id: number
source: string
deviceId: string
deviceName: string | null
mode: string
updatedAt: string | null
}
export interface DeviceLogConfigData {
globalMode: string
overrides: DeviceLogOverride[]
}
export interface DeviceLogQuery {
source?: string
keyword?: string
startDate?: string
endDate?: string
page: number
pageSize: number
}
/** 分页查询日志文件列表:GET /api/admin/device-logs/files */
export async function fetchDeviceLogFiles(params: DeviceLogQuery): Promise<DeviceLogPage> {
const query: Record<string, string | number> = { page: params.page, pageSize: params.pageSize }
if (params.source) query.source = params.source
if (params.keyword) query.keyword = params.keyword
if (params.startDate) query.startDate = params.startDate
if (params.endDate) query.endDate = params.endDate
const { data } = await http.get('/api/admin/device-logs/files', { params: query })
return unwrap<DeviceLogPage>(data)
}
/** 查看日志尾部内容:GET /api/admin/device-logs/content */
export async function fetchDeviceLogContent(fileId: number, maxBytes?: number): Promise<DeviceLogContent> {
const query: Record<string, number> = { fileId }
if (maxBytes) query.maxBytes = maxBytes
const { data } = await http.get('/api/admin/device-logs/content', { params: query })
return unwrap<DeviceLogContent>(data)
}
/** 完整日志下载地址(同域 cookie 鉴权,直接给 a[href] 或 window.open 用)。 */
export function deviceLogDownloadUrl(fileId: number): string {
return `/api/admin/device-logs/download?fileId=${fileId}`
}
/** 删除日志文件(片段与元数据,不可恢复):DELETE /api/admin/device-logs/{id} */
export async function deleteDeviceLogFile(id: number): Promise<void> {
const { data } = await http.delete(`/api/admin/device-logs/${id}`)
unwrap<unknown>(data)
}
/** 采集配置(全局默认 + 终端覆盖):GET /api/admin/device-logs/config */
export async function fetchDeviceLogConfig(keyword?: string): Promise<DeviceLogConfigData> {
const { data } = await http.get('/api/admin/device-logs/config', {
params: keyword ? { keyword } : undefined,
})
return unwrap<DeviceLogConfigData>(data)
}
/** 最近上报过的终端(覆盖选择用):GET /api/admin/device-logs/devices */
export async function fetchDeviceLogDevices(): Promise<DeviceLogDevice[]> {
const { data } = await http.get('/api/admin/device-logs/devices')
return unwrap<DeviceLogDevice[]>(data)
}
/** 设置全局采集模式:PUT /api/admin/device-logs/config/global */
export async function updateDeviceLogGlobalMode(mode: string): Promise<void> {
const { data } = await http.put('/api/admin/device-logs/config/global', undefined, { params: { mode } })
unwrap<unknown>(data)
}
/** 设置/更新终端覆盖:PUT /api/admin/device-logs/config/device */
export async function updateDeviceLogOverride(
source: string,
deviceId: string,
deviceName: string | null,
mode: string,
): Promise<void> {
const { data } = await http.put('/api/admin/device-logs/config/device', undefined, {
params: { source, deviceId, deviceName: deviceName || undefined, mode },
})
unwrap<unknown>(data)
}
/** 删除终端覆盖(回落到全局默认):DELETE /api/admin/device-logs/config/device/{id} */
export async function deleteDeviceLogOverride(id: number): Promise<void> {
const { data } = await http.delete(`/api/admin/device-logs/config/device/${id}`)
unwrap<unknown>(data)
}
@@ -57,19 +57,18 @@
@input="onKeywordInput" @input="onKeywordInput"
/> />
<div class="bell-search-days"> <div class="bell-search-days">
<input <el-date-picker
v-model="startDate" :model-value="dateRangeValue"
type="date" type="daterange"
class="bell-date-input" value-format="YYYY-MM-DD"
aria-label="始日期" start-placeholder="始日期"
@change="reload" end-placeholder="结束日期"
/> range-separator=""
<span class="bell-date-sep"></span> size="small"
<input class="bell-date-picker"
v-model="endDate" popper-class="bell-date-popper"
type="date" :clearable="true"
class="bell-date-input" @update:model-value="onDateRangeChange"
aria-label="结束日期"
@change="reload" @change="reload"
/> />
<button v-if="hasFilter" type="button" class="bell-search-reset" @click="resetFilters"> <button v-if="hasFilter" type="button" class="bell-search-reset" @click="resetFilters">
@@ -171,6 +170,23 @@ const groupedItems = computed(() => groupNotificationsByDay(items.value))
const totalPages = computed(() => Math.max(1, Math.ceil(total.value / PAGE_SIZE))) const totalPages = computed(() => Math.max(1, Math.ceil(total.value / PAGE_SIZE)))
const hasFilter = computed(() => Boolean(keyword.value.trim() || startDate.value || endDate.value)) const hasFilter = computed(() => Boolean(keyword.value.trim() || startDate.value || endDate.value))
/**
* el-date-picker 的绑定桥:日期区间在内部仍用 startDate/endDate 两个 ref 表示
* hasFilter/normalizedDayRange/resetFilters 都基于它们)。
* 此前模板写的是 v-model="dateRange",而 dateRange 从未声明 → vue-tsc 直接报 TS2339 构建失败,
* 且日期筛选完全不下发参数。
*/
const dateRangeValue = computed<[string, string] | null>(() =>
startDate.value && endDate.value ? [startDate.value, endDate.value] : null
)
function onDateRangeChange(value: [string, string] | null) {
const start = value?.[0]
const end = value?.[1]
startDate.value = start ? String(start) : ''
endDate.value = end ? String(end) : ''
}
let pollTimer: number | null = null let pollTimer: number | null = null
let searchTimer: number | null = null let searchTimer: number | null = null
@@ -77,7 +77,7 @@ watch(
node-key="id" node-key="id"
show-checkbox show-checkbox
default-expand-all default-expand-all
:props="{ label: 'name', children: 'children' }" :props="{ label: 'name', children: 'children', disabled: 'disabled' }"
@check="onCheck" @check="onCheck"
/> />
</div> </div>
@@ -89,7 +89,7 @@ watch(
node-key="id" node-key="id"
show-checkbox show-checkbox
default-expand-all default-expand-all
:props="{ label: 'name', children: 'children' }" :props="{ label: 'name', children: 'children', disabled: 'disabled' }"
@check="onCheck" @check="onCheck"
/> />
</div> </div>
@@ -16,6 +16,11 @@ export interface MenuOptionNode {
parentId: number | null parentId: number | null
/** 所属菜单类型(admin 后台 / app 前端客户端),提交时按类型分区落库。 */ /** 所属菜单类型(admin 后台 / app 前端客户端),提交时按类型分区落库。 */
type: string type: string
/**
* 当前操作者无权授予(后端 grantable=false)时置灰:仍展示并回显已勾选,
* 但不允许改勾选。非超管只能授自己已有的菜单,勾到越权项会让整笔保存回滚。
*/
disabled?: boolean
children?: MenuOptionNode[] children?: MenuOptionNode[]
} }
@@ -39,6 +44,8 @@ export function parsePermissionMenuItem(raw: unknown, type = ''): MenuOptionNode
sort: sortRaw === null ? 0 : sortRaw, sort: sortRaw === null ? 0 : sortRaw,
parentId: parentId === null ? null : parentId, parentId: parentId === null ? null : parentId,
type, type,
// 缺省(菜单管理页等未标记的接口)按可授予处理,保持旧行为
disabled: record.grantable === false,
} }
} }
@@ -43,6 +43,10 @@ const lockedGroupId = computed<number | null>(() => {
return groups.value.length === 1 ? groups.value[0].id : null return groups.value.length === 1 ? groups.value[0].id : null
}) })
const jumpPage = ref('') const jumpPage = ref('')
/** 顺序翻页游标:上一页响应给的 nextLastId;仅在"下一页"时使用,跳页/筛选时清空走 OFFSET。 */
const pageCursor = ref<number | null>(null)
/** 本次请求实际使用的游标(load 时决定) */
let pendingCursor: number | null = null
const totalPages = computed(() => Math.max(1, Math.ceil(total.value / pageSize.value))) const totalPages = computed(() => Math.max(1, Math.ceil(total.value / pageSize.value)))
@@ -85,10 +89,15 @@ function stopExportWait(): void {
async function load(): Promise<void> { async function load(): Promise<void> {
loading.value = true loading.value = true
try { try {
const result = await fetchDedupeTotalList(toDedupeListParams(filter, page.value, pageSize.value)) const result = await fetchDedupeTotalList(
toDedupeListParams(filter, page.value, pageSize.value, pendingCursor),
)
rows.value = result.items rows.value = result.items
total.value = result.total total.value = result.total
if (result.page >= 1) page.value = result.page if (result.page >= 1) page.value = result.page
// 本页响应回传的游标留作"下一页"用;空页则清空(没有更多)
pageCursor.value = result.nextLastId ?? null
pendingCursor = null
} catch (error) { } catch (error) {
ElMessage.error(error instanceof Error ? error.message : '去重数据加载失败') ElMessage.error(error instanceof Error ? error.message : '去重数据加载失败')
} finally { } finally {
@@ -98,11 +107,15 @@ async function load(): Promise<void> {
function apply(): void { function apply(): void {
page.value = 1 page.value = 1
pageCursor.value = null
pendingCursor = null
void load() void load()
} }
function changePage(next: number): void { function changePage(next: number): void {
if (next < 1 || next > totalPages.value) return if (next < 1 || next > totalPages.value) return
// 只有"顺序下一页"用 keyset 游标(避免深分页 offset);跳页/回退走 OFFSET,行为不变
pendingCursor = next === page.value + 1 ? pageCursor.value : null
page.value = next page.value = next
void load() void load()
} }
@@ -119,6 +132,8 @@ function goJump(): void {
function changeSize(size: number) { function changeSize(size: number) {
pageSize.value = size pageSize.value = size
page.value = 1 page.value = 1
pageCursor.value = null
pendingCursor = null
void load() void load()
} }
@@ -24,6 +24,8 @@ export interface AsinListParams {
groupId?: number | null groupId?: number | null
/** 国家代码(如 DE、UK)。 */ /** 国家代码(如 DE、UK)。 */
country?: string country?: string
/** 顺序翻页游标(上一页返回的 nextLastId):传了就忽略 page 偏移,走 keyset。 */
lastId?: number
} }
/** 序列化到 Java 分页接口的查询参数(snake_case)。 */ /** 序列化到 Java 分页接口的查询参数(snake_case)。 */
@@ -36,6 +38,8 @@ export interface AsinPageQuery {
end_date?: string end_date?: string
group_id?: number group_id?: number
country?: string country?: string
/** 顺序翻页游标(keyset):传了就忽略 page 偏移。 */
last_id?: number
} }
function finiteInt(value: unknown): number | null { function finiteInt(value: unknown): number | null {
@@ -77,5 +81,6 @@ export function toAsinPageQuery(params: AsinListParams): AsinPageQuery {
if (params.endDate) query.end_date = params.endDate if (params.endDate) query.end_date = params.endDate
if (params.groupId != null) query.group_id = params.groupId if (params.groupId != null) query.group_id = params.groupId
if (params.country) query.country = params.country if (params.country) query.country = params.country
if (typeof params.lastId === 'number' && params.lastId > 0) query.last_id = params.lastId
return query return query
} }
@@ -30,8 +30,10 @@ export function toDedupeListParams(
state: DedupeTotalFilterState, state: DedupeTotalFilterState,
page: number, page: number,
pageSize: number, pageSize: number,
lastId?: number | null,
): AsinListParams { ): AsinListParams {
const params: AsinListParams = { page, pageSize } const params: AsinListParams = { page, pageSize }
if (typeof lastId === 'number' && lastId > 0) params.lastId = lastId
const keyword = (state.keyword || '').trim() const keyword = (state.keyword || '').trim()
const username = (state.username || '').trim() const username = (state.username || '').trim()
const country = (state.country || '').trim() const country = (state.country || '').trim()
@@ -18,6 +18,8 @@ export interface DedupeTotalPageResult {
total: number total: number
page: number page: number
pageSize: number pageSize: number
/** 顺序翻页游标:本页最后一行 id;下一页回传它即可走 keyset。 */
nextLastId?: number
} }
export function emptyDedupeTotalPage(): DedupeTotalPageResult { export function emptyDedupeTotalPage(): DedupeTotalPageResult {
@@ -65,6 +67,8 @@ export function parseDedupeTotalPage(payload: unknown): DedupeTotalPageResult {
} }
if (typeof record.total === 'number') out.total = Math.floor(record.total) if (typeof record.total === 'number') out.total = Math.floor(record.total)
if (typeof record.page === 'number' && record.page >= 1) out.page = Math.floor(record.page) if (typeof record.page === 'number' && record.page >= 1) out.page = Math.floor(record.page)
const rawNextLastId = record.nextLastId ?? record.next_last_id
if (typeof rawNextLastId === 'number' && rawNextLastId > 0) out.nextLastId = Math.floor(rawNextLastId)
const rawSize = record.pageSize ?? record.page_size const rawSize = record.pageSize ?? record.page_size
if (typeof rawSize === 'number' && rawSize >= 1) out.pageSize = Math.floor(rawSize) if (typeof rawSize === 'number' && rawSize >= 1) out.pageSize = Math.floor(rawSize)
return out return out
@@ -0,0 +1,719 @@
<script setup lang="ts">
/** 日志管理页:桌面客户端 / 麦象采集机上报的日志浏览(仅超管)。
* 支持来源/日期/关键字筛选,尾部内容查看(自动滚到底、可向前加载更早内容)、
* 完整下载、删除;「采集配置」可调全量/精选模式(全局默认 + 终端覆盖)。 */
import { computed, nextTick, onMounted, ref } from 'vue'
import OldPagination from '@/components/OldPagination.vue'
import { ElMessage } from 'element-plus'
import { formatDateTime } from '@/utils/datetime'
import {
deleteDeviceLogFile,
deleteDeviceLogOverride,
deviceLogDownloadUrl,
fetchDeviceLogConfig,
fetchDeviceLogContent,
fetchDeviceLogDevices,
fetchDeviceLogFiles,
updateDeviceLogGlobalMode,
updateDeviceLogOverride,
type DeviceLogContent,
type DeviceLogDevice,
type DeviceLogFileRow,
type DeviceLogOverride,
} from '@/api/device-logs'
const SOURCE_OPTIONS = [
{ value: '', label: '全部来源' },
{ value: 'client', label: '桌面客户端' },
{ value: 'maixiang', label: '麦象采集机' },
]
function sourceLabel(source: string): string {
if (source === 'client') return '桌面客户端'
if (source === 'maixiang') return '麦象采集机'
return source || '—'
}
function formatBytes(bytes: number | null | undefined): string {
if (bytes == null || !Number.isFinite(bytes) || bytes <= 0) return '—'
if (bytes < 1024) return `${bytes} B`
const units = ['KB', 'MB', 'GB']
let value = bytes / 1024
let unitIndex = 0
while (value >= 1024 && unitIndex < units.length - 1) {
value /= 1024
unitIndex += 1
}
return `${value >= 100 ? value.toFixed(0) : value.toFixed(1)} ${units[unitIndex]}`
}
const loading = ref(false)
const rows = ref<DeviceLogFileRow[]>([])
const total = ref(0)
const page = ref(1)
const pageSize = ref(15)
const sourceFilter = ref('')
const keyword = ref('')
const startDate = ref('')
const endDate = ref('')
const retentionDays = ref(7)
const totalPages = computed(() => Math.max(1, Math.ceil(total.value / pageSize.value)))
async function load() {
loading.value = true
try {
const result = await fetchDeviceLogFiles({
source: sourceFilter.value || undefined,
keyword: keyword.value.trim() || undefined,
startDate: startDate.value || undefined,
endDate: endDate.value || undefined,
page: page.value,
pageSize: pageSize.value,
})
rows.value = result?.items || []
total.value = Number(result?.total || 0)
if (result?.retentionDays) retentionDays.value = result.retentionDays
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '日志列表加载失败')
} finally {
loading.value = false
}
}
function search() {
page.value = 1
load()
}
function reset() {
sourceFilter.value = ''
keyword.value = ''
startDate.value = ''
endDate.value = ''
page.value = 1
load()
}
function changePage(next: number) {
if (next < 1 || next > totalPages.value) return
page.value = next
load()
}
function changeSize(size: number) {
pageSize.value = size
page.value = 1
load()
}
// ---------------------------------------------------------------- 内容查看
const TAIL_STEP = 256 * 1024
const TAIL_MAX = 8 * 1024 * 1024
const viewerVisible = ref(false)
const viewerLoading = ref(false)
const viewerFile = ref<DeviceLogFileRow | null>(null)
const viewerContent = ref<DeviceLogContent | null>(null)
const viewerMaxBytes = ref(TAIL_STEP)
const viewerPre = ref<HTMLElement | null>(null)
async function openViewer(row: DeviceLogFileRow) {
viewerFile.value = row
viewerMaxBytes.value = TAIL_STEP
viewerContent.value = null
viewerVisible.value = true
await loadContent(true)
}
async function loadContent(scrollToBottom: boolean) {
if (!viewerFile.value) return
viewerLoading.value = true
try {
viewerContent.value = await fetchDeviceLogContent(viewerFile.value.id, viewerMaxBytes.value)
if (scrollToBottom) {
await nextTick()
if (viewerPre.value) viewerPre.value.scrollTop = viewerPre.value.scrollHeight
}
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '日志内容加载失败')
} finally {
viewerLoading.value = false
}
}
function loadMore() {
viewerMaxBytes.value = Math.min(viewerMaxBytes.value * 2, TAIL_MAX)
loadContent(false)
}
async function remove(row: DeviceLogFileRow) {
if (!window.confirm(`确认删除「${row.fileName}」(${row.deviceName || row.deviceId})的云端日志?删除后不可恢复。`)) return
try {
await deleteDeviceLogFile(row.id)
ElMessage.success('已删除')
load()
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '删除失败')
}
}
// ---------------------------------------------------------------- 采集配置
const configVisible = ref(false)
const configLoading = ref(false)
const globalMode = ref('full')
const overrides = ref<DeviceLogOverride[]>([])
const devices = ref<DeviceLogDevice[]>([])
const newOverrideKey = ref('')
const newOverrideMode = ref('selected')
const MODE_OPTIONS = [
{ value: 'full', label: '全量采集' },
{ value: 'selected', label: '精选采集' },
]
function modeLabel(mode: string): string {
return mode === 'selected' ? '精选' : '全量'
}
async function openConfig() {
configVisible.value = true
await loadConfig()
}
async function loadConfig() {
configLoading.value = true
try {
const data = await fetchDeviceLogConfig()
globalMode.value = data.globalMode || 'full'
overrides.value = data.overrides || []
devices.value = (await fetchDeviceLogDevices()) || []
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '采集配置加载失败')
} finally {
configLoading.value = false
}
}
async function saveGlobalMode(mode: string) {
if (mode === globalMode.value) return
try {
await updateDeviceLogGlobalMode(mode)
globalMode.value = mode
ElMessage.success(`全局采集模式已切换为「${modeLabel(mode)}`)
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '保存失败')
}
}
async function addOverride() {
if (!newOverrideKey.value) {
ElMessage.warning('请先选择终端')
return
}
const [source, deviceId] = newOverrideKey.value.split('|')
const device = devices.value.find((item) => item.source === source && item.deviceId === deviceId)
try {
await updateDeviceLogOverride(source, deviceId, device?.deviceName || null, newOverrideMode.value)
ElMessage.success('终端覆盖已保存')
newOverrideKey.value = ''
await loadConfig()
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '保存失败')
}
}
async function removeOverride(row: DeviceLogOverride) {
const who = row.deviceName || row.deviceId
if (!window.confirm(`确认删除终端「${who}」的采集覆盖(回落到全局默认)?`)) return
try {
await deleteDeviceLogOverride(row.id)
ElMessage.success('已删除')
await loadConfig()
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '删除失败')
}
}
onMounted(load)
</script>
<template>
<div class="device-logs-view">
<section class="panel-box">
<div class="logs-head">
<h3>日志文件列表</h3>
<div class="logs-head-tools">
<span class="retention-tip">云端仅保留 {{ retentionDays }} </span>
<button class="btn btn-ghost" type="button" @click="openConfig">采集配置</button>
<button class="btn" type="button" @click="load">刷新</button>
</div>
</div>
<div class="form-row logs-filter-row">
<div class="form-group" style="min-width: 150px">
<label>来源</label>
<select v-model="sourceFilter">
<option v-for="option in SOURCE_OPTIONS" :key="option.value" :value="option.value">{{ option.label }}</option>
</select>
</div>
<div class="form-group" style="min-width: 220px">
<label>设备 / 文件名</label>
<input v-model="keyword" type="text" placeholder="模糊搜索设备名、设备ID、文件名" @keyup.enter="search" />
</div>
<div class="form-group" style="min-width: 150px">
<label>日志日期</label>
<input v-model="startDate" type="date" />
</div>
<div class="form-group" style="min-width: 150px">
<label>日志日期</label>
<input v-model="endDate" type="date" />
</div>
<div class="form-group">
<label>&nbsp;</label>
<div class="filter-actions">
<button class="btn" type="button" @click="search">查询</button>
<button class="btn btn-ghost" type="button" @click="reset">重置</button>
</div>
</div>
</div>
<div class="logs-table-scroll">
<table>
<thead>
<tr>
<th style="width: 110px">来源</th>
<th style="width: 230px">设备</th>
<th style="width: 130px">用户</th>
<th style="width: 230px">文件名</th>
<th style="width: 110px">日志日期</th>
<th style="width: 100px">已收大小</th>
<th style="width: 80px">片段数</th>
<th style="width: 170px">最后更新</th>
<th style="width: 190px">操作</th>
</tr>
</thead>
<tbody>
<template v-if="rows.length">
<tr v-for="row in rows" :key="row.id">
<td>
<span class="source-pill" :class="row.source === 'maixiang' ? 'is-maixiang' : 'is-client'">
{{ sourceLabel(row.source) }}
</span>
</td>
<td>
<span class="device-name" :title="row.deviceId">{{ row.deviceName || '—' }}</span>
<span class="device-id" :title="row.deviceId">{{ row.deviceId }}</span>
</td>
<td>
<span v-if="row.username" class="user-name">{{ row.username }}</span>
<span v-else-if="row.uid" class="user-name">UID {{ row.uid }}</span>
<span v-else class="dim"></span>
</td>
<td>
<span class="file-name" :title="row.fileName">{{ row.fileName }}</span>
</td>
<td>{{ row.logDate }}</td>
<td>{{ formatBytes(row.uploadedBytes) }}</td>
<td>{{ row.partCount ?? 0 }}</td>
<td>{{ row.lastUploadAt ? formatDateTime(row.lastUploadAt) : '—' }}</td>
<td class="ops-cell">
<button class="btn btn-sm" type="button" @click="openViewer(row)">查看</button>
<a class="btn btn-sm btn-ghost dl-link" :href="deviceLogDownloadUrl(row.id)" download>下载</a>
<button class="btn btn-sm btn-danger" type="button" @click="remove(row)">删除</button>
</td>
</tr>
</template>
<tr v-else-if="loading">
<td colspan="9" class="empty-tip">加载中...</td>
</tr>
<tr v-else>
<td colspan="9" class="empty-tip">
{{ keyword || sourceFilter || startDate || endDate ? '暂无匹配日志' : '暂无日志上报(客户端/采集机上报后自动出现在这里)' }}
</td>
</tr>
</tbody>
</table>
</div>
<OldPagination :total="total" :page="page" :page-size="pageSize" @change="changePage" @size-change="changeSize" />
</section>
<el-dialog v-model="viewerVisible" :title="viewerFile ? `${viewerFile.fileName}${viewerFile.deviceName || viewerFile.deviceId}` : '日志内容'" width="900px" top="5vh">
<div class="viewer-toolbar">
<span class="viewer-meta">
云端已收 {{ formatBytes(viewerContent?.totalBytes ?? viewerFile?.uploadedBytes) }}
<template v-if="viewerContent"> · 当前展示 {{ formatBytes(viewerContent.shownBytes) }}</template>
<template v-if="viewerContent?.truncated"> · 更早内容未加载</template>
</span>
<span class="viewer-actions">
<button class="btn btn-sm btn-ghost" type="button" :disabled="viewerLoading" @click="loadContent(true)">刷新</button>
<button
class="btn btn-sm btn-ghost"
type="button"
:disabled="viewerLoading || !viewerContent?.truncated || viewerMaxBytes >= TAIL_MAX"
@click="loadMore"
>加载更早</button>
<a v-if="viewerFile" class="btn btn-sm btn-ghost dl-link" :href="deviceLogDownloadUrl(viewerFile.id)" download>下载完整日志</a>
</span>
</div>
<pre ref="viewerPre" class="log-pre">{{ viewerLoading && !viewerContent ? '加载中...' : (viewerContent?.content || '(暂无内容)') }}</pre>
</el-dialog>
<el-dialog v-model="configVisible" title="采集配置" width="720px">
<div class="config-block">
<h4>全局默认模式</h4>
<p class="config-desc">对未单独配置的终端生效全量=上传日志目录内全部文件精选=排除低价值大日志客户端排除 pywebview麦象排除 kk-browser / 控制台 / 测试日志终端在下一次上报周期 1 分钟内跟随新配置</p>
<div class="mode-switch">
<button
v-for="option in MODE_OPTIONS"
:key="option.value"
class="mode-btn"
:class="{ 'is-active': globalMode === option.value }"
type="button"
@click="saveGlobalMode(option.value)"
>{{ option.label }}</button>
</div>
</div>
<div class="config-block">
<h4>终端覆盖</h4>
<div class="override-add">
<select v-model="newOverrideKey" class="override-select">
<option value="">选择终端最近上报的设备</option>
<option v-for="device in devices" :key="`${device.source}|${device.deviceId}`" :value="`${device.source}|${device.deviceId}`">
{{ sourceLabel(device.source) }} · {{ device.deviceName || device.deviceId }}{{ device.deviceId }}
</option>
</select>
<select v-model="newOverrideMode">
<option v-for="option in MODE_OPTIONS" :key="option.value" :value="option.value">{{ option.label }}</option>
</select>
<button class="btn btn-sm" type="button" @click="addOverride">添加/更新覆盖</button>
</div>
<table class="override-table">
<thead>
<tr>
<th style="width: 100px">来源</th>
<th>设备</th>
<th style="width: 80px">模式</th>
<th style="width: 160px">更新时间</th>
<th style="width: 90px">操作</th>
</tr>
</thead>
<tbody>
<tr v-for="row in overrides" :key="row.id">
<td>{{ sourceLabel(row.source) }}</td>
<td>
<span class="device-name">{{ row.deviceName || row.deviceId }}</span>
<span class="device-id">{{ row.deviceId }}</span>
</td>
<td>{{ modeLabel(row.mode) }}</td>
<td>{{ row.updatedAt ? formatDateTime(row.updatedAt) : '—' }}</td>
<td class="ops-cell">
<button class="btn btn-sm btn-danger" type="button" @click="removeOverride(row)">删除</button>
</td>
</tr>
<tr v-if="!overrides.length">
<td colspan="5" class="empty-tip">{{ configLoading ? '加载中...' : '暂无终端覆盖(全部跟随全局默认)' }}</td>
</tr>
</tbody>
</table>
</div>
<template #footer>
<el-button @click="configVisible = false">关闭</el-button>
</template>
</el-dialog>
</div>
</template>
<style scoped>
/* 沿用「记录与版本」系列的旧后台面板视觉语言。 */
.device-logs-view {
font-family: inherit;
color: #24384d;
}
.panel-box {
width: 100%;
min-width: 0;
padding: 20px 22px 24px;
border: 1px solid #d8e3ee;
border-radius: 14px;
background: linear-gradient(145deg, #ffffff, #f9fbfd);
box-shadow: 0 1px 2px rgba(39, 67, 94, 0.04), 0 14px 30px -24px rgba(39, 67, 94, 0.28);
}
h3 {
margin: 0;
font-size: 15px;
font-weight: 650;
color: #24384d;
letter-spacing: 0.2px;
}
.logs-head {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
margin-bottom: 16px;
}
.logs-head-tools {
display: flex;
align-items: center;
gap: 10px;
}
.retention-tip {
color: #8598ab;
font-size: 12.5px;
}
.logs-filter-row {
margin: 0 0 16px;
}
.form-row {
display: flex;
flex-wrap: wrap;
align-items: flex-end;
gap: 14px 18px;
}
.form-group {
display: flex;
flex-direction: column;
gap: 7px;
}
.form-group label {
color: #5b6f83;
font-size: 12.5px;
font-weight: 600;
}
.form-group input,
.form-group select {
min-width: 0;
min-height: 42px;
padding: 8px 12px;
border: 1px solid #cbd9e6;
border-radius: 9px;
background: #f8fbfd;
color: #24384d;
font-size: 13.5px;
font-family: inherit;
color-scheme: light;
outline: none;
}
.form-group input:focus,
.form-group select:focus {
background: #ffffff;
border-color: #5f85ad;
box-shadow: 0 0 0 3px rgba(95, 133, 173, 0.16);
}
.filter-actions {
display: flex;
gap: 10px;
}
.btn {
display: inline-flex;
align-items: center;
justify-content: center;
gap: 8px;
min-height: 42px;
padding: 9px 18px;
border: 1px solid #4f78a5;
border-radius: 9px;
background: linear-gradient(135deg, #5f85ad, #4f78a5);
color: #ffffff;
font-family: inherit;
font-size: 13.5px;
cursor: pointer;
box-shadow: 0 8px 18px -14px rgba(79, 120, 165, 0.72);
text-decoration: none;
}
.btn:hover:not(:disabled) {
background: linear-gradient(135deg, #7094ba, #5d83ac);
}
.btn:disabled {
cursor: not-allowed;
opacity: 0.55;
}
.btn-ghost {
background: #ffffff;
border-color: #c7d7e5;
color: #4f78a5;
box-shadow: none;
}
.btn-ghost:hover:not(:disabled) {
background: #edf5fb;
border-color: #95b1cb;
color: #2f5d8b;
}
.btn-danger {
background: linear-gradient(135deg, #c06d77, #b35f6a);
border-color: #b35f6a;
}
.btn-danger:hover:not(:disabled) {
background: linear-gradient(135deg, #cb7c84, #b96570);
}
.btn-sm {
min-height: 32px;
padding: 4px 12px;
font-size: 12.5px;
border-radius: 7px;
}
.logs-table-scroll {
width: 100%;
overflow-x: auto;
}
table {
width: 100%;
border-collapse: collapse;
font-size: 13px;
}
th {
padding: 10px 12px;
text-align: left;
color: #5b6f83;
font-weight: 650;
font-size: 12.5px;
border-bottom: 1px solid #dbe6f0;
white-space: nowrap;
}
td {
padding: 10px 12px;
border-bottom: 1px solid #eaf1f7;
vertical-align: top;
}
.source-pill {
display: inline-block;
padding: 2px 10px;
border-radius: 999px;
font-size: 12px;
font-weight: 600;
}
.source-pill.is-client {
background: #e8f1fb;
color: #37618f;
}
.source-pill.is-maixiang {
background: #eef7ec;
color: #3f7a42;
}
.device-name {
display: block;
font-weight: 600;
word-break: break-all;
}
.device-id {
display: block;
color: #8598ab;
font-size: 11.5px;
word-break: break-all;
}
.user-name {
font-weight: 600;
}
.file-name {
display: block;
word-break: break-all;
}
.dim {
color: #9db0c2;
}
.ops-cell {
display: flex;
gap: 8px;
flex-wrap: wrap;
border-bottom: none;
}
.empty-tip {
padding: 26px 0;
text-align: center;
color: #8598ab;
}
.dl-link {
text-decoration: none;
}
.viewer-toolbar {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
margin-bottom: 10px;
}
.viewer-meta {
color: #5b6f83;
font-size: 12.5px;
}
.viewer-actions {
display: flex;
gap: 8px;
}
.log-pre {
max-height: 62vh;
margin: 0;
padding: 12px 14px;
overflow: auto;
border: 1px solid #d8e3ee;
border-radius: 10px;
background: #0f1c29;
color: #d7e4f1;
font-family: Consolas, 'Courier New', monospace;
font-size: 12.5px;
line-height: 1.55;
white-space: pre;
}
.config-block {
margin-bottom: 18px;
}
.config-block h4 {
margin: 0 0 6px;
font-size: 13.5px;
color: #24384d;
}
.config-desc {
margin: 0 0 10px;
color: #66798d;
font-size: 12.5px;
line-height: 1.6;
}
.mode-switch {
display: flex;
gap: 10px;
}
.mode-btn {
min-height: 38px;
padding: 8px 20px;
border: 1px solid #c7d7e5;
border-radius: 9px;
background: #ffffff;
color: #4f78a5;
font-size: 13.5px;
font-family: inherit;
cursor: pointer;
}
.mode-btn.is-active {
border-color: #4f78a5;
background: linear-gradient(135deg, #5f85ad, #4f78a5);
color: #ffffff;
font-weight: 600;
}
.override-add {
display: flex;
gap: 10px;
margin-bottom: 12px;
}
.override-add select {
min-height: 38px;
padding: 6px 10px;
border: 1px solid #cbd9e6;
border-radius: 9px;
background: #f8fbfd;
color: #24384d;
font-size: 13px;
font-family: inherit;
color-scheme: light;
}
.override-select {
flex: 1;
min-width: 0;
}
.override-table td {
vertical-align: middle;
}
</style>
@@ -2,7 +2,8 @@
import { formatDateTime } from '@/utils/datetime' import { formatDateTime } from '@/utils/datetime'
/** 教程管理页:工具台首页「立即下载教程」的包体管理(上传新包 / 列表 / 下载 / 删除)。 /** 教程管理页:工具台首页「立即下载教程」的包体管理(上传新包 / 列表 / 下载 / 删除)。
* 上传走浏览器直传 MinIOpresign → PUT 带进度 → confirm 落库),与软件版本管理页同一套链路; * 上传走浏览器直传 MinIOpresign → PUT 带进度 → confirm 落库),与软件版本管理页同一套链路;
* 工具台始终下载"最新上传"的包(列表第一条即当前生效)。 */ * 上传时可填版本号(仅展示与排序用);工具台始终下载"最新上传"的包(不随列表排序变化)。
* 列表默认按上传时间降序,「版本号」「上传时间」表头可点击切换升降序。 */
import { computed, onMounted, ref, watch } from 'vue' import { computed, onMounted, ref, watch } from 'vue'
import OldPagination from '@/components/OldPagination.vue' import OldPagination from '@/components/OldPagination.vue'
import { ElMessage } from 'element-plus' import { ElMessage } from 'element-plus'
@@ -22,12 +23,64 @@ const filteredItems = computed(() => {
/** 全站分页统一:客户端分页(10/20/50/100)。 */ /** 全站分页统一:客户端分页(10/20/50/100)。 */
const page = ref(1) const page = ref(1)
const pageSize = ref(10) const pageSize = ref(10)
const pagedItems = computed(() => filteredItems.value.slice((page.value - 1) * pageSize.value, page.value * pageSize.value))
/** 排序:默认按上传时间降序(最新上传在最前,与后端返回顺序一致);点表头切换升降序。 */
type SortKey = 'version' | 'createdAt'
const sortKey = ref<SortKey>('createdAt')
const sortAsc = ref(false)
function toggleSort(key: SortKey) {
if (sortKey.value === key) {
sortAsc.value = !sortAsc.value
} else {
sortKey.value = key
sortAsc.value = false
}
page.value = 1
}
/** 排序标记:未激活 ▲▼,激活时只留方向箭头(比 ⇅/↕ 字形支持好,避免 Windows 缺字形显示方框)。 */
function sortMark(key: SortKey): string {
if (sortKey.value !== key) return '▲▼'
return sortAsc.value ? '▲' : '▼'
}
function compareText(a: string, b: string): number {
return a.localeCompare(b, 'zh-Hans-CN', { numeric: true, sensitivity: 'base' })
}
const sortedItems = computed(() => {
const rows = [...filteredItems.value]
rows.sort((a, b) => {
// 版本号为空的历史行始终排在末尾,避免切换排序时"无版本"占满首页。
if (sortKey.value === 'version') {
const av = a.version
const bv = b.version
if (!av || !bv) {
if (!av && !bv) return 0
return av ? -1 : 1
}
const diff = compareText(av, bv)
return sortAsc.value ? diff : -diff
}
const at = a.createdAt
const bt = b.createdAt
if (!at || !bt) {
if (!at && !bt) return 0
return at ? -1 : 1
}
const diff = compareText(at, bt)
return sortAsc.value ? diff : -diff
})
return rows
})
const pagedItems = computed(() => sortedItems.value.slice((page.value - 1) * pageSize.value, page.value * pageSize.value))
function changePage(p: number) { page.value = p } function changePage(p: number) { page.value = p }
function changeSize(size: number) { pageSize.value = size; page.value = 1 } function changeSize(size: number) { pageSize.value = size; page.value = 1 }
// 搜索导致数据收缩时回钳页码,避免停在空页。 // 搜索/排序导致数据收缩时回钳页码,避免停在空页。
watch(filteredItems, () => { watch(sortedItems, () => {
page.value = Math.min(page.value, Math.max(1, Math.ceil(filteredItems.value.length / pageSize.value))) page.value = Math.min(page.value, Math.max(1, Math.ceil(sortedItems.value.length / pageSize.value)))
}) })
/** 当前生效包 = 列表第一条(工具台下发的就是它)。 */ /** 当前生效包 = 列表第一条(工具台下发的就是它)。 */
@@ -80,6 +133,7 @@ async function removeOne(row: TutorialPackageItem) {
const uploadVisible = ref(false) const uploadVisible = ref(false)
const uploading = ref(false) const uploading = ref(false)
const uploadPercent = ref(0) const uploadPercent = ref(0)
const newVersion = ref('')
const pickedFile = ref<File | null>(null) const pickedFile = ref<File | null>(null)
/** 弹窗内成功/失败文案留驻(对齐软件版本页交互)。 */ /** 弹窗内成功/失败文案留驻(对齐软件版本页交互)。 */
const uploadMsg = ref('') const uploadMsg = ref('')
@@ -103,6 +157,7 @@ function onFileChange(file: File) {
function openUpload() { function openUpload() {
uploadMsg.value = '' uploadMsg.value = ''
uploadMsgOk.value = false uploadMsgOk.value = false
newVersion.value = ''
pickedFile.value = null pickedFile.value = null
uploadVisible.value = true uploadVisible.value = true
} }
@@ -126,12 +181,13 @@ async function submitUpload() {
uploading.value = true uploading.value = true
try { try {
// 浏览器直传 MinIOpresign → PUT(进度条)→ confirm 落库。 // 浏览器直传 MinIOpresign → PUT(进度条)→ confirm 落库。
await uploadTutorialPackage(pickedFile.value, pickedFile.value.name, (p) => { await uploadTutorialPackage(pickedFile.value, pickedFile.value.name, newVersion.value.trim(), (p) => {
uploadPercent.value = p uploadPercent.value = p
uploadMsg.value = p >= 100 ? '上传完成,正在登记教程包...' : `正在上传:${p}%` uploadMsg.value = p >= 100 ? '上传完成,正在登记教程包...' : `正在上传:${p}%`
}) })
uploadMsg.value = '上传成功,工具台「立即下载教程」已切换为该包' uploadMsg.value = '上传成功,工具台「立即下载教程」已切换为该包'
uploadMsgOk.value = true uploadMsgOk.value = true
newVersion.value = ''
pickedFile.value = null pickedFile.value = null
uploadPercent.value = 0 uploadPercent.value = 0
load() load()
@@ -165,8 +221,13 @@ onMounted(load)
<input type="checkbox" :checked="pagedItems.length > 0 && pagedItems.every((r) => selectedIds.has(r.id))" @change="toggleSelectAllPage" :disabled="!pagedItems.length" /> <input type="checkbox" :checked="pagedItems.length > 0 && pagedItems.every((r) => selectedIds.has(r.id))" @change="toggleSelectAllPage" :disabled="!pagedItems.length" />
</th> </th>
<th style="width: 260px">文件名</th> <th style="width: 260px">文件名</th>
<th style="width: 110px">
<button class="sort-th sort-version" type="button" @click="toggleSort('version')">版本号<span class="sort-mark">{{ sortMark('version') }}</span></button>
</th>
<th style="width: 110px">大小</th> <th style="width: 110px">大小</th>
<th style="width: 150px">上传时间</th> <th style="width: 150px">
<button class="sort-th sort-time" type="button" @click="toggleSort('createdAt')">上传时间<span class="sort-mark">{{ sortMark('createdAt') }}</span></button>
</th>
<th>下载链接</th> <th>下载链接</th>
<th style="width: 170px">操作</th> <th style="width: 170px">操作</th>
</tr> </tr>
@@ -181,6 +242,10 @@ onMounted(load)
<span class="file-name" :title="row.fileName">{{ row.fileName }}</span> <span class="file-name" :title="row.fileName">{{ row.fileName }}</span>
<span v-if="row.id === activeId" class="tag-active">当前生效</span> <span v-if="row.id === activeId" class="tag-active">当前生效</span>
</td> </td>
<td>
<span v-if="row.version" class="version-cell" :title="row.version">{{ row.version }}</span>
<span v-else class="dim"></span>
</td>
<td>{{ formatFileSize(row.fileSize) }}</td> <td>{{ formatFileSize(row.fileSize) }}</td>
<td>{{ formatDateTime(row.createdAt) }}</td> <td>{{ formatDateTime(row.createdAt) }}</td>
<td> <td>
@@ -195,10 +260,10 @@ onMounted(load)
</tr> </tr>
</template> </template>
<tr v-else-if="loading"> <tr v-else-if="loading">
<td colspan="6" class="empty-tip">加载中...</td> <td colspan="7" class="empty-tip">加载中...</td>
</tr> </tr>
<tr v-else> <tr v-else>
<td colspan="6" class="empty-tip">{{ keyword ? '暂无匹配教程包' : '暂无教程包记录' }}</td> <td colspan="7" class="empty-tip">{{ keyword ? '暂无匹配教程包' : '暂无教程包记录' }}</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
@@ -209,6 +274,10 @@ onMounted(load)
<el-dialog v-model="uploadVisible" title="上传教程包" width="520px"> <el-dialog v-model="uploadVisible" title="上传教程包" width="520px">
<p class="upload-desc">上传教程 ZIP 包后工具台首页立即下载教程将以下载该包为准以最新上传的为主</p> <p class="upload-desc">上传教程 ZIP 包后工具台首页立即下载教程将以下载该包为准以最新上传的为主</p>
<el-form label-width="110px"> <el-form label-width="110px">
<el-form-item label="版本号">
<el-input v-model="newVersion" placeholder="例如:v2026.09 或留空" maxlength="64" />
<p class="zip-hint">仅作展示与排序用可留空不影响工具台按最新上传下载</p>
</el-form-item>
<el-form-item label="ZIP 压缩包" required> <el-form-item label="ZIP 压缩包" required>
<el-upload :auto-upload="false" :limit="1" accept=".zip" :on-change="(file: any) => onFileChange(file.raw)" :on-remove="() => (pickedFile = null)"> <el-upload :auto-upload="false" :limit="1" accept=".zip" :on-change="(file: any) => onFileChange(file.raw)" :on-remove="() => (pickedFile = null)">
<el-button>选择文件</el-button> <el-button>选择文件</el-button>
@@ -368,6 +437,46 @@ h3 {
text-overflow: ellipsis; text-overflow: ellipsis;
vertical-align: middle; vertical-align: middle;
} }
/* 表头排序:按钮铺满单元格,仅 hover 时加深字色,保持旧版表头观感。 */
.sort-th {
display: inline-flex;
align-items: center;
gap: 4px;
width: 100%;
padding: 0;
border: 0;
background: none;
color: inherit;
font-family: inherit;
font-size: inherit;
font-weight: inherit;
letter-spacing: inherit;
text-align: left;
cursor: pointer;
}
.sort-th:hover {
color: #2f5d8b;
}
.sort-mark {
color: #8293a5;
font-size: 12px;
}
.sort-th:hover .sort-mark {
color: #5f85ad;
}
.sort-version {
width: 110px;
}
.sort-time {
width: 150px;
}
.version-cell {
display: inline-block;
max-width: 100%;
overflow: hidden;
text-overflow: ellipsis;
vertical-align: middle;
}
.tag-active { .tag-active {
display: inline-block; display: inline-block;
margin-left: 6px; margin-left: 6px;
@@ -22,11 +22,12 @@ export interface TutorialUploadTarget {
objectKey: string objectKey: string
uploadUrl: string uploadUrl: string
fileUrl: string fileUrl: string
version: string
} }
/** 第一步:向后端申请直传 PUT 预签名地址(登录会话签发,3 分钟有效)。 */ /** 第一步:向后端申请直传 PUT 预签名地址(登录会话签发,3 分钟有效)。 */
export async function requestTutorialPresign(fileName: string): Promise<TutorialUploadTarget> { export async function requestTutorialPresign(fileName: string, version = ''): Promise<TutorialUploadTarget> {
const { data } = await http.post<unknown>(TUTORIAL_PRESIGN_ENDPOINT, null, { params: { file_name: fileName } }) const { data } = await http.post<unknown>(TUTORIAL_PRESIGN_ENDPOINT, null, { params: { file_name: fileName, version } })
const core = (unwrap(data) ?? {}) as Record<string, unknown> const core = (unwrap(data) ?? {}) as Record<string, unknown>
const uploadUrl = typeof core.upload_url === 'string' ? core.upload_url : '' const uploadUrl = typeof core.upload_url === 'string' ? core.upload_url : ''
const objectKey = typeof core.object_key === 'string' ? core.object_key : '' const objectKey = typeof core.object_key === 'string' ? core.object_key : ''
@@ -37,13 +38,14 @@ export async function requestTutorialPresign(fileName: string): Promise<Tutorial
objectKey, objectKey,
uploadUrl, uploadUrl,
fileUrl: typeof core.file_url === 'string' ? core.file_url : '', fileUrl: typeof core.file_url === 'string' ? core.file_url : '',
version: typeof core.version === 'string' ? core.version : version.trim(),
} }
} }
/** 第三步:直传完成后通知后端校验对象并写入记录(data.item 为新记录行)。 */ /** 第三步:直传完成后通知后端校验对象并写入记录(data.item 为新记录行)。 */
export async function confirmTutorialPackage(objectKey: string, fileName: string): Promise<TutorialPackageItem | null> { export async function confirmTutorialPackage(objectKey: string, fileName: string, version = ''): Promise<TutorialPackageItem | null> {
const { data } = await http.post<unknown>(TUTORIAL_CONFIRM_ENDPOINT, null, { const { data } = await http.post<unknown>(TUTORIAL_CONFIRM_ENDPOINT, null, {
params: { object_key: objectKey, file_name: fileName }, params: { object_key: objectKey, file_name: fileName, version },
}) })
return parseTutorialPackageUpload(data) return parseTutorialPackageUpload(data)
} }
@@ -63,9 +65,10 @@ export async function deleteTutorialPackages(ids: number[]): Promise<number> {
export async function uploadTutorialPackage( export async function uploadTutorialPackage(
file: Blob, file: Blob,
fileName: string, fileName: string,
version = '',
onProgress?: (percent: number) => void, onProgress?: (percent: number) => void,
): Promise<TutorialPackageItem | null> { ): Promise<TutorialPackageItem | null> {
const target = await requestTutorialPresign(fileName) const target = await requestTutorialPresign(fileName, version)
await directPut.put(target.uploadUrl, file, { await directPut.put(target.uploadUrl, file, {
headers: { 'Content-Type': 'application/octet-stream' }, headers: { 'Content-Type': 'application/octet-stream' },
onUploadProgress: (event) => { onUploadProgress: (event) => {
@@ -74,5 +77,5 @@ export async function uploadTutorialPackage(
onProgress(total > 0 ? Math.min(Math.round((event.loaded / total) * 100), 100) : 0) onProgress(total > 0 ? Math.min(Math.round((event.loaded / total) * 100), 100) : 0)
}, },
}) })
return confirmTutorialPackage(target.objectKey, fileName) return confirmTutorialPackage(target.objectKey, fileName, target.version)
} }
@@ -4,6 +4,8 @@
export interface TutorialPackageItem { export interface TutorialPackageItem {
id: number id: number
fileName: string fileName: string
/** 版本号(上传时填写,V126 之前的历史行为空串) */
version: string
objectKey: string objectKey: string
fileSize: number fileSize: number
fileUrl: string fileUrl: string
@@ -19,6 +19,7 @@ export function toTutorialPackageItem(raw: unknown): TutorialPackageItem | null
return { return {
id, id,
fileName: text(r.file_name ?? r.fileName), fileName: text(r.file_name ?? r.fileName),
version: text(r.version),
objectKey: text(r.object_key ?? r.objectKey), objectKey: text(r.object_key ?? r.objectKey),
fileSize: numberOrNull(r.file_size ?? r.fileSize) ?? 0, fileSize: numberOrNull(r.file_size ?? r.fileSize) ?? 0,
fileUrl: text(r.file_url ?? r.fileUrl), fileUrl: text(r.file_url ?? r.fileUrl),
+1
View File
@@ -25,6 +25,7 @@ export const adminPages: AdminPageDef[] = [
{ path: 'records/history', menuKey: 'admin_history', title: '生成记录', load: () => import('@/pages/records/RecordsHistoryPage.vue') }, { path: 'records/history', menuKey: 'admin_history', title: '生成记录', load: () => import('@/pages/records/RecordsHistoryPage.vue') },
{ path: 'records/software-version', menuKey: 'admin_version', title: '软件版本管理', load: () => import('@/pages/records/RecordsSoftwareVersionPage.vue') }, { path: 'records/software-version', menuKey: 'admin_version', title: '软件版本管理', load: () => import('@/pages/records/RecordsSoftwareVersionPage.vue') },
{ path: 'records/tutorial', menuKey: 'admin_tutorial', title: '教程管理', load: () => import('@/pages/records/RecordsTutorialPage.vue') }, { path: 'records/tutorial', menuKey: 'admin_tutorial', title: '教程管理', load: () => import('@/pages/records/RecordsTutorialPage.vue') },
{ path: 'records/device-logs', menuKey: 'admin_device_logs', title: '日志管理', load: () => import('@/pages/records/DeviceLogsPage.vue') },
{ path: 'records/digital-human-version', menuKey: 'digital_human_version', title: '数字人版本管理', load: () => import('@/pages/records/RecordsDigitalHumanVersionPage.vue') }, { path: 'records/digital-human-version', menuKey: 'digital_human_version', title: '数字人版本管理', load: () => import('@/pages/records/RecordsDigitalHumanVersionPage.vue') },
{ path: 'records/image-video-tasks', menuKey: 'admin_image_video_tasks', title: '视频任务记录', load: () => import('@/pages/tasks/ImageVideoTasksPage.vue') }, { path: 'records/image-video-tasks', menuKey: 'admin_image_video_tasks', title: '视频任务记录', load: () => import('@/pages/tasks/ImageVideoTasksPage.vue') },
{ path: 'shop-center/data-tasks', menuKey: 'admin_shop_data_crawl_tasks', title: '店铺数据记录', load: () => import('@/pages/tasks/ShopDataTasksPage.vue') }, { path: 'shop-center/data-tasks', menuKey: 'admin_shop_data_crawl_tasks', title: '店铺数据记录', load: () => import('@/pages/tasks/ShopDataTasksPage.vue') },
@@ -14,16 +14,23 @@ test('align_tutorial_page_registered', () => {
test('align_tutorial_page_wiring', () => { test('align_tutorial_page_wiring', () => {
const page = readSource('src/pages/records/RecordsTutorialPage.vue') const page = readSource('src/pages/records/RecordsTutorialPage.vue')
// 上传入口:选择 zip → 直传(进度条)→ 成功提示留驻弹窗。 // 上传入口:版本号(可空)→ 选择 zip → 直传(进度条)→ 成功提示留驻弹窗。
assert.match(page, /上传教程包/, '存在上传入口按钮') assert.match(page, /上传教程包/, '存在上传入口按钮')
assert.match(page, /uploadTutorialPackage/, '上传走教程包直传链路') assert.match(page, /uploadTutorialPackage/, '上传走教程包直传链路')
assert.match(page, /label="版本号"/, '上传弹窗提供版本号输入')
assert.match(page, /newVersion\.value\.trim\(\)/, '版本号去空白后随上传提交')
assert.match(page, /请选择 zip 压缩包/, '未选文件时提示') assert.match(page, /请选择 zip 压缩包/, '未选文件时提示')
assert.match(page, /仅支持 \.zip 格式/, '格式校验提示') assert.match(page, /仅支持 \.zip 格式/, '格式校验提示')
assert.match(page, /上传成功,工具台「立即下载教程」已切换为该包/, '成功提示说明生效位置') assert.match(page, /上传成功,工具台「立即下载教程」已切换为该包/, '成功提示说明生效位置')
assert.match(page, /上传完成,正在登记教程包/, '进度文案') assert.match(page, /上传完成,正在登记教程包/, '进度文案')
// 列表:当前生效标记 + 下载 + 删除 + 空态。 // 列表:版本号列 + 上传时间列 + 当前生效标记 + 下载 + 删除 + 空态。
assert.match(page, /当前生效/, '最新上传的包标记当前生效') assert.match(page, /当前生效/, '最新上传的包标记当前生效')
assert.match(page, /formatFileSize/, '展示包体大小') assert.match(page, /formatFileSize/, '展示包体大小')
assert.match(page, /row\.version/, '展示版本号列')
assert.match(page, /toggleSort\('version'\)/, '版本号表头可点击排序')
assert.match(page, /toggleSort\('createdAt'\)/, '上传时间表头可点击排序')
assert.match(page, /const sortKey = ref<SortKey>\('createdAt'\)/, '默认排序字段为上传时间')
assert.match(page, /const sortAsc = ref\(false\)/, '默认降序')
assert.match(page, /下载/, '操作列提供下载') assert.match(page, /下载/, '操作列提供下载')
assert.match(page, /<a v-if="row\.fileUrl" class="btn btn-sm dl-btn"[^>]*download/, '下载链接带 download 强制下载') assert.match(page, /<a v-if="row\.fileUrl" class="btn btn-sm dl-btn"[^>]*download/, '下载链接带 download 强制下载')
assert.match(page, /确认删除选中的/, '批量删除二次确认') assert.match(page, /确认删除选中的/, '批量删除二次确认')
@@ -37,7 +44,8 @@ test('align_tutorial_api_contract', () => {
assert.match(api, /\$\{TUTORIAL_UPLOAD_ENDPOINT\}\/presign/, '直传预签名端点') assert.match(api, /\$\{TUTORIAL_UPLOAD_ENDPOINT\}\/presign/, '直传预签名端点')
assert.match(api, /\$\{TUTORIAL_UPLOAD_ENDPOINT\}\/confirm/, '确认端点') assert.match(api, /\$\{TUTORIAL_UPLOAD_ENDPOINT\}\/confirm/, '确认端点')
assert.match(api, /\$\{TUTORIAL_UPLOAD_ENDPOINT\}\/delete/, '删除端点') assert.match(api, /\$\{TUTORIAL_UPLOAD_ENDPOINT\}\/delete/, '删除端点')
assert.match(api, /params: \{ object_key: objectKey, file_name: fileName \}/, '确认回传对象 key 与文件名') assert.match(api, /params: \{ file_name: fileName, version \}/, '预签名回传文件名与版本号')
assert.match(api, /params: \{ object_key: objectKey, file_name: fileName, version \}/, '确认回传对象 key、文件名与版本号')
assert.match(api, /withCredentials: false/, '直传实例不挂会话拦截器') assert.match(api, /withCredentials: false/, '直传实例不挂会话拦截器')
}) })
@@ -46,4 +54,5 @@ test('align_tutorial_model_parsers', () => {
assert.match(model, /r\.file_name \?\? r\.fileName/, '兼容 snake/camel 文件名') assert.match(model, /r\.file_name \?\? r\.fileName/, '兼容 snake/camel 文件名')
assert.match(model, /r\.file_url \?\? r\.fileUrl/, '兼容 snake/camel 下载链接') assert.match(model, /r\.file_url \?\? r\.fileUrl/, '兼容 snake/camel 下载链接')
assert.match(model, /r\.file_size \?\? r\.fileSize/, '兼容 snake/camel 文件大小') assert.match(model, /r\.file_size \?\? r\.fileSize/, '兼容 snake/camel 文件大小')
assert.match(model, /version: text\(r\.version\)/, '解析版本号(缺失为空串,兼容历史行)')
}) })
@@ -0,0 +1,86 @@
import test from 'node:test'
import assert from 'node:assert/strict'
import { readSource } from './helpers.ts'
import {
buildMenuOptionTree,
compactDirectGrantIds,
parseMenuOptionList,
parsePermissionMenuItem,
} from '../src/pages/account/user-menu-auth.ts'
// 2026-09-16 线上事故:非超管(普通管理员)在授权树里勾到自己无权授予的菜单后,
// 后端 ensureGrantable 抛 403 并回滚整笔事务——创建用户与保存权限双双失败,
// 前端只显示「普通管理员只能分配自己已有的菜单权限」。
// 修复:菜单列表按操作者标记 grantable,前端把不可授予的节点置灰不可勾。
//
// 注意:这里是「置灰」而非「隐藏」。授权保存是整树替换,隐藏会让超管早先授予、
// 而操作者自己没有的菜单在提交时被当作取消勾选删掉(与 09-13「权限自己没掉」同类)。
test('align_user_menu_grantable_false_maps_to_disabled_node', () => {
const locked = parsePermissionMenuItem(
{ id: 5, name: '查询ASIN', parent_id: null, sort_order: 1, grantable: false },
'admin',
)
assert.equal(locked?.disabled, true, 'grantable=false → 节点置灰')
const allowed = parsePermissionMenuItem(
{ id: 6, name: '店铺管理', parent_id: null, sort_order: 2, grantable: true },
'admin',
)
assert.equal(allowed?.disabled, false, 'grantable=true → 可勾选')
// 菜单管理页等未标记 grantable 的接口必须保持旧行为(全部可勾选)
const unmarked = parsePermissionMenuItem({ id: 7, name: '菜单权限配置', parent_id: null, sort_order: 3 }, 'admin')
assert.equal(unmarked?.disabled, false, '缺省 grantable 视为可授予')
})
test('align_user_menu_grantable_survives_tree_build', () => {
const nodes = parseMenuOptionList(
[
{ id: 98, name: '账号与权限', parent_id: null, sort_order: 1, grantable: false },
{ id: 7, name: '用户管理', parent_id: 98, sort_order: 1, grantable: false },
{ id: 100, name: '店铺管理', parent_id: null, sort_order: 2, grantable: true },
],
'admin',
)
const tree = buildMenuOptionTree(nodes)
const account = tree.find((node) => node.id === 98)
assert.equal(account?.disabled, true, '分组节点置灰')
assert.equal(account?.children?.[0]?.disabled, true, '子节点置灰随树保留')
assert.equal(tree.find((node) => node.id === 100)?.disabled, false, '可授予节点不受影响')
})
test('align_user_menu_grantable_disabled_node_still_compactable', () => {
// 已持有但无权授予的节点会保持勾选并原样提交,压缩逻辑不能因 disabled 漏掉它
const tree = buildMenuOptionTree(
parseMenuOptionList(
[
{ id: 98, name: '账号与权限', parent_id: null, sort_order: 1, grantable: false },
{ id: 7, name: '用户管理', parent_id: 98, sort_order: 1, grantable: false },
],
'admin',
),
)
assert.deepEqual(compactDirectGrantIds([98, 7], tree), [98], '父级已勾选时仍压缩掉后代')
})
test('align_user_menu_grantable_wired_end_to_end', () => {
const tree = readSource('src/pages/account/UserMenuAuthTree.vue')
assert.match(tree, /disabled: 'disabled'/, 'el-tree 按 disabled 键置灰节点')
const vo = readSource(
'../backend-java/src/main/java/com/nanri/aiimage/modules/permission/model/vo/PermissionMenuItemVo.java',
)
assert.match(vo, /private Boolean grantable;/, 'VO 暴露 grantable')
const controller = readSource(
'../backend-java/src/main/java/com/nanri/aiimage/modules/permission/controller/PermissionMenuController.java',
)
assert.match(controller, /permissionMenuService\.list\(requireAdmin\(request\), menuType\)/, '列表接口传入操作者')
const service = readSource(
'../backend-java/src/main/java/com/nanri/aiimage/modules/permission/service/PermissionMenuService.java',
)
assert.match(service, /resolveGrantableMenuIds/, '按操作者计算可授予集')
assert.match(service, /ensureGrantable\(operator, grantIds, userId\)/, '保存校验传入目标用户以放行既有授权')
})
@@ -0,0 +1,28 @@
# 单次提交事务段耗时基线(task-138)
> 本文档由 `TxDurationBenchmarkTest` 断言存在与内容,用于冻结「Python 结果提交」单次调用的
> 耗时上界,防止提交路径劣化。**不要删除**,劣化时更新数值并说明原因。
## 测量对象
`SimilarAsinTaskService.submitResult(taskId, request)` 的 mock 环境单次调用,拆成两段:
| 段 | 含义 |
|---|---|
| 计算段(prepare) | 事务外的纯计算:行裁剪、校验、payload 序列化与哈希 |
| 事务段(persist | `inNewTransaction` 内的落库:分片 upsert、scope state 更新 |
## 基线数值
- **单次事务段基线: 5**(毫秒,mock 环境,本地实测)
- **200 分片**负载基线: 10 秒(`twoHundredChunkLoadBounded` 的上界)
- 单次提交总耗时上界: 500 毫秒(`totalDurationReasonablePerSubmission`
`noRegressionAgainstDocumentedBaseline` 按「当前 ≤ 基线 × 3」判定劣化,因此事务段超过
15ms 即视为回归。
## 说明
- 该基线取的是 mock 环境(Mapper 全部打桩)而非真实 MySQL 的耗时,用于**相对劣化**而非绝对性能。
- 真实环境的事务段耗时会显著高于此值,本基线不适用于容量规划。
- 测量机器与 JDK 变更后如需调整,请同步更新本文件数值。
@@ -0,0 +1,36 @@
package com.nanri.aiimage.common.exception;
/**
* 业务错误码常量。
*
* <p>历史上 40901 被两种语义复用:「任务已结束」(幂等忽略)与「任务正在处理中」(分布式锁竞争,
* 应重试);而 GlobalExceptionHandler 把 40901 一律转成 HTTP 200 + success=true
* 导致锁竞争时 Python worker 误判回传成功并停止重试,分片数据静默丢失。
* 现拆分为两个码:
* <ul>
* <li>{@link #TASK_ALREADY_FINISHED}:任务已结束,重复提交无意义 → 幂等忽略,响应 success=true</li>
* <li>{@link #TASK_BUSY}:任务正被其它请求持锁推进 → 响应 success=false,调用方应稍后重试。</li>
* </ul>
*/
public final class BusinessCodes {
private BusinessCodes() {
}
/** 任务已结束,拒绝重复提交。语义:幂等忽略,响应 success=true,调用方不应重试。 */
public static final int TASK_ALREADY_FINISHED = 40901;
/** 任务正在处理中(分布式锁竞争)。语义:资源忙,响应 success=false,调用方应稍后重试。 */
public static final int TASK_BUSY = 40902;
/** 任务归属其它实例,需转发。 */
public static final int TASK_OWNER_FORWARD = 40903;
/**
* 提交结果的目标任务已不存在(通常是被删除)。
* 语义:本次提交无意义,响应 success=false 且带该码,调用方应放弃而不是反复重试。
* 与 {@link #TASK_ALREADY_FINISHED} 的区别:那个还留了任务记录(可幂等忽略),
* 这个任务已经没了——如实报错,否则任务被误删时结果会被静默吞掉。
*/
public static final int TASK_NOT_FOUND = 40401;
}
@@ -1,18 +1,21 @@
package com.nanri.aiimage.common.exception; package com.nanri.aiimage.common.exception;
import com.nanri.aiimage.common.api.ApiResponse; import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.common.security.DeviceSessionPolicy;
import com.nanri.aiimage.common.service.TaskOwnerForwardService; import com.nanri.aiimage.common.service.TaskOwnerForwardService;
import com.nanri.aiimage.config.TaskOperationLockConfig; import com.nanri.aiimage.config.TaskOperationLockConfig;
import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletRequest;
import jakarta.validation.ConstraintViolationException; import jakarta.validation.ConstraintViolationException;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity; import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.MethodArgumentNotValidException; import org.springframework.web.bind.MethodArgumentNotValidException;
import org.springframework.web.bind.annotation.ExceptionHandler; import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice; import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.context.request.async.AsyncRequestNotUsableException; import org.springframework.web.context.request.async.AsyncRequestNotUsableException;
import org.springframework.web.server.ResponseStatusException; import org.springframework.web.server.ResponseStatusException;
import org.springframework.web.servlet.resource.NoResourceFoundException;
import java.io.IOException; import java.io.IOException;
@@ -52,26 +55,64 @@ public class GlobalExceptionHandler {
? ApiResponse.fail(forwardEx.getMessage()) ? ApiResponse.fail(forwardEx.getMessage())
: ApiResponse.fail(forwardEx.getCode(), forwardEx.getMessage()); : ApiResponse.fail(forwardEx.getCode(), forwardEx.getMessage());
} catch (Exception forwardEx) { } catch (Exception forwardEx) {
// 转发失败是**瞬时基础设施故障**(归属实例正在滚动重启),不是业务结论,
// 更不能表达成「任务不存活」。原先返回 ApiResponse.fail(40903) —— HTTP 200
// 加 data:null,而客户端那句 bool((resp.json().get("data") or {}).get("alive"))
// 会把「拿不到数据」折叠成 alive=false,于是客户端把**健康的长任务主动停掉**:
// 2026-09-18 任务 28616 就是这么死的(归属节点 server-110 重启窗口内,心跳经
// nginx 落到 server-121,转发 3 次 Connection refused 后返回空 data)。
// 改为 503 + 空 body:新客户端按状态码判为「未知」继续跑;老客户端因 body 不是
// JSON、resp.json() 抛异常,同样落到「未知」。顺带让这类故障在 HTTP 指标里可见
// (原先记成 200,监控完全看不到滚动重启期间丢了多少心跳)。
log.warn("[instance-routing] forward failed taskId={} operation={} owner={} current={} msg={}", log.warn("[instance-routing] forward failed taskId={} operation={} owner={} current={} msg={}",
ex.getTaskId(), ex.getOperation(), ex.getOwnerInstanceId(), ex.getCurrentInstanceId(), ex.getTaskId(), ex.getOperation(), ex.getOwnerInstanceId(), ex.getCurrentInstanceId(),
forwardEx.getMessage(), forwardEx); forwardEx.getMessage(), forwardEx);
return ApiResponse.fail(40903, "任务归属实例转发失败: " + forwardEx.getMessage()); return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE).build();
} }
} }
@ExceptionHandler(BusinessException.class) @ExceptionHandler(BusinessException.class)
public ApiResponse<Void> handleBusinessException(BusinessException ex) { public ApiResponse<Void> handleBusinessException(BusinessException ex, HttpServletRequest request) {
if (Integer.valueOf(40901).equals(ex.getCode())) { // 业务异常此前完全不记日志:2026-09-16 线上「保存权限/创建用户」双双失败时,
// 服务端只留 RequestTraceFilter 的 200 一行,根因只能靠反推响应体字节数才定位到。
// 401/4011(未登录、被顶下线)属于轮询类接口的常态噪声,降为 debug 以免淹没真实业务错。
if (isRoutineAuthNoise(ex.getCode())) {
log.debug("[business] {} {} code={} message={}", request.getMethod(), request.getRequestURI(),
ex.getCode(), ex.getMessage());
} else {
log.warn("[business] {} {} code={} message={}", request.getMethod(), request.getRequestURI(),
ex.getCode(), ex.getMessage());
}
if (Integer.valueOf(BusinessCodes.TASK_ALREADY_FINISHED).equals(ex.getCode())) {
// 幂等忽略:任务已结束时的重复提交无副作用,按成功返回,避免客户端反复重试
return ApiResponse.success("任务已结束,忽略重复提交", null); return ApiResponse.success("任务已结束,忽略重复提交", null);
} }
if (Integer.valueOf(BusinessCodes.TASK_BUSY).equals(ex.getCode())) {
// 锁竞争:必须如实返回失败 + 可重试码,否则 worker 会把「未落库」当成功而停止重试
return ApiResponse.fail(BusinessCodes.TASK_BUSY, ex.getMessage());
}
return ex.getCode() == null return ex.getCode() == null
? ApiResponse.fail(ex.getMessage()) ? ApiResponse.fail(ex.getMessage())
: ApiResponse.fail(ex.getCode(), ex.getMessage()); : ApiResponse.fail(ex.getCode(), ex.getMessage());
} }
@ExceptionHandler(NoResourceFoundException.class)
public ResponseEntity<ApiResponse<Void>> handleNoResourceFoundException(NoResourceFoundException ex) {
// 静态资源 404。绝大部分是外部扫描器在探测 /.env、/credentials、aliyun.json、oss.json
// 这类云凭据文件(线上单节点一天 580 条)。此前落到 handleException 里,既刷 ERROR 堆栈,
// 又把探测响应伪装成 HTTP 200;这里降为 debug 并如实返回 404。
log.debug("static resource not found: {}", ex.getMessage());
return ResponseEntity.status(HttpStatus.NOT_FOUND).body(ApiResponse.fail("资源不存在"));
}
/** 未登录 / 登录态失效 / 被其他设备顶下线:按 401 语义的常态噪声,不占 WARN。 */
private boolean isRoutineAuthNoise(Integer code) {
return Integer.valueOf(401).equals(code)
|| Integer.valueOf(DeviceSessionPolicy.CODE_KICKED).equals(code);
}
@ExceptionHandler(MethodArgumentNotValidException.class) @ExceptionHandler(MethodArgumentNotValidException.class)
public ApiResponse<Void> handleMethodArgumentNotValidException(MethodArgumentNotValidException ex) { public ApiResponse<Void> handleMethodArgumentNotValidException(MethodArgumentNotValidException ex) { String message = ex.getBindingResult().getFieldError() != null
String message = ex.getBindingResult().getFieldError() != null
? ex.getBindingResult().getFieldError().getDefaultMessage() ? ex.getBindingResult().getFieldError().getDefaultMessage()
: "参数校验失败"; : "参数校验失败";
return ApiResponse.fail(message); return ApiResponse.fail(message);
@@ -100,7 +141,9 @@ public class GlobalExceptionHandler {
return ApiResponse.fail("客户端已断开连接"); return ApiResponse.fail("客户端已断开连接");
} }
log.error("Unhandled exception", ex); log.error("Unhandled exception", ex);
return ApiResponse.fail("服务异常: " + ex.getMessage()); // 不再回传原始异常信息:SQL 报错、类名与内部路径会直接暴露给调用方,便于攻击者
// 摸清技术栈与表结构。详情只进日志(上方 log.error 已带完整堆栈),对外统一文案。
return ApiResponse.fail("服务器内部错误,请稍后重试");
} }
private boolean isClientAbort(Throwable ex) { private boolean isClientAbort(Throwable ex) {
@@ -1,7 +1,7 @@
package com.nanri.aiimage.modules.permission.mapper; package com.nanri.aiimage.common.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper; import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.nanri.aiimage.modules.permission.model.entity.AdminUserEntity; import com.nanri.aiimage.common.model.entity.AdminUserEntity;
import org.apache.ibatis.annotations.Mapper; import org.apache.ibatis.annotations.Mapper;
@Mapper @Mapper
@@ -1,7 +1,7 @@
package com.nanri.aiimage.modules.shopkey.mapper; package com.nanri.aiimage.common.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper; import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.nanri.aiimage.modules.shopkey.model.entity.ShopManageGroupEntity; import com.nanri.aiimage.common.model.entity.ShopManageGroupEntity;
import org.apache.ibatis.annotations.Mapper; import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param; import org.apache.ibatis.annotations.Param;
import org.apache.ibatis.annotations.Select; import org.apache.ibatis.annotations.Select;
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.permission.model.entity; package com.nanri.aiimage.common.model.entity;
import com.baomidou.mybatisplus.annotation.IdType; import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableField; import com.baomidou.mybatisplus.annotation.TableField;
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.shopkey.model.entity; package com.nanri.aiimage.common.model.entity;
import com.baomidou.mybatisplus.annotation.IdType; import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableField; import com.baomidou.mybatisplus.annotation.TableField;
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.productrisk.model.vo; package com.nanri.aiimage.common.model.vo;
import com.fasterxml.jackson.annotation.JsonProperty; import com.fasterxml.jackson.annotation.JsonProperty;
import io.swagger.v3.oas.annotations.media.Schema; import io.swagger.v3.oas.annotations.media.Schema;
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.productrisk.model.vo; package com.nanri.aiimage.common.model.vo;
import com.fasterxml.jackson.annotation.JsonProperty; import com.fasterxml.jackson.annotation.JsonProperty;
import io.swagger.v3.oas.annotations.media.Schema; import io.swagger.v3.oas.annotations.media.Schema;
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.productrisk.model.vo; package com.nanri.aiimage.common.model.vo;
import com.fasterxml.jackson.annotation.JsonProperty; import com.fasterxml.jackson.annotation.JsonProperty;
import io.swagger.v3.oas.annotations.media.Schema; import io.swagger.v3.oas.annotations.media.Schema;
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.productrisk.model.vo; package com.nanri.aiimage.common.model.vo;
import io.swagger.v3.oas.annotations.media.Schema; import io.swagger.v3.oas.annotations.media.Schema;
import lombok.Data; import lombok.Data;
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.productrisk.model.vo; package com.nanri.aiimage.common.model.vo;
import io.swagger.v3.oas.annotations.media.Schema; import io.swagger.v3.oas.annotations.media.Schema;
import lombok.Data; import lombok.Data;
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.productrisk.model.vo; package com.nanri.aiimage.common.model.vo;
import com.fasterxml.jackson.annotation.JsonAlias; import com.fasterxml.jackson.annotation.JsonAlias;
import com.fasterxml.jackson.annotation.JsonProperty; import com.fasterxml.jackson.annotation.JsonProperty;
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.productrisk.model.vo; package com.nanri.aiimage.common.model.vo;
import com.fasterxml.jackson.annotation.JsonAlias; import com.fasterxml.jackson.annotation.JsonAlias;
import com.fasterxml.jackson.annotation.JsonProperty; import com.fasterxml.jackson.annotation.JsonProperty;
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.productrisk.model.vo; package com.nanri.aiimage.common.model.vo;
import io.swagger.v3.oas.annotations.media.Schema; import io.swagger.v3.oas.annotations.media.Schema;
import lombok.Data; import lombok.Data;
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.productrisk.model.vo; package com.nanri.aiimage.common.model.vo;
import io.swagger.v3.oas.annotations.media.Schema; import io.swagger.v3.oas.annotations.media.Schema;
import lombok.Data; import lombok.Data;
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.productrisk.model.vo; package com.nanri.aiimage.common.model.vo;
import com.fasterxml.jackson.annotation.JsonProperty; import com.fasterxml.jackson.annotation.JsonProperty;
import com.nanri.aiimage.modules.shopmatch.model.vo.ShopMatchTaskStageVo; import com.nanri.aiimage.modules.shopmatch.model.vo.ShopMatchTaskStageVo;
@@ -0,0 +1,111 @@
package com.nanri.aiimage.common.module;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
/**
* 任务模块清单的**单一来源**(2026-09 全维度审查 G6)。
*
* 背景:新增一个工具模块此前要同时改 5 处枚举(结果文件 Job 白名单、按天清理清单、
* 站内通知模块名、任务心跳分支、陈旧判死巡检线),其中 4 处没有任何自检,
* 漏改表现为"功能静默不生效"。现在把**数据驱动的三处**(结果文件 Job、按天清理、通知中文名)
* 统一从这里派生,并由 {@code TaskModuleCoverageTest} 守住覆盖面。
*
* 心跳与陈旧判死两处是代码分支驱动(switch / 逐模块委派),无法纯数据派生,
* 新增模块时仍需实现对应 Handler,测试会提示缺失。
*/
public final class TaskModuleRegistry {
/**
* 单个任务模块的声明。
*
* @param type moduleType(与 biz_file_task.module_type 一致)
* @param label 站内通知/界面用中文名
* @param resultFileJob 是否产出结果文件(需要注册 ResultFileJobHandler
* @param ageCleanup 是否参与按天清理(ModuleCleanupProperties
* @param delegatedStaleCheck 是否以"委派"方式并入 stale-check 巡检线
* DeleteBrandStaleTaskService.delegatedStaleChecks,必须逐模块登记动作)
* @param selfScheduledStaleCheck 该模块的陈旧判死自带更快的调度(publish 60s / collect-data 30s),
* 刻意不并入 2 分钟一轮的巡检线——并入会显著拉长判死时延。
* 这类模块的"不在委托名单"是设计差异,不是漏接;由覆盖面测试固定。
*/
public record Module(String type, String label, boolean resultFileJob, boolean ageCleanup,
boolean delegatedStaleCheck, boolean selfScheduledStaleCheck) {
}
private static final List<Module> MODULES = List.of(
new Module("PUBLISH", "上架", true, false, false, true),
new Module("DEDUPE", "数据去重", false, true, false, false),
new Module("SPLIT", "数据拆分", false, true, false, false),
new Module("CONVERT", "格式转换", false, true, false, false),
new Module("DELETE_BRAND", "删除ASIN", true, true, false, false),
new Module("PRODUCT_RISK_RESOLVE", "商品风险解决", true, true, false, false),
new Module("PRICE_TRACK", "跟价", true, true, false, false),
new Module("SHOP_MATCH", "定时匹配", true, true, false, false),
new Module("PATROL_DELETE", "巡店删除", true, true, false, false),
new Module("QUERY_ASIN", "查询ASIN", true, true, false, false),
new Module("WITHDRAW", "取款", true, true, false, false),
new Module("APPEARANCE_PATENT", "外观专利检测", true, true, true, false),
new Module("SIMILAR_ASIN", "货源查询", true, true, true, false),
new Module("COLLECT_DATA", "采集数据", true, true, false, true),
new Module("SHOP_DATA_CRAWL", "店铺数据抓取", true, false, true, false),
new Module("BRAND", "品牌检测", true, false, true, false)
);
private TaskModuleRegistry() {
}
public static List<Module> modules() {
return MODULES;
}
/** 全部 moduleType。 */
public static Set<String> moduleTypes() {
return MODULES.stream().map(Module::type).collect(Collectors.toUnmodifiableSet());
}
/** moduleType → 中文名(站内通知等展示用)。 */
public static Map<String, String> labels() {
Map<String, String> labels = new LinkedHashMap<>();
for (Module module : MODULES) {
labels.put(module.type(), module.label());
}
return Map.copyOf(labels);
}
/** 产出结果文件的模块(需要注册 Handler 的模块)。 */
public static Set<String> resultFileJobModuleTypes() {
return MODULES.stream().filter(Module::resultFileJob).map(Module::type)
.collect(Collectors.toUnmodifiableSet());
}
/** 参与按天清理的模块。 */
public static Set<String> ageCleanupModuleTypes() {
return MODULES.stream().filter(Module::ageCleanup).map(Module::type)
.collect(Collectors.toUnmodifiableSet());
}
/** 自带更快巡检节奏、刻意不并入集中巡检线的模块。 */
public static Set<String> selfScheduledStaleCheckModuleTypes() {
return MODULES.stream().filter(Module::selfScheduledStaleCheck).map(Module::type)
.collect(Collectors.toUnmodifiableSet());
}
/** 以"委派"方式并入 stale-check 巡检线的模块(DeleteBrandStaleTaskService 必须逐模块登记动作)。 */
public static Set<String> delegatedStaleCheckModuleTypes() {
return MODULES.stream().filter(Module::delegatedStaleCheck).map(Module::type)
.collect(Collectors.toUnmodifiableSet());
}
public static String labelOf(String moduleType) {
for (Module module : MODULES) {
if (module.type().equals(moduleType)) {
return module.label();
}
}
return moduleType;
}
}
@@ -1,24 +1,26 @@
package com.nanri.aiimage.modules.admin.support; package com.nanri.aiimage.common.security;
import com.nanri.aiimage.common.exception.BusinessException; import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.modules.auth.service.JwtService; import com.nanri.aiimage.common.security.JwtService;
import com.nanri.aiimage.modules.auth.support.DeviceSessionPolicy; import com.nanri.aiimage.common.security.DeviceSessionPolicy;
import com.nanri.aiimage.modules.permission.mapper.AdminUserMapper; import com.nanri.aiimage.common.mapper.AdminUserMapper;
import com.nanri.aiimage.modules.permission.model.entity.AdminUserEntity; import com.nanri.aiimage.common.model.entity.AdminUserEntity;
import io.jsonwebtoken.Claims; import io.jsonwebtoken.Claims;
import jakarta.servlet.http.Cookie; import jakarta.servlet.http.Cookie;
import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletRequest;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.http.HttpHeaders; import org.springframework.http.HttpHeaders;
import org.springframework.stereotype.Component; import org.springframework.stereotype.Component;
import com.nanri.aiimage.modules.auth.config.AuthProperties; import com.nanri.aiimage.common.security.AuthProperties;
import org.springframework.beans.factory.annotation.Value; import org.springframework.beans.factory.annotation.Value;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.nio.file.Files; import java.nio.file.Files;
import java.nio.file.Path; import java.nio.file.Path;
@Slf4j
@Component @Component
@RequiredArgsConstructor @RequiredArgsConstructor
public class AdminAuthSupport { public class AdminAuthSupport {
@@ -62,8 +64,30 @@ public class AdminAuthSupport {
return user; return user;
} }
/** 当前用户必须是管理员或超级管理员,否则抛 403。 */ /**
public AdminUserEntity requireAdmin(HttpServletRequest request) { * 解析当前请求 JWT **签名的**设备标识deviceId claim识别不出时返回空串
*
* <p> tokentoken 过期/非法内部令牌通道调用一律返回空串调用方必须把空串
* 当作"来源不明"做保守判定绝不据此放宽任何限制只认签名 claim不接受
* X-Device-Id 请求头头由客户端可控 {@link DeviceSessionPolicy} 类注释</p>
*
* <p>本方法只做识别不做鉴权因此解析失败不抛异常仅记日志后返回空串
* 避免把匿名/内部调用直接升级成 401</p>
*/
public String currentDeviceId(HttpServletRequest request) {
String token = resolveToken(request);
if (token == null || token.isBlank()) {
return "";
}
try {
return DeviceSessionPolicy.claimDeviceId(jwtService.parse(token));
} catch (Exception ex) {
log.warn("[auth] 解析 token 取设备标识失败,按来源不明处理: {}", ex.getMessage());
return "";
}
}
/** 当前用户必须是管理员或超级管理员,否则抛 403。 */ public AdminUserEntity requireAdmin(HttpServletRequest request) {
AdminUserEntity user = requireUser(request); AdminUserEntity user = requireUser(request);
String role = currentRole(user); String role = currentRole(user);
if (role == null) { if (role == null) {
@@ -143,7 +167,10 @@ public class AdminAuthSupport {
if (expectedToken.isBlank() || suppliedToken == null || suppliedToken.isBlank()) { if (expectedToken.isBlank() || suppliedToken == null || suppliedToken.isBlank()) {
return false; return false;
} }
return expectedToken.equals(suppliedToken.trim()); // 常量时间比较逐字节 equals 可被计时侧信道逐位试探出内部令牌
return java.security.MessageDigest.isEqual(
expectedToken.getBytes(java.nio.charset.StandardCharsets.UTF_8),
suppliedToken.trim().getBytes(java.nio.charset.StandardCharsets.UTF_8));
} }
/** /**
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.auth.config; package com.nanri.aiimage.common.security;
import lombok.Data; import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.boot.context.properties.ConfigurationProperties;
@@ -1,4 +1,4 @@
package com.nanri.aiimage.modules.auth.support; package com.nanri.aiimage.common.security;
import com.nanri.aiimage.common.exception.BusinessException; import com.nanri.aiimage.common.exception.BusinessException;
import io.jsonwebtoken.Claims; import io.jsonwebtoken.Claims;
@@ -1,6 +1,6 @@
package com.nanri.aiimage.modules.auth.service; package com.nanri.aiimage.common.security;
import com.nanri.aiimage.modules.auth.config.AuthProperties; import com.nanri.aiimage.common.security.AuthProperties;
import io.jsonwebtoken.Claims; import io.jsonwebtoken.Claims;
import io.jsonwebtoken.JwtException; import io.jsonwebtoken.JwtException;
import io.jsonwebtoken.Jwts; import io.jsonwebtoken.Jwts;
@@ -26,6 +26,26 @@ public class JwtService {
private volatile SecretKey cachedKey; private volatile SecretKey cachedKey;
@org.springframework.beans.factory.annotation.Value("${spring.profiles.active:}")
private String activeProfiles;
/**
* 生产 profile 下密钥缺失即拒绝启动2026-09 全维度审查
*
* <p>此前未配置时静默回退到内置默认密钥那是公开值任何人可据此伪造 token
* 一次 env 丢失就会让全站鉴权形同虚设只留一行 warn 日志不足以拦住发布
* 本地/测试 profile 保持宽松否则开发无法启动
*/
@jakarta.annotation.PostConstruct
void requireSecretInServerProfile() {
boolean serverProfile = activeProfiles != null && activeProfiles.contains("server");
if (serverProfile && (props.getJwtSecret() == null || props.getJwtSecret().isBlank())) {
throw new IllegalStateException(
"server profile 下必须配置 aiimage.auth.jwt-secret(环境变量 AIIMAGE_JWT_SECRET)——"
+ "缺失时会回退到公开的默认密钥,token 可被任意伪造");
}
}
private SecretKey signingKey() { private SecretKey signingKey() {
SecretKey key = cachedKey; SecretKey key = cachedKey;
if (key == null) { if (key == null) {
@@ -0,0 +1,179 @@
package com.nanri.aiimage.common.service;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import org.springframework.data.redis.core.StringRedisTemplate;
import java.time.Duration;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicLong;
/**
* 跨节点的小容量状态存储(2026-09 全维度审查 D2)。
*
* <p>用途:导入/导出任务的进度、归属用户、分组等状态。原先只存节点本地内存,
* 客户端轮询落到另一节点就报"任务不存在"nginx 的 user_id 亲和只覆盖常态)。
* 现在本地 Map 作快路径、Redis 作跨节点真源,任何节点都能读到。
*
* <p>写节流:调用方可能**逐行**刷新进度(数十万行),逐次写 Redis 不可接受;
* 默认 500ms 内只写一次(终态用 {@link #putNow} 立即落库)。
*
* <p>语义取舍:读-改-写不保证原子(进度类状态可接受);{@code redis}/{@code objectMapper}
* 为空时退化为纯本地(单测与未注入场景)。
*/
@Slf4j
public final class NodeSharedStore<K, V> {
private static final long DEFAULT_WRITE_THROTTLE_MILLIS = 500L;
private final String keyPrefix;
private final Duration ttl;
private final Class<V> valueType;
private final StringRedisTemplate redis;
private final ObjectMapper objectMapper;
private final long writeThrottleMillis;
private final ConcurrentHashMap<K, V> local = new ConcurrentHashMap<>();
private final ConcurrentHashMap<K, AtomicLong> lastWriteAt = new ConcurrentHashMap<>();
public NodeSharedStore(String keyPrefix,
Duration ttl,
Class<V> valueType,
StringRedisTemplate redis,
ObjectMapper objectMapper) {
this(keyPrefix, ttl, valueType, redis, objectMapper, DEFAULT_WRITE_THROTTLE_MILLIS);
}
public NodeSharedStore(String keyPrefix,
Duration ttl,
Class<V> valueType,
StringRedisTemplate redis,
ObjectMapper objectMapper,
long writeThrottleMillis) {
this.keyPrefix = keyPrefix;
this.ttl = ttl;
this.valueType = valueType;
this.redis = redis;
this.objectMapper = objectMapper;
this.writeThrottleMillis = Math.max(0L, writeThrottleMillis);
}
/** 本地优先;本地没有则读 Redis 并回填本地(跨节点可见)。 */
public V get(K key) {
if (key == null) {
return null;
}
V cached = local.get(key);
if (cached != null) {
return cached;
}
V remote = readRemote(key);
if (remote != null) {
local.put(key, remote);
}
return remote;
}
public boolean containsKey(K key) {
return get(key) != null;
}
/** 写入并(按节流)同步到 Redis。 */
public void put(K key, V value) {
if (key == null || value == null) {
return;
}
local.put(key, value);
if (!throttleAllowsWrite(key)) {
return;
}
writeRemote(key, value);
}
/** 立即写入 Redis(终态、归属等一次性状态用)。 */
public void putNow(K key, V value) {
if (key == null || value == null) {
return;
}
local.put(key, value);
writeRemote(key, value);
}
/** 删除(本地 + Redis),返回删除前的值(可能为 null)。 */
public V remove(K key) {
if (key == null) {
return null;
}
V previous = local.remove(key);
lastWriteAt.remove(key);
if (redis != null) {
try {
redis.delete(fullKey(key));
} catch (Exception ex) {
log.warn("[node-shared-store] 删除远端状态失败 key={} msg={}", fullKey(key), ex.getMessage());
}
}
return previous;
}
/** 本地快照(仅用于日志/统计;不含其它节点的写入)。 */
public int localSize() {
return local.size();
}
/**
* 本节点已知条目的快照(副本,可安全遍历)。
*
* <p>用途:保留期清理等维护动作需要遍历键;跨节点的过期回收由 Redis TTL 兜底,
* 因此这里只返回本节点写入过的条目即可。
*/
public java.util.Map<K, V> localEntriesSnapshot() {
return new java.util.LinkedHashMap<>(local);
}
private boolean throttleAllowsWrite(K key) {
if (writeThrottleMillis <= 0L) {
return true;
}
long now = System.currentTimeMillis();
AtomicLong last = lastWriteAt.computeIfAbsent(key, ignored -> new AtomicLong(0L));
long previous = last.get();
if (now - previous < writeThrottleMillis) {
return false;
}
return last.compareAndSet(previous, now);
}
private V readRemote(K key) {
if (redis == null || objectMapper == null) {
return null;
}
try {
String json = redis.opsForValue().get(fullKey(key));
if (json == null || json.isBlank()) {
return null;
}
return objectMapper.readValue(json, valueType);
} catch (Exception ex) {
log.warn("[node-shared-store] 读取远端状态失败 key={} msg={}", fullKey(key), ex.getMessage());
return null;
}
}
private void writeRemote(K key, V value) {
if (redis == null || objectMapper == null) {
return;
}
try {
String json = objectMapper.writeValueAsString(value);
redis.opsForValue().set(fullKey(key), json, ttl);
} catch (Exception ex) {
// 写失败不影响本地进度(下次 put 会重试),只留线索
log.warn("[node-shared-store] 写入远端状态失败 key={} msg={}", fullKey(key), ex.getMessage());
}
}
private String fullKey(K key) {
return keyPrefix + ":" + key;
}
}
@@ -0,0 +1,46 @@
package com.nanri.aiimage.common.service;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Component;
import java.util.Objects;
/**
* 结果文件下载直链解析(2026-09 全维度审查去重)。
*
* <p>此前 11 个业务模块各写一份逐字相同的 {@code resolveResultDownloadUrl},下载鉴权口径
* 或 OSS 直链规则一调整就要改 11 处;similarasin 已自行分叉成返回 record 的第 12 种写法,
* 说明"改的时候漏一个"已经开始发生。
*
* <p>文件名解析({@code resolveResultDownloadFilename}**未**收口:各模块兜底文件名策略
* 确实不同(模块名+id / 源文件名派生 stem),属业务差异而非重复。
*/
@Component
@RequiredArgsConstructor
public class ResultDownloadResolver {
private final FileResultMapper fileResultMapper;
private final OssStorageService ossStorageService;
/**
* 解析结果文件下载直链。
*
* @param resultId 结果行 idbiz_file_result
* @param userId 当前用户 id,必须与结果行归属一致
* @param moduleType 期望的模块类型
*/
public String resolveUrl(Long resultId, Long userId, String moduleType) {
FileResultEntity row = fileResultMapper.selectById(resultId);
if (row == null || !moduleType.equals(row.getModuleType()) || !Objects.equals(userId, row.getUserId())) {
throw new BusinessException("记录不存在");
}
if (row.getResultFileUrl() == null || row.getResultFileUrl().isBlank()) {
throw new BusinessException("暂无可下载文件");
}
return ossStorageService.generateFreshDownloadUrl(row.getResultFileUrl());
}
}
@@ -13,10 +13,12 @@ import org.springframework.http.ResponseEntity;
import org.springframework.http.client.SimpleClientHttpRequestFactory; import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.util.StreamUtils; import org.springframework.util.StreamUtils;
import org.springframework.web.client.ResourceAccessException;
import org.springframework.web.client.RestClient; import org.springframework.web.client.RestClient;
import org.springframework.web.util.ContentCachingRequestWrapper; import org.springframework.web.util.ContentCachingRequestWrapper;
import java.io.IOException; import java.io.IOException;
import java.net.ConnectException;
import java.util.Enumeration; import java.util.Enumeration;
import java.util.List; import java.util.List;
import java.util.Set; import java.util.Set;
@@ -28,6 +30,11 @@ public class TaskOwnerForwardService {
public static final String FORWARDED_HEADER = "X-AIIMAGE-Owner-Forwarded"; public static final String FORWARDED_HEADER = "X-AIIMAGE-Owner-Forwarded";
/** 连接类失败的重试次数(含首次)。对端滚动重启时通常几秒内即可恢复。 */
private static final int CONNECT_RETRY_TIMES = 3;
/** 第 n 次重试前的退避:1s、2s(总等待不超过 3s,不长时间占用请求线程)。 */
private static final long CONNECT_RETRY_BACKOFF_MILLIS = 1000L;
private static final Set<String> HOP_BY_HOP_HEADERS = Set.of( private static final Set<String> HOP_BY_HOP_HEADERS = Set.of(
"connection", "connection",
"keep-alive", "keep-alive",
@@ -58,12 +65,60 @@ public class TaskOwnerForwardService {
HttpHeaders headers = copyForwardHeaders(request, ex.getCurrentInstanceId()); HttpHeaders headers = copyForwardHeaders(request, ex.getCurrentInstanceId());
log.info("[instance-routing] forward {} taskId={} operation={} owner={} current={} url={}", log.info("[instance-routing] forward {} taskId={} operation={} owner={} current={} url={}",
method, ex.getTaskId(), ex.getOperation(), ex.getOwnerInstanceId(), ex.getCurrentInstanceId(), url); method, ex.getTaskId(), ex.getOperation(), ex.getOwnerInstanceId(), ex.getCurrentInstanceId(), url);
return forwardWithConnectRetry(method, url, headers, body, ex);
}
/**
* 转发带连接级重试。
*
* <p>对端实例在部署窗口(原地换 JAR + 两节点滚动重启)内会有几秒的 Connection refused。
* 连接都没建立起来说明请求没到达对端,此时重放是安全的;而读超时不重试——对端可能
* 已经在处理,盲目重放会造成重复提交。线上由此丢过用户提交的结果。
*/
private ResponseEntity<byte[]> forwardWithConnectRetry(HttpMethod method, String url,
HttpHeaders headers, byte[] body,
TaskOwnerMismatchException ex) {
RuntimeException lastError = null;
for (int attempt = 1; attempt <= CONNECT_RETRY_TIMES; attempt++) {
try {
return restClient().method(method) return restClient().method(method)
.uri(url) .uri(url)
.headers(target -> target.addAll(headers)) .headers(target -> target.addAll(headers))
.body(body) .body(body)
.retrieve() .retrieve()
.toEntity(byte[].class); .toEntity(byte[].class);
} catch (ResourceAccessException accessError) {
if (!isConnectFailure(accessError)) {
throw accessError;
}
lastError = accessError;
log.warn("[instance-routing] 转发连接失败,第 {}/{} 次 url={} taskId={} 原因={}",
attempt, CONNECT_RETRY_TIMES, url, ex.getTaskId(), accessError.getMessage());
if (attempt < CONNECT_RETRY_TIMES) {
sleepQuietly(CONNECT_RETRY_BACKOFF_MILLIS * attempt);
}
}
}
throw lastError;
}
private static boolean isConnectFailure(Throwable error) {
Throwable cursor = error;
while (cursor != null) {
if (cursor instanceof ConnectException) {
return true;
}
cursor = cursor.getCause();
}
return false;
}
private static void sleepQuietly(long millis) {
try {
Thread.sleep(millis);
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
}
} }
private String resolveUrl(TaskOwnerMismatchException ex, String path) { private String resolveUrl(TaskOwnerMismatchException ex, String path) {
@@ -0,0 +1,70 @@
package com.nanri.aiimage.common.util;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.function.Function;
/**
* 有界 LRU 缓存:容量超限时自动淘汰最久未使用的条目。
*
* <p>用于按「外部端点」缓存长生命周期资源(HttpClient / RestClient)。这类资源各自持有
* 连接池与 selector 线程,无界累积会持续泄漏线程与内存:代理端点每次提取往往是新的
* IP:port(jikip 提取),无上限的缓存只增不减。
*
* <p>淘汰时只从缓存移除引用,不做显式关闭:JDK 的 HttpClientImpl 注册了 Cleaner
* 对象不可达后由 GC 回收并关闭其 selector 线程;显式关闭反而可能打断仍在途的请求。
*/
public final class BoundedLruCache<K, V> {
/** 默认容量:代理端点数量级远小于此,足够覆盖热点端点又不至于累积。 */
public static final int DEFAULT_MAX_SIZE = 64;
private final int maxSize;
private final Map<K, V> store;
public BoundedLruCache() {
this(DEFAULT_MAX_SIZE);
}
public BoundedLruCache(int maxSize) {
this.maxSize = Math.max(1, maxSize);
// accessOrder=true 使 get 也刷新顺序(真正的 LRU);synchronizedMap 保证其线程安全
this.store = Collections.synchronizedMap(new LinkedHashMap<>(16, 0.75f, true) {
@Override
protected boolean removeEldestEntry(Map.Entry<K, V> eldest) {
return size() > BoundedLruCache.this.maxSize;
}
});
}
/**
* 取缓存值,缺失时用 loader 计算并放入。
*
* <p>与 {@code ConcurrentHashMap.computeIfAbsent} 不同,此处不保证 loader 的原子性:
* 并发首次访问同一 key 时可能各自构造一次,随后其中一个覆盖另一个。对
* HttpClient/RestClient 这类构造廉价且幂等的资源可接受,换来的是锁粒度更小。
*/
public V computeIfAbsent(K key, Function<K, V> loader) {
V existing = store.get(key);
if (existing != null) {
return existing;
}
V created = loader.apply(key);
store.put(key, created);
return created;
}
public int size() {
return store.size();
}
/** 当前容量上限,供日志与测试断言使用。 */
public int maxSize() {
return maxSize;
}
public void clear() {
store.clear();
}
}
@@ -56,6 +56,13 @@ public final class ExcelStreamReader {
default void onHeader(String sheetName, Integer sheetNo, Map<Integer, String> headerMap) throws Exception { default void onHeader(String sheetName, Integer sheetNo, Map<Integer, String> headerMap) throws Exception {
} }
/**
* 表头之后回调本次 sheet 的近似总行数(EasyExcel 基于 sheet 尺寸,可能为 null)。
* 流式解析无法在读完前得到精确行数,需要展示进度/做前置上限校验的调用方可用它近似。
*/
default void onSheetTotal(String sheetName, Integer sheetNo, Integer approximateTotalRows) throws Exception {
}
void onRow(String sheetName, Integer sheetNo, int rowIndex, Map<Integer, String> headerMap, Map<Integer, String> rowMap) throws Exception; void onRow(String sheetName, Integer sheetNo, int rowIndex, Map<Integer, String> headerMap, Map<Integer, String> rowMap) throws Exception;
} }
@@ -77,6 +84,7 @@ public final class ExcelStreamReader {
currentHeaderMap = normalizedHeadMap; currentHeaderMap = normalizedHeadMap;
try { try {
handler.onHeader(sheetName(context), sheetNo(context), currentHeaderMap); handler.onHeader(sheetName(context), sheetNo(context), currentHeaderMap);
handler.onSheetTotal(sheetName(context), sheetNo(context), approximateTotalRows(context));
} catch (RuntimeException ex) { } catch (RuntimeException ex) {
throw ex; throw ex;
} catch (Exception ex) { } catch (Exception ex) {
@@ -109,6 +117,15 @@ public final class ExcelStreamReader {
public void doAfterAllAnalysed(AnalysisContext context) { public void doAfterAllAnalysed(AnalysisContext context) {
} }
private Integer approximateTotalRows(AnalysisContext context) {
try {
return context.readSheetHolder() == null ? null
: context.readSheetHolder().getApproximateTotalRowNumber();
} catch (Exception ex) {
return null;
}
}
private String sheetName(AnalysisContext context) { private String sheetName(AnalysisContext context) {
return context.readSheetHolder() == null ? "" : context.readSheetHolder().getSheetName(); return context.readSheetHolder() == null ? "" : context.readSheetHolder().getSheetName();
} }
@@ -0,0 +1,138 @@
package com.nanri.aiimage.common.util;
import lombok.extern.slf4j.Slf4j;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.nio.file.StandardOpenOption;
import java.util.ArrayList;
import java.util.Collection;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Set;
/**
* 待删对象队列的本地落盘日志:进程重启后仍能恢复「待删除对象」清单。
*
* <p>使用场景:对象存储删除失败后的补偿队列原本只在节点内存里,重启即丢,
* 对应对象会一直残留在桶里直到生命周期规则过期。这里用一行一个 objectKey 的
* 追加日志做最小持久化——入队追加、队列收敛后整体重写、启动时回放。
*
* <p>并发:所有方法内部同步,调用方无需额外加锁;文件损坏/读写异常只记日志不抛出,
* 保证补偿链路本身不会因为落盘失败而中断业务。
*
* <p>上限:{@code maxEntries} 用于防止日志在异常堆积时无界增长(超出后丢弃最早的记录,
* 与内存队列的容量准入语义一致——丢的是「待删对象」,最坏结果是对象残留)。
*/
@Slf4j
public final class PendingDeleteJournal {
private final Path path;
private final int maxEntries;
private final Object lock = new Object();
public PendingDeleteJournal(Path path, int maxEntries) {
this.path = path;
this.maxEntries = Math.max(1, maxEntries);
}
public Path getPath() {
return path;
}
/** 追加一条待删对象(重复追加由回放时的 Set 语义去重)。 */
public void record(String objectKey) {
if (objectKey == null || objectKey.isBlank()) {
return;
}
synchronized (lock) {
try {
Path parent = path.getParent();
if (parent != null) {
Files.createDirectories(parent);
}
Files.writeString(path, sanitize(objectKey) + System.lineSeparator(),
StandardCharsets.UTF_8, StandardOpenOption.CREATE, StandardOpenOption.APPEND);
} catch (Exception ex) {
log.warn("[delete-journal] 追加待删对象失败 path={} msg={}", path, ex.getMessage());
}
}
}
/** 启动回放:返回日志中的待删对象(按首次出现顺序去重)。 */
public List<String> readAll() {
synchronized (lock) {
if (!Files.isRegularFile(path)) {
return List.of();
}
List<String> lines;
try {
lines = Files.readAllLines(path, StandardCharsets.UTF_8);
} catch (Exception ex) {
log.warn("[delete-journal] 读取待删对象日志失败 path={} msg={}", path, ex.getMessage());
return List.of();
}
Set<String> unique = new LinkedHashSet<>();
for (String line : lines) {
if (line == null) {
continue;
}
String key = line.trim();
if (!key.isEmpty()) {
unique.add(key);
}
}
return new ArrayList<>(unique);
}
}
/** 队列收敛(成功删除/引用仍在)后整体重写为剩余的待删对象;剩余为空则删除日志。 */
public void rewrite(Collection<String> remaining) {
synchronized (lock) {
Set<String> keep = new LinkedHashSet<>();
if (remaining != null) {
for (String key : remaining) {
if (key != null && !key.isBlank()) {
keep.add(sanitize(key));
if (keep.size() >= maxEntries) {
break;
}
}
}
}
try {
if (keep.isEmpty()) {
Files.deleteIfExists(path);
return;
}
Path parent = path.getParent();
if (parent != null) {
Files.createDirectories(parent);
}
Path tmp = path.resolveSibling(path.getFileName() + ".tmp");
StringBuilder content = new StringBuilder();
for (String key : keep) {
content.append(key).append(System.lineSeparator());
}
Files.writeString(tmp, content.toString(), StandardCharsets.UTF_8,
StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING);
try {
Files.move(tmp, path, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE);
} catch (IOException atomicUnsupported) {
// 少数文件系统不支持 ATOMIC_MOVE,退化为普通替换
Files.move(tmp, path, StandardCopyOption.REPLACE_EXISTING);
}
} catch (Exception ex) {
log.warn("[delete-journal] 重写待删对象日志失败 path={} msg={}", path, ex.getMessage());
}
}
}
/** 单行一条记录:去掉换行避免破坏行结构。 */
private static String sanitize(String objectKey) {
return objectKey.replace('\n', ' ').replace('\r', ' ').trim();
}
}
@@ -0,0 +1,52 @@
package com.nanri.aiimage.common.util;
import java.net.URI;
/**
* 敏感串脱敏工具(2026-09 全维度审查后收口)。
*
* <p>背景:代理提取链接形态为 {@code http://user:pass@host:port},代码中曾有多处
* 直接把整串打进日志,导致用户代理账号密码长期留在应用日志与 docker logs 里。
*/
public final class SecretMasking {
private SecretMasking() {
}
/** 通用掩码:保留前 4 与后 4 字符;过短则整体掩掉。 */
public static String mask(String value) {
if (value == null || value.isBlank()) {
return "";
}
String text = value.trim();
if (text.length() <= 8) {
return "****";
}
return text.substring(0, 4) + "****" + text.substring(text.length() - 4);
}
/** 代理掩码:隐去账号密码,保留 scheme://host:port 便于运维核对。 */
public static String maskProxy(String value) {
if (value == null || value.isBlank()) {
return "";
}
try {
URI uri = URI.create(value.trim());
if (uri.getHost() == null || uri.getHost().isBlank()) {
return mask(value);
}
StringBuilder masked = new StringBuilder();
masked.append(uri.getScheme() == null ? "http" : uri.getScheme()).append("://");
if (uri.getUserInfo() != null && !uri.getUserInfo().isBlank()) {
masked.append("***@");
}
masked.append(uri.getHost());
if (uri.getPort() > 0) {
masked.append(':').append(uri.getPort());
}
return masked.toString();
} catch (Exception ex) {
return mask(value);
}
}
}
@@ -0,0 +1,47 @@
package com.nanri.aiimage.common.util;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
/**
* 有界线程池工厂(2026-09 全维度审查补)。
*
* <p>业务里多处使用 {@code Executors.newFixedThreadPool}:它内部是**无界**
* {@code LinkedBlockingQueue},任务堆积时永远不会触发拒绝策略,会把内存吃满
* (表现为 OOM,或整机因 GC 变慢导致所有任务一起劣化)。
* 统一改为有界队列 + {@code CallerRunsPolicy}:队列满时在提交线程执行,形成天然背压。
*/
public final class ThreadPools {
private ThreadPools() {
}
/** 默认队列容量:足以吸收突发,又不至于无界堆积。 */
public static final int DEFAULT_QUEUE_CAPACITY = 512;
/** 有界固定线程池(daemon 线程,空闲可回收)。 */
public static ExecutorService boundedFixed(String threadNamePrefix, int threads) {
return boundedFixed(threadNamePrefix, threads, DEFAULT_QUEUE_CAPACITY);
}
/** 有界固定线程池(显式队列容量)。 */
public static ExecutorService boundedFixed(String threadNamePrefix, int threads, int queueCapacity) {
ThreadPoolExecutor executor = new ThreadPoolExecutor(
Math.max(1, threads),
Math.max(1, threads),
// keepAliveTime 必须 > 0:下面开了 allowCoreThreadTimeOut
// 传 0 会让构造器直接抛 "Core threads must have nonzero keep alive times"
60L, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(Math.max(1, queueCapacity)),
runnable -> {
Thread thread = new Thread(runnable, threadNamePrefix);
thread.setDaemon(true);
return thread;
},
new ThreadPoolExecutor.CallerRunsPolicy());
executor.allowCoreThreadTimeOut(true);
return executor;
}
}
@@ -3,7 +3,8 @@ package com.nanri.aiimage.config;
import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.api.ApiResponse; import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.common.exception.BusinessException; import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.modules.admin.support.AdminAuthSupport; import com.nanri.aiimage.common.security.AdminAuthSupport;
import com.nanri.aiimage.common.security.DeviceSessionPolicy;
import jakarta.servlet.FilterChain; import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException; import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletRequest;
@@ -63,6 +64,24 @@ public class AdminApiGuardFilter extends OncePerRequestFilter {
private static final String[] USER_TOOL_PREFIXES = { private static final String[] USER_TOOL_PREFIXES = {
"/api/collect-data", "/api/collect-data",
"/api/price-track", "/api/price-track",
// 2026-09 全维度审查补:以下前缀此前完全在守卫范围之外。
// /api/files 匿名可上传(2GB/次,可耗尽临时盘);/api/digital-human 匿名可发布/删除版本;
// /api/brand 的 fileUrl 曾可直接请求任意地址(/api/image-video 与 /api/task-file-jobs
// 客户端零调用,已移入下方无条件名单)。
"/api/files",
"/api/digital-human",
"/api/brand",
"/api/appearance-patent",
"/api/similar-asin",
"/api/query-asin",
"/api/patrol-delete",
"/api/product-risk-resolve",
"/api/shop-match",
"/api/shop-data-crawl",
"/api/withdraw",
// /api/tasks/{taskId}/interrupted 仅凭 taskId 即可把 RUNNING 任务置为 FAILED
// 匿名遍历 taskId 就能批量打断线上任务
"/api/tasks",
}; };
/** /**
@@ -72,6 +91,16 @@ public class AdminApiGuardFilter extends OncePerRequestFilter {
private static final String[] SELF_SERVICE_PREFIXES = { private static final String[] SELF_SERVICE_PREFIXES = {
"/api/user-secrets", "/api/user-secrets",
"/api/notifications", "/api/notifications",
// 2026-09-14:这三组前缀**桌面端 Python 侧零调用**(实测),只有带 JWT 的网页端在用,
// 因此不必等客户端铺开即可无条件收紧(其余用户态前缀仍在 user-tool-guard-enabled 开关后面)
"/api/image-video",
"/api/task-file-jobs",
// 其全部接口已在 controller 内 requireAdmin(含可换取员工店铺登录令牌的 /shops/open),
// 纳入守卫是"鉴权失败返回统一 401 体"的第二层
"/api/ziniao",
// 2026-09 全维度审查补:内部端点此前仅靠 controller 自校验令牌,纳入守卫后
// 不带令牌的请求直接 401(带可信令牌的仍由 doFilterInternal 放行)
"/api/internal",
}; };
private final AdminAuthSupport adminAuthSupport; private final AdminAuthSupport adminAuthSupport;
@@ -125,7 +154,14 @@ public class AdminApiGuardFilter extends OncePerRequestFilter {
ApiResponse<Void> body = ex.getCode() == null ApiResponse<Void> body = ex.getCode() == null
? ApiResponse.fail(ex.getMessage()) ? ApiResponse.fail(ex.getMessage())
: ApiResponse.fail(ex.getCode(), ex.getMessage()); : ApiResponse.fail(ex.getCode(), ex.getMessage());
// 401(登录已过期)与被顶下线是前端定时轮询(/api/notifications/summary、/api/user-secrets 等)
// 的常态:线上单节点一天近 3000 条,会把真实业务错误淹没。与 GlobalExceptionHandler
// 的 isRoutineAuthNoise 同一口径降为 debug。
if (isRoutineAuthNoise(ex.getCode())) {
log.debug("[admin-guard] {} {} rejected: {}", request.getMethod(), request.getRequestURI(), ex.getMessage());
} else {
log.warn("[admin-guard] {} {} rejected: {}", request.getMethod(), request.getRequestURI(), ex.getMessage()); log.warn("[admin-guard] {} {} rejected: {}", request.getMethod(), request.getRequestURI(), ex.getMessage());
}
response.setStatus(HttpServletResponse.SC_OK); response.setStatus(HttpServletResponse.SC_OK);
response.setContentType("application/json;charset=UTF-8"); response.setContentType("application/json;charset=UTF-8");
response.getWriter().write(objectMapper.writeValueAsString(body)); response.getWriter().write(objectMapper.writeValueAsString(body));
@@ -140,6 +176,12 @@ public class AdminApiGuardFilter extends OncePerRequestFilter {
chain.doFilter(request, response); chain.doFilter(request, response);
} }
/** 登录态过期 / 被其他设备顶下线:前端轮询的常态噪声,不占 WARN。 */
private static boolean isRoutineAuthNoise(Integer code) {
return Integer.valueOf(401).equals(code)
|| Integer.valueOf(DeviceSessionPolicy.CODE_KICKED).equals(code);
}
/** 命中受保护前缀(/api/admin、/debug、自助接口、用户态工具前缀及其子路径)才进入鉴权,其余请求直接放行。 */ /** 命中受保护前缀(/api/admin、/debug、自助接口、用户态工具前缀及其子路径)才进入鉴权,其余请求直接放行。 */
private boolean isGuarded(String uri) { private boolean isGuarded(String uri) {
if (matchesPrefix(uri, ADMIN_API_PREFIX) || matchesPrefix(uri, DEBUG_PREFIX)) { if (matchesPrefix(uri, ADMIN_API_PREFIX) || matchesPrefix(uri, DEBUG_PREFIX)) {
@@ -30,15 +30,24 @@ public class AppearancePatentProperties {
private int llmFirstAttemptReadTimeoutMillis = 60000; private int llmFirstAttemptReadTimeoutMillis = 60000;
private int llmBatchSize = 10; private int llmBatchSize = 10;
/** /**
* 批内行级并发数,默认等于批量大小 * 批内行级并发数。批次串行提交,每行串行发 2 个 LLM 请求,
* 故该值≈单任务对 LLM 网关的瞬时并发;默认与品牌检测同为 5,避免多任务并行时成倍放大。
*/ */
private int llmRowConcurrency = 10; private int llmRowConcurrency = 5;
/** /**
* 每行每个 LLM 请求的重试次数(含首次) * 每行每个 LLM 请求的重试次数(含首次)
*/ */
private int llmRetryTimes = 3; private int llmRetryTimes = 3;
private int staleTimeoutMinutes = 30; private int staleTimeoutMinutes = 30;
private String staleFinalizeCron = "0 */2 * * * *"; private String staleFinalizeCron = "0 */2 * * * *";
/**
* 二次判死线(心跳正常但连续 N 分钟无结果分片上报)阈值,分钟;<=0 关闭本线。
* 既有判定以 Redis heartbeat stale 为主信号,但心跳随 Python HTTP 心跳每分钟刷新——
* 主线程卡死时心跳线程照发,任务永远不 stale(与生产 28131 同型缺口)。
* 本线改看 biz_task_scope_state.last_chunk_at(仅分片上传时刷新)。
*/
private int noResultUploadTimeoutMinutes = 180;
/** /**
* 末尾不足一批的数据等待该时长后强制提交检测。 * 末尾不足一批的数据等待该时长后强制提交检测。
* Python 回传超时后会直接封口;这个配置处理 Python 慢回传但仍未超时的零头批次。 * Python 回传超时后会直接封口;这个配置处理 Python 慢回传但仍未超时的零头批次。
@@ -10,10 +10,27 @@ public class BrandCheckProperties {
private String path = "/brand_check"; private String path = "/brand_check";
private String token = ""; private String token = "";
private String defaultStrategy = "Terms"; private String defaultStrategy = "Terms";
/** 单品牌商标查询的重试次数(含首次),连续失败超过该次数才判定为查询失败。 */ /**
private int retryTimes = 3; * 单品牌商标查询的重试次数(含首次),连续失败超过该次数才判定为查询失败。
/** 每次查询失败后到下一次重试前的等待毫秒数。 */ * 原为 3:16890 偶发限流几秒内即恢复,3 次(前两次间隔各 1s)恢复不了就把结论
* 写成「查询失败」,对客户是硬伤;2026-09-14 与客户端品牌一致提到 10 次。
*/
private int retryTimes = 10;
/** 每次查询失败后到下一次重试前的等待毫秒数(基准值,按重试轮次递增)。 */
private int retryIntervalMillis = 1000; private int retryIntervalMillis = 1000;
/**
* 单次重试等待的上限毫秒数。等待按 retryIntervalMillis × 第几次重试 递增后封顶,
* 10 次尝试合计约 45s,既能覆盖限流窗口又不会让单个品牌长时间占住查询线程。
*/
private int retryMaxIntervalMillis = 10000;
/**
* 一次批量检查的总耗时上限(毫秒):整批用尽后不再重试,剩余品牌按「查询失败」收尾。
* 上限的意义不是省时间,而是给「分片回传」这类同步调用方一个时延上界——上游卡死时
* 单品牌 10 次重试曾把一次回传拖到 103.5 秒(taskId 28599),客户端重试预算耗尽后
* 中止了整个采集。首轮请求始终执行,故上游只是略慢时不会误降级。
* 设为 0 或负数表示不限制。
*/
private int totalTimeoutMillis = 90000;
private int connectTimeoutMillis = 10000; private int connectTimeoutMillis = 10000;
private int readTimeoutMillis = 60000; private int readTimeoutMillis = 60000;
} }
@@ -10,4 +10,11 @@ public class BrandProgressProperties {
private long failedTtlHours = 2; private long failedTtlHours = 2;
private long heartbeatTimeoutMinutes = 15; private long heartbeatTimeoutMinutes = 15;
private String staleCheckCron = "0 */2 * * * *"; private String staleCheckCron = "0 */2 * * * *";
/**
* 二次判死线(心跳正常但连续 N 分钟无结果上报)阈值,分钟;<=0 关闭本线。
* 既有心跳线以 updated_at/last_heartbeat_at 陈旧为判据,而前端心跳会持续刷新它们——
* 主线程卡死时心跳线程照发,任务永远命不中。本线改看结果上报时写入的 last_result_at。
*/
private long noResultUploadTimeoutMinutes = 180;
} }
@@ -45,6 +45,19 @@ public class DeleteBrandProgressProperties {
*/ */
private long withdrawStaleTimeoutMinutes = 30; private long withdrawStaleTimeoutMinutes = 30;
/**
* 「心跳正常但连续 N 分钟无结果分片上报」的二次判死阈值(分钟),默认 3 小时。
*
* <p>既有各模块心跳线的候选条件是 updated_at 陈旧,而客户端心跳会持续刷新 updated_at——
* 主线程卡死时心跳线程照发,任务永远命不中(生产 28131 卡死 12h+ 仍 RUNNING)。
* 本线改用 biz_task_scope_state.last_chunk_at(只随结果分片上报刷新)作判据,
* 从未上报(无 scope 行或全 NULL)的任务跳过,避免误杀首批较慢的正常任务。
*/
private long noResultUploadTimeoutMinutes = 180;
/** 二次判死线开关:false = 整段不扫描(观察期与回滚用,改环境变量即生效)。 */
private boolean noResultUploadCheckEnabled = true;
/** /**
* 组装结果文件时允许缺失的最大分片数:缺失不超过该值时降级组装(缺失行状态写"未回传"), * 组装结果文件时允许缺失的最大分片数:缺失不超过该值时降级组装(缺失行状态写"未回传"),
* 保证任务不因少量分片丢失整体白跑;超出时任务失败,但仍会在重试耗尽后尝试 * 保证任务不因少量分片丢失整体白跑;超出时任务失败,但仍会在重试耗尽后尝试
@@ -0,0 +1,38 @@
package com.nanri.aiimage.config;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
/**
* 设备日志对象存储配置:指向主机B 独立部署的 MinIO 实例(非业务 MinIO)。
*
* <p>日志体积大、只保留 7 天,独立实例便于单独设生命周期规则与容量管理,
* 不挤占业务桶(nanri-ai-images 等)。endpoint 为空时上报接口直接失败,
* 不做静默回退(避免日志悄悄落到其他存储上而无人知情)。
*/
@Data
@ConfigurationProperties(prefix = "aiimage.device-log-oss")
public class DeviceLogOssProperties {
private String endpoint;
private String accessKeyId;
private String accessKeySecret;
private String bucket;
/**
* 日志保留天数:查询侧按此过滤(早于今天的 N-1 天不展示),
* 对象过期由 MinIO 桶生命周期规则在部署时同步设置(两侧口径保持一致)。
*/
private Integer retentionDays;
public boolean configured() {
return endpoint != null && !endpoint.isBlank()
&& accessKeyId != null && !accessKeyId.isBlank()
&& accessKeySecret != null && !accessKeySecret.isBlank()
&& bucket != null && !bucket.isBlank();
}
public int retentionDaysOrDefault() {
return retentionDays == null || retentionDays < 1 ? 7 : retentionDays;
}
}
@@ -1,17 +1,21 @@
package com.nanri.aiimage.config; package com.nanri.aiimage.config;
import com.nanri.aiimage.common.util.BoundedLruCache;
import lombok.extern.slf4j.Slf4j;
import org.springframework.http.client.ClientHttpRequestFactory; import org.springframework.http.client.ClientHttpRequestFactory;
import org.springframework.http.client.JdkClientHttpRequestFactory; import org.springframework.http.client.JdkClientHttpRequestFactory;
import java.io.IOException;
import java.io.InputStream;
import java.net.Authenticator; import java.net.Authenticator;
import java.net.InetSocketAddress; import java.net.InetSocketAddress;
import java.net.PasswordAuthentication; import java.net.PasswordAuthentication;
import java.net.ProxySelector; import java.net.ProxySelector;
import java.net.URI; import java.net.URI;
import java.net.http.HttpClient; import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration; import java.time.Duration;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
/** /**
* Task 77:外部 HTTP 客户端统一连接复用池。 * Task 77:外部 HTTP 客户端统一连接复用池。
@@ -20,6 +24,7 @@ import java.util.concurrent.ConcurrentHashMap;
* 每次请求都重新建连。各客户端按自身超时创建独立的 * 每次请求都重新建连。各客户端按自身超时创建独立的
* JdkClientHttpRequestFactory(共享底层连接池),RestClient 单例懒加载。 * JdkClientHttpRequestFactory(共享底层连接池),RestClient 单例懒加载。
*/ */
@Slf4j
public class HttpClientPool { public class HttpClientPool {
private static volatile HttpClient sharedHttpClient; private static volatile HttpClient sharedHttpClient;
@@ -55,6 +60,38 @@ public class HttpClientPool {
} }
} }
/**
* 打开远程文件流(带超时),调用方负责关闭返回的流。
*
* <p>替代裸 {@code URI.create(url).toURL().openStream()}:后者走 JVM 默认超时(0 = 无限),
* 上游半开连接或挂起时会把 Tomcat 工作线程无限占用(管理端批量打包可同时挂多个)。
* 返回的流是流式的,适用于「服务端代理下载 OSS 文件转发给浏览器」这类不落盘场景。
*
* @param url 远程地址
* @param timeout 等待响应超时(连接建立 + 响应头);非法值钳制到 1 秒
* @throws IOException 非 2xx 响应或网络异常
* @throws InterruptedException 线程被中断
*/
public static InputStream openStreamWithTimeout(String url, Duration timeout) throws IOException, InterruptedException {
Duration effective = (timeout == null || timeout.isZero() || timeout.isNegative())
? Duration.ofSeconds(1)
: timeout;
HttpRequest request = HttpRequest.newBuilder(URI.create(url))
.timeout(effective)
.GET()
.build();
HttpResponse<InputStream> response = sharedHttpClient().send(request, HttpResponse.BodyHandlers.ofInputStream());
if (response.statusCode() / 100 != 2) {
try {
response.body().close();
} catch (Exception ignored) {
// 关闭失败不影响错误上报
}
throw new IOException("远程文件返回非 2xx: HTTP " + response.statusCode());
}
return response.body();
}
/** 按 readTimeout(毫秒)创建共享连接池工厂;非法值钳制到最小正数。 */ /** 按 readTimeout(毫秒)创建共享连接池工厂;非法值钳制到最小正数。 */
public static ClientHttpRequestFactory requestFactory(int readTimeoutMillis) { public static ClientHttpRequestFactory requestFactory(int readTimeoutMillis) {
return requestFactory(readTimeoutMillis, null); return requestFactory(readTimeoutMillis, null);
@@ -67,8 +104,12 @@ public class HttpClientPool {
public static ClientHttpRequestFactory requestFactory(int readTimeoutMillis, String proxyUrl) { public static ClientHttpRequestFactory requestFactory(int readTimeoutMillis, String proxyUrl) {
long safeReadTimeout = Math.max(1L, readTimeoutMillis); long safeReadTimeout = Math.max(1L, readTimeoutMillis);
long callTimeout = configuredCallTimeoutMillis; long callTimeout = configuredCallTimeoutMillis;
if (callTimeout > 0L) { if (callTimeout > 0L && safeReadTimeout > callTimeout) {
safeReadTimeout = Math.min(safeReadTimeout, callTimeout); // 读超时以调用方显式值为准,不再被全局 call-timeout 截断:
// LLM 长思考配的是 180sllm-read-timeout-millis),曾被静默压到 90s
// 导致请求在 90s 被掐断 → 上层重试 → 付费网关二次计费(2026-09-15 修复)。
// 各调用方的超时已由各自的 HttpConfigResolver 钳制,此处不再二次收敛。
log.debug("读超时 {}ms 超过全局 call-timeout {}ms,按调用方显式值生效", safeReadTimeout, callTimeout);
} }
JdkClientHttpRequestFactory factory = JdkClientHttpRequestFactory factory =
new JdkClientHttpRequestFactory(httpClientFor(proxyUrl)); new JdkClientHttpRequestFactory(httpClientFor(proxyUrl));
@@ -130,5 +171,6 @@ public class HttpClientPool {
private record ProxyEndpoint(String host, int port, String userInfo) { private record ProxyEndpoint(String host, int port, String userInfo) {
} }
private static final Map<ProxyEndpoint, HttpClient> PROXY_CLIENTS = new ConcurrentHashMap<>(); private static final BoundedLruCache<ProxyEndpoint, HttpClient> PROXY_CLIENTS =
new BoundedLruCache<>(BoundedLruCache.DEFAULT_MAX_SIZE);
} }
@@ -1,5 +1,6 @@
package com.nanri.aiimage.config; package com.nanri.aiimage.config;
import com.nanri.aiimage.common.module.TaskModuleRegistry;
import lombok.Data; import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.boot.context.properties.ConfigurationProperties;
@@ -19,5 +20,6 @@ public class ModuleCleanupProperties {
private int batchSize = 500; private int batchSize = 500;
// SHOP_DATA_CRAWL keeps one per-shop daily workbook in its task service // SHOP_DATA_CRAWL keeps one per-shop daily workbook in its task service
// and must not be removed by the age-based sweep. // and must not be removed by the age-based sweep.
private List<String> moduleTypes = new ArrayList<>(List.of("DEDUPE", "SPLIT", "CONVERT", "DELETE_BRAND", "PRODUCT_RISK_RESOLVE", "PRICE_TRACK", "SHOP_MATCH", "PATROL_DELETE", "QUERY_ASIN", "WITHDRAW", "APPEARANCE_PATENT", "SIMILAR_ASIN", "COLLECT_DATA")); /** 参与按天清理的模块:取自模块注册表(G6)。 */
private List<String> moduleTypes = new ArrayList<>(TaskModuleRegistry.ageCleanupModuleTypes());
} }
@@ -41,6 +41,38 @@ public class NotificationProperties {
/** jikip 代理接口探测开关(复用余额查询接口判活,使用 user-secret 的 jikip 配置)。 */ /** jikip 代理接口探测开关(复用余额查询接口判活,使用 user-secret 的 jikip 配置)。 */
private boolean jikipProbeEnabled = true; private boolean jikipProbeEnabled = true;
/** 麦象(18960 任务调度)异常扫描开关:任务停滞/失败/队列积压 → 管理员通知。 */
private boolean maixiangScanEnabled = true;
/**
* 麦象后台接口令牌(18960 console token)。留空=跳过麦象异常扫描——
* 该令牌与「跟价任务 API 地址」(priceTrackApiUrl) 一起构成后台只读接口的访问凭据。
*/
private String maixiangConsoleToken = "";
/** 麦象批量任务停滞阈值(分钟):status=0/1 且超过该时长无更新视为卡住。 */
private int maixiangStuckMinutes = 60;
/** 麦象单任务滞留阈值(分钟):创建超时仍未完成(status=0/1)视为滞留/无人消费。 */
private int maixiangSingleStuckMinutes = 30;
/** 麦象任务失败告警阈值(条):近 30 分钟窗口内失败数达到该值才告警。 */
private int maixiangFailMinCount = 1;
/** 麦象队列积压阈值(条):task:queue 待处理数达到该值告警。 */
private int maixiangQueuePendingThreshold = 300;
/** 麦象队列积压阈值(条):task:processing 处理中数达到该值告警。 */
private int maixiangQueueProcessingThreshold = 100;
/** 已读通知保留天数(超期自动清理),默认 90 天。 */ /** 已读通知保留天数(超期自动清理),默认 90 天。 */
private int readRetentionDays = 90; private int readRetentionDays = 90;
/**
* 未读通知保留天数,默认 180 天(比已读长一倍)。
*
* <p>未读通知此前永不清理:不看铃铛的用户会无限累积。保留期给得更宽,
* 是因为未读意味着"用户可能还没看到",但也不能永远留着。
*/
private int unreadRetentionDays = 180;
} }
@@ -4,6 +4,6 @@ import org.springframework.boot.context.properties.EnableConfigurationProperties
import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Configuration;
@Configuration @Configuration
@EnableConfigurationProperties({OssProperties.class, TransientStorageProperties.class, StorageProperties.class, BrandProgressProperties.class, DeleteBrandProgressProperties.class, BrandCheckProperties.class, ZiniaoProperties.class, ModuleCleanupProperties.class, TaskPressureProperties.class, TaskImageCacheCleanupProperties.class, AppearancePatentProperties.class, SimilarAsinProperties.class, ImageVideoProperties.class, InstanceRoutingProperties.class, CapacityPlanProperties.class, UserSecretProperties.class, NotificationProperties.class}) @EnableConfigurationProperties({OssProperties.class, TransientStorageProperties.class, StorageProperties.class, BrandProgressProperties.class, DeleteBrandProgressProperties.class, BrandCheckProperties.class, ZiniaoProperties.class, ModuleCleanupProperties.class, TaskPressureProperties.class, TaskImageCacheCleanupProperties.class, AppearancePatentProperties.class, SimilarAsinProperties.class, ImageVideoProperties.class, InstanceRoutingProperties.class, CapacityPlanProperties.class, UserSecretProperties.class, NotificationProperties.class, DeviceLogOssProperties.class})
public class PropertiesConfig { public class PropertiesConfig {
} }
@@ -20,8 +20,15 @@ public class SchedulingConfig {
private static final ZoneId BUSINESS_ZONE = ZoneId.of("Asia/Shanghai"); private static final ZoneId BUSINESS_ZONE = ZoneId.of("Asia/Shanghai");
/**
* 调度线程池:承载全站 30+ 个 @Scheduled(含 imagevideo 1s 派发、5s 轮询、结果文件 worker 15s 等高频任务)。
*
* <p>此前默认 4 线程:任一慢任务(如结果文件组装被内联执行时)都会把兜底类任务
* StaleTaskRepair 心跳判死、陈旧扫描、历史清理)顺延,而兜底任务被顺延会直接放大线上故障面。
* 提到 16 并保持可配(aiimage.scheduling.pool-size)。
*/
@Bean @Bean
public TaskScheduler taskScheduler(@Value("${aiimage.scheduling.pool-size:4}") int poolSize) { public TaskScheduler taskScheduler(@Value("${aiimage.scheduling.pool-size:16}") int poolSize) {
ThreadPoolTaskScheduler scheduler = new ThreadPoolTaskScheduler(); ThreadPoolTaskScheduler scheduler = new ThreadPoolTaskScheduler();
scheduler.setPoolSize(Math.max(1, poolSize)); scheduler.setPoolSize(Math.max(1, poolSize));
scheduler.setThreadNamePrefix("aiimage-scheduling-"); scheduler.setThreadNamePrefix("aiimage-scheduling-");
@@ -23,6 +23,14 @@ public class SimilarAsinProperties {
private int staleTimeoutMinutes = 30; private int staleTimeoutMinutes = 30;
private String staleFinalizeCron = "0 */2 * * * *"; private String staleFinalizeCron = "0 */2 * * * *";
/**
* 二次判死线(心跳正常但连续 N 分钟无结果分片上报)阈值,分钟;<=0 关闭本线。
* 既有判定以 Redis heartbeat stale 为主信号,但心跳随 Python HTTP 心跳每分钟刷新——
* 主线程卡死时心跳线程照发,任务永远不 stale(与生产 28131 同型缺口)。
* 本线改看 biz_task_scope_state.last_chunk_at(仅分片上传时刷新)。
*/
private int noResultUploadTimeoutMinutes = 180;
/** /**
* 末尾零头 batch 的强制 flush 阈值(分钟):当不足 llmBatchSize 的零头 row * 末尾零头 batch 的强制 flush 阈值(分钟):当不足 llmBatchSize 的零头 row
* 长时间挂着(Python 慢回传)时触发提交。 * 长时间挂着(Python 慢回传)时触发提交。
@@ -33,6 +33,7 @@ import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.beans.factory.ObjectProvider; import org.springframework.beans.factory.ObjectProvider;
import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value; import org.springframework.beans.factory.annotation.Value;
import com.nanri.aiimage.common.module.TaskModuleRegistry;
import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Configuration;
import org.springframework.core.task.TaskExecutor; import org.springframework.core.task.TaskExecutor;
@@ -52,11 +53,11 @@ import java.util.concurrent.Semaphore;
public class TaskFileJobConfig { public class TaskFileJobConfig {
/** 结果文件 Job 支持的全部 moduleType(启动校验枚举源,见 ResultFileJobHandlerRegistry.validateCoverage */ /** 结果文件 Job 支持的全部 moduleType(启动校验枚举源,见 ResultFileJobHandlerRegistry.validateCoverage */
public static final Set<String> RESULT_FILE_JOB_MODULE_TYPES = Set.of( /**
"SHOP_MATCH", "PRICE_TRACK", "PRODUCT_RISK_RESOLVE", * 结果文件 Job 支持的 moduleType:取自模块注册表(G6),
"PUBLISH", "QUERY_ASIN", "SHOP_DATA_CRAWL", "WITHDRAW", * 与 {@code ResultFileJobHandlerRegistry.validateCoverage} 的启动自检配合使用。
"PATROL_DELETE", "APPEARANCE_PATENT", "SIMILAR_ASIN", */
"DELETE_BRAND", "BRAND", "COLLECT_DATA"); public static final Set<String> RESULT_FILE_JOB_MODULE_TYPES = TaskModuleRegistry.resultFileJobModuleTypes();
@Bean("taskFileJobDispatchExecutor") @Bean("taskFileJobDispatchExecutor")
public TaskExecutor taskFileJobDispatchExecutor( public TaskExecutor taskFileJobDispatchExecutor(
@@ -29,7 +29,14 @@ public class TransientStorageProperties {
*/ */
private long maxTotalConcurrentOperations = 0; private long maxTotalConcurrentOperations = 0;
private long acquirePermitTimeoutMillis = 2000; private long acquirePermitTimeoutMillis = 2000;
private long baseRetryDelayMillis = 500; /**
* 首次重试前的基础退避。
*
* <p>线上高频的重试诱因是 `unexpected end of stream`——那是**立即失败**(连接被 RustFS
* 重置后 OkHttp 读响应即报错),不是等超时,所以 500ms 基本是白等:每天上千次累计十几分钟。
* 降到 200ms 保留退避语义(真遇到服务端过载仍会退让),又不至于让用户等太久。
*/
private long baseRetryDelayMillis = 200;
private long maxRetryDelayMillis = 5000; private long maxRetryDelayMillis = 5000;
private long retryJitterMillis = 250; private long retryJitterMillis = 250;
private long failureWindowSeconds = 60; private long failureWindowSeconds = 60;
@@ -37,7 +44,19 @@ public class TransientStorageProperties {
private long failureCooldownMillis = 10000; private long failureCooldownMillis = 10000;
private int dispatcherMaxRequests = 56; private int dispatcherMaxRequests = 56;
private int dispatcherMaxRequestsPerHost = 56; private int dispatcherMaxRequestsPerHost = 56;
/**
* 空闲连接保留数。
*
* <p>2026-09-17 曾试过设 0(彻底不复用)来验证"unexpected end of stream 是复用死连接导致的"
* 这一假设——**实测照旧失败**(新容器起来后第一次请求就中招)。至此已排除公网链路、
* keepAlive 过长、连接复用三项;用 mc 并发压 200 个小对象也全部成功,说明服务端没问题。
* 剩余方向指向 MinIO Java SDK / OkHttp 与 RustFS 的协议细节,故恢复默认的连接复用。
*/
private int connectionPoolMaxIdle = 5; private int connectionPoolMaxIdle = 5;
/**
* 空闲连接在池里的保留时长。曾由 300000 调到 30000 试图减少 unexpected end of stream
* 实测无改善(该现象与连接复用无关,见 {@link #connectionPoolMaxIdle} 的排查记录),故恢复原值。
*/
private long connectionPoolKeepAliveMillis = 300000; private long connectionPoolKeepAliveMillis = 300000;
private long warnPayloadBytes = 5L * 1024 * 1024; private long warnPayloadBytes = 5L * 1024 * 1024;
private long maxPayloadBytes = 50L * 1024 * 1024; private long maxPayloadBytes = 50L * 1024 * 1024;
@@ -48,6 +67,12 @@ public class TransientStorageProperties {
*/ */
private long maxDecompressedPayloadBytes = 100L * 1024 * 1024; private long maxDecompressedPayloadBytes = 100L * 1024 * 1024;
private boolean fallbackToLocalOnOversize = true; private boolean fallbackToLocalOnOversize = true;
/**
* 上传失败(已重试+熔断)时是否回落到本机磁盘。默认 false:
* 多实例/容器化部署下 local 指针只有写入它的那个实例能读,宁可让本次写入失败,
* 也不要把跨节点不可读的脏指针交给调用方;仅单机部署才应打开。
*/
private boolean fallbackToLocalOnError = false;
private boolean deleteRetryEnabled = true; private boolean deleteRetryEnabled = true;
private String deleteRetryCron = "0 */5 * * * *"; private String deleteRetryCron = "0 */5 * * * *";
private int deleteRetryQueueCapacity = 10000; private int deleteRetryQueueCapacity = 10000;
@@ -22,6 +22,13 @@ public class UserSecretProperties {
/** 单轮巡检时间预算(分钟),超时中断本轮。 */ /** 单轮巡检时间预算(分钟),超时中断本轮。 */
private int checkBudgetMinutes = 20; private int checkBudgetMinutes = 20;
/**
* 检测请求使用的 LLM 模型:独立于业务任务模型(业务用 gemini-3.8-flash 等),
* 选便宜的可用模型,只验证密钥有效性与链路连通,降低每次检测与巡检的成本。
* 用 lite 而非 mini:mini 在中继分组下无可用渠道(503 model_not_found),实测 lite 可路由。
*/
private String checkModel = "doubao-seed-2-0-lite-260215";
/** /**
* 检测出口代理提取链接(选配):配置后检测请求优先经该代理 IP 发出, * 检测出口代理提取链接(选配):配置后检测请求优先经该代理 IP 发出,
* 代理不可用时自动回退直连;留空则全部直连。 * 代理不可用时自动回退直连;留空则全部直连。
@@ -1,8 +1,8 @@
package com.nanri.aiimage.modules.admin.controller; package com.nanri.aiimage.modules.admin.controller;
import com.nanri.aiimage.common.api.ApiResponse; import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.modules.admin.support.AdminAuthSupport; import com.nanri.aiimage.common.security.AdminAuthSupport;
import com.nanri.aiimage.modules.permission.model.entity.AdminUserEntity; import com.nanri.aiimage.common.model.entity.AdminUserEntity;
import com.nanri.aiimage.modules.permission.model.vo.PermissionMenuItemVo; import com.nanri.aiimage.modules.permission.model.vo.PermissionMenuItemVo;
import com.nanri.aiimage.modules.permission.service.PermissionMenuService; import com.nanri.aiimage.modules.permission.service.PermissionMenuService;
import com.nanri.aiimage.modules.permission.service.support.AdminMenuTreeBuilder; import com.nanri.aiimage.modules.permission.service.support.AdminMenuTreeBuilder;
@@ -76,8 +76,10 @@ public class AdminConsoleController {
@Operation(summary = "当前登录管理员的可见后台菜单树") @Operation(summary = "当前登录管理员的可见后台菜单树")
public ApiResponse<Map<String, Object>> currentUserMenus(HttpServletRequest request) { public ApiResponse<Map<String, Object>> currentUserMenus(HttpServletRequest request) {
AdminUserEntity operator = adminAuthSupport.requireAdminOrInternal(request); AdminUserEntity operator = adminAuthSupport.requireAdminOrInternal(request);
// 补全祖先分组:部分授权用户(只授权了子页面)也要看到「一级分组 + 子页面」层级,
// 与超管的菜单组织顺序一致;分组节点无页面路由,不构成权限扩展。
List<PermissionMenuItemVo> menus = permissionMenuService.getUserColumnPermissions( List<PermissionMenuItemVo> menus = permissionMenuService.getUserColumnPermissions(
operator, operator.getId(), PermissionMenuService.MENU_TYPE_ADMIN); operator, operator.getId(), PermissionMenuService.MENU_TYPE_ADMIN, true);
menus = AdminMenuTreeBuilder.filterByMenuType(menus, PermissionMenuService.MENU_TYPE_ADMIN); menus = AdminMenuTreeBuilder.filterByMenuType(menus, PermissionMenuService.MENU_TYPE_ADMIN);
List<Map<String, Object>> items = AdminMenuTreeBuilder.toMapList(AdminMenuTreeBuilder.build(menus)); List<Map<String, Object>> items = AdminMenuTreeBuilder.toMapList(AdminMenuTreeBuilder.build(menus));
return ApiResponse.success(Map.of("items", items)); return ApiResponse.success(Map.of("items", items));
@@ -5,8 +5,8 @@ import com.nanri.aiimage.modules.admin.model.dto.AdminUserCreateRequest;
import com.nanri.aiimage.modules.admin.model.dto.AdminUserUpdateRequest; import com.nanri.aiimage.modules.admin.model.dto.AdminUserUpdateRequest;
import com.nanri.aiimage.modules.admin.model.vo.AdminUserListVo; import com.nanri.aiimage.modules.admin.model.vo.AdminUserListVo;
import com.nanri.aiimage.modules.admin.service.AdminUserService; import com.nanri.aiimage.modules.admin.service.AdminUserService;
import com.nanri.aiimage.modules.admin.support.AdminAuthSupport; import com.nanri.aiimage.common.security.AdminAuthSupport;
import com.nanri.aiimage.modules.permission.model.entity.AdminUserEntity; import com.nanri.aiimage.common.model.entity.AdminUserEntity;
import io.swagger.v3.oas.annotations.Operation; import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag; import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletRequest;
@@ -8,14 +8,14 @@ import com.nanri.aiimage.modules.admin.model.dto.AdminUserUpdateRequest;
import com.nanri.aiimage.modules.admin.model.vo.AdminBriefVo; import com.nanri.aiimage.modules.admin.model.vo.AdminBriefVo;
import com.nanri.aiimage.modules.admin.model.vo.AdminUserItemVo; import com.nanri.aiimage.modules.admin.model.vo.AdminUserItemVo;
import com.nanri.aiimage.modules.admin.model.vo.AdminUserListVo; import com.nanri.aiimage.modules.admin.model.vo.AdminUserListVo;
import com.nanri.aiimage.modules.admin.support.AdminAuthSupport; import com.nanri.aiimage.common.security.AdminAuthSupport;
import com.nanri.aiimage.modules.admin.util.PinyinAbbrUtil; import com.nanri.aiimage.modules.admin.util.PinyinAbbrUtil;
import com.nanri.aiimage.modules.auth.util.WerkzeugPasswordEncoder; import com.nanri.aiimage.modules.auth.util.WerkzeugPasswordEncoder;
import com.nanri.aiimage.modules.permission.mapper.AdminUserMapper; import com.nanri.aiimage.common.mapper.AdminUserMapper;
import com.nanri.aiimage.modules.permission.model.dto.UserColumnPermissionUpdateRequest; import com.nanri.aiimage.modules.permission.model.dto.UserColumnPermissionUpdateRequest;
import com.nanri.aiimage.modules.permission.model.entity.AdminUserEntity; import com.nanri.aiimage.common.model.entity.AdminUserEntity;
import com.nanri.aiimage.modules.permission.service.PermissionMenuService; import com.nanri.aiimage.modules.permission.service.PermissionMenuService;
import com.nanri.aiimage.modules.usersecret.service.UserApiSecretService; import com.nanri.aiimage.modules.admin.spi.UserSecretCleanupPort;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.springframework.dao.DuplicateKeyException; import org.springframework.dao.DuplicateKeyException;
@@ -43,7 +43,7 @@ public class AdminUserService {
private final WerkzeugPasswordEncoder passwordEncoder; private final WerkzeugPasswordEncoder passwordEncoder;
private final AdminAuthSupport adminAuthSupport; private final AdminAuthSupport adminAuthSupport;
private final PermissionMenuService permissionMenuService; private final PermissionMenuService permissionMenuService;
private final UserApiSecretService userApiSecretService; private final UserSecretCleanupPort userSecretCleanupPort;
public AdminUserListVo listUsers(AdminUserEntity currentUser, Integer page, Integer pageSize, public AdminUserListVo listUsers(AdminUserEntity currentUser, Integer page, Integer pageSize,
String username, Long createdById, String roleFilter) { String username, Long createdById, String roleFilter) {
@@ -303,7 +303,7 @@ public class AdminUserService {
if (affected == 0) { if (affected == 0) {
throw new BusinessException("用户不存在"); throw new BusinessException("用户不存在");
} }
int secretRows = userApiSecretService.adminClearByUser(uid); int secretRows = userSecretCleanupPort.adminClearByUser(uid);
log.info("[admin-user] 用户已删除 uid={} 级联清理密钥行={}", uid, secretRows); log.info("[admin-user] 用户已删除 uid={} 级联清理密钥行={}", uid, secretRows);
} }
@@ -0,0 +1,12 @@
package com.nanri.aiimage.modules.admin.spi;
/**
* 删除用户时级联清理其密钥数据(2026-09 边界收敛:admin → usersecret 的类依赖改为端口)。
*
* <p>实现方在 usersecret 模块;返回清理的行数用于日志。
*/
public interface UserSecretCleanupPort {
/** 清理该用户的全部密钥相关行,返回受影响行数。 */
int adminClearByUser(Long userId);
}
@@ -0,0 +1,45 @@
package com.nanri.aiimage.modules.appconfig.controller;
import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.modules.appconfig.service.KdFlowService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.servlet.http.HttpServletRequest;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;
import java.util.Map;
/**
* 工作台「开店流程」模块访问密码校验(公开接口,密码本身就是凭据,不额外要求登录态)。
*
* <p>客户端只在用户点开「开店流程」分组时调用一次;返回体只给 ok 与中文提示,
* 不回显服务端配置的密码。
*/
@Slf4j
@RestController
@RequiredArgsConstructor
@Tag(name = "开店流程访问校验", description = "工作台「开店流程」模块访问密码的服务端校验")
public class KdFlowController {
private final KdFlowService kdFlowService;
@PostMapping("/api/kd-flow/verify")
@Operation(summary = "校验开店流程访问密码",
description = "密码存 app_config.kd_flow_password;改密码只需 UPDATE 该行,客户端无需重新发布")
public ApiResponse<Map<String, Object>> verify(@RequestBody(required = false) Map<String, String> body,
HttpServletRequest request) {
String input = body == null ? null : body.get("password");
boolean ok = kdFlowService.matches(input);
// 只记输入长度与结果,绝不回显密码本身
log.info("[开店流程] 校验请求 remoteAddr={} 输入为空={} 结果={}",
request.getRemoteAddr(), input == null || input.isBlank(), ok ? "通过" : "拒绝");
if (!ok) {
return ApiResponse.fail("密码错误");
}
return ApiResponse.success("验证通过", Map.of("ok", true));
}
}
@@ -0,0 +1,9 @@
package com.nanri.aiimage.modules.appconfig.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.nanri.aiimage.modules.appconfig.model.entity.AppConfigEntity;
import org.apache.ibatis.annotations.Mapper;
@Mapper
public interface AppConfigMapper extends BaseMapper<AppConfigEntity> {
}
@@ -0,0 +1,32 @@
package com.nanri.aiimage.modules.appconfig.model.entity;
import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableId;
import com.baomidou.mybatisplus.annotation.TableName;
import lombok.Data;
import java.time.LocalDateTime;
/**
* 通用应用配置(键值)。首个用途:工作台「开店流程」模块访问密码(key = kd_flow_password)。
* <p>只放这类低价值、需要"改一行即生效"的口令,不放密钥类敏感配置。
*/
@Data
@TableName("app_config")
public class AppConfigEntity {
@TableId(type = IdType.AUTO)
private Long id;
/** 配置键(唯一) */
private String configKey;
/** 配置值 */
private String configValue;
/** 说明 */
private String remark;
/** 更新时间,由数据库 CURRENT_TIMESTAMP 维护 */
private LocalDateTime updatedAt;
}
@@ -0,0 +1,48 @@
package com.nanri.aiimage.modules.appconfig.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.nanri.aiimage.modules.appconfig.mapper.AppConfigMapper;
import com.nanri.aiimage.modules.appconfig.model.entity.AppConfigEntity;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
/**
* 工作台「开店流程」模块访问密码的服务端校验。
*
* <p>此前密码写死在客户端源码(KD_FLOW_PASSWORD),改密码必须重新打包装包发给全部用户;
* 改由服务端比对后,改密码只需 UPDATE app_config 一行(key = kd_flow_password)。
*
* <p>不缓存:调用频次极低(用户点一次分组头一次),且改密码后应立即生效。
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class KdFlowService {
/** app_config 中存放开店流程访问密码的键名 */
public static final String PASSWORD_KEY = "kd_flow_password";
private final AppConfigMapper appConfigMapper;
/** 读取服务端配置的密码;未配置返回 null。 */
public String configuredPassword() {
AppConfigEntity row = appConfigMapper.selectOne(new LambdaQueryWrapper<AppConfigEntity>()
.eq(AppConfigEntity::getConfigKey, PASSWORD_KEY)
.last("LIMIT 1"));
return row == null ? null : row.getConfigValue();
}
/** 校验用户输入的密码。未配置密码时一律判失败(宁可锁死也不放行)。 */
public boolean matches(String input) {
String expect = configuredPassword();
if (expect == null || expect.isBlank()) {
log.warn("[开店流程] app_config 未配置 {},本次校验一律判失败", PASSWORD_KEY);
return false;
}
String actual = input == null ? "" : input.trim();
boolean ok = expect.equals(actual);
log.info("[开店流程] 服务端校验 输入长度={} 结果={}", actual.length(), ok ? "通过" : "不通过");
return ok;
}
}
@@ -34,6 +34,7 @@ import java.net.URI;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import com.nanri.aiimage.modules.task.model.dto.TaskProgressLightRequest; import com.nanri.aiimage.modules.task.model.dto.TaskProgressLightRequest;
import com.nanri.aiimage.modules.task.model.vo.TaskProgressLightBatchVo; import com.nanri.aiimage.modules.task.model.vo.TaskProgressLightBatchVo;
import com.nanri.aiimage.modules.task.service.TaskProgressOwnershipSupport;
@RestController @RestController
@RequiredArgsConstructor @RequiredArgsConstructor
@@ -43,6 +44,7 @@ import com.nanri.aiimage.modules.task.model.vo.TaskProgressLightBatchVo;
public class AppearancePatentController { public class AppearancePatentController {
private final AppearancePatentTaskService service; private final AppearancePatentTaskService service;
private final TaskProgressOwnershipSupport progressOwnershipSupport;
@PostMapping("/parse") @PostMapping("/parse")
@Operation(summary = "解析 Excel 并创建任务", description = "解析上传后的 Excel 文件,提取 id、ASIN、国家、URL、标题等字段。返回给前端的数据只包含整数 id 和 n_1 行;n_2、n_3 等子行会保存在 OSS 解析载荷中,用于最终结果补齐。创建后的任务状态为 PENDING,不会自动推送 Python。") @Operation(summary = "解析 Excel 并创建任务", description = "解析上传后的 Excel 文件,提取 id、ASIN、国家、URL、标题等字段。返回给前端的数据只包含整数 id 和 n_1 行;n_2、n_3 等子行会保存在 OSS 解析载荷中,用于最终结果补齐。创建后的任务状态为 PENDING,不会自动推送 Python。")
@@ -93,21 +95,26 @@ public class AppearancePatentController {
@RequestParam("user_id") Long userId, @RequestParam("user_id") Long userId,
@Parameter(description = "历史记录条数,默认 50,最大 100", example = "50") @Parameter(description = "历史记录条数,默认 50,最大 100", example = "50")
@RequestParam(value = "limit", required = false, defaultValue = "50") Integer limit) { @RequestParam(value = "limit", required = false, defaultValue = "50") Integer limit) {
return ApiResponse.success(service.history(userId, limit)); // 上限钳制(2026-09):文档写"最大 100"但此前无实际校验
int safeLimit = limit == null ? 100 : Math.min(Math.max(1, limit), 100);
return ApiResponse.success(service.history(userId, safeLimit));
} }
@PostMapping("/tasks/progress/batch") @PostMapping("/tasks/progress/batch")
@Operation(summary = "批量查询任务进度", description = "前端只对活跃任务调用该接口,建议 6 秒一次。接口只返回轻量任务状态,不返回明细结果。") @Operation(summary = "批量查询任务进度", description = "前端只对活跃任务调用该接口,建议 6 秒一次。接口只返回轻量任务状态,不返回明细结果。")
public ApiResponse<AppearancePatentTaskBatchVo> progress(@Valid @RequestBody AppearancePatentTaskBatchRequest request) { public ApiResponse<AppearancePatentTaskBatchVo> progress(@Valid @RequestBody AppearancePatentTaskBatchRequest request) {
return ApiResponse.success(service.progressBatch(request.getTaskIds())); // 归属过滤:只查询属于该用户的任务(userId 未传时保持原行为,兼容未升级调用方)
return ApiResponse.success(service.progressBatch(progressOwnershipSupport
.filterOwnedTaskIds(request.getTaskIds(), request.getUserId())));
} }
@PostMapping("/tasks/progress/light") @PostMapping("/tasks/progress/light")
@Operation(summary = "批量查询任务轻量进度(前端轮询专用)", @Operation(summary = "批量查询任务轻量进度(前端轮询专用)",
description = "与 progress/batch 同输入,只返回轻量字段(taskId/status/statusCode/fileStatus/fileError/fileReady/updatedAt)," description = "与 progress/batch 同输入,只返回轻量字段(taskId/status/statusCode/fileStatus/fileError/fileReady/updatedAt/rowsVersion),"
+ "查明细/result 行,响应体更小;旧 progress/batch 端点保留不动。") + "返回行明细内容(仅多一次结果行版本聚合查询),响应体更小;旧 progress/batch 端点保留不动。")
public ApiResponse<TaskProgressLightBatchVo> progressLight(@Valid @RequestBody TaskProgressLightRequest request) { public ApiResponse<TaskProgressLightBatchVo> progressLight(@Valid @RequestBody TaskProgressLightRequest request) {
return ApiResponse.success(service.progressLight(request.getTaskIds())); // 透传 userId:此前传 null = 不过滤归属,匿名遍历 taskId 即可读他人任务状态(2026-09 审查)
return ApiResponse.success(service.progressLight(request.getTaskIds(), request.getUserId()));
} }
@PostMapping("/tasks/{taskId}/activate") @PostMapping("/tasks/{taskId}/activate")
@@ -12,4 +12,8 @@ public class AppearancePatentTaskBatchRequest {
@NotEmpty @NotEmpty
@Schema(description = "需要查询进度的任务 ID 列表。前端只传正在轮询的活跃任务;后端会批量查询,避免每个任务单独请求。", example = "[3938,3939]", requiredMode = Schema.RequiredMode.REQUIRED) @Schema(description = "需要查询进度的任务 ID 列表。前端只传正在轮询的活跃任务;后端会批量查询,避免每个任务单独请求。", example = "[3938,3939]", requiredMode = Schema.RequiredMode.REQUIRED)
private List<Long> taskIds; private List<Long> taskIds;
/** 当前用户 ID(可省略;传入时仅返回该用户的任务,用于进度接口归属过滤)。 */
@Schema(description = "当前用户 ID(可省略;传入时仅返回该用户的任务)", example = "1")
private Long userId;
} }
@@ -0,0 +1,30 @@
package com.nanri.aiimage.modules.appearancepatent.service;
import com.nanri.aiimage.modules.task.model.dto.TaskHeartbeatRequest;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.spi.TaskModuleHeartbeatSpi;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
/**
* 外观专利检测 的任务心跳实现(2026-09 全维度审查 G5)。
*
* <p>心跳逻辑(缓存刷新 / 任务缓存回写)从 task 模块收回本模块,task 侧只依赖 SPI 接口,
* 消除 task → 业务模块的编译期依赖。
*/
@Service
@RequiredArgsConstructor
public class AppearancePatentTaskHeartbeatSpi implements TaskModuleHeartbeatSpi {
private final AppearancePatentTaskCacheService cacheService;
@Override
public String moduleType() {
return "APPEARANCE_PATENT";
}
@Override
public void touchHeartbeat(Long taskId, TaskHeartbeatRequest request) {
cacheService.touchTaskHeartbeat(taskId);
}
}
@@ -0,0 +1,60 @@
package com.nanri.aiimage.modules.appearancepatent.service;
import com.nanri.aiimage.modules.appearancepatent.model.dto.AppearancePatentParsedPayloadDto;
import com.nanri.aiimage.modules.appearancepatent.model.vo.AppearancePatentParsedGroupVo;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.spi.ClientTaskPullSpi;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
/**
* 外观专利的客户端兜底拉取实现。
*
* <p>Python 消费端需要 groups(解析分组,页面上是现拉 /queue-payload 再入队),
* 这里直接复用同一个 service 方法;payload 与页面保持一致(含 prompt / api_key)。
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class AppearancePatentTaskPullSpiImpl implements ClientTaskPullSpi {
private static final String QUEUE_TYPE = "appearance-patent-run";
private final AppearancePatentTaskService taskService;
private final AppearancePatentTaskCacheService taskCacheService;
@Override
public String moduleType() {
return AppearancePatentTaskService.MODULE_TYPE;
}
@Override
public Map<String, Object> buildQueuePayload(FileTaskEntity task) {
AppearancePatentParsedPayloadDto payload = taskService.queuePayload(task.getId(), task.getUserId());
List<AppearancePatentParsedGroupVo> groups = payload.getGroups() == null ? List.of() : payload.getGroups();
if (groups.isEmpty()) {
// 空 groups 在 Python 侧会被静默跳过("groups/rows is empty, skip"),宁可在服务端直接判失败
log.warn("[appearance-patent] 兜底拉取失败:解析分组为空 taskId={}", task.getId());
return null;
}
Map<String, Object> data = new LinkedHashMap<>();
data.put("taskId", task.getId());
data.put("user_id", task.getUserId());
data.put("prompt", payload.getAiPrompt());
data.put("api_key", payload.getApiKey());
data.put("groups", groups);
log.info("[appearance-patent] 兜底载荷已组装 taskId={} groups={}", task.getId(), groups.size());
return Map.of("type", QUEUE_TYPE, "data", data);
}
@Override
public void onClaimed(FileTaskEntity task) {
// 对齐 activate:刷新模块缓存心跳,让页面立刻看到 RUNNING
taskCacheService.touchTaskHeartbeat(task.getId());
}
}
@@ -7,6 +7,7 @@ import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.exception.BusinessException; import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.modules.appearancepatent.service.support.AppearancePatentExcelParser;
import com.nanri.aiimage.common.exception.TaskOwnerMismatchException; import com.nanri.aiimage.common.exception.TaskOwnerMismatchException;
import com.nanri.aiimage.common.service.DistributedJobLockService; import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.common.util.GroupResultPropagator; import com.nanri.aiimage.common.util.GroupResultPropagator;
@@ -39,6 +40,7 @@ import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper; import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper; import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper; import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.dto.TaskScopeLastChunkDto;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity; import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity; import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity; import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
@@ -67,6 +69,8 @@ import org.springframework.stereotype.Service;
import org.springframework.transaction.PlatformTransactionManager; import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.annotation.Transactional; import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.TransactionDefinition; import org.springframework.transaction.TransactionDefinition;
import org.springframework.transaction.support.TransactionSynchronization;
import org.springframework.transaction.support.TransactionSynchronizationManager;
import org.springframework.transaction.support.TransactionTemplate; import org.springframework.transaction.support.TransactionTemplate;
import java.io.File; import java.io.File;
@@ -81,6 +85,7 @@ import java.util.ArrayList;
import java.util.Comparator; import java.util.Comparator;
import java.util.LinkedHashMap; import java.util.LinkedHashMap;
import java.util.LinkedHashSet; import java.util.LinkedHashSet;
import java.util.HashMap;
import java.util.List; import java.util.List;
import java.util.Locale; import java.util.Locale;
import java.util.Map; import java.util.Map;
@@ -103,8 +108,8 @@ public class AppearancePatentTaskService {
public static final String MODULE_TYPE = "APPEARANCE_PATENT"; public static final String MODULE_TYPE = "APPEARANCE_PATENT";
public TaskProgressLightBatchVo progressLight(List<Long> taskIds) { public TaskProgressLightBatchVo progressLight(List<Long> taskIds, Long userId) {
return taskProgressLightAssembler.assemble(MODULE_TYPE, null, taskIds); return taskProgressLightAssembler.assemble(MODULE_TYPE, userId, taskIds);
} }
private static final String STATUS_PENDING = "PENDING"; private static final String STATUS_PENDING = "PENDING";
private static final String STATUS_RUNNING = "RUNNING"; private static final String STATUS_RUNNING = "RUNNING";
@@ -121,6 +126,8 @@ public class AppearancePatentTaskService {
private final LocalFileStorageService localFileStorageService; private final LocalFileStorageService localFileStorageService;
private final OssStorageService ossStorageService; private final OssStorageService ossStorageService;
/** 结果下载直链解析(2026-09 从本类抽到 common,消除 11 处逐字重复) */
private final com.nanri.aiimage.common.service.ResultDownloadResolver resultDownloadResolver;
private final StorageProperties storageProperties; private final StorageProperties storageProperties;
private final FileTaskMapper fileTaskMapper; private final FileTaskMapper fileTaskMapper;
private final FileResultMapper fileResultMapper; private final FileResultMapper fileResultMapper;
@@ -277,12 +284,20 @@ public class AppearancePatentTaskService {
throw new BusinessException("任务不存在"); throw new BusinessException("任务不存在");
} }
ensureTaskOwnedByCurrentInstance(task, "activate"); ensureTaskOwnedByCurrentInstance(task, "activate");
if (STATUS_SUCCESS.equals(task.getStatus()) || STATUS_FAILED.equals(task.getStatus())) { // 只允许 PENDING→RUNNING(条件更新):与客户端「兜底拉取」的原子认领互斥,
// 谁先翻转谁执行,避免页面与客户端重复执行同一任务
int updated = fileTaskMapper.update(null, new LambdaUpdateWrapper<FileTaskEntity>()
.eq(FileTaskEntity::getId, taskId)
.eq(FileTaskEntity::getStatus, STATUS_PENDING)
.set(FileTaskEntity::getStatus, STATUS_RUNNING)
.set(FileTaskEntity::getUpdatedAt, LocalDateTime.now()));
if (updated == 0) {
FileTaskEntity latest = fileTaskMapper.selectById(taskId);
if (latest != null && STATUS_RUNNING.equals(latest.getStatus())) {
throw new BusinessException("任务已在执行中(可能已由客户端自动接管),无需重复启动");
}
throw new BusinessException("任务已结束"); throw new BusinessException("任务已结束");
} }
task.setStatus(STATUS_RUNNING);
task.setUpdatedAt(LocalDateTime.now());
fileTaskMapper.updateById(task);
taskCacheService.touchTaskHeartbeat(taskId); taskCacheService.touchTaskHeartbeat(taskId);
} }
@@ -409,6 +424,53 @@ public class AppearancePatentTaskService {
public void submitResult(Long taskId, AppearancePatentSubmitResultRequest request) { public void submitResult(Long taskId, AppearancePatentSubmitResultRequest request) {
submitResultLocked(taskId, request); submitResultLocked(taskId, request);
// 分片补传后自动恢复终态失败的组装 job —— 走与删除品牌同一套口径,
// 此前外观专利没有接该入口,分片补齐后只能人工重置 job(线上任务 28459 即如此)。
maybeRecoverTerminalFailedAssemble(taskId);
}
/**
* 补传恢复:此前因分片缺失导致组装 job 重试耗尽(终态失败),客户端补传缺口后
* 把失败的组装 job 重置为 PENDING 重新派发({@code resetTerminalFailedForRecovery} 自行补发 dispatch 事件)。
*
* <p>best-effort:恢复失败不得影响补传本身——分片已经落库,恢复只是让后续组装继续推进。
* 常态(无终态失败 job)下只查两次即返回,不触发分片扫描。
*/
private void maybeRecoverTerminalFailedAssemble(Long taskId) {
if (taskId == null || taskId <= 0) {
return;
}
try {
FileResultEntity result = findResultRecord(taskId);
if (result == null) {
return;
}
if (taskFileJobService.hasSuccessfulAssembleJob(taskId, MODULE_TYPE, result.getId())) {
return;
}
if (!taskFileJobService.isTerminalFailedAssembleJob(taskId, MODULE_TYPE, result.getId())) {
return;
}
if (!isResultSubmissionComplete(taskId)) {
log.info("[appearance-patent] 组装 job 终态失败但分片仍未补传完整,暂不恢复 taskId={} resultId={}",
taskId, result.getId());
return;
}
log.info("[appearance-patent] 分片已补传完整,恢复终态失败的组装 job taskId={} resultId={}",
taskId, result.getId());
taskFileJobService.resetTerminalFailedForRecovery(taskId, MODULE_TYPE, result.getId());
} catch (Exception ex) {
log.warn("[appearance-patent] 补传恢复检查失败(不影响本次补传)taskId={} err={}", taskId, ex.getMessage(), ex);
}
}
/** 按 task 取结果行(不创建);不存在返回 null。 */
private FileResultEntity findResultRecord(Long taskId) {
List<FileResultEntity> rows = fileResultMapper.selectList(new LambdaQueryWrapper<FileResultEntity>()
.eq(FileResultEntity::getTaskId, taskId)
.eq(FileResultEntity::getModuleType, MODULE_TYPE)
.last("limit 1"));
return rows == null || rows.isEmpty() ? null : rows.getFirst();
} }
private void submitResultLocked(Long taskId, AppearancePatentSubmitResultRequest request) { private void submitResultLocked(Long taskId, AppearancePatentSubmitResultRequest request) {
@@ -509,20 +571,28 @@ public class AppearancePatentTaskService {
scheduleLlmPipelineForSubmittedChunk(context); scheduleLlmPipelineForSubmittedChunk(context);
} }
/**
* 删除任务。
*
* <p>事务边界:远端载荷删除与缓存清理移到事务提交后执行 —— 单次删除超时 120s、重试 3 次,
* 放在 DB 事务里会让连接被长时间占用(项目在 TaskScopePayloadStorageService 已写明该约束)。
*/
@Transactional @Transactional
public void deleteTask(Long taskId, Long userId) { public void deleteTask(Long taskId, Long userId) {
FileTaskEntity task = fileTaskMapper.selectById(taskId); FileTaskEntity task = fileTaskMapper.selectById(taskId);
if (task == null || !MODULE_TYPE.equals(task.getModuleType()) || !Objects.equals(userId, task.getUserId())) { if (task == null || !MODULE_TYPE.equals(task.getModuleType()) || !Objects.equals(userId, task.getUserId())) {
throw new BusinessException("任务不存在"); throw new BusinessException("任务不存在");
} }
List<String> payloads = collectTransientTaskPayloads(taskId);
fileResultMapper.delete(new LambdaQueryWrapper<FileResultEntity>().eq(FileResultEntity::getTaskId, taskId).eq(FileResultEntity::getModuleType, MODULE_TYPE)); fileResultMapper.delete(new LambdaQueryWrapper<FileResultEntity>().eq(FileResultEntity::getTaskId, taskId).eq(FileResultEntity::getModuleType, MODULE_TYPE));
deleteTransientTaskPayloads(taskId);
taskScopeStateMapper.delete(new LambdaQueryWrapper<TaskScopeStateEntity>().eq(TaskScopeStateEntity::getTaskId, taskId).eq(TaskScopeStateEntity::getModuleType, MODULE_TYPE)); taskScopeStateMapper.delete(new LambdaQueryWrapper<TaskScopeStateEntity>().eq(TaskScopeStateEntity::getTaskId, taskId).eq(TaskScopeStateEntity::getModuleType, MODULE_TYPE));
taskChunkMapper.delete(new LambdaQueryWrapper<TaskChunkEntity>().eq(TaskChunkEntity::getTaskId, taskId).eq(TaskChunkEntity::getModuleType, MODULE_TYPE)); taskChunkMapper.delete(new LambdaQueryWrapper<TaskChunkEntity>().eq(TaskChunkEntity::getTaskId, taskId).eq(TaskChunkEntity::getModuleType, MODULE_TYPE));
taskCacheService.deleteTaskCache(taskId);
// 同步清理 task_file_job,避免被删除任务遗留的 PENDING/FAILED 行被 TaskResultFileJobWorker 反复扫描出 task not found。 // 同步清理 task_file_job,避免被删除任务遗留的 PENDING/FAILED 行被 TaskResultFileJobWorker 反复扫描出 task not found。
taskFileJobService.deleteTaskJobs(taskId, MODULE_TYPE); taskFileJobService.deleteTaskJobs(taskId, MODULE_TYPE);
fileTaskMapper.deleteById(taskId); fileTaskMapper.deleteById(taskId);
// 事务提交后再做远端删除与缓存清理
deletePayloadsAfterCommit(payloads, taskId);
runAfterCommit(() -> taskCacheService.deleteTaskCache(taskId));
} }
public void deleteHistory(Long resultId, Long userId) { public void deleteHistory(Long resultId, Long userId) {
@@ -536,14 +606,8 @@ public class AppearancePatentTaskService {
} }
public String resolveResultDownloadUrl(Long resultId, Long userId) { public String resolveResultDownloadUrl(Long resultId, Long userId) {
FileResultEntity row = fileResultMapper.selectById(resultId); // 2026-09 去重:与原实现等价(并修掉 userId.equals 的潜在 NPE),实现收口到 common
if (row == null || !MODULE_TYPE.equals(row.getModuleType()) || !Objects.equals(userId, row.getUserId())) { return resultDownloadResolver.resolveUrl(resultId, userId, MODULE_TYPE);
throw new BusinessException("记录不存在");
}
if (row.getResultFileUrl() == null || row.getResultFileUrl().isBlank()) {
throw new BusinessException("暂无可下载文件");
}
return ossStorageService.generateFreshDownloadUrl(row.getResultFileUrl());
} }
public String resolveResultDownloadFilename(Long resultId, Long userId) { public String resolveResultDownloadFilename(Long resultId, Long userId) {
@@ -596,6 +660,7 @@ public class AppearancePatentTaskService {
} }
} }
} }
finalizeNoUploadStaleTasks();
} }
public void debugFinalizeStaleTask(Long taskId) { public void debugFinalizeStaleTask(Long taskId) {
@@ -656,6 +721,64 @@ public class AppearancePatentTaskService {
return updatedMillis <= thresholdMillis; return updatedMillis <= thresholdMillis;
} }
/**
* 二次判死:Python 心跳正常(Redis heartbeat 新鲜)但连续 N 分钟无结果分片上报。
*
* <p>既有判定以 Redis heartbeat 为 stale 主信号(P1-7),但该心跳随 Python 的 HTTP 心跳
* 每分钟刷新——主线程卡死时心跳线程照发,任务永远不 stale(与生产 28131 同型缺口)。
* 本线改看 biz_task_scope_state.last_chunk_at(仅在 persistSubmittedChunk 上传分片时刷新),
* 命中后走既有 finalizeStaleTask(封口上传 + LLM 收尾,不粗暴杀任务)。
* 从未上报(无 scope 行或全 NULL)的任务跳过,避免误杀首批较慢的正常任务。
*/
private void finalizeNoUploadStaleTasks() {
long minutes = properties.getNoResultUploadTimeoutMinutes();
if (minutes <= 0) {
return;
}
List<FileTaskEntity> tasks = listStaleFinalizeCandidates();
if (tasks.isEmpty()) {
return;
}
LocalDateTime cutoff = LocalDateTime.now().minusMinutes(minutes);
long heartbeatThresholdMillis = LocalDateTime.now()
.minusMinutes(Math.max(1, properties.getStaleTimeoutMinutes()))
.atZone(ZoneId.systemDefault()).toInstant().toEpochMilli();
Map<Long, LocalDateTime> lastResultAtByTaskId = new HashMap<>();
for (TaskScopeLastChunkDto dto : taskScopeStateMapper.selectLastChunkAtByTaskIds(
tasks.stream().map(FileTaskEntity::getId).toList())) {
if (dto.taskId() != null && dto.lastChunkAt() != null) {
lastResultAtByTaskId.put(dto.taskId(), dto.lastChunkAt());
}
}
for (FileTaskEntity task : tasks) {
if (isHeartbeatStale(task, heartbeatThresholdMillis)) {
// 心跳已 stale:归既有心跳线处理
continue;
}
LocalDateTime lastResultAt = lastResultAtByTaskId.get(task.getId());
if (lastResultAt == null || lastResultAt.isAfter(cutoff)) {
continue;
}
TaskDistributedLockService.LockHandle lockHandle = acquireTaskLock(task.getId(), 0L);
if (lockHandle == null) {
continue;
}
try (lockHandle) {
log.warn("[appearance-patent] 心跳正常但 {} 分钟无结果分片上报,按卡死收尾 taskId={} lastResultAt={}",
minutes, task.getId(), lastResultAt);
String error = "Python heartbeat alive but no result chunk uploaded for " + minutes + " minutes";
if (transactionManager != null) {
inNewTransaction(() -> {
finalizeStaleTask(task.getId(), error);
return null;
});
} else {
finalizeStaleTask(task.getId(), error);
}
}
}
}
/** /**
* /result 提交的纯计算阶段(无事务注解):读任务 + 校验归属 + 分组展开 + * /result 提交的纯计算阶段(无事务注解):读任务 + 校验归属 + 分组展开 +
* 序列化 + payload 存储 + 哈希预计算,全部在事务开始前完成。 * 序列化 + payload 存储 + 哈希预计算,全部在事务开始前完成。
@@ -989,6 +1112,14 @@ public class AppearancePatentTaskService {
long activeAssembleJobs = taskFileJobService.countActiveAssembleJobs(taskId, MODULE_TYPE); long activeAssembleJobs = taskFileJobService.countActiveAssembleJobs(taskId, MODULE_TYPE);
log.info("[appearance-patent] stale recovery probe taskId={} uploadComplete={} activeAssembleJobs={} persistedRows={}", log.info("[appearance-patent] stale recovery probe taskId={} uploadComplete={} activeAssembleJobs={} persistedRows={}",
taskId, uploadComplete, activeAssembleJobs, hasPersistedResultRows(taskId)); taskId, uploadComplete, activeAssembleJobs, hasPersistedResultRows(taskId));
// 已有「重试耗尽且已终态收尾」的 assemble job:说明恢复已经试过、缺失是永久的。
// 再重建只会每 30 秒空转一轮,而且恢复过程刷新任务心跳会让任务永远 RUNNING
// (线上任务 28459 实测:48 分钟里每隔 30 秒重建一次 job)。返回 false 交给
// finalizeStaleTask 按失败收尾,用户看到明确失败而不是无限等待。
if (taskFileJobService.hasExhaustedAssembleJob(taskId, MODULE_TYPE)) {
log.warn("[appearance-patent] stale recovery 放弃:已有重试耗尽的 assemble job,按失败收尾 taskId={}", taskId);
return false;
}
if (!hasPersistedResultRows(taskId)) { if (!hasPersistedResultRows(taskId)) {
log.info("[appearance-patent] stale recovery aborted because no persisted rows taskId={}", taskId); log.info("[appearance-patent] stale recovery aborted because no persisted rows taskId={}", taskId);
return false; return false;
@@ -1139,6 +1270,7 @@ public class AppearancePatentTaskService {
if (rows == null || rows.isEmpty()) { if (rows == null || rows.isEmpty()) {
return; return;
} }
String conflictDetail = "未发生冲突";
for (int attempt = 1; attempt <= CHUNK_PAYLOAD_MERGE_RETRY_LIMIT; attempt++) { for (int attempt = 1; attempt <= CHUNK_PAYLOAD_MERGE_RETRY_LIMIT; attempt++) {
TaskChunkEntity chunk = taskChunkMapper.selectOne(new LambdaQueryWrapper<TaskChunkEntity>() TaskChunkEntity chunk = taskChunkMapper.selectOne(new LambdaQueryWrapper<TaskChunkEntity>()
.eq(TaskChunkEntity::getTaskId, taskId) .eq(TaskChunkEntity::getTaskId, taskId)
@@ -1180,13 +1312,35 @@ public class AppearancePatentTaskService {
transientPayloadStorageService.deleteReplacedPayloadIfNeeded(oldPayload, storedPayload); transientPayloadStorageService.deleteReplacedPayloadIfNeeded(oldPayload, storedPayload);
return; return;
} }
transientPayloadStorageService.deletePayloadIfPresent(storedPayload); // CAS 冲突:读回行上的当前哈希,否则只有一句 conflict 无从判断是谁改的(线上 28459 至今未能定位)
String currentHash = currentPayloadHash(chunk.getId());
conflictDetail = "期望hash=" + oldPayloadHash + " 当前hash=" + currentHash;
if (attempt < CHUNK_PAYLOAD_MERGE_RETRY_LIMIT) { if (attempt < CHUNK_PAYLOAD_MERGE_RETRY_LIMIT) {
log.warn("[appearance-patent] chunk payload CAS conflict, retry merge taskId={} scopeHash={} chunk={} attempt={}/{}", log.warn("[appearance-patent] chunk payload CAS conflict, retry merge taskId={} scopeHash={} chunk={} attempt={}/{} 期望hash={} 当前hash={}",
taskId, scopeHash, chunkIndex, attempt, CHUNK_PAYLOAD_MERGE_RETRY_LIMIT); taskId, scopeHash, chunkIndex, attempt, CHUNK_PAYLOAD_MERGE_RETRY_LIMIT, oldPayloadHash, currentHash);
// 还要重试:这次写的对象会被下次重写,先删掉避免堆积
transientPayloadStorageService.deletePayloadIfPresent(storedPayload);
} else {
// 终局失败:保留这次写入的版本化对象,作为读路径「同槽位兄弟对象」兜底的恢复源。
// 行没指过去不该让该分片永久判死——删掉它才是线上 28459 丢数据的形态。
log.error("[appearance-patent] chunk payload 合并冲突重试耗尽,保留兜底对象 taskId={} scopeHash={} chunk={} 期望hash={} 当前hash={} 保留对象={}",
taskId, scopeHash, chunkIndex, oldPayloadHash, currentHash, storedPayload);
} }
} }
throw new IllegalStateException("appearance patent chunk payload update conflict"); throw new IllegalStateException("appearance patent chunk payload update conflict " + conflictDetail);
}
/** 读回 chunk 行当前的 payload_hash,用于 CAS 冲突定位(行已不存在/读取失败时返回可读标记)。 */
private String currentPayloadHash(Long chunkId) {
if (chunkId == null) {
return "chunkId 为空";
}
try {
TaskChunkEntity latest = taskChunkMapper.selectById(chunkId);
return latest == null ? "行已不存在" : latest.getPayloadHash();
} catch (Exception ex) {
return "读取失败:" + ex.getMessage();
}
} }
private List<AppearancePatentResultRowDto> expandRows(List<AppearancePatentResultRowDto> representatives, private List<AppearancePatentResultRowDto> expandRows(List<AppearancePatentResultRowDto> representatives,
@@ -2296,45 +2450,35 @@ public class AppearancePatentTaskService {
} }
/**
* 解析单个源文件(2026-09 审查 C5 改造)。
*
* <p>改用流式解析器 {@link AppearancePatentExcelParser}EasyExcel SAX ✓,语义与旧 DOM 实现一致、
* 已有单测)取出行级原始值,再在本方法内做原有的业务处理(分组键、状态过滤、字段补齐),
* 不再把用户源文件整表读进堆(几十万行曾会占 1~2GB)。
*/
private ParsedWorkbook parseWorkbook(File input, AppearancePatentSourceFileDto source) { private ParsedWorkbook parseWorkbook(File input, AppearancePatentSourceFileDto source) {
DataFormatter formatter = new DataFormatter(); int maxParseRows = Math.max(1, properties.getMaxParseRows());
try (FileInputStream fis = new FileInputStream(input); Workbook workbook = WorkbookFactory.create(fis)) { AppearancePatentExcelParser.ParsedSheet sheet = new AppearancePatentExcelParser().parse(input, maxParseRows);
Sheet sheet = workbook.getSheetAt(0); List<String> headers = sheet.headers();
Row header = sheet.getRow(0);
if (header == null) {
throw new BusinessException("Excel 表头为空");
}
Map<String, Integer> headerMap = buildHeaderMap(header, formatter);
List<String> headers = readHeaders(header, formatter);
int idCol = findRequiredHeader(headerMap, "id");
int asinCol = findRequiredHeader(headerMap, "asin");
int countryCol = findRequiredHeader(headerMap, "国家", "country");
int priceCol = findOptionalHeaderExact(headerMap, "价格", "price");
int skuCol = findOptionalHeaderExact(headerMap,
"sku", "seller sku", "seller_sku", "merchant sku", "merchant_sku", "商品sku", "商品 sku", "库存sku");
int urlCol = findOptionalHeaderExact(headerMap,
"url", "rul", "link", "image", "img", "pic", "picture",
"链接", "商品链接", "图片", "商品图片", "主图", "商品主图", "图片链接", "主图链接");
int titleCol = findOptionalHeaderExact(headerMap,
"标题", "title", "listing title", "product title", "商品标题", "商品名称", "产品名称");
int statusCol = FailedStatusRowFilter.findStatusColumnIndex(headers); int statusCol = FailedStatusRowFilter.findStatusColumnIndex(headers);
boolean includeBlankStatusRows = statusCol >= 0 && isAppearanceResultWorkbook(headers);
Map<String, Integer> headerIndex = new LinkedHashMap<>();
for (int i = 0; i < headers.size(); i++) {
headerIndex.putIfAbsent(headers.get(i), i);
}
List<ParsedAppearanceRow> parsedRows = new ArrayList<>(); List<ParsedAppearanceRow> parsedRows = new ArrayList<>();
int total = 0; int total = 0;
int dropped = 0; int dropped = 0;
int validRows = 0; int validRows = 0;
int maxParseRows = Math.max(1, properties.getMaxParseRows());
String currentBlockBaseId = ""; String currentBlockBaseId = "";
String currentGroupKey = ""; String currentGroupKey = "";
for (int i = 1; i <= sheet.getLastRowNum(); i++) { for (AppearancePatentExcelParser.AppearanceExcelRow parsed : sheet.rows()) {
Row row = sheet.getRow(i); String id = parsed.id() == null ? "" : parsed.id();
if (row == null) { String asin = parsed.asin() == null ? "" : parsed.asin().toUpperCase(Locale.ROOT);
continue; String country = parsed.country() == null ? "" : parsed.country();
}
String id = cell(row, idCol, formatter);
String asin = cell(row, asinCol, formatter).toUpperCase(Locale.ROOT);
String country = cell(row, countryCol, formatter);
if (id.isBlank() && asin.isBlank() && country.isBlank()) { if (id.isBlank() && asin.isBlank() && country.isBlank()) {
continue; continue;
} }
@@ -2350,7 +2494,7 @@ public class AppearancePatentTaskService {
AppearancePatentParsedRowVo vo = new AppearancePatentParsedRowVo(); AppearancePatentParsedRowVo vo = new AppearancePatentParsedRowVo();
vo.setSourceFileKey(source.getFileKey()); vo.setSourceFileKey(source.getFileKey());
vo.setSourceFilename(firstNonBlank(source.getOriginalFilename(), input.getName())); vo.setSourceFilename(firstNonBlank(source.getOriginalFilename(), input.getName()));
vo.setRowIndex(i + 1); vo.setRowIndex(parsed.rowIndex());
vo.setSourceId(id); vo.setSourceId(id);
vo.setDisplayId(normalizeDisplayId(id)); vo.setDisplayId(normalizeDisplayId(id));
String rowBaseId = baseId(vo.getDisplayId()); String rowBaseId = baseId(vo.getDisplayId());
@@ -2362,18 +2506,20 @@ public class AppearancePatentTaskService {
vo.setRowToken(buildRowToken(source.getFileKey(), vo.getRowIndex())); vo.setRowToken(buildRowToken(source.getFileKey(), vo.getRowIndex()));
vo.setAsin(asin); vo.setAsin(asin);
vo.setCountry(country); vo.setCountry(country);
vo.setPrice(priceCol >= 0 ? cell(row, priceCol, formatter) : ""); vo.setPrice(parsed.price() == null ? "" : parsed.price());
vo.setSku(skuCol >= 0 ? cell(row, skuCol, formatter) : ""); vo.setSku(parsed.sku() == null ? "" : parsed.sku());
vo.setUrl(urlCol >= 0 ? cell(row, urlCol, formatter) : ""); vo.setUrl(parsed.url() == null ? "" : parsed.url());
vo.setTitle(titleCol >= 0 ? cell(row, titleCol, formatter) : ""); vo.setTitle(parsed.title() == null ? "" : parsed.title());
vo.setValues(readRowValues(row, headers, formatter)); vo.setValues(parsed.values() == null ? new LinkedHashMap<>() : new LinkedHashMap<>(parsed.values()));
parsedRows.add(new ParsedAppearanceRow(vo, statusCol >= 0 ? cell(row, statusCol, formatter) : "")); String statusValue = statusCol >= 0 && parsed.values() != null
? parsed.values().getOrDefault(statusHeaderName(headers, statusCol), "")
: "";
parsedRows.add(new ParsedAppearanceRow(vo, statusValue == null ? "" : statusValue));
} }
List<AppearancePatentParsedRowVo> allRows = parsedRows.stream() List<AppearancePatentParsedRowVo> allRows = parsedRows.stream()
.map(ParsedAppearanceRow::row) .map(ParsedAppearanceRow::row)
.toList(); .toList();
hydratePromptFields(allRows); hydratePromptFields(allRows);
boolean includeBlankStatusRows = statusCol >= 0 && isAppearanceResultWorkbook(headers);
FailedStatusRowFilter.FilterResult<ParsedAppearanceRow> filteredRows = FailedStatusRowFilter.FilterResult<ParsedAppearanceRow> filteredRows =
FailedStatusRowFilter.retainRows( FailedStatusRowFilter.retainRows(
parsedRows, parsedRows,
@@ -2392,13 +2538,12 @@ public class AppearancePatentTaskService {
if (allRows.isEmpty()) { if (allRows.isEmpty()) {
throw new BusinessException("no valid appearance patent rows"); throw new BusinessException("no valid appearance patent rows");
} }
return new ParsedWorkbook(total, dropped, headers, allRows); return new ParsedWorkbook(total, dropped, new ArrayList<>(headers), allRows);
} catch (BusinessException ex) {
throw ex;
} catch (Exception ex) {
log.warn("[appearance-patent] parse failed file={} err={}", input, ex.getMessage());
throw new BusinessException("解析 Excel 失败");
} }
/** 状态列的原始表头名(流式行只带"表头→值"映射,状态值需按表头名取回)。 */
private static String statusHeaderName(List<String> headers, int statusCol) {
return statusCol >= 0 && statusCol < headers.size() ? headers.get(statusCol) : "";
} }
private void hydratePromptFields(List<AppearancePatentParsedRowVo> rows) { private void hydratePromptFields(List<AppearancePatentParsedRowVo> rows) {
@@ -2848,12 +2993,35 @@ public class AppearancePatentTaskService {
} catch (Exception ex) { } catch (Exception ex) {
log.warn("[appearance-patent] read chunk payload failed taskId={} chunk={} err={}", log.warn("[appearance-patent] read chunk payload failed taskId={} chunk={} err={}",
chunk.getTaskId(), chunk.getChunkIndex(), ex.getMessage()); chunk.getTaskId(), chunk.getChunkIndex(), ex.getMessage());
if (isPayloadMissing(ex)) {
// payload 对象已不在(被清理或从未写入):重试多少次都读不回来。继续抛会让
// ASSEMBLE_RESULT job 的终态回调每轮重跑兜底组装 → 再读同一个缺失对象 → 无限循环
// (线上任务 28459 每 10~30 秒重试一次)。跳过该分片,让任务按已有分片出部分结果,
// 与品牌/相似ASIN「失败也产出可下载的部分结果」同一口径。
log.warn("[appearance-patent] chunk payload 已不存在,跳过该分片(任务按已有分片出结果)"
+ " taskId={} chunk={}", chunk.getTaskId(), chunk.getChunkIndex());
return rows;
}
throw new BusinessException("appearance patent chunk payload read failed chunk=" throw new BusinessException("appearance patent chunk payload read failed chunk="
+ chunk.getChunkIndex() + ": " + ex.getMessage(), ex); + chunk.getChunkIndex() + ": " + ex.getMessage(), ex);
} }
return rows; return rows;
} }
/** payload 对象已不存在(RustFS 返回 NoSuchKeymessage 为 "The specified key does not exist.")。
* 只有这种"重试也没用"的缺失才允许跳过;网络类失败仍照旧抛出以便重试。 */
private static boolean isPayloadMissing(Throwable error) {
Throwable cursor = error;
while (cursor != null) {
String message = cursor.getMessage();
if (message != null && message.contains("does not exist")) {
return true;
}
cursor = cursor.getCause();
}
return false;
}
private String rowKey(AppearancePatentParsedRowVo row) { private String rowKey(AppearancePatentParsedRowVo row) {
if (row == null) { if (row == null) {
return ""; return "";
@@ -2935,18 +3103,27 @@ public class AppearancePatentTaskService {
} }
} }
private void deleteTransientTaskPayloads(Long taskId) { /** 只读收集任务范围/分片载荷指针,供事务提交后做远端删除。 */
private List<String> collectTransientTaskPayloads(Long taskId) {
if (taskId == null || taskId <= 0) { if (taskId == null || taskId <= 0) {
return; return List.of();
} }
List<String> payloads = new ArrayList<>();
List<TaskScopeStateEntity> scopes = taskScopeStateMapper.selectList(new LambdaQueryWrapper<TaskScopeStateEntity>() List<TaskScopeStateEntity> scopes = taskScopeStateMapper.selectList(new LambdaQueryWrapper<TaskScopeStateEntity>()
.select(TaskScopeStateEntity::getParsedPayloadJson, TaskScopeStateEntity::getStateJson) .select(TaskScopeStateEntity::getParsedPayloadJson, TaskScopeStateEntity::getStateJson)
.eq(TaskScopeStateEntity::getTaskId, taskId) .eq(TaskScopeStateEntity::getTaskId, taskId)
.eq(TaskScopeStateEntity::getModuleType, MODULE_TYPE)); .eq(TaskScopeStateEntity::getModuleType, MODULE_TYPE));
if (scopes != null) { if (scopes != null) {
for (TaskScopeStateEntity scope : scopes) { for (TaskScopeStateEntity scope : scopes) {
transientPayloadStorageService.deletePayloadIfPresent(scope.getParsedPayloadJson()); if (scope == null) {
transientPayloadStorageService.deletePayloadIfPresent(scope.getStateJson()); continue;
}
if (scope.getParsedPayloadJson() != null && !scope.getParsedPayloadJson().isBlank()) {
payloads.add(scope.getParsedPayloadJson());
}
if (scope.getStateJson() != null && !scope.getStateJson().isBlank()) {
payloads.add(scope.getStateJson());
}
} }
} }
List<TaskChunkEntity> chunks = taskChunkMapper.selectList(new LambdaQueryWrapper<TaskChunkEntity>() List<TaskChunkEntity> chunks = taskChunkMapper.selectList(new LambdaQueryWrapper<TaskChunkEntity>()
@@ -2955,10 +3132,50 @@ public class AppearancePatentTaskService {
.eq(TaskChunkEntity::getModuleType, MODULE_TYPE)); .eq(TaskChunkEntity::getModuleType, MODULE_TYPE));
if (chunks != null) { if (chunks != null) {
for (TaskChunkEntity chunk : chunks) { for (TaskChunkEntity chunk : chunks) {
transientPayloadStorageService.deletePayloadIfPresent(chunk.getPayloadJson()); if (chunk != null && chunk.getPayloadJson() != null && !chunk.getPayloadJson().isBlank()) {
payloads.add(chunk.getPayloadJson());
} }
} }
} }
return payloads;
}
/** 事务提交后逐条删远端载荷;无事务时立即执行(与 TaskResultItemService 同款兜底)。 */
private void deletePayloadsAfterCommit(List<String> payloads, Long taskId) {
if (payloads == null || payloads.isEmpty()) {
return;
}
runAfterCommit(() -> {
for (String payload : payloads) {
try {
transientPayloadStorageService.deletePayloadIfPresent(payload);
} catch (Exception ex) {
log.warn("[appearance-patent] 载荷删除失败 taskId={} msg={}", taskId, ex.getMessage());
}
}
});
}
/**
* 删除范围/分片的远端载荷(保留给无事务的清理链路调用)。
*/
private void deleteTransientTaskPayloads(Long taskId) {
deletePayloadsAfterCommit(collectTransientTaskPayloads(taskId), taskId);
}
/** 有活动事务则注册 afterCommit,否则立即执行。 */
private void runAfterCommit(Runnable action) {
if (TransactionSynchronizationManager.isSynchronizationActive()) {
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
@Override
public void afterCommit() {
action.run();
}
});
return;
}
action.run();
}
private record SubmitContext(FileTaskEntity task, private record SubmitContext(FileTaskEntity task,
String scopeKey, String scopeKey,
@@ -1,17 +1,17 @@
package com.nanri.aiimage.modules.appearancepatent.service.support; package com.nanri.aiimage.modules.appearancepatent.service.support;
import com.nanri.aiimage.common.exception.BusinessException; import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.common.util.ExcelStreamReader;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.poi.ss.usermodel.DataFormatter;
import org.apache.poi.ss.usermodel.Row;
import org.apache.poi.ss.usermodel.Sheet;
import org.apache.poi.ss.usermodel.Workbook;
import org.apache.poi.ss.usermodel.WorkbookFactory;
import java.io.BufferedInputStream;
import java.io.File; import java.io.File;
import java.io.FileInputStream; import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream; import java.io.InputStream;
import java.io.PushbackInputStream;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Arrays;
import java.util.LinkedHashMap; import java.util.LinkedHashMap;
import java.util.List; import java.util.List;
import java.util.Locale; import java.util.Locale;
@@ -37,9 +37,8 @@ public class AppearancePatentExcelParser {
if (input == null) { if (input == null) {
throw new IllegalArgumentException("input must not be null"); throw new IllegalArgumentException("input must not be null");
} }
try (FileInputStream fis = new FileInputStream(input); try (InputStream inputStream = new FileInputStream(input)) {
Workbook workbook = WorkbookFactory.create(fis)) { return readStreaming(inputStream, maxRows);
return parseWorkbook(workbook, maxRows);
} catch (BusinessException ex) { } catch (BusinessException ex) {
throw ex; throw ex;
} catch (Exception ex) { } catch (Exception ex) {
@@ -52,8 +51,8 @@ public class AppearancePatentExcelParser {
if (input == null) { if (input == null) {
throw new IllegalArgumentException("input must not be null"); throw new IllegalArgumentException("input must not be null");
} }
try (Workbook workbook = WorkbookFactory.create(input)) { try {
return parseWorkbook(workbook, DEFAULT_MAX_ROWS); return readStreaming(input, DEFAULT_MAX_ROWS);
} catch (BusinessException ex) { } catch (BusinessException ex) {
throw ex; throw ex;
} catch (Exception ex) { } catch (Exception ex) {
@@ -62,83 +61,154 @@ public class AppearancePatentExcelParser {
} }
} }
private ParsedSheet parseWorkbook(Workbook workbook, int maxRows) { /**
int safeMaxRows = Math.max(1, maxRows); * 流式解析(EasyExcel SAX),替代 POI WorkbookFactory 全量 DOM 加载:
DataFormatter formatter = new DataFormatter(); * 大表不再整表驻留堆内存,行数上限在迭代过程中即时生效(超限抛错)。
Sheet sheet = workbook.getSheetAt(0); * 语义与原 POI 路径一致:cell 归一化、表头别名匹配、空行跳过、必填表头缺失抛错、无字段截断。
Row header = sheet.getRow(0); * 注意不关闭传入的 InputStream(由调用方负责)。
if (header == null) { */
private ParsedSheet readStreaming(InputStream inputStream, int maxRows) throws Exception {
// EasyExcel 会把非 zip 文本当 CSV 解析成功;原 WorkbookFactory 只认 xlsx/xls
// 这里先做文件魔数校验,保持「垃圾文件→解析 Excel 失败」的语义并拒绝 CSV 误解析。
PushbackInputStream pb = new PushbackInputStream(new BufferedInputStream(inputStream, 8192), 8);
requireExcelMagic(pb);
SheetContext ctx = new SheetContext(Math.max(1, maxRows));
ExcelStreamReader.readFirstSheet(pb, new ExcelStreamReader.SheetRowHandler() {
@Override
public void onHeader(String sheetName, Integer sheetNo, Map<Integer, String> headerMap) {
ctx.initHeader(headerMap);
}
@Override
public void onRow(String sheetName, Integer sheetNo, int rowIndex,
Map<Integer, String> headerMap, Map<Integer, String> rowMap) {
ctx.consumeRow(rowIndex, rowMap);
}
});
if (ctx.headers == null) {
// 空表 / 表头行整行为空(EasyExcel 不上报表头回调):与旧实现 header == null 一致
throw new BusinessException("Excel 表头为空"); throw new BusinessException("Excel 表头为空");
} }
Map<String, Integer> headerMap = buildHeaderMap(header, formatter); return new ParsedSheet(ctx.headers, ctx.rows);
List<String> headers = readHeaders(header, formatter);
int idCol = findRequiredHeader(headerMap, "id");
int asinCol = findRequiredHeader(headerMap, "asin");
int countryCol = findRequiredHeader(headerMap, "国家", "country");
int priceCol = findOptionalHeaderExact(headerMap, "价格", "price");
int skuCol = findOptionalHeaderExact(headerMap,
"sku", "seller sku", "seller_sku", "merchant sku", "merchant_sku", "商品sku", "商品 sku", "库存sku");
int urlCol = findOptionalHeaderExact(headerMap,
"url", "rul", "link", "image", "img", "pic", "picture",
"链接", "商品链接", "图片", "商品图片", "主图", "商品主图", "图片链接", "主图链接");
int titleCol = findOptionalHeaderExact(headerMap,
"标题", "title", "listing title", "product title", "商品标题", "商品名称", "产品名称");
List<AppearanceExcelRow> rows = new ArrayList<>();
for (int i = 1; i <= sheet.getLastRowNum(); i++) {
Row row = sheet.getRow(i);
if (row == null) {
continue;
}
String id = cell(row, idCol, formatter);
String asin = cell(row, asinCol, formatter).toUpperCase(Locale.ROOT);
String country = cell(row, countryCol, formatter);
if (id.isBlank() && asin.isBlank() && country.isBlank()) {
continue;
}
if (rows.size() >= safeMaxRows) {
throw new BusinessException("解析行数超过上限: " + safeMaxRows);
}
rows.add(new AppearanceExcelRow(
i + 1,
id,
asin,
country,
priceCol >= 0 ? cell(row, priceCol, formatter) : "",
skuCol >= 0 ? cell(row, skuCol, formatter) : "",
urlCol >= 0 ? cell(row, urlCol, formatter) : "",
titleCol >= 0 ? cell(row, titleCol, formatter) : "",
readRowValues(row, headers, formatter)));
}
return new ParsedSheet(headers, rows);
} }
private Map<String, Integer> buildHeaderMap(Row header, DataFormatter formatter) { /** 校验文件头魔数:xlsx=PK(zip)、xls=OLE2(CFD)。判非抛「解析 Excel 失败」;用 unread 回退已读字节。 */
private void requireExcelMagic(PushbackInputStream in) throws IOException {
byte[] head = new byte[8];
int n = 0;
while (n < head.length) {
int r = in.read(head, n, head.length - n);
if (r < 0) {
break;
}
n += r;
}
if (n > 0) {
in.unread(head, 0, n);
}
boolean isZip = n >= 4 && head[0] == 'P' && head[1] == 'K' && (head[2] == 3 || head[2] == 5 || head[2] == 7);
boolean isOle2 = n >= 8
&& (head[0] & 0xFF) == 0xD0 && (head[1] & 0xFF) == 0xCF
&& (head[2] & 0xFF) == 0x11 && (head[3] & 0xFF) == 0xE0;
if (!isZip && !isOle2) {
log.warn("[appearance-patent] parse rejected non-excel magic head={}", Arrays.copyOf(head, Math.max(n, 0)));
throw new BusinessException("解析 Excel 失败");
}
}
/** 单表解析上下文:表头就绪后逐行累积结果行。 */
private final class SheetContext {
private final int maxRows;
private final List<AppearanceExcelRow> rows = new ArrayList<>();
private List<String> headers;
private int idCol;
private int asinCol;
private int countryCol;
private int priceCol;
private int skuCol;
private int urlCol;
private int titleCol;
SheetContext(int maxRows) {
this.maxRows = maxRows;
}
void initHeader(Map<Integer, String> rawHeaderMap) {
// EasyExcel 回调给出 列号 → 表头文本,与旧 Row 遍历等价(缺列一般为 null/空串)
int lastColumnCount = lastColumnCount(rawHeaderMap);
Map<String, Integer> map = new LinkedHashMap<>(); Map<String, Integer> map = new LinkedHashMap<>();
for (int i = 0; i < header.getLastCellNum(); i++) { List<String> headerNames = new ArrayList<>();
String val = normalize(formatter.formatCellValue(header.getCell(i))); for (int i = 0; i < lastColumnCount; i++) {
String val = normalize(rawHeaderMap.getOrDefault(i, ""));
headerNames.add(val.isBlank() ? "" + (i + 1) : val);
if (!val.isBlank()) { if (!val.isBlank()) {
map.putIfAbsent(val.toLowerCase(Locale.ROOT), i); map.putIfAbsent(val.toLowerCase(Locale.ROOT), i);
} }
} }
return map; this.headers = headerNames;
this.idCol = findRequiredHeader(map, "id");
this.asinCol = findRequiredHeader(map, "asin");
this.countryCol = findRequiredHeader(map, "国家", "country");
this.priceCol = findOptionalHeaderExact(map, "价格", "price");
this.skuCol = findOptionalHeaderExact(map,
"sku", "seller sku", "seller_sku", "merchant sku", "merchant_sku", "商品sku", "商品 sku", "库存sku");
this.urlCol = findOptionalHeaderExact(map,
"url", "rul", "link", "image", "img", "pic", "picture",
"链接", "商品链接", "图片", "商品图片", "主图", "商品主图", "图片链接", "主图链接");
this.titleCol = findOptionalHeaderExact(map,
"标题", "title", "listing title", "product title", "商品标题", "商品名称", "产品名称");
} }
private List<String> readHeaders(Row header, DataFormatter formatter) { void consumeRow(int rowIndex, Map<Integer, String> rowMap) {
List<String> headers = new ArrayList<>(); if (headers == null) {
for (int i = 0; i < header.getLastCellNum(); i++) { return;
String val = normalize(formatter.formatCellValue(header.getCell(i)));
headers.add(val.isBlank() ? "" + (i + 1) : val);
} }
return headers; // EasyExcel rowIndex 从 0 起(0 为表头),POI 原实现行号同样 0 起并 +1 展示
String id = streamCell(rowMap, idCol);
String asin = streamCell(rowMap, asinCol).toUpperCase(Locale.ROOT);
String country = streamCell(rowMap, countryCol);
if (id.isBlank() && asin.isBlank() && country.isBlank()) {
return;
}
if (rows.size() >= maxRows) {
throw new BusinessException("解析行数超过上限: " + maxRows);
} }
private Map<String, String> readRowValues(Row row, List<String> headers, DataFormatter formatter) {
Map<String, String> values = new LinkedHashMap<>(); Map<String, String> values = new LinkedHashMap<>();
for (int i = 0; i < headers.size(); i++) { for (int i = 0; i < headers.size(); i++) {
values.put(headers.get(i), cell(row, i, formatter)); values.put(headers.get(i), streamCell(rowMap, i));
}
rows.add(new AppearanceExcelRow(
rowIndex + 1,
id,
asin,
country,
priceCol >= 0 ? streamCell(rowMap, priceCol) : "",
skuCol >= 0 ? streamCell(rowMap, skuCol) : "",
urlCol >= 0 ? streamCell(rowMap, urlCol) : "",
titleCol >= 0 ? streamCell(rowMap, titleCol) : "",
values));
}
private int lastColumnCount(Map<Integer, String> rowMap) {
if (rowMap == null || rowMap.isEmpty()) {
return 0;
}
int lastColumnCount = 0;
for (Integer columnIndex : rowMap.keySet()) {
if (columnIndex != null && columnIndex >= lastColumnCount) {
lastColumnCount = columnIndex + 1;
}
}
return lastColumnCount;
}
private String streamCell(Map<Integer, String> rowMap, int col) {
if (col < 0 || rowMap == null) {
return "";
}
return normalize(rowMap.get(col));
} }
return values;
} }
private int findRequiredHeader(Map<String, Integer> map, String... names) { private int findRequiredHeader(Map<String, Integer> map, String... names) {
@@ -177,10 +247,6 @@ public class AppearancePatentExcelParser {
return normalized.replaceAll("[\\s_\\-()()\\[\\]{}:/\\\\]+", ""); return normalized.replaceAll("[\\s_\\-()()\\[\\]{}:/\\\\]+", "");
} }
private String cell(Row row, int col, DataFormatter formatter) {
return col < 0 ? "" : normalize(formatter.formatCellValue(row.getCell(col)));
}
private String normalize(String val) { private String normalize(String val) {
return val == null ? "" : val.replace(String.valueOf((char) 0xFEFF), "").replace((char) 0x3000, ' ').trim().replaceAll("\\s+", " "); return val == null ? "" : val.replace(String.valueOf((char) 0xFEFF), "").replace((char) 0x3000, ' ').trim().replaceAll("\\s+", " ");
} }
@@ -3,12 +3,12 @@ package com.nanri.aiimage.modules.auth.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper; import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper; import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.nanri.aiimage.common.exception.BusinessException; import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.modules.auth.config.AuthProperties; import com.nanri.aiimage.common.security.AuthProperties;
import com.nanri.aiimage.modules.auth.mapper.LoginUserMapper; import com.nanri.aiimage.modules.auth.mapper.LoginUserMapper;
import com.nanri.aiimage.modules.auth.model.dto.LoginRequest; import com.nanri.aiimage.modules.auth.model.dto.LoginRequest;
import com.nanri.aiimage.modules.auth.model.entity.LoginUserEntity; import com.nanri.aiimage.modules.auth.model.entity.LoginUserEntity;
import com.nanri.aiimage.modules.auth.model.vo.LoginResultVo; import com.nanri.aiimage.modules.auth.model.vo.LoginResultVo;
import com.nanri.aiimage.modules.auth.support.DeviceSessionPolicy; import com.nanri.aiimage.common.security.DeviceSessionPolicy;
import com.nanri.aiimage.modules.auth.util.WerkzeugPasswordEncoder; import com.nanri.aiimage.modules.auth.util.WerkzeugPasswordEncoder;
import com.nanri.aiimage.modules.permission.service.PermissionMenuService; import com.nanri.aiimage.modules.permission.service.PermissionMenuService;
import io.jsonwebtoken.Claims; import io.jsonwebtoken.Claims;
@@ -18,6 +18,10 @@ import org.springframework.http.ResponseCookie;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import java.time.Duration; import java.time.Duration;
import java.time.Instant;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import com.nanri.aiimage.common.security.JwtService;
@Service @Service
@RequiredArgsConstructor @RequiredArgsConstructor
@@ -30,6 +34,50 @@ public class AuthService {
private final PermissionMenuService permissionMenuService; private final PermissionMenuService permissionMenuService;
private final AuthProperties authProperties; private final AuthProperties authProperties;
/**
* 登录失败计数与锁定(2026-09 全维度审查补):生产是公网域名,此前无任何失败限制
* 即可无限撞库。内存实现(双节点各自计数,防护效果减半但不引入新依赖),
* 达到阈值后锁定 15 分钟。
*/
private static final int LOGIN_FAIL_LIMIT = 10;
private static final Duration LOGIN_LOCK_DURATION = Duration.ofMinutes(15);
private final Map<String, FailRecord> loginFailures = new ConcurrentHashMap<>();
/** 失败计数;lockedUntil 非空表示已锁定。 */
private record FailRecord(int count, Instant lockedUntil) {
}
private void assertLoginNotLocked(String username) {
FailRecord record = loginFailures.get(username);
if (record != null && record.lockedUntil() != null && record.lockedUntil().isAfter(Instant.now())) {
long minutes = Duration.between(Instant.now(), record.lockedUntil()).toMinutes() + 1;
log.warn("[auth] 登录已锁定 username={} 剩余约 {} 分钟", username, minutes);
throw new BusinessException("登录失败次数过多,请 " + minutes + " 分钟后再试");
}
}
private void recordLoginFailure(String username) {
// 防无界增长:积累较多时清掉未锁定的过期记录(登录接口调用频次低)
if (loginFailures.size() > 1000) {
Instant now = Instant.now();
loginFailures.entrySet().removeIf(e -> e.getValue().lockedUntil() == null
|| e.getValue().lockedUntil().isBefore(now));
}
loginFailures.compute(username, (key, old) -> {
int count = (old == null ? 0 : old.count()) + 1;
Instant lockedUntil = count >= LOGIN_FAIL_LIMIT ? Instant.now().plus(LOGIN_LOCK_DURATION) : null;
if (lockedUntil != null) {
log.warn("[auth] 登录失败达阈值,锁定 username={} count={} minutes={}",
username, count, LOGIN_LOCK_DURATION.toMinutes());
}
return new FailRecord(count, lockedUntil);
});
}
private void clearLoginFailures(String username) {
loginFailures.remove(username);
}
public LoginResultVo login(LoginRequest request) { public LoginResultVo login(LoginRequest request) {
String username = trim(request.getUsername()); String username = trim(request.getUsername());
String password = request.getPassword() == null ? "" : request.getPassword(); String password = request.getPassword() == null ? "" : request.getPassword();
@@ -41,12 +89,16 @@ public class AuthService {
throw new BusinessException("缺少设备ID,请在桌面端打开"); throw new BusinessException("缺少设备ID,请在桌面端打开");
} }
assertLoginNotLocked(username);
LoginUserEntity user = loginUserMapper.selectOne(new LambdaQueryWrapper<LoginUserEntity>() LoginUserEntity user = loginUserMapper.selectOne(new LambdaQueryWrapper<LoginUserEntity>()
.eq(LoginUserEntity::getUsername, username) .eq(LoginUserEntity::getUsername, username)
.last("LIMIT 1")); .last("LIMIT 1"));
if (user == null || !passwordEncoder.matches(password, user.getPasswordHash())) { if (user == null || !passwordEncoder.matches(password, user.getPasswordHash())) {
recordLoginFailure(username);
throw new BusinessException("用户名或密码错误"); throw new BusinessException("用户名或密码错误");
} }
clearLoginFailures(username);
boolean isAdmin = user.getIsAdmin() != null && user.getIsAdmin() == 1; boolean isAdmin = user.getIsAdmin() != null && user.getIsAdmin() == 1;
// 单设备登录:登录成功即把账号绑定到当前设备(last-login-wins),原设备下一次请求被顶下线 // 单设备登录:登录成功即把账号绑定到当前设备(last-login-wins),原设备下一次请求被顶下线
@@ -95,10 +95,14 @@ public class BrandCheckClient {
public BrandCheckBatchResult checkAll(List<String> brands, String strategy) { public BrandCheckBatchResult checkAll(List<String> brands, String strategy) {
List<String> distinctBrands = distinctNonBlank(brands); List<String> distinctBrands = distinctNonBlank(brands);
// 整批共用一个耗时预算:上游 16890 卡死时,单品牌 10 次重试曾把一次分片回传拖到
// 103.5 秒(taskId 28599),客户端重试预算耗尽后中止了整个采集。预算用尽即停止重试。
long budgetMillis = properties.getTotalTimeoutMillis();
long deadlineNanos = budgetMillis > 0L ? System.nanoTime() + budgetMillis * 1_000_000L : Long.MAX_VALUE;
List<CompletableFuture<BrandCheckOutcome>> futures = new ArrayList<>(distinctBrands.size()); List<CompletableFuture<BrandCheckOutcome>> futures = new ArrayList<>(distinctBrands.size());
for (String brand : distinctBrands) { for (String brand : distinctBrands) {
futures.add(CompletableFuture.supplyAsync( futures.add(CompletableFuture.supplyAsync(
() -> checkOneBrand(brand, strategy), checkExecutor)); () -> checkOneBrand(brand, strategy, deadlineNanos), checkExecutor));
} }
List<Object> failedData = new ArrayList<>(); List<Object> failedData = new ArrayList<>();
List<Object> queryFailedData = new ArrayList<>(); List<Object> queryFailedData = new ArrayList<>();
@@ -110,20 +114,29 @@ public class BrandCheckClient {
return new BrandCheckBatchResult(distinctBrands, failedData, queryFailedData); return new BrandCheckBatchResult(distinctBrands, failedData, queryFailedData);
} }
private BrandCheckOutcome checkOneBrand(String brand, String strategy) { private BrandCheckOutcome checkOneBrand(String brand, String strategy, long deadlineNanos) {
int attempts = Math.max(1, properties.getRetryTimes()); int attempts = Math.max(1, properties.getRetryTimes());
BrandCheckResponse response = null; BrandCheckResponse response = null;
Exception lastFailure = null; Exception lastFailure = null;
for (int attempt = 1; attempt <= attempts; attempt++) { for (int attempt = 1; attempt <= attempts; attempt++) {
// 预算用尽就不再重试,按查询失败收尾。只掐「重试」不打断已发出的请求,
// 故最坏耗时 ≈ 预算 + 一次请求的读超时;首轮始终执行,避免上游只是慢一点时被误降级。
if (attempt > 1 && System.nanoTime() >= deadlineNanos) {
log.warn("[brand-check] 整批耗时预算用尽,停止重试 brand={} attempt={}/{} lastErr={}",
brand, attempt, attempts,
lastFailure == null ? "query_faild_data 持续非空" : lastFailure.getMessage());
break;
}
try { try {
response = check(brand, strategy); response = check(brand, strategy);
} catch (Exception ex) { } catch (Exception ex) {
lastFailure = ex; lastFailure = ex;
response = null; response = null;
if (attempt < attempts) { if (attempt < attempts) {
log.warn("[brand-check] 请求异常将重试 brand={} attempt={}/{} err={}", long retryDelayMillis = retryDelayMillis(attempt);
brand, attempt, attempts, ex.getMessage()); log.warn("[brand-check] 请求异常将重试 brand={} attempt={}/{} 等待={}ms err={}",
sleepBeforeRetry(); brand, attempt, attempts, retryDelayMillis, ex.getMessage());
sleepBeforeRetry(retryDelayMillis);
} }
continue; continue;
} }
@@ -132,9 +145,10 @@ public class BrandCheckClient {
return new BrandCheckOutcome(nullToEmpty(response.getFaildData()), List.of()); return new BrandCheckOutcome(nullToEmpty(response.getFaildData()), List.of());
} }
if (attempt < attempts) { if (attempt < attempts) {
log.warn("[brand-check] 服务端返回查询失败将重试 brand={} attempt={}/{}", long retryDelayMillis = retryDelayMillis(attempt);
brand, attempt, attempts); log.warn("[brand-check] 服务端返回查询失败将重试 brand={} attempt={}/{} 等待={}ms",
sleepBeforeRetry(); brand, attempt, attempts, retryDelayMillis);
sleepBeforeRetry(retryDelayMillis);
} }
} }
log.warn("[brand-check] 重试耗尽判定查询失败 brand={} attempts={} lastErr={}", log.warn("[brand-check] 重试耗尽判定查询失败 brand={} attempts={} lastErr={}",
@@ -144,8 +158,18 @@ public class BrandCheckClient {
response == null ? List.of(brand) : nullToEmpty(response.getQueryFaildData())); response == null ? List.of(brand) : nullToEmpty(response.getQueryFaildData()));
} }
private void sleepBeforeRetry() { /**
long delayMillis = Math.max(0L, properties.getRetryIntervalMillis()); * 第 attempt 次重试前的等待毫秒数:按基准间隔随轮次递增后封顶。
* 限流窗口通常只有几秒,固定 1s 间隔反复打过去救不回来;递增等待能覆盖窗口,
* 封顶则保证单个品牌不会长时间占住查询线程(并发度只有 3)。
*/
private long retryDelayMillis(int attempt) {
long base = Math.max(0L, properties.getRetryIntervalMillis());
long cap = Math.max(base, properties.getRetryMaxIntervalMillis());
return Math.min(base * Math.max(1, attempt), cap);
}
private void sleepBeforeRetry(long delayMillis) {
if (delayMillis <= 0L) { if (delayMillis <= 0L) {
return; return;
} }
@@ -0,0 +1,56 @@
package com.nanri.aiimage.modules.brand.controller;
import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.common.security.AdminAuthSupport;
import com.nanri.aiimage.modules.brand.model.dto.BrandTaskAbortRequest;
import com.nanri.aiimage.modules.brand.service.BrandTaskService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.servlet.http.HttpServletRequest;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.Map;
/**
* 内部接口:品牌任务中止上报,供主机 A 品牌检测服务(15126)在爬取不可继续时调用
* (如 WIPO 连续限流熔断)。Java 侧用已收到的结果分片部分组装结果文件并落 failed +
* 真实原因,避免任务悬挂到心跳超时被判「前端长时间无响应」、已跑出的数据无法下载。
*
* <p>鉴权:仅凭 X-Internal-Token(与容器 AIIMAGE_INTERNAL_TOKEN / 宿主机
* ~/.aiimage/internal-token 同值)。/api/internal 前缀虽在 AdminApiGuardFilter 兜底
* 名单内、可信令牌会放行,controller 仍须自校验——防止配置漂移时匿名可达。
*/
@RestController
@RequiredArgsConstructor
@Slf4j
@RequestMapping("/api/internal/brand")
@Tag(name = "内部接口", description = "服务间调用(X-Internal-Token 鉴权)。")
public class InternalBrandTaskController {
private final BrandTaskService brandTaskService;
private final AdminAuthSupport adminAuthSupport;
@PostMapping("/tasks/{taskId}/abort")
@Operation(summary = "上报品牌任务中止(爬取方调用)",
description = "用已收到的结果分片部分组装结果文件(未检测品牌单独成 sheet)并落 failed + 真实原因;幂等,终态任务直接返回。")
public ApiResponse<Map<String, Object>> abortTask(HttpServletRequest request,
@PathVariable Long taskId,
@RequestBody(required = false) BrandTaskAbortRequest body) {
if (!adminAuthSupport.isTrustedInternalToken(request)) {
log.warn("[internal-brand-abort] 拒绝未携带可信内部令牌的请求 taskId={} remoteAddr={}",
taskId, request.getRemoteAddr());
throw new BusinessException(401, "未授权");
}
String errorMessage = body == null ? null : body.getErrorMessage();
log.info("[internal-brand-abort] 收到中止上报 taskId={} remoteAddr={} msg={}",
taskId, request.getRemoteAddr(), errorMessage);
return ApiResponse.success(brandTaskService.abortTask(taskId, errorMessage));
}
}
@@ -3,7 +3,31 @@ package com.nanri.aiimage.modules.brand.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper; import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.nanri.aiimage.modules.brand.model.entity.BrandCrawlTaskEntity; import com.nanri.aiimage.modules.brand.model.entity.BrandCrawlTaskEntity;
import org.apache.ibatis.annotations.Mapper; import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param;
import org.apache.ibatis.annotations.Select;
import java.time.LocalDateTime;
import java.util.List;
@Mapper @Mapper
public interface BrandCrawlTaskMapper extends BaseMapper<BrandCrawlTaskEntity> { public interface BrandCrawlTaskMapper extends BaseMapper<BrandCrawlTaskEntity> {
/**
* 查询超过保留期的终态品牌检测任务 id(保留期清理用,只取 id 不拉整行——历史行的
* file_paths/result_paths JSON 字段可能很大)。
*
* <p>终态集合与 {@code BrandTaskService} 的状态机一致(success/failed/cancelled),
* pending/running 绝不返回(删了正在跑的任务,结果回传会找不到任务行)。
* 时间线用 updated_at,与 BrandTaskStaleRepairSpiImpl 的陈旧判定同款口径,
* 可命中 V120 的 idx_brand_crawl_task_status_updated(status, updated_at) 索引。
*/
@Select("""
SELECT id FROM brand_crawl_tasks
WHERE status IN ('success', 'failed', 'cancelled')
AND updated_at < #{cutoff}
ORDER BY id ASC
LIMIT #{batchSize}
""")
List<Long> selectExpiredTerminalTaskIds(@Param("cutoff") LocalDateTime cutoff,
@Param("batchSize") int batchSize);
} }
@@ -18,4 +18,7 @@ public class BrandFileAggregateCacheDto {
private Boolean completed = false; private Boolean completed = false;
private List<BrandInvalidBrandDto> invalidBrands = new ArrayList<>(); private List<BrandInvalidBrandDto> invalidBrands = new ArrayList<>();
private List<String> queryFailedBrands = new ArrayList<>(); private List<String> queryFailedBrands = new ArrayList<>();
/** 已判定保留的品牌:与 invalidBrands / queryFailedBrands 一起构成「已检测品牌」,
* 失败任务的未检测品牌 = 源文件品牌 - 三者并集(部分组装时写「未检测品牌」sheet)。 */
private List<String> keptBrands = new ArrayList<>();
} }
@@ -0,0 +1,13 @@
package com.nanri.aiimage.modules.brand.model.dto;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.Data;
@Data
@Schema(description = "品牌任务中止上报请求(内部接口,由爬取方 15126 调用)。")
public class BrandTaskAbortRequest {
@Schema(description = "中止原因,会原样写入任务 error_message,前端任务列表展示该文案。",
example = "连续 8 次请求被 WIPO 限流(返回 Forbidden),已中止任务;请检查代理配置或错峰重跑")
private String errorMessage;
}
@@ -0,0 +1,97 @@
package com.nanri.aiimage.modules.brand.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.nanri.aiimage.modules.brand.mapper.BrandCrawlTaskMapper;
import com.nanri.aiimage.modules.brand.model.entity.BrandCrawlTaskEntity;
import com.nanri.aiimage.modules.task.model.dto.TaskHeartbeatRequest;
import com.nanri.aiimage.modules.task.model.vo.TaskHeartbeatVo;
import com.nanri.aiimage.modules.task.spi.BrandTaskHeartbeatSpi;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.time.LocalDateTime;
/**
* 品牌检测任务(brand_crawl_tasks)的心跳/中断实现(2026-09 全维度审查 G5)。
*
* <p>品牌任务有独立的表与状态字面量(running/pending/cancelled),原先这段逻辑散在
* {@code TaskHeartbeatService} 里并直接依赖本模块的 Mapper 与进度缓存;现整体收回本模块。
*/
@Service
@RequiredArgsConstructor
@Slf4j
public class BrandCrawlTaskHeartbeatSpi implements BrandTaskHeartbeatSpi {
private static final String MODULE_BRAND = "BRAND";
private static final String BRAND_STATUS_RUNNING = "running";
private final BrandCrawlTaskMapper brandCrawlTaskMapper;
private final BrandTaskProgressCacheService brandTaskProgressCacheService;
@Override
public TaskHeartbeatVo heartbeat(Long taskId, TaskHeartbeatRequest request) {
BrandCrawlTaskEntity task = selectTask(taskId);
if (task == null) {
return null;
}
String status = task.getStatus();
if (!BRAND_STATUS_RUNNING.equals(status)) {
log.warn("[task-heartbeat] brand task is not running taskId={} actualUserId={} status={}",
task.getId(), task.getUserId(), status);
return TaskHeartbeatVo.notAlive(MODULE_BRAND, status, "task is not running");
}
int updated = brandCrawlTaskMapper.update(null, new LambdaUpdateWrapper<BrandCrawlTaskEntity>()
.eq(BrandCrawlTaskEntity::getId, task.getId())
.eq(BrandCrawlTaskEntity::getStatus, BRAND_STATUS_RUNNING)
.set(BrandCrawlTaskEntity::getUpdatedAt, LocalDateTime.now()));
if (updated <= 0) {
BrandCrawlTaskEntity latest = brandCrawlTaskMapper.selectById(task.getId());
log.warn("[task-heartbeat] brand task heartbeat update missed taskId={} actualUserId={} status={} latestStatus={}",
task.getId(), task.getUserId(), status,
latest == null ? null : latest.getStatus());
return TaskHeartbeatVo.notAlive(MODULE_BRAND, latest == null ? status : latest.getStatus(),
"task is not running");
}
brandTaskProgressCacheService.touchHeartbeat(
task.getId(),
request == null ? null : request.getPhase(),
request == null ? null : request.getCurrent(),
request == null ? null : request.getTotal());
return TaskHeartbeatVo.alive(MODULE_BRAND, BRAND_STATUS_RUNNING);
}
@Override
public TaskHeartbeatVo markInterrupted(Long taskId, String reason) {
BrandCrawlTaskEntity task = selectTask(taskId);
if (task == null) {
return null;
}
String status = task.getStatus();
if ("running".equalsIgnoreCase(status) || "pending".equalsIgnoreCase(status)) {
// 与 file 分支统一改为条件更新:整行 updateById 会拿读取快照覆盖并发写入的字段
// (客户端重启上报与品牌任务自身状态流转同时发生时)
int updated = brandCrawlTaskMapper.update(null, new LambdaUpdateWrapper<BrandCrawlTaskEntity>()
.eq(BrandCrawlTaskEntity::getId, task.getId())
.in(BrandCrawlTaskEntity::getStatus, "running", "pending", "RUNNING", "PENDING")
.set(BrandCrawlTaskEntity::getStatus, "cancelled")
.set(BrandCrawlTaskEntity::getErrorMessage, reason)
.set(BrandCrawlTaskEntity::getUpdatedAt, LocalDateTime.now()));
if (updated > 0) {
log.warn("[task-interrupted] brand task marked cancelled by client restart taskId={} reason={}",
taskId, reason);
return TaskHeartbeatVo.notAlive(MODULE_BRAND, "cancelled", "marked cancelled");
}
}
log.info("[task-interrupted] brand task not in running/pending, skipped taskId={} status={}", taskId, status);
return TaskHeartbeatVo.notAlive(MODULE_BRAND, status, "task is not running");
}
private BrandCrawlTaskEntity selectTask(Long taskId) {
LambdaQueryWrapper<BrandCrawlTaskEntity> brandQuery = new LambdaQueryWrapper<BrandCrawlTaskEntity>()
.eq(BrandCrawlTaskEntity::getId, taskId)
.last("limit 1");
return brandCrawlTaskMapper.selectOne(brandQuery);
}
}
@@ -7,8 +7,8 @@ import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
/** /**
* BRAND 结果文件 Job Handler04 注册表)。 * BRAND 结果文件 Job Handler04 注册表)。
* 注意:resultFileUrl 解析特例(resolveResultObjectKey,无 resultId 也走) * resultFileUrl 解析特例(resolveResultObjectKey,无 resultId 也走)由本 Handler 的
* 保留在 Worker 公共路径 resolveResultFileUrlHandler 不接管 URL 解析 * resolveResultFileUrl 钩子承担(2026-09:原实现在 Worker 内直接 import 本模块,构成 task → 业务依赖)
* cleanup 为空(原 cleanupAfterSuccess 无 BRAND 分支)。 * cleanup 为空(原 cleanupAfterSuccess 无 BRAND 分支)。
*/ */
public class BrandResultFileJobHandler implements ResultFileJobHandler { public class BrandResultFileJobHandler implements ResultFileJobHandler {
@@ -32,4 +32,12 @@ public class BrandResultFileJobHandler implements ResultFileJobHandler {
brandTaskService.processResultFileJob(job); brandTaskService.processResultFileJob(job);
return true; return true;
} }
@Override
public String resolveResultFileUrl(TaskFileJobEntity job) {
if (job == null || job.getTaskId() == null) {
return null;
}
return brandTaskService.resolveResultObjectKey(job.getTaskId());
}
} }
@@ -4,11 +4,13 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.config.BrandProgressProperties; import com.nanri.aiimage.config.BrandProgressProperties;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.springframework.data.redis.core.StringRedisTemplate; import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import java.time.Duration; import java.time.Duration;
import java.time.Instant; import java.time.Instant;
import java.util.LinkedHashMap; import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map; import java.util.Map;
@Service @Service
@@ -21,6 +23,9 @@ public class BrandTaskProgressCacheService {
public static final String PHASE_FAILED = "failed"; public static final String PHASE_FAILED = "failed";
private static final Duration FINALIZE_LOCK_TTL = Duration.ofMinutes(10); private static final Duration FINALIZE_LOCK_TTL = Duration.ofMinutes(10);
/** 本实例(进程)的锁持有者标识:释放锁时用它校验"锁还是我的"。 */
private final String lockOwnerToken = java.util.UUID.randomUUID().toString();
private final StringRedisTemplate stringRedisTemplate; private final StringRedisTemplate stringRedisTemplate;
private final BrandProgressProperties brandProgressProperties; private final BrandProgressProperties brandProgressProperties;
@SuppressWarnings("unused") @SuppressWarnings("unused")
@@ -55,6 +60,8 @@ public class BrandTaskProgressCacheService {
values.put("finished_files", String.valueOf(Math.max(finishedFiles, 0))); values.put("finished_files", String.valueOf(Math.max(finishedFiles, 0)));
values.put("updated_at", now); values.put("updated_at", now);
values.put("last_heartbeat_at", now); values.put("last_heartbeat_at", now);
// 结果上报专属信号(二次判死线用):心跳/touchHeartbeat 不写它,只有结果回传才刷新
values.put("last_result_at", now);
try { try {
stringRedisTemplate.opsForHash().putAll(key, values); stringRedisTemplate.opsForHash().putAll(key, values);
stringRedisTemplate.expire(key, ttl()); stringRedisTemplate.expire(key, ttl());
@@ -129,8 +136,10 @@ public class BrandTaskProgressCacheService {
public boolean acquireFinalizeLock(Long taskId) { public boolean acquireFinalizeLock(Long taskId) {
try { try {
// value 用持有者 token(而非时间戳):释放时需要它来校验"锁还是我的",
// 否则锁因 TTL 到期被他方持有后,本线程的裸 delete 会误删他人的锁
Boolean ok = stringRedisTemplate.opsForValue() Boolean ok = stringRedisTemplate.opsForValue()
.setIfAbsent(buildFinalizeLockKey(taskId), String.valueOf(Instant.now().toEpochMilli()), FINALIZE_LOCK_TTL); .setIfAbsent(buildFinalizeLockKey(taskId), lockOwnerToken, FINALIZE_LOCK_TTL);
return Boolean.TRUE.equals(ok); return Boolean.TRUE.equals(ok);
} catch (Exception ex) { } catch (Exception ex) {
log.warn("[brand-progress-cache] acquire finalize lock degraded taskId={} msg={}", taskId, ex.getMessage()); log.warn("[brand-progress-cache] acquire finalize lock degraded taskId={} msg={}", taskId, ex.getMessage());
@@ -140,7 +149,12 @@ public class BrandTaskProgressCacheService {
public void releaseFinalizeLock(Long taskId) { public void releaseFinalizeLock(Long taskId) {
try { try {
stringRedisTemplate.delete(buildFinalizeLockKey(taskId)); // Lua 原子校验后删除(2026-09 全维度审查):此前是裸 delete,锁已过期(TTL 到期、
// 他人已持有)时会把别人的锁删掉,导致同一任务被两个线程同时收尾。
stringRedisTemplate.execute(new DefaultRedisScript<>(
"if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end",
Long.class),
List.of(buildFinalizeLockKey(taskId)), lockOwnerToken);
} catch (Exception ex) { } catch (Exception ex) {
log.warn("[brand-progress-cache] release finalize lock degraded taskId={} msg={}", taskId, ex.getMessage()); log.warn("[brand-progress-cache] release finalize lock degraded taskId={} msg={}", taskId, ex.getMessage());
} }
@@ -0,0 +1,93 @@
package com.nanri.aiimage.modules.brand.service;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.modules.brand.mapper.BrandCrawlTaskMapper;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.time.Duration;
import java.time.LocalDateTime;
import java.util.List;
/**
* 品牌检测任务(brand_crawl_tasks)的保留期清理(2026-09 审核:该表只增不删,永久累积)。
*
* <p>该表自建、不写 biz_file_task,故不在 ModuleHistoryCleanupService 的清理名单里,
* 此前没有任何删除路径。这里只查过期终态任务的 id,逐个走
* {@link BrandTaskService#deleteTask(Long)} 既有删除入口 —— 它已处理任务行删除 +
* 存储数据清理(brandTaskStorageService.deleteTaskData+ 进度缓存清理,本类不重新实现删除逻辑。
*
* <p>双节点用 job 锁保证单实例执行;每批小批量(默认 50)逐个删,避免单次跑太久占住锁。
*/
@Service
@RequiredArgsConstructor
@Slf4j
public class BrandTaskRetentionService {
/** 单轮最多处理的批次数(每批 batchSize 个任务),剩余留给下一轮。 */
static final int MAX_BATCHES_PER_RUN = 20;
private final BrandCrawlTaskMapper brandCrawlTaskMapper;
private final BrandTaskService brandTaskService;
private final DistributedJobLockService distributedJobLockService;
@Value("${aiimage.brand.task-retention-days:90}")
private int retentionDays = 90;
@Value("${aiimage.brand.task-retention-batch-size:50}")
private int retentionBatchSize = 50;
@Scheduled(cron = "${aiimage.brand.task-retention-cron:0 45 4 * * *}")
public void purgeExpiredTasks() {
int days = Math.max(1, retentionDays);
int batchSize = Math.max(1, retentionBatchSize);
LocalDateTime cutoff = LocalDateTime.now().minusDays(days);
// 单轮 20 批 × 50 个任务的删除(含存储数据清理)可能跑较久,锁 TTL 给足 30 分钟
DistributedJobLockService.LockHandle lockHandle =
distributedJobLockService.tryLock("brand:task-retention", Duration.ofMinutes(30));
if (lockHandle == null) {
log.info("[brand-retention] 任务保留清理跳过:另一实例持锁");
return;
}
try (lockHandle) {
int totalDeleted = 0;
int totalFailed = 0;
int batches = 0;
while (batches < MAX_BATCHES_PER_RUN) {
// 只取 id:历史行的 file_paths/result_paths JSON 字段可能很大
List<Long> taskIds = brandCrawlTaskMapper.selectExpiredTerminalTaskIds(cutoff, batchSize);
if (taskIds.isEmpty()) {
break;
}
batches++;
int deletedInBatch = 0;
for (Long taskId : taskIds) {
try {
brandTaskService.deleteTask(taskId);
deletedInBatch++;
} catch (Exception ex) {
// 单个任务删除失败只记日志继续:一个坏任务不能卡住整轮
log.warn("[brand-retention] 任务删除失败 taskId={} msg={}", taskId, ex.getMessage(), ex);
}
}
totalDeleted += deletedInBatch;
totalFailed += taskIds.size() - deletedInBatch;
if (deletedInBatch == 0) {
log.warn("[brand-retention] 整批任务均删除失败,提前结束本轮以免反复重试同一批");
break;
}
if (taskIds.size() < batchSize) {
break;
}
}
log.info("[brand-retention] 任务保留清理完成 cutoff={} retentionDays={} deleted={} failed={} batches={}",
cutoff, days, totalDeleted, totalFailed, batches);
} catch (Exception ex) {
log.warn("[brand-retention] 任务保留清理失败 cutoff={} msg={}", cutoff, ex.getMessage(), ex);
}
}
}
@@ -8,7 +8,9 @@ import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.exception.BusinessException; import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.common.service.DistributedJobLockService; import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.common.util.ExcelStreamReader;
import com.nanri.aiimage.config.BrandProgressProperties; import com.nanri.aiimage.config.BrandProgressProperties;
import com.nanri.aiimage.config.HttpClientPool;
import com.nanri.aiimage.config.StorageProperties; import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.modules.brand.mapper.BrandCrawlTaskMapper; import com.nanri.aiimage.modules.brand.mapper.BrandCrawlTaskMapper;
import com.nanri.aiimage.modules.brand.model.dto.BrandCrawlResultFileDto; import com.nanri.aiimage.modules.brand.model.dto.BrandCrawlResultFileDto;
@@ -38,12 +40,7 @@ import com.nanri.aiimage.modules.task.service.TaskProgressSnapshotService;
import jakarta.annotation.PreDestroy; import jakarta.annotation.PreDestroy;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.poi.ss.usermodel.Cell;
import org.apache.poi.ss.usermodel.DataFormatter;
import org.apache.poi.ss.usermodel.Row;
import org.apache.poi.ss.usermodel.Sheet; import org.apache.poi.ss.usermodel.Sheet;
import org.apache.poi.ss.usermodel.Workbook;
import org.apache.poi.ss.usermodel.WorkbookFactory;
import org.apache.poi.xssf.streaming.SXSSFWorkbook; import org.apache.poi.xssf.streaming.SXSSFWorkbook;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
@@ -53,6 +50,8 @@ import java.io.FileOutputStream;
import java.io.IOException; import java.io.IOException;
import java.io.InputStream; import java.io.InputStream;
import java.net.URI; import java.net.URI;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.file.Files; import java.nio.file.Files;
import java.time.Duration; import java.time.Duration;
import java.time.Instant; import java.time.Instant;
@@ -82,6 +81,21 @@ public class BrandTaskService {
private static final DateTimeFormatter DATETIME_FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm"); private static final DateTimeFormatter DATETIME_FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm");
private static final Duration STALE_CHECK_LOCK_TTL = Duration.ofMinutes(10); private static final Duration STALE_CHECK_LOCK_TTL = Duration.ofMinutes(10);
/** 源文件下载请求超时:比普通 API 调用宽松(源文件可能几十 MB),但必须有上限。 */
private static final Duration SOURCE_DOWNLOAD_TIMEOUT = Duration.ofMinutes(5);
/** SSRF 防护:禁止请求云元数据与环回地址(正常源文件都在自家 OSS/MinIO 域名上)。 */
private static final java.util.Set<String> BLOCKED_SOURCE_HOSTS = java.util.Set.of(
"169.254.169.254", "metadata.google.internal", "metadata", "localhost",
"127.0.0.1", "0.0.0.0", "::1", "[::1]");
private static boolean isBlockedSourceHost(String host) {
if (host == null || host.isBlank()) {
return true;
}
String normalized = host.trim().toLowerCase(java.util.Locale.ROOT);
return BLOCKED_SOURCE_HOSTS.contains(normalized) || normalized.endsWith(".localhost");
}
private static final long RESULT_SUBMIT_WAIT_MILLIS = 5 * 60 * 1000L; private static final long RESULT_SUBMIT_WAIT_MILLIS = 5 * 60 * 1000L;
private static final String STATUS_PENDING = "pending"; private static final String STATUS_PENDING = "pending";
private static final String STATUS_RUNNING = "running"; private static final String STATUS_RUNNING = "running";
@@ -92,8 +106,9 @@ public class BrandTaskService {
/** 结果文件并发上传数:OSS/MinIO 上传互不依赖,3 并发平衡收益与内存占用。 */ /** 结果文件并发上传数:OSS/MinIO 上传互不依赖,3 并发平衡收益与内存占用。 */
private static final int RESULT_UPLOAD_CONCURRENCY = 3; private static final int RESULT_UPLOAD_CONCURRENCY = 3;
private final ExecutorService resultUploadExecutor = Executors.newFixedThreadPool( // 有界队列线程池:newFixedThreadPool 用的是无界队列,任务堆积时不会拒绝、会把内存吃满
RESULT_UPLOAD_CONCURRENCY, namedThreadFactory("brand-result-upload")); private final ExecutorService resultUploadExecutor = com.nanri.aiimage.common.util.ThreadPools
.boundedFixed("brand-result-upload", RESULT_UPLOAD_CONCURRENCY);
@PreDestroy @PreDestroy
void shutdownResultUploadExecutor() { void shutdownResultUploadExecutor() {
@@ -671,23 +686,8 @@ public class BrandTaskService {
} }
brandTaskProgressCacheService.updatePhase(taskId, BrandTaskProgressCacheService.PHASE_ASSEMBLING, totalCount, totalCount); brandTaskProgressCacheService.updatePhase(taskId, BrandTaskProgressCacheService.PHASE_ASSEMBLING, totalCount, totalCount);
File outputDir = FileUtil.mkdir(FileUtil.file(storageProperties.getLocalTempDir(), "brand-result", String.valueOf(taskId)));
List<OutputEntry> outputEntries = new ArrayList<>();
try { try {
for (BrandSourceFileDto sourceFile : sourceFiles) { Map<String, Object> resultPaths = assembleAndUploadResult(taskId, strategy, sourceFiles, cachedByUrl, aggregates, false);
BrandFileAggregateCacheDto aggregate = aggregates.get(sourceFile.getFileUrl());
BrandParsedFileCacheDto cachedFile = cachedByUrl.get(sourceFile.getFileUrl());
if (cachedFile == null) {
throw new BusinessException("缺少原始缓存数据: " + sourceFile.getFileUrl());
}
File sourceLocalFile = resolveSourceFile(sourceFile);
String originalFilename = resolveOriginalFilename(sourceFile, sourceLocalFile);
File outputFile = buildNamedOutputFile(outputDir, buildResultFilename(originalFilename));
writeBrandWorkbook(outputFile, strategy, cachedFile, aggregate);
outputEntries.add(new OutputEntry(sourceLocalFile, originalFilename, outputFile, originalFilename));
}
brandTaskProgressCacheService.updatePhase(taskId, BrandTaskProgressCacheService.PHASE_UPLOADING, totalCount, totalCount);
Map<String, Object> resultPaths = buildAndUploadResult(taskId, outputEntries);
int updated = brandCrawlTaskMapper.update(null, new LambdaUpdateWrapper<BrandCrawlTaskEntity>() int updated = brandCrawlTaskMapper.update(null, new LambdaUpdateWrapper<BrandCrawlTaskEntity>()
.eq(BrandCrawlTaskEntity::getId, taskId) .eq(BrandCrawlTaskEntity::getId, taskId)
.ne(BrandCrawlTaskEntity::getStatus, STATUS_CANCELLED) .ne(BrandCrawlTaskEntity::getStatus, STATUS_CANCELLED)
@@ -721,11 +721,150 @@ public class BrandTaskService {
throw businessException; throw businessException;
} }
throw new BusinessException(ex.getMessage()); throw new BusinessException(ex.getMessage());
}
}
/**
* 组装并上传结果文件。partial=false 为成功收尾(要求全部文件分片收齐,由调用方校验);
* partial=true 为失败中止的部分组装:用已收到的分片出结果,未检测品牌单独成 sheet。
*/
private Map<String, Object> assembleAndUploadResult(Long taskId,
String strategy,
List<BrandSourceFileDto> sourceFiles,
Map<String, BrandParsedFileCacheDto> cachedByUrl,
Map<String, BrandFileAggregateCacheDto> aggregates,
boolean partial) throws IOException {
File outputDir = FileUtil.mkdir(FileUtil.file(storageProperties.getLocalTempDir(), "brand-result", String.valueOf(taskId)));
List<OutputEntry> outputEntries = new ArrayList<>();
try {
for (BrandSourceFileDto sourceFile : sourceFiles) {
BrandParsedFileCacheDto cachedFile = cachedByUrl.get(sourceFile.getFileUrl());
if (cachedFile == null) {
if (!partial) {
throw new BusinessException("缺少原始缓存数据: " + sourceFile.getFileUrl());
}
log.warn("[brand-assemble] taskId={} 原始缓存数据缺失,跳过该文件 fileUrl={}", taskId, sourceFile.getFileUrl());
continue;
}
BrandFileAggregateCacheDto aggregate = aggregates.get(sourceFile.getFileUrl());
if (aggregate == null) {
if (!partial) {
throw new BusinessException("缺少结果聚合数据: " + sourceFile.getFileUrl());
}
aggregate = new BrandFileAggregateCacheDto();
aggregate.setFileUrl(sourceFile.getFileUrl());
}
List<String> undetectedBrands = partial ? resolveUndetectedBrands(cachedFile, aggregate) : List.of();
File sourceLocalFile = resolveSourceFile(sourceFile);
String originalFilename = resolveOriginalFilename(sourceFile, sourceLocalFile);
File outputFile = buildNamedOutputFile(outputDir, buildResultFilename(originalFilename));
writeBrandWorkbook(outputFile, strategy, cachedFile, aggregate, undetectedBrands);
outputEntries.add(new OutputEntry(sourceLocalFile, originalFilename, outputFile, originalFilename));
}
if (outputEntries.isEmpty()) {
throw new BusinessException("没有可组装的结果文件");
}
brandTaskProgressCacheService.updatePhase(taskId, BrandTaskProgressCacheService.PHASE_UPLOADING,
sourceFiles.size(), sourceFiles.size());
return buildAndUploadResult(taskId, outputEntries);
} finally { } finally {
cleanupBrandResultTempFiles(outputEntries, outputDir); cleanupBrandResultTempFiles(outputEntries, outputDir);
} }
} }
/** 未检测品牌 = 源文件品牌 -(保留 ∪ 不符合品牌 ∪ 查询失败品牌);按源文件出现顺序去重。 */
private List<String> resolveUndetectedBrands(BrandParsedFileCacheDto cachedFile, BrandFileAggregateCacheDto aggregate) {
List<Map<String, Object>> sourceRows = cachedFile.getRows() == null ? List.of() : cachedFile.getRows();
Set<String> handled = new LinkedHashSet<>();
handled.addAll(normalizeBrandSet(aggregate.getKeptBrands()));
handled.addAll(normalizeBrandSetFromInvalids(aggregate.getInvalidBrands()));
handled.addAll(normalizeBrandSet(aggregate.getQueryFailedBrands()));
LinkedHashSet<String> undetected = new LinkedHashSet<>();
for (Map<String, Object> rowData : sourceRows) {
String brand = normalizeCellText(Objects.toString(rowData.getOrDefault("品牌", ""), ""));
if (!brand.isBlank() && !handled.contains(brand)) {
undetected.add(brand);
}
}
return new ArrayList<>(undetected);
}
/**
* 爬取方(15126)中止上报:任务不可能再收到剩余分片时调用(如 WIPO 连续限流熔断)。
* 用已收到的分片部分组装结果文件并落 failed + 真实原因——否则 Java 侧任务会悬挂到
* 心跳超时被判「前端长时间无响应」,且已跑出的数据因没有 result_paths 无法下载。
* 幂等:任务已是终态时直接返回,不覆盖既有结果。
*/
public Map<String, Object> abortTask(Long taskId, String errorMessage) {
if (taskId == null || taskId <= 0) {
throw new BusinessException("taskId invalid");
}
try (TaskDistributedLockService.LockHandle ignored = acquireTaskLockOrThrow(taskId, RESULT_SUBMIT_WAIT_MILLIS)) {
BrandCrawlTaskEntity task = requireTask(taskId);
String status = blankToDefault(task.getStatus(), STATUS_PENDING);
Map<String, Object> data = new LinkedHashMap<>();
data.put("taskId", taskId);
boolean hasResult = task.getResultPaths() != null && !task.getResultPaths().isBlank();
// success/cancelled 不动;failed 已有结果也不重复组装。failed 且无结果
// (如被心跳超时兜底判失败的历史任务)允许补组装——存量补救路径。
if (STATUS_SUCCESS.equalsIgnoreCase(status) || STATUS_CANCELLED.equalsIgnoreCase(status)
|| (STATUS_FAILED.equalsIgnoreCase(status) && hasResult)) {
log.info("[brand-abort] taskId={} 已是终态且无需补组装 status={} hasResult={},跳过",
taskId, status, hasResult);
data.put("status", status);
data.put("resultGenerated", false);
return data;
}
String message = blankToDefault(errorMessage,
blankToDefault(task.getErrorMessage(), "品牌检测任务已中止"));
List<BrandSourceFileDto> sourceFiles = parseSourceFiles(task.getFilePaths());
int totalCount = sourceFiles.size();
Map<String, BrandParsedFileCacheDto> cachedByUrl =
indexCachedFiles(brandTaskStorageService.getParsedPayload(taskId));
// 先从分片重建聚合再读取:缓存的 state_json 可能是旧版本(缺后加字段,
// 如 keptBrands)或与已落库分片不一致,直接读会把已检测品牌误判为未检测
for (BrandSourceFileDto sourceFile : sourceFiles) {
brandTaskStorageService.refreshFileAggregate(taskId, sourceFile.getFileUrl());
}
Map<String, BrandFileAggregateCacheDto> aggregates = brandTaskStorageService.getAllFileAggregates(taskId);
boolean hasChunk = aggregates.values().stream()
.anyMatch(item -> item != null && defaultInteger(item.getReceivedChunkCount()) > 0);
Map<String, Object> resultPaths = null;
if (hasChunk && !cachedByUrl.isEmpty() && !sourceFiles.isEmpty()) {
try {
resultPaths = assembleAndUploadResult(taskId, normalizeStrategy(task.getStrategy()),
sourceFiles, cachedByUrl, aggregates, true);
log.info("[brand-abort] taskId={} 部分结果组装完成 files={}", taskId, sourceFiles.size());
} catch (Exception ex) {
log.warn("[brand-abort] taskId={} 部分结果组装失败(仅标记失败) msg={}", taskId, ex.getMessage(), ex);
}
} else {
log.info("[brand-abort] taskId={} 无已收到分片,跳过结果组装 receivedAggregates={}", taskId, aggregates.size());
}
int finishedCount = brandTaskStorageService.countCompletedFiles(taskId);
LambdaUpdateWrapper<BrandCrawlTaskEntity> wrapper = new LambdaUpdateWrapper<BrandCrawlTaskEntity>()
.eq(BrandCrawlTaskEntity::getId, taskId)
.in(BrandCrawlTaskEntity::getStatus, STATUS_PENDING, STATUS_RUNNING, STATUS_FAILED)
.set(BrandCrawlTaskEntity::getStatus, STATUS_FAILED)
.set(BrandCrawlTaskEntity::getErrorMessage, message)
.set(BrandCrawlTaskEntity::getProgressCurrent, finishedCount)
.set(BrandCrawlTaskEntity::getProgressTotal, totalCount);
if (resultPaths != null) {
wrapper.set(BrandCrawlTaskEntity::getResultPaths, JSONUtil.toJsonStr(resultPaths));
}
int updated = brandCrawlTaskMapper.update(null, wrapper);
if (updated > 0) {
brandTaskProgressCacheService.markFailed(taskId, message);
saveBrandProgressSnapshot(taskId, STATUS_FAILED, totalCount, finishedCount, 1, message);
}
log.info("[brand-abort] taskId={} aborted updated={} resultGenerated={} finishedFiles={}/{} msg={}",
taskId, updated, resultPaths != null, finishedCount, totalCount, message);
data.put("status", STATUS_FAILED);
data.put("resultGenerated", resultPaths != null);
return data;
}
}
private void ensureNoDuplicateResultFiles(List<BrandCrawlResultFileDto> resultFiles) { private void ensureNoDuplicateResultFiles(List<BrandCrawlResultFileDto> resultFiles) {
Set<String> seen = new LinkedHashSet<>(); Set<String> seen = new LinkedHashSet<>();
for (BrandCrawlResultFileDto resultFile : resultFiles) { for (BrandCrawlResultFileDto resultFile : resultFiles) {
@@ -812,21 +951,49 @@ public class BrandTaskService {
} }
private File downloadSourceFile(String fileUrl) { private File downloadSourceFile(String fileUrl) {
URI uri;
try { try {
URI uri = URI.create(fileUrl); uri = URI.create(fileUrl);
} catch (Exception ex) {
log.warn("[brand] 源文件地址非法 fileUrl={} err={}", fileUrl, ex.getMessage());
throw new BusinessException("下载源文件失败: 地址非法");
}
// SSRF 防护(2026-09 全维度审查):fileUrl 来自请求体,若不校验则匿名调用方可让服务端
// 请求内网/云元数据地址(169.254.169.254 等)。正常源文件都落在自家 OSS/MinIO 上。
if (isBlockedSourceHost(uri.getHost())) {
log.warn("[brand] 拒绝下载疑似 SSRF 的源文件地址 host={} fileUrl={}", uri.getHost(), fileUrl);
throw new BusinessException("源文件地址不合法");
}
String filename = FileUtil.getName(uri.getPath()); String filename = FileUtil.getName(uri.getPath());
if (filename == null || filename.isBlank()) { if (filename == null || filename.isBlank()) {
filename = "brand-source.xlsx"; filename = "brand-source.xlsx";
} }
String suffix = FileUtil.extName(filename); String suffix = FileUtil.extName(filename);
File downloadDir = FileUtil.mkdir(FileUtil.file(storageProperties.getLocalTempDir(), "brand-source-download")); File downloadDir = FileUtil.mkdir(FileUtil.file(storageProperties.getLocalTempDir(), "brand-source-download"));
try {
File tempFile = Files.createTempFile(downloadDir.toPath(), "brand_", suffix.isBlank() ? "" : "." + suffix).toFile(); File tempFile = Files.createTempFile(downloadDir.toPath(), "brand_", suffix.isBlank() ? "" : "." + suffix).toFile();
try (InputStream inputStream = uri.toURL().openStream()) { // 走统一连接池 + 显式超时。此前 uri.toURL().openStream() 无任何超时(JVM 默认 0 = 无限),
// 上游半开连接会把 Tomcat 工作线程无限占用;超时值比普通 API 调用宽松(源文件可能几十 MB)
HttpRequest downloadRequest = HttpRequest.newBuilder(uri)
.timeout(SOURCE_DOWNLOAD_TIMEOUT)
.GET()
.build();
HttpResponse<InputStream> response = HttpClientPool.sharedHttpClient()
.send(downloadRequest, HttpResponse.BodyHandlers.ofInputStream());
if (response.statusCode() / 100 != 2) {
log.warn("[brand] 下载源文件返回非 2xx fileUrl={} status={}", fileUrl, response.statusCode());
throw new BusinessException("下载源文件失败: HTTP " + response.statusCode());
}
try (InputStream inputStream = response.body()) {
FileUtil.writeFromStream(inputStream, tempFile); FileUtil.writeFromStream(inputStream, tempFile);
} }
return tempFile; return tempFile;
} catch (BusinessException ex) {
throw ex;
} catch (Exception ex) { } catch (Exception ex) {
throw new BusinessException("下载源文件失败"); // 此前该 catch 既不记日志也不带 cause,线上无法定位
log.warn("[brand] 下载源文件失败 fileUrl={} err={}", fileUrl, ex.getMessage(), ex);
throw new BusinessException("下载源文件失败", ex);
} }
} }
@@ -835,54 +1002,60 @@ public class BrandTaskService {
} }
private ParsedBrandFile parseBrandFile(File inputFile) { private ParsedBrandFile parseBrandFile(File inputFile) {
DataFormatter formatter = new DataFormatter(); // 2026-09 全维度审查 C5:改逐行流式解析(ExcelStreamReader → EasyExcel SAX)。
try (FileInputStream fis = new FileInputStream(inputFile); Workbook workbook = WorkbookFactory.create(fis)) { // 此前 WorkbookFactory.create 把整表读成 DOM,品牌源文件(几十万行)会整表驻留堆内存。
Sheet sheet = workbook.getSheetAt(0); // 解析结果(columns/rows/uniqueBrands/行号)与错误文案不变,行号沿用 sheet 绝对 0 基行号 +1。
Row headerRow = sheet.getRow(0); if (!hasExcelMagic(inputFile)) {
if (headerRow == null) { log.warn("[brand] 源文件不是 Excel(疑似 CSV/文本),拒绝解析 file={}", inputFile.getName());
throw new BusinessException("Excel 表头为空"); throw new BusinessException("读取 Excel 失败");
} }
List<String> columns = extractHeaders(headerRow, formatter); BrandFileSheetContext context = new BrandFileSheetContext();
if (columns.isEmpty()) { try {
throw new BusinessException("未读取到有效表头"); ExcelStreamReader.readFirstSheet(inputFile, context);
} } catch (BusinessException ex) {
Map<String, Integer> headerIndexes = buildHeaderIndexes(headerRow, formatter, columns); throw ex;
List<Map<String, Object>> rows = new ArrayList<>();
Set<String> uniqueBrands = new LinkedHashSet<>();
for (int rowNum = 1; rowNum <= sheet.getLastRowNum(); rowNum++) {
Row row = sheet.getRow(rowNum);
if (row == null) {
continue;
}
Map<String, Object> rowData = new LinkedHashMap<>();
for (String column : columns) {
Integer index = headerIndexes.get(column);
String value = index == null ? "" : normalizeCellText(formatter.formatCellValue(row.getCell(index)));
rowData.put(column, value);
}
String brand = normalizeCellText(Objects.toString(rowData.getOrDefault("品牌", ""), ""));
if (brand.isBlank()) {
rowData.put("__rowIndex", rowNum + 1);
rows.add(rowData);
continue;
}
if (uniqueBrands.add(brand)) {
rowData.put("__rowIndex", rowNum + 1);
rows.add(rowData);
}
}
return new ParsedBrandFile(sheet.getSheetName(), columns, rows, new ArrayList<>(uniqueBrands));
} catch (IOException ex) { } catch (IOException ex) {
throw new BusinessException("读取 Excel 失败"); throw new BusinessException("读取 Excel 失败");
} }
return context.finish();
} }
private List<String> extractHeaders(Row headerRow, DataFormatter formatter) { /**
* 文件魔数校验:xlsx=PK(zip)、xls=OLE2(CFD)。
* EasyExcel 会把非 zip 文本当 CSV 静默解析成功,而旧 POI WorkbookFactory 只认 xlsx/xls(垃圾文件直接失败),
* 这里保持「非 Excel 源文件 → 读取失败」的语义(与 SimilarAsinExcelParser 同口径)。
*/
private boolean hasExcelMagic(File file) {
byte[] head = new byte[8];
try (InputStream inputStream = new FileInputStream(file)) {
int n = inputStream.readNBytes(head, 0, head.length);
boolean isZip = n >= 4 && head[0] == 'P' && head[1] == 'K' && (head[2] == 3 || head[2] == 5 || head[2] == 7);
boolean isOle2 = n >= 8
&& (head[0] & 0xFF) == 0xD0 && (head[1] & 0xFF) == 0xCF
&& (head[2] & 0xFF) == 0x11 && (head[3] & 0xFF) == 0xE0;
return isZip || isOle2;
} catch (IOException ex) {
return false;
}
}
/** 品牌源文件流式解析上下文:表头解析 + 按品牌去重累积结果行(内存不随整表 DOM 放大)。 */
private final class BrandFileSheetContext implements ExcelStreamReader.SheetRowHandler {
private final List<Map<String, Object>> rows = new ArrayList<>();
private final Set<String> uniqueBrands = new LinkedHashSet<>();
private List<String> columns;
private Map<String, Integer> headerIndexes;
private String sheetName = "";
@Override
public void onHeader(String sheetName, Integer sheetNo, Map<Integer, String> headerMap) {
this.sheetName = sheetName == null ? "" : sheetName;
int lastColumnCount = lastColumnCount(headerMap);
List<String> headers = new ArrayList<>(); List<String> headers = new ArrayList<>();
Set<String> seen = new LinkedHashSet<>(); Set<String> seen = new LinkedHashSet<>();
for (int i = 0; i < headerRow.getLastCellNum(); i++) { for (int i = 0; i < lastColumnCount; i++) {
Cell cell = headerRow.getCell(i); String value = normalizeHeaderValue(headerMap.get(i));
String value = normalizeHeaderValue(cell == null ? null : formatter.formatCellValue(cell));
if (value.isBlank() || seen.contains(value)) { if (value.isBlank() || seen.contains(value)) {
continue; continue;
} }
@@ -892,24 +1065,70 @@ public class BrandTaskService {
break; break;
} }
} }
return headers; if (headers.isEmpty()) {
throw new BusinessException("未读取到有效表头");
} }
Map<String, Integer> indexes = new LinkedHashMap<>();
private Map<String, Integer> buildHeaderIndexes(Row headerRow, DataFormatter formatter, List<String> columns) { Set<String> allowed = new LinkedHashSet<>(headers);
Map<String, Integer> headerIndexes = new LinkedHashMap<>(); for (int i = 0; i < lastColumnCount; i++) {
Set<String> allowed = new LinkedHashSet<>(columns); String value = normalizeHeaderValue(headerMap.get(i));
for (int i = 0; i < headerRow.getLastCellNum(); i++) { if (value.isBlank() || indexes.containsKey(value) || !allowed.contains(value)) {
Cell cell = headerRow.getCell(i);
String value = normalizeHeaderValue(cell == null ? null : formatter.formatCellValue(cell));
if (value.isBlank() || headerIndexes.containsKey(value) || !allowed.contains(value)) {
continue; continue;
} }
headerIndexes.put(value, i); indexes.put(value, i);
if ("缩略图地址8".equals(value)) { if ("缩略图地址8".equals(value)) {
break; break;
} }
} }
return headerIndexes; this.columns = headers;
this.headerIndexes = indexes;
}
@Override
public void onRow(String sheetName, Integer sheetNo, int rowIndex,
Map<Integer, String> headerMap, Map<Integer, String> rowMap) {
if (columns == null) {
// 表头行整行为空时 EasyExcel 不上报表头回调:与旧实现 headerRow == null 一致
throw new BusinessException("Excel 表头为空");
}
Map<String, Object> rowData = new LinkedHashMap<>();
for (String column : columns) {
Integer index = headerIndexes.get(column);
String value = index == null ? "" : normalizeCellText(rowMap.get(index));
rowData.put(column, value);
}
String brand = normalizeCellText(Objects.toString(rowData.getOrDefault("品牌", ""), ""));
if (brand.isBlank()) {
rowData.put("__rowIndex", rowIndex + 1);
rows.add(rowData);
return;
}
if (uniqueBrands.add(brand)) {
rowData.put("__rowIndex", rowIndex + 1);
rows.add(rowData);
}
}
private ParsedBrandFile finish() {
if (columns == null) {
// 空表(无任何行)或表头行缺失:与旧实现 headerRow == null 一致
throw new BusinessException("Excel 表头为空");
}
return new ParsedBrandFile(sheetName, columns, rows, new ArrayList<>(uniqueBrands));
}
private int lastColumnCount(Map<Integer, String> headerMap) {
if (headerMap == null || headerMap.isEmpty()) {
return 0;
}
int lastColumnCount = 0;
for (Integer columnIndex : headerMap.keySet()) {
if (columnIndex != null && columnIndex >= lastColumnCount) {
lastColumnCount = columnIndex + 1;
}
}
return lastColumnCount;
}
} }
private String normalizeHeaderValue(String value) { private String normalizeHeaderValue(String value) {
@@ -939,7 +1158,8 @@ public class BrandTaskService {
private void writeBrandWorkbook(File outputFile, private void writeBrandWorkbook(File outputFile,
String strategy, String strategy,
BrandParsedFileCacheDto cachedFile, BrandParsedFileCacheDto cachedFile,
BrandFileAggregateCacheDto resultFile) throws IOException { BrandFileAggregateCacheDto resultFile,
List<String> undetectedBrands) throws IOException {
String actualStrategy = normalizeStrategy(strategy); String actualStrategy = normalizeStrategy(strategy);
SXSSFWorkbook workbook = new SXSSFWorkbook(200); SXSSFWorkbook workbook = new SXSSFWorkbook(200);
workbook.setCompressTempFiles(true); workbook.setCompressTempFiles(true);
@@ -953,6 +1173,10 @@ public class BrandTaskService {
} }
Set<String> invalidBrandSet = normalizeBrandSetFromInvalids(resultFile.getInvalidBrands()); Set<String> invalidBrandSet = normalizeBrandSetFromInvalids(resultFile.getInvalidBrands());
// 未检测品牌(任务中止、分片没到齐):主 sheet 剔除这些行、整体挪到独立 sheet,
// 避免用户把「没查过」的行误当成「已通过检测」上架
Set<String> undetectedBrandSet = normalizeBrandSet(undetectedBrands);
List<Map<String, Object>> undetectedRows = new ArrayList<>();
List<Map<String, Object>> sourceRows = cachedFile.getRows() == null ? List.of() : cachedFile.getRows(); List<Map<String, Object>> sourceRows = cachedFile.getRows() == null ? List.of() : cachedFile.getRows();
int writeRowIndex = 1; int writeRowIndex = 1;
for (Map<String, Object> rowData : sourceRows) { for (Map<String, Object> rowData : sourceRows) {
@@ -960,6 +1184,10 @@ public class BrandTaskService {
if (!brand.isBlank() && invalidBrandSet.contains(brand)) { if (!brand.isBlank() && invalidBrandSet.contains(brand)) {
continue; continue;
} }
if (!brand.isBlank() && undetectedBrandSet.contains(brand)) {
undetectedRows.add(rowData);
continue;
}
var row = mainSheet.createRow(writeRowIndex++); var row = mainSheet.createRow(writeRowIndex++);
for (int colIndex = 0; colIndex < columns.size(); colIndex++) { for (int colIndex = 0; colIndex < columns.size(); colIndex++) {
String column = columns.get(colIndex); String column = columns.get(colIndex);
@@ -967,6 +1195,23 @@ public class BrandTaskService {
} }
} }
if (!undetectedRows.isEmpty()) {
var undetectedSheet = workbook.createSheet("未检测品牌");
var undetectedHeader = undetectedSheet.createRow(0);
for (int i = 0; i < columns.size(); i++) {
undetectedHeader.createCell(i).setCellValue(columns.get(i));
}
for (int i = 0; i < undetectedRows.size(); i++) {
Map<String, Object> rowData = undetectedRows.get(i);
var row = undetectedSheet.createRow(i + 1);
for (int colIndex = 0; colIndex < columns.size(); colIndex++) {
String column = columns.get(colIndex);
row.createCell(colIndex).setCellValue(Objects.toString(rowData.getOrDefault(column, ""), ""));
}
}
applyBrandSheetWidths(undetectedSheet, columns.size());
}
var invalidSheet = workbook.createSheet("不符合品牌"); var invalidSheet = workbook.createSheet("不符合品牌");
var invalidHeader = invalidSheet.createRow(0); var invalidHeader = invalidSheet.createRow(0);
invalidHeader.createCell(0).setCellValue("品牌"); invalidHeader.createCell(0).setCellValue("品牌");
@@ -1292,6 +1537,55 @@ public class BrandTaskService {
failStaleRunningTask(task.getId(), "前端长时间无响应,任务已自动失败"); failStaleRunningTask(task.getId(), "前端长时间无响应,任务已自动失败");
} }
} }
failNoUploadStaleRunningTasks();
}
}
/**
* 二次判死:前端心跳正常但连续 N 分钟无结果上报(治「心跳续命」的假活任务)。
*
* <p>既有心跳线候选条件是 updated_at/last_heartbeat_at 陈旧,而前端心跳会持续刷新它们——
* 主线程卡死时心跳线程照发,任务永远命不中(与生产 28131 同型缺口)。
* 本线候选取「心跳新鲜 + 创建超过 N 分钟」,判据用结果上报时写入 progress hash 的
* last_result_at(仅 saveProgressFromResult 写);从未上报(无该字段)跳过,避免误杀首批较慢的正常任务。
*/
private void failNoUploadStaleRunningTasks() {
long minutes = brandProgressProperties.getNoResultUploadTimeoutMinutes();
if (minutes <= 0) {
return;
}
LocalDateTime heartbeatThreshold = LocalDateTime.now()
.minusMinutes(brandProgressProperties.getHeartbeatTimeoutMinutes());
LocalDateTime cutoff = LocalDateTime.now().minusMinutes(minutes);
List<BrandCrawlTaskEntity> runningTasks = brandCrawlTaskMapper.selectList(new LambdaQueryWrapper<BrandCrawlTaskEntity>()
.eq(BrandCrawlTaskEntity::getStatus, STATUS_RUNNING)
.ge(BrandCrawlTaskEntity::getUpdatedAt, heartbeatThreshold)
.lt(BrandCrawlTaskEntity::getCreatedAt, cutoff));
for (BrandCrawlTaskEntity task : runningTasks) {
TaskDistributedLockService.LockHandle taskLockHandle = acquireTaskLock(task.getId(), 0L);
if (taskLockHandle == null) {
continue;
}
try (taskLockHandle) {
Map<Object, Object> progress = brandTaskProgressCacheService.getProgress(task.getId());
long lastResultAt = 0L;
try {
lastResultAt = Long.parseLong(String.valueOf(progress.getOrDefault("last_result_at", "0")));
} catch (Exception ignored) {
}
if (lastResultAt <= 0L) {
// 从未上报结果:保守跳过(首批可能较慢)
continue;
}
LocalDateTime lastResult = LocalDateTime.ofInstant(Instant.ofEpochMilli(lastResultAt), ZoneId.systemDefault());
if (lastResult.isAfter(cutoff)) {
continue;
}
log.warn("[brand-stale-check] no-upload failing taskId={} lastResultAt={} timeoutMinutes={}",
task.getId(), lastResult, minutes);
failStaleRunningTask(task.getId(),
"连续 " + minutes + " 分钟无结果回传,疑似客户端任务卡死,任务已自动失败(最后结果上报 " + lastResult + "");
}
} }
} }
@@ -0,0 +1,47 @@
package com.nanri.aiimage.modules.brand.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.nanri.aiimage.modules.brand.mapper.BrandCrawlTaskMapper;
import com.nanri.aiimage.modules.brand.model.entity.BrandCrawlTaskEntity;
import com.nanri.aiimage.modules.task.spi.BrandTaskStaleRepairSpi;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
import java.time.LocalDateTime;
import java.util.List;
/**
* brand_crawl_tasks 的陈旧修复实现(2026-09 全维度审查 G5:逻辑从 task 模块收回本模块)。
*/
@Service
@RequiredArgsConstructor
public class BrandTaskStaleRepairSpiImpl implements BrandTaskStaleRepairSpi {
private static final String STATUS_PENDING = "pending";
private static final String STATUS_RUNNING = "running";
private static final String STATUS_FAILED = "failed";
private final BrandCrawlTaskMapper brandCrawlTaskMapper;
@Override
public List<Long> failStaleBrandTasks(LocalDateTime cutoff, int limit) {
List<BrandCrawlTaskEntity> stale = brandCrawlTaskMapper.selectList(
new LambdaQueryWrapper<BrandCrawlTaskEntity>()
.select(BrandCrawlTaskEntity::getId)
.in(BrandCrawlTaskEntity::getStatus, STATUS_PENDING, STATUS_RUNNING)
.lt(BrandCrawlTaskEntity::getUpdatedAt, cutoff)
.last("limit " + Math.max(1, limit)));
if (stale.isEmpty()) {
return List.of();
}
List<Long> ids = stale.stream().map(BrandCrawlTaskEntity::getId).toList();
brandCrawlTaskMapper.update(null, new LambdaUpdateWrapper<BrandCrawlTaskEntity>()
.in(BrandCrawlTaskEntity::getId, ids)
.in(BrandCrawlTaskEntity::getStatus, STATUS_PENDING, STATUS_RUNNING)
.lt(BrandCrawlTaskEntity::getUpdatedAt, cutoff)
.set(BrandCrawlTaskEntity::getStatus, STATUS_FAILED)
.set(BrandCrawlTaskEntity::getErrorMessage, "任务长期无心跳,已自动失败"));
return ids;
}
}
@@ -13,9 +13,12 @@ import com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity; import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService; import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.dao.DuplicateKeyException; import org.springframework.dao.DuplicateKeyException;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional; import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.support.TransactionSynchronization;
import org.springframework.transaction.support.TransactionSynchronizationManager;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.security.MessageDigest; import java.security.MessageDigest;
@@ -27,6 +30,7 @@ import java.util.Map;
@Service @Service
@RequiredArgsConstructor @RequiredArgsConstructor
@Slf4j
public class BrandTaskStorageService { public class BrandTaskStorageService {
private static final String MODULE_TYPE = "BRAND"; private static final String MODULE_TYPE = "BRAND";
@@ -236,6 +240,13 @@ public class BrandTaskStorageService {
return new ChunkStoreResult(false, completed, countCompletedFiles(taskId), aggregate); return new ChunkStoreResult(false, completed, countCompletedFiles(taskId), aggregate);
} }
/**
* 删除任务的全部范围/分片数据。
*
* <p>事务边界:远端载荷删除移到事务提交后执行 —— 单次删除超时 120s、重试 3 次,
* 放在 DB 事务里会让连接被长时间占用(项目在 TaskScopePayloadStorageService 已写明该约束)。
* 提交后再删还有个好处:引用计数查询看到的是删除完成后的 DB 状态,判断更准确。
*/
@Transactional @Transactional
public void deleteTaskData(Long taskId) { public void deleteTaskData(Long taskId) {
if (taskId == null || taskId <= 0) { if (taskId == null || taskId <= 0) {
@@ -245,27 +256,68 @@ public class BrandTaskStorageService {
.select(TaskScopeStateEntity::getParsedPayloadJson, TaskScopeStateEntity::getStateJson) .select(TaskScopeStateEntity::getParsedPayloadJson, TaskScopeStateEntity::getStateJson)
.eq(TaskScopeStateEntity::getTaskId, taskId) .eq(TaskScopeStateEntity::getTaskId, taskId)
.eq(TaskScopeStateEntity::getModuleType, MODULE_TYPE)); .eq(TaskScopeStateEntity::getModuleType, MODULE_TYPE));
if (states != null) {
for (TaskScopeStateEntity state : states) {
transientPayloadStorageService.deletePayloadIfPresent(state.getParsedPayloadJson());
transientPayloadStorageService.deletePayloadIfPresent(state.getStateJson());
}
}
List<TaskChunkEntity> chunks = taskChunkMapper.selectList(new LambdaQueryWrapper<TaskChunkEntity>() List<TaskChunkEntity> chunks = taskChunkMapper.selectList(new LambdaQueryWrapper<TaskChunkEntity>()
.select(TaskChunkEntity::getPayloadJson) .select(TaskChunkEntity::getPayloadJson)
.eq(TaskChunkEntity::getTaskId, taskId) .eq(TaskChunkEntity::getTaskId, taskId)
.eq(TaskChunkEntity::getModuleType, MODULE_TYPE)); .eq(TaskChunkEntity::getModuleType, MODULE_TYPE));
if (chunks != null) {
for (TaskChunkEntity chunk : chunks) {
transientPayloadStorageService.deletePayloadIfPresent(chunk.getPayloadJson());
}
}
taskChunkMapper.delete(new LambdaQueryWrapper<TaskChunkEntity>() taskChunkMapper.delete(new LambdaQueryWrapper<TaskChunkEntity>()
.eq(TaskChunkEntity::getTaskId, taskId) .eq(TaskChunkEntity::getTaskId, taskId)
.eq(TaskChunkEntity::getModuleType, MODULE_TYPE)); .eq(TaskChunkEntity::getModuleType, MODULE_TYPE));
taskScopeStateMapper.delete(new LambdaQueryWrapper<TaskScopeStateEntity>() taskScopeStateMapper.delete(new LambdaQueryWrapper<TaskScopeStateEntity>()
.eq(TaskScopeStateEntity::getTaskId, taskId) .eq(TaskScopeStateEntity::getTaskId, taskId)
.eq(TaskScopeStateEntity::getModuleType, MODULE_TYPE)); .eq(TaskScopeStateEntity::getModuleType, MODULE_TYPE));
deletePayloadsAfterCommit(states, chunks, taskId);
}
/** 事务提交后逐条删远端载荷;无事务时立即执行(与 TaskResultItemService 同款兜底)。 */
private void deletePayloadsAfterCommit(List<TaskScopeStateEntity> states,
List<TaskChunkEntity> chunks,
Long taskId) {
List<String> payloads = new ArrayList<>();
if (states != null) {
for (TaskScopeStateEntity state : states) {
if (state == null) {
continue;
}
if (state.getParsedPayloadJson() != null && !state.getParsedPayloadJson().isBlank()) {
payloads.add(state.getParsedPayloadJson());
}
if (state.getStateJson() != null && !state.getStateJson().isBlank()) {
payloads.add(state.getStateJson());
}
}
}
if (chunks != null) {
for (TaskChunkEntity chunk : chunks) {
if (chunk != null && chunk.getPayloadJson() != null && !chunk.getPayloadJson().isBlank()) {
payloads.add(chunk.getPayloadJson());
}
}
}
if (payloads.isEmpty()) {
return;
}
if (TransactionSynchronizationManager.isSynchronizationActive()) {
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
@Override
public void afterCommit() {
deletePayloadsNow(payloads, taskId);
}
});
return;
}
deletePayloadsNow(payloads, taskId);
}
private void deletePayloadsNow(List<String> payloads, Long taskId) {
for (String payload : payloads) {
try {
transientPayloadStorageService.deletePayloadIfPresent(payload);
} catch (Exception ex) {
// 事务已提交,远端删除失败只记日志,不影响任务数据清理结果
log.warn("[brand-storage] 载荷删除失败 taskId={} msg={}", taskId, ex.getMessage());
}
}
} }
private void saveAggregate(Long taskId, private void saveAggregate(Long taskId,
@@ -325,6 +377,7 @@ public class BrandTaskStorageService {
aggregate.setCompleted(false); aggregate.setCompleted(false);
aggregate.setInvalidBrands(new ArrayList<>()); aggregate.setInvalidBrands(new ArrayList<>());
aggregate.setQueryFailedBrands(new ArrayList<>()); aggregate.setQueryFailedBrands(new ArrayList<>());
aggregate.setKeptBrands(new ArrayList<>());
return aggregate; return aggregate;
} }
@@ -362,6 +415,12 @@ public class BrandTaskStorageService {
if (file.getQueryFailedBrands() != null && !file.getQueryFailedBrands().isEmpty()) { if (file.getQueryFailedBrands() != null && !file.getQueryFailedBrands().isEmpty()) {
aggregate.getQueryFailedBrands().addAll(file.getQueryFailedBrands()); aggregate.getQueryFailedBrands().addAll(file.getQueryFailedBrands());
} }
if (file.getKeptRows() != null && !file.getKeptRows().isEmpty()) {
if (aggregate.getKeptBrands() == null) {
aggregate.setKeptBrands(new ArrayList<>());
}
aggregate.getKeptBrands().addAll(file.getKeptRows());
}
} }
private BrandFileAggregateCacheDto rebuildAggregate(Long taskId, String scopeKey, String scopeHash) { private BrandFileAggregateCacheDto rebuildAggregate(Long taskId, String scopeKey, String scopeHash) {
@@ -379,6 +438,7 @@ public class BrandTaskStorageService {
aggregate.setCompleted(false); aggregate.setCompleted(false);
aggregate.setInvalidBrands(new ArrayList<>()); aggregate.setInvalidBrands(new ArrayList<>());
aggregate.setQueryFailedBrands(new ArrayList<>()); aggregate.setQueryFailedBrands(new ArrayList<>());
aggregate.setKeptBrands(new ArrayList<>());
} }
return aggregate; return aggregate;
} }
@@ -466,11 +526,8 @@ public class BrandTaskStorageService {
try { try {
MessageDigest digest = MessageDigest.getInstance("SHA-256"); MessageDigest digest = MessageDigest.getInstance("SHA-256");
byte[] bytes = digest.digest(normalize(value).getBytes(StandardCharsets.UTF_8)); byte[] bytes = digest.digest(normalize(value).getBytes(StandardCharsets.UTF_8));
StringBuilder sb = new StringBuilder(bytes.length * 2); // HexFormat 替代逐字节 String.format("%02x"):后者每次走 32 次 Formatter(热路径按行调用)
for (byte b : bytes) { return java.util.HexFormat.of().formatHex(bytes);
sb.append(String.format("%02x", b));
}
return sb.toString();
} catch (Exception ex) { } catch (Exception ex) {
throw new IllegalStateException("failed to hash brand scope", ex); throw new IllegalStateException("failed to hash brand scope", ex);
} }
@@ -29,6 +29,7 @@ import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController; import org.springframework.web.bind.annotation.RestController;
import com.nanri.aiimage.modules.task.model.dto.TaskProgressLightRequest; import com.nanri.aiimage.modules.task.model.dto.TaskProgressLightRequest;
import com.nanri.aiimage.modules.task.model.vo.TaskProgressLightBatchVo; import com.nanri.aiimage.modules.task.model.vo.TaskProgressLightBatchVo;
import com.nanri.aiimage.modules.task.service.TaskProgressOwnershipSupport;
@RestController @RestController
@RequiredArgsConstructor @RequiredArgsConstructor
@@ -37,6 +38,7 @@ import com.nanri.aiimage.modules.task.model.vo.TaskProgressLightBatchVo;
public class CollectDataController { public class CollectDataController {
private final CollectDataService service; private final CollectDataService service;
private final TaskProgressOwnershipSupport progressOwnershipSupport;
@PostMapping("/parse") @PostMapping("/parse")
@Operation(summary = "解析 Excel 并创建任务", description = "解析上传后的 Excel 文件,按行入库到 biz_collect_data_item,并保存任务筛选条件。任务初始状态为 PENDING。") @Operation(summary = "解析 Excel 并创建任务", description = "解析上传后的 Excel 文件,按行入库到 biz_collect_data_item,并保存任务筛选条件。任务初始状态为 PENDING。")
@@ -112,15 +114,18 @@ public class CollectDataController {
@PostMapping("/tasks/progress/batch") @PostMapping("/tasks/progress/batch")
@Operation(summary = "批量查询任务进度") @Operation(summary = "批量查询任务进度")
public ApiResponse<CollectDataTaskBatchVo> progressBatch(@Valid @RequestBody CollectDataTaskBatchRequest request) { public ApiResponse<CollectDataTaskBatchVo> progressBatch(@Valid @RequestBody CollectDataTaskBatchRequest request) {
return ApiResponse.success(service.progressBatch(request.getTaskIds())); // 归属过滤:只查询属于该用户的任务(userId 未传时保持原行为,兼容未升级调用方)
return ApiResponse.success(service.progressBatch(progressOwnershipSupport
.filterOwnedTaskIds(request.getTaskIds(), request.getUserId())));
} }
@PostMapping("/tasks/progress/light") @PostMapping("/tasks/progress/light")
@Operation(summary = "批量查询任务轻量进度(前端轮询专用)", @Operation(summary = "批量查询任务轻量进度(前端轮询专用)",
description = "与 progress/batch 同输入,只返回轻量字段(taskId/status/statusCode/fileStatus/fileError/fileReady/updatedAt)," description = "与 progress/batch 同输入,只返回轻量字段(taskId/status/statusCode/fileStatus/fileError/fileReady/updatedAt/rowsVersion),"
+ "查明细/result 行,响应体更小;旧 progress/batch 端点保留不动。") + "返回行明细内容(仅多一次结果行版本聚合查询),响应体更小;旧 progress/batch 端点保留不动。")
public ApiResponse<TaskProgressLightBatchVo> progressLight(@Valid @RequestBody TaskProgressLightRequest request) { public ApiResponse<TaskProgressLightBatchVo> progressLight(@Valid @RequestBody TaskProgressLightRequest request) {
return ApiResponse.success(service.progressLight(request.getTaskIds())); // 透传 userId:此前传 null = 不过滤归属,匿名遍历 taskId 即可读他人任务状态(2026-09 审查)
return ApiResponse.success(service.progressLight(request.getTaskIds(), request.getUserId()));
} }
@DeleteMapping("/tasks/{taskId}") @DeleteMapping("/tasks/{taskId}")
@@ -12,4 +12,8 @@ public class CollectDataTaskBatchRequest {
@NotEmpty @NotEmpty
@Schema(description = "任务 ID 列表", requiredMode = Schema.RequiredMode.REQUIRED) @Schema(description = "任务 ID 列表", requiredMode = Schema.RequiredMode.REQUIRED)
private List<Long> taskIds; private List<Long> taskIds;
/** 当前用户 ID(可省略;传入时仅返回该用户的任务,用于进度接口归属过滤)。 */
@Schema(description = "当前用户 ID(可省略;传入时仅返回该用户的任务)", example = "1")
private Long userId;
} }
@@ -0,0 +1,29 @@
package com.nanri.aiimage.modules.collectdata.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.nanri.aiimage.modules.collectdata.mapper.CollectDataItemMapper;
import com.nanri.aiimage.modules.collectdata.model.entity.CollectDataItemEntity;
import com.nanri.aiimage.modules.task.spi.CollectDataItemCleanupSpi;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
import java.util.List;
/**
* 采集明细行的历史清理实现(2026-09 全维度审查 G5:逻辑从 task 模块收回本模块)。
*/
@Service
@RequiredArgsConstructor
public class CollectDataItemCleanupSpiImpl implements CollectDataItemCleanupSpi {
private final CollectDataItemMapper collectDataItemMapper;
@Override
public int deleteItemsByTaskIds(List<Long> taskIds) {
if (taskIds == null || taskIds.isEmpty()) {
return 0;
}
return collectDataItemMapper.delete(new LambdaQueryWrapper<CollectDataItemEntity>()
.in(CollectDataItemEntity::getTaskId, taskIds));
}
}
@@ -4,9 +4,11 @@ import cn.hutool.core.util.IdUtil;
import cn.hutool.core.io.FileUtil; import cn.hutool.core.io.FileUtil;
import cn.hutool.crypto.digest.DigestUtil; import cn.hutool.crypto.digest.DigestUtil;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper; import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.fasterxml.jackson.core.type.TypeReference; import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.exception.BusinessCodes;
import com.nanri.aiimage.common.exception.BusinessException; import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.common.util.ExcelStreamReader; import com.nanri.aiimage.common.util.ExcelStreamReader;
import com.nanri.aiimage.modules.collectdata.mapper.CollectDataCountryPrefMapper; import com.nanri.aiimage.modules.collectdata.mapper.CollectDataCountryPrefMapper;
@@ -49,6 +51,7 @@ import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskResultItemMapper; import com.nanri.aiimage.modules.task.mapper.TaskResultItemMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper; import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.dto.TaskHeartbeatRequest; import com.nanri.aiimage.modules.task.model.dto.TaskHeartbeatRequest;
import com.nanri.aiimage.modules.task.model.dto.TaskScopeLastChunkDto;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity; import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity; import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity; import com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity;
@@ -104,8 +107,8 @@ public class CollectDataService {
*/ */
private static final String LEGACY_MODULE_TYPE = "collectdata"; private static final String LEGACY_MODULE_TYPE = "collectdata";
public TaskProgressLightBatchVo progressLight(List<Long> taskIds) { public TaskProgressLightBatchVo progressLight(List<Long> taskIds, Long userId) {
return taskProgressLightAssembler.assemble(MODULE_TYPE, null, taskIds); return taskProgressLightAssembler.assemble(MODULE_TYPE, userId, taskIds);
} }
public static final int DEFAULT_PAGE_SIZE = 50; public static final int DEFAULT_PAGE_SIZE = 50;
@@ -177,6 +180,10 @@ public class CollectDataService {
@Value("${aiimage.collect-data.stale-timeout-minutes:30}") @Value("${aiimage.collect-data.stale-timeout-minutes:30}")
private long staleTimeoutMinutes; private long staleTimeoutMinutes;
/** 二次判死线(心跳正常但连续 N 分钟无结果分片上报)阈值,分钟;<=0 关闭本线。 */
@Value("${aiimage.collect-data.no-result-upload-timeout-minutes:180}")
private long noResultUploadTimeoutMinutes;
@Value("${aiimage.collect-data.max-source-file-bytes:0}") @Value("${aiimage.collect-data.max-source-file-bytes:0}")
private Long maxSourceFileBytes; private Long maxSourceFileBytes;
@@ -379,42 +386,73 @@ public class CollectDataService {
@Transactional @Transactional
public void activateTask(Long taskId, Long userId) { public void activateTask(Long taskId, Long userId) {
FileTaskEntity task = requireTask(taskId, userId); FileTaskEntity task = requireTask(taskId, userId);
if (STATUS_SUCCESS.equals(task.getStatus()) || STATUS_FAILED.equals(task.getStatus())) { // 只允许 PENDING→RUNNING(条件更新):既堵住 TOCTOU/fail 抢先标 FAILED 后被整行
// updateById 复活成 RUNNING),又与客户端「兜底拉取」的原子认领互斥,谁先翻转谁执行
int updated = fileTaskMapper.update(null, new LambdaUpdateWrapper<FileTaskEntity>()
.eq(FileTaskEntity::getId, task.getId())
.eq(FileTaskEntity::getStatus, STATUS_PENDING)
.set(FileTaskEntity::getStatus, STATUS_RUNNING)
.set(FileTaskEntity::getUpdatedAt, LocalDateTime.now()));
if (updated == 0) {
FileTaskEntity latest = fileTaskMapper.selectById(task.getId());
if (latest != null && STATUS_RUNNING.equals(latest.getStatus())) {
throw new BusinessException("任务已在执行中(可能已由客户端自动接管),无需重复启动");
}
throw new BusinessException("任务已结束"); throw new BusinessException("任务已结束");
} }
task.setStatus(STATUS_RUNNING);
task.setUpdatedAt(LocalDateTime.now());
fileTaskMapper.updateById(task);
} }
@Transactional @Transactional
public void failTask(Long taskId, Long userId, String error) { public void failTask(Long taskId, Long userId, String error) {
FileTaskEntity task = requireTask(taskId, userId); FileTaskEntity task = requireTask(taskId, userId);
if (STATUS_SUCCESS.equals(task.getStatus()) || STATUS_FAILED.equals(task.getStatus())) {
return;
}
String message = firstNonBlank(error, "collect-data task dispatch failed"); String message = firstNonBlank(error, "collect-data task dispatch failed");
FileResultEntity result = ensureTaskResult(task); FileResultEntity result = ensureTaskResult(task);
CollectDataStats stats = loadStats(task); CollectDataStats stats = loadStats(task);
boolean alreadyTerminal = STATUS_SUCCESS.equals(task.getStatus()) || STATUS_FAILED.equals(task.getStatus());
if (!alreadyTerminal) {
result.setSuccess(0); result.setSuccess(0);
result.setErrorMessage(message); result.setErrorMessage(message);
result.setRowCount(stats.finalRowCount); result.setRowCount(stats.finalRowCount);
fileResultMapper.updateById(result); fileResultMapper.updateById(result);
task.setStatus(STATUS_FAILED);
task.setErrorMessage(message);
task.setFailedFileCount(1);
task.setUpdatedAt(LocalDateTime.now());
task.setFinishedAt(LocalDateTime.now());
persistStats(task, stats); persistStats(task, stats);
fileTaskMapper.updateById(task); }
// 条件更新:客户端报错(/fail)与结果文件组装完成(processResultFileJob 写 SUCCESS
// 可能并发 —— 无条件 updateById 会把已生成的 SUCCESS 覆盖成 FAILED(用户拿不到下载)或反之
int updated = fileTaskMapper.update(null, new LambdaUpdateWrapper<FileTaskEntity>()
.eq(FileTaskEntity::getId, task.getId())
.notIn(FileTaskEntity::getStatus, STATUS_SUCCESS, STATUS_FAILED)
.set(FileTaskEntity::getStatus, STATUS_FAILED)
.set(FileTaskEntity::getErrorMessage, message)
.set(FileTaskEntity::getFailedFileCount, 1)
.set(FileTaskEntity::getUpdatedAt, LocalDateTime.now())
.set(FileTaskEntity::getFinishedAt, LocalDateTime.now()));
if (updated == 0) {
log.info("[collect-data] failTask 跳过写入:任务已是终态 taskId={} status={}", taskId, task.getStatus());
}
} }
@Transactional /**
* 进度心跳。
*
* <p>事务边界:Redis 任务锁在事务外获取(自旋等待最长 TASK_LOCK_WAIT_MILLIS
* 放在 @Transactional 里会白占一个 Hikari 连接),DB 段(统计持久化 + 任务行更新)
* 仍在一个事务内。
*/
public void updateProgress(Long taskId, TaskHeartbeatRequest request) { public void updateProgress(Long taskId, TaskHeartbeatRequest request) {
if (taskId == null || taskId <= 0 || request == null) { if (taskId == null || taskId <= 0 || request == null) {
return; return;
} }
try (TaskDistributedLockService.LockHandle ignored = acquireTaskLockOrThrow(taskId)) { try (TaskDistributedLockService.LockHandle ignored = acquireTaskLockOrThrow(taskId)) {
if (transactionTemplate == null) {
// 单测场景(@InjectMocks 未注入事务模板):退化为直接执行 DB 段
updateProgressLocked(taskId, request);
return;
}
transactionTemplate.executeWithoutResult(status -> updateProgressLocked(taskId, request));
}
}
private void updateProgressLocked(Long taskId, TaskHeartbeatRequest request) {
FileTaskEntity task = fileTaskMapper.selectById(taskId); FileTaskEntity task = fileTaskMapper.selectById(taskId);
if (task == null || !MODULE_TYPE.equals(task.getModuleType()) || !STATUS_RUNNING.equals(task.getStatus())) { if (task == null || !MODULE_TYPE.equals(task.getModuleType()) || !STATUS_RUNNING.equals(task.getStatus())) {
return; return;
@@ -471,7 +509,6 @@ public class CollectDataService {
fileTaskMapper.updateById(task); fileTaskMapper.updateById(task);
lastProgressFlushMillis = System.currentTimeMillis(); lastProgressFlushMillis = System.currentTimeMillis();
} }
}
/** /**
* 节流判定:progressThrottleMillis>0 且距上次实际落库未超过窗口 → 合并写(跳过 UPDATE)。 * 节流判定:progressThrottleMillis>0 且距上次实际落库未超过窗口 → 合并写(跳过 UPDATE)。
@@ -509,6 +546,7 @@ public class CollectDataService {
for (FileTaskEntity task : tasks) { for (FileTaskEntity task : tasks) {
finalizeStaleTask(task.getId(), threshold, timeoutMinutes); finalizeStaleTask(task.getId(), threshold, timeoutMinutes);
} }
finalizeNoUploadStaleTasks(threshold);
} }
} }
@@ -548,6 +586,86 @@ public class CollectDataService {
} }
} }
/**
* 二次判死:心跳正常但连续 N 分钟无结果分片上报(治「心跳续命」的假活任务)。
*
* <p>既有心跳线候选条件是 updated_at 陈旧,而客户端心跳会持续刷新 updated_at——
* 主线程卡死(浏览器自动化等待/异常)时心跳线程照发,任务永远命不中。
* 本线候选取「心跳新鲜(既有线放过)+ 创建超过 N 分钟」,判据用
* biz_task_scope_state.last_chunk_at(只随结果分片上报刷新);
* 从未上报(无 scope 行或全 NULL)的任务跳过,避免误杀首批较慢的正常任务。
*/
private void finalizeNoUploadStaleTasks(LocalDateTime heartbeatThreshold) {
long minutes = noResultUploadTimeoutMinutes;
if (minutes <= 0) {
return;
}
LocalDateTime cutoff = LocalDateTime.now().minusMinutes(minutes);
List<FileTaskEntity> candidates = fileTaskMapper.selectList(new LambdaQueryWrapper<FileTaskEntity>()
.eq(FileTaskEntity::getModuleType, MODULE_TYPE)
.eq(FileTaskEntity::getStatus, STATUS_RUNNING)
.ge(FileTaskEntity::getUpdatedAt, heartbeatThreshold)
.lt(FileTaskEntity::getCreatedAt, cutoff)
.orderByAsc(FileTaskEntity::getCreatedAt)
.last("limit 200"));
if (candidates.isEmpty()) {
return;
}
Map<Long, LocalDateTime> lastResultAtByTaskId = new HashMap<>();
for (TaskScopeLastChunkDto dto : taskScopeStateMapper.selectLastChunkAtByTaskIds(
candidates.stream().map(FileTaskEntity::getId).toList())) {
if (dto.taskId() != null && dto.lastChunkAt() != null) {
lastResultAtByTaskId.put(dto.taskId(), dto.lastChunkAt());
}
}
for (FileTaskEntity task : candidates) {
LocalDateTime lastResultAt = lastResultAtByTaskId.get(task.getId());
if (lastResultAt == null || lastResultAt.isAfter(cutoff)) {
continue;
}
finalizeNoUploadStaleTask(task.getId(), lastResultAt, minutes);
}
}
/** 单个任务的二次判死收尾:锁内复查心跳后,有分片→组装部分工作簿;无分片→标失败。 */
private void finalizeNoUploadStaleTask(Long taskId, LocalDateTime lastResultAt, long minutes) {
try (TaskDistributedLockService.LockHandle lock =
taskDistributedLockService.acquire(MODULE_TYPE, taskId, 0L)) {
if (lock == null) {
return;
}
FileTaskEntity task = fileTaskMapper.selectById(taskId);
LocalDateTime heartbeatFreshAfter = LocalDateTime.now().minusMinutes(Math.max(1L, staleTimeoutMinutes));
if (task == null
|| !MODULE_TYPE.equals(task.getModuleType())
|| !STATUS_RUNNING.equals(task.getStatus())
|| task.getUpdatedAt() == null
|| task.getUpdatedAt().isBefore(heartbeatFreshAfter)) {
// 已终态、或心跳在排队期间回落到陈旧(交回既有心跳线处理)
return;
}
if (taskFileJobService.countUnfinishedAssembleJobs(taskId, MODULE_TYPE) > 0L) {
return;
}
FileResultEntity result = ensureTaskResult(task);
CollectDataStats stats = loadStats(task);
if (hasReceivedChunks(taskId)) {
enqueueFinalWorkbook(task, result, stats);
log.warn("[collect-data] no-upload stale task enqueued partial workbook taskId={} lastResultAt={} timeoutMinutes={} finalRows={}",
taskId, lastResultAt, minutes, stats.finalRowCount);
return;
}
markTaskFailed(task, result,
"连续 " + minutes + " 分钟无结果回传,疑似客户端任务卡死,任务已自动失败(最后结果上报 " + lastResultAt + "",
stats);
log.warn("[collect-data] no-upload stale task failed without chunks taskId={} lastResultAt={} timeoutMinutes={}",
taskId, lastResultAt, minutes);
} catch (Exception ex) {
log.warn("[collect-data] no-upload stale task finalization failed taskId={} msg={}",
taskId, ex.getMessage(), ex);
}
}
public CollectDataDashboardVo dashboard(Long userId) { public CollectDataDashboardVo dashboard(Long userId) {
CollectDataDashboardVo vo = new CollectDataDashboardVo(); CollectDataDashboardVo vo = new CollectDataDashboardVo();
vo.setPendingTaskCount(countActiveTasks(userId)); vo.setPendingTaskCount(countActiveTasks(userId));
@@ -664,6 +782,39 @@ public class CollectDataService {
throw new BusinessException("request is empty"); throw new BusinessException("request is empty");
} }
ensureRustfsPayloadStorageEnabled(); ensureRustfsPayloadStorageEnabled();
// 锁外预检:任务不存在/已结束时立即失败,不为终态任务白跑去重查询与品牌检测。
// 只做快速失败,并发正确性仍由锁内的重读复核保证。
FileTaskEntity probe = fileTaskMapper.selectById(taskId);
if (probe == null || !MODULE_TYPE.equals(probe.getModuleType())) {
throw new BusinessException("任务不存在");
}
if (STATUS_SUCCESS.equals(probe.getStatus()) || STATUS_FAILED.equals(probe.getStatus())) {
log.warn("[collect-data] 任务已结束,拒绝重复提交 taskId={} status={}", taskId, probe.getStatus());
throw new BusinessException(40901, "任务已结束,拒绝重复提交");
}
// 归一化 / 去重过滤 / 品牌检测放在锁外:品牌检测是同步远程调用,上游 16890 抖动时
// 单品牌 10 次重试合计上百秒(taskId 28599 实测:chunk 回传在锁内等品牌检测 103.5 秒,
// 期间心跳与客户端重试全部撞 40902 拿不到锁,客户端 5 次重试预算耗尽后中止整个采集)。
// 这几步只依赖本批入参、不写任务状态,放锁外不改变 chunk 落库的串行语义。
List<CollectDataResultRowVo> rows = normalizeSubmitRows(request.getItems());
buildParseLimits().validateChunkRowCount(rows.size());
// 「结果文件」sheet 需要 Python 回传的全量数据(不经任何后端过滤),
// 因此 chunk payload 写入未筛除空 ASIN 的全量 rows;
// 仅当 ASIN 非空时才进入去重 / 无效品牌 / 品牌检测链路并落 biz_task_result_item。
List<CollectDataResultRowVo> rowsForFiltering = new ArrayList<>(rows.size());
for (CollectDataResultRowVo row : rows) {
if (row.getAsin() != null && !row.getAsin().isBlank()) {
rowsForFiltering.add(row);
}
}
long prepareStartAt = System.currentTimeMillis();
CollectDataBatchQuery.FilterResult filtered = collectDataBatchQuery.filter(rowsForFiltering);
CollectDataBrandBatchFilter.BrandBatchOutcome brandOutcome = brandBatchFilter.filter(filtered.kept());
log.info("[collect-data] 锁外预处理完成 taskId={} rows={} 去重后={} 品牌检测耗时={}ms",
taskId, rows.size(), filtered.kept().size(), System.currentTimeMillis() - prepareStartAt);
try (TaskDistributedLockService.LockHandle ignored = acquireTaskLockOrThrow(taskId)) { try (TaskDistributedLockService.LockHandle ignored = acquireTaskLockOrThrow(taskId)) {
FileTaskEntity task = fileTaskMapper.selectById(taskId); FileTaskEntity task = fileTaskMapper.selectById(taskId);
if (task == null || !MODULE_TYPE.equals(task.getModuleType())) { if (task == null || !MODULE_TYPE.equals(task.getModuleType())) {
@@ -701,25 +852,23 @@ public class CollectDataService {
return buildSubmitVo(task, result, chunkIndex, chunkTotal, request, 0); return buildSubmitVo(task, result, chunkIndex, chunkTotal, request, 0);
} }
List<CollectDataResultRowVo> rows = normalizeSubmitRows(request.getItems());
buildParseLimits().validateChunkRowCount(rows.size());
CollectDataStats stats = loadStats(task); CollectDataStats stats = loadStats(task);
stats.receivedRows += rows.size(); stats.receivedRows += rows.size();
stats.currentChunkRows = rows.size(); stats.currentChunkRows = rows.size();
// 「结果文件」sheet 需要 Python 回传的全量数据(不经任何后端过滤),
// 因此 chunk payload 写入未筛除空 ASIN 的全量 rows;
// 仅当 ASIN 非空时才进入去重 / 无效品牌 / 品牌检测链路并落 biz_task_result_item。
List<CollectDataResultRowVo> rowsForFiltering = new ArrayList<>(rows.size());
for (CollectDataResultRowVo row : rows) {
if (row.getAsin() != null && !row.getAsin().isBlank()) {
rowsForFiltering.add(row);
}
}
CollectDataBatchQuery.FilterResult filtered = collectDataBatchQuery.filter(rowsForFiltering);
stats.dedupeFilteredCount += filtered.dedupeFilteredCount(); stats.dedupeFilteredCount += filtered.dedupeFilteredCount();
stats.invalidFilteredCount += filtered.invalidFilteredCount(); stats.invalidFilteredCount += filtered.invalidFilteredCount();
List<CollectDataResultRowVo> accepted = filterByBrandCheck(filtered.kept(), stats); stats.brandRejectedCount += brandOutcome.rejected().size();
stats.brandQueryFailedCount += brandOutcome.queryFailed().size();
// 只有被服务端判定为无效品牌(rejected)的行才写入无效品牌表;
// queryFailed 是品牌检测服务端到端失败(如缺 X-Token 422/超时/限流),
// 一并写入会把故障期间被误伤的品牌永久拉黑,后续同品牌商品全部被
// invalidFiltered 过滤(task-27265 实测:通过 Python 过滤的 21 行中
// 8 行查询失败被写表后误杀,最终结果 Excel 0 行)。rejected 为空时
// 不触发任何写入,避免空批次无意义调用。
if (!brandOutcome.rejected().isEmpty()) {
invalidAsinBatchWriter.writeBatch(brandOutcome.rejected());
}
List<CollectDataResultRowVo> accepted = brandOutcome.accepted();
// 结果明细改为 chunk 级存储:整个 chunk 的 accepted 行共享一个 // 结果明细改为 chunk 级存储:整个 chunk 的 accepted 行共享一个
// RustFS 对象(deterministic key,同 chunk 重提覆盖同一对象), // RustFS 对象(deterministic key,同 chunk 重提覆盖同一对象),
// biz_task_result_item.payload_json 只存 {chunk, offset, payload} 引用。 // biz_task_result_item.payload_json 只存 {chunk, offset, payload} 引用。
@@ -754,6 +903,14 @@ public class CollectDataService {
if (request.getError() != null && !request.getError().isBlank()) { if (request.getError() != null && !request.getError().isBlank()) {
markTaskFailed(task, result, request.getError(), stats); markTaskFailed(task, result, request.getError(), stats);
// 失败但已收到分片:照常组装结果文件,让用户能下载已采集的数据。
// 此前失败分支只标失败不组装,已落库的数据也没有任何结果文件可下载
// taskId 2859955 个分片全部收到、187 行明细已落库,用户却拿不到文件)。
if (hasReceivedChunks(taskId)) {
enqueueFinalWorkbook(task, result, stats);
log.warn("[collect-data] 任务失败仍组装部分结果 taskId={} error={} finalRows={}",
taskId, request.getError(), stats.finalRowCount);
}
} else if (Boolean.TRUE.equals(request.getDone())) { } else if (Boolean.TRUE.equals(request.getDone())) {
enqueueFinalWorkbook(task, result, stats); enqueueFinalWorkbook(task, result, stats);
} else { } else {
@@ -815,22 +972,6 @@ public class CollectDataService {
return rows; return rows;
} }
private List<CollectDataResultRowVo> filterByBrandCheck(List<CollectDataResultRowVo> rows, CollectDataStats stats) {
CollectDataBrandBatchFilter.BrandBatchOutcome outcome = brandBatchFilter.filter(rows);
stats.brandRejectedCount += outcome.rejected().size();
stats.brandQueryFailedCount += outcome.queryFailed().size();
// 只有被服务端判定为无效品牌(rejected)的行才写入无效品牌表;
// queryFailed 是品牌检测服务端到端失败(如缺 X-Token 422/超时/限流),
// 一并写入会把故障期间被误伤的品牌永久拉黑,后续同品牌商品全部被
// invalidFiltered 过滤(task-27265 实测:通过 Python 过滤的 21 行中
// 8 行查询失败被写表后误杀,最终结果 Excel 0 行)。rejected 为空时
// 不触发任何写入,避免空批次无意义调用。
if (!outcome.rejected().isEmpty()) {
invalidAsinBatchWriter.writeBatch(outcome.rejected());
}
return outcome.accepted();
}
private void persistChunk(Long taskId, private void persistChunk(Long taskId,
String scopeKey, String scopeKey,
String scopeHash, String scopeHash,
@@ -893,7 +1034,8 @@ public class CollectDataService {
result.setResultFileSize(0L); result.setResultFileSize(0L);
result.setResultContentType(CONTENT_TYPE_XLSX); result.setResultContentType(CONTENT_TYPE_XLSX);
result.setRowCount(stats.finalRowCount); result.setRowCount(stats.finalRowCount);
result.setErrorMessage(null); // 不清 errorMessage:失败任务的部分结果组装也走这里,清掉会让用户看不到真实失败原因
// (成功路径的 errorMessage 本来就为 null,无需清理)。
fileResultMapper.updateById(result); fileResultMapper.updateById(result);
task.setUpdatedAt(LocalDateTime.now()); task.setUpdatedAt(LocalDateTime.now());
fileTaskMapper.updateById(task); fileTaskMapper.updateById(task);
@@ -942,6 +1084,22 @@ public class CollectDataService {
stats.summaries, stats.summaries,
batch -> streamRawRows(task.getId(), batch)); batch -> streamRawRows(task.getId(), batch));
String objectKey = ossStorageService.uploadResultFile(xlsx, MODULE_TYPE); String objectKey = ossStorageService.uploadResultFile(xlsx, MODULE_TYPE);
// 复用同一份 stats 更新 finalRowCount 后再持久化,避免重复 loadStats 丢失 summaries。
stats.finalRowCount = (int) finalRowCount;
persistStats(task, stats);
// 失败原因先留存:下面的乐观写入会清空 result.errorMessage,任务已被判失败时要用它恢复。
String failureReason = result.getErrorMessage();
if (failureReason == null || failureReason.isBlank()) {
failureReason = task.getErrorMessage();
}
if (failureReason == null || failureReason.isBlank()) {
failureReason = "任务失败,结果文件为已采集的部分数据";
}
// 结果行先按成功乐观写入:保持「结果文件先于任务成功落库」的时序,
// 万一进程在这两步之间退出,任务仍是 RUNNING,会被陈旧巡检重新组装(可自愈)。
result.setResultFilename(filename); result.setResultFilename(filename);
result.setResultFileUrl(objectKey); result.setResultFileUrl(objectKey);
result.setResultFileSize(xlsx.length()); result.setResultFileSize(xlsx.length());
@@ -951,16 +1109,27 @@ public class CollectDataService {
result.setErrorMessage(null); result.setErrorMessage(null);
fileResultMapper.updateById(result); fileResultMapper.updateById(result);
// 复用同一份 stats 更新 finalRowCount 后再持久化,避免重复 loadStats 丢失 summaries。 // 条件更新:任务可能已被判失败(客户端上报失败 / 陈旧判死与结果文件组装并发)——
stats.finalRowCount = (int) finalRowCount; // 无条件 updateById 会把 FAILED 覆盖成 SUCCESS(错误信息被清、用户看到「假成功」)。
persistStats(task, stats); // 结果文件已生成,故失败态下仍保留文件,只是不覆盖状态。
task.setStatus(STATUS_SUCCESS); int updated = fileTaskMapper.update(null, new LambdaUpdateWrapper<FileTaskEntity>()
task.setSuccessFileCount(1); .eq(FileTaskEntity::getId, task.getId())
task.setFailedFileCount(0); .ne(FileTaskEntity::getStatus, STATUS_FAILED)
task.setErrorMessage(null); .set(FileTaskEntity::getStatus, STATUS_SUCCESS)
task.setUpdatedAt(LocalDateTime.now()); .set(FileTaskEntity::getSuccessFileCount, 1)
task.setFinishedAt(LocalDateTime.now()); .set(FileTaskEntity::getFailedFileCount, 0)
fileTaskMapper.updateById(task); .set(FileTaskEntity::getErrorMessage, null)
.set(FileTaskEntity::getUpdatedAt, LocalDateTime.now())
.set(FileTaskEntity::getFinishedAt, LocalDateTime.now()));
if (updated == 0) {
// 任务已是 FAILED:结果记录改回失败语义并保留真实原因,但文件 URL 照常保留,
// 用户看到「失败 + 原因」的同时仍能下载已采集的部分结果。
result.setSuccess(0);
result.setErrorMessage(failureReason);
fileResultMapper.updateById(result);
log.warn("[collect-data] 结果文件已生成但任务已是 FAILED,保留失败态与原因 taskId={} rows={} reason={}",
task.getId(), finalRowCount, failureReason);
}
} finally { } finally {
FileUtil.del(xlsx); FileUtil.del(xlsx);
} }
@@ -1085,7 +1254,8 @@ public class CollectDataService {
private TaskDistributedLockService.LockHandle acquireTaskLockOrThrow(Long taskId) { private TaskDistributedLockService.LockHandle acquireTaskLockOrThrow(Long taskId) {
TaskDistributedLockService.LockHandle lockHandle = taskDistributedLockService.acquire(MODULE_TYPE, taskId, TASK_LOCK_WAIT_MILLIS); TaskDistributedLockService.LockHandle lockHandle = taskDistributedLockService.acquire(MODULE_TYPE, taskId, TASK_LOCK_WAIT_MILLIS);
if (lockHandle == null) { if (lockHandle == null) {
throw new BusinessException(40901, "任务正在处理,请稍后再试"); log.warn("[collect-data] 任务锁竞争,拒绝本次提交 taskId={} waitMillis={}", taskId, TASK_LOCK_WAIT_MILLIS);
throw new BusinessException(BusinessCodes.TASK_BUSY, "任务正在处理,请稍后再试");
} }
return lockHandle; return lockHandle;
} }
@@ -0,0 +1,28 @@
package com.nanri.aiimage.modules.collectdata.service;
import com.nanri.aiimage.modules.task.model.dto.TaskHeartbeatRequest;
import com.nanri.aiimage.modules.task.spi.TaskModuleHeartbeatSpi;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
/**
* 采集数据的任务心跳实现(2026-09 全维度审查 G5)。
*
* <p>进度由采集模块自己的导入进度表维护,心跳直接把客户端上报的进度写进去。
*/
@Service
@RequiredArgsConstructor
public class CollectDataTaskHeartbeatSpi implements TaskModuleHeartbeatSpi {
private final CollectDataService collectDataService;
@Override
public String moduleType() {
return "COLLECT_DATA";
}
@Override
public void touchHeartbeat(Long taskId, TaskHeartbeatRequest request) {
collectDataService.updateProgress(taskId, request);
}
}
@@ -0,0 +1,71 @@
package com.nanri.aiimage.modules.collectdata.service;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.spi.ClientTaskPullSpi;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.util.LinkedHashMap;
import java.util.Map;
/**
* 集采(collect-data)的客户端兜底拉取实现。
*
* <p>Python 消费端需要 taskId / totalRows / pageSize / filters(明细行自行按 /items 分页拉取)。
* filters 取自任务行 request_json 里 parse 时落库的那份(CollectDataService#persistParsedTask),
* 序列化后就是 Python 读取的 camelCase 键(countryCode/minAmount/...)。
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class CollectDataTaskPullSpiImpl implements ClientTaskPullSpi {
private static final String QUEUE_TYPE = "collect-data-run";
private static final String TASK_TYPE = "collect-data";
private final ObjectMapper objectMapper;
@Override
public String moduleType() {
return CollectDataService.MODULE_TYPE;
}
@Override
public Map<String, Object> buildQueuePayload(FileTaskEntity task) {
JsonNode request = parseJson(task.getRequestJson());
if (request == null) {
log.warn("[collect-data] 兜底拉取失败:任务请求参数缺失或不可解析 taskId={}", task.getId());
return null;
}
JsonNode filtersNode = request.get("filters");
Map<String, Object> filters = filtersNode == null || filtersNode.isNull()
? Map.of()
: objectMapper.convertValue(filtersNode, Map.class);
JsonNode stats = parseJson(task.getResultJson());
int totalRows = stats == null ? 0 : stats.path("totalRows").asInt(0);
Map<String, Object> data = new LinkedHashMap<>();
data.put("taskId", task.getId());
data.put("taskNo", task.getTaskNo());
data.put("taskType", TASK_TYPE);
data.put("totalRows", totalRows);
data.put("pageSize", CollectDataService.DEFAULT_PAGE_SIZE);
data.put("filters", filters);
log.info("[collect-data] 兜底载荷已组装 taskId={} totalRows={} filters={}", task.getId(), totalRows, filters);
return Map.of("type", QUEUE_TYPE, "data", data);
}
private JsonNode parseJson(String json) {
if (json == null || json.isBlank()) {
return null;
}
try {
return objectMapper.readTree(json);
} catch (Exception ex) {
log.warn("[collect-data] 兜底拉取解析任务 JSON 失败 err={}", ex.getMessage());
return null;
}
}
}
@@ -148,11 +148,8 @@ public class CollectDataResultDetailCodec {
try { try {
MessageDigest digest = MessageDigest.getInstance("SHA-256"); MessageDigest digest = MessageDigest.getInstance("SHA-256");
byte[] bytes = digest.digest((value == null ? "" : value).getBytes(StandardCharsets.UTF_8)); byte[] bytes = digest.digest((value == null ? "" : value).getBytes(StandardCharsets.UTF_8));
StringBuilder sb = new StringBuilder(bytes.length * 2); // HexFormat 替代逐字节 String.format("%02x"):后者每次走 32 次 Formatter(热路径按行调用)
for (byte b : bytes) { return java.util.HexFormat.of().formatHex(bytes);
sb.append(String.format("%02x", b));
}
return sb.toString();
} catch (Exception ex) { } catch (Exception ex) {
throw new IllegalStateException("chunk detail ref hash failed", ex); throw new IllegalStateException("chunk detail ref hash failed", ex);
} }
@@ -1,6 +1,7 @@
package com.nanri.aiimage.modules.collectdata.util; package com.nanri.aiimage.modules.collectdata.util;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper; import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.modules.collectdata.model.vo.CollectDataResultRowVo; import com.nanri.aiimage.modules.collectdata.model.vo.CollectDataResultRowVo;
import com.nanri.aiimage.modules.task.mapper.TaskResultItemMapper; import com.nanri.aiimage.modules.task.mapper.TaskResultItemMapper;
import com.nanri.aiimage.modules.task.model.entity.TaskResultItemEntity; import com.nanri.aiimage.modules.task.model.entity.TaskResultItemEntity;
@@ -133,12 +134,14 @@ public class CollectDataResultItemBatchWriter {
} }
int written = 0; int written = 0;
int failedBatches = 0;
for (int from = 0; from < toUpsert.size(); from += batchSize) { for (int from = 0; from < toUpsert.size(); from += batchSize) {
int to = Math.min(from + batchSize, toUpsert.size()); int to = Math.min(from + batchSize, toUpsert.size());
List<TaskResultItemEntity> batch = toUpsert.subList(from, to); List<TaskResultItemEntity> batch = toUpsert.subList(from, to);
try { try {
written += taskResultItemMapper.upsertBatch(batch); written += taskResultItemMapper.upsertBatch(batch);
} catch (RuntimeException ex) { } catch (RuntimeException ex) {
failedBatches++;
log.warn("[collect-data] upsert result item batch failed, skip batch {}..{} taskId={}", log.warn("[collect-data] upsert result item batch failed, skip batch {}..{} taskId={}",
from, to, taskId, ex); from, to, taskId, ex);
// 失败批的新行未落库,从增量计数中扣除,避免任务内累计虚高; // 失败批的新行未落库,从增量计数中扣除,避免任务内累计虚高;
@@ -150,6 +153,16 @@ public class CollectDataResultItemBatchWriter {
} }
} }
} }
if (failedBatches > 0) {
// biz_task_result_item 是结果 Excel「明细」sheet 的唯一数据源:静默跳批会让任务
// 以 SUCCESS 收尾但明细缺行,且与「结果文件」sheet 的汇总数量对不上(假成功)。
// 抛出让本次 chunk 提交明确失败:workersearch_spider)识别 success=false 后会重试,
// 重提按 payload_hash 幂等(已落库行跳过、未落库行补插),最终收敛为完整数据。
log.error("[collect-data] 明细写入存在失败批次,拒绝本次提交 taskId={} failedBatches={} totalBatches={}",
taskId, failedBatches, (toUpsert.size() + batchSize - 1) / batchSize);
throw new BusinessException("采集结果明细写入失败,请稍后重试(失败批次 "
+ failedBatches + "/" + ((toUpsert.size() + batchSize - 1) / batchSize) + "");
}
return new UpsertCounts(written, skipped, newlyInserted); return new UpsertCounts(written, skipped, newlyInserted);
} }
@@ -157,11 +170,8 @@ public class CollectDataResultItemBatchWriter {
try { try {
java.security.MessageDigest digest = java.security.MessageDigest.getInstance("SHA-256"); java.security.MessageDigest digest = java.security.MessageDigest.getInstance("SHA-256");
byte[] bytes = digest.digest((value == null ? "" : value).getBytes(java.nio.charset.StandardCharsets.UTF_8)); byte[] bytes = digest.digest((value == null ? "" : value).getBytes(java.nio.charset.StandardCharsets.UTF_8));
StringBuilder sb = new StringBuilder(bytes.length * 2); // HexFormat 替代逐字节 String.format("%02x"):后者每次走 32 次 Formatter(热路径按行调用)
for (byte b : bytes) { return java.util.HexFormat.of().formatHex(bytes);
sb.append(String.format("%02x", b));
}
return sb.toString();
} catch (Exception ex) { } catch (Exception ex) {
throw new IllegalStateException("结果明细 hash 计算失败", ex); throw new IllegalStateException("结果明细 hash 计算失败", ex);
} }
@@ -278,12 +278,14 @@ public class ConvertRunService {
File outputDir = FileUtil.mkdir(FileUtil.file(storageProperties.getLocalTempDir(), "convert-result")); File outputDir = FileUtil.mkdir(FileUtil.file(storageProperties.getLocalTempDir(), "convert-result"));
List<GeneratedConvertFile> generatedFiles = new ArrayList<>(); List<GeneratedConvertFile> generatedFiles = new ArrayList<>();
Map<String, BufferedWriter> writers = new LinkedHashMap<>(); Map<String, BufferedWriter> writers = new LinkedHashMap<>();
try {
// 创建循环纳入 try:第 N 个 writer 创建失败时,前 N-1 个已打开的句柄会因 finally
// 尚未生效而泄漏(同族 SplitRunService.SplitChunkWriter 已用 try/finally 处理)
for (String outputFilename : outputFilenames) { for (String outputFilename : outputFilenames) {
File outputFile = buildNamedOutputFile(outputDir, outputFilename); File outputFile = buildNamedOutputFile(outputDir, outputFilename);
generatedFiles.add(new GeneratedConvertFile(outputFilename, outputFile)); generatedFiles.add(new GeneratedConvertFile(outputFilename, outputFile));
writers.put(outputFilename, Files.newBufferedWriter(outputFile.toPath(), StandardCharsets.UTF_8)); writers.put(outputFilename, Files.newBufferedWriter(outputFile.toPath(), StandardCharsets.UTF_8));
} }
try {
streamTxtRowsToOutputs(inputFile, templateEntity, writers); streamTxtRowsToOutputs(inputFile, templateEntity, writers);
} finally { } finally {
IOException closeException = null; IOException closeException = null;
@@ -2,7 +2,7 @@ package com.nanri.aiimage.modules.dedupe.controller;
import com.nanri.aiimage.common.api.ApiResponse; import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.common.exception.BusinessException; import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.modules.admin.support.AdminAuthSupport; import com.nanri.aiimage.common.security.AdminAuthSupport;
import com.nanri.aiimage.common.util.DownloadHeaderUtil; import com.nanri.aiimage.common.util.DownloadHeaderUtil;
import com.nanri.aiimage.modules.dedupe.model.dto.DedupeTotalDataCreateRequest; import com.nanri.aiimage.modules.dedupe.model.dto.DedupeTotalDataCreateRequest;
import com.nanri.aiimage.modules.dedupe.model.dto.DedupeTotalDataUpdateRequest; import com.nanri.aiimage.modules.dedupe.model.dto.DedupeTotalDataUpdateRequest;
@@ -12,7 +12,7 @@ import com.nanri.aiimage.modules.dedupe.model.vo.DedupeTotalDataImportVo;
import com.nanri.aiimage.modules.dedupe.model.vo.DedupeTotalDataItemVo; import com.nanri.aiimage.modules.dedupe.model.vo.DedupeTotalDataItemVo;
import com.nanri.aiimage.modules.dedupe.model.vo.DedupeTotalDataPageVo; import com.nanri.aiimage.modules.dedupe.model.vo.DedupeTotalDataPageVo;
import com.nanri.aiimage.modules.dedupe.service.DedupeTotalDataService; import com.nanri.aiimage.modules.dedupe.service.DedupeTotalDataService;
import com.nanri.aiimage.modules.permission.model.entity.AdminUserEntity; import com.nanri.aiimage.common.model.entity.AdminUserEntity;
import com.nanri.aiimage.modules.permission.service.PermissionMenuService; import com.nanri.aiimage.modules.permission.service.PermissionMenuService;
import io.swagger.v3.oas.annotations.Operation; import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter; import io.swagger.v3.oas.annotations.Parameter;
@@ -88,10 +88,12 @@ public class DedupeTotalDataController {
@RequestParam(name = "end_date", required = false) @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate endDate, @RequestParam(name = "end_date", required = false) @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate endDate,
@Parameter(description = "分组 ID") @RequestParam(name = "group_id", required = false) Long groupId, @Parameter(description = "分组 ID") @RequestParam(name = "group_id", required = false) Long groupId,
@Parameter(description = "国家代码(如 DE、UK") @RequestParam(name = "country", required = false) String country, @Parameter(description = "国家代码(如 DE、UK") @RequestParam(name = "country", required = false) String country,
@Parameter(description = "顺序翻页游标(上一页返回的 nextLastId;传了就忽略 page 偏移)")
@RequestParam(name = "last_id", required = false) Long lastId,
HttpServletRequest request) { HttpServletRequest request) {
RequestOperator operator = requireDedupeTotalDataAccess(request); RequestOperator operator = requireDedupeTotalDataAccess(request);
return ApiResponse.success(dedupeTotalDataService.page( return ApiResponse.success(dedupeTotalDataService.page(
page, pageSize, keyword, username, startDate, endDate, groupId, country, operator.id())); page, pageSize, keyword, username, startDate, endDate, groupId, country, lastId, operator.id()));
} }
@GetMapping("/export") @GetMapping("/export")

Some files were not shown because too many files have changed in this diff Show More