Compare commits

...

38 Commits

Author SHA1 Message Date
huangzd1997 5b8105ec2b feat(后台管理): 实体管理列表统一展示创建时间/更新时间
用户管理、菜单管理、不符合ASIN、数据去重总数据、查询ASIN、最低价ASIN、
商品类目、密钥管理共 8 个实体管理列表补齐两列。分组管理/店铺密钥/店铺管理
此前已带创建+修改时间,任务列表与统计报表(撞款监控、密钥用量、日志、
记录与版本)不含实体更新语义,均未改动。

关键点——时间列必须由数据库维护,否则新列是假的:
这些表的更新走 selectById → 改字段 → updateById,实体带着读出的旧
updated_at 一起写回。MySQL 规则是「UPDATE 显式给某列赋值时不触发该列的
ON UPDATE 自动更新」,不禁写就会把旧值写回去,更新时间永远冻结在首次写入
时刻。按 V125(biz_file_result)既有样板,给 7 个实体标注
@TableField(insertStrategy=NEVER, updateStrategy=NEVER)。

- V131:users / columns 补 updated_at(幂等 ADD COLUMN,仿 V125 写法)。
  存量行被回填为迁移执行时刻,非真实历史变更时间(历史上无记录,无法还原)
- 实体/VO:AdminUserEntity、PermissionMenuEntity、InvalidAsinDataEntity、
  DedupeTotalDataEntity、ProductCategoryEntity、QueryAsinEntity、
  SkipPriceAsinEntity 加/改写 updatedAt;AdminUserItemVo、PermissionMenuItemVo、
  InvalidAsinDataItemVo、DedupeTotalDataItemVo 补 updatedAt;
  AdminUserSecretRowVo 补 createdAt(行级首次配置时间 = 三模块最早)
- 查询ASIN/最低价ASIN 后端 VO 与前端 model 本就有两字段,仅补渲染
- 前端 8 页表格加列,同步修正空态/加载行的 colspan(手写表格,不同步会错位)
- 测试:align-query-asin / align-skip-price 原断言「不允许有更新时间列」
  (像素复刻旧版),按新需求改为断言两列存在;新增 e2e list-time-columns
  覆盖 8 页表头与真实时间值渲染
2026-09-19 15:53:01 +08:00
huangzd1997 5fcb449946 chore(客户端更新日志): 补 4.0.31 条目(购物车未知时最低价也生效) 2026-09-19 15:39:20 +08:00
huangzd1997 ff68426c69 chore(客户端更新日志): 补 4.0.30 条目(跟价最低价跌破自动调回) 2026-09-19 15:24:30 +08:00
huangzd1997 46be044121 fix(实例路由): 转发失败不再伪装成「任务不存活」
归属节点滚动重启时跨节点转发会连接失败,原来返回 ApiResponse.fail(40903) ——
HTTP 200 + data:null。客户端把「拿不到数据」解析成 alive=false,于是把健康的长
任务主动停掉(2026-09-18 任务 28616:跑到 66/253 被自杀,只跑到第 2/6 页)。

改为 503 + 空 body:新客户端按状态码判未知继续跑;老客户端因 body 不是 JSON、
resp.json() 抛异常也落到未知——两边都不会再把未知当成死。顺带让这类故障在
HTTP 指标里可见(原先记成 200,滚动重启期间丢了多少心跳监控完全看不到)。
2026-09-18 16:25:33 +08:00
huangzd1997 e0303f9cba feat(上架): 店铺互斥改为按(设备,店铺),同店可在不同客户端并行跑不同国家
背景:紫鸟浏览器会话是每台客户端一份,同一家店可以在两台机器上并行;
国家是会话内的可切换状态(SwitchingCountries,断线重连还会再切),
所以真正必须串行的只有"同一台设备上的同一家店"。原来的全局按店铺互斥
把跨机器的合法并行也挡了(2026-09-18 任务 28624 被 28616 误挡)。

- V130:biz_publish_file 增加 device_id 列
- AdminAuthSupport.currentDeviceId():只读 JWT 签名的 deviceId claim,
  不信任客户端可控的 X-Device-Id 请求头
- activateFile 互斥键改为 (device_id, shop_name);device 为空(旧客户端
  token 无该 claim / 内部令牌调用)时退回全局店铺互斥,保守不放宽;
  存量 RUNNING 行的 NULL device 视为"来源不明",同样保守拦到跑完为止
- 前端与客户端无需改动:页面 axios 对 /newApi 已带 Authorization Bearer
2026-09-18 16:08:08 +08:00
huangzd1997 9d39705c77 fix(采集数据): 品牌检测移出任务锁 + 失败任务仍产出可下载的部分结果
taskId 28599 事故根因:分片回传在任务锁内同步做品牌检测,上游 16890 卡死时
单次持锁 103.5 秒(costMs=103556),期间该任务的心跳与客户端重试全部撞
40902「任务正在处理」,客户端 5 次重试预算耗尽后中止了整个采集。

- submitResult:行归一化 / 去重过滤 / 品牌检测移到任务锁外,锁内只保留落库与终态
- BrandCheckClient:整批加 90 秒总耗时上限(超时按查询失败降级,首轮始终执行),
  单次回传时延从「无上界」收敛到约 90 秒 + 一次读超时
- 失败且已收到分片时照常组装部分结果;任务已 FAILED 则保留失败态与真实失败原因,
  只补结果文件(乐观写结果行 + 条件更新不匹配再补偿写回,避免崩溃时留下「成功但无下载」)
- productrisk / shopmatch 按 pricetrack 的 preserveFailure 样板补齐,含组装落地时
  不再把失败行「洗成成功」
- 前端:采集 / 跟价 / 产品风险 / 店铺匹配的下载按钮放宽到失败任务
2026-09-18 15:53:15 +08:00
huangzd1997 986df86e89 fix(任务恢复): 停止意图优先于自动续跑 + 未覆盖模块的中断可见
自查上一提交时发现两处遗漏:

- 用户点了「停止循环」而子任务恰好因客户端中断失败时,续派逻辑跑在停止检查之前,
  会把循环从停止意图里拉回 RUNNING 自己转起来。改为停止请求优先:直接 STOPPED。
- 模块不支持自动续跑的中断任务此前完全无痕迹(用户只看到失败)。
  这类模块(上架/改价/审批/商品管理采集/跟价非循环任务)的续跑载荷需要用户在页面上
  选的执行参数(ziniao_version 等),而这份选择只存在于派发那一刻的浏览器里、没落到
  request_json —— 自动重排队会用错参数,所以它们**不纳入**续跑(白名单的设计原则就是
  「执行参数全在服务端」)。现补一条计数查询把这类中断数量带进 stale-check summary
  的 resume(u=N),运维据此人工重跑;将来把参数回写落库后即可纳入白名单。

测试:PriceTrackLoopRunServiceTest 新增「已请求停止时中断失败不续派而是停止」,
TaskResumeServiceTest 新增「统计不支持续跑的中断任务数」。
2026-09-18 15:48:33 +08:00
huangzd1997 bd359411a9 feat(任务恢复): 客户端中断的任务自动续跑(保留失败记录 + 重新排队)
2026-09-18 任务 28587(跟价,uid 977 店铺「张美莺」)在客户端 09:51/09:57 被更新脚本
重启后中断,任务一直挂 RUNNING 到 stale 兜底;跟价循环还有第二重问题——子任务失败
即整个循环判 FAILED,一次客户端更新就让无限循环彻底停下。

契约:中断任务**保留失败记录**(用户看得到「因客户端重启中断」),同时由服务端自动
重新排队续跑,长任务不再因为一次更新整个白跑。

- V129:biz_file_task 加 resume_of_task_id / resume_attempt,
  biz_price_track_loop_run 加 resume_attempt(代数封顶用);
- 新增 TaskResumeService:扫描最近 30 分钟内因客户端中断而失败、未续跑过、代数未超限的
  任务,复制请求参数重新排队成 PENDING,交给已有兜底拉取通道(客户端每分钟 pull-pending)
  领走执行——不新造第二套派发机制。只对注册了 ClientTaskPullSpi 的模块生效
  (相似ASIN/采集/外观专利),上架/改价等写操作模块刻意排除,避免盲目重跑;
- 幂等:按 resume_of_task_id 反查,同一原任务不会重复排队;
- 挂在 stale-check 巡检线(每 2 分钟、有分布式锁),并用隔离壳包住异常:
  续跑失败绝不拖垮判死主流程(判死优先级更高);
- PriceTrackLoopRunService:子任务因「客户端异常中断」失败时不终止循环,
  清 active_task_id 后保持 RUNNING,客户端下次 dispatchNext 拿到同一店铺/同一轮,
  即原地续跑(页内已处理 ASIN 由服务端 skip_asins 去重,不会重复改价);
  连续重派超过 3 次才真正判失败,避免会话持续不可用时无限重开浏览器
  (28587 就是这样白烧了 5 小时)。子任务成功一轮后计数归零;
- application.yml:aiimage.task-resume.enabled 默认跟随 client-task-pull 开关
  (兜底拉取关着时续跑任务无人领取,只会积压被告死)。

测试:TaskResumeServiceTest 6 例(排队/幂等/开关/无实现/插入失败/无归属用户)、
PriceTrackLoopRunServiceTest 新增 4 例(中断续跑/达上限/非中断仍终止/成功归零),
连带更新 DeleteBrandStaleTaskServiceTest 与 TaskModuleCoverageTest 的构造参数。
2026-09-18 15:42:07 +08:00
huangzd1997 3634ea1d62 feat(撞款): 采集明细落库后自动触发重扫,重复检查无需等次日 00:00
- shopdatacrawl 新增 DuplicateCheckRefreshPort 端口,TaskService 在明细落库成功后请求重扫
- shopduplicatecheck 新增 DuplicateCheckRefreshScheduler:10s 合并窗口去抖、单飞、
  扫描锁忙自动重试(最多 6 次),失败不影响采集归档
- 复用现有 SCAN_LOCK,与定时扫描、手动「重新分析」互斥;新增 5 个调度器单测
2026-09-17 23:08:15 +08:00
huangzd1997 3ce0569c59 fix(提示语): 失败提示优先展示后端真实原因,修正 10+ 处误导文案
上架/删除品牌/去重/转换/拆分/撤回/查ASIN/产品风险/跟价/巡店/店铺匹配/视频工作台:
- 解析失败不再只报通用文案,逐条展示后端文件级 errorMessage(上架新增「店铺未找到」专属弹窗)
- 匹配失败不再一律「请检查店铺名」,按 matchStatus 区分 CONFLICT/PENDING
- 去重/转换/拆分全失败弹窗优先列各文件真实原因,取不到才回退猜测文案
- dispatch-guard 新增 fileErrors 选项与 collectDistinctErrors 导出
2026-09-17 23:02:15 +08:00
huangzd1997 d2d95f0b71 feat(上架): 激活时按店铺互斥,同一店铺不允许两个任务同时跑
2026-09-17 事故(28519/28520/28521 相继提交同一店铺「林洪武」):同一店铺被多个
任务并发打开,客户端 startBrowser 全部返回 -10000,三个任务一起失败。

- PublishTaskService.activateFile:激活前查是否已有其他任务在同一店铺 RUNNING,
  有则拒绝并带出占用中的任务号(激活是任务真正开跑的唯一入口,能最早拦住);
  校验与更新之间仍有极小竞态窗口,真正串行由客户端店铺锁保证,这层负责尽早提示;
- 前端 BrandPublishTab:文件派发失败的原因回显给用户。原来错误只写进日志、提示
  固定为"已记录并继续",用户会误以为是文件问题而反复重传,看不到真正原因;
- 新增 3 个后端单测:同店铺被占用则拒绝、无占用则放行、店铺名为空跳过校验。
2026-09-17 15:48:51 +08:00
huangzd1997 2a51006888 feat(前端防连点): 工具页提交按钮加全局冷却,并精简 4.0.28 更新日志
各工具页的「开始上架 / 启动任务 / 匹配店铺」按钮在提交逻辑跑完后立刻恢复可点,
手快连点会重复发起(2026-09-17 上架事故:同一店铺被接连提交三次,服务端并存多个
同店铺任务,客户端并发打开同一店铺时全部失败)。

- 新增 shared/utils/submit-guard.ts:冷却按按钮元素各自计算(WeakMap,点 A 不影响
  B),在捕获阶段拦截、抢在 Vue 的 @click 之前;冷却自首次有效点击起算,被拦的
  点击不会把冷却越拖越长;
- main.ts 全局安装 installSubmitGuard(),覆盖所有 .btn-run 主按钮,页面零改动;
- 13 个单测覆盖边界:不同按钮互不影响、被拦不延长冷却、非法时长回落默认等;
- client-changelog 每条精简到 40 字以内(超出面板显示不下,已有守护测试)。
2026-09-17 15:41:41 +08:00
huangzd1997 a89de129ea chore(客户端更新日志): 追加 4.0.28 条目
4.0.28 修复两个用户可感知的问题:
- 多个任务同时操作同一店铺导致「打开店铺失败」→ 同一店铺改排队执行;
- 紫鸟更新内核期间直接报「打开店铺失败」→ 改为等待更新完成并显示进度。
2026-09-17 15:32:47 +08:00
huangzd1997 ddefcbed56 feat(任务恢复): 外观专利接上「补传恢复」;合并冲突保留兜底对象并可诊断
28459 的两个遗留项:
1) 外观专利缺「补传恢复」入口——分片缺失致组装 job 重试耗尽后,客户端补传缺口
   也无法自动重跑,只能人工重置 job。TaskFileJobService 早有
   resetTerminalFailedForRecovery,但只被删除品牌模块接了。现按同一口径在分片
   提交成功后检查并恢复;best-effort,恢复失败不影响补传本身。
2) 合并 CAS 冲突不可诊断且会删掉唯一的兜底对象——原实现每次冲突都删掉刚写入的
   版本化对象,重试耗尽即抛异常、行仍指向旧指针;一旦旧对象也不在,该分片永久
   读不到(28459 的 chunk-462/473/484 正是这个形态)。现在冲突时读回行上当前
   哈希并写进异常与日志;终局失败保留最后一个对象,作为读路径「同槽位兄弟对象」
   兜底的恢复源。外观专利与相似ASIN 同一口径。

测试:新增 17 个用例(补传恢复 8 / 外观专利合并冲突 5 / 相似ASIN 合并冲突 4),
RED 均已确认;全量 mvn test 3099 个 0 失败。
2026-09-17 11:04:38 +08:00
huangzd1997 3137299bfe fix(临时载荷): 收口删除守卫 + chunk 读兜底,修「对象被误删→任务永久失败」
线上任务 28459(外观专利)三个分片的载荷对象被删、DB 行仍指向已删的确定性
key,组装读不到 → 整单 FAILED。数据其实还在同槽位的版本化对象里,是行指错了。

三条收口:
- 上传补偿删除加准入:只有末段带 UUID 的版本化 key(本次写入独占)才允许入队。
  确定性 key 会被重传复用,删它就删掉了行仍在引用的对象;而删除队列
  (deleteObjectFromRetry -> removeObject)本身不做引用反查,准入必须卡在入队处。
- 引用守卫阈值 >1 收紧为 >0:原来把「恰好 1 行引用」当作调用方自己那行而放行,
  但调用方无法证明归属。物理删除本就约定在 DB 行删除之后执行,正常路径引用数
  必为 0;宁可留孤儿(有保留期清理兜底),也不删可能仍被引用的对象。
- chunk 读兜底:指针对象确已缺失(NoSuchKey)时回退同槽位版本化对象 chunk-N-*。
  仅限 chunk 槽位 + 确为缺失两个条件,避免误配无关对象或掩盖真实故障。

测试:新增 31 个用例(补偿删除准入 / 引用守卫 / 兄弟对象兜底),均先确认 RED
再实现;另更新 3 个断言旧行为的既有用例。全量 mvn test 3082 个 0 失败。
2026-09-17 10:42:53 +08:00
huangzd1997 1fe3368c5a perf(rustfs): 重试基础退避 500ms→200ms
线上高频的重试诱因是 unexpected end of stream——那是**立即失败**(RustFS 重置连接后
OkHttp 读响应即报错),不是等超时,500ms 基本是白等;每天上千次累计十几分钟。
降到 200ms 保留退避语义(真遇服务端过载仍会让步),又不让用户多等。
2026-09-17 01:55:35 +08:00
huangzd1997 7aea3a0a50 fix(rustfs): 修 unexpected end of stream 根因——零长度请求体缺 Content-Length
抓包定位(90 秒内 515 次 HTTP 411 Length Required):OkHttp 对长度为 0 的请求体
不写 Content-Length,请求于是既无 Content-Length 也无 Transfer-Encoding——HTTP/1.1
不允许这样,RustFS 直接回 411;客户端读到不完整响应就报 unexpected end of stream
再重试。业务里上传空内容(content 为 null/空)是常态,所以每天上千次。

修复用 network interceptor 在协议层补 Content-Length: 0:普通 interceptor 里加的
头会被 BridgeInterceptor 按 body 长度覆盖,加了不生效。

实测:部署后同样 90 秒抓包,411 从 515 次降到 0,两节点「确定性错误不重试」归零。
此前排除的公网链路、keepAlive、连接复用三项确实都不是原因——抓包才是对的入口。
2026-09-17 01:36:55 +08:00
huangzd1997 188aedec84 fix(任务收尾): 打破「数据永久缺失 → 卡死恢复无限重建 job」的死循环
线上任务 28459(appearance_patent)卡在 RUNNING:chunk 462/473/484 的 payload
对象已不存在,组装失败 → stale recovery 每 30 秒重建一次 assemble job 再失败,
而恢复过程又会刷新任务心跳、任务永远判不出「卡死」,48 分钟空转近百轮。

- appearance-patent:读 chunk 时若 payload 对象已不存在(RustFS NoSuchKey),
  跳过该分片让任务按已有分片出部分结果,不再抛异常把组装永久拖死;
  similar-asin 在原有 typeMismatch 跳过分支旁补同款处理
- TaskFileJobService.hasExhaustedAssembleJob + stale recovery:已有「重试耗尽且
  已终态收尾」的 assemble job 时放弃恢复,交给 finalizeStaleTask 按失败收尾
  —— 用户看到明确失败,而不是无限 RUNNING

实测:部署后两节点 28459 相关日志、rustfs 确定性错误、stale recovery 重建全部归零。
2026-09-17 01:19:14 +08:00
huangzd1997 d5952945dd fix(rustfs): 确定性错误不再重试;回退两处经实测无效的连接池实验
- 新增 isNonRetryable:NoSuchKey / AccessDenied / 签名错误等确定性错误立即失败,
  不再白打两次请求(线上 read 一个已清理的 chunk 会连打 3 次并留下一条 ERROR)
- connectionPoolMaxIdle 0→5、keepAlive 30000→300000 回退:
  实测证明 unexpected end of stream 与连接复用无关——禁用复用后新容器起来的
  第一次请求照样中招。至此已排除公网链路、keepAlive 过长、连接复用三项;
  另用 mc 并发压 200 个小对象全部成功,说明 RustFS 服务端无问题。
  剩余方向指向 MinIO SDK(8.5.17,已是最新)/OkHttp 与 RustFS 的协议细节,
  需抓包对比 mc 与 Java 的请求才能定位。数据不丢(重试都能成功),影响是每次 +600ms。
2026-09-17 00:58:01 +08:00
huangzd1997 6e1689dfe5 docs(更新日志): 追加客户端 4.0.27 条目 2026-09-17 00:33:29 +08:00
huangzd1997 4f16a02658 fix(稳定性): 修任务锁超时/实例转发/扫描降噪,补 rustfs 结论日志
线上日志扫描发现的缺陷批量修复:
- TaskFileJobService.resetStuckRunningJobsDetailed 去掉跨行长事务:两节点
  @Scheduled 各取 200 行在同一事务里逐行 UPDATE,互等行锁导致 biz_task_file_job
  每天上百次 Lock wait timeout。改逐行独立提交(本就有 CAS 条件,幂等)
- TaskOwnerForwardService 连接类失败重试 3 次:两节点滚动重启窗口内的
  Connection refused 会直接丢掉用户提交的结果(读超时不重放,避免重复提交)
- AdminApiGuardFilter 把 401/被顶下线降为 debug(线上单节点一天近 3000 条噪声)
- NoResourceFoundException 单独处理:如实返回 404,不再刷 ERROR 堆栈、不再伪装
  200(每天 580 条,绝大多数是外部扫描 /.env、/credentials、aliyun.json 等)
- RustfsObjectStorageService 补「重试后成功 / 重试耗尽」结论日志:一天上千条
  retrying 却无结论,无法判断数据是否落盘
- 连接池 keep-alive 300s→30s,减少复用已被对端关闭的空闲连接
- SimilarAsin 提交结果遇到已删除任务时带 TASK_NOT_FOUND(40401),客户端据此
  停止每 30 秒一轮的无谓重试
- 修 SimilarAsinResultRowDto 的 GBK 乱码 JsonAlias("浠锋牸" → "价格")
2026-09-17 00:32:48 +08:00
huangzd1997 5e9a59b327 test(鉴权): 兜底过滤器用例改用不存在的样例豁免前缀
exemptPrefixes 是配置化操作出口(aiimage.security.admin-guard-exempt-prefixes,
默认空、生产未配置任何值),原用例拿已下线的 /api/admin/shop-credential-checks
当样例路径,容易被误读成「该端点已配好豁免」——那条路径是 2026-09-04 引入
豁免机制时随手取的,并非真的配过。换成显然不存在的 /api/admin/example-exempt,
并注明不要改成真实业务端点。
2026-09-17 00:13:27 +08:00
huangzd1997 ab09cff427 docs(更新日志): 追加客户端 4.0.26 条目 2026-09-16 22:55:31 +08:00
huangzd1997 47a9520a82 fix(跟价): 失败任务也产出并保留部分结果文件,不再只剩一条失败记录
会话掉线这类「跑到第 N 页才断」的场景,任务本就该是 FAILED,但已经跑出来的行
(哪些 ASIN 真实改价过)必须随任务一起交付,否则用户无从对账、也无法只重跑漏掉的。

- submitResult 的 error / success=false 分支改走 finalizeFailedShop:先按失败落库
  (success=0 + 原因),若合并后仍有可用行就排队组装部分结果文件;一行可用数据都没有时
  才退化成原来的纯失败(并清掉合并缓存)
- markResultFilePending / assembleShopResult 增加 preserveFailure 语义:失败行只挂文件,
  不把 success 洗成 1、也不清 errorMessage
- updateTaskStatusFromLatestRows:失败行正在组装文件时不让任务提前终态,
  否则前端一停轮询、下载按钮永不出现(失败任务的结果文件同样要能下载)

PriceTrackTaskServiceTest 新增 2 条契约测试(部分结果排队 + 组装完成后仍失败且可下载),
跟价模块测试 20/20 通过。
2026-09-16 20:33:31 +08:00
huangzd1997 7a6c3c3fa3 fix(菜单权限): 授权树按操作者置灰不可授予项 + 放行保留既有授权
普通管理员勾到无权授予的菜单后,ensureGrantable 抛 403,而 createUser/updateUser
都是 @Transactional,整笔回滚——线上表现就是「后台不能保存权限,也不能创建用户」
(2026-09-16 周丽娥账号创建用户与保存 uid=45 权限双双失败,库里查无新用户)。
根因是校验 2026-07-28 就加了,但授权树一直拿全量菜单,两边规则不一致。

- permission-menus 按操作者标记 grantable;授权树据此置灰(不隐藏:授权是整树替换,
  隐藏会把超管授予过、操作者自己没有的菜单当取消勾选删掉,与 09-13「权限自己没掉」同类)
- ensureGrantable 放行目标已持有的授权,只拦新增,不构成提权
- GlobalExceptionHandler 补业务异常日志:此前普通业务异常一行都不记,本次排查只能靠
  反推响应体字节数(nginx body_bytes_sent 含 chunked 开销)才定位到根因
2026-09-16 18:46:21 +08:00
huangzd1997 803b5d583d docs(更新日志): 追加客户端 4.0.25 条目
跟价「指定ASIN模式」搜索打到屏外的导航搜索框,客户端 4.0.25 已修(app_client 29c12f2)。
2026-09-16 17:15:06 +08:00
huangzd1997 1c52cd529b fix(去重): 历史任务弹层补下载/删除按钮(漏了 history-item-actions 插槽)
任务面板的 #item-actions 只作用于「当前任务」列表,历史结果全部只在
#history-item-actions 弹层里渲染。16 个工具页里唯独去重页只给了前者,
表现为用户反馈「历史任务里压根没下载按钮」(既无下载也无删除)。
接口数据(success/resultId/downloadUrl)完整、类型检查与单测均不报错,
只能靠点开历史弹层才能发现。
2026-09-16 17:05:11 +08:00
huangzd1997 ab28b168ec fix(去重): 主链接行位于其子链接行之后时未被丢弃 + 导出表头与数据行错列
线上任务 28422(文件 新数据变体完.xlsx)结果中 16 个主ID同时保留了主链接与
子链接,违反所选规则(keepIntegerIds=false / keepUnderscoreIds=true /
keepIntegerMainIdsWhenNoSubIds=true)。

根因:旧实现把整数主链接行暂存,依赖「后到的同主ID子链接行」把它丢弃。
当顺序为「子链接在前、主链接在后」时,子行到来时暂存区尚空,无人记录该主ID
已有子链接,主链接一路存活到 flush。改用 IdRuleRowPicker 先收集、收尾统一按
「该主ID是否出现过子链接行」判定,与源文件顺序无关;补 subRows/mainRows/
droppedMainRows 排查日志。

同时修一处潜在错列:表头按 selectedColumns 原顺序写、数据行按
orderedSelectedColumns(id/ASIN/国家/价格/品牌 提前)写,两者不同序时整体
错列;本任务所选列恰为导出优先级顺序故未暴露。

实测:用线上源文件重算,输出 7273 → 7257 行(正好少掉那 16 个主/子并存的主ID)。
2026-09-16 16:33:04 +08:00
huangzd1997 367b4b7553 chore(更新日志): 补 4.0.24 条目(跟价改版为行内输入+底部「全部保存」提交) 2026-09-16 16:03:42 +08:00
huangzd1997 25c7323c47 chore(更新日志): 补 4.0.23 条目(跟价适配亚马逊新版商品页) 2026-09-16 13:40:59 +08:00
huangzd1997 aea0e16279 feat(站内通知): 铃铛支持按类型大类筛选(系统异常/任务异常/配置异常/系统通知)
原来铃铛只能按关键字和日期筛,任务失败、密钥欠费、麦象异常全混在一起,
用户没法只看自己关心的那类。

后端 NotificationService 增加 scene→大类映射(未知 scene 归「系统通知」兜底),
列表接口新增 category 参数,落到 SQL 是 scene IN (...);「系统通知」作兜底类
还要纳入未登记的 scene,故表达为「= system 或 不在其它三类里」。列表项返回
category,前端不必各自维护一份 scene 映射。前后端两个通知接口同步加参数。

前端铃铛筛选区新增一行类型 chip(全部 + 四类),与关键字、日期一起参与
hasFilter 与重置;空分类不下发,避免后端查询落空。
2026-09-16 10:32:11 +08:00
huangzd1997 f566573fce fix(菜单权限): 后台侧边栏为部分授权用户补全祖先分组(二级菜单不再平铺)
只勾选分组内子页面、未勾选分组本身时,getUserColumnPermissions 只返回
直接授权+展开后代,分组容器节点缺失,AdminMenuTreeBuilder 把子页面当根节点,
前端侧边栏渲染成一列平铺;且子页面 sort_order 是全局值、跨分组穿插显示,
与超管的「分组+组内顺序」视图完全对不上(线上 31 个账号如此,含 uid=972 阿武)。

修复:getUserColumnPermissions 增加 includeAncestorGroups 重载(默认 false),
true 时沿 parentId 链把有可见后代的祖先并入集合——分组无页面路由,仅还原
展示层级,不构成授权扩展;只有 current-user/menus(后台侧边栏)启用。

刻意保持原语义的出口:/permission-users/{id}/column-permissions 与登录响应
(桌面端 tool-catalog「组键命中即整组放行」,补组键会误放行整组工具)、
dedupe/invalidasin 的精确 key 校验。

新增 3 个测试:祖先补全含多层链、树组装还原分组与组内顺序、默认出口不含祖先。
2026-09-16 09:58:33 +08:00
huangzd1997 8fcceb3226 fix(撞款扫描): 保留期 90→7 天,并单独兜住最新 SUCCESS 行
该表每行含整份聚合 payload(线上实测约 2.3MB/行,24 行占 55MB),
而读取侧只认最新一行(selectLatestFullRow/LightRow 都是 ORDER BY id DESC LIMIT 1),
历史行纯占磁盘。保留期降到 7 天,行数下限仍由 PROTECTED_ROWS=10 兜底。

同时补一条保护线:连续失败多日时,最新 SUCCESS 行会滑出「最新 N 行」保护窗口,
再按时间线被删掉,界面就空白了——现在取「第 N 新行」与「最新 SUCCESS 行」
两者更靠前的那个 id 作为保护线,确保它永不被删。
2026-09-16 09:25:30 +08:00
huangzd1997 ed0d6575c8 feat(前端): 追加客户端 4.0.22 更新日志条目
包体积 -38%、启动时清理本地垃圾、启动与连接优化三条面向用户的说明。
version 与 web_config 发布记录一致(4.0.22),否则更新面板按本机版本筛选匹配不到。
2026-09-16 09:09:05 +08:00
huangzd1997 f2ada02383 feat(保留期清理): 补齐 7 处只增不删的数据;修结果对象孤儿
审核发现一批表/元数据只增不删,且对象存储存在活跃泄漏:

- module-cleanup 漏收 task_result_item / task_result_payload 的 payload 指针,
  每晚删完行就在 json-server 桶留下孤儿对象;指针收集触顶从"截断后照样删整组"
  改为任务组二分拆分,拆到单任务仍触顶则整组不删并记 error(截断会让指针随行消失)
- 结果文件对象此前从不回收(只删 DB 行、对象留在桶里),现改为事务提交后
  按 file_result / task_file_job 的 result_file_url 逐个回收
- 新增 7 个保留期清理:device_log_file 元数据、price_track_loop_run、撞款扫描行、
  密钥用量日统计、紫鸟记忆过期行、PUBLISH 任务、BRAND 任务。后两者刻意不复用
  module-cleanup(BRAND 不写 biz_file_task;PUBLISH 单任务可达数万行),
  改为调各自既有的业务删除入口,保证结果对象与子表一起回收
- 未读通知补保留期(180 天,已读仍 90 天);ZiniaoMemoryStoreService.deleteExpired
  此前全库无调用方(实现了没接线),现已挂定时任务

全部沿用既有范式:@Scheduled + 分布式锁单实例 + 分批 + 单轮批次数上限 + 失败只记日志。
2026-09-16 01:59:28 +08:00
huangzd1997 5e5816cd74 fix(站内通知): 麦象异常扫描滞留检查改用 update_time 过滤(18960 console 新增 update_time_end)——原按 create_time 过滤 + desc(id) 分页只能看到最新批次,积压深处的老任务永远看不到(2026-09-15 滞留在队 9 小时未被告警指出的漏报根因) 2026-09-16 01:05:26 +08:00
huangzd1997 d189d94c3b fix(更新日志): 面板随指定版本下拉联动;本机已是最新时展示本机条目
- 「指定版本更新」下拉选中版本时日志切到该版本自身的条目(升级/回退统一语义),
  未收录显示"暂无更新说明"(区块不消失),选回空项回默认区间;
  列表加载后自动默认选中第一条不触发切换
- 本机版本==线上最新时回退显示本机版本条目(原来整块消失),灰度/回滚仍隐藏
- 精简 4.0.20/4.0.21 超长文案,修复既有红测试(更新说明限 40 字)
2026-09-16 01:02:13 +08:00
huangzd1997 79b5d40327 feat(设备日志/后台配置/密钥检测): 补齐已上线未提交的设备日志与app_config,并合入检测模型解耦
三部分均已部署到双节点(当前线上 JAR 009efc3e),本次补齐仓库状态,避免"已上线未提交"
在后续最小构建比对里被误判。

- 设备日志管理:modules/devicelog + DeviceLogOssProperties(主机B 独立 MinIO,仅内网)
  + V127 device_log_file/device_log_config + admin-vue 日志管理页;客户端/麦象按 offset
  增量上报(X-Internal-Token),查询仅超管
- 后台通用配置:modules/appconfig + V128 app_config 键值表;工作台「开店流程」密码改服务端
  校验(POST /api/kd-flow/verify),改密码只需 UPDATE 该行、客户端无需重新发版
- 密钥检测模型解耦:新增 aiimage.user-secret.check-model(env AIIMAGE_USER_SECRET_CHECK_MODEL),
  默认 doubao-seed-2-0-lite-260215 —— 同系列 mini 在中继分组下无可用渠道(503 model_not_found),
  实测 lite 可路由;UserSecretModule 去掉 LlmTarget 改 resolveLlmHost,探测日志带 model=
- 前端:密钥面板「检测配置密钥」按钮不再折行;client-changelog 补 4.0.19/4.0.20/4.0.21 条目

注:application.yml 与 PropertiesConfig 同时承载上述多个部分,故未按功能拆分提交
2026-09-15 23:53:05 +08:00
205 changed files with 10141 additions and 375 deletions
@@ -0,0 +1,44 @@
import { expect, test, type Page } from '@playwright/test'
// 后台管理列表「创建时间 + 更新时间」两列验收(2026-09-19 需求):
// 8 个实体管理列表统一展示创建/更新时间,数据取自各列表 VO。
// mock 夹具统一给 createdAt=2026-03-01 09:15:00、updatedAt=2026-09-18 16:42:30
// 页面统一经 formatDateTime 归一为「YYYY-MM-DD HH:mm:ss」。
const CREATED = '2026-03-01 09:15:00'
const UPDATED = '2026-09-18 16:42:30'
const PAGES: Array<{ title: string; path: string; heading: string }> = [
{ title: '用户管理', path: '/admin-vue/account/users', heading: '用户列表' },
{ title: '菜单管理', path: '/admin-vue/account/menus', heading: '菜单' },
{ title: '密钥管理', path: '/admin-vue/account/user-secrets', heading: '用户密钥列表' },
{ title: '不符合ASIN数据', path: '/admin-vue/asin-center/invalid', heading: '品牌数据列表' },
{ title: '数据去重总数据', path: '/admin-vue/asin-center/registry', heading: '数据去重总数据' },
{ title: '查询ASIN', path: '/admin-vue/asin-center/query', heading: '查询' },
{ title: '最低价ASIN', path: '/admin-vue/asin-center/skip-price', heading: '最低价' },
{ title: '商品类目', path: '/admin-vue/asin-center/categories', heading: '商品类目' },
]
async function openPage(page: Page, path: string) {
await page.goto(path)
await expect(page.locator('.panel-box table tbody tr').first()).toBeVisible({ timeout: 15000 })
}
for (const spec of PAGES) {
test(`${spec.title}:表格展示创建时间与更新时间`, async ({ page }) => {
const errors: string[] = []
page.on('pageerror', (error) => errors.push(error.message))
await openPage(page, spec.path)
const table = page.locator('.panel-box table').first()
// 表头出现两列
await expect(table.locator('thead')).toContainText('创建时间')
await expect(table.locator('thead')).toContainText('更新时间')
// 数据行渲染出真实时间值(非空、非占位符)
await expect(table.locator('tbody')).toContainText(CREATED)
await expect(table.locator('tbody')).toContainText(UPDATED)
expect(errors).toEqual([])
})
}
@@ -220,13 +220,183 @@ const server = createServer((req, res) => {
return json(res, {
success: true,
data: {
items: [{ id: 1, username: 'admin', role: 'super_admin', creatorUsername: 'system', createdAt: '2026-01-01 00:00:00' }],
items: [{ id: 1, username: 'admin', role: 'super_admin', creatorUsername: 'system', created_at: '2026-03-01 09:15:00', updated_at: '2026-09-18 16:42:30' }],
total: 1,
},
})
}
if (url === '/api/admin/permission-menus') {
return json(res, { success: true, data: { items: [] } })
// 菜单管理列表:一条根节点,带创建/更新时间(验收时间列)。
return json(res, {
success: true,
data: [
{
id: 1,
name: '用户管理',
column_key: 'admin_users',
parent_id: null,
menu_type: 'admin',
route_path: 'account/users',
sort_order: 10,
created_at: '2026-03-01T09:15:00',
updated_at: '2026-09-18T16:42:30',
},
],
})
}
if (url === '/api/admin/invalid-asin-data') {
return json(res, {
success: true,
data: {
items: [
{
id: 1,
dataValue: 'B0INVALID1',
brand: '测试品牌',
groupId: 1,
groupName: '测试分组',
recordSource: 'MANUAL',
createdAt: '2026-03-01 09:15:00',
updatedAt: '2026-09-18 16:42:30',
},
],
total: 1,
page: 1,
pageSize: 10,
},
})
}
if (url === '/api/admin/dedupe-total-data') {
return json(res, {
success: true,
data: {
items: [
{
id: 1,
dataValue: 'B0DEDUPE01',
country: 'DE',
groupId: 1,
groupName: '测试分组',
uploaderUserId: 1,
username: 'admin',
createdAt: '2026-03-01 09:15:00',
updatedAt: '2026-09-18 16:42:30',
},
],
total: 1,
page: 1,
pageSize: 10,
},
})
}
if (url === '/api/admin/query-asins') {
return json(res, {
success: true,
data: {
items: [
{
id: 1,
groupId: 1,
groupName: '测试分组',
shopName: '测试店铺',
asinDe: 'B0QUERY001',
asinUk: '',
asinFr: '',
asinIt: '',
asinEs: '',
createdAt: '2026-03-01 09:15:00',
updatedAt: '2026-09-18 16:42:30',
},
],
total: 1,
page: 1,
pageSize: 10,
},
})
}
if (url === '/api/admin/skip-price-asins') {
return json(res, {
success: true,
data: {
items: [
{
id: 1,
groupId: 1,
groupName: '测试分组',
shopName: '测试店铺',
asinDe: 'B0SKIP0001',
minimumPriceDe: 9.99,
asinUk: '',
minimumPriceUk: null,
asinFr: '',
minimumPriceFr: null,
asinIt: '',
minimumPriceIt: null,
asinEs: '',
minimumPriceEs: null,
createdAt: '2026-03-01 09:15:00',
updatedAt: '2026-09-18 16:42:30',
},
],
total: 1,
page: 1,
pageSize: 10,
},
})
}
if (url === '/api/admin/product-categories/children') {
return json(res, {
success: true,
data: {
items: [
{
id: 1,
parentId: null,
name: '护肤品',
categoryKey: 'skincare',
sortOrder: 10,
description: '测试备注',
isBuiltin: true,
childCount: 0,
level: 0,
path: '护肤品',
createdAt: '2026-03-01 09:15:00',
updatedAt: '2026-09-18 16:42:30',
},
],
total: 1,
page: 1,
pageSize: 10,
hasMore: false,
},
})
}
if (url === '/api/admin/user-secrets') {
const emptyModule = { moduleKey: '', moduleLabel: '', masked: '', full: '', exists: false, checkStatus: 'unknown', checkCode: '', checkMessage: '', checkLatencyMs: null, checkedAt: null, updatedAt: null }
return json(res, {
success: true,
data: {
items: [
{
userId: 1,
username: 'admin',
groups: [],
leaderUsername: '',
similarAsin: { ...emptyModule, moduleKey: 'similar-asin', moduleLabel: '货源查询密钥' },
appearancePatent: { ...emptyModule, moduleKey: 'appearance-patent', moduleLabel: '外观专利密钥' },
proxy: { ...emptyModule, moduleKey: 'proxy', moduleLabel: '代理设置' },
status: 'unknown',
statusMessage: '',
createdAt: '2026-03-01 09:15:00',
updatedAt: '2026-09-18 16:42:30',
},
],
total: 1,
page: 1,
pageSize: 15,
groupOptions: [],
},
})
}
if (url === '/api/admin/shop-manage-groups') {
return json(res, { success: true, data: { items: [] } })
+135
View File
@@ -0,0 +1,135 @@
import { http } from './http'
import { unwrap } from './envelope'
/** 日志文件行(桌面客户端 / 麦象采集机上报)。 */
export interface DeviceLogFileRow {
id: number
source: string
deviceId: string
deviceName: string | null
username: string | null
uid: number | null
fileName: string
logDate: string
uploadedBytes: number
partCount: number
lastUploadAt: string | null
createdAt: string | null
}
export interface DeviceLogPage {
items: DeviceLogFileRow[]
total: number
page: number
pageSize: number
/** 云端日志保留天数(页面提示用)。 */
retentionDays: number
}
export interface DeviceLogContent {
fileId: number
fileName: string
content: string
totalBytes: number
shownBytes: number
truncated: boolean
}
export interface DeviceLogDevice {
source: string
deviceId: string
deviceName: string | null
lastUploadAt: string | null
}
export interface DeviceLogOverride {
id: number
source: string
deviceId: string
deviceName: string | null
mode: string
updatedAt: string | null
}
export interface DeviceLogConfigData {
globalMode: string
overrides: DeviceLogOverride[]
}
export interface DeviceLogQuery {
source?: string
keyword?: string
startDate?: string
endDate?: string
page: number
pageSize: number
}
/** 分页查询日志文件列表:GET /api/admin/device-logs/files */
export async function fetchDeviceLogFiles(params: DeviceLogQuery): Promise<DeviceLogPage> {
const query: Record<string, string | number> = { page: params.page, pageSize: params.pageSize }
if (params.source) query.source = params.source
if (params.keyword) query.keyword = params.keyword
if (params.startDate) query.startDate = params.startDate
if (params.endDate) query.endDate = params.endDate
const { data } = await http.get('/api/admin/device-logs/files', { params: query })
return unwrap<DeviceLogPage>(data)
}
/** 查看日志尾部内容:GET /api/admin/device-logs/content */
export async function fetchDeviceLogContent(fileId: number, maxBytes?: number): Promise<DeviceLogContent> {
const query: Record<string, number> = { fileId }
if (maxBytes) query.maxBytes = maxBytes
const { data } = await http.get('/api/admin/device-logs/content', { params: query })
return unwrap<DeviceLogContent>(data)
}
/** 完整日志下载地址(同域 cookie 鉴权,直接给 a[href] 或 window.open 用)。 */
export function deviceLogDownloadUrl(fileId: number): string {
return `/api/admin/device-logs/download?fileId=${fileId}`
}
/** 删除日志文件(片段与元数据,不可恢复):DELETE /api/admin/device-logs/{id} */
export async function deleteDeviceLogFile(id: number): Promise<void> {
const { data } = await http.delete(`/api/admin/device-logs/${id}`)
unwrap<unknown>(data)
}
/** 采集配置(全局默认 + 终端覆盖):GET /api/admin/device-logs/config */
export async function fetchDeviceLogConfig(keyword?: string): Promise<DeviceLogConfigData> {
const { data } = await http.get('/api/admin/device-logs/config', {
params: keyword ? { keyword } : undefined,
})
return unwrap<DeviceLogConfigData>(data)
}
/** 最近上报过的终端(覆盖选择用):GET /api/admin/device-logs/devices */
export async function fetchDeviceLogDevices(): Promise<DeviceLogDevice[]> {
const { data } = await http.get('/api/admin/device-logs/devices')
return unwrap<DeviceLogDevice[]>(data)
}
/** 设置全局采集模式:PUT /api/admin/device-logs/config/global */
export async function updateDeviceLogGlobalMode(mode: string): Promise<void> {
const { data } = await http.put('/api/admin/device-logs/config/global', undefined, { params: { mode } })
unwrap<unknown>(data)
}
/** 设置/更新终端覆盖:PUT /api/admin/device-logs/config/device */
export async function updateDeviceLogOverride(
source: string,
deviceId: string,
deviceName: string | null,
mode: string,
): Promise<void> {
const { data } = await http.put('/api/admin/device-logs/config/device', undefined, {
params: { source, deviceId, deviceName: deviceName || undefined, mode },
})
unwrap<unknown>(data)
}
/** 删除终端覆盖(回落到全局默认):DELETE /api/admin/device-logs/config/device/{id} */
export async function deleteDeviceLogOverride(id: number): Promise<void> {
const { data } = await http.delete(`/api/admin/device-logs/config/device/${id}`)
unwrap<unknown>(data)
}
@@ -30,6 +30,8 @@ export interface AdminUserSecretRow {
proxy: AdminUserSecretModule
status: string
statusMessage: string
/** 首次配置时间(三模块中最早);从未配置为 null。 */
createdAt: string | null
updatedAt: string | null
}
@@ -23,6 +23,8 @@ export function toAdminUserItem(raw: unknown): AdminUser | null {
if (createdById !== null) item.createdById = createdById
const createdAt = text(r.created_at)
if (createdAt) item.createdAt = createdAt
const updatedAt = text(r.updated_at)
if (updatedAt) item.updatedAt = updatedAt
const creator = text(r.creator_username)
if (creator) item.creatorUsername = creator
const abbr = text(r.pinyin_abbr)
@@ -241,6 +241,7 @@ onMounted(loadMenus)
<th style="width: 100px">菜单类型</th>
<th style="width: 120px">上级菜单</th>
<th style="width: 170px">创建时间</th>
<th style="width: 170px">更新时间</th>
<th style="width: 170px">操作</th>
</tr>
</thead>
@@ -271,6 +272,7 @@ onMounted(loadMenus)
<td><span class="menu-type-text">{{ menuTypeLabel(row.menuType) }}</span></td>
<td>{{ parentNameOf(row) }}</td>
<td>{{ formatDateTime(row.createdAt) }}</td>
<td>{{ formatDateTime(row.updatedAt) }}</td>
<td class="ops-cell">
<button class="btn btn-sm" type="button" @click="openEdit(row)">编辑</button>
<button
@@ -286,10 +288,10 @@ onMounted(loadMenus)
</tr>
</template>
<tr v-else-if="loading">
<td colspan="7" class="empty-tip">加载中...</td>
<td colspan="8" class="empty-tip">加载中...</td>
</tr>
<tr v-else>
<td colspan="7" class="empty-tip">{{ filterName || filterType ? '暂无匹配菜单' : '暂无菜单,请先在上方新增' }}</td>
<td colspan="8" class="empty-tip">{{ filterName || filterType ? '暂无匹配菜单' : '暂无菜单,请先在上方新增' }}</td>
</tr>
</tbody>
</table>
@@ -77,7 +77,7 @@ watch(
node-key="id"
show-checkbox
default-expand-all
:props="{ label: 'name', children: 'children' }"
:props="{ label: 'name', children: 'children', disabled: 'disabled' }"
@check="onCheck"
/>
</div>
@@ -89,7 +89,7 @@ watch(
node-key="id"
show-checkbox
default-expand-all
:props="{ label: 'name', children: 'children' }"
:props="{ label: 'name', children: 'children', disabled: 'disabled' }"
@check="onCheck"
/>
</div>
@@ -233,6 +233,8 @@ onMounted(load)
<th style="width: 230px">外观专利密钥</th>
<th style="width: 380px">代理设置</th>
<th style="width: 130px">状态</th>
<th style="width: 170px">创建时间</th>
<th style="width: 170px">更新时间</th>
<th style="width: 170px">操作</th>
</tr>
</thead>
@@ -278,6 +280,8 @@ onMounted(load)
{{ rowStatusMeta(row.status).label }}
</span>
</td>
<td>{{ formatDateTime(row.createdAt) }}</td>
<td>{{ formatDateTime(row.updatedAt) }}</td>
<td class="ops-cell">
<button
class="btn btn-sm"
@@ -292,10 +296,10 @@ onMounted(load)
</tr>
</template>
<tr v-else-if="loading">
<td :colspan="isSuperAdmin ? 8 : 7" class="empty-tip">加载中...</td>
<td :colspan="isSuperAdmin ? 10 : 9" class="empty-tip">加载中...</td>
</tr>
<tr v-else>
<td :colspan="isSuperAdmin ? 8 : 7" class="empty-tip">{{ keyword || statusFilter || groupFilter ? '暂无匹配记录' : '暂无用户密钥记录' }}</td>
<td :colspan="isSuperAdmin ? 10 : 9" class="empty-tip">{{ keyword || statusFilter || groupFilter ? '暂无匹配记录' : '暂无用户密钥记录' }}</td>
</tr>
</tbody>
</table>
@@ -169,6 +169,7 @@ onMounted(loadUsers)
<th style="width: 140px">角色</th>
<th style="width: 160px">所属管理员</th>
<th style="width: 180px">创建时间</th>
<th style="width: 180px">更新时间</th>
<th style="width: 160px">操作</th>
</tr>
</thead>
@@ -180,6 +181,7 @@ onMounted(loadUsers)
<td>{{ roleLabel(row.role) }}</td>
<td>{{ row.creatorUsername || '-' }}</td>
<td>{{ formatDateTime(row.createdAt) }}</td>
<td>{{ formatDateTime(row.updatedAt) }}</td>
<td class="ops-cell">
<button
class="btn btn-sm"
@@ -203,10 +205,10 @@ onMounted(loadUsers)
</tr>
</template>
<tr v-else-if="loading">
<td colspan="6" class="empty-tip">加载中...</td>
<td colspan="7" class="empty-tip">加载中...</td>
</tr>
<tr v-else>
<td colspan="6" class="empty-tip">{{ userListEmptyHint() }}</td>
<td colspan="7" class="empty-tip">{{ userListEmptyHint() }}</td>
</tr>
</tbody>
</table>
@@ -15,6 +15,7 @@ export interface MenuManageNode {
routePath: string
sortOrder: number
createdAt?: string
updatedAt?: string
children?: MenuManageNode[]
}
@@ -90,6 +91,8 @@ export function parseMenuManageItem(raw: unknown): MenuManageNode | null {
if (rootColumnKey) node.rootColumnKey = rootColumnKey
const createdAt = text(record.created_at)
if (createdAt) node.createdAt = createdAt
const updatedAt = text(record.updated_at)
if (updatedAt) node.updatedAt = updatedAt
return node
}
@@ -16,6 +16,11 @@ export interface MenuOptionNode {
parentId: number | null
/** 所属菜单类型(admin 后台 / app 前端客户端),提交时按类型分区落库。 */
type: string
/**
* 当前操作者无权授予(后端 grantable=false)时置灰:仍展示并回显已勾选,
* 但不允许改勾选。非超管只能授自己已有的菜单,勾到越权项会让整笔保存回滚。
*/
disabled?: boolean
children?: MenuOptionNode[]
}
@@ -39,6 +44,8 @@ export function parsePermissionMenuItem(raw: unknown, type = ''): MenuOptionNode
sort: sortRaw === null ? 0 : sortRaw,
parentId: parentId === null ? null : parentId,
type,
// 缺省(菜单管理页等未标记的接口)按可授予处理,保持旧行为
disabled: record.grantable === false,
}
}
@@ -202,6 +202,7 @@ onMounted(() => {
<th v-if="isSuperAdmin" style="width: 140px">分组</th>
<th style="width: 110px">来源</th>
<th style="width: 170px">创建时间</th>
<th style="width: 170px">更新时间</th>
<th style="width: 150px">操作</th>
</tr>
</thead>
@@ -218,6 +219,7 @@ onMounted(() => {
</span>
</td>
<td>{{ formatDateTime(row.createdAt) }}</td>
<td>{{ formatDateTime(row.updatedAt) }}</td>
<td class="ops-cell">
<button class="btn btn-sm" type="button" @click="openEdit(row)">编辑</button>
<button class="btn btn-sm btn-danger" type="button" @click="remove(row)">删除</button>
@@ -225,10 +227,10 @@ onMounted(() => {
</tr>
</template>
<tr v-else-if="loading">
<td :colspan="isSuperAdmin ? 7 : 6" class="empty-tip">加载中...</td>
<td :colspan="isSuperAdmin ? 8 : 7" class="empty-tip">加载中...</td>
</tr>
<tr v-else>
<td :colspan="isSuperAdmin ? 7 : 6" class="empty-tip">暂无数据</td>
<td :colspan="isSuperAdmin ? 8 : 7" class="empty-tip">暂无数据</td>
</tr>
</tbody>
</table>
@@ -375,6 +375,7 @@ onMounted(() => {
<th>用户名</th>
<th v-if="isSuperAdmin">分组</th>
<th>创建时间</th>
<th>更新时间</th>
<th>操作</th>
</tr>
</thead>
@@ -387,6 +388,7 @@ onMounted(() => {
<td>{{ row.username || '-' }}</td>
<td v-if="isSuperAdmin">{{ row.groupName || '未分组' }}</td>
<td>{{ formatDateTime(row.createdAt) }}</td>
<td>{{ formatDateTime(row.updatedAt) }}</td>
<td class="ops-cell">
<button class="btn btn-sm" type="button" @click="openEdit(row)">编辑</button>
<button class="btn btn-sm btn-danger" type="button" @click="removeRow(row)">删除</button>
@@ -394,10 +396,10 @@ onMounted(() => {
</tr>
</template>
<tr v-else-if="loading">
<td :colspan="isSuperAdmin ? 7 : 6" class="empty-tip">加载中...</td>
<td :colspan="isSuperAdmin ? 8 : 7" class="empty-tip">加载中...</td>
</tr>
<tr v-else>
<td :colspan="isSuperAdmin ? 7 : 6" class="empty-tip">暂无总数据</td>
<td :colspan="isSuperAdmin ? 8 : 7" class="empty-tip">暂无总数据</td>
</tr>
</tbody>
</table>
@@ -15,6 +15,7 @@ import {
} from './product-category-model.ts'
import { categoryDraftError, createProductCategoryDraft, toCategorySaveRequest, type ProductCategoryDraft } from './product-category-editor-model.ts'
import { buildProductCategoryExportUrl } from './product-category-export.ts'
import { formatDateTime } from '@/utils/datetime'
import OldPagination from '@/components/OldPagination.vue'
const loading = ref(false)
@@ -330,6 +331,8 @@ onMounted(loadTree)
<th style="width: 90px">排序</th>
<th style="width: 120px">来源</th>
<th>备注</th>
<th style="width: 170px">创建时间</th>
<th style="width: 170px">更新时间</th>
<th style="width: 150px">操作</th>
</tr>
</thead>
@@ -337,7 +340,7 @@ onMounted(loadTree)
<template v-if="treeRows.length">
<template v-for="row in treeRows" :key="String(row.node.id)">
<tr v-if="isCategoryLoadMoreNode(row.node)" class="load-more-row">
<td colspan="6">
<td colspan="8">
<div class="load-more-cell">
<span class="tree-indent" :style="{ width: `${(row.depth + 1) * 24}px` }"></span>
<button class="btn btn-sm btn-secondary" type="button" :disabled="moreLoading(row.node)" @click="loadMoreChildren(row.node)">
@@ -376,6 +379,8 @@ onMounted(loadTree)
<span v-if="row.node.description" class="node-desc" :title="row.node.description">{{ row.node.description }}</span>
<span v-else class="dim"></span>
</td>
<td>{{ formatDateTime(row.node.createdAt) }}</td>
<td>{{ formatDateTime(row.node.updatedAt) }}</td>
<td class="ops-cell">
<button class="btn btn-sm" type="button" @click="openEdit(row.node)">编辑</button>
<button
@@ -392,10 +397,10 @@ onMounted(loadTree)
</template>
</template>
<tr v-else-if="loading">
<td colspan="6" class="empty-tip">加载中...</td>
<td colspan="8" class="empty-tip">加载中...</td>
</tr>
<tr v-else>
<td colspan="6" class="empty-tip">暂无商品类目</td>
<td colspan="8" class="empty-tip">暂无商品类目</td>
</tr>
</tbody>
<tbody v-else>
@@ -411,6 +416,8 @@ onMounted(loadTree)
<span v-if="row.description" class="node-desc" :title="row.description">{{ row.description }}</span>
<span v-else class="dim"></span>
</td>
<td>{{ formatDateTime(row.createdAt) }}</td>
<td>{{ formatDateTime(row.updatedAt) }}</td>
<td class="ops-cell">
<button class="btn btn-sm" type="button" @click="openEdit(row)">编辑</button>
<button
@@ -426,10 +433,10 @@ onMounted(loadTree)
</tr>
</template>
<tr v-else-if="searchLoading">
<td colspan="6" class="empty-tip">搜索中...</td>
<td colspan="8" class="empty-tip">搜索中...</td>
</tr>
<tr v-else>
<td colspan="6" class="empty-tip">暂无搜索结果</td>
<td colspan="8" class="empty-tip">暂无搜索结果</td>
</tr>
</tbody>
</table>
@@ -4,6 +4,7 @@
import { computed, onMounted, reactive, ref } from 'vue'
import { ElMessage, ElMessageBox } from 'element-plus'
import CopyText from '@/components/CopyText.vue'
import { formatDateTime } from '@/utils/datetime'
import { createQueryAsin, fetchQueryAsinList, fetchShopNamesByGroup } from './query-asin-api.ts'
import { QUERY_ASIN_COUNTRIES, queryAsinDisplayRows, type QueryAsinItem } from './query-asin-model.ts'
import { asinCountryLabel } from './asin-country.ts'
@@ -394,6 +395,8 @@ onMounted(() => {
<th style="width: 15%">店铺名</th>
<th style="width: 24%">ASIN</th>
<th style="width: 12%">国家</th>
<th style="width: 170px">创建时间</th>
<th style="width: 170px">更新时间</th>
<th style="width: 9%">操作</th>
</tr>
</thead>
@@ -413,6 +416,8 @@ onMounted(() => {
</td>
<td>{{ asinCountryLabel(row.country || '') }}</td>
<template v-if="row.isFirst">
<td :rowspan="row.rowspan">{{ formatDateTime(row.item.createdAt) }}</td>
<td :rowspan="row.rowspan">{{ formatDateTime(row.item.updatedAt) }}</td>
<td :rowspan="row.rowspan" class="asin-col-actions">
<button class="btn btn-sm" type="button" @click="openConfig(row.item as QueryAsinItem)">配置</button>
<button class="btn btn-sm btn-danger" type="button" @click="removeShopAsin(row.item as QueryAsinItem)">删除</button>
@@ -421,10 +426,10 @@ onMounted(() => {
</tr>
</template>
<tr v-else-if="loading">
<td :colspan="isSuperAdmin ? 6 : 5" class="empty-tip">加载中...</td>
<td :colspan="isSuperAdmin ? 8 : 7" class="empty-tip">加载中...</td>
</tr>
<tr v-else>
<td :colspan="isSuperAdmin ? 6 : 5" class="empty-tip">暂无数据</td>
<td :colspan="isSuperAdmin ? 8 : 7" class="empty-tip">暂无数据</td>
</tr>
</tbody>
</table>
@@ -4,6 +4,7 @@
import { computed, onMounted, reactive, ref } from 'vue'
import { ElMessage, ElMessageBox } from 'element-plus'
import CopyText from '@/components/CopyText.vue'
import { formatDateTime } from '@/utils/datetime'
import { createSkipPriceAsin, fetchSkipPriceList } from './skip-price-api.ts'
import { skipPriceDisplayRows, type SkipPriceItem } from './skip-price-model.ts'
import { asinCountryLabel } from './asin-country.ts'
@@ -447,6 +448,8 @@ onMounted(() => {
<th style="width: 24%">ASIN</th>
<th style="width: 12%">国家</th>
<th style="width: 10%">最低价</th>
<th style="width: 170px">创建时间</th>
<th style="width: 170px">更新时间</th>
<th style="width: 9%">操作</th>
</tr>
</thead>
@@ -467,6 +470,8 @@ onMounted(() => {
<td>{{ asinCountryLabel(row.country || '') }}</td>
<td class="price-cell">{{ row.minimumPrice !== '' ? row.minimumPrice : '-' }}</td>
<template v-if="row.isFirst">
<td :rowspan="row.rowspan">{{ formatDateTime(row.item.createdAt) }}</td>
<td :rowspan="row.rowspan">{{ formatDateTime(row.item.updatedAt) }}</td>
<td :rowspan="row.rowspan" class="asin-col-actions">
<button class="btn btn-sm" type="button" @click="openConfig(row.item as SkipPriceItem)">配置</button>
<button class="btn btn-sm btn-danger" type="button" @click="removeRow(row.item as SkipPriceItem)">删除</button>
@@ -475,10 +480,10 @@ onMounted(() => {
</tr>
</template>
<tr v-else-if="loading">
<td :colspan="isSuperAdmin ? 7 : 6" class="empty-tip">加载中...</td>
<td :colspan="isSuperAdmin ? 9 : 8" class="empty-tip">加载中...</td>
</tr>
<tr v-else>
<td :colspan="isSuperAdmin ? 7 : 6" class="empty-tip">暂无数据</td>
<td :colspan="isSuperAdmin ? 9 : 8" class="empty-tip">暂无数据</td>
</tr>
</tbody>
</table>
@@ -11,6 +11,7 @@ export interface DedupeTotalItem {
uploaderUserId: number | null
username: string
createdAt?: string
updatedAt?: string
}
export interface DedupeTotalPageResult {
@@ -51,6 +52,8 @@ export function toDedupeTotalItem(raw: unknown): DedupeTotalItem | null {
}
const createdAt = text(record.createdAt ?? record.created_at)
if (createdAt) item.createdAt = createdAt
const updatedAt = text(record.updatedAt ?? record.updated_at)
if (updatedAt) item.updatedAt = updatedAt
return item
}
@@ -14,6 +14,8 @@ export interface ProductCategoryNode {
childCount: number
level: number | null
path: string
createdAt?: string
updatedAt?: string
children: ProductCategoryNode[]
}
@@ -72,6 +74,10 @@ export function toProductCategoryNode(raw: unknown): ProductCategoryNode | null
path: text(record.path),
children: [],
}
const createdAt = text(record.createdAt ?? record.created_at)
if (createdAt) node.createdAt = createdAt
const updatedAt = text(record.updatedAt ?? record.updated_at)
if (updatedAt) node.updatedAt = updatedAt
if (Array.isArray(record.children)) {
node.children = record.children
.map((child) => toProductCategoryNode(child))
@@ -12,6 +12,7 @@ export interface InvalidAsinItem {
groupName: string
recordSource: string
createdAt?: string
updatedAt?: string
}
export interface InvalidAsinPageResult {
@@ -74,6 +75,8 @@ export function toInvalidAsinItem(raw: unknown): InvalidAsinItem | null {
}
const createdAt = text(record.createdAt ?? record.created_at)
if (createdAt) item.createdAt = createdAt
const updatedAt = text(record.updatedAt ?? record.updated_at)
if (updatedAt) item.updatedAt = updatedAt
return item
}
@@ -0,0 +1,719 @@
<script setup lang="ts">
/** 日志管理页:桌面客户端 / 麦象采集机上报的日志浏览(仅超管)。
* 支持来源/日期/关键字筛选,尾部内容查看(自动滚到底、可向前加载更早内容)、
* 完整下载、删除;「采集配置」可调全量/精选模式(全局默认 + 终端覆盖)。 */
import { computed, nextTick, onMounted, ref } from 'vue'
import OldPagination from '@/components/OldPagination.vue'
import { ElMessage } from 'element-plus'
import { formatDateTime } from '@/utils/datetime'
import {
deleteDeviceLogFile,
deleteDeviceLogOverride,
deviceLogDownloadUrl,
fetchDeviceLogConfig,
fetchDeviceLogContent,
fetchDeviceLogDevices,
fetchDeviceLogFiles,
updateDeviceLogGlobalMode,
updateDeviceLogOverride,
type DeviceLogContent,
type DeviceLogDevice,
type DeviceLogFileRow,
type DeviceLogOverride,
} from '@/api/device-logs'
const SOURCE_OPTIONS = [
{ value: '', label: '全部来源' },
{ value: 'client', label: '桌面客户端' },
{ value: 'maixiang', label: '麦象采集机' },
]
function sourceLabel(source: string): string {
if (source === 'client') return '桌面客户端'
if (source === 'maixiang') return '麦象采集机'
return source || '—'
}
function formatBytes(bytes: number | null | undefined): string {
if (bytes == null || !Number.isFinite(bytes) || bytes <= 0) return '—'
if (bytes < 1024) return `${bytes} B`
const units = ['KB', 'MB', 'GB']
let value = bytes / 1024
let unitIndex = 0
while (value >= 1024 && unitIndex < units.length - 1) {
value /= 1024
unitIndex += 1
}
return `${value >= 100 ? value.toFixed(0) : value.toFixed(1)} ${units[unitIndex]}`
}
const loading = ref(false)
const rows = ref<DeviceLogFileRow[]>([])
const total = ref(0)
const page = ref(1)
const pageSize = ref(15)
const sourceFilter = ref('')
const keyword = ref('')
const startDate = ref('')
const endDate = ref('')
const retentionDays = ref(7)
const totalPages = computed(() => Math.max(1, Math.ceil(total.value / pageSize.value)))
async function load() {
loading.value = true
try {
const result = await fetchDeviceLogFiles({
source: sourceFilter.value || undefined,
keyword: keyword.value.trim() || undefined,
startDate: startDate.value || undefined,
endDate: endDate.value || undefined,
page: page.value,
pageSize: pageSize.value,
})
rows.value = result?.items || []
total.value = Number(result?.total || 0)
if (result?.retentionDays) retentionDays.value = result.retentionDays
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '日志列表加载失败')
} finally {
loading.value = false
}
}
function search() {
page.value = 1
load()
}
function reset() {
sourceFilter.value = ''
keyword.value = ''
startDate.value = ''
endDate.value = ''
page.value = 1
load()
}
function changePage(next: number) {
if (next < 1 || next > totalPages.value) return
page.value = next
load()
}
function changeSize(size: number) {
pageSize.value = size
page.value = 1
load()
}
// ---------------------------------------------------------------- 内容查看
const TAIL_STEP = 256 * 1024
const TAIL_MAX = 8 * 1024 * 1024
const viewerVisible = ref(false)
const viewerLoading = ref(false)
const viewerFile = ref<DeviceLogFileRow | null>(null)
const viewerContent = ref<DeviceLogContent | null>(null)
const viewerMaxBytes = ref(TAIL_STEP)
const viewerPre = ref<HTMLElement | null>(null)
async function openViewer(row: DeviceLogFileRow) {
viewerFile.value = row
viewerMaxBytes.value = TAIL_STEP
viewerContent.value = null
viewerVisible.value = true
await loadContent(true)
}
async function loadContent(scrollToBottom: boolean) {
if (!viewerFile.value) return
viewerLoading.value = true
try {
viewerContent.value = await fetchDeviceLogContent(viewerFile.value.id, viewerMaxBytes.value)
if (scrollToBottom) {
await nextTick()
if (viewerPre.value) viewerPre.value.scrollTop = viewerPre.value.scrollHeight
}
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '日志内容加载失败')
} finally {
viewerLoading.value = false
}
}
function loadMore() {
viewerMaxBytes.value = Math.min(viewerMaxBytes.value * 2, TAIL_MAX)
loadContent(false)
}
async function remove(row: DeviceLogFileRow) {
if (!window.confirm(`确认删除「${row.fileName}」(${row.deviceName || row.deviceId})的云端日志?删除后不可恢复。`)) return
try {
await deleteDeviceLogFile(row.id)
ElMessage.success('已删除')
load()
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '删除失败')
}
}
// ---------------------------------------------------------------- 采集配置
const configVisible = ref(false)
const configLoading = ref(false)
const globalMode = ref('full')
const overrides = ref<DeviceLogOverride[]>([])
const devices = ref<DeviceLogDevice[]>([])
const newOverrideKey = ref('')
const newOverrideMode = ref('selected')
const MODE_OPTIONS = [
{ value: 'full', label: '全量采集' },
{ value: 'selected', label: '精选采集' },
]
function modeLabel(mode: string): string {
return mode === 'selected' ? '精选' : '全量'
}
async function openConfig() {
configVisible.value = true
await loadConfig()
}
async function loadConfig() {
configLoading.value = true
try {
const data = await fetchDeviceLogConfig()
globalMode.value = data.globalMode || 'full'
overrides.value = data.overrides || []
devices.value = (await fetchDeviceLogDevices()) || []
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '采集配置加载失败')
} finally {
configLoading.value = false
}
}
async function saveGlobalMode(mode: string) {
if (mode === globalMode.value) return
try {
await updateDeviceLogGlobalMode(mode)
globalMode.value = mode
ElMessage.success(`全局采集模式已切换为「${modeLabel(mode)}`)
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '保存失败')
}
}
async function addOverride() {
if (!newOverrideKey.value) {
ElMessage.warning('请先选择终端')
return
}
const [source, deviceId] = newOverrideKey.value.split('|')
const device = devices.value.find((item) => item.source === source && item.deviceId === deviceId)
try {
await updateDeviceLogOverride(source, deviceId, device?.deviceName || null, newOverrideMode.value)
ElMessage.success('终端覆盖已保存')
newOverrideKey.value = ''
await loadConfig()
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '保存失败')
}
}
async function removeOverride(row: DeviceLogOverride) {
const who = row.deviceName || row.deviceId
if (!window.confirm(`确认删除终端「${who}」的采集覆盖(回落到全局默认)?`)) return
try {
await deleteDeviceLogOverride(row.id)
ElMessage.success('已删除')
await loadConfig()
} catch (error) {
ElMessage.error(error instanceof Error ? error.message : '删除失败')
}
}
onMounted(load)
</script>
<template>
<div class="device-logs-view">
<section class="panel-box">
<div class="logs-head">
<h3>日志文件列表</h3>
<div class="logs-head-tools">
<span class="retention-tip">云端仅保留 {{ retentionDays }} </span>
<button class="btn btn-ghost" type="button" @click="openConfig">采集配置</button>
<button class="btn" type="button" @click="load">刷新</button>
</div>
</div>
<div class="form-row logs-filter-row">
<div class="form-group" style="min-width: 150px">
<label>来源</label>
<select v-model="sourceFilter">
<option v-for="option in SOURCE_OPTIONS" :key="option.value" :value="option.value">{{ option.label }}</option>
</select>
</div>
<div class="form-group" style="min-width: 220px">
<label>设备 / 文件名</label>
<input v-model="keyword" type="text" placeholder="模糊搜索设备名、设备ID、文件名" @keyup.enter="search" />
</div>
<div class="form-group" style="min-width: 150px">
<label>日志日期</label>
<input v-model="startDate" type="date" />
</div>
<div class="form-group" style="min-width: 150px">
<label>日志日期</label>
<input v-model="endDate" type="date" />
</div>
<div class="form-group">
<label>&nbsp;</label>
<div class="filter-actions">
<button class="btn" type="button" @click="search">查询</button>
<button class="btn btn-ghost" type="button" @click="reset">重置</button>
</div>
</div>
</div>
<div class="logs-table-scroll">
<table>
<thead>
<tr>
<th style="width: 110px">来源</th>
<th style="width: 230px">设备</th>
<th style="width: 130px">用户</th>
<th style="width: 230px">文件名</th>
<th style="width: 110px">日志日期</th>
<th style="width: 100px">已收大小</th>
<th style="width: 80px">片段数</th>
<th style="width: 170px">最后更新</th>
<th style="width: 190px">操作</th>
</tr>
</thead>
<tbody>
<template v-if="rows.length">
<tr v-for="row in rows" :key="row.id">
<td>
<span class="source-pill" :class="row.source === 'maixiang' ? 'is-maixiang' : 'is-client'">
{{ sourceLabel(row.source) }}
</span>
</td>
<td>
<span class="device-name" :title="row.deviceId">{{ row.deviceName || '—' }}</span>
<span class="device-id" :title="row.deviceId">{{ row.deviceId }}</span>
</td>
<td>
<span v-if="row.username" class="user-name">{{ row.username }}</span>
<span v-else-if="row.uid" class="user-name">UID {{ row.uid }}</span>
<span v-else class="dim"></span>
</td>
<td>
<span class="file-name" :title="row.fileName">{{ row.fileName }}</span>
</td>
<td>{{ row.logDate }}</td>
<td>{{ formatBytes(row.uploadedBytes) }}</td>
<td>{{ row.partCount ?? 0 }}</td>
<td>{{ row.lastUploadAt ? formatDateTime(row.lastUploadAt) : '—' }}</td>
<td class="ops-cell">
<button class="btn btn-sm" type="button" @click="openViewer(row)">查看</button>
<a class="btn btn-sm btn-ghost dl-link" :href="deviceLogDownloadUrl(row.id)" download>下载</a>
<button class="btn btn-sm btn-danger" type="button" @click="remove(row)">删除</button>
</td>
</tr>
</template>
<tr v-else-if="loading">
<td colspan="9" class="empty-tip">加载中...</td>
</tr>
<tr v-else>
<td colspan="9" class="empty-tip">
{{ keyword || sourceFilter || startDate || endDate ? '暂无匹配日志' : '暂无日志上报(客户端/采集机上报后自动出现在这里)' }}
</td>
</tr>
</tbody>
</table>
</div>
<OldPagination :total="total" :page="page" :page-size="pageSize" @change="changePage" @size-change="changeSize" />
</section>
<el-dialog v-model="viewerVisible" :title="viewerFile ? `${viewerFile.fileName}${viewerFile.deviceName || viewerFile.deviceId}` : '日志内容'" width="900px" top="5vh">
<div class="viewer-toolbar">
<span class="viewer-meta">
云端已收 {{ formatBytes(viewerContent?.totalBytes ?? viewerFile?.uploadedBytes) }}
<template v-if="viewerContent"> · 当前展示 {{ formatBytes(viewerContent.shownBytes) }}</template>
<template v-if="viewerContent?.truncated"> · 更早内容未加载</template>
</span>
<span class="viewer-actions">
<button class="btn btn-sm btn-ghost" type="button" :disabled="viewerLoading" @click="loadContent(true)">刷新</button>
<button
class="btn btn-sm btn-ghost"
type="button"
:disabled="viewerLoading || !viewerContent?.truncated || viewerMaxBytes >= TAIL_MAX"
@click="loadMore"
>加载更早</button>
<a v-if="viewerFile" class="btn btn-sm btn-ghost dl-link" :href="deviceLogDownloadUrl(viewerFile.id)" download>下载完整日志</a>
</span>
</div>
<pre ref="viewerPre" class="log-pre">{{ viewerLoading && !viewerContent ? '加载中...' : (viewerContent?.content || '(暂无内容)') }}</pre>
</el-dialog>
<el-dialog v-model="configVisible" title="采集配置" width="720px">
<div class="config-block">
<h4>全局默认模式</h4>
<p class="config-desc">对未单独配置的终端生效全量=上传日志目录内全部文件精选=排除低价值大日志客户端排除 pywebview麦象排除 kk-browser / 控制台 / 测试日志终端在下一次上报周期 1 分钟内跟随新配置</p>
<div class="mode-switch">
<button
v-for="option in MODE_OPTIONS"
:key="option.value"
class="mode-btn"
:class="{ 'is-active': globalMode === option.value }"
type="button"
@click="saveGlobalMode(option.value)"
>{{ option.label }}</button>
</div>
</div>
<div class="config-block">
<h4>终端覆盖</h4>
<div class="override-add">
<select v-model="newOverrideKey" class="override-select">
<option value="">选择终端最近上报的设备</option>
<option v-for="device in devices" :key="`${device.source}|${device.deviceId}`" :value="`${device.source}|${device.deviceId}`">
{{ sourceLabel(device.source) }} · {{ device.deviceName || device.deviceId }}{{ device.deviceId }}
</option>
</select>
<select v-model="newOverrideMode">
<option v-for="option in MODE_OPTIONS" :key="option.value" :value="option.value">{{ option.label }}</option>
</select>
<button class="btn btn-sm" type="button" @click="addOverride">添加/更新覆盖</button>
</div>
<table class="override-table">
<thead>
<tr>
<th style="width: 100px">来源</th>
<th>设备</th>
<th style="width: 80px">模式</th>
<th style="width: 160px">更新时间</th>
<th style="width: 90px">操作</th>
</tr>
</thead>
<tbody>
<tr v-for="row in overrides" :key="row.id">
<td>{{ sourceLabel(row.source) }}</td>
<td>
<span class="device-name">{{ row.deviceName || row.deviceId }}</span>
<span class="device-id">{{ row.deviceId }}</span>
</td>
<td>{{ modeLabel(row.mode) }}</td>
<td>{{ row.updatedAt ? formatDateTime(row.updatedAt) : '—' }}</td>
<td class="ops-cell">
<button class="btn btn-sm btn-danger" type="button" @click="removeOverride(row)">删除</button>
</td>
</tr>
<tr v-if="!overrides.length">
<td colspan="5" class="empty-tip">{{ configLoading ? '加载中...' : '暂无终端覆盖(全部跟随全局默认)' }}</td>
</tr>
</tbody>
</table>
</div>
<template #footer>
<el-button @click="configVisible = false">关闭</el-button>
</template>
</el-dialog>
</div>
</template>
<style scoped>
/* 沿用「记录与版本」系列的旧后台面板视觉语言。 */
.device-logs-view {
font-family: inherit;
color: #24384d;
}
.panel-box {
width: 100%;
min-width: 0;
padding: 20px 22px 24px;
border: 1px solid #d8e3ee;
border-radius: 14px;
background: linear-gradient(145deg, #ffffff, #f9fbfd);
box-shadow: 0 1px 2px rgba(39, 67, 94, 0.04), 0 14px 30px -24px rgba(39, 67, 94, 0.28);
}
h3 {
margin: 0;
font-size: 15px;
font-weight: 650;
color: #24384d;
letter-spacing: 0.2px;
}
.logs-head {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
margin-bottom: 16px;
}
.logs-head-tools {
display: flex;
align-items: center;
gap: 10px;
}
.retention-tip {
color: #8598ab;
font-size: 12.5px;
}
.logs-filter-row {
margin: 0 0 16px;
}
.form-row {
display: flex;
flex-wrap: wrap;
align-items: flex-end;
gap: 14px 18px;
}
.form-group {
display: flex;
flex-direction: column;
gap: 7px;
}
.form-group label {
color: #5b6f83;
font-size: 12.5px;
font-weight: 600;
}
.form-group input,
.form-group select {
min-width: 0;
min-height: 42px;
padding: 8px 12px;
border: 1px solid #cbd9e6;
border-radius: 9px;
background: #f8fbfd;
color: #24384d;
font-size: 13.5px;
font-family: inherit;
color-scheme: light;
outline: none;
}
.form-group input:focus,
.form-group select:focus {
background: #ffffff;
border-color: #5f85ad;
box-shadow: 0 0 0 3px rgba(95, 133, 173, 0.16);
}
.filter-actions {
display: flex;
gap: 10px;
}
.btn {
display: inline-flex;
align-items: center;
justify-content: center;
gap: 8px;
min-height: 42px;
padding: 9px 18px;
border: 1px solid #4f78a5;
border-radius: 9px;
background: linear-gradient(135deg, #5f85ad, #4f78a5);
color: #ffffff;
font-family: inherit;
font-size: 13.5px;
cursor: pointer;
box-shadow: 0 8px 18px -14px rgba(79, 120, 165, 0.72);
text-decoration: none;
}
.btn:hover:not(:disabled) {
background: linear-gradient(135deg, #7094ba, #5d83ac);
}
.btn:disabled {
cursor: not-allowed;
opacity: 0.55;
}
.btn-ghost {
background: #ffffff;
border-color: #c7d7e5;
color: #4f78a5;
box-shadow: none;
}
.btn-ghost:hover:not(:disabled) {
background: #edf5fb;
border-color: #95b1cb;
color: #2f5d8b;
}
.btn-danger {
background: linear-gradient(135deg, #c06d77, #b35f6a);
border-color: #b35f6a;
}
.btn-danger:hover:not(:disabled) {
background: linear-gradient(135deg, #cb7c84, #b96570);
}
.btn-sm {
min-height: 32px;
padding: 4px 12px;
font-size: 12.5px;
border-radius: 7px;
}
.logs-table-scroll {
width: 100%;
overflow-x: auto;
}
table {
width: 100%;
border-collapse: collapse;
font-size: 13px;
}
th {
padding: 10px 12px;
text-align: left;
color: #5b6f83;
font-weight: 650;
font-size: 12.5px;
border-bottom: 1px solid #dbe6f0;
white-space: nowrap;
}
td {
padding: 10px 12px;
border-bottom: 1px solid #eaf1f7;
vertical-align: top;
}
.source-pill {
display: inline-block;
padding: 2px 10px;
border-radius: 999px;
font-size: 12px;
font-weight: 600;
}
.source-pill.is-client {
background: #e8f1fb;
color: #37618f;
}
.source-pill.is-maixiang {
background: #eef7ec;
color: #3f7a42;
}
.device-name {
display: block;
font-weight: 600;
word-break: break-all;
}
.device-id {
display: block;
color: #8598ab;
font-size: 11.5px;
word-break: break-all;
}
.user-name {
font-weight: 600;
}
.file-name {
display: block;
word-break: break-all;
}
.dim {
color: #9db0c2;
}
.ops-cell {
display: flex;
gap: 8px;
flex-wrap: wrap;
border-bottom: none;
}
.empty-tip {
padding: 26px 0;
text-align: center;
color: #8598ab;
}
.dl-link {
text-decoration: none;
}
.viewer-toolbar {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
margin-bottom: 10px;
}
.viewer-meta {
color: #5b6f83;
font-size: 12.5px;
}
.viewer-actions {
display: flex;
gap: 8px;
}
.log-pre {
max-height: 62vh;
margin: 0;
padding: 12px 14px;
overflow: auto;
border: 1px solid #d8e3ee;
border-radius: 10px;
background: #0f1c29;
color: #d7e4f1;
font-family: Consolas, 'Courier New', monospace;
font-size: 12.5px;
line-height: 1.55;
white-space: pre;
}
.config-block {
margin-bottom: 18px;
}
.config-block h4 {
margin: 0 0 6px;
font-size: 13.5px;
color: #24384d;
}
.config-desc {
margin: 0 0 10px;
color: #66798d;
font-size: 12.5px;
line-height: 1.6;
}
.mode-switch {
display: flex;
gap: 10px;
}
.mode-btn {
min-height: 38px;
padding: 8px 20px;
border: 1px solid #c7d7e5;
border-radius: 9px;
background: #ffffff;
color: #4f78a5;
font-size: 13.5px;
font-family: inherit;
cursor: pointer;
}
.mode-btn.is-active {
border-color: #4f78a5;
background: linear-gradient(135deg, #5f85ad, #4f78a5);
color: #ffffff;
font-weight: 600;
}
.override-add {
display: flex;
gap: 10px;
margin-bottom: 12px;
}
.override-add select {
min-height: 38px;
padding: 6px 10px;
border: 1px solid #cbd9e6;
border-radius: 9px;
background: #f8fbfd;
color: #24384d;
font-size: 13px;
font-family: inherit;
color-scheme: light;
}
.override-select {
flex: 1;
min-width: 0;
}
.override-table td {
vertical-align: middle;
}
</style>
+1
View File
@@ -25,6 +25,7 @@ export const adminPages: AdminPageDef[] = [
{ path: 'records/history', menuKey: 'admin_history', title: '生成记录', load: () => import('@/pages/records/RecordsHistoryPage.vue') },
{ path: 'records/software-version', menuKey: 'admin_version', title: '软件版本管理', load: () => import('@/pages/records/RecordsSoftwareVersionPage.vue') },
{ path: 'records/tutorial', menuKey: 'admin_tutorial', title: '教程管理', load: () => import('@/pages/records/RecordsTutorialPage.vue') },
{ path: 'records/device-logs', menuKey: 'admin_device_logs', title: '日志管理', load: () => import('@/pages/records/DeviceLogsPage.vue') },
{ path: 'records/digital-human-version', menuKey: 'digital_human_version', title: '数字人版本管理', load: () => import('@/pages/records/RecordsDigitalHumanVersionPage.vue') },
{ path: 'records/image-video-tasks', menuKey: 'admin_image_video_tasks', title: '视频任务记录', load: () => import('@/pages/tasks/ImageVideoTasksPage.vue') },
{ path: 'shop-center/data-tasks', menuKey: 'admin_shop_data_crawl_tasks', title: '店铺数据记录', load: () => import('@/pages/tasks/ShopDataTasksPage.vue') },
+1
View File
@@ -12,6 +12,7 @@ export interface AdminUser {
createdById?: number | null
creatorUsername?: string
createdAt?: string
updatedAt?: string
pinyinAbbr?: string
}
@@ -82,7 +82,9 @@ test('align_query_asin_page_layout_wiring', () => {
assert.match(page, /新增 ASIN/, '顶栏补新增 ASIN 按钮')
assert.match(page, /:rowspan="row\.rowspan"/, '表格用原生 rowspan 合并(像素复刻旧版)')
assert.match(page, /CopyText/, 'ASIN 单元格点击复制')
assert.doesNotMatch(page, /更新时间/, '去掉更新时间列(参考无此列)')
// 需求变更:后台管理列表统一展示「创建时间 + 更新时间」,覆盖早期「不加更新时间」的像素对齐。
assert.match(page, /创建时间/, '表格含创建时间列')
assert.match(page, /更新时间/, '表格含更新时间列')
assert.match(page, /queryAsinDisplayRows/, '行展开走纯模型')
assert.match(page, /请完整填写分组、店铺名、国家和 ASIN/, '新增弹窗校验文案对齐')
assert.match(page, /请先选择分组/, '店铺下拉未选分组时占位对齐')
@@ -82,7 +82,9 @@ test('align_skip_price_page_layout_wiring', () => {
assert.match(page, /新增 ASIN/, '顶栏补新增 ASIN 按钮')
assert.match(page, /:rowspan="row\.rowspan"/, '表格用原生 rowspan 合并(像素复刻旧版)')
assert.match(page, /CopyText/, 'ASIN 单元格点击复制')
assert.doesNotMatch(page, /更新时间/, '去掉更新时间列(参考无此列)')
// 需求变更:后台管理列表统一展示「创建时间 + 更新时间」,覆盖早期「不加更新时间」的像素对齐。
assert.match(page, /创建时间/, '表格含创建时间列')
assert.match(page, /更新时间/, '表格含更新时间列')
assert.match(page, /skipPriceDisplayRows/, '行展开走纯模型')
assert.match(page, /请完整填写分组、店铺名、国家和 ASIN/, '新增弹窗校验文案对齐')
assert.match(page, /最低价格式不正确/, '最低价格式校验对齐')
@@ -0,0 +1,86 @@
import test from 'node:test'
import assert from 'node:assert/strict'
import { readSource } from './helpers.ts'
import {
buildMenuOptionTree,
compactDirectGrantIds,
parseMenuOptionList,
parsePermissionMenuItem,
} from '../src/pages/account/user-menu-auth.ts'
// 2026-09-16 线上事故:非超管(普通管理员)在授权树里勾到自己无权授予的菜单后,
// 后端 ensureGrantable 抛 403 并回滚整笔事务——创建用户与保存权限双双失败,
// 前端只显示「普通管理员只能分配自己已有的菜单权限」。
// 修复:菜单列表按操作者标记 grantable,前端把不可授予的节点置灰不可勾。
//
// 注意:这里是「置灰」而非「隐藏」。授权保存是整树替换,隐藏会让超管早先授予、
// 而操作者自己没有的菜单在提交时被当作取消勾选删掉(与 09-13「权限自己没掉」同类)。
test('align_user_menu_grantable_false_maps_to_disabled_node', () => {
const locked = parsePermissionMenuItem(
{ id: 5, name: '查询ASIN', parent_id: null, sort_order: 1, grantable: false },
'admin',
)
assert.equal(locked?.disabled, true, 'grantable=false → 节点置灰')
const allowed = parsePermissionMenuItem(
{ id: 6, name: '店铺管理', parent_id: null, sort_order: 2, grantable: true },
'admin',
)
assert.equal(allowed?.disabled, false, 'grantable=true → 可勾选')
// 菜单管理页等未标记 grantable 的接口必须保持旧行为(全部可勾选)
const unmarked = parsePermissionMenuItem({ id: 7, name: '菜单权限配置', parent_id: null, sort_order: 3 }, 'admin')
assert.equal(unmarked?.disabled, false, '缺省 grantable 视为可授予')
})
test('align_user_menu_grantable_survives_tree_build', () => {
const nodes = parseMenuOptionList(
[
{ id: 98, name: '账号与权限', parent_id: null, sort_order: 1, grantable: false },
{ id: 7, name: '用户管理', parent_id: 98, sort_order: 1, grantable: false },
{ id: 100, name: '店铺管理', parent_id: null, sort_order: 2, grantable: true },
],
'admin',
)
const tree = buildMenuOptionTree(nodes)
const account = tree.find((node) => node.id === 98)
assert.equal(account?.disabled, true, '分组节点置灰')
assert.equal(account?.children?.[0]?.disabled, true, '子节点置灰随树保留')
assert.equal(tree.find((node) => node.id === 100)?.disabled, false, '可授予节点不受影响')
})
test('align_user_menu_grantable_disabled_node_still_compactable', () => {
// 已持有但无权授予的节点会保持勾选并原样提交,压缩逻辑不能因 disabled 漏掉它
const tree = buildMenuOptionTree(
parseMenuOptionList(
[
{ id: 98, name: '账号与权限', parent_id: null, sort_order: 1, grantable: false },
{ id: 7, name: '用户管理', parent_id: 98, sort_order: 1, grantable: false },
],
'admin',
),
)
assert.deepEqual(compactDirectGrantIds([98, 7], tree), [98], '父级已勾选时仍压缩掉后代')
})
test('align_user_menu_grantable_wired_end_to_end', () => {
const tree = readSource('src/pages/account/UserMenuAuthTree.vue')
assert.match(tree, /disabled: 'disabled'/, 'el-tree 按 disabled 键置灰节点')
const vo = readSource(
'../backend-java/src/main/java/com/nanri/aiimage/modules/permission/model/vo/PermissionMenuItemVo.java',
)
assert.match(vo, /private Boolean grantable;/, 'VO 暴露 grantable')
const controller = readSource(
'../backend-java/src/main/java/com/nanri/aiimage/modules/permission/controller/PermissionMenuController.java',
)
assert.match(controller, /permissionMenuService\.list\(requireAdmin\(request\), menuType\)/, '列表接口传入操作者')
const service = readSource(
'../backend-java/src/main/java/com/nanri/aiimage/modules/permission/service/PermissionMenuService.java',
)
assert.match(service, /resolveGrantableMenuIds/, '按操作者计算可授予集')
assert.match(service, /ensureGrantable\(operator, grantIds, userId\)/, '保存校验传入目标用户以放行既有授权')
})
@@ -25,4 +25,12 @@ public final class BusinessCodes {
/** 任务归属其它实例,需转发。 */
public static final int TASK_OWNER_FORWARD = 40903;
/**
* 提交结果的目标任务已不存在(通常是被删除)。
* 语义:本次提交无意义,响应 success=false 且带该码,调用方应放弃而不是反复重试。
* 与 {@link #TASK_ALREADY_FINISHED} 的区别:那个还留了任务记录(可幂等忽略),
* 这个任务已经没了——如实报错,否则任务被误删时结果会被静默吞掉。
*/
public static final int TASK_NOT_FOUND = 40401;
}
@@ -1,18 +1,21 @@
package com.nanri.aiimage.common.exception;
import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.common.security.DeviceSessionPolicy;
import com.nanri.aiimage.common.service.TaskOwnerForwardService;
import com.nanri.aiimage.config.TaskOperationLockConfig;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.validation.ConstraintViolationException;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.MethodArgumentNotValidException;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.context.request.async.AsyncRequestNotUsableException;
import org.springframework.web.server.ResponseStatusException;
import org.springframework.web.servlet.resource.NoResourceFoundException;
import java.io.IOException;
@@ -52,22 +55,40 @@ public class GlobalExceptionHandler {
? ApiResponse.fail(forwardEx.getMessage())
: ApiResponse.fail(forwardEx.getCode(), forwardEx.getMessage());
} catch (Exception forwardEx) {
// 转发失败是**瞬时基础设施故障**(归属实例正在滚动重启),不是业务结论,
// 更不能表达成「任务不存活」。原先返回 ApiResponse.fail(40903) —— HTTP 200
// 加 data:null,而客户端那句 bool((resp.json().get("data") or {}).get("alive"))
// 会把「拿不到数据」折叠成 alive=false,于是客户端把**健康的长任务主动停掉**:
// 2026-09-18 任务 28616 就是这么死的(归属节点 server-110 重启窗口内,心跳经
// nginx 落到 server-121,转发 3 次 Connection refused 后返回空 data)。
// 改为 503 + 空 body:新客户端按状态码判为「未知」继续跑;老客户端因 body 不是
// JSON、resp.json() 抛异常,同样落到「未知」。顺带让这类故障在 HTTP 指标里可见
// (原先记成 200,监控完全看不到滚动重启期间丢了多少心跳)。
log.warn("[instance-routing] forward failed taskId={} operation={} owner={} current={} msg={}",
ex.getTaskId(), ex.getOperation(), ex.getOwnerInstanceId(), ex.getCurrentInstanceId(),
forwardEx.getMessage(), forwardEx);
return ApiResponse.fail(40903, "任务归属实例转发失败: " + forwardEx.getMessage());
return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE).build();
}
}
@ExceptionHandler(BusinessException.class)
public ApiResponse<Void> handleBusinessException(BusinessException ex) {
public ApiResponse<Void> handleBusinessException(BusinessException ex, HttpServletRequest request) {
// 业务异常此前完全不记日志:2026-09-16 线上「保存权限/创建用户」双双失败时,
// 服务端只留 RequestTraceFilter 的 200 一行,根因只能靠反推响应体字节数才定位到。
// 401/4011(未登录、被顶下线)属于轮询类接口的常态噪声,降为 debug 以免淹没真实业务错。
if (isRoutineAuthNoise(ex.getCode())) {
log.debug("[business] {} {} code={} message={}", request.getMethod(), request.getRequestURI(),
ex.getCode(), ex.getMessage());
} else {
log.warn("[business] {} {} code={} message={}", request.getMethod(), request.getRequestURI(),
ex.getCode(), ex.getMessage());
}
if (Integer.valueOf(BusinessCodes.TASK_ALREADY_FINISHED).equals(ex.getCode())) {
// 幂等忽略:任务已结束时的重复提交无副作用,按成功返回,避免客户端反复重试
return ApiResponse.success("任务已结束,忽略重复提交", null);
}
if (Integer.valueOf(BusinessCodes.TASK_BUSY).equals(ex.getCode())) {
// 锁竞争:必须如实返回失败 + 可重试码,否则 worker 会把「未落库」当成功而停止重试
log.warn("[business] 任务忙,调用方应稍后重试: {}", ex.getMessage());
return ApiResponse.fail(BusinessCodes.TASK_BUSY, ex.getMessage());
}
return ex.getCode() == null
@@ -75,9 +96,23 @@ public class GlobalExceptionHandler {
: ApiResponse.fail(ex.getCode(), ex.getMessage());
}
@ExceptionHandler(NoResourceFoundException.class)
public ResponseEntity<ApiResponse<Void>> handleNoResourceFoundException(NoResourceFoundException ex) {
// 静态资源 404。绝大部分是外部扫描器在探测 /.env、/credentials、aliyun.json、oss.json
// 这类云凭据文件(线上单节点一天 580 条)。此前落到 handleException 里,既刷 ERROR 堆栈,
// 又把探测响应伪装成 HTTP 200;这里降为 debug 并如实返回 404。
log.debug("static resource not found: {}", ex.getMessage());
return ResponseEntity.status(HttpStatus.NOT_FOUND).body(ApiResponse.fail("资源不存在"));
}
/** 未登录 / 登录态失效 / 被其他设备顶下线:按 401 语义的常态噪声,不占 WARN。 */
private boolean isRoutineAuthNoise(Integer code) {
return Integer.valueOf(401).equals(code)
|| Integer.valueOf(DeviceSessionPolicy.CODE_KICKED).equals(code);
}
@ExceptionHandler(MethodArgumentNotValidException.class)
public ApiResponse<Void> handleMethodArgumentNotValidException(MethodArgumentNotValidException ex) {
String message = ex.getBindingResult().getFieldError() != null
public ApiResponse<Void> handleMethodArgumentNotValidException(MethodArgumentNotValidException ex) { String message = ex.getBindingResult().getFieldError() != null
? ex.getBindingResult().getFieldError().getDefaultMessage()
: "参数校验失败";
return ApiResponse.fail(message);
@@ -1,5 +1,6 @@
package com.nanri.aiimage.common.model.entity;
import com.baomidou.mybatisplus.annotation.FieldStrategy;
import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableField;
import com.baomidou.mybatisplus.annotation.TableId;
@@ -24,5 +25,13 @@ public class AdminUserEntity {
private Long createdById;
@TableField("created_at")
private LocalDateTime createdAt;
/**
* 更新时间(V131 新增):由数据库维护(DEFAULT CURRENT_TIMESTAMP + ON UPDATE CURRENT_TIMESTAMP)。
*
* <p>禁止应用显式写:MySQL 在 UPDATE 语句显式给该列赋值时不会触发自动更新,
* 而本表写回多为 selectById → 改字段 → updateById(实体带着旧值),一旦写回就会冻结更新时间。
*/
@TableField(value = "updated_at", insertStrategy = FieldStrategy.NEVER, updateStrategy = FieldStrategy.NEVER)
private LocalDateTime updatedAt;
private String machine;
}
@@ -9,6 +9,7 @@ import io.jsonwebtoken.Claims;
import jakarta.servlet.http.Cookie;
import jakarta.servlet.http.HttpServletRequest;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.http.HttpHeaders;
import org.springframework.stereotype.Component;
@@ -19,6 +20,7 @@ import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
@Slf4j
@Component
@RequiredArgsConstructor
public class AdminAuthSupport {
@@ -62,8 +64,30 @@ public class AdminAuthSupport {
return user;
}
/** 当前用户必须是管理员或超级管理员,否则抛 403。 */
public AdminUserEntity requireAdmin(HttpServletRequest request) {
/**
* 解析当前请求 JWT 中**签名的**设备标识(deviceId claim);识别不出时返回空串。
*
* <p>无 token、token 过期/非法、内部令牌通道调用一律返回空串——调用方必须把空串
* 当作"来源不明"做保守判定,绝不据此放宽任何限制。只认签名 claim,不接受
* X-Device-Id 请求头(头由客户端可控,见 {@link DeviceSessionPolicy} 类注释)。</p>
*
* <p>本方法只做识别、不做鉴权,因此解析失败不抛异常,仅记日志后返回空串,
* 避免把匿名/内部调用直接升级成 401。</p>
*/
public String currentDeviceId(HttpServletRequest request) {
String token = resolveToken(request);
if (token == null || token.isBlank()) {
return "";
}
try {
return DeviceSessionPolicy.claimDeviceId(jwtService.parse(token));
} catch (Exception ex) {
log.warn("[auth] 解析 token 取设备标识失败,按来源不明处理: {}", ex.getMessage());
return "";
}
}
/** 当前用户必须是管理员或超级管理员,否则抛 403。 */ public AdminUserEntity requireAdmin(HttpServletRequest request) {
AdminUserEntity user = requireUser(request);
String role = currentRole(user);
if (role == null) {
@@ -13,10 +13,12 @@ import org.springframework.http.ResponseEntity;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.stereotype.Service;
import org.springframework.util.StreamUtils;
import org.springframework.web.client.ResourceAccessException;
import org.springframework.web.client.RestClient;
import org.springframework.web.util.ContentCachingRequestWrapper;
import java.io.IOException;
import java.net.ConnectException;
import java.util.Enumeration;
import java.util.List;
import java.util.Set;
@@ -28,6 +30,11 @@ public class TaskOwnerForwardService {
public static final String FORWARDED_HEADER = "X-AIIMAGE-Owner-Forwarded";
/** 连接类失败的重试次数(含首次)。对端滚动重启时通常几秒内即可恢复。 */
private static final int CONNECT_RETRY_TIMES = 3;
/** 第 n 次重试前的退避:1s、2s(总等待不超过 3s,不长时间占用请求线程)。 */
private static final long CONNECT_RETRY_BACKOFF_MILLIS = 1000L;
private static final Set<String> HOP_BY_HOP_HEADERS = Set.of(
"connection",
"keep-alive",
@@ -58,12 +65,60 @@ public class TaskOwnerForwardService {
HttpHeaders headers = copyForwardHeaders(request, ex.getCurrentInstanceId());
log.info("[instance-routing] forward {} taskId={} operation={} owner={} current={} url={}",
method, ex.getTaskId(), ex.getOperation(), ex.getOwnerInstanceId(), ex.getCurrentInstanceId(), url);
return forwardWithConnectRetry(method, url, headers, body, ex);
}
/**
* 转发带连接级重试。
*
* <p>对端实例在部署窗口(原地换 JAR + 两节点滚动重启)内会有几秒的 Connection refused。
* 连接都没建立起来说明请求没到达对端,此时重放是安全的;而读超时不重试——对端可能
* 已经在处理,盲目重放会造成重复提交。线上由此丢过用户提交的结果。
*/
private ResponseEntity<byte[]> forwardWithConnectRetry(HttpMethod method, String url,
HttpHeaders headers, byte[] body,
TaskOwnerMismatchException ex) {
RuntimeException lastError = null;
for (int attempt = 1; attempt <= CONNECT_RETRY_TIMES; attempt++) {
try {
return restClient().method(method)
.uri(url)
.headers(target -> target.addAll(headers))
.body(body)
.retrieve()
.toEntity(byte[].class);
} catch (ResourceAccessException accessError) {
if (!isConnectFailure(accessError)) {
throw accessError;
}
lastError = accessError;
log.warn("[instance-routing] 转发连接失败,第 {}/{} 次 url={} taskId={} 原因={}",
attempt, CONNECT_RETRY_TIMES, url, ex.getTaskId(), accessError.getMessage());
if (attempt < CONNECT_RETRY_TIMES) {
sleepQuietly(CONNECT_RETRY_BACKOFF_MILLIS * attempt);
}
}
}
throw lastError;
}
private static boolean isConnectFailure(Throwable error) {
Throwable cursor = error;
while (cursor != null) {
if (cursor instanceof ConnectException) {
return true;
}
cursor = cursor.getCause();
}
return false;
}
private static void sleepQuietly(long millis) {
try {
Thread.sleep(millis);
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
}
}
private String resolveUrl(TaskOwnerMismatchException ex, String path) {
@@ -4,6 +4,7 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.common.security.AdminAuthSupport;
import com.nanri.aiimage.common.security.DeviceSessionPolicy;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
@@ -153,7 +154,14 @@ public class AdminApiGuardFilter extends OncePerRequestFilter {
ApiResponse<Void> body = ex.getCode() == null
? ApiResponse.fail(ex.getMessage())
: ApiResponse.fail(ex.getCode(), ex.getMessage());
// 401(登录已过期)与被顶下线是前端定时轮询(/api/notifications/summary、/api/user-secrets 等)
// 的常态:线上单节点一天近 3000 条,会把真实业务错误淹没。与 GlobalExceptionHandler
// 的 isRoutineAuthNoise 同一口径降为 debug。
if (isRoutineAuthNoise(ex.getCode())) {
log.debug("[admin-guard] {} {} rejected: {}", request.getMethod(), request.getRequestURI(), ex.getMessage());
} else {
log.warn("[admin-guard] {} {} rejected: {}", request.getMethod(), request.getRequestURI(), ex.getMessage());
}
response.setStatus(HttpServletResponse.SC_OK);
response.setContentType("application/json;charset=UTF-8");
response.getWriter().write(objectMapper.writeValueAsString(body));
@@ -168,6 +176,12 @@ public class AdminApiGuardFilter extends OncePerRequestFilter {
chain.doFilter(request, response);
}
/** 登录态过期 / 被其他设备顶下线:前端轮询的常态噪声,不占 WARN。 */
private static boolean isRoutineAuthNoise(Integer code) {
return Integer.valueOf(401).equals(code)
|| Integer.valueOf(DeviceSessionPolicy.CODE_KICKED).equals(code);
}
/** 命中受保护前缀(/api/admin、/debug、自助接口、用户态工具前缀及其子路径)才进入鉴权,其余请求直接放行。 */
private boolean isGuarded(String uri) {
if (matchesPrefix(uri, ADMIN_API_PREFIX) || matchesPrefix(uri, DEBUG_PREFIX)) {
@@ -23,6 +23,14 @@ public class BrandCheckProperties {
* 10 次尝试合计约 45s,既能覆盖限流窗口又不会让单个品牌长时间占住查询线程。
*/
private int retryMaxIntervalMillis = 10000;
/**
* 一次批量检查的总耗时上限(毫秒):整批用尽后不再重试,剩余品牌按「查询失败」收尾。
* 上限的意义不是省时间,而是给「分片回传」这类同步调用方一个时延上界——上游卡死时
* 单品牌 10 次重试曾把一次回传拖到 103.5 秒(taskId 28599),客户端重试预算耗尽后
* 中止了整个采集。首轮请求始终执行,故上游只是略慢时不会误降级。
* 设为 0 或负数表示不限制。
*/
private int totalTimeoutMillis = 90000;
private int connectTimeoutMillis = 10000;
private int readTimeoutMillis = 60000;
}
@@ -0,0 +1,38 @@
package com.nanri.aiimage.config;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
/**
* 设备日志对象存储配置:指向主机B 独立部署的 MinIO 实例(非业务 MinIO)。
*
* <p>日志体积大、只保留 7 天,独立实例便于单独设生命周期规则与容量管理,
* 不挤占业务桶(nanri-ai-images 等)。endpoint 为空时上报接口直接失败,
* 不做静默回退(避免日志悄悄落到其他存储上而无人知情)。
*/
@Data
@ConfigurationProperties(prefix = "aiimage.device-log-oss")
public class DeviceLogOssProperties {
private String endpoint;
private String accessKeyId;
private String accessKeySecret;
private String bucket;
/**
* 日志保留天数:查询侧按此过滤(早于今天的 N-1 天不展示),
* 对象过期由 MinIO 桶生命周期规则在部署时同步设置(两侧口径保持一致)。
*/
private Integer retentionDays;
public boolean configured() {
return endpoint != null && !endpoint.isBlank()
&& accessKeyId != null && !accessKeyId.isBlank()
&& accessKeySecret != null && !accessKeySecret.isBlank()
&& bucket != null && !bucket.isBlank();
}
public int retentionDaysOrDefault() {
return retentionDays == null || retentionDays < 1 ? 7 : retentionDays;
}
}
@@ -67,4 +67,12 @@ public class NotificationProperties {
/** 已读通知保留天数(超期自动清理),默认 90 天。 */
private int readRetentionDays = 90;
/**
* 未读通知保留天数,默认 180 天(比已读长一倍)。
*
* <p>未读通知此前永不清理:不看铃铛的用户会无限累积。保留期给得更宽,
* 是因为未读意味着"用户可能还没看到",但也不能永远留着。
*/
private int unreadRetentionDays = 180;
}
@@ -4,6 +4,6 @@ import org.springframework.boot.context.properties.EnableConfigurationProperties
import org.springframework.context.annotation.Configuration;
@Configuration
@EnableConfigurationProperties({OssProperties.class, TransientStorageProperties.class, StorageProperties.class, BrandProgressProperties.class, DeleteBrandProgressProperties.class, BrandCheckProperties.class, ZiniaoProperties.class, ModuleCleanupProperties.class, TaskPressureProperties.class, TaskImageCacheCleanupProperties.class, AppearancePatentProperties.class, SimilarAsinProperties.class, ImageVideoProperties.class, InstanceRoutingProperties.class, CapacityPlanProperties.class, UserSecretProperties.class, NotificationProperties.class})
@EnableConfigurationProperties({OssProperties.class, TransientStorageProperties.class, StorageProperties.class, BrandProgressProperties.class, DeleteBrandProgressProperties.class, BrandCheckProperties.class, ZiniaoProperties.class, ModuleCleanupProperties.class, TaskPressureProperties.class, TaskImageCacheCleanupProperties.class, AppearancePatentProperties.class, SimilarAsinProperties.class, ImageVideoProperties.class, InstanceRoutingProperties.class, CapacityPlanProperties.class, UserSecretProperties.class, NotificationProperties.class, DeviceLogOssProperties.class})
public class PropertiesConfig {
}
@@ -29,7 +29,14 @@ public class TransientStorageProperties {
*/
private long maxTotalConcurrentOperations = 0;
private long acquirePermitTimeoutMillis = 2000;
private long baseRetryDelayMillis = 500;
/**
* 首次重试前的基础退避。
*
* <p>线上高频的重试诱因是 `unexpected end of stream`——那是**立即失败**(连接被 RustFS
* 重置后 OkHttp 读响应即报错),不是等超时,所以 500ms 基本是白等:每天上千次累计十几分钟。
* 降到 200ms 保留退避语义(真遇到服务端过载仍会退让),又不至于让用户等太久。
*/
private long baseRetryDelayMillis = 200;
private long maxRetryDelayMillis = 5000;
private long retryJitterMillis = 250;
private long failureWindowSeconds = 60;
@@ -37,7 +44,19 @@ public class TransientStorageProperties {
private long failureCooldownMillis = 10000;
private int dispatcherMaxRequests = 56;
private int dispatcherMaxRequestsPerHost = 56;
/**
* 空闲连接保留数。
*
* <p>2026-09-17 曾试过设 0(彻底不复用)来验证"unexpected end of stream 是复用死连接导致的"
* 这一假设——**实测照旧失败**(新容器起来后第一次请求就中招)。至此已排除公网链路、
* keepAlive 过长、连接复用三项;用 mc 并发压 200 个小对象也全部成功,说明服务端没问题。
* 剩余方向指向 MinIO Java SDK / OkHttp 与 RustFS 的协议细节,故恢复默认的连接复用。
*/
private int connectionPoolMaxIdle = 5;
/**
* 空闲连接在池里的保留时长。曾由 300000 调到 30000 试图减少 unexpected end of stream
* 实测无改善(该现象与连接复用无关,见 {@link #connectionPoolMaxIdle} 的排查记录),故恢复原值。
*/
private long connectionPoolKeepAliveMillis = 300000;
private long warnPayloadBytes = 5L * 1024 * 1024;
private long maxPayloadBytes = 50L * 1024 * 1024;
@@ -22,6 +22,13 @@ public class UserSecretProperties {
/** 单轮巡检时间预算(分钟),超时中断本轮。 */
private int checkBudgetMinutes = 20;
/**
* 检测请求使用的 LLM 模型:独立于业务任务模型(业务用 gemini-3.8-flash 等),
* 选便宜的可用模型,只验证密钥有效性与链路连通,降低每次检测与巡检的成本。
* 用 lite 而非 mini:mini 在中继分组下无可用渠道(503 model_not_found),实测 lite 可路由。
*/
private String checkModel = "doubao-seed-2-0-lite-260215";
/**
* 检测出口代理提取链接(选配):配置后检测请求优先经该代理 IP 发出,
* 代理不可用时自动回退直连;留空则全部直连。
@@ -76,8 +76,10 @@ public class AdminConsoleController {
@Operation(summary = "当前登录管理员的可见后台菜单树")
public ApiResponse<Map<String, Object>> currentUserMenus(HttpServletRequest request) {
AdminUserEntity operator = adminAuthSupport.requireAdminOrInternal(request);
// 补全祖先分组:部分授权用户(只授权了子页面)也要看到「一级分组 + 子页面」层级,
// 与超管的菜单组织顺序一致;分组节点无页面路由,不构成权限扩展。
List<PermissionMenuItemVo> menus = permissionMenuService.getUserColumnPermissions(
operator, operator.getId(), PermissionMenuService.MENU_TYPE_ADMIN);
operator, operator.getId(), PermissionMenuService.MENU_TYPE_ADMIN, true);
menus = AdminMenuTreeBuilder.filterByMenuType(menus, PermissionMenuService.MENU_TYPE_ADMIN);
List<Map<String, Object>> items = AdminMenuTreeBuilder.toMapList(AdminMenuTreeBuilder.build(menus));
return ApiResponse.success(Map.of("items", items));
@@ -16,6 +16,8 @@ public class AdminUserItemVo {
private String creatorUsername;
@JsonProperty("created_at")
private String createdAt;
@JsonProperty("updated_at")
private String updatedAt;
@JsonProperty("pinyin_abbr")
private String pinyinAbbr;
}
@@ -338,6 +338,8 @@ public class AdminUserService {
vo.setCreatorUsername(creatorId == null ? "" : creatorMap.getOrDefault(creatorId, ""));
LocalDateTime createdAt = entity.getCreatedAt();
vo.setCreatedAt(createdAt == null ? "" : createdAt.format(CREATED_AT_FORMATTER));
LocalDateTime updatedAt = entity.getUpdatedAt();
vo.setUpdatedAt(updatedAt == null ? "" : updatedAt.format(CREATED_AT_FORMATTER));
vo.setPinyinAbbr(PinyinAbbrUtil.abbr(entity.getUsername()));
return vo;
}
@@ -0,0 +1,45 @@
package com.nanri.aiimage.modules.appconfig.controller;
import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.modules.appconfig.service.KdFlowService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.servlet.http.HttpServletRequest;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;
import java.util.Map;
/**
* 工作台开店流程模块访问密码校验公开接口密码本身就是凭据不额外要求登录态
*
* <p>客户端只在用户点开开店流程分组时调用一次返回体只给 ok 与中文提示
* 不回显服务端配置的密码
*/
@Slf4j
@RestController
@RequiredArgsConstructor
@Tag(name = "开店流程访问校验", description = "工作台「开店流程」模块访问密码的服务端校验")
public class KdFlowController {
private final KdFlowService kdFlowService;
@PostMapping("/api/kd-flow/verify")
@Operation(summary = "校验开店流程访问密码",
description = "密码存 app_config.kd_flow_password;改密码只需 UPDATE 该行,客户端无需重新发布")
public ApiResponse<Map<String, Object>> verify(@RequestBody(required = false) Map<String, String> body,
HttpServletRequest request) {
String input = body == null ? null : body.get("password");
boolean ok = kdFlowService.matches(input);
// 只记输入长度与结果绝不回显密码本身
log.info("[开店流程] 校验请求 remoteAddr={} 输入为空={} 结果={}",
request.getRemoteAddr(), input == null || input.isBlank(), ok ? "通过" : "拒绝");
if (!ok) {
return ApiResponse.fail("密码错误");
}
return ApiResponse.success("验证通过", Map.of("ok", true));
}
}
@@ -0,0 +1,9 @@
package com.nanri.aiimage.modules.appconfig.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.nanri.aiimage.modules.appconfig.model.entity.AppConfigEntity;
import org.apache.ibatis.annotations.Mapper;
@Mapper
public interface AppConfigMapper extends BaseMapper<AppConfigEntity> {
}
@@ -0,0 +1,32 @@
package com.nanri.aiimage.modules.appconfig.model.entity;
import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableId;
import com.baomidou.mybatisplus.annotation.TableName;
import lombok.Data;
import java.time.LocalDateTime;
/**
* 通用应用配置键值首个用途工作台开店流程模块访问密码key = kd_flow_password
* <p>只放这类低价值需要"改一行即生效"的口令不放密钥类敏感配置
*/
@Data
@TableName("app_config")
public class AppConfigEntity {
@TableId(type = IdType.AUTO)
private Long id;
/** 配置键(唯一) */
private String configKey;
/** 配置值 */
private String configValue;
/** 说明 */
private String remark;
/** 更新时间,由数据库 CURRENT_TIMESTAMP 维护 */
private LocalDateTime updatedAt;
}
@@ -0,0 +1,48 @@
package com.nanri.aiimage.modules.appconfig.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.nanri.aiimage.modules.appconfig.mapper.AppConfigMapper;
import com.nanri.aiimage.modules.appconfig.model.entity.AppConfigEntity;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
/**
* 工作台开店流程模块访问密码的服务端校验
*
* <p>此前密码写死在客户端源码KD_FLOW_PASSWORD改密码必须重新打包装包发给全部用户
* 改由服务端比对后改密码只需 UPDATE app_config 一行key = kd_flow_password
*
* <p>不缓存调用频次极低用户点一次分组头一次且改密码后应立即生效
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class KdFlowService {
/** app_config 中存放开店流程访问密码的键名 */
public static final String PASSWORD_KEY = "kd_flow_password";
private final AppConfigMapper appConfigMapper;
/** 读取服务端配置的密码;未配置返回 null。 */
public String configuredPassword() {
AppConfigEntity row = appConfigMapper.selectOne(new LambdaQueryWrapper<AppConfigEntity>()
.eq(AppConfigEntity::getConfigKey, PASSWORD_KEY)
.last("LIMIT 1"));
return row == null ? null : row.getConfigValue();
}
/** 校验用户输入的密码。未配置密码时一律判失败(宁可锁死也不放行)。 */
public boolean matches(String input) {
String expect = configuredPassword();
if (expect == null || expect.isBlank()) {
log.warn("[开店流程] app_config 未配置 {},本次校验一律判失败", PASSWORD_KEY);
return false;
}
String actual = input == null ? "" : input.trim();
boolean ok = expect.equals(actual);
log.info("[开店流程] 服务端校验 输入长度={} 结果={}", actual.length(), ok ? "通过" : "不通过");
return ok;
}
}
@@ -424,6 +424,53 @@ public class AppearancePatentTaskService {
public void submitResult(Long taskId, AppearancePatentSubmitResultRequest request) {
submitResultLocked(taskId, request);
// 分片补传后自动恢复终态失败的组装 job 走与删除品牌同一套口径
// 此前外观专利没有接该入口分片补齐后只能人工重置 job线上任务 28459 即如此
maybeRecoverTerminalFailedAssemble(taskId);
}
/**
* 补传恢复此前因分片缺失导致组装 job 重试耗尽终态失败客户端补传缺口后
* 把失败的组装 job 重置为 PENDING 重新派发{@code resetTerminalFailedForRecovery} 自行补发 dispatch 事件
*
* <p>best-effort恢复失败不得影响补传本身分片已经落库恢复只是让后续组装继续推进
* 常态无终态失败 job下只查两次即返回不触发分片扫描
*/
private void maybeRecoverTerminalFailedAssemble(Long taskId) {
if (taskId == null || taskId <= 0) {
return;
}
try {
FileResultEntity result = findResultRecord(taskId);
if (result == null) {
return;
}
if (taskFileJobService.hasSuccessfulAssembleJob(taskId, MODULE_TYPE, result.getId())) {
return;
}
if (!taskFileJobService.isTerminalFailedAssembleJob(taskId, MODULE_TYPE, result.getId())) {
return;
}
if (!isResultSubmissionComplete(taskId)) {
log.info("[appearance-patent] 组装 job 终态失败但分片仍未补传完整,暂不恢复 taskId={} resultId={}",
taskId, result.getId());
return;
}
log.info("[appearance-patent] 分片已补传完整,恢复终态失败的组装 job taskId={} resultId={}",
taskId, result.getId());
taskFileJobService.resetTerminalFailedForRecovery(taskId, MODULE_TYPE, result.getId());
} catch (Exception ex) {
log.warn("[appearance-patent] 补传恢复检查失败(不影响本次补传)taskId={} err={}", taskId, ex.getMessage(), ex);
}
}
/** 按 task 取结果行(不创建);不存在返回 null。 */
private FileResultEntity findResultRecord(Long taskId) {
List<FileResultEntity> rows = fileResultMapper.selectList(new LambdaQueryWrapper<FileResultEntity>()
.eq(FileResultEntity::getTaskId, taskId)
.eq(FileResultEntity::getModuleType, MODULE_TYPE)
.last("limit 1"));
return rows == null || rows.isEmpty() ? null : rows.getFirst();
}
private void submitResultLocked(Long taskId, AppearancePatentSubmitResultRequest request) {
@@ -1065,6 +1112,14 @@ public class AppearancePatentTaskService {
long activeAssembleJobs = taskFileJobService.countActiveAssembleJobs(taskId, MODULE_TYPE);
log.info("[appearance-patent] stale recovery probe taskId={} uploadComplete={} activeAssembleJobs={} persistedRows={}",
taskId, uploadComplete, activeAssembleJobs, hasPersistedResultRows(taskId));
// 已有重试耗尽且已终态收尾 assemble job说明恢复已经试过缺失是永久的
// 再重建只会每 30 秒空转一轮而且恢复过程刷新任务心跳会让任务永远 RUNNING
// 线上任务 28459 实测48 分钟里每隔 30 秒重建一次 job返回 false 交给
// finalizeStaleTask 按失败收尾用户看到明确失败而不是无限等待
if (taskFileJobService.hasExhaustedAssembleJob(taskId, MODULE_TYPE)) {
log.warn("[appearance-patent] stale recovery 放弃:已有重试耗尽的 assemble job,按失败收尾 taskId={}", taskId);
return false;
}
if (!hasPersistedResultRows(taskId)) {
log.info("[appearance-patent] stale recovery aborted because no persisted rows taskId={}", taskId);
return false;
@@ -1215,6 +1270,7 @@ public class AppearancePatentTaskService {
if (rows == null || rows.isEmpty()) {
return;
}
String conflictDetail = "未发生冲突";
for (int attempt = 1; attempt <= CHUNK_PAYLOAD_MERGE_RETRY_LIMIT; attempt++) {
TaskChunkEntity chunk = taskChunkMapper.selectOne(new LambdaQueryWrapper<TaskChunkEntity>()
.eq(TaskChunkEntity::getTaskId, taskId)
@@ -1256,13 +1312,35 @@ public class AppearancePatentTaskService {
transientPayloadStorageService.deleteReplacedPayloadIfNeeded(oldPayload, storedPayload);
return;
}
transientPayloadStorageService.deletePayloadIfPresent(storedPayload);
// CAS 冲突读回行上的当前哈希否则只有一句 conflict 无从判断是谁改的线上 28459 至今未能定位
String currentHash = currentPayloadHash(chunk.getId());
conflictDetail = "期望hash=" + oldPayloadHash + " 当前hash=" + currentHash;
if (attempt < CHUNK_PAYLOAD_MERGE_RETRY_LIMIT) {
log.warn("[appearance-patent] chunk payload CAS conflict, retry merge taskId={} scopeHash={} chunk={} attempt={}/{}",
taskId, scopeHash, chunkIndex, attempt, CHUNK_PAYLOAD_MERGE_RETRY_LIMIT);
log.warn("[appearance-patent] chunk payload CAS conflict, retry merge taskId={} scopeHash={} chunk={} attempt={}/{} 期望hash={} 当前hash={}",
taskId, scopeHash, chunkIndex, attempt, CHUNK_PAYLOAD_MERGE_RETRY_LIMIT, oldPayloadHash, currentHash);
// 还要重试这次写的对象会被下次重写先删掉避免堆积
transientPayloadStorageService.deletePayloadIfPresent(storedPayload);
} else {
// 终局失败保留这次写入的版本化对象作为读路径同槽位兄弟对象兜底的恢复源
// 行没指过去不该让该分片永久判死删掉它才是线上 28459 丢数据的形态
log.error("[appearance-patent] chunk payload 合并冲突重试耗尽,保留兜底对象 taskId={} scopeHash={} chunk={} 期望hash={} 当前hash={} 保留对象={}",
taskId, scopeHash, chunkIndex, oldPayloadHash, currentHash, storedPayload);
}
}
throw new IllegalStateException("appearance patent chunk payload update conflict");
throw new IllegalStateException("appearance patent chunk payload update conflict " + conflictDetail);
}
/** 读回 chunk 行当前的 payload_hash,用于 CAS 冲突定位(行已不存在/读取失败时返回可读标记)。 */
private String currentPayloadHash(Long chunkId) {
if (chunkId == null) {
return "chunkId 为空";
}
try {
TaskChunkEntity latest = taskChunkMapper.selectById(chunkId);
return latest == null ? "行已不存在" : latest.getPayloadHash();
} catch (Exception ex) {
return "读取失败:" + ex.getMessage();
}
}
private List<AppearancePatentResultRowDto> expandRows(List<AppearancePatentResultRowDto> representatives,
@@ -2915,12 +2993,35 @@ public class AppearancePatentTaskService {
} catch (Exception ex) {
log.warn("[appearance-patent] read chunk payload failed taskId={} chunk={} err={}",
chunk.getTaskId(), chunk.getChunkIndex(), ex.getMessage());
if (isPayloadMissing(ex)) {
// payload 对象已不在被清理或从未写入重试多少次都读不回来继续抛会让
// ASSEMBLE_RESULT job 的终态回调每轮重跑兜底组装 再读同一个缺失对象 无限循环
// 线上任务 28459 10~30 秒重试一次跳过该分片让任务按已有分片出部分结果
// 与品牌/相似ASIN失败也产出可下载的部分结果同一口径
log.warn("[appearance-patent] chunk payload 已不存在,跳过该分片(任务按已有分片出结果)"
+ " taskId={} chunk={}", chunk.getTaskId(), chunk.getChunkIndex());
return rows;
}
throw new BusinessException("appearance patent chunk payload read failed chunk="
+ chunk.getChunkIndex() + ": " + ex.getMessage(), ex);
}
return rows;
}
/** payload 对象已不存在RustFS 返回 NoSuchKeymessage "The specified key does not exist."
* 只有这种"重试也没用"的缺失才允许跳过网络类失败仍照旧抛出以便重试 */
private static boolean isPayloadMissing(Throwable error) {
Throwable cursor = error;
while (cursor != null) {
String message = cursor.getMessage();
if (message != null && message.contains("does not exist")) {
return true;
}
cursor = cursor.getCause();
}
return false;
}
private String rowKey(AppearancePatentParsedRowVo row) {
if (row == null) {
return "";
@@ -95,10 +95,14 @@ public class BrandCheckClient {
public BrandCheckBatchResult checkAll(List<String> brands, String strategy) {
List<String> distinctBrands = distinctNonBlank(brands);
// 整批共用一个耗时预算上游 16890 卡死时单品牌 10 次重试曾把一次分片回传拖到
// 103.5 taskId 28599客户端重试预算耗尽后中止了整个采集预算用尽即停止重试
long budgetMillis = properties.getTotalTimeoutMillis();
long deadlineNanos = budgetMillis > 0L ? System.nanoTime() + budgetMillis * 1_000_000L : Long.MAX_VALUE;
List<CompletableFuture<BrandCheckOutcome>> futures = new ArrayList<>(distinctBrands.size());
for (String brand : distinctBrands) {
futures.add(CompletableFuture.supplyAsync(
() -> checkOneBrand(brand, strategy), checkExecutor));
() -> checkOneBrand(brand, strategy, deadlineNanos), checkExecutor));
}
List<Object> failedData = new ArrayList<>();
List<Object> queryFailedData = new ArrayList<>();
@@ -110,11 +114,19 @@ public class BrandCheckClient {
return new BrandCheckBatchResult(distinctBrands, failedData, queryFailedData);
}
private BrandCheckOutcome checkOneBrand(String brand, String strategy) {
private BrandCheckOutcome checkOneBrand(String brand, String strategy, long deadlineNanos) {
int attempts = Math.max(1, properties.getRetryTimes());
BrandCheckResponse response = null;
Exception lastFailure = null;
for (int attempt = 1; attempt <= attempts; attempt++) {
// 预算用尽就不再重试按查询失败收尾只掐重试不打断已发出的请求
// 故最坏耗时 预算 + 一次请求的读超时首轮始终执行避免上游只是慢一点时被误降级
if (attempt > 1 && System.nanoTime() >= deadlineNanos) {
log.warn("[brand-check] 整批耗时预算用尽,停止重试 brand={} attempt={}/{} lastErr={}",
brand, attempt, attempts,
lastFailure == null ? "query_faild_data 持续非空" : lastFailure.getMessage());
break;
}
try {
response = check(brand, strategy);
} catch (Exception ex) {
@@ -3,7 +3,31 @@ package com.nanri.aiimage.modules.brand.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.nanri.aiimage.modules.brand.model.entity.BrandCrawlTaskEntity;
import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param;
import org.apache.ibatis.annotations.Select;
import java.time.LocalDateTime;
import java.util.List;
@Mapper
public interface BrandCrawlTaskMapper extends BaseMapper<BrandCrawlTaskEntity> {
/**
* 查询超过保留期的终态品牌检测任务 id保留期清理用只取 id 不拉整行历史行的
* file_paths/result_paths JSON 字段可能很大
*
* <p>终态集合与 {@code BrandTaskService} 的状态机一致success/failed/cancelled
* pending/running 绝不返回删了正在跑的任务结果回传会找不到任务行
* 时间线用 updated_at BrandTaskStaleRepairSpiImpl 的陈旧判定同款口径
* 可命中 V120 idx_brand_crawl_task_status_updated(status, updated_at) 索引
*/
@Select("""
SELECT id FROM brand_crawl_tasks
WHERE status IN ('success', 'failed', 'cancelled')
AND updated_at < #{cutoff}
ORDER BY id ASC
LIMIT #{batchSize}
""")
List<Long> selectExpiredTerminalTaskIds(@Param("cutoff") LocalDateTime cutoff,
@Param("batchSize") int batchSize);
}
@@ -0,0 +1,93 @@
package com.nanri.aiimage.modules.brand.service;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.modules.brand.mapper.BrandCrawlTaskMapper;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.time.Duration;
import java.time.LocalDateTime;
import java.util.List;
/**
* 品牌检测任务brand_crawl_tasks的保留期清理2026-09 审核该表只增不删永久累积
*
* <p>该表自建不写 biz_file_task故不在 ModuleHistoryCleanupService 的清理名单里
* 此前没有任何删除路径这里只查过期终态任务的 id逐个走
* {@link BrandTaskService#deleteTask(Long)} 既有删除入口 它已处理任务行删除 +
* 存储数据清理brandTaskStorageService.deleteTaskData+ 进度缓存清理本类不重新实现删除逻辑
*
* <p>双节点用 job 锁保证单实例执行每批小批量默认 50逐个删避免单次跑太久占住锁
*/
@Service
@RequiredArgsConstructor
@Slf4j
public class BrandTaskRetentionService {
/** 单轮最多处理的批次数(每批 batchSize 个任务),剩余留给下一轮。 */
static final int MAX_BATCHES_PER_RUN = 20;
private final BrandCrawlTaskMapper brandCrawlTaskMapper;
private final BrandTaskService brandTaskService;
private final DistributedJobLockService distributedJobLockService;
@Value("${aiimage.brand.task-retention-days:90}")
private int retentionDays = 90;
@Value("${aiimage.brand.task-retention-batch-size:50}")
private int retentionBatchSize = 50;
@Scheduled(cron = "${aiimage.brand.task-retention-cron:0 45 4 * * *}")
public void purgeExpiredTasks() {
int days = Math.max(1, retentionDays);
int batchSize = Math.max(1, retentionBatchSize);
LocalDateTime cutoff = LocalDateTime.now().minusDays(days);
// 单轮 20 × 50 个任务的删除含存储数据清理可能跑较久 TTL 给足 30 分钟
DistributedJobLockService.LockHandle lockHandle =
distributedJobLockService.tryLock("brand:task-retention", Duration.ofMinutes(30));
if (lockHandle == null) {
log.info("[brand-retention] 任务保留清理跳过:另一实例持锁");
return;
}
try (lockHandle) {
int totalDeleted = 0;
int totalFailed = 0;
int batches = 0;
while (batches < MAX_BATCHES_PER_RUN) {
// 只取 id历史行的 file_paths/result_paths JSON 字段可能很大
List<Long> taskIds = brandCrawlTaskMapper.selectExpiredTerminalTaskIds(cutoff, batchSize);
if (taskIds.isEmpty()) {
break;
}
batches++;
int deletedInBatch = 0;
for (Long taskId : taskIds) {
try {
brandTaskService.deleteTask(taskId);
deletedInBatch++;
} catch (Exception ex) {
// 单个任务删除失败只记日志继续一个坏任务不能卡住整轮
log.warn("[brand-retention] 任务删除失败 taskId={} msg={}", taskId, ex.getMessage(), ex);
}
}
totalDeleted += deletedInBatch;
totalFailed += taskIds.size() - deletedInBatch;
if (deletedInBatch == 0) {
log.warn("[brand-retention] 整批任务均删除失败,提前结束本轮以免反复重试同一批");
break;
}
if (taskIds.size() < batchSize) {
break;
}
}
log.info("[brand-retention] 任务保留清理完成 cutoff={} retentionDays={} deleted={} failed={} batches={}",
cutoff, days, totalDeleted, totalFailed, batches);
} catch (Exception ex) {
log.warn("[brand-retention] 任务保留清理失败 cutoff={} msg={}", cutoff, ex.getMessage(), ex);
}
}
}
@@ -782,6 +782,39 @@ public class CollectDataService {
throw new BusinessException("request is empty");
}
ensureRustfsPayloadStorageEnabled();
// 锁外预检任务不存在/已结束时立即失败不为终态任务白跑去重查询与品牌检测
// 只做快速失败并发正确性仍由锁内的重读复核保证
FileTaskEntity probe = fileTaskMapper.selectById(taskId);
if (probe == null || !MODULE_TYPE.equals(probe.getModuleType())) {
throw new BusinessException("任务不存在");
}
if (STATUS_SUCCESS.equals(probe.getStatus()) || STATUS_FAILED.equals(probe.getStatus())) {
log.warn("[collect-data] 任务已结束,拒绝重复提交 taskId={} status={}", taskId, probe.getStatus());
throw new BusinessException(40901, "任务已结束,拒绝重复提交");
}
// 归一化 / 去重过滤 / 品牌检测放在锁外品牌检测是同步远程调用上游 16890 抖动时
// 单品牌 10 次重试合计上百秒taskId 28599 实测chunk 回传在锁内等品牌检测 103.5
// 期间心跳与客户端重试全部撞 40902 拿不到锁客户端 5 次重试预算耗尽后中止整个采集
// 这几步只依赖本批入参不写任务状态放锁外不改变 chunk 落库的串行语义
List<CollectDataResultRowVo> rows = normalizeSubmitRows(request.getItems());
buildParseLimits().validateChunkRowCount(rows.size());
// 结果文件sheet 需要 Python 回传的全量数据不经任何后端过滤
// 因此 chunk payload 写入未筛除空 ASIN 的全量 rows
// 仅当 ASIN 非空时才进入去重 / 无效品牌 / 品牌检测链路并落 biz_task_result_item
List<CollectDataResultRowVo> rowsForFiltering = new ArrayList<>(rows.size());
for (CollectDataResultRowVo row : rows) {
if (row.getAsin() != null && !row.getAsin().isBlank()) {
rowsForFiltering.add(row);
}
}
long prepareStartAt = System.currentTimeMillis();
CollectDataBatchQuery.FilterResult filtered = collectDataBatchQuery.filter(rowsForFiltering);
CollectDataBrandBatchFilter.BrandBatchOutcome brandOutcome = brandBatchFilter.filter(filtered.kept());
log.info("[collect-data] 锁外预处理完成 taskId={} rows={} 去重后={} 品牌检测耗时={}ms",
taskId, rows.size(), filtered.kept().size(), System.currentTimeMillis() - prepareStartAt);
try (TaskDistributedLockService.LockHandle ignored = acquireTaskLockOrThrow(taskId)) {
FileTaskEntity task = fileTaskMapper.selectById(taskId);
if (task == null || !MODULE_TYPE.equals(task.getModuleType())) {
@@ -819,25 +852,23 @@ public class CollectDataService {
return buildSubmitVo(task, result, chunkIndex, chunkTotal, request, 0);
}
List<CollectDataResultRowVo> rows = normalizeSubmitRows(request.getItems());
buildParseLimits().validateChunkRowCount(rows.size());
CollectDataStats stats = loadStats(task);
stats.receivedRows += rows.size();
stats.currentChunkRows = rows.size();
// 结果文件sheet 需要 Python 回传的全量数据不经任何后端过滤
// 因此 chunk payload 写入未筛除空 ASIN 的全量 rows
// 仅当 ASIN 非空时才进入去重 / 无效品牌 / 品牌检测链路并落 biz_task_result_item
List<CollectDataResultRowVo> rowsForFiltering = new ArrayList<>(rows.size());
for (CollectDataResultRowVo row : rows) {
if (row.getAsin() != null && !row.getAsin().isBlank()) {
rowsForFiltering.add(row);
}
}
CollectDataBatchQuery.FilterResult filtered = collectDataBatchQuery.filter(rowsForFiltering);
stats.dedupeFilteredCount += filtered.dedupeFilteredCount();
stats.invalidFilteredCount += filtered.invalidFilteredCount();
List<CollectDataResultRowVo> accepted = filterByBrandCheck(filtered.kept(), stats);
stats.brandRejectedCount += brandOutcome.rejected().size();
stats.brandQueryFailedCount += brandOutcome.queryFailed().size();
// 只有被服务端判定为无效品牌rejected的行才写入无效品牌表
// queryFailed 是品牌检测服务端到端失败如缺 X-Token 422/超时/限流
// 一并写入会把故障期间被误伤的品牌永久拉黑后续同品牌商品全部被
// invalidFiltered 过滤task-27265 实测通过 Python 过滤的 21 行中
// 8 行查询失败被写表后误杀最终结果 Excel 0 rejected 为空时
// 不触发任何写入避免空批次无意义调用
if (!brandOutcome.rejected().isEmpty()) {
invalidAsinBatchWriter.writeBatch(brandOutcome.rejected());
}
List<CollectDataResultRowVo> accepted = brandOutcome.accepted();
// 结果明细改为 chunk 级存储整个 chunk accepted 行共享一个
// RustFS 对象deterministic key chunk 重提覆盖同一对象
// biz_task_result_item.payload_json 只存 {chunk, offset, payload} 引用
@@ -872,6 +903,14 @@ public class CollectDataService {
if (request.getError() != null && !request.getError().isBlank()) {
markTaskFailed(task, result, request.getError(), stats);
// 失败但已收到分片照常组装结果文件让用户能下载已采集的数据
// 此前失败分支只标失败不组装已落库的数据也没有任何结果文件可下载
// taskId 2859955 个分片全部收到187 行明细已落库用户却拿不到文件
if (hasReceivedChunks(taskId)) {
enqueueFinalWorkbook(task, result, stats);
log.warn("[collect-data] 任务失败仍组装部分结果 taskId={} error={} finalRows={}",
taskId, request.getError(), stats.finalRowCount);
}
} else if (Boolean.TRUE.equals(request.getDone())) {
enqueueFinalWorkbook(task, result, stats);
} else {
@@ -933,22 +972,6 @@ public class CollectDataService {
return rows;
}
private List<CollectDataResultRowVo> filterByBrandCheck(List<CollectDataResultRowVo> rows, CollectDataStats stats) {
CollectDataBrandBatchFilter.BrandBatchOutcome outcome = brandBatchFilter.filter(rows);
stats.brandRejectedCount += outcome.rejected().size();
stats.brandQueryFailedCount += outcome.queryFailed().size();
// 只有被服务端判定为无效品牌rejected的行才写入无效品牌表
// queryFailed 是品牌检测服务端到端失败如缺 X-Token 422/超时/限流
// 一并写入会把故障期间被误伤的品牌永久拉黑后续同品牌商品全部被
// invalidFiltered 过滤task-27265 实测通过 Python 过滤的 21 行中
// 8 行查询失败被写表后误杀最终结果 Excel 0 rejected 为空时
// 不触发任何写入避免空批次无意义调用
if (!outcome.rejected().isEmpty()) {
invalidAsinBatchWriter.writeBatch(outcome.rejected());
}
return outcome.accepted();
}
private void persistChunk(Long taskId,
String scopeKey,
String scopeHash,
@@ -1011,7 +1034,8 @@ public class CollectDataService {
result.setResultFileSize(0L);
result.setResultContentType(CONTENT_TYPE_XLSX);
result.setRowCount(stats.finalRowCount);
result.setErrorMessage(null);
// 不清 errorMessage失败任务的部分结果组装也走这里清掉会让用户看不到真实失败原因
// 成功路径的 errorMessage 本来就为 null无需清理
fileResultMapper.updateById(result);
task.setUpdatedAt(LocalDateTime.now());
fileTaskMapper.updateById(task);
@@ -1060,6 +1084,22 @@ public class CollectDataService {
stats.summaries,
batch -> streamRawRows(task.getId(), batch));
String objectKey = ossStorageService.uploadResultFile(xlsx, MODULE_TYPE);
// 复用同一份 stats 更新 finalRowCount 后再持久化避免重复 loadStats 丢失 summaries
stats.finalRowCount = (int) finalRowCount;
persistStats(task, stats);
// 失败原因先留存下面的乐观写入会清空 result.errorMessage任务已被判失败时要用它恢复
String failureReason = result.getErrorMessage();
if (failureReason == null || failureReason.isBlank()) {
failureReason = task.getErrorMessage();
}
if (failureReason == null || failureReason.isBlank()) {
failureReason = "任务失败,结果文件为已采集的部分数据";
}
// 结果行先按成功乐观写入保持结果文件先于任务成功落库的时序
// 万一进程在这两步之间退出任务仍是 RUNNING会被陈旧巡检重新组装可自愈
result.setResultFilename(filename);
result.setResultFileUrl(objectKey);
result.setResultFileSize(xlsx.length());
@@ -1069,10 +1109,7 @@ public class CollectDataService {
result.setErrorMessage(null);
fileResultMapper.updateById(result);
// 复用同一份 stats 更新 finalRowCount 后再持久化避免重复 loadStats 丢失 summaries
stats.finalRowCount = (int) finalRowCount;
persistStats(task, stats);
// 条件更新任务可能已被 /fail 标为 FAILED客户端报错与结果文件组装并发
// 条件更新任务可能已被判失败客户端上报失败 / 陈旧判死与结果文件组装并发
// 无条件 updateById 会把 FAILED 覆盖成 SUCCESS错误信息被清用户看到假成功
// 结果文件已生成故失败态下仍保留文件只是不覆盖状态
int updated = fileTaskMapper.update(null, new LambdaUpdateWrapper<FileTaskEntity>()
@@ -1085,7 +1122,13 @@ public class CollectDataService {
.set(FileTaskEntity::getUpdatedAt, LocalDateTime.now())
.set(FileTaskEntity::getFinishedAt, LocalDateTime.now()));
if (updated == 0) {
log.warn("[collect-data] 结果文件已生成但任务已是 FAILED,保留失败态不覆盖 taskId={}", task.getId());
// 任务已是 FAILED结果记录改回失败语义并保留真实原因但文件 URL 照常保留
// 用户看到失败 + 原因的同时仍能下载已采集的部分结果
result.setSuccess(0);
result.setErrorMessage(failureReason);
fileResultMapper.updateById(result);
log.warn("[collect-data] 结果文件已生成但任务已是 FAILED,保留失败态与原因 taskId={} rows={} reason={}",
task.getId(), finalRowCount, failureReason);
}
} finally {
FileUtil.del(xlsx);
@@ -1,6 +1,8 @@
package com.nanri.aiimage.modules.dedupe.model.entity;
import com.baomidou.mybatisplus.annotation.FieldStrategy;
import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableField;
import com.baomidou.mybatisplus.annotation.TableId;
import com.baomidou.mybatisplus.annotation.TableName;
import lombok.Data;
@@ -19,5 +21,12 @@ public class DedupeTotalDataEntity {
private Long uploaderUserId;
private String uploaderUsername;
private LocalDateTime createdAt;
/**
* 更新时间由数据库维护DEFAULT CURRENT_TIMESTAMP + ON UPDATE CURRENT_TIMESTAMP
*
* <p>禁止应用显式写MySQL UPDATE 语句显式给该列赋值时不会触发自动更新
* 而本表写回为 selectById 改字段 updateById实体带着旧值一旦写回就会冻结更新时间
*/
@TableField(value = "updated_at", insertStrategy = FieldStrategy.NEVER, updateStrategy = FieldStrategy.NEVER)
private LocalDateTime updatedAt;
}
@@ -32,4 +32,7 @@ public class DedupeTotalDataItemVo {
@Schema(description = "创建时间")
private LocalDateTime createdAt;
@Schema(description = "更新时间")
private LocalDateTime updatedAt;
}
@@ -42,6 +42,7 @@ import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.Semaphore;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Supplier;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
@@ -487,7 +488,7 @@ public class DedupeRunService {
readResult.scannedRows = new AtomicInteger(0);
readResult.filteredFbaRows = new AtomicInteger(0);
PendingMainIdGroup pendingMainIdGroup = new PendingMainIdGroup();
IdRuleRowPicker rowPicker = new IdRuleRowPicker(keepIntegerIds, keepUnderscoreIds, keepIntegerMainIdsWhenNoSubIds);
// 表头列索引缓存 onHeader 填充数据行为空时使用
final Map<String, Integer>[] headerIndexCache = new Map[]{Map.of()};
@@ -526,23 +527,16 @@ public class DedupeRunService {
}
Integer idColumnIndex = headerIndex.get("id");
if (idColumnIndex == null) {
readResult.rows.add(buildCandidateRow(rowMap, headerIndex, orderedSelectedColumns, null));
rowPicker.addAlways(buildCandidateRow(rowMap, headerIndex, orderedSelectedColumns, null));
return;
}
appendRowByIdRule(
rowPicker.select(
normalizeCellText(cellText(rowMap, idColumnIndex)),
rowMap,
headerIndex,
orderedSelectedColumns,
keepIntegerIds,
keepUnderscoreIds,
keepIntegerMainIdsWhenNoSubIds,
pendingMainIdGroup,
readResult.rows
() -> buildCandidateRow(rowMap, headerIndex, orderedSelectedColumns, null)
);
}
});
pendingMainIdGroup.flush(readResult.rows);
readResult.rows = rowPicker.resolve();
long readNs = elapsedNs(readStartNs);
Set<String> candidateAsinValues = new HashSet<>();
@@ -560,8 +554,9 @@ public class DedupeRunService {
try (SXSSFWorkbook outputWorkbook = new SXSSFWorkbook(200)) {
org.apache.poi.ss.usermodel.Sheet outputSheet = outputWorkbook.createSheet(WorkbookUtil.createSafeSheetName(readResult.sheetName.isBlank() ? "Sheet1" : readResult.sheetName));
Row outputHeaderRow = outputSheet.createRow(0);
for (int i = 0; i < selectedColumns.size(); i++) {
outputHeaderRow.createCell(i).setCellValue(selectedColumns.get(i));
// 表头必须与数据行同序数据按 orderedSelectedColumns 取值表头写 selectedColumns 会整体错列
for (int i = 0; i < orderedSelectedColumns.size(); i++) {
outputHeaderRow.createCell(i).setCellValue(orderedSelectedColumns.get(i));
}
Set<String> writtenAsinValues = new HashSet<>();
@@ -659,37 +654,83 @@ public class DedupeRunService {
return new DedupeCandidateRow(selectedValues, asinValue);
}
private void appendRowByIdRule(String idValue, Map<Integer, String> rowMap,
Map<String, Integer> headerIndex, List<String> selectedColumns,
boolean keepIntegerIds, boolean keepUnderscoreIds,
boolean keepIntegerMainIdsWhenNoSubIds,
PendingMainIdGroup pendingMainIdGroup,
List<DedupeCandidateRow> rows) {
/**
* ID 保留规则挑选输出行
*
* <p>主链接行是否保留只取决于该主 ID 是否出现过子链接行与两类行在源文件中的
* 先后顺序无关旧实现用暂存主链接行 + 后到的同主 ID 子链接行把它丢弃的方式
* 一旦顺序是子链接在前主链接在后子行到来时暂存区尚空无人记录该主 ID 已有
* 子链接主链接就会一路存活到收尾导致同一主 ID 的主/子链接同时出现在结果里</p>
*/
static final class IdRuleRowPicker {
private final boolean keepIntegerIds;
private final boolean keepUnderscoreIds;
private final boolean keepIntegerMainIdsWhenNoSubIds;
/** 出现过子链接行的主 ID 集合,与源文件顺序无关 */
private final Set<String> mainIdsWithSubRows = new HashSet<>();
private final List<PickedRow> pickedRows = new ArrayList<>();
private int subRowCount = 0;
private int mainRowCount = 0;
IdRuleRowPicker(boolean keepIntegerIds, boolean keepUnderscoreIds, boolean keepIntegerMainIdsWhenNoSubIds) {
this.keepIntegerIds = keepIntegerIds;
this.keepUnderscoreIds = keepUnderscoreIds;
this.keepIntegerMainIdsWhenNoSubIds = keepIntegerMainIdsWhenNoSubIds;
}
/** ID 不参与保留规则判定的行(如无 id 列或 id 为空)直接保留。 */
void addAlways(DedupeCandidateRow row) {
pickedRows.add(new PickedRow(row, null));
}
/** 按 ID 形态与保留规则挑选;rowSupplier 仅在确定保留时才求值。 */
void select(String idValue, Supplier<DedupeCandidateRow> rowSupplier) {
if (idValue == null || idValue.isBlank()) {
return;
}
String mainId = extractMainId(idValue);
if (pendingMainIdGroup.hasDifferentMainId(mainId)) {
pendingMainIdGroup.flush(rows);
}
if (isUnderscoreId(idValue)) {
pendingMainIdGroup.discardIfSameMainId(mainId);
subRowCount++;
mainIdsWithSubRows.add(mainId);
if (keepUnderscoreIds) {
rows.add(buildCandidateRow(rowMap, headerIndex, selectedColumns, null));
pickedRows.add(new PickedRow(rowSupplier.get(), null));
}
return;
}
if (isIntegerId(idValue)) {
if (keepIntegerIds) {
rows.add(buildCandidateRow(rowMap, headerIndex, selectedColumns, null));
pickedRows.add(new PickedRow(rowSupplier.get(), null));
return;
}
if (keepIntegerMainIdsWhenNoSubIds) {
pendingMainIdGroup.add(mainId, buildCandidateRow(rowMap, headerIndex, selectedColumns, null));
mainRowCount++;
pickedRows.add(new PickedRow(rowSupplier.get(), mainId));
}
}
}
/** 收尾统一判定:带条件的主链接行仅在该主 ID 没有子链接行时保留。 */
List<DedupeCandidateRow> resolve() {
List<DedupeCandidateRow> resolved = new ArrayList<>(pickedRows.size());
int droppedMainRowCount = 0;
for (PickedRow picked : pickedRows) {
if (picked.conditionalMainId() != null && mainIdsWithSubRows.contains(picked.conditionalMainId())) {
droppedMainRowCount++;
continue;
}
resolved.add(picked.row());
}
log.info("dedupe id rules subRows={} mainRows={} mainIdsWithSubRows={} droppedMainRows={} keptRows={}",
subRowCount, mainRowCount, mainIdsWithSubRows.size(), droppedMainRowCount, resolved.size());
return resolved;
}
/** conditionalMainId 非空表示该行是主链接行,需等收尾时看同主 ID 有无子链接行 */
private record PickedRow(DedupeCandidateRow row, String conditionalMainId) {
}
}
private String cellText(Map<Integer, String> rowMap, Integer columnIndex) {
if (columnIndex == null) {
return "";
@@ -730,7 +771,7 @@ public class DedupeRunService {
return String.join("/", parts);
}
private String extractMainId(String text) {
static String extractMainId(String text) {
if (text == null || text.isBlank()) {
return "";
}
@@ -744,7 +785,7 @@ public class DedupeRunService {
return "";
}
private boolean isIntegerId(String text) {
static boolean isIntegerId(String text) {
if (text == null || text.isEmpty()) {
return false;
}
@@ -756,7 +797,7 @@ public class DedupeRunService {
return true;
}
private boolean isUnderscoreId(String text) {
static boolean isUnderscoreId(String text) {
if (text == null || text.length() < 3) {
return false;
}
@@ -952,38 +993,6 @@ public class DedupeRunService {
private AtomicInteger filteredFbaRows;
}
private record DedupeCandidateRow(List<String> selectedValues, String asinValue) {
}
private static final class PendingMainIdGroup {
private String mainId;
private final List<DedupeCandidateRow> rows = new ArrayList<>();
private void add(String nextMainId, DedupeCandidateRow row) {
if (hasDifferentMainId(nextMainId)) {
rows.clear();
}
mainId = nextMainId;
rows.add(row);
}
private boolean hasDifferentMainId(String nextMainId) {
return mainId != null && (nextMainId == null || nextMainId.isBlank() || !mainId.equals(nextMainId));
}
private void discardIfSameMainId(String nextMainId) {
if (mainId != null && mainId.equals(nextMainId)) {
rows.clear();
mainId = null;
}
}
private void flush(List<DedupeCandidateRow> outputRows) {
if (!rows.isEmpty()) {
outputRows.addAll(rows);
rows.clear();
}
mainId = null;
}
record DedupeCandidateRow(List<String> selectedValues, String asinValue) {
}
}
@@ -1398,6 +1398,7 @@ public class DedupeTotalDataService {
vo.setUploaderUserId(entity.getUploaderUserId());
vo.setUsername(entity.getUploaderUsername());
vo.setCreatedAt(entity.getCreatedAt());
vo.setUpdatedAt(entity.getUpdatedAt());
return vo;
}
@@ -27,6 +27,7 @@ import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskHeartbeatPositionService;
import com.nanri.aiimage.modules.task.service.TaskResumeService;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
@@ -92,6 +93,8 @@ public class DeleteBrandStaleTaskService {
private final ShopDataCrawlTaskService shopDataCrawlTaskService;
private final TaskScopeStateMapper taskScopeStateMapper;
private final TaskHeartbeatPositionService taskHeartbeatPositionService;
/** 客户端中断任务的自动续跑(保留失败记录 + 重排队续跑任务,V129)。 */
private final TaskResumeService taskResumeService;
@Value("${aiimage.temp-dir.retention-hours:24}")
private long tempDirRetentionHours;
@@ -114,13 +117,16 @@ public class DeleteBrandStaleTaskService {
ShopMatchStaleCheckStats queryAsinStats = failStaleQueryAsinTasks();
ShopMatchStaleCheckStats withdrawStats = failStaleWithdrawTasks();
NoResultUploadStaleCheckStats noUploadStats = failNoResultUploadTasks();
// 客户端重启中断的任务除了标失败客户端上报用户能看到原因还要重新排队
// 一条 PENDING 续跑任务交给客户端兜底拉取执行否则长任务一遇客户端更新就整个白跑
TaskResumeService.ResumeStats resumeStats = resumeInterruptedSafely();
for (Map.Entry<String, Runnable> delegated : delegatedStaleChecks().entrySet()) {
runModuleStaleCheck(delegated.getKey(), delegated.getValue());
}
// 周期每 2 分钟一轮各模块 summary 合并为单行避免定期刷屏
// 注意每段占位符数量必须与实参一致此前每段 5 个占位符只传 4 个参数
// 导致 withdraw 之后的取值整体错位末尾 elapsedMs/thread 打成字面量
log.info("[stale-check] summary product-risk(s={} f={} x={} p={}) price-track(s={} f={} x={} p={}) shop-match(s={} f={} x={} p={}) patrol-delete(s={} f={} x={} p={}) query-asin(s={} f={} x={} p={}) withdraw(s={} f={} x={} p={}) no-upload(c={} f={} x={}) elapsedMs={} thread={}",
log.info("[stale-check] summary product-risk(s={} f={} x={} p={}) price-track(s={} f={} x={} p={}) shop-match(s={} f={} x={} p={}) patrol-delete(s={} f={} x={} p={}) query-asin(s={} f={} x={} p={}) withdraw(s={} f={} x={} p={}) no-upload(c={} f={} x={}) resume(s={} r={} k={} u={}) elapsedMs={} thread={}",
stats.scannedTaskCount, stats.finalizedTaskCount, stats.failedTaskCount, stats.skippedTaskCount,
priceTrackStats.scannedTaskCount, priceTrackStats.finalizedTaskCount, priceTrackStats.failedTaskCount, priceTrackStats.skippedTaskCount,
shopMatchStats.scannedTaskCount, shopMatchStats.finalizedTaskCount, shopMatchStats.failedTaskCount, shopMatchStats.skippedTaskCount,
@@ -128,11 +134,27 @@ public class DeleteBrandStaleTaskService {
queryAsinStats.scannedTaskCount, queryAsinStats.finalizedTaskCount, queryAsinStats.failedTaskCount, queryAsinStats.skippedTaskCount,
withdrawStats.scannedTaskCount, withdrawStats.finalizedTaskCount, withdrawStats.failedTaskCount, withdrawStats.skippedTaskCount,
noUploadStats.scannedTaskCount, noUploadStats.failedTaskCount, noUploadStats.skippedTaskCount,
resumeStats.scannedTaskCount, resumeStats.resumedTaskCount, resumeStats.skippedTaskCount,
resumeStats.unsupportedTaskCount,
System.currentTimeMillis() - startedAt,
Thread.currentThread().getName());
}
}
/**
* 自动续跑巡检的隔离壳续跑失败绝不能拖垮判死主流程
* 判死是任务状态的兜底不做会留下永远 RUNNING 的孤儿优先级高于续跑
* 这里失败只记日志并返回空统计2 分钟后的下一轮自然重试
*/
private TaskResumeService.ResumeStats resumeInterruptedSafely() {
try {
return taskResumeService.resumeInterruptedTasks();
} catch (Exception ex) {
log.warn("[task-resume] 自动续跑巡检失败(不影响本轮判死): {}", ex.getMessage(), ex);
return new TaskResumeService.ResumeStats();
}
}
/**
* 委派式陈旧判死moduleType 处理动作
* {@code TaskModuleRegistry} delegatedStaleCheck=true 的模块都必须在这里登记
@@ -0,0 +1,159 @@
package com.nanri.aiimage.modules.devicelog.controller;
import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.common.model.entity.AdminUserEntity;
import com.nanri.aiimage.common.security.AdminAuthSupport;
import com.nanri.aiimage.modules.devicelog.model.entity.DeviceLogConfigEntity;
import com.nanri.aiimage.modules.devicelog.model.entity.DeviceLogFileEntity;
import com.nanri.aiimage.modules.devicelog.model.vo.DeviceLogContentVo;
import com.nanri.aiimage.modules.devicelog.model.vo.DeviceLogPageVo;
import com.nanri.aiimage.modules.devicelog.service.DeviceLogConfigService;
import com.nanri.aiimage.modules.devicelog.service.DeviceLogService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.format.annotation.DateTimeFormat;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PutMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import java.io.IOException;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.time.LocalDate;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
/**
* 后台日志管理桌面客户端与麦象采集机的日志浏览仅超管
* 列表/内容/下载/删除 + 采集配置全局默认与终端覆盖
*/
@RestController
@RequiredArgsConstructor
@Slf4j
@RequestMapping("/api/admin/device-logs")
@Tag(name = "日志管理(后台)", description = "设备日志列表、内容查看、下载与采集配置(仅超管)。")
public class AdminDeviceLogController {
private final DeviceLogService deviceLogService;
private final DeviceLogConfigService deviceLogConfigService;
private final AdminAuthSupport adminAuthSupport;
@GetMapping("/files")
@Operation(summary = "日志文件分页列表",
description = "source=client/maixiangkeyword 模糊匹配设备名/设备ID/文件名;日期为闭区间(早于保留窗口自动收紧)。")
public ApiResponse<DeviceLogPageVo> files(
HttpServletRequest request,
@Parameter(description = "来源") @RequestParam(required = false) String source,
@Parameter(description = "关键字(设备/文件名)") @RequestParam(required = false) String keyword,
@Parameter(description = "起始日期(含)") @RequestParam(required = false)
@DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate startDate,
@Parameter(description = "结束日期(含)") @RequestParam(required = false)
@DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate endDate,
@RequestParam(defaultValue = "1") Long page,
@RequestParam(defaultValue = "20") Long pageSize) {
requireSuperAdmin(request);
return ApiResponse.success(deviceLogService.page(source, keyword, startDate, endDate, page, pageSize));
}
@GetMapping("/content")
@Operation(summary = "查看日志尾部内容", description = "默认取最后 256KB(自最早行边界起);truncated=true 时可用更大 maxBytes 再取。")
public ApiResponse<DeviceLogContentVo> content(
HttpServletRequest request,
@RequestParam Long fileId,
@Parameter(description = "期望返回的明文字节数(16KB ~ 8MB)") @RequestParam(required = false) Long maxBytes) {
requireSuperAdmin(request);
return ApiResponse.success(deviceLogService.readTail(fileId, maxBytes));
}
@GetMapping("/download")
@Operation(summary = "下载完整日志(按偏移拼接解压)")
public void download(HttpServletRequest request, HttpServletResponse response,
@RequestParam Long fileId) throws IOException {
requireSuperAdmin(request);
DeviceLogFileEntity row = deviceLogService.requireFile(fileId);
String downloadName = row.getFileName().replace('/', '_').replace('\\', '_');
response.setContentType("text/plain;charset=UTF-8");
response.setHeader("Content-Disposition", "attachment; filename*=UTF-8''"
+ URLEncoder.encode(downloadName, StandardCharsets.UTF_8).replace("+", "%20"));
deviceLogService.streamDownload(fileId, response.getOutputStream());
}
@DeleteMapping("/{id}")
@Operation(summary = "删除日志文件(片段与元数据,不可恢复)")
public ApiResponse<Map<String, Object>> delete(HttpServletRequest request,
@PathVariable Long id) {
requireSuperAdmin(request);
int deletedParts = deviceLogService.deleteFile(id);
return ApiResponse.success("已删除", Map.of("deletedParts", deletedParts));
}
@GetMapping("/config")
@Operation(summary = "采集配置:全局默认 + 终端覆盖列表")
public ApiResponse<Map<String, Object>> config(HttpServletRequest request,
@RequestParam(required = false) String keyword) {
requireSuperAdmin(request);
Map<String, Object> data = new LinkedHashMap<>();
data.put("globalMode", deviceLogConfigService.globalMode());
List<DeviceLogConfigEntity> overrides = deviceLogConfigService.listOverrides(keyword);
data.put("overrides", overrides);
return ApiResponse.success(data);
}
@GetMapping("/devices")
@Operation(summary = "最近上报的终端列表(覆盖选择用)")
public ApiResponse<List<Map<String, Object>>> devices(HttpServletRequest request) {
requireSuperAdmin(request);
return ApiResponse.success(deviceLogService.recentDevices());
}
@PutMapping("/config/global")
@Operation(summary = "设置全局采集模式", description = "mode=full(全量)/ selected(精选)")
public ApiResponse<Map<String, Object>> updateGlobal(HttpServletRequest request,
@RequestParam String mode) {
requireSuperAdmin(request);
deviceLogConfigService.setGlobalMode(mode);
return ApiResponse.success(Map.of("globalMode", deviceLogConfigService.globalMode()));
}
@PutMapping("/config/device")
@Operation(summary = "设置/更新终端采集模式覆盖")
public ApiResponse<Map<String, Object>> updateDevice(HttpServletRequest request,
@RequestParam String source,
@RequestParam String deviceId,
@RequestParam(required = false) String deviceName,
@RequestParam String mode) {
requireSuperAdmin(request);
DeviceLogConfigEntity row = deviceLogConfigService.upsertOverride(source, deviceId, deviceName, mode);
return ApiResponse.success(Map.of("id", row.getId()));
}
@DeleteMapping("/config/device/{id}")
@Operation(summary = "删除终端覆盖(回落到全局默认)")
public ApiResponse<Boolean> deleteOverride(HttpServletRequest request, @PathVariable Long id) {
requireSuperAdmin(request);
return ApiResponse.success("已删除", deviceLogConfigService.deleteOverride(id));
}
/**
* 日志可能含账号/代理等敏感信息这里比常规后台更严仅超管requireAdmin 不含
*/
private void requireSuperAdmin(HttpServletRequest request) {
AdminUserEntity user = adminAuthSupport.requireAdmin(request);
if (!"super_admin".equals(adminAuthSupport.currentRole(user))) {
log.warn("[device-log] 非超管访问日志管理被拒 userId={} username={}",
user.getId(), user.getUsername());
throw new BusinessException(403, "仅超级管理员可访问日志管理");
}
}
}
@@ -0,0 +1,102 @@
package com.nanri.aiimage.modules.devicelog.controller;
import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.common.security.AdminAuthSupport;
import com.nanri.aiimage.modules.devicelog.service.DeviceLogConfigService;
import com.nanri.aiimage.modules.devicelog.service.DeviceLogService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.servlet.http.HttpServletRequest;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.format.annotation.DateTimeFormat;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;
import java.io.IOException;
import java.time.LocalDate;
import java.util.LinkedHashMap;
import java.util.Map;
/**
* 设备日志上报桌面客户端 / 麦象采集机增量片段上传进度对齐采集配置拉取
* 仅内部令牌X-Internal-Token可调
*/
@RestController
@RequiredArgsConstructor
@Slf4j
@RequestMapping("/api/internal/device-logs")
@Tag(name = "内部接口", description = "服务间调用(X-Internal-Token 鉴权)。")
public class InternalDeviceLogController {
private final DeviceLogService deviceLogService;
private final DeviceLogConfigService deviceLogConfigService;
private final AdminAuthSupport adminAuthSupport;
@PostMapping("/upload")
@Operation(summary = "上报日志增量片段",
description = "multipart:元数据字段 + filegzip 片段)。offset 必须等于服务端已收字节数;"
+ "重复片段幂等跳过(skipped=true);偏移不连续返回 code=409,调用方应从 uploadedBytes 重读。")
public ApiResponse<Map<String, Object>> upload(
HttpServletRequest request,
@RequestParam("source") String source,
@RequestParam("deviceId") String deviceId,
@RequestParam(value = "deviceName", required = false) String deviceName,
@RequestParam(value = "uid", required = false) Long uid,
@RequestParam("fileName") String fileName,
@RequestParam("logDate") @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate logDate,
@RequestParam("offset") long offset,
@RequestParam("plainBytes") long plainBytes,
@RequestParam("file") MultipartFile file) throws IOException {
requireInternal(request, "日志上报");
if (file == null || file.isEmpty()) {
return ApiResponse.fail("file 片段为空");
}
DeviceLogService.PartResult result = deviceLogService.recordPart(source, deviceId, deviceName, uid,
fileName, logDate, offset, plainBytes, file.getBytes());
Map<String, Object> data = new LinkedHashMap<>();
data.put("uploadedBytes", result.uploadedBytes());
data.put("partCount", result.partCount());
data.put("accepted", result.accepted());
data.put("skipped", result.skipped());
return ApiResponse.success(data);
}
@GetMapping("/state")
@Operation(summary = "查询某文件服务端已收进度", description = "客户端本地进度丢失/被拒后从此对齐。")
public ApiResponse<Map<String, Object>> state(
HttpServletRequest request,
@RequestParam("source") String source,
@RequestParam("deviceId") String deviceId,
@RequestParam("fileName") String fileName,
@RequestParam("logDate") @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate logDate) {
requireInternal(request, "进度查询");
return ApiResponse.success(deviceLogService.state(source, deviceId, fileName, logDate));
}
@GetMapping("/config")
@Operation(summary = "拉取生效的采集配置", description = "终端覆盖 > 全局默认;返回 mode 与精选模式排除清单(glob)。")
public ApiResponse<Map<String, Object>> config(
HttpServletRequest request,
@RequestParam("source") String source,
@RequestParam("deviceId") String deviceId) {
requireInternal(request, "配置拉取");
DeviceLogConfigService.EffectiveConfig config = deviceLogConfigService.resolve(source, deviceId);
Map<String, Object> data = new LinkedHashMap<>();
data.put("mode", config.mode());
data.put("exclude", config.exclude());
return ApiResponse.success(data);
}
private void requireInternal(HttpServletRequest request, String scene) {
if (!adminAuthSupport.isTrustedInternalToken(request)) {
log.warn("[device-log] 拒绝未携带可信内部令牌的{}请求 remoteAddr={}", scene, request.getRemoteAddr());
throw new BusinessException(401, "未授权");
}
}
}
@@ -0,0 +1,9 @@
package com.nanri.aiimage.modules.devicelog.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.nanri.aiimage.modules.devicelog.model.entity.DeviceLogConfigEntity;
import org.apache.ibatis.annotations.Mapper;
@Mapper
public interface DeviceLogConfigMapper extends BaseMapper<DeviceLogConfigEntity> {
}
@@ -0,0 +1,26 @@
package com.nanri.aiimage.modules.devicelog.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.nanri.aiimage.modules.devicelog.model.entity.DeviceLogFileEntity;
import org.apache.ibatis.annotations.Delete;
import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param;
import java.time.LocalDate;
@Mapper
public interface DeviceLogFileMapper extends BaseMapper<DeviceLogFileEntity> {
/**
* 按日志日期分批删除保留期外的元数据行
*
* <p>只删 {@code log_date} 早于 cutoff 的行这条线正好是查询侧可见窗口的边界
* "早已看不见、只剩占位"的行删掉不影响任何读取路径
*/
@Delete("""
DELETE FROM device_log_file
WHERE log_date < #{cutoff}
LIMIT #{batchSize}
""")
int deleteOlderThanBatch(@Param("cutoff") LocalDate cutoff, @Param("batchSize") int batchSize);
}
@@ -0,0 +1,32 @@
package com.nanri.aiimage.modules.devicelog.model.entity;
import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableId;
import com.baomidou.mybatisplus.annotation.TableName;
import lombok.Data;
import java.time.LocalDateTime;
/**
* 日志采集配置scope=global 为全局默认source/device_id 存空串占位
* scope=device 为终端级覆盖按来源+设备精确命中优先于全局
*/
@Data
@TableName("device_log_config")
public class DeviceLogConfigEntity {
@TableId(type = IdType.AUTO)
private Long id;
/** global / device。 */
private String scope;
/** 对应 device_log_file.sourceglobal 行存空串)。 */
private String source;
/** 对应 device_log_file.device_idglobal 行存空串)。 */
private String deviceId;
/** 覆盖行记录的设备展示名(列表展示用)。 */
private String deviceName;
/** full(全量)/ selected(精选)。 */
private String mode;
private LocalDateTime createdAt;
private LocalDateTime updatedAt;
}
@@ -0,0 +1,37 @@
package com.nanri.aiimage.modules.devicelog.model.entity;
import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableId;
import com.baomidou.mybatisplus.annotation.TableName;
import lombok.Data;
import java.time.LocalDate;
import java.time.LocalDateTime;
/**
* 设备日志文件元数据对象内容存独立 MinIO本表只存索引与进度
* 一个来源 + 设备 + 文件名 + 日志日期一行uploadedBytes/partCount 随增量上报推进
*/
@Data
@TableName("device_log_file")
public class DeviceLogFileEntity {
@TableId(type = IdType.AUTO)
private Long id;
/** 来源:client(桌面客户端)/ maixiang(麦象采集机)。 */
private String source;
private String deviceId;
/** 展示名:客户端登录用户名或机器名。 */
private String deviceName;
/** 桌面客户端当前登录用户 idusers.id),maixiang 上报为空。 */
private Long uid;
/** 日志文件名(客户端可能含子目录,如 API/2026_09_15.log)。 */
private String fileName;
private LocalDate logDate;
/** 已上传的明文字节数(客户端增量断点由此对齐)。 */
private Long uploadedBytes;
private Integer partCount;
private LocalDateTime lastUploadAt;
private LocalDateTime createdAt;
private LocalDateTime updatedAt;
}
@@ -0,0 +1,19 @@
package com.nanri.aiimage.modules.devicelog.model.vo;
import lombok.Data;
/** 日志内容(按尾部截取)。 */
@Data
public class DeviceLogContentVo {
private Long fileId;
private String fileName;
/** 已解压的日志文本(自最早行边界起,保证不截出半行)。 */
private String content;
/** 服务端已收到的日志总字节数。 */
private long totalBytes;
/** 本次实际返回的字节数。 */
private long shownBytes;
/** true=内容被截断(更早的历史未返回,可加大 maxBytes 再取)。 */
private boolean truncated;
}
@@ -0,0 +1,25 @@
package com.nanri.aiimage.modules.devicelog.model.vo;
import lombok.Data;
import java.time.LocalDate;
import java.time.LocalDateTime;
/** 日志文件列表行。 */
@Data
public class DeviceLogFileVo {
private Long id;
private String source;
private String deviceId;
private String deviceName;
/** 关联用户展示名(users.username;解析不到时为空)。 */
private String username;
private Long uid;
private String fileName;
private LocalDate logDate;
private Long uploadedBytes;
private Integer partCount;
private LocalDateTime lastUploadAt;
private LocalDateTime createdAt;
}
@@ -0,0 +1,17 @@
package com.nanri.aiimage.modules.devicelog.model.vo;
import lombok.Data;
import java.util.List;
/** 日志文件分页结果。 */
@Data
public class DeviceLogPageVo {
private List<DeviceLogFileVo> items;
private long total;
private long page;
private long pageSize;
/** 服务端保留天数(前端提示「日志仅保留 N 天」)。 */
private int retentionDays;
}
@@ -0,0 +1,169 @@
package com.nanri.aiimage.modules.devicelog.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.modules.devicelog.mapper.DeviceLogConfigMapper;
import com.nanri.aiimage.modules.devicelog.model.entity.DeviceLogConfigEntity;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
/**
* 日志采集配置全局默认 + 终端覆盖超管在后台日志管理调整
*
* <p>上报端桌面客户端 / 麦象定期拉取生效配置全量上传目录内全部日志
* 精选模式只上传关键日志排除清单见 {@link #selectedExcludes}随配置一起下发
* 调整清单无需发客户端版本
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class DeviceLogConfigService {
public static final String MODE_FULL = "full";
public static final String MODE_SELECTED = "selected";
/** 精选模式排除清单(glob,按来源;相对日志目录的文件名)。 */
private static final Map<String, List<String>> SELECTED_EXCLUDES = Map.of(
"client", List.of("pywebview.log"),
"maixiang", List.of("kk-browser.log*", "*_console.log", "test*.log"));
private final DeviceLogConfigMapper deviceLogConfigMapper;
/** 生效配置(终端覆盖 > 全局默认 > 兜底全量)。 */
public record EffectiveConfig(String mode, List<String> exclude) {
}
public EffectiveConfig resolve(String source, String deviceId) {
DeviceLogConfigEntity override = deviceLogConfigMapper.selectOne(
new LambdaQueryWrapper<DeviceLogConfigEntity>()
.eq(DeviceLogConfigEntity::getScope, "device")
.eq(DeviceLogConfigEntity::getSource, source)
.eq(DeviceLogConfigEntity::getDeviceId, deviceId)
.last("limit 1"));
String mode = override != null ? override.getMode() : globalMode();
return new EffectiveConfig(mode, selectedExcludes(mode, source));
}
public String globalMode() {
DeviceLogConfigEntity global = findGlobal();
return global == null ? MODE_FULL : global.getMode();
}
public List<String> selectedExcludes(String mode, String source) {
if (!MODE_SELECTED.equals(mode)) {
return List.of();
}
List<String> excludes = SELECTED_EXCLUDES.get(source);
if (excludes == null) {
log.warn("[device-log] 来源 {} 无精选排除清单,精选模式将等价全量", source);
return List.of();
}
return new ArrayList<>(excludes);
}
public void setGlobalMode(String mode) {
String safeMode = normalizeMode(mode);
DeviceLogConfigEntity global = findGlobal();
if (global == null) {
global = new DeviceLogConfigEntity();
global.setScope("global");
global.setSource("");
global.setDeviceId("");
global.setMode(safeMode);
deviceLogConfigMapper.insert(global);
log.info("[device-log] 全局采集模式初始化 mode={}", safeMode);
return;
}
if (safeMode.equals(global.getMode())) {
return;
}
deviceLogConfigMapper.updateById(withMode(global, safeMode));
log.info("[device-log] 全局采集模式更新 {} → {}", global.getMode(), safeMode);
}
public List<DeviceLogConfigEntity> listOverrides(String keyword) {
LambdaQueryWrapper<DeviceLogConfigEntity> qw = new LambdaQueryWrapper<DeviceLogConfigEntity>()
.eq(DeviceLogConfigEntity::getScope, "device");
if (keyword != null && !keyword.isBlank()) {
String kw = keyword.trim();
qw.and(w -> w.like(DeviceLogConfigEntity::getDeviceId, kw)
.or().like(DeviceLogConfigEntity::getDeviceName, kw));
}
return deviceLogConfigMapper.selectList(qw
.orderByDesc(DeviceLogConfigEntity::getUpdatedAt)
.last("limit 500"));
}
public DeviceLogConfigEntity upsertOverride(String source, String deviceId, String deviceName, String mode) {
String safeSource = requireText(source, "source", 32);
String safeDeviceId = requireText(deviceId, "deviceId", 128);
String safeMode = normalizeMode(mode);
DeviceLogConfigEntity row = deviceLogConfigMapper.selectOne(
new LambdaQueryWrapper<DeviceLogConfigEntity>()
.eq(DeviceLogConfigEntity::getScope, "device")
.eq(DeviceLogConfigEntity::getSource, safeSource)
.eq(DeviceLogConfigEntity::getDeviceId, safeDeviceId)
.last("limit 1"));
if (row == null) {
row = new DeviceLogConfigEntity();
row.setScope("device");
row.setSource(safeSource);
row.setDeviceId(safeDeviceId);
row.setDeviceName(deviceName);
row.setMode(safeMode);
deviceLogConfigMapper.insert(row);
log.info("[device-log] 新增终端覆盖 source={} device={} mode={}", safeSource, safeDeviceId, safeMode);
return row;
}
DeviceLogConfigEntity update = withMode(row, safeMode);
if (deviceName != null && !deviceName.isBlank()) {
update.setDeviceName(deviceName);
}
deviceLogConfigMapper.updateById(update);
log.info("[device-log] 终端覆盖更新 source={} device={} mode={}", safeSource, safeDeviceId, safeMode);
return update;
}
public boolean deleteOverride(Long id) {
if (id == null) {
throw new BusinessException(400, "id 不能为空");
}
int deleted = deviceLogConfigMapper.deleteById(id);
log.info("[device-log] 终端覆盖删除 id={} deleted={}", id, deleted);
return deleted > 0;
}
private DeviceLogConfigEntity findGlobal() {
return deviceLogConfigMapper.selectOne(new LambdaQueryWrapper<DeviceLogConfigEntity>()
.eq(DeviceLogConfigEntity::getScope, "global")
.last("limit 1"));
}
private static DeviceLogConfigEntity withMode(DeviceLogConfigEntity row, String mode) {
DeviceLogConfigEntity update = new DeviceLogConfigEntity();
update.setId(row.getId());
update.setMode(mode);
return update;
}
private static String normalizeMode(String mode) {
String trimmed = mode == null ? "" : mode.trim().toLowerCase();
if (!MODE_FULL.equals(trimmed) && !MODE_SELECTED.equals(trimmed)) {
throw new BusinessException(400, "mode 只支持 full / selected");
}
return trimmed;
}
private static String requireText(String value, String field, int maxLength) {
String trimmed = value == null ? "" : value.trim();
if (trimmed.isEmpty() || trimmed.length() > maxLength) {
throw new BusinessException(400, field + " 非法");
}
return trimmed;
}
}
@@ -0,0 +1,77 @@
package com.nanri.aiimage.modules.devicelog.service;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.DeviceLogOssProperties;
import com.nanri.aiimage.modules.devicelog.mapper.DeviceLogFileMapper;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.time.Duration;
import java.time.LocalDate;
/**
* 设备日志元数据的保留期清理
*
* <p>对象侧一直有生命周期主机B 独立 MinIO 由运维 mc 定时任务按 7 天回收
* {@code device_log_file} 的元数据行此前**只增不删**客户端每 60 秒上报一轮
* 每台设备每天登记若干来源+设备+文件名+日期长期运行会无限累积
*
* <p>查询侧本就只展示保留期内的行{@link DeviceLogService#page} 用同一个
* {@code retentionDays} 过滤这里按完全相同的边界删掉早已不可见的行
* 保留天数直接复用 {@code aiimage.device-log-oss.retention-days}
* 与对象侧共用同一个配置项避免两边各写一份天数后悄悄漂移
*
* <p>对象本身仍由桶生命周期负责回收本任务不碰对象两条时间线按对象修改时间
* 日志日期衡量可能有几天错位但对象最终仍会被桶规则删除不会永久残留
*
* <p>双节点用 job 锁保证单实例执行分批删除并限制单轮批次数避免一次跑太久占住锁
*/
@Service
@RequiredArgsConstructor
@Slf4j
public class DeviceLogRetentionService {
/** 单轮最多删除的批次数(每批 batchSize 行),剩余留给下一轮。 */
static final int MAX_BATCHES_PER_RUN = 20;
private final DeviceLogFileMapper deviceLogFileMapper;
private final DeviceLogOssProperties properties;
private final DistributedJobLockService distributedJobLockService;
@Value("${aiimage.device-log.retention-batch-size:2000}")
private int retentionBatchSize = 2000;
@Scheduled(cron = "${aiimage.device-log.retention-cron:0 40 4 * * *}")
public void purgeExpiredMetadata() {
int days = properties.retentionDaysOrDefault();
int batchSize = Math.max(100, retentionBatchSize);
// 与查询侧可见窗口同一条边界page() 只展示 log_date >= 今天-(days-1) 的行
LocalDate cutoff = LocalDate.now().minusDays(days - 1L);
DistributedJobLockService.LockHandle lockHandle =
distributedJobLockService.tryLock("device-log:metadata-retention", Duration.ofMinutes(15));
if (lockHandle == null) {
log.info("[device-log] 元数据保留清理跳过:另一实例持锁");
return;
}
try (lockHandle) {
int totalDeleted = 0;
int batches = 0;
while (batches < MAX_BATCHES_PER_RUN) {
int deleted = deviceLogFileMapper.deleteOlderThanBatch(cutoff, batchSize);
batches++;
totalDeleted += deleted;
if (deleted < batchSize) {
break;
}
}
log.info("[device-log] 元数据保留清理完成 cutoff={} retentionDays={} deleted={} batches={}",
cutoff, days, totalDeleted, batches);
} catch (Exception ex) {
log.warn("[device-log] 元数据保留清理失败 cutoff={} msg={}", cutoff, ex.getMessage(), ex);
}
}
}
@@ -0,0 +1,458 @@
package com.nanri.aiimage.modules.devicelog.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.common.mapper.AdminUserMapper;
import com.nanri.aiimage.common.model.entity.AdminUserEntity;
import com.nanri.aiimage.config.DeviceLogOssProperties;
import com.nanri.aiimage.modules.devicelog.mapper.DeviceLogFileMapper;
import com.nanri.aiimage.modules.devicelog.model.entity.DeviceLogFileEntity;
import com.nanri.aiimage.modules.devicelog.model.vo.DeviceLogContentVo;
import com.nanri.aiimage.modules.devicelog.model.vo.DeviceLogFileVo;
import com.nanri.aiimage.modules.devicelog.model.vo.DeviceLogPageVo;
import com.nanri.aiimage.modules.devicelog.storage.DeviceLogStorageService;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Deque;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.function.Function;
import java.util.regex.Pattern;
import java.util.zip.GZIPInputStream;
/**
* 设备日志增量片段接收桌面客户端 / 麦象采集机上报与后台查询
*
* <p>存储模型一个来源+设备+文件名+日期一行元数据内容以 gzip 片段对象按
* 起始偏移命名存独立 MinIOdevice-logs//{offset}.log.gz客户端按本地
* uploadedBytes 断点续传服务端条件推进偏移查看/下载时按偏移顺序拼接解压
* 片段 key 带偏移而非序号重复上报与乱序重试都会覆盖同一对象天然幂等
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class DeviceLogService {
/** 单片段明文上限(客户端按 2MB 切片,此处留防线余量)。 */
private static final long MAX_PART_PLAIN_BYTES = 8L * 1024 * 1024;
private static final long MAX_PART_GZIP_BYTES = 8L * 1024 * 1024;
private static final long DEFAULT_TAIL_BYTES = 256L * 1024;
private static final long MIN_TAIL_BYTES = 16L * 1024;
private static final long MAX_TAIL_BYTES = 8L * 1024 * 1024;
private static final long MAX_PAGE_SIZE = 100L;
/** 来源白名单形态:小写字母开头,长度 ≤32(client / maixiang / 未来新来源)。 */
private static final Pattern SOURCE_PATTERN = Pattern.compile("^[a-z][a-z0-9_-]{0,31}$");
/** 对象 key 段落清洗:路径分隔符、通配符、控制字符一律换成下划线。 */
private static final Pattern UNSAFE_SEGMENT = Pattern.compile("[\\\\/:*?\"<>|\\x00-\\x1F]+");
private final DeviceLogFileMapper deviceLogFileMapper;
private final AdminUserMapper adminUserMapper;
private final DeviceLogStorageService storage;
private final DeviceLogOssProperties properties;
/** 上报处理结果。 */
public record PartResult(long uploadedBytes, int partCount, boolean accepted, boolean skipped) {
}
// ---------------------------------------------------------------- 上报
/**
* 接收一个增量片段
*
* @param offset 客户端认为的已上传明文偏移必须等于服务端记录值才能追加
* @param plainBytes 本片段解压后的明文字节数客户端告知服务端只存 gzip 不解压
*/
public PartResult recordPart(String source, String deviceId, String deviceName, Long uid,
String fileName, LocalDate logDate, long offset, long plainBytes,
byte[] gzipBytes) {
String safeSource = normalizeSource(source);
String safeDeviceId = requireSegment(deviceId, "deviceId", 128);
String safeFileName = requireSegment(fileName, "fileName", 255);
if (logDate == null) {
throw new BusinessException(400, "logDate 不能为空");
}
if (offset < 0) {
throw new BusinessException(400, "offset 非法");
}
if (plainBytes <= 0 || plainBytes > MAX_PART_PLAIN_BYTES) {
throw new BusinessException(400, "plainBytes 非法(1 ~ " + MAX_PART_PLAIN_BYTES + "");
}
if (gzipBytes == null || gzipBytes.length == 0 || gzipBytes.length > MAX_PART_GZIP_BYTES) {
throw new BusinessException(400, "片段内容为空或超过上限");
}
if (!storage.enabled()) {
log.error("[device-log] 上报被拒:对象存储未就绪 source={} device={} file={}", safeSource, safeDeviceId, safeFileName);
throw new BusinessException(503, "日志存储未就绪,请稍后重试");
}
DeviceLogFileEntity row = findOrCreate(safeSource, safeDeviceId, safeFileName, logDate, deviceName, uid);
long current = row.getUploadedBytes() == null ? 0L : row.getUploadedBytes();
if (offset < current) {
// 重试/重复上报内容已收过幂等跳过返回服务端权威进度供客户端对齐
log.info("[device-log] 片段重复,幂等跳过 source={} device={} file={} date={} offset={} current={}",
safeSource, safeDeviceId, safeFileName, logDate, offset, current);
return new PartResult(current, nvl(row.getPartCount()), false, true);
}
if (offset > current) {
// 出现空洞客户端本地进度领先于服务端拒绝让客户端从服务端进度重读
log.warn("[device-log] 片段偏移不连续 source={} device={} file={} date={} offset={} current={}",
safeSource, safeDeviceId, safeFileName, logDate, offset, current);
throw new BusinessException(409, "偏移不连续,请从 uploadedBytes=" + current + " 重新读取");
}
String objectKey = objectKeyPrefix(safeSource, safeDeviceId, logDate, safeFileName)
+ String.format("%012d.log.gz", offset);
storage.putPart(objectKey, gzipBytes);
// 条件推进uploaded_bytes 与读取时一致才更新双节点并发时另一方以 0 行影响放弃以库中值为准
int updated = deviceLogFileMapper.update(null, new LambdaUpdateWrapper<DeviceLogFileEntity>()
.eq(DeviceLogFileEntity::getId, row.getId())
.eq(DeviceLogFileEntity::getUploadedBytes, current)
.set(DeviceLogFileEntity::getUploadedBytes, offset + plainBytes)
.setSql("part_count = part_count + 1")
.set(DeviceLogFileEntity::getLastUploadAt, LocalDateTime.now())
.set(deviceName != null && !deviceName.isBlank(), DeviceLogFileEntity::getDeviceName, deviceName)
.set(uid != null, DeviceLogFileEntity::getUid, uid));
if (updated <= 0) {
DeviceLogFileEntity latest = deviceLogFileMapper.selectById(row.getId());
long latestBytes = latest == null || latest.getUploadedBytes() == null ? current : latest.getUploadedBytes();
log.warn("[device-log] 并发推进冲突,以库中值为准 id={} offset={} 库中={}", row.getId(), offset, latestBytes);
return new PartResult(latestBytes, latest == null ? 0 : nvl(latest.getPartCount()), false, true);
}
long after = offset + plainBytes;
log.info("[device-log] 已收片段 source={} device={} file={} date={} offset={} +{}B → {}B key={}",
safeSource, safeDeviceId, safeFileName, logDate, offset, plainBytes, after, objectKey);
return new PartResult(after, nvl(row.getPartCount()) + 1, true, false);
}
/** 客户端进度对齐:返回服务端已持有的偏移与片段数。 */
public Map<String, Object> state(String source, String deviceId, String fileName, LocalDate logDate) {
String safeSource = normalizeSource(source);
DeviceLogFileEntity row = find(safeSource, requireSegment(deviceId, "deviceId", 128),
requireSegment(fileName, "fileName", 255), logDate);
return Map.of(
"exists", row != null,
"uploadedBytes", row == null || row.getUploadedBytes() == null ? 0L : row.getUploadedBytes(),
"parts", row == null ? 0 : nvl(row.getPartCount()));
}
// ---------------------------------------------------------------- 查询
public DeviceLogPageVo page(String source, String keyword, LocalDate startDate, LocalDate endDate,
Long pageParam, Long pageSizeParam) {
int retentionDays = properties.retentionDaysOrDefault();
LocalDate minDate = LocalDate.now().minusDays(retentionDays - 1L);
LocalDate from = startDate == null || startDate.isBefore(minDate) ? minDate : startDate;
long safePage = pageParam == null || pageParam < 1 ? 1L : pageParam;
long safeSize = pageSizeParam == null || pageSizeParam < 1
? 20L : Math.min(pageSizeParam, MAX_PAGE_SIZE);
Function<Boolean, LambdaQueryWrapper<DeviceLogFileEntity>> wrapperBuilder = countOnly -> {
LambdaQueryWrapper<DeviceLogFileEntity> qw = new LambdaQueryWrapper<>();
if (source != null && !source.isBlank()) {
qw.eq(DeviceLogFileEntity::getSource, source.trim());
}
if (keyword != null && !keyword.isBlank()) {
String kw = keyword.trim();
qw.and(w -> w.like(DeviceLogFileEntity::getDeviceName, kw)
.or().like(DeviceLogFileEntity::getDeviceId, kw)
.or().like(DeviceLogFileEntity::getFileName, kw));
}
qw.ge(DeviceLogFileEntity::getLogDate, from);
if (endDate != null) {
qw.le(DeviceLogFileEntity::getLogDate, endDate);
}
return qw;
};
Long totalValue = deviceLogFileMapper.selectCount(wrapperBuilder.apply(true));
long total = totalValue == null ? 0L : totalValue;
long offset = Math.max(0L, (safePage - 1) * safeSize);
List<DeviceLogFileEntity> rows = total == 0 ? List.of()
: deviceLogFileMapper.selectList(wrapperBuilder.apply(false)
.orderByDesc(DeviceLogFileEntity::getLastUploadAt)
.orderByDesc(DeviceLogFileEntity::getId)
.last("limit " + offset + "," + safeSize));
Map<Long, String> usernameOf = resolveUsernames(rows);
List<DeviceLogFileVo> items = new ArrayList<>(rows.size());
for (DeviceLogFileEntity row : rows) {
items.add(toVo(row, usernameOf));
}
DeviceLogPageVo vo = new DeviceLogPageVo();
vo.setItems(items);
vo.setTotal(total);
vo.setPage(safePage);
vo.setPageSize(safeSize);
vo.setRetentionDays(retentionDays);
log.info("[device-log] 列表查询 source={} keyword={} 起={} 止={} page={} size={} 命中={}",
source, keyword, from, endDate, safePage, safeSize, total);
return vo;
}
/** 最近上报过的终端(来源+设备去重,供后台配置终端覆盖时选择)。 */
public List<Map<String, Object>> recentDevices() {
LocalDate minDate = LocalDate.now().minusDays(properties.retentionDaysOrDefault() - 1L);
return deviceLogFileMapper.selectMaps(new QueryWrapper<DeviceLogFileEntity>()
.select("source",
"device_id AS deviceId",
"MAX(device_name) AS deviceName",
"MAX(last_upload_at) AS lastUploadAt")
.ge("log_date", minDate)
.groupBy("source", "device_id")
.orderByDesc("lastUploadAt")
.last("limit 200"));
}
/** 尾部内容:从最新片段往前读,尽量凑满 maxBytes(自最早行边界起截取,不出现半行)。 */
public DeviceLogContentVo readTail(Long fileId, Long maxBytesParam) {
DeviceLogFileEntity row = requireFile(fileId);
long maxBytes = maxBytesParam == null ? DEFAULT_TAIL_BYTES
: Math.max(MIN_TAIL_BYTES, Math.min(maxBytesParam, MAX_TAIL_BYTES));
String prefix = objectKeyPrefix(row);
List<String> partKeys = storage.listParts(prefix);
Deque<byte[]> chunks = new ArrayDeque<>();
long acc = 0;
int idx = partKeys.size() - 1;
for (; idx >= 0 && acc < maxBytes; idx--) {
byte[] plain;
try {
plain = gunzip(storage.readPartBytes(partKeys.get(idx)));
} catch (Exception ex) {
log.warn("[device-log] 片段读取/解压失败,跳过 key={} err={}", partKeys.get(idx), ex.getMessage());
continue;
}
chunks.addFirst(plain);
acc += plain.length;
}
boolean truncated = idx >= 0;
ByteArrayOutputStream merged = new ByteArrayOutputStream((int) Math.min(acc, Integer.MAX_VALUE));
for (byte[] chunk : chunks) {
merged.write(chunk, 0, chunk.length);
}
byte[] bytes = merged.toByteArray();
if (bytes.length > maxBytes) {
int cut = (int) (bytes.length - maxBytes);
int nl = indexOfNewline(bytes, cut);
// 从行边界开始截不留半行但若这样会切掉全部内容超长行退回按字节截
if (nl >= 0 && nl + 1 < bytes.length) {
cut = nl + 1;
}
bytes = Arrays.copyOfRange(bytes, cut, bytes.length);
truncated = true;
}
DeviceLogContentVo vo = new DeviceLogContentVo();
vo.setFileId(row.getId());
vo.setFileName(row.getFileName());
vo.setContent(new String(bytes, StandardCharsets.UTF_8));
vo.setTotalBytes(row.getUploadedBytes() == null ? 0L : row.getUploadedBytes());
vo.setShownBytes(bytes.length);
vo.setTruncated(truncated);
log.info("[device-log] 内容查看 id={} file={} 总大小={}B 返回={}B 截断={}",
row.getId(), row.getFileName(), vo.getTotalBytes(), bytes.length, truncated);
return vo;
}
/** 按偏移顺序流式拼接全部片段(解压后写响应,无整文件内存占用)。 */
public void streamDownload(Long fileId, OutputStream out) throws IOException {
DeviceLogFileEntity row = requireFile(fileId);
List<String> partKeys = storage.listParts(objectKeyPrefix(row));
if (partKeys.isEmpty()) {
throw new BusinessException(404, "该日志暂无内容");
}
for (String key : partKeys) {
try (InputStream raw = storage.openPartStream(key);
GZIPInputStream gz = new GZIPInputStream(raw)) {
gz.transferTo(out);
}
}
out.flush();
log.info("[device-log] 下载拼接完成 id={} file={} 片段数={}", row.getId(), row.getFileName(), partKeys.size());
}
/** 删除日志文件(片段对象 + 元数据行)。返回删除的对象数。 */
public int deleteFile(Long fileId) {
DeviceLogFileEntity row = requireFile(fileId);
List<String> partKeys = storage.listParts(objectKeyPrefix(row));
List<String> failed = storage.deleteParts(partKeys);
deviceLogFileMapper.deleteById(fileId);
log.info("[device-log] 删除日志 id={} file={} 片段总数={} 删除失败={}",
row.getId(), row.getFileName(), partKeys.size(), failed.size());
return partKeys.size() - failed.size();
}
// ---------------------------------------------------------------- 内部
/** 按 id 取日志文件行(不存在抛 404)。 */
public DeviceLogFileEntity requireFile(Long fileId) {
if (fileId == null) {
throw new BusinessException(400, "fileId 不能为空");
}
DeviceLogFileEntity row = deviceLogFileMapper.selectById(fileId);
if (row == null) {
throw new BusinessException(404, "日志文件不存在或已清理");
}
return row;
}
private DeviceLogFileEntity findOrCreate(String source, String deviceId, String fileName,
LocalDate logDate, String deviceName, Long uid) {
DeviceLogFileEntity row = find(source, deviceId, fileName, logDate);
if (row != null) {
return row;
}
DeviceLogFileEntity entity = new DeviceLogFileEntity();
entity.setSource(source);
entity.setDeviceId(deviceId);
entity.setFileName(fileName);
entity.setLogDate(logDate);
entity.setDeviceName(deviceName);
entity.setUid(uid);
entity.setUploadedBytes(0L);
entity.setPartCount(0);
try {
deviceLogFileMapper.insert(entity);
log.info("[device-log] 登记新日志文件 id={} source={} device={} file={} date={}",
entity.getId(), source, deviceId, fileName, logDate);
return entity;
} catch (Exception ex) {
// 双节点并发首传同一文件唯一键冲突后复用已有行
DeviceLogFileEntity existing = find(source, deviceId, fileName, logDate);
if (existing != null) {
log.info("[device-log] 并发登记同一文件,复用已有行 id={} file={}", existing.getId(), fileName);
return existing;
}
throw ex;
}
}
private DeviceLogFileEntity find(String source, String deviceId, String fileName, LocalDate logDate) {
if (logDate == null) {
return null;
}
return deviceLogFileMapper.selectOne(new LambdaQueryWrapper<DeviceLogFileEntity>()
.eq(DeviceLogFileEntity::getSource, source)
.eq(DeviceLogFileEntity::getDeviceId, deviceId)
.eq(DeviceLogFileEntity::getFileName, fileName)
.eq(DeviceLogFileEntity::getLogDate, logDate)
.last("limit 1"));
}
private String objectKeyPrefix(DeviceLogFileEntity row) {
return objectKeyPrefix(row.getSource(), row.getDeviceId(), row.getLogDate(), row.getFileName());
}
private String objectKeyPrefix(String source, String deviceId, LocalDate logDate, String fileName) {
return String.format("device-logs/%s/%s/%s/%s/",
safeSegment(source), safeSegment(deviceId), logDate, safeSegment(fileName));
}
private Map<Long, String> resolveUsernames(List<DeviceLogFileEntity> rows) {
List<Long> uids = rows.stream()
.map(DeviceLogFileEntity::getUid)
.filter(Objects::nonNull)
.distinct()
.toList();
if (uids.isEmpty()) {
return Map.of();
}
List<AdminUserEntity> users = adminUserMapper.selectBatchIds(uids);
Map<Long, String> map = new java.util.HashMap<>();
for (AdminUserEntity user : users) {
map.put(user.getId(), user.getUsername());
}
return map;
}
private DeviceLogFileVo toVo(DeviceLogFileEntity row, Map<Long, String> usernameOf) {
DeviceLogFileVo vo = new DeviceLogFileVo();
vo.setId(row.getId());
vo.setSource(row.getSource());
vo.setDeviceId(row.getDeviceId());
vo.setDeviceName(row.getDeviceName());
vo.setUid(row.getUid());
vo.setUsername(row.getUid() == null ? null : usernameOf.get(row.getUid()));
vo.setFileName(row.getFileName());
vo.setLogDate(row.getLogDate());
vo.setUploadedBytes(row.getUploadedBytes());
vo.setPartCount(row.getPartCount());
vo.setLastUploadAt(row.getLastUploadAt());
vo.setCreatedAt(row.getCreatedAt());
return vo;
}
private String normalizeSource(String source) {
String trimmed = source == null ? "" : source.trim();
if (!SOURCE_PATTERN.matcher(trimmed).matches()) {
log.warn("[device-log] 非法来源被拒 source={}", source);
throw new BusinessException(400, "source 非法(小写字母开头,数字/下划线/中划线,≤32)");
}
return trimmed;
}
private String requireSegment(String value, String field, int maxLength) {
String trimmed = value == null ? "" : value.trim();
if (trimmed.isEmpty()) {
throw new BusinessException(400, field + " 不能为空");
}
if (trimmed.length() > maxLength) {
throw new BusinessException(400, field + " 超长(>" + maxLength + "");
}
return trimmed;
}
/** 清洗对象 key 段落:路径分隔符等换成下划线,剔除「.」「..」防穿越。 */
private String safeSegment(String value) {
String cleaned = UNSAFE_SEGMENT.matcher(value == null ? "" : value.trim()).replaceAll("_");
if (cleaned.isEmpty() || cleaned.equals(".") || cleaned.equals("..")) {
return "_";
}
return cleaned;
}
private static byte[] gunzip(byte[] gz) throws IOException {
try (GZIPInputStream in = new GZIPInputStream(new ByteArrayInputStream(gz));
ByteArrayOutputStream out = new ByteArrayOutputStream(Math.max(64, gz.length * 4))) {
in.transferTo(out);
return out.toByteArray();
}
}
private static int indexOfNewline(byte[] bytes, int from) {
for (int i = Math.max(0, from); i < bytes.length; i++) {
if (bytes[i] == '\n') {
return i;
}
}
return -1;
}
private static int nvl(Integer value) {
return value == null ? 0 : value;
}
}
@@ -0,0 +1,177 @@
package com.nanri.aiimage.modules.devicelog.storage;
import com.nanri.aiimage.config.DeviceLogOssProperties;
import io.minio.BucketExistsArgs;
import io.minio.GetObjectArgs;
import io.minio.GetObjectResponse;
import io.minio.ListObjectsArgs;
import io.minio.MakeBucketArgs;
import io.minio.MinioClient;
import io.minio.PutObjectArgs;
import io.minio.RemoveObjectsArgs;
import io.minio.Result;
import io.minio.messages.DeleteError;
import io.minio.messages.DeleteObject;
import io.minio.messages.Item;
import jakarta.annotation.PostConstruct;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
/**
* 设备日志的独立对象存储客户端主机B 自建 MinIO非业务 OSS
*
* <p>只存 gzip 片段原文不做重压缩桶的 7 天过期规则在部署时由运维用 mc 配置
* 本类不负责生命周期管理
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class DeviceLogStorageService {
private final DeviceLogOssProperties properties;
private volatile MinioClient client;
@PostConstruct
void init() {
if (!properties.configured()) {
log.warn("[device-log] aiimage.device-log-oss 未配置(endpoint/凭据/桶),"
+ "日志上报与管理接口将不可用;生产必须通过 AIIMAGE_DEVICE_LOG_OSS_* 环境变量注入");
return;
}
try {
MinioClient built = MinioClient.builder()
.endpoint(properties.getEndpoint())
.credentials(properties.getAccessKeyId(), properties.getAccessKeySecret())
.build();
// 收紧超时日志接口不能被慢存储拖挂默认读超时 5 分钟
built.setTimeout(10_000, 60_000, 60_000);
boolean exists = built.bucketExists(BucketExistsArgs.builder()
.bucket(properties.getBucket()).build());
if (!exists) {
built.makeBucket(MakeBucketArgs.builder().bucket(properties.getBucket()).build());
log.info("[device-log] 已创建日志桶 bucket={}", properties.getBucket());
}
this.client = built;
log.info("[device-log] 日志对象存储已就绪 endpoint={} bucket={} 保留天数={}",
properties.getEndpoint(), properties.getBucket(), properties.retentionDaysOrDefault());
} catch (Exception ex) {
log.error("[device-log] 日志对象存储初始化失败 endpoint={} bucket={},日志功能不可用: {}",
properties.getEndpoint(), properties.getBucket(), ex.getMessage(), ex);
}
}
public boolean enabled() {
return client != null;
}
public String bucket() {
return properties.getBucket();
}
/** 写入一个 gzip 片段(同 key 覆盖写,幂等)。 */
public void putPart(String objectKey, byte[] gzipBytes) {
MinioClient c = requireClient();
try (ByteArrayInputStream in = new ByteArrayInputStream(gzipBytes)) {
c.putObject(PutObjectArgs.builder()
.bucket(bucket())
.object(objectKey)
.stream(in, gzipBytes.length, -1)
.contentType("application/gzip")
.build());
} catch (Exception ex) {
throw new IllegalStateException("写日志对象失败 key=" + objectKey + " err=" + ex.getMessage(), ex);
}
}
/** 列出前缀下全部对象 key(按 key 升序;key 内的 offset 为零填充,字典序即偏移序)。 */
public List<String> listParts(String prefix) {
MinioClient c = requireClient();
List<String> keys = new ArrayList<>();
try {
Iterable<Result<Item>> results = c.listObjects(ListObjectsArgs.builder()
.bucket(bucket())
.prefix(prefix)
.recursive(true)
.build());
for (Result<Item> result : results) {
keys.add(result.get().objectName());
}
} catch (Exception ex) {
throw new IllegalStateException("列日志对象失败 prefix=" + prefix + " err=" + ex.getMessage(), ex);
}
keys.sort(String::compareTo);
return keys;
}
/** 读取一个 gzip 片段的原始字节(未解压)。 */
public byte[] readPartBytes(String objectKey) {
MinioClient c = requireClient();
try (GetObjectResponse response = c.getObject(GetObjectArgs.builder()
.bucket(bucket()).object(objectKey).build())) {
return response.readAllBytes();
} catch (Exception ex) {
throw new IllegalStateException("读日志对象失败 key=" + objectKey + " err=" + ex.getMessage(), ex);
}
}
/** 打开一个 gzip 片段的流(调用方负责关闭;下载拼接时避免整段进内存)。 */
public InputStream openPartStream(String objectKey) {
MinioClient c = requireClient();
try {
return c.getObject(GetObjectArgs.builder()
.bucket(bucket()).object(objectKey).build());
} catch (Exception ex) {
throw new IllegalStateException("打开日志对象流失败 key=" + objectKey + " err=" + ex.getMessage(), ex);
}
}
/** 批量删除对象;返回删除失败的 key 列表。 */
public List<String> deleteParts(List<String> objectKeys) {
if (objectKeys == null || objectKeys.isEmpty()) {
return List.of();
}
MinioClient c = requireClient();
List<DeleteObject> targets = objectKeys.stream().map(DeleteObject::new).toList();
List<String> failed = new ArrayList<>();
try {
Iterable<Result<DeleteError>> results = c.removeObjects(RemoveObjectsArgs.builder()
.bucket(bucket()).objects(targets).build());
for (Result<DeleteError> result : results) {
DeleteError error = result.get();
failed.add(error.objectName());
log.warn("[device-log] 删除对象失败 key={} err={}", error.objectName(), error.message());
}
} catch (Exception ex) {
throw new IllegalStateException("批量删除日志对象失败 err=" + ex.getMessage(), ex);
}
return failed;
}
/** 探测连通性(上传接口的错误提示用)。 */
public boolean ping() {
if (client == null) {
return false;
}
try {
return client.bucketExists(BucketExistsArgs.builder().bucket(bucket()).build());
} catch (Exception ex) {
log.warn("[device-log] 存储连通性探测失败: {}", ex.getMessage());
return false;
}
}
private MinioClient requireClient() {
MinioClient c = client;
if (c == null) {
throw new IllegalStateException("日志对象存储未配置或初始化失败");
}
return c;
}
}
@@ -5,10 +5,14 @@ import io.micrometer.core.instrument.DistributionSummary;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import io.minio.GetObjectArgs;
import io.minio.ListObjectsArgs;
import io.minio.MinioClient;
import io.minio.PutObjectArgs;
import io.minio.RemoveObjectArgs;
import io.minio.Result;
import io.minio.StatObjectArgs;
import io.minio.errors.ErrorResponseException;
import io.minio.messages.Item;
import lombok.extern.slf4j.Slf4j;
import okhttp3.ConnectionPool;
import okhttp3.Dispatcher;
@@ -20,7 +24,11 @@ import org.springframework.stereotype.Service;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.UUID;
import java.util.concurrent.Semaphore;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.TimeUnit;
@@ -38,6 +46,17 @@ public class RustfsObjectStorageService {
private static final String OP_STAT = "stat";
private static final String OP_TOTAL = "total";
/** S3 确定性错误码:重试不会改变结果,应立即失败而不是白打两次请求。 */
private static final Set<String> NON_RETRYABLE_S3_CODES = Set.of(
"NoSuchKey", "NoSuchBucket", "NoSuchVersion",
"AccessDenied", "InvalidAccessKeyId", "SignatureDoesNotMatch", "InvalidBucketName");
/** 标准 UUID 字符串长度(8-4-4-4-12),用于识别版本化对象 key。 */
private static final int UUID_STRING_LENGTH = 36;
/** 兄弟对象兜底列出的上限:只为找回同槽位的版本化对象,不需要列全。 */
private static final int MAX_SIBLING_LIST_KEYS = 50;
private final TransientStorageProperties properties;
private final ObjectProvider<MeterRegistry> meterRegistryProvider;
private final ObjectProvider<RustfsDeleteRetryService> deleteRetryServiceProvider;
@@ -99,7 +118,22 @@ public class RustfsObjectStorageService {
return uploadBytes(objectKey, bytes, verifyAfterUpload);
}
/**
* 三参重载是否做上传失败补偿删除按对象 key 形态自动判定
*
* <p>只有版本化 key末段以 UUID 结尾是本次写入独占的确定性 key 会被重传重写复用
* 删它就可能删掉别的 DB 行仍在引用的对象2026-09-17 线上任务 28459 的载荷对象就是这么丢的
*/
public String uploadBytes(String objectKey, byte[] content, boolean verifyAfterUpload) {
return uploadBytes(objectKey, content, verifyAfterUpload, isVersionedObjectKey(objectKey));
}
/**
* @param compensateDeleteOnFailure put 已完成但后续可见性校验失败时是否把该对象排进删除补偿队列
* 仅当调用方能确认该对象不会被其它写入复用时才可传 true
*/
public String uploadBytes(String objectKey, byte[] content, boolean verifyAfterUpload,
boolean compensateDeleteOnFailure) {
long deadlineNanos = operationDeadlineNanos();
if (!isConfigured()) {
throw new IllegalStateException("transient storage is not configured");
@@ -129,13 +163,45 @@ public class RustfsObjectStorageService {
}
return uploadedObjectKey;
} catch (RuntimeException ex) {
if (putCompleted.get()) {
if (putCompleted.get() && compensateDeleteOnFailure) {
enqueueDeleteRetry(objectKey, ex);
} else if (putCompleted.get()) {
// 共享 key 会被重传重写此处删除可能删掉别的行正在引用的对象交给保留期清理兜底
// 线上任务 28459 chunk-462/473/484 就是被这条无条件删除队列删掉的
log.warn("[rustfs] 跳过上传失败补偿删除(对象非本次独占,可能被复用)objectKey={} err={}",
objectKey, ex.getMessage());
}
throw ex;
}
}
/**
* 对象 key 是否为本次写入独占的版本化 key末段去掉 {@code .json} 后缀 UUID 结尾
*
* <p>只看末段UUID 出现在中间段 scopeHash不代表该对象被独占解析失败一律按共享处理
* 保守宁可留下孤儿对象也不删掉可能仍被引用的对象
*/
static boolean isVersionedObjectKey(String objectKey) {
if (objectKey == null) {
return false;
}
String key = objectKey.trim();
if (key.endsWith(".json")) {
key = key.substring(0, key.length() - ".json".length());
}
int slash = key.lastIndexOf('/');
String lastSegment = slash < 0 ? key : key.substring(slash + 1);
if (lastSegment.length() < UUID_STRING_LENGTH) {
return false;
}
try {
UUID.fromString(lastSegment.substring(lastSegment.length() - UUID_STRING_LENGTH));
return true;
} catch (IllegalArgumentException ex) {
return false;
}
}
public String readObjectAsString(String objectKey) {
byte[] bytes = readObjectBytes(objectKey);
return new String(bytes, StandardCharsets.UTF_8);
@@ -175,6 +241,43 @@ public class RustfsObjectStorageService {
});
}
/**
* 列出前缀下的对象 key按最后修改时间倒序最新在前
*
* <p>只服务于指针指向的对象已不存在需要找回同槽位的版本化兄弟对象这一兜底路径
* 因此刻意不做重试不参与失败窗口记账列出失败直接抛错由调用方按原错误语义处理
*/
public List<String> listObjectKeysNewestFirst(String prefix, int limit) {
if (!isConfigured()) {
throw new IllegalStateException("transient storage is not configured");
}
int safeLimit = Math.max(1, Math.min(limit, MAX_SIBLING_LIST_KEYS));
long deadlineNanos = operationDeadlineNanos();
List<String[]> entries = new ArrayList<>();
try {
Iterable<Result<Item>> results = buildClient(deadlineNanos).listObjects(ListObjectsArgs.builder()
.bucket(properties.getBucket())
.prefix(prefix == null ? "" : prefix)
.recursive(true)
.maxKeys(safeLimit)
.build());
for (Result<Item> result : results) {
Item item = result.get();
entries.add(new String[]{item.objectName(),
item.lastModified() == null ? "" : item.lastModified().toString()});
}
} catch (Exception ex) {
throw new IllegalStateException("transient storage list failed prefix=" + prefix
+ " err=" + ex.getMessage(), ex);
}
entries.sort((left, right) -> right[1].compareTo(left[1]));
List<String> keys = new ArrayList<>(entries.size());
for (String[] entry : entries) {
keys.add(entry[0]);
}
return keys;
}
public void deleteObject(String objectKey) {
deleteObject(objectKey, true, operationDeadlineNanos());
}
@@ -238,6 +341,12 @@ public class RustfsObjectStorageService {
resetFailureWindow(operation);
}
recordOperation(operation, "success", elapsedNanos(startedAt));
if (attempt > 1) {
// 重试后成功必须留 INFO 结论线上一天上千条 "operation failed, retrying"
// 却没有任何结论日志无法判断这些上传最后到底落盘了没有
log.info("[rustfs] 重试后成功 operation={} objectKey={} attempt={}/{} durationMs={} bytes={}",
operation, objectKey, attempt, maxRetries, elapsedMillis(startedAt), bytes);
}
log.debug("[rustfs] operation success operation={} objectKey={} attempt={}/{} durationMs={} bytes={}",
operation, objectKey, attempt, maxRetries, elapsedMillis(startedAt), bytes);
return result;
@@ -245,6 +354,15 @@ public class RustfsObjectStorageService {
last = ex;
recordOperation(operation, attempt < maxRetries ? "retry" : "failure", elapsedNanos(startedAt));
recordFailure(operation, objectKey, ex);
if (isNonRetryable(ex)) {
// 确定性错误NoSuchKey / AccessDenied重试多少次结果都一样
// 线上 read 一个已被清理的 chunk 就会连打 3 次请求还被记成 ERROR
log.warn("[rustfs] 确定性错误,不重试 operation={} objectKey={} err={}",
operation, objectKey, ex.getMessage());
throw ex instanceof RuntimeException runtimeException
? runtimeException
: new IllegalStateException(ex);
}
if (attempt < maxRetries) {
delayMillis = retryDelayMillis(attempt);
log.warn("[rustfs] operation failed, retrying operation={} objectKey={} attempt={}/{} delayMs={} err={}",
@@ -262,6 +380,9 @@ public class RustfsObjectStorageService {
operation, objectKey, deadlineNanos);
}
}
// 重试耗尽 ERROR 结论 + 最后一次错误否则只看到一串 retrying无从判断是否真丢数据
log.error("[rustfs] 重试耗尽,最终失败 operation={} objectKey={} maxRetries={} 最后一次错误={}",
operation, objectKey, maxRetries, last == null ? "" : last.getMessage(), last);
throw new IllegalStateException("failed to " + operation + " payload in transient storage", last);
} finally {
if (totalAcquired) {
@@ -270,6 +391,27 @@ public class RustfsObjectStorageService {
}
}
/**
* 是否为重试也没用的确定性错误NoSuchKey / AccessDenied / 签名错误
*
* <p>只有网络类与 5xx 类失败才值得重试确定性错误重试多少次结果都一样
*/
private static boolean isNonRetryable(Throwable error) {
Throwable cursor = error;
while (cursor != null) {
if (cursor instanceof ErrorResponseException responseException) {
String code = responseException.errorResponse() == null
? null
: responseException.errorResponse().code();
if (code != null && NON_RETRYABLE_S3_CODES.contains(code)) {
return true;
}
}
cursor = cursor.getCause();
}
return false;
}
private void acquirePermit(String operation, String objectKey, Semaphore semaphore, long deadlineNanos) {
try {
checkDeadline(operation, objectKey, deadlineNanos);
@@ -336,6 +478,22 @@ public class RustfsObjectStorageService {
Math.max(1L, properties.getConnectionPoolKeepAliveMillis()),
TimeUnit.MILLISECONDS))
.retryOnConnectionFailure(true)
.addNetworkInterceptor(chain -> {
okhttp3.Request request = chain.request();
okhttp3.RequestBody body = request.body();
// OkHttp 长度为 0 的请求体不会写 Content-Length于是请求既无
// Content-Length 也无 Transfer-EncodingHTTP/1.1 不允许这样
// RustFS 对此直接回 411 Length Required客户端读到不完整响应就报
// unexpected end of stream进而重试线上抓包实测 90 秒内 515 411
// 而上传空内容content null/在业务里是常态
// network interceptor 在协议层补上该头普通 interceptor 会被
// BridgeInterceptor body 长度覆盖掉加了也不生效
if (body != null && body.contentLength() == 0L
&& request.header("Content-Length") == null) {
request = request.newBuilder().header("Content-Length", "0").build();
}
return chain.proceed(request);
})
.build();
}
return httpClient;
@@ -1,6 +1,8 @@
package com.nanri.aiimage.modules.invalidasin.model.entity;
import com.baomidou.mybatisplus.annotation.FieldStrategy;
import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableField;
import com.baomidou.mybatisplus.annotation.TableId;
import com.baomidou.mybatisplus.annotation.TableName;
import lombok.Data;
@@ -18,5 +20,12 @@ public class InvalidAsinDataEntity {
private Long groupId;
private String recordSource;
private LocalDateTime createdAt;
/**
* 更新时间由数据库维护DEFAULT CURRENT_TIMESTAMP + ON UPDATE CURRENT_TIMESTAMP
*
* <p>禁止应用显式写MySQL UPDATE 语句显式给该列赋值时不会触发自动更新
* 而本表写回为 selectById 改字段 updateById实体带着旧值一旦写回就会冻结更新时间
*/
@TableField(value = "updated_at", insertStrategy = FieldStrategy.NEVER, updateStrategy = FieldStrategy.NEVER)
private LocalDateTime updatedAt;
}
@@ -29,4 +29,7 @@ public class InvalidAsinDataItemVo {
@Schema(description = "创建时间")
private LocalDateTime createdAt;
@Schema(description = "更新时间")
private LocalDateTime updatedAt;
}
@@ -245,6 +245,7 @@ public class InvalidAsinDataService {
vo.setGroupName(groupName == null ? "" : groupName);
vo.setRecordSource(isManualRecord(entity) ? RECORD_SOURCE_MANUAL : RECORD_SOURCE_AUTO);
vo.setCreatedAt(entity.getCreatedAt());
vo.setUpdatedAt(entity.getUpdatedAt());
return vo;
}
}
@@ -44,23 +44,27 @@ public class MaixiangConsoleClient {
return hasText(properties.getPriceTrackApiUrl()) && hasText(properties.getMaixiangConsoleToken());
}
/** 批量任务列表(all_task)。createdBefore 非空时只取创建时间不晚于该时刻的任务(SQL 字符串比较)。 */
public BatchTaskPage batchTasks(int status, LocalDateTime createdBefore, int pageSize) {
/**
* 批量任务列表all_taskupdateTimeBefore 非空时只取最后更新不晚于该时刻的任务
* SQL 字符串比较 update_time 过滤而不是 create_timedesc(id) 分页下按创建时间过滤
* 只能看到最新创建的一批积压深处的老任务真正停滞的永远看不到 2026-09-15 滞留漏报的根因
*/
public BatchTaskPage batchTasks(int status, LocalDateTime updateTimeBefore, int pageSize) {
String url = buildUrl("/api/console/batch/tasks",
"status=" + status,
"page=1",
"page_size=" + pageSize,
createdBefore == null ? null : "end_time=" + TIME_FORMAT.format(createdBefore));
updateTimeBefore == null ? null : "update_time_end=" + TIME_FORMAT.format(updateTimeBefore));
return parseBatchTaskPage(fetch(url, "批量任务列表 status=" + status));
}
/** 单任务列表(task_record)。 */
public SingleTaskPage singleTasks(int status, LocalDateTime createdBefore, int pageSize) {
/** 单任务列表(task_record)。updateTimeBefore 语义同 {@link #batchTasks(int, LocalDateTime, int)}。 */
public SingleTaskPage singleTasks(int status, LocalDateTime updateTimeBefore, int pageSize) {
String url = buildUrl("/api/console/tasks",
"status=" + status,
"page=1",
"page_size=" + pageSize,
createdBefore == null ? null : "end_time=" + TIME_FORMAT.format(createdBefore));
updateTimeBefore == null ? null : "update_time_end=" + TIME_FORMAT.format(updateTimeBefore));
return parseSingleTaskPage(fetch(url, "单任务列表 status=" + status));
}
@@ -44,20 +44,24 @@ public class AdminNotificationController {
@GetMapping
@Operation(summary = "通知分页列表",
description = "onlyUnread=true 时只返回未读;keyword 模糊匹配标题/内容;startDate/endDate 为年月日闭区间。")
description = "onlyUnread=true 时只返回未读;keyword 模糊匹配标题/内容;"
+ "category 按类型大类筛选(system_error/task_error/config_error/system_notice);"
+ "startDate/endDate 为年月日闭区间。")
public ApiResponse<NotificationPageVo> page(
HttpServletRequest request,
@Parameter(description = "页码") @RequestParam(defaultValue = "1") Long page,
@Parameter(description = "每页数量") @RequestParam(defaultValue = "20") Long pageSize,
@Parameter(description = "只看未读") @RequestParam(required = false, defaultValue = "false") Boolean onlyUnread,
@Parameter(description = "关键字(标题/内容)") @RequestParam(required = false) String keyword,
@Parameter(description = "类型大类:system_error/task_error/config_error/system_notice")
@RequestParam(required = false) String category,
@Parameter(description = "起始日期(yyyy-MM-dd,含)")
@RequestParam(required = false) @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate startDate,
@Parameter(description = "结束日期(yyyy-MM-dd,含)")
@RequestParam(required = false) @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate endDate) {
Long userId = currentAdminId(request);
return ApiResponse.success(notificationService.page(userId, NotificationService.AUDIENCE_ADMIN,
NotificationPageQuery.of(page, pageSize, onlyUnread, keyword, startDate, endDate)));
NotificationPageQuery.of(page, pageSize, onlyUnread, keyword, category, startDate, endDate)));
}
@PostMapping("/{id}/read")
@@ -44,20 +44,24 @@ public class NotificationController {
@GetMapping
@Operation(summary = "通知分页列表",
description = "onlyUnread=true 时只返回未读;keyword 模糊匹配标题/内容;startDate/endDate 为年月日闭区间。")
description = "onlyUnread=true 时只返回未读;keyword 模糊匹配标题/内容;"
+ "category 按类型大类筛选(system_error/task_error/config_error/system_notice);"
+ "startDate/endDate 为年月日闭区间。")
public ApiResponse<NotificationPageVo> page(
HttpServletRequest request,
@Parameter(description = "页码") @RequestParam(defaultValue = "1") Long page,
@Parameter(description = "每页数量") @RequestParam(defaultValue = "20") Long pageSize,
@Parameter(description = "只看未读") @RequestParam(required = false, defaultValue = "false") Boolean onlyUnread,
@Parameter(description = "关键字(标题/内容)") @RequestParam(required = false) String keyword,
@Parameter(description = "类型大类:system_error/task_error/config_error/system_notice")
@RequestParam(required = false) String category,
@Parameter(description = "起始日期(yyyy-MM-dd,含)")
@RequestParam(required = false) @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate startDate,
@Parameter(description = "结束日期(yyyy-MM-dd,含)")
@RequestParam(required = false) @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate endDate) {
Long userId = currentUserId(request);
return ApiResponse.success(notificationService.page(userId, NotificationService.AUDIENCE_USER,
NotificationPageQuery.of(page, pageSize, onlyUnread, keyword, startDate, endDate)));
NotificationPageQuery.of(page, pageSize, onlyUnread, keyword, category, startDate, endDate)));
}
@PostMapping("/{id}/read")
@@ -5,8 +5,9 @@ import lombok.Data;
import java.time.LocalDate;
/**
* 通知列表查询条件分页 + 未读过滤 + 关键字标题/内容模糊+ 创建日期区间年月日闭区间
* 铃铛面板的按天搜索内容搜索都走这里空值表示不限制
* 通知列表查询条件分页 + 未读过滤 + 关键字标题/内容模糊+ 创建日期区间年月日闭区间
* + 类型大类system_error/task_error/config_error/system_notice
* 铃铛面板的按天搜索内容搜索类型筛选都走这里空值表示不限制
*/
@Data
public class NotificationPageQuery {
@@ -15,16 +16,19 @@ public class NotificationPageQuery {
private Long pageSize;
private Boolean onlyUnread;
private String keyword;
private String category;
private LocalDate startDate;
private LocalDate endDate;
public static NotificationPageQuery of(Long page, Long pageSize, Boolean onlyUnread,
String keyword, LocalDate startDate, LocalDate endDate) {
String keyword, String category,
LocalDate startDate, LocalDate endDate) {
NotificationPageQuery query = new NotificationPageQuery();
query.setPage(page);
query.setPageSize(pageSize);
query.setOnlyUnread(onlyUnread);
query.setKeyword(keyword);
query.setCategory(category);
query.setStartDate(startDate);
query.setEndDate(endDate);
return query;
@@ -11,6 +11,8 @@ public class NotificationItemVo {
private Long id;
/** 场景:secret_balance/secret_invalid/task_failed/service_down/system */
private String scene;
/** 类型大类(铃铛筛选用):system_error/task_error/config_error/system_notice */
private String category;
/** 级别:info/warning/error */
private String level;
private String title;
@@ -104,7 +104,7 @@ public class MaixiangAnomalyScanner {
}
}
/** 批量任务停滞:创建早于阈值、仍处 status=0/1 update_time 不再推进。 */
/** 批量任务停滞:最后更新早于阈值、仍处 status=0/1(服务端按 update_time 过滤后取回)。 */
void checkStuckBatchTasks(NotificationDispatchService.AdminAudience audience, LocalDateTime now) {
int thresholdMinutes = Math.max(1, properties.getMaixiangStuckMinutes());
LocalDateTime cutoff = now.minusMinutes(thresholdMinutes);
@@ -112,7 +112,7 @@ public class MaixiangAnomalyScanner {
for (int status : new int[]{0, 1}) {
candidates.addAll(consoleClient.batchTasks(status, cutoff, PAGE_SIZE).items());
}
// 创建时间早于 cutoff 但仍在正常推进的大任务update_time 晚于 cutoff不算停滞
// 服务端已按 update_time cutoff 过滤这里再兜一层防止接口参数被忽略时误报正常推进的大任务
List<BatchTaskItem> stuck = new ArrayList<>();
for (BatchTaskItem task : candidates) {
if (task.updateTime() != null && !task.updateTime().isAfter(cutoff)) {
@@ -135,7 +135,7 @@ public class MaixiangAnomalyScanner {
push(audience, "麦象批量任务停滞", content, "maixiang_stuck_batch:" + now.format(DAY_FORMAT));
}
/** 单任务滞留:创建早于阈值、仍处 status=0/1(跟价等单任务应秒级完成)。 */
/** 单任务滞留:最后更新早于阈值、仍处 status=0/1(跟价等单任务应秒级完成)。 */
void checkStuckSingleTasks(NotificationDispatchService.AdminAudience audience, LocalDateTime now) {
int thresholdMinutes = Math.max(1, properties.getMaixiangSingleStuckMinutes());
LocalDateTime cutoff = now.minusMinutes(thresholdMinutes);
@@ -273,15 +273,17 @@ public class NotificationScanScheduler {
"service_down:" + serviceKey + ":" + hour, null);
}
/** 已读通知保留期清理:每天最多一次。 */
/** 通知保留期清理:每天最多一次;已读与未读各按自己的保留天数。 */
private void cleanupExpiredIfNeeded() {
LocalDate today = LocalDate.now();
if (today.equals(lastCleanupDate)) {
return;
}
lastCleanupDate = today;
LocalDateTime cutoff = LocalDateTime.now().minusDays(Math.max(1, properties.getReadRetentionDays()));
notificationService.cleanupReadBefore(cutoff);
LocalDateTime readCutoff = LocalDateTime.now().minusDays(Math.max(1, properties.getReadRetentionDays()));
notificationService.cleanupReadBefore(readCutoff);
LocalDateTime unreadCutoff = LocalDateTime.now().minusDays(Math.max(1, properties.getUnreadRetentionDays()));
notificationService.cleanupUnreadBefore(unreadCutoff);
}
private String dedupeKeyOf(BucketKey key) {
@@ -15,6 +15,7 @@ import org.springframework.stereotype.Service;
import java.time.LocalDateTime;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
/**
* 站内通知存取桌面端用户audience=user与后台管理员audience=admin共用一张表
@@ -40,6 +41,27 @@ public class NotificationService {
public static final String SCENE_MAIXIANG_ANOMALY = "maixiang_anomaly";
public static final String SCENE_SYSTEM = "system";
/**
* 通知大类scene 太细技术语义铃铛按用户看得懂的四类归并筛选
* null/空分类表示不筛选 {@link #scenesOfCategory}
*/
public static final String CATEGORY_SYSTEM_ERROR = "system_error";
public static final String CATEGORY_TASK_ERROR = "task_error";
public static final String CATEGORY_CONFIG_ERROR = "config_error";
public static final String CATEGORY_SYSTEM_NOTICE = "system_notice";
/**
* scene 大类未知 scene历史遗留或新增未登记归入系统通知
* 保证任一通知只属于一类筛选项互斥且完备
*/
private static final Map<String, String> SCENE_TO_CATEGORY = Map.of(
SCENE_SERVICE_DOWN, CATEGORY_SYSTEM_ERROR,
SCENE_MAIXIANG_ANOMALY, CATEGORY_SYSTEM_ERROR,
SCENE_TASK_FAILED, CATEGORY_TASK_ERROR,
SCENE_SECRET_BALANCE, CATEGORY_CONFIG_ERROR,
SCENE_SECRET_INVALID, CATEGORY_CONFIG_ERROR,
SCENE_SYSTEM, CATEGORY_SYSTEM_NOTICE);
private static final long MAX_PAGE_SIZE = 100L;
private static final int TITLE_MAX_LENGTH = 128;
private static final int CONTENT_MAX_LENGTH = 512;
@@ -143,12 +165,35 @@ public class NotificationService {
vo.setPage(safePage);
vo.setPageSize(safeSize);
vo.setUnreadCount(unreadCount(userId, audience));
log.info("[notification] 列表查询 userId={} audience={} page={} size={} onlyUnread={} keyword={} 起={} 止={} 命中={}",
userId, audience, safePage, safeSize, onlyUnread, normalize(safe.getKeyword()),
safe.getStartDate(), safe.getEndDate(), total);
log.info("[notification] 列表查询 userId={} audience={} page={} size={} onlyUnread={} category={} keyword={} 起={} 止={} 命中={}",
userId, audience, safePage, safeSize, onlyUnread, normalize(safe.getCategory()),
normalize(safe.getKeyword()), safe.getStartDate(), safe.getEndDate(), total);
return vo;
}
/** scene → 用户可见大类;未知 scene(历史遗留/新增未登记)一律归「系统通知」。 */
public static String categoryOf(String scene) {
return SCENE_TO_CATEGORY.getOrDefault(normalize(scene), CATEGORY_SYSTEM_NOTICE);
}
/**
* 大类对应的 scene 集合供查询侧做 scene IN (...) 过滤
* 空值或未知大类返回空列表调用方据此跳过该筛选不报错避免前端传错值就查不到数据
*/
public static List<String> scenesOfCategory(String category) {
String wanted = normalize(category);
if (wanted.isEmpty()) {
return List.of();
}
List<String> scenes = new ArrayList<>();
for (Map.Entry<String, String> entry : SCENE_TO_CATEGORY.entrySet()) {
if (entry.getValue().equals(wanted)) {
scenes.add(entry.getKey());
}
}
return scenes;
}
/** 摘要:未读数 + 最新通知 id(前端轮询判断是否有新通知)。 */
public NotificationSummaryVo summary(Long userId, String audience) {
NotificationSummaryVo vo = new NotificationSummaryVo();
@@ -217,6 +262,17 @@ public class NotificationService {
return deleted;
}
/** 清理指定时间之前仍未读的通知(保留期由调用方决定,比已读给得更宽)。 */
public int cleanupUnreadBefore(LocalDateTime cutoff) {
int deleted = userNotificationMapper.delete(new LambdaQueryWrapper<UserNotificationEntity>()
.isNull(UserNotificationEntity::getReadAt)
.lt(UserNotificationEntity::getCreatedAt, cutoff));
if (deleted > 0) {
log.info("[notification] 清理历史未读通知 cutoff={} 删除={} 条", cutoff, deleted);
}
return deleted;
}
private LambdaQueryWrapper<UserNotificationEntity> baseWrapper(Long userId, String audience, boolean onlyUnread,
NotificationPageQuery query) {
LambdaQueryWrapper<UserNotificationEntity> wrapper = new LambdaQueryWrapper<UserNotificationEntity>()
@@ -231,9 +287,11 @@ public class NotificationService {
/**
* 列表筛选关键字模糊匹配标题/内容日期按年月日闭区间
* 起日 00:00 起含止日次日 00:00 前不含避免当天 23:59 漏行
* 起日 00:00 起含止日次日 00:00 前不含避免当天 23:59 漏行
* 类型按大类映射成 scene 集合过滤
*/
private void applyFilters(LambdaQueryWrapper<UserNotificationEntity> wrapper, NotificationPageQuery query) {
applyCategoryFilter(wrapper, query.getCategory());
String keyword = normalize(query.getKeyword());
if (!keyword.isEmpty()) {
wrapper.and(nested -> nested.like(UserNotificationEntity::getTitle, keyword)
@@ -247,8 +305,28 @@ public class NotificationService {
}
}
private boolean existsByDedupeKey(String dedupeKey) {
return selectByDedupeKey(dedupeKey) != null;
/**
* 类型筛选系统通知是兜底类 scene=system 外还要包含所有未登记 scene
* 否则直接改库塞入的 scene 在筛选时会凭空消失故表达为= system 不在其它三类里
* 未知分类一律不追加条件避免前端传错值就查不到任何数据
*/
private void applyCategoryFilter(LambdaQueryWrapper<UserNotificationEntity> wrapper, String category) {
String wanted = normalize(category);
List<String> scenes = wanted.isEmpty() ? List.of() : scenesOfCategory(wanted);
if (scenes.isEmpty()) {
return;
}
if (CATEGORY_SYSTEM_NOTICE.equals(wanted)) {
List<String> classified = new ArrayList<>(SCENE_TO_CATEGORY.keySet());
classified.remove(SCENE_SYSTEM);
wrapper.and(nested -> nested.eq(UserNotificationEntity::getScene, SCENE_SYSTEM)
.or().notIn(UserNotificationEntity::getScene, classified));
return;
}
wrapper.in(UserNotificationEntity::getScene, scenes);
}
private boolean existsByDedupeKey(String dedupeKey) { return selectByDedupeKey(dedupeKey) != null;
}
private UserNotificationEntity selectByDedupeKey(String dedupeKey) {
@@ -262,6 +340,7 @@ public class NotificationService {
NotificationItemVo vo = new NotificationItemVo();
vo.setId(row.getId());
vo.setScene(row.getScene());
vo.setCategory(categoryOf(row.getScene()));
vo.setLevel(row.getLevel());
vo.setTitle(row.getTitle());
vo.setContent(row.getContent());
@@ -271,7 +350,7 @@ public class NotificationService {
return vo;
}
private String normalize(String value) {
private static String normalize(String value) {
return value == null ? "" : value.trim();
}
@@ -56,8 +56,8 @@ public class PermissionMenuController {
@Operation(summary = "查询菜单权限列表")
public ApiResponse<List<PermissionMenuItemVo>> listMenus(HttpServletRequest request,
@Parameter(description = "菜单类型: app/admin") @RequestParam(required = false) String menuType) {
requireAdmin(request);
return ApiResponse.success(permissionMenuService.list(menuType));
// 传入操作者授权树据此把非超管无权授予的菜单标记为不可勾选
return ApiResponse.success(permissionMenuService.list(requireAdmin(request), menuType));
}
@PostMapping("/permission-menus")
@@ -1,5 +1,6 @@
package com.nanri.aiimage.modules.permission.model.entity;
import com.baomidou.mybatisplus.annotation.FieldStrategy;
import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableField;
import com.baomidou.mybatisplus.annotation.TableId;
@@ -22,4 +23,12 @@ public class PermissionMenuEntity {
private String routePath;
private Integer sortOrder;
private LocalDateTime createdAt;
/**
* 更新时间V131 新增由数据库维护DEFAULT CURRENT_TIMESTAMP + ON UPDATE CURRENT_TIMESTAMP
*
* <p>禁止应用显式写MySQL UPDATE 语句显式给该列赋值时不会触发自动更新
* 而本表写回多为 selectById 改字段 updateById实体带着旧值一旦写回就会冻结更新时间
*/
@TableField(value = "updated_at", insertStrategy = FieldStrategy.NEVER, updateStrategy = FieldStrategy.NEVER)
private LocalDateTime updatedAt;
}
@@ -24,4 +24,12 @@ public class PermissionMenuItemVo {
private Integer sortOrder;
@JsonProperty("created_at")
private LocalDateTime createdAt;
@JsonProperty("updated_at")
private LocalDateTime updatedAt;
/**
* 当前操作者能否把该菜单授予他人仅授权树接口list填充菜单 CRUD 回显等
* 其他接口留空语义为不限制
*/
private Boolean grantable;
}
@@ -57,8 +57,35 @@ public class PermissionMenuService {
/** Returns the flat menu catalog, including each item's direct parent ID. */
public List<PermissionMenuItemVo> list(String menuType) {
return list(null, menuType);
}
/**
* 授权树用非超管只能授予自己已持有的菜单 {@link #ensureGrantable} 同一判据
* 这里把结论标进 {@code grantable}让前端把不可授予的节点置灰避免勾了才在保存时
* 整单被 403 回滚创建用户与保存权限共用这一棵树勾到一个越权项会连建号一起失败
* operator 为空或超管时不限制
*/
public List<PermissionMenuItemVo> list(AdminUserEntity operator, String menuType) {
List<PermissionMenuEntity> menus = loadMenus(menuType);
return toItemVos(menus, menus);
List<PermissionMenuItemVo> items = toItemVos(menus, menus);
Set<Long> grantableIds = resolveGrantableMenuIds(operator);
for (PermissionMenuItemVo item : items) {
item.setGrantable(grantableIds == null || grantableIds.contains(item.getId()));
}
return items;
}
/**
* 操作者可授予的菜单 id 全集自身直接授权 + 其后代
*
* @return null 表示不限制超管或无操作者
*/
private Set<Long> resolveGrantableMenuIds(AdminUserEntity operator) {
if (operator == null || operator.getId() == null || isSuperAdmin(operator)) {
return null;
}
return expandDescendantIds(new LinkedHashSet<>(loadDirectColumnIds(operator.getId())), loadMenus(null));
}
/**
@@ -262,6 +289,20 @@ public class PermissionMenuService {
public List<PermissionMenuItemVo> getUserColumnPermissions(AdminUserEntity operator,
Long userId,
String menuType) {
return getUserColumnPermissions(operator, userId, menuType, false);
}
/**
* Controller-facing variant with target-user scope checks.
*
* <p>includeAncestorGroups=true 时额外把有可见后代的祖先节点并入返回集合仅用于
* 后台侧边栏还原二级分组层级分组自身无页面路由不代表授权扩展权限判定与客户端
* 组键命中即整组放行的权限键集合语义都不受影响默认 false 保持原行为</p>
*/
public List<PermissionMenuItemVo> getUserColumnPermissions(AdminUserEntity operator,
Long userId,
String menuType,
boolean includeAncestorGroups) {
AdminUserEntity user = getUserById(userId);
ensureTargetAccessible(operator, user);
List<PermissionMenuEntity> menus = loadMenus(menuType);
@@ -273,8 +314,9 @@ public class PermissionMenuService {
return List.of();
}
Set<Long> effectiveIds = expandDescendantIds(directIds, menus);
Set<Long> visibleIds = includeAncestorGroups ? includeAncestorIds(effectiveIds, menus) : effectiveIds;
List<PermissionMenuEntity> effectiveMenus = menus.stream()
.filter(menu -> menu.getId() != null && effectiveIds.contains(menu.getId()))
.filter(menu -> menu.getId() != null && visibleIds.contains(menu.getId()))
.toList();
return toItemVos(effectiveMenus, menus);
}
@@ -528,7 +570,7 @@ public class PermissionMenuService {
.filter(id -> !protectedIds.contains(id))
.toList();
}
ensureGrantable(operator, grantIds);
ensureGrantable(operator, grantIds, userId);
LinkedHashSet<Long> finalGrantIds = new LinkedHashSet<>(grantIds);
if (!protectedIds.isEmpty()) {
@@ -724,6 +766,44 @@ public class PermissionMenuService {
return effective;
}
/**
* 把可见节点沿 parentId 链上的祖先并入集合仅供后台侧边栏还原一级分组 + 子页面
* 的展示层级分组节点无页面路由不代表授权扩展权限判定另有独立方法父链缺失
* 孤儿数据时截断出现自环/成环时由 visited 兜底终止
*/
private Set<Long> includeAncestorIds(Set<Long> effectiveIds, List<PermissionMenuEntity> menus) {
Map<Long, PermissionMenuEntity> menuMap = new HashMap<>();
for (PermissionMenuEntity menu : menus) {
if (menu.getId() != null) {
menuMap.put(menu.getId(), menu);
}
}
Set<Long> withAncestors = new LinkedHashSet<>(effectiveIds);
int addedCount = 0;
for (Long id : effectiveIds) {
PermissionMenuEntity current = menuMap.get(id);
Set<Long> visited = new HashSet<>();
while (current != null && current.getId() != null && visited.add(current.getId())) {
Long parentId = current.getParentId();
if (parentId == null) {
break;
}
PermissionMenuEntity parent = menuMap.get(parentId);
if (parent == null) {
break;
}
if (withAncestors.add(parent.getId())) {
addedCount++;
}
current = parent;
}
}
if (addedCount > 0) {
log.debug("[menu-tree-ancestor] 侧边栏树补充祖先分组节点 {} 个(仅展示层级,不改变授权)", addedCount);
}
return withAncestors;
}
private PermissionMenuEntity getMenuById(Long id) {
PermissionMenuEntity entity = permissionMenuMapper.selectById(id);
if (entity == null) {
@@ -780,14 +860,25 @@ public class PermissionMenuService {
}
}
/** Throws 403 when a non-super admin requests grants outside their own effective set. */
private void ensureGrantable(AdminUserEntity operator, List<Long> requestedIds) {
/**
* Throws 403 when a non-super admin **新增** grants outside their own effective set.
*
* <p>目标用户已持有的直接授权不参与校验那是既有事实通常由超管分配普通管理员
* 编辑该用户时整树提交会把它原样带回若一并判为越权整笔事务会回滚连建号改密
* 都做不成生产 2026-09-16 现象勾到一个越权项创建用户与保存权限双双失败
* 只放行保留已有不放行新增因此不构成提权
*/
private void ensureGrantable(AdminUserEntity operator, List<Long> requestedIds, Long targetUserId) {
if (operator == null || isSuperAdmin(operator)) {
return;
}
List<PermissionMenuEntity> menus = loadMenus(null);
Set<Long> effective = expandDescendantIds(new LinkedHashSet<>(loadDirectColumnIds(operator.getId())), menus);
// 操作者自身 id 缺失异常数据时按无任何可授予项从严处理不放行
Set<Long> effective = Objects.requireNonNullElse(resolveGrantableMenuIds(operator), Set.of());
Set<Long> keptIds = targetUserId == null
? Set.of()
: new LinkedHashSet<>(loadDirectColumnIds(targetUserId));
Set<Long> denied = requestedIds.stream()
.filter(id -> !keptIds.contains(id))
.filter(id -> !effective.contains(id))
.collect(Collectors.toCollection(LinkedHashSet::new));
if (!denied.isEmpty()) {
@@ -1069,6 +1160,7 @@ public class PermissionMenuService {
vo.setRoutePath(entity.getRoutePath());
vo.setSortOrder(entity.getSortOrder());
vo.setCreatedAt(entity.getCreatedAt());
vo.setUpdatedAt(entity.getUpdatedAt());
return vo;
}
@@ -2,8 +2,26 @@ package com.nanri.aiimage.modules.pricetrack.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.nanri.aiimage.modules.pricetrack.model.entity.PriceTrackLoopRunEntity;
import org.apache.ibatis.annotations.Delete;
import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param;
import java.time.LocalDateTime;
@Mapper
public interface PriceTrackLoopRunMapper extends BaseMapper<PriceTrackLoopRunEntity> {
/**
* 分批删除已结束且超过保留期的循环批次行
*
* <p>只删终态行RUNNING 的批次删了会让后续 dispatch / childFinished 找不到记录
* 时间线用 COALESCE 兜底终态行本应写 finished_at历史行可能只有 updated_at
*/
@Delete("""
DELETE FROM biz_price_track_loop_run
WHERE status IN ('SUCCESS', 'FAILED', 'STOPPED')
AND COALESCE(finished_at, updated_at, created_at) < #{cutoff}
LIMIT #{batchSize}
""")
int deleteFinishedBefore(@Param("cutoff") LocalDateTime cutoff, @Param("batchSize") int batchSize);
}
@@ -29,6 +29,8 @@ public class PriceTrackLoopRunEntity {
@TableField(value = "active_task_id", updateStrategy = FieldStrategy.ALWAYS)
private Long activeTaskId;
private Boolean stopRequested;
/** 因客户端中断自动重派当前轮的次数:封顶用,避免会话持续不可用时无限重派(V129)。 */
private Integer resumeAttempt;
private String errorMessage;
private LocalDateTime createdAt;
private LocalDateTime updatedAt;
@@ -0,0 +1,70 @@
package com.nanri.aiimage.modules.pricetrack.service;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.modules.pricetrack.mapper.PriceTrackLoopRunMapper;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.time.Duration;
import java.time.LocalDateTime;
/**
* 跟价循环批次表biz_price_track_loop_run的保留期清理
*
* <p>每次循环跟价插一行且只在请求停止时更新全模块此前没有任何删除路径纯遗漏
* 读取侧只有两个入口 id 取单条用户当下正在看的那一轮当前是否有活跃循环
* 没有任何历史列表查询所以终态批次过了保留期即可安全删除
*
* <p>只删终态SUCCESS/FAILED/STOPPEDRUNNING 一律不动正在跑的循环被删会让
* 后续 dispatch / childFinished 找不到记录而中断
*/
@Service
@RequiredArgsConstructor
@Slf4j
public class PriceTrackLoopRunRetentionService {
/** 单轮最多删除的批次数,剩余留给下一轮。 */
static final int MAX_BATCHES_PER_RUN = 20;
private final PriceTrackLoopRunMapper loopRunMapper;
private final DistributedJobLockService distributedJobLockService;
@Value("${aiimage.price-track.loop-run-retention-days:30}")
private int retentionDays = 30;
@Value("${aiimage.price-track.loop-run-retention-batch-size:500}")
private int retentionBatchSize = 500;
@Scheduled(cron = "${aiimage.price-track.loop-run-retention-cron:0 20 4 * * *}")
public void purgeExpiredLoopRuns() {
int days = Math.max(1, retentionDays);
int batchSize = Math.max(50, retentionBatchSize);
LocalDateTime cutoff = LocalDateTime.now().minusDays(days);
DistributedJobLockService.LockHandle lockHandle =
distributedJobLockService.tryLock("price-track:loop-run-retention", Duration.ofMinutes(15));
if (lockHandle == null) {
log.info("[price-track-loop] 批次保留清理跳过:另一实例持锁");
return;
}
try (lockHandle) {
int totalDeleted = 0;
int batches = 0;
while (batches < MAX_BATCHES_PER_RUN) {
int deleted = loopRunMapper.deleteFinishedBefore(cutoff, batchSize);
batches++;
totalDeleted += deleted;
if (deleted < batchSize) {
break;
}
}
log.info("[price-track-loop] 批次保留清理完成 cutoff={} retentionDays={} deleted={} batches={}",
cutoff, days, totalDeleted, batches);
} catch (Exception ex) {
log.warn("[price-track-loop] 批次保留清理失败 cutoff={} msg={}", cutoff, ex.getMessage(), ex);
}
}
}
@@ -39,6 +39,14 @@ public class PriceTrackLoopRunService {
private static final String STATUS_STOPPED = "STOPPED";
private static final String EXECUTION_MODE_FINITE = "FINITE";
private static final String EXECUTION_MODE_INFINITE = "INFINITE";
/** 客户端异常中断的任务 errorMessage 前缀(TaskHeartbeatService.markInterrupted 写入)。 */
private static final String CLIENT_INTERRUPT_ERROR_PREFIX = "客户端异常中断";
/**
* 中断后自动重派当前轮的次数上限
* 会话持续不可用账号被风控紫鸟未就绪不封顶会无限重派无限重开浏览器
* 2026-09-18 任务 28587 就是这样白烧了 5 小时
*/
private static final int MAX_AUTO_RESUME_ATTEMPT = 3;
private final PriceTrackLoopRunMapper loopRunMapper;
private final FileTaskMapper fileTaskMapper;
@@ -259,6 +267,33 @@ public class PriceTrackLoopRunService {
entity.setActiveTaskId(null);
entity.setUpdatedAt(LocalDateTime.now());
if (STATUS_FAILED.equals(task.getStatus())) {
// 用户已请求停止时绝不续派停止意图优先于自动恢复否则点完停止循环还会自己转起来
if (Boolean.TRUE.equals(entity.getStopRequested())) {
markStopped(entity, null);
return;
}
// 客户端重启中断markInterrupted 写入的前缀不终止循环清空 active_task_id 后保持
// RUNNING客户端下次 dispatchNext 会拿到**同一店铺同一轮次** childTaskRequest
// 等于原地续跑页内已处理的 ASIN 由服务端 skip_asins 去重不会重复改价
if (isClientInterrupt(task) && currentResumeAttempt(entity) < MAX_AUTO_RESUME_ATTEMPT) {
int attempt = currentResumeAttempt(entity) + 1;
entity.setStatus(STATUS_RUNNING);
entity.setErrorMessage(null);
entity.setFinishedAt(null);
entity.setResumeAttempt(attempt);
loopRunMapper.updateById(entity);
log.warn("[price-track-loop] 子任务因客户端中断失败,自动重派当前轮 loopRunId={} childTaskId={} "
+ "round={} shopIndex={} resumeAttempt={}/{} error={}",
entity.getId(), childTaskId, entity.getCurrentRound(), entity.getCurrentShopIndex(),
attempt, MAX_AUTO_RESUME_ATTEMPT, task.getErrorMessage());
return;
}
if (isClientInterrupt(task)) {
log.warn("[price-track-loop] 客户端中断续跑已达上限 {} 次,终止循环 loopRunId={} childTaskId={} "
+ "round={} shopIndex={}",
MAX_AUTO_RESUME_ATTEMPT, entity.getId(), childTaskId,
entity.getCurrentRound(), entity.getCurrentShopIndex());
}
entity.setStatus(STATUS_FAILED);
entity.setErrorMessage(task.getErrorMessage() == null || task.getErrorMessage().isBlank()
? "子任务执行失败"
@@ -269,6 +304,8 @@ public class PriceTrackLoopRunService {
entity.getId(), childTaskId, entity.getCurrentRound(), entity.getCurrentShopIndex(), entity.getErrorMessage());
return;
}
// 子任务成功 中断续跑计数归零否则历史上的中断会一直占用封顶额度
entity.setResumeAttempt(0);
List<PriceTrackMatchShopsVo.PriceTrackShopQueueItem> items = parseShops(entity);
if (items.isEmpty()) {
entity.setStatus(STATUS_FAILED);
@@ -300,6 +337,16 @@ public class PriceTrackLoopRunService {
entity.getId(), childTaskId, entity.getStatus(), entity.getCurrentRound(), entity.getCurrentShopIndex(), entity.getActiveTaskId());
}
/** 子任务失败原因是否为「客户端异常中断」(客户端重启上报,可自动重派续跑)。 */
private static boolean isClientInterrupt(FileTaskEntity task) {
String message = task == null ? null : task.getErrorMessage();
return message != null && message.startsWith(CLIENT_INTERRUPT_ERROR_PREFIX);
}
private static int currentResumeAttempt(PriceTrackLoopRunEntity entity) {
return entity.getResumeAttempt() == null ? 0 : entity.getResumeAttempt();
}
private void reconcileWithTerminalChild(PriceTrackLoopRunEntity entity) {
if (entity == null || entity.getActiveTaskId() == null || isTerminal(entity.getStatus())) {
return;
@@ -653,13 +653,11 @@ public class PriceTrackTaskService {
}
changed = true;
if (payload.getError() != null && !payload.getError().isBlank()) {
markResultFailed(fr, payload.getError());
priceTrackTaskCacheService.removeShopMergedPayload(taskId, shopKey);
finalizeFailedShop(fr, shopKey, payload, payload.getError());
continue;
}
if (Boolean.FALSE.equals(payload.getSuccess())) {
markResultFailed(fr, "shop processing failed");
priceTrackTaskCacheService.removeShopMergedPayload(taskId, shopKey);
finalizeFailedShop(fr, shopKey, payload, "shop processing failed");
continue;
}
handleSkipAsinDeletionSignals(shopKey, payload);
@@ -815,6 +813,39 @@ public class PriceTrackTaskService {
updateTaskStatusFromLatestRows(task, latest);
}
/**
* 失败店铺的收尾任务照旧标 FAILED原因回显给用户但已经把跑出来的行组装成
* 部分结果文件随任务交付否则用户只看到失败却拿不到哪些 ASIN 实际已被
* 跟价/改价的记录会话掉线这类"跑了前几页才断"的场景尤其需要
*
* <p>组装出来的行仍是失败态success=0 + errorMessage任务状态因此不变
* 只是多了个可下载的文件只有一行可用数据都没有时才退化成纯失败
*/
private void finalizeFailedShop(FileResultEntity fr, String shopKey,
PriceTrackSubmitResultRequest.ShopResult payload, String errorMessage) {
String finalMessage = errorMessage;
try {
PriceTrackSubmitResultRequest.ShopResult merged = mergeShopPayload(fr.getTaskId(), shopKey, payload);
int rows = countPayloadRows(merged);
if (rows > 0) {
markResultFailed(fr, errorMessage);
enqueueResultFileAssembly(fr, shopKey, merged, true);
priceTrackTaskCacheService.removeShopMergedPayload(fr.getTaskId(), shopKey);
log.info("[price-track] 失败店铺仍产出部分结果 taskId={} shop={} rows={} error={}",
fr.getTaskId(), shopKey, rows, errorMessage);
return;
}
log.info("[price-track] 失败店铺无可用行,不产出结果文件 taskId={} shop={} error={}",
fr.getTaskId(), shopKey, errorMessage);
} catch (Exception ex) {
log.warn("[price-track] 失败店铺部分结果排队失败,仅标记失败 taskId={} shop={} msg={}",
fr.getTaskId(), shopKey, ex.getMessage(), ex);
finalMessage = errorMessage + "(部分结果组装排队失败:" + ex.getMessage() + "";
}
markResultFailed(fr, finalMessage);
priceTrackTaskCacheService.removeShopMergedPayload(fr.getTaskId(), shopKey);
}
private void markResultFailed(FileResultEntity fr, String message) {
fr.setSuccess(0);
fr.setErrorMessage(message);
@@ -841,11 +872,18 @@ public class PriceTrackTaskService {
java.io.File workRoot = cn.hutool.core.io.FileUtil.mkdir(
cn.hutool.core.io.FileUtil.file(System.getProperty("java.io.tmpdir"), "price-track-result", String.valueOf(result.getTaskId())));
java.io.File xlsx = cn.hutool.core.io.FileUtil.file(workRoot, stem + ".xlsx");
// 失败店铺的部分结果errorMessage 已写明原因组装完成时保留失败态只挂文件
// 否则组装一落地就把行"洗成成功"用户再也看不到这个店铺其实没跑完
boolean partialFailure = result.getErrorMessage() != null && !result.getErrorMessage().isBlank();
try {
excelAssemblyService.writeWorkbook(xlsx, countries);
String objectKey = ossStorageService.uploadResultFile(xlsx, MODULE_TYPE);
if (partialFailure) {
result.setSuccess(0);
} else {
result.setSuccess(1);
result.setErrorMessage(null);
}
result.setResultFilename(stem + ".xlsx");
result.setResultFileUrl(objectKey);
result.setResultFileSize(xlsx.length());
@@ -885,9 +923,20 @@ public class PriceTrackTaskService {
private void enqueueResultFileAssembly(FileResultEntity result,
String shopKey,
PriceTrackSubmitResultRequest.ShopResult payload) {
enqueueResultFileAssembly(result, shopKey, payload, false);
}
/**
* @param preserveFailure 失败店铺的部分结果保留 success=0 + 失败原因只把文件挂上去
* 任务状态不变 FAILED用户仍能下载已跑出来的行
*/
private void enqueueResultFileAssembly(FileResultEntity result,
String shopKey,
PriceTrackSubmitResultRequest.ShopResult payload,
boolean preserveFailure) {
applyServerModifyCounts(result.getTaskId(), payload);
taskResultPayloadService.saveLatest(result.getTaskId(), MODULE_TYPE, shopKey, payload);
markResultFilePending(result, shopKey, payload);
markResultFilePending(result, shopKey, payload, preserveFailure);
taskFileJobService.enqueueAssembleResult(result.getTaskId(), MODULE_TYPE, result.getId(), shopKey);
}
@@ -993,15 +1042,24 @@ public class PriceTrackTaskService {
private void markResultFilePending(FileResultEntity result,
String shopKey,
PriceTrackSubmitResultRequest.ShopResult payload) {
PriceTrackSubmitResultRequest.ShopResult payload,
boolean preserveFailure) {
Map<String, List<PriceTrackSubmitResultRequest.AsinResult>> countries =
excelAssemblyService.normalizeCountriesMap(payload.getCountries());
String displayName = payload.getShopName() != null && !payload.getShopName().isBlank()
? payload.getShopName().trim()
: shopKey;
String stem = safeFileStem(displayName);
if (preserveFailure) {
// 失败店铺的部分结果成功态与失败原因都不能动只标文件名已定文件待组装
result.setSuccess(0);
if (result.getErrorMessage() == null || result.getErrorMessage().isBlank()) {
result.setErrorMessage("店铺未跑完,仅产出部分结果");
}
} else {
result.setSuccess(1);
result.setErrorMessage(null);
}
result.setResultFilename(stem + ".xlsx");
result.setResultFileUrl(null);
result.setResultFileSize(0L);
@@ -2164,6 +2222,12 @@ public class PriceTrackTaskService {
}
ok++;
} else if (failed) {
// 失败行也可能正在组装"部分结果"文件文件没落地前不能让任务提前终态
// 否则前端一停轮询下载按钮永远不出现失败任务的结果文件同样要能下载
if (isResultAwaitingFileAssembly(fr, jobMap.get(fr.getId()))) {
allDone = false;
continue;
}
fail++;
allErrors.add(fr.getSourceFilename() + ": " + fr.getErrorMessage());
} else {
@@ -652,9 +652,8 @@ public class ProductRiskTaskService {
}
matchedShopCount++;
if (payload.getError() != null && !payload.getError().isBlank()) {
markResultFailed(fr, payload.getError());
batchErrors.add(shopKey + ": " + payload.getError());
productRiskTaskCacheService.removeShopMergedPayload(taskId, shopKey);
finalizeFailedShop(fr, shopKey, mergeShopPayload(taskId, shopKey, payload),
payload.getError(), batchErrors);
continue;
}
@@ -759,9 +758,8 @@ public class ProductRiskTaskService {
}
if (cachedPayload.getError() != null && !cachedPayload.getError().isBlank()) {
markResultFailed(fr, cachedPayload.getError());
batchErrors.add(shopKey + ": " + cachedPayload.getError());
productRiskTaskCacheService.removeShopMergedPayload(taskId, shopKey);
// 陈旧收尾同样保留已跑出来的行失败原因照常回显文件顺手组装
finalizeFailedShop(fr, shopKey, cachedPayload, cachedPayload.getError(), batchErrors);
changed = true;
continue;
}
@@ -852,6 +850,9 @@ public class ProductRiskTaskService {
String stem = safeFileStem(displayName);
File xlsx = FileUtil.file(workRoot, stem + ".xlsx");
File zip = FileUtil.file(workRoot, stem + ".zip");
// 失败店铺的部分结果errorMessage 已写明原因组装完成时保留失败态只挂文件
// 否则组装一落地就把行"洗成成功"用户再也看不到这个店铺其实没跑完
boolean partialFailure = fr.getErrorMessage() != null && !fr.getErrorMessage().isBlank();
try {
excelAssemblyService.writeWorkbook(xlsx, displayName, countries);
ZipUtil.zip(zip, false, xlsx);
@@ -861,8 +862,12 @@ public class ProductRiskTaskService {
fr.setResultFileSize(zip.length());
fr.setResultContentType(CONTENT_TYPE_ZIP);
fr.setRowCount(excelAssemblyService.countRows(countries));
if (partialFailure) {
fr.setSuccess(0);
} else {
fr.setSuccess(1);
fr.setErrorMessage(null);
}
fileResultMapper.updateById(fr);
} finally {
FileUtil.del(xlsx);
@@ -899,13 +904,56 @@ public class ProductRiskTaskService {
}
private void enqueueResultFileAssembly(FileResultEntity result, String shopKey, ProductRiskShopPayloadDto payload) {
enqueueResultFileAssembly(result, shopKey, payload, false);
}
/**
* @param preserveFailure 失败店铺的部分结果保留 success=0 + 失败原因只把文件挂上去
* 任务状态不变 FAILED用户仍能下载已跑出来的行
*/
private void enqueueResultFileAssembly(FileResultEntity result, String shopKey,
ProductRiskShopPayloadDto payload, boolean preserveFailure) {
taskResultPayloadService.saveLatest(result.getTaskId(), MODULE_TYPE, shopKey, payload);
taskResultItemService.replaceResultSnapshot(result.getTaskId(), MODULE_TYPE, result.getId(), shopKey, payload);
markResultFilePending(result, shopKey, payload);
markResultFilePending(result, shopKey, payload, preserveFailure);
taskFileJobService.enqueueAssembleResult(result.getTaskId(), MODULE_TYPE, result.getId(), shopKey);
}
private void markResultFilePending(FileResultEntity result, String shopKey, ProductRiskShopPayloadDto payload) {
/**
* 失败店铺的收尾任务照旧标 FAILED原因回显给用户但已经把跑出来的行组装成
* 部分结果文件随任务交付否则用户只看到失败却拿不到哪些 ASIN 实际已被处理
* 的记录会话掉线这类"跑了几个国家才断"的场景尤其需要一行可用数据都没有时才退化成纯失败
*/
private void finalizeFailedShop(FileResultEntity fr, String shopKey,
ProductRiskShopPayloadDto mergedPayload, String errorMessage,
List<String> batchErrors) {
int rows = mergedPayload == null ? 0 : countPayloadRows(mergedPayload);
String message = errorMessage;
if (rows > 0) {
markResultFailed(fr, errorMessage);
try {
enqueueResultFileAssembly(fr, shopKey, mergedPayload, true);
batchErrors.add(shopKey + ": " + errorMessage);
productRiskTaskCacheService.removeShopMergedPayload(fr.getTaskId(), shopKey);
log.info("[product-risk] 失败店铺仍产出部分结果 taskId={} shop={} rows={} error={}",
fr.getTaskId(), shopKey, rows, errorMessage);
return;
} catch (Exception ex) {
message = errorMessage + "(部分结果组装排队失败:" + (ex.getMessage() == null ? "未知错误" : ex.getMessage()) + "";
log.warn("[product-risk] 失败店铺部分结果排队失败,仅标记失败 taskId={} shop={} msg={}",
fr.getTaskId(), shopKey, ex.getMessage(), ex);
}
} else {
log.info("[product-risk] 失败店铺无可用行,不产出结果文件 taskId={} shop={} error={}",
fr.getTaskId(), shopKey, errorMessage);
}
markResultFailed(fr, message);
batchErrors.add(shopKey + ": " + message);
productRiskTaskCacheService.removeShopMergedPayload(fr.getTaskId(), shopKey);
}
private void markResultFilePending(FileResultEntity result, String shopKey,
ProductRiskShopPayloadDto payload, boolean preserveFailure) {
Map<String, List<ProductRiskRowDto>> countries = excelAssemblyService.normalizeCountriesMap(payload.getCountries());
String displayName = payload.getShopName() != null && !payload.getShopName().isBlank()
? payload.getShopName().trim()
@@ -916,8 +964,16 @@ public class ProductRiskTaskService {
result.setResultFileSize(0L);
result.setResultContentType(CONTENT_TYPE_ZIP);
result.setRowCount(excelAssemblyService.countRows(countries));
if (preserveFailure) {
// 失败店铺的部分结果成功态与失败原因都不能动只标文件名已定文件待组装
result.setSuccess(0);
if (result.getErrorMessage() == null || result.getErrorMessage().isBlank()) {
result.setErrorMessage("店铺未跑完,仅产出部分结果");
}
} else {
result.setSuccess(1);
result.setErrorMessage(null);
}
fileResultMapper.updateById(result);
}
@@ -1,6 +1,7 @@
package com.nanri.aiimage.modules.publish.controller;
import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.common.security.AdminAuthSupport;
import com.nanri.aiimage.modules.publish.model.dto.PublishParseRequest;
import com.nanri.aiimage.modules.publish.model.dto.PublishSubmitResultRequest;
import com.nanri.aiimage.modules.publish.model.dto.PublishTaskBatchRequest;
@@ -14,6 +15,7 @@ import com.nanri.aiimage.modules.publish.service.PublishTaskService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.validation.Valid;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.DeleteMapping;
@@ -34,6 +36,7 @@ import com.nanri.aiimage.modules.task.model.vo.TaskProgressLightBatchVo;
public class PublishController {
private final PublishTaskService publishTaskService;
private final AdminAuthSupport adminAuthSupport;
@PostMapping("/parse")
@Operation(
@@ -57,15 +60,18 @@ public class PublishController {
@PostMapping("/tasks/{taskId}/files/{fileId}/activate")
@Operation(
summary = "激活任务中的单个文件",
description = "前端派发 Python 队列前调用。taskId 和 fileId 用于定位文件,user_id 可省略;如传入则校验任务归属。同一任务同一时间只允许一个 RUNNING 文件;重复激活当前 RUNNING 文件为幂等操作。")
description = "前端派发 Python 队列前调用。taskId 和 fileId 用于定位文件,user_id 可省略;如传入则校验任务归属。同一任务同一时间只允许一个 RUNNING 文件;重复激活当前 RUNNING 文件为幂等操作。"
+ "店铺互斥按 (发起方设备, 店铺) 判定:同一台机器上同一店铺只允许一个任务在跑(该机器上该店铺只有一个紫鸟浏览器会话,并发会互相切换国家);不同设备各自持有独立会话,允许同一店铺并行跑不同国家。"
+ "设备号取自 JWT 签名的 deviceId claim,缺失时退回按店铺全局互斥。")
public ApiResponse<Void> activateFile(
@Parameter(description = "上架任务 ID", required = true, example = "9001")
@PathVariable Long taskId,
@Parameter(description = "任务内文件 ID", required = true, example = "9101")
@PathVariable Long fileId,
@Parameter(description = "可选的任务所属用户 ID;省略时由 taskId 反查", required = false, example = "1")
@RequestParam(value = "user_id", required = false) Long userId) {
publishTaskService.activateFile(taskId, fileId, userId);
@RequestParam(value = "user_id", required = false) Long userId,
HttpServletRequest request) {
publishTaskService.activateFile(taskId, fileId, userId, adminAuthSupport.currentDeviceId(request));
return ApiResponse.success(null);
}
@@ -20,6 +20,8 @@ public class PublishFileEntity {
private Integer matched;
private String shopId;
private Long matchedUserId;
/** 激活该文件时客户端所在设备(JWT 签名的 deviceId);空表示来源不明,按全局店铺互斥保守处理。 */
private String deviceId;
private String platform;
private String companyName;
private String matchStatus;
@@ -0,0 +1,97 @@
package com.nanri.aiimage.modules.publish.service;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.time.Duration;
import java.time.LocalDateTime;
import java.util.List;
/**
* 上架任务的保留期清理2026-09 审核PUBLISH 的任务历史此前完全没有清理永久累积
*
* <p>涉及 biz_file_task / biz_file_result / biz_publish_item每个上传 Excel 的每一行落一行
* 单任务可达数万行/ biz_publish_file此前只有用户手动删任务才回收这里刻意不复用
* ModuleHistoryCleanupService上架单任务行数太大且删除必须经过业务删除入口回收结果对象
*
* <p>删除动作复用 {@link PublishTaskService#deleteTaskForRetention(Long)}与用户删任务
* 同一套删除实现明细/结果/分片载荷 + 事务提交后回收结果对象本类只做查一批过期 id
* 逐个调用 计数该入口保留状态校验只删终态任务PENDING/RUNNING 不会被碰
*
* <p>双节点用 job 锁保证单实例执行每批小批量默认 50逐个删避免单次事务过长
*/
@Service
@RequiredArgsConstructor
@Slf4j
public class PublishTaskRetentionService {
/** 单轮最多处理的批次数(每批 batchSize 个任务),剩余留给下一轮。 */
static final int MAX_BATCHES_PER_RUN = 20;
private final FileTaskMapper fileTaskMapper;
private final PublishTaskService publishTaskService;
private final DistributedJobLockService distributedJobLockService;
@Value("${aiimage.publish.task-retention-days:90}")
private int retentionDays = 90;
@Value("${aiimage.publish.task-retention-batch-size:50}")
private int retentionBatchSize = 50;
@Scheduled(cron = "${aiimage.publish.task-retention-cron:0 30 4 * * *}")
public void purgeExpiredTasks() {
int days = Math.max(1, retentionDays);
int batchSize = Math.max(1, retentionBatchSize);
LocalDateTime cutoff = LocalDateTime.now().minusDays(days);
// 单轮 20 × 50 个任务的域内删除含结果对象回收可能跑较久 TTL 给足 30 分钟
DistributedJobLockService.LockHandle lockHandle =
distributedJobLockService.tryLock("publish:task-retention", Duration.ofMinutes(30));
if (lockHandle == null) {
log.info("[publish-retention] 任务保留清理跳过:另一实例持锁");
return;
}
try (lockHandle) {
int totalDeleted = 0;
int totalFailed = 0;
int batches = 0;
while (batches < MAX_BATCHES_PER_RUN) {
// 只取 id上架任务行可携带很大的 request_json/result_json不拉整行
List<Long> taskIds = fileTaskMapper.selectExpiredTerminalTaskIds(
PublishTaskService.MODULE_TYPE, cutoff, batchSize);
if (taskIds.isEmpty()) {
break;
}
batches++;
int deletedInBatch = 0;
for (Long taskId : taskIds) {
try {
publishTaskService.deleteTaskForRetention(taskId);
deletedInBatch++;
} catch (Exception ex) {
// 单个任务删除失败只记日志继续一个坏任务不能卡住整轮
log.warn("[publish-retention] 任务删除失败 taskId={} msg={}", taskId, ex.getMessage(), ex);
}
}
totalDeleted += deletedInBatch;
totalFailed += taskIds.size() - deletedInBatch;
if (deletedInBatch == 0) {
log.warn("[publish-retention] 整批任务均删除失败,提前结束本轮以免反复重试同一批");
break;
}
if (taskIds.size() < batchSize) {
break;
}
}
log.info("[publish-retention] 任务保留清理完成 cutoff={} retentionDays={} deleted={} failed={} batches={}",
cutoff, days, totalDeleted, totalFailed, batches);
} catch (Exception ex) {
log.warn("[publish-retention] 任务保留清理失败 cutoff={} msg={}", cutoff, ex.getMessage(), ex);
}
}
}
@@ -96,6 +96,12 @@ public class PublishTaskService {
private static final String STATUS_RUNNING = "RUNNING";
private static final String STATUS_SUCCESS = "SUCCESS";
private static final String STATUS_FAILED = "FAILED";
/**
* 保留期清理允许删除的终态集合 ModuleHistoryCleanupService TERMINAL_STATUSES 一致
* PENDING/RUNNING 绝不在此列PUBLISH 实际只会写 SUCCESS/FAILED取消态是无害的超集
*/
private static final Set<String> RETENTION_TERMINAL_STATUSES =
Set.of(STATUS_SUCCESS, STATUS_FAILED, "CANCELLED", "CANCELED");
private static final DateTimeFormatter TIME_FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");
private final LocalFileStorageService localFileStorageService;
@@ -168,7 +174,14 @@ public class PublishTaskService {
}
}
public void activateFile(Long taskId, Long fileId, Long userId) {
/**
* 激活任务中的单个文件
*
* @param deviceId 发起方设备标识JWT 签名的 deviceId claim空串/空白表示来源不明
* 旧客户端 token 无该 claim内部令牌调用此时退回全局店铺互斥
*/
public void activateFile(Long taskId, Long fileId, Long userId, String deviceId) {
String device = deviceId == null ? "" : deviceId.trim();
try (TaskDistributedLockService.LockHandle lock =
taskDistributedLockService.acquire(MODULE_TYPE, taskId)) {
if (lock == null) {
@@ -192,16 +205,61 @@ public class PublishTaskService {
if (runningFile != null) {
throw new BusinessException("同一任务已有文件正在执行: " + runningFile.getSourceFilename());
}
// 店铺级互斥同一台设备上同一店铺同一时刻只允许一个上架任务在跑
// 2026-09-17 事故28519/28520/28521 相继提交同一店铺林洪武同一台机器上同一
// 店铺被多个任务并发打开客户端 startBrowser 全部返回 -10000三个任务一起失败
// 激活是任务真正开跑的唯一入口在这里挡掉并带出占用中的任务号用户才知道要等谁
//
// 互斥键是 (设备, 店铺) 而不是店铺紫鸟浏览器会话是**每台机器一份**不同客户端
// 各自持有独立会话同一家店可以在两台机器上并行跑不同国家2026-09-18 任务 28624
// 在另一台机器上被 28616 误挡真正必须串行的是同一台设备那里只有一个会话
// 两个任务会互相切换国家add_product.py SwitchingCountries 是会话级状态任务
// 中途断线重连还会再切一次轻则失败重则把 A 国家的商品提交进 B 国家的店铺
// 设备号取自 JWT 签名的 deviceId claim为空旧客户端 token 无该 claim / 内部令牌
// 调用时退回改动前的全局店铺互斥保守不放宽
//
// 注意本校验与随后的状态更新之间仍有极小竞态窗口两个请求恰好同时通过校验
// 真正的串行由客户端店铺锁保证这一层的目的是尽早给出明确提示避免白传文件与重复执行
String shopName = file.getShopName();
if (shopName != null && !shopName.isBlank()) {
LambdaQueryWrapper<PublishFileEntity> shopRunningQuery = new LambdaQueryWrapper<PublishFileEntity>()
.eq(PublishFileEntity::getShopName, shopName)
.eq(PublishFileEntity::getStatus, STATUS_RUNNING)
.ne(PublishFileEntity::getTaskId, taskId)
.orderByAsc(PublishFileEntity::getId)
.last("limit 1");
if (device.isEmpty()) {
log.info("[publish] 激活无设备标识,按全局店铺互斥判定 taskId={} fileId={} shop={}",
taskId, fileId, shopName);
} else {
// 本设备的 RUNNING 以及设备未知的存量行旧客户端/内部调用NULL 或空串
// 后者无法判断落在哪台机器上一律保守视为可能同机
shopRunningQuery.and(wrapper -> wrapper
.eq(PublishFileEntity::getDeviceId, device)
.or().isNull(PublishFileEntity::getDeviceId)
.or().eq(PublishFileEntity::getDeviceId, ""));
}
PublishFileEntity shopRunning = publishFileMapper.selectOne(shopRunningQuery);
if (shopRunning != null) {
log.warn("[publish] 店铺互斥拦截 taskId={} fileId={} shop={} device={} 占用任务={} 占用设备={}",
taskId, fileId, shopName, device,
shopRunning.getTaskId(), shopRunning.getDeviceId());
throw new BusinessException("店铺「" + shopName + "」已有上架任务正在执行(任务 "
+ shopRunning.getTaskId() + "),请等它完成后再提交");
}
}
int updated = publishFileMapper.update(null, new LambdaUpdateWrapper<PublishFileEntity>()
.eq(PublishFileEntity::getId, fileId)
.eq(PublishFileEntity::getTaskId, taskId)
.eq(PublishFileEntity::getStatus, STATUS_PENDING)
.set(PublishFileEntity::getStatus, STATUS_RUNNING)
.set(PublishFileEntity::getDeviceId, device.isEmpty() ? null : device)
.set(PublishFileEntity::getUpdatedAt, LocalDateTime.now())
.set(PublishFileEntity::getErrorMessage, null));
if (updated <= 0) {
throw new BusinessException("文件激活失败,请刷新后重试");
}
log.info("[publish] 文件激活成功 taskId={} fileId={} shop={} device={}", taskId, fileId, shopName, device);
if (STATUS_PENDING.equals(task.getStatus())) {
fileTaskMapper.update(null, new LambdaUpdateWrapper<FileTaskEntity>()
.eq(FileTaskEntity::getId, taskId)
@@ -566,6 +624,37 @@ public class PublishTaskService {
@Transactional
public void deleteTask(Long taskId, Long userId) {
FileTaskEntity task = requireTaskForDeletion(taskId, userId);
deleteTaskWithRelatedRows(task);
}
/**
* 保留期清理专用删除入口内部调用定时任务没有用户身份
*
* <p> {@link #deleteTask(Long, Long)} 复用同一套删除实现明细行/结果记录/分片载荷 +
* 事务提交后回收结果对象仅跳过用户归属校验状态校验保留 只允许删终态任务
* PENDING/RUNNING 一律不删正在跑的任务被删会让分片回传结果组装找不到任务行
* 行已被删除或非上架任务时幂等跳过不抛异常
*/
@Transactional
public void deleteTaskForRetention(Long taskId) {
if (taskId == null || taskId <= 0) {
throw new BusinessException("taskId 不合法");
}
FileTaskEntity task = fileTaskMapper.selectById(taskId);
if (task == null || !MODULE_TYPE.equals(task.getModuleType())) {
log.info("[publish] 保留期删除跳过:任务已不存在或非上架任务 taskId={}", taskId);
return;
}
if (!isRetentionTerminal(task.getStatus())) {
log.warn("[publish] 保留期删除跳过非终态任务 taskId={} status={}", taskId, task.getStatus());
return;
}
deleteTaskWithRelatedRows(task);
}
/** 删除任务行与全部关联数据,并在事务提交后回收远端对象(deleteTask 与保留期清理共用)。 */
private void deleteTaskWithRelatedRows(FileTaskEntity task) {
Long taskId = task.getId();
List<FileResultEntity> results = fileResultMapper.selectList(new LambdaQueryWrapper<FileResultEntity>()
.eq(FileResultEntity::getTaskId, taskId)
.eq(FileResultEntity::getModuleType, MODULE_TYPE));
@@ -1953,6 +2042,11 @@ public class PublishTaskService {
return STATUS_SUCCESS.equals(status) || STATUS_FAILED.equals(status);
}
/** 保留期清理的终态判定:大小写不敏感,PENDING/RUNNING 一律返回 false。 */
private static boolean isRetentionTerminal(String status) {
return status != null && RETENTION_TERMINAL_STATUSES.contains(status.trim().toUpperCase(Locale.ROOT));
}
private List<Long> normalizeTaskIds(List<Long> taskIds) {
if (taskIds == null) {
return List.of();
@@ -27,6 +27,7 @@ import com.nanri.aiimage.modules.shopdatacrawl.model.vo.ShopDataCrawlTaskBatchVo
import com.nanri.aiimage.modules.shopdatacrawl.model.vo.ShopDataCrawlCreateTaskVo;
import com.nanri.aiimage.modules.shopdatacrawl.model.vo.ShopDataCrawlHistoryVo;
import com.nanri.aiimage.modules.shopdatacrawl.model.vo.ShopDataCrawlResultItemVo;
import com.nanri.aiimage.modules.shopdatacrawl.spi.DuplicateCheckRefreshPort;
import com.nanri.aiimage.common.model.vo.ProductRiskDashboardVo;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
@@ -121,6 +122,7 @@ public class ShopDataCrawlTaskService {
private final InstanceMetadata instanceMetadata;
private final ShopDataCrawlDailyFileService dailyFileService;
private final ShopDataCrawlItemStoreService shopDataCrawlItemStoreService;
private final DuplicateCheckRefreshPort duplicateCheckRefreshPort;
private final PlatformTransactionManager transactionManager;
private final TaskProgressLightAssembler taskProgressLightAssembler;
@@ -2116,6 +2118,13 @@ public class ShopDataCrawlTaskService {
shopDataCrawlItemStoreService.saveShopBatchFromSnapshot(
snapshot.getShopName(), itemBatchDate, accumulatedItems,
snapshot.getResultId(), task.getId(), baseDailyFile == null ? null : baseDailyFile.getId());
// 明细已落库请求撞款重扫异步合并执行不阻塞归档端口契约保证不抛错
try {
duplicateCheckRefreshPort.requestRefresh("shop-data-crawl:" + snapshot.getShopName());
} catch (RuntimeException ex) {
log.warn("[shop-data-crawl] 请求撞款重扫失败(忽略,不影响归档) shop={} msg={}",
snapshot.getShopName(), ex.getMessage());
}
int rowCount = excelAssemblyService.writeWorkbook(outputXlsx, accumulatedItems);
String objectKey = ossStorageService.uploadResultFile(outputXlsx, MODULE_TYPE);
if (blank(objectKey)) {
@@ -0,0 +1,17 @@
package com.nanri.aiimage.modules.shopdatacrawl.spi;
/**
* 采集明细就绪后的撞款重扫触发端口2026-09店铺数据采集落库后即时刷新重复检查
*
* <p>实现方在 shopduplicatecheck 模块{@code ShopDataDuplicateCheckScanService}
* 契约实现必须异步执行去抖合并不得阻塞调用方不得向外抛出异常
*/
public interface DuplicateCheckRefreshPort {
/**
* 请求一次撞款重扫异步合并窗口内的多次触发聚合为一次扫描
*
* @param reason 触发来源仅用于日志排查
*/
void requestRefresh(String reason);
}
@@ -4,9 +4,13 @@ import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.nanri.aiimage.modules.shopduplicatecheck.model.dto.ScanLightRowDto;
import com.nanri.aiimage.modules.shopduplicatecheck.model.dto.ScanFullRowDto;
import com.nanri.aiimage.modules.shopduplicatecheck.model.entity.ShopDataDuplicateScanEntity;
import org.apache.ibatis.annotations.Delete;
import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param;
import org.apache.ibatis.annotations.Select;
import java.time.LocalDateTime;
@Mapper
public interface ShopDataDuplicateScanMapper extends BaseMapper<ShopDataDuplicateScanEntity> {
@@ -20,4 +24,19 @@ public interface ShopDataDuplicateScanMapper extends BaseMapper<ShopDataDuplicat
+ "created_at AS createdAt FROM shop_data_duplicate_scan "
+ "WHERE status = 'SUCCESS' ORDER BY id DESC LIMIT 1")
ScanFullRowDto selectLatestFullRow();
/** 第 N 新的行 id(offset 从 0 计);行数不足时返回 null,供保留清理计算保护线。 */
@Select("SELECT id FROM shop_data_duplicate_scan ORDER BY id DESC LIMIT 1 OFFSET #{offset}")
Long selectNthNewestId(@Param("offset") int offset);
/** 分批删除保留期外、且早于保护线的历史扫描行。 */
@Delete("""
DELETE FROM shop_data_duplicate_scan
WHERE created_at < #{cutoff}
AND id < #{protectFromId}
LIMIT #{batchSize}
""")
int deleteOlderThanBatch(@Param("cutoff") LocalDateTime cutoff,
@Param("protectFromId") long protectFromId,
@Param("batchSize") int batchSize);
}
@@ -9,6 +9,7 @@ import com.nanri.aiimage.modules.shopdatacrawl.model.dto.ShopDataCrawlCountryRes
import com.nanri.aiimage.modules.shopdatacrawl.model.dto.ShopDataCrawlRowDto;
import com.nanri.aiimage.modules.shopdatacrawl.model.vo.ShopDataCrawlResultItemVo;
import com.nanri.aiimage.modules.shopdatacrawl.service.ShopDataCrawlItemStoreService;
import com.nanri.aiimage.modules.shopdatacrawl.spi.DuplicateCheckRefreshPort;
import com.nanri.aiimage.modules.shopduplicatecheck.mapper.ShopDataDuplicateScanMapper;
import com.nanri.aiimage.modules.shopduplicatecheck.mapper.ShopDuplicateCheckItemMapper;
import com.nanri.aiimage.modules.shopduplicatecheck.mapper.ShopDuplicateCheckSourceMapper;
@@ -21,6 +22,7 @@ import com.nanri.aiimage.modules.shopduplicatecheck.model.entity.ShopDataDuplica
import com.nanri.aiimage.modules.shopduplicatecheck.model.payload.DuplicateScanPayload;
import com.nanri.aiimage.modules.shopduplicatecheck.model.payload.DuplicateScanSummary;
import com.nanri.aiimage.modules.shopduplicatecheck.service.support.DuplicateCheckAggregator;
import com.nanri.aiimage.modules.shopduplicatecheck.service.support.DuplicateCheckRefreshScheduler;
import com.nanri.aiimage.modules.shopduplicatecheck.service.support.DuplicateCheckWorkbookParser;
import com.nanri.aiimage.modules.shopduplicatecheck.service.support.RawRow;
import com.nanri.aiimage.modules.shopduplicatecheck.service.support.ShopParsed;
@@ -43,14 +45,14 @@ import java.util.Set;
import java.util.concurrent.atomic.AtomicLong;
/**
* 撞款扫描每日 00:00 定时 + force 同步全量重扫
* 撞款扫描每日 00:00 定时 + force 同步全量重扫 + 采集落库触发的异步合并重扫
* 数据源 = 采集明细表biz_shop_data_crawl_item采集先落库再更新文件
* 直接查库聚合 shops/items/summary 落库 shop_data_duplicate_scan输出契约不变
* 双实例通过 Redis 分布式锁防重扫描失败落 FAILED 行并上抛force 场景由端点转 409/500
*/
@Service
@Slf4j
public class ShopDataDuplicateCheckScanService {
public class ShopDataDuplicateCheckScanService implements DuplicateCheckRefreshPort {
public static final String SCAN_LOCK = "shop-data-duplicate-check:scan";
static final DateTimeFormatter SCANNED_AT_FORMAT = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");
@@ -71,6 +73,9 @@ public class ShopDataDuplicateCheckScanService {
private final AtomicLong cachedRowId = new AtomicLong(-1L);
private volatile CachedScan cachedScan;
/** 采集落库触发的异步合并重扫调度器(单飞 + 去抖 + 锁忙重试)。 */
private final DuplicateCheckRefreshScheduler refreshScheduler;
@Autowired
public ShopDataDuplicateCheckScanService(ShopDataDuplicateScanMapper scanMapper,
ShopDuplicateCheckSourceMapper sourceMapper,
@@ -86,6 +91,7 @@ public class ShopDataDuplicateCheckScanService {
this.objectMapper = objectMapper;
this.itemMapper = itemMapper;
this.itemStoreService = itemStoreService;
this.refreshScheduler = new DuplicateCheckRefreshScheduler(this::runRefreshOnce);
}
/** 读侧视图:scanned_at 为最新 SUCCESS 行 created_atyyyy-MM-dd HH:mm:ss)。 */
@@ -122,6 +128,30 @@ public class ShopDataDuplicateCheckScanService {
}
}
/** 采集明细落库后的重扫请求(端口实现):异步合并执行,不阻塞、不抛错。 */
@Override
public void requestRefresh(String reason) {
refreshScheduler.request(reason);
}
/** 调度器单次扫描动作:锁被占返回 LOCK_BUSY 供其重试;失败只记日志(FAILED 行已落库)。 */
private DuplicateCheckRefreshScheduler.Outcome runRefreshOnce() {
try {
scanNow();
return DuplicateCheckRefreshScheduler.Outcome.DONE;
} catch (BusinessException ex) {
if (ex.getCode() != null && ex.getCode() == 409) {
log.info("[shop-duplicate-check] 自动重扫未执行:其它扫描进行中 msg={}", ex.getMessage());
return DuplicateCheckRefreshScheduler.Outcome.LOCK_BUSY;
}
log.warn("[shop-duplicate-check] 自动重扫失败 code={} msg={}", ex.getCode(), ex.getMessage());
return DuplicateCheckRefreshScheduler.Outcome.FAILED;
} catch (Exception ex) {
log.error("[shop-duplicate-check] 自动重扫异常", ex);
return DuplicateCheckRefreshScheduler.Outcome.FAILED;
}
}
/** 最新 SUCCESS 扫描视图;无扫描结果返回 null。 */
public DuplicateScanView loadLatest() {
ScanLightRowDto light = scanMapper.selectLatestLightRow();
@@ -0,0 +1,89 @@
package com.nanri.aiimage.modules.shopduplicatecheck.service;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.modules.shopduplicatecheck.mapper.ShopDataDuplicateScanMapper;
import com.nanri.aiimage.modules.shopduplicatecheck.model.dto.ScanLightRowDto;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.time.Duration;
import java.time.LocalDateTime;
/**
* 撞款扫描结果表shop_data_duplicate_scan的保留期清理
*
* <p>扫描每晚自动跑一轮管理员还能手动触发每次插入一行**含整份聚合 payload 的结果**
* 但读取侧只认最新一行{@code selectLatestLightRow} / {@code selectLatestFullRow} 都是
* {@code ORDER BY id DESC LIMIT 1}历史行纯属占用磁盘此前无任何删除路径
*
* <p>保护策略无论多旧始终保留按 id 最新的若干行
* 万一扫描停摆很久界面上仍能看到最后一份结果而不是被清理任务顺手抹掉
*/
@Service
@RequiredArgsConstructor
@Slf4j
public class ShopDataDuplicateScanRetentionService {
/** 单轮最多删除的批次数,剩余留给下一轮。 */
static final int MAX_BATCHES_PER_RUN = 20;
/** 无论时间多久都保留的最新行数(最新 SUCCESS 行必然在其中)。 */
static final int PROTECTED_ROWS = 10;
private final ShopDataDuplicateScanMapper scanMapper;
private final DistributedJobLockService distributedJobLockService;
@Value("${aiimage.shop-duplicate-check.scan-retention-days:7}")
private int retentionDays = 7;
@Value("${aiimage.shop-duplicate-check.scan-retention-batch-size:200}")
private int retentionBatchSize = 200;
@Scheduled(cron = "${aiimage.shop-duplicate-check.scan-retention-cron:0 50 3 * * *}")
public void purgeExpiredScans() {
int days = Math.max(1, retentionDays);
int batchSize = Math.max(20, retentionBatchSize);
LocalDateTime cutoff = LocalDateTime.now().minusDays(days);
DistributedJobLockService.LockHandle lockHandle =
distributedJobLockService.tryLock("shop-duplicate-check:scan-retention", Duration.ofMinutes(15));
if (lockHandle == null) {
log.info("[shop-duplicate-check] 扫描结果保留清理跳过:另一实例持锁");
return;
}
try (lockHandle) {
// 两条保护线取更靠前更小的那个 id保护线之内的行一律不删
// 最新的 PROTECTED_ROWS 万一扫描停摆很久界面上仍能看到最后一份结果
// 最新 SUCCESS 读侧只认它连续失败多日时它可能已滑出保护窗口
// 按时间线会被删掉界面直接空白所以单独兜一条
Long nthNewestId = scanMapper.selectNthNewestId(PROTECTED_ROWS - 1);
ScanLightRowDto latestSuccess = scanMapper.selectLatestLightRow();
long protectFromId = minId(nthNewestId, latestSuccess == null ? null : latestSuccess.getId());
int totalDeleted = 0;
int batches = 0;
while (batches < MAX_BATCHES_PER_RUN) {
int deleted = scanMapper.deleteOlderThanBatch(cutoff, protectFromId, batchSize);
batches++;
totalDeleted += deleted;
if (deleted < batchSize) {
break;
}
}
log.info("[shop-duplicate-check] 扫描结果保留清理完成 cutoff={} retentionDays={} protectFromId={} deleted={} batches={}",
cutoff, days, protectFromId, totalDeleted, batches);
} catch (Exception ex) {
log.warn("[shop-duplicate-check] 扫描结果保留清理失败 cutoff={} msg={}", cutoff, ex.getMessage(), ex);
}
}
/** 取两条保护线里更靠前(更小)的 id;都为 null 表示无需保护(等价于不设限)。 */
private static long minId(Long first, Long second) {
if (first == null) {
return second == null ? Long.MAX_VALUE : second;
}
return second == null ? first : Math.min(first, second);
}
}
@@ -0,0 +1,138 @@
package com.nanri.aiimage.modules.shopduplicatecheck.service.support;
import com.nanri.aiimage.common.util.ThreadPools;
import lombok.extern.slf4j.Slf4j;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.Supplier;
/**
* 采集落库触发的撞款重扫调度器合并窗口去抖 + 单飞 + 锁忙重试
*
* <p>语义{@link #request} 只置位并异步执行永不阻塞调用方永不向外抛错
* 合并窗口内的多次触发聚合为一次扫描扫描动作执行期间到达的触发在下一轮执行
* 扫描因分布式锁被占用未执行{@link Outcome#LOCK_BUSY}时按固定间隔重试有限次
*/
@Slf4j
public class DuplicateCheckRefreshScheduler {
/** 单次扫描动作的终态:完成 / 锁被占(可重试)/ 失败(不重试,等下次触发或定时扫描)。 */
public enum Outcome {
DONE, LOCK_BUSY, FAILED
}
private static final long DEFAULT_DEBOUNCE_MILLIS = 10_000L;
private static final long DEFAULT_LOCK_RETRY_MILLIS = 20_000L;
private static final int DEFAULT_MAX_LOCK_RETRIES = 6;
private final Supplier<Outcome> scanAction;
private final long debounceMillis;
private final long lockRetryMillis;
private final int maxLockRetries;
private final ExecutorService executor;
private final AtomicBoolean pending = new AtomicBoolean(false);
private final AtomicBoolean running = new AtomicBoolean(false);
public DuplicateCheckRefreshScheduler(Supplier<Outcome> scanAction) {
this(scanAction, DEFAULT_DEBOUNCE_MILLIS, DEFAULT_LOCK_RETRY_MILLIS, DEFAULT_MAX_LOCK_RETRIES);
}
/** 测试用:注入更短的窗口与重试参数。 */
DuplicateCheckRefreshScheduler(Supplier<Outcome> scanAction, long debounceMillis,
long lockRetryMillis, int maxLockRetries) {
this.scanAction = scanAction;
this.debounceMillis = Math.max(0L, debounceMillis);
this.lockRetryMillis = Math.max(0L, lockRetryMillis);
this.maxLockRetries = Math.max(0, maxLockRetries);
this.executor = ThreadPools.boundedFixed("shop-dup-refresh", 1, 8);
}
/** 请求一次重扫(异步、去抖合并)。调用方不被阻塞,也不会收到异常。 */
public void request(String reason) {
pending.set(true);
if (running.compareAndSet(false, true)) {
submit(reason);
}
}
private void submit(String reason) {
try {
log.info("[shop-duplicate-check] 触发撞款重扫(异步合并执行,窗口={}ms) reason={}", debounceMillis, reason);
executor.execute(this::drain);
} catch (Exception ex) {
// 提交失败如线程池拒绝时复位单飞标记避免后续触发被永久吞掉
running.set(false);
log.warn("[shop-duplicate-check] 撞款重扫任务提交失败 reason={} msg={}", reason, ex.getMessage());
}
}
private void drain() {
try {
while (true) {
// 合并窗口窗口内到达的多次触发聚合为同一轮扫描
if (!sleepQuietly(debounceMillis)) {
return;
}
if (!pending.compareAndSet(true, false)) {
return;
}
int lockRetries = 0;
while (true) {
Outcome outcome = runOnceSafely();
if (outcome != Outcome.LOCK_BUSY) {
break;
}
if (lockRetries >= maxLockRetries) {
log.warn("[shop-duplicate-check] 撞款重扫连续 {} 次未取得扫描锁,放弃本轮(等待下次触发或定时扫描)",
lockRetries + 1);
break;
}
lockRetries++;
log.info("[shop-duplicate-check] 撞款重扫未取得扫描锁,{}ms 后重试(第 {}/{} 次)",
lockRetryMillis, lockRetries, maxLockRetries);
if (!sleepQuietly(lockRetryMillis)) {
return;
}
}
}
} finally {
running.set(false);
// 竞态兜底running 复位前到达的触发可能没能提交补一次
if (pending.get() && running.compareAndSet(false, true)) {
submit("race-guard");
}
}
}
/** 执行一次扫描动作;动作自身异常也被吸收(调度器对外零抛出)。 */
private Outcome runOnceSafely() {
long startedAt = System.currentTimeMillis();
try {
Outcome outcome = scanAction.get();
long elapsed = System.currentTimeMillis() - startedAt;
if (outcome == Outcome.DONE) {
log.info("[shop-duplicate-check] 采集后自动重扫完成 耗时={}ms", elapsed);
} else if (outcome == Outcome.FAILED) {
log.warn("[shop-duplicate-check] 采集后自动重扫失败 耗时={}ms", elapsed);
}
return outcome == null ? Outcome.FAILED : outcome;
} catch (Exception ex) {
log.error("[shop-duplicate-check] 采集后自动重扫异常", ex);
return Outcome.FAILED;
}
}
private static boolean sleepQuietly(long millis) {
if (millis <= 0) {
return true;
}
try {
Thread.sleep(millis);
return true;
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
return false;
}
}
}
@@ -1,5 +1,6 @@
package com.nanri.aiimage.modules.shopkey.model.entity;
import com.baomidou.mybatisplus.annotation.FieldStrategy;
import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableField;
import com.baomidou.mybatisplus.annotation.TableId;
@@ -39,6 +40,12 @@ public class QueryAsinEntity {
@TableField("created_at")
private LocalDateTime createdAt;
@TableField("updated_at")
/**
* 更新时间由数据库维护DEFAULT CURRENT_TIMESTAMP + ON UPDATE CURRENT_TIMESTAMP
*
* <p>禁止应用显式写本表更新走 selectById 改字段 updateById实体带着旧值
* 显式写回旧值会触发 MySQL 显式赋值不自动更新规则把更新时间冻结在首次写入时刻
*/
@TableField(value = "updated_at", insertStrategy = FieldStrategy.NEVER, updateStrategy = FieldStrategy.NEVER)
private LocalDateTime updatedAt;
}

Some files were not shown because too many files have changed in this diff Show More