Compare commits

...

52 Commits

Author SHA1 Message Date
huangzd1997 152ea6eec0 下线店铺管理-检测店铺密码功能:后台 UI 按钮/徽标、Python 代理路由、Java 接口与 service 全套移除,V105 迁移删除 biz_shop_credential_check 表及历史记录
Build Backend JAR / build (push) Has been cancelled
2026-09-02 16:55:54 +08:00
huangzd1997 664047dbe0 修复店铺数据抓取新店铺归档 daily_file 报 user_id 无默认值
线上现象:店铺数据抓取任务 26401 失败,错误
"Field 'user_id' doesn't have a default value",
INSERT 列清单不含 user_id(MyBatis-Plus 对 null 字段跳过该列)。

根因链:
1. V95(97ae8a7)累计文件改为店铺级共享(去 user 维度),
   persistDailyAggregation 新建 daily_file 时删掉了 setUserId;
2. 但 V95 只改了唯一键为 (shop_key, business_date),未把 user_id 列改可空;
3. 表列 user_id 仍为 NOT NULL 无默认值 → 新店铺首次插入必报错;
   已有 daily_file 行的店铺走 update 分支不受影响,因此只在
   新店铺(如 26401)出现。

修复:
- persistDailyAggregation 新建 daily_file 时补回 setUserId(row.getUserId()),
  新店铺也写入正确归属;
- V104 迁移把 user_id 改可空,兼容存量/历史数据(保留兼容,不再使用);
- 新增回归断言(新建累计文件必须写入 user_id),
  shopdatacrawl 全包 255 测试全绿,双机已部署验证。
2026-09-02 14:23:48 +08:00
huangzd1997 1aea4674e3 修复货源查询 404 行被过滤且状态列置空:Python 404 的 ASIN 保留导出并标记「ASIN未找到」
线上现象:货源采集任务 26402(1212.xlsx)结果文件中存在 Python 端
404 Not Found 的行没有图,且状态列空白——这些行实际已完成采集但被后端过滤。

根因链:
1. Python 404 行 title/url/图全空 → isMissingRequired 必填校验拦截 →
   markRowsFailed(status=FAILED,error 保留 "404 Not Found");
2. isExportableResultRow 对「其它错误需带 title/url/图」的判断把这类 404 行
   过滤掉,即使存在也被 isSilentOutputFailure 把状态列置空。

修复:
- 新增 isAsinNotFound 判定(404 / not found / 商品未找到 等文案,大小写不敏感);
- resolveResultStatus 在图片判定后、rawStatus→失败 映射前插入
  「ASIN未找到」状态;isExportableResultRow 对 404 行直接放行;
- writeResultWorkbook 状态列 gate 排除 404 行(不再被置空);
- 新增 2 条单元测试(404/中文文案命中、普通技术失败不误判)。
2026-09-02 13:47:58 +08:00
huangzd1997 69eb825d29 修复采集数据结果明细 module_type 写读大小写不一致:写入端误用小写 collectdata 导致 task-50 之后所有采集任务明细行生成结果文件时读不到
线上现象:任务 26334 采集完成(去重 42/无效品牌 62/品牌拒绝 6/保留 85)
但结果文件「采集数据结果」sheet 只有表头 0 行,历史卡片「保留 0」。

根因链:
1. CollectDataResultItemBatchWriter(task-50 新增写入端)常量误写为小写
   "collectdata",与 CollectDataService.MODULE_TYPE(大写 COLLECT_DATA)不一致;
2. loadFinalRows 按大写查 biz_task_result_item 命中 0 行 → 组装空表 →
   stats.finalRowCount 被 rows.size()=0 覆盖,UI 显示保留 0;
3. 生产库 5386 行小写 collectdata(8/31 起约 15 个任务)全部读不到,
   大写 COLLECT_DATA 仅存 1067 行历史数据。

修复:
- 写入常量改回大写 COLLECT_DATA(与读取端一致),现有行查询 in(大写,小写)
  兼容存量,旧任务重试幂等命中不重复插入;
- loadFinalRows / deleteTask / deleteHistory 查询改 in(MODULE_TYPE, LEGACY_MODULE_TYPE),
  历史小写行可读出并正确清理(RustFS 对象不泄漏);
- 新增 2 条回归测试(写入 module_type 必须与读取端常量一致、小写存量行重试命中),
  collectdata 全包 168 测试全绿。
2026-09-02 13:47:52 +08:00
huangzd1997 162ed7e406 task-168: LLM 客户端配置接入(LlmHttpConfigResolver 从 aiimage.http-client.* 读取超时/重试,默认值同现状、钳制、可覆盖)+ 8 条测试 2026-09-02 08:16:12 +08:00
huangzd1997 8ca5b1d53b task-167: aiimage.http-client.* 配置命名空间(connect/read/call 超时与重试默认值同现状、非法值钳制、env 覆盖)+ 8 条测试;surefire 加 MaxMetaspaceSize 修复 fork 内存崩溃 2026-09-02 08:00:19 +08:00
huangzd1997 f2736729fa task-166: HTTP 客户端契约表盘点(45 项超时/重试配置、198 个调用点扫描器 + 8 条测试 + 报告快照,不改生产代码) 2026-09-02 07:43:27 +08:00
huangzd1997 bde9e75eed task-165: 巡检只读契约(巡检器无删除/状态修改/写库方法集合断言、调度只调 inspect*、可重复、报表隔离)+ 8 条测试 2026-09-02 07:42:10 +08:00
huangzd1997 34f614ea32 task-164: 清理失败不阻断契约(单文件失败其余继续、失败路径可见、不上抛、下轮重试、引用跳过不计失败)+ 8 条测试 2026-09-02 07:35:57 +08:00
huangzd1997 7a43ed6c43 task-163: 引用中文件不清契约(job/result 引用跳过、引用解除后可清、判定失败保守跳过、批量混合、幂等)+ 8 条测试 2026-09-02 07:30:53 +08:00
huangzd1997 0e4d680aef task-162: 路径穿越输入矩阵(../、..\、绝对路径、%2e 编码、全角点、混合分隔符、空字节全部拒绝)+ 9 条测试 2026-09-02 07:25:49 +08:00
huangzd1997 035ec9ac79 task-161: 巡检任务开关(aiimage.inspection.enabled 默认 disabled、cron 调度、分布式锁双实例去重、单巡检异常隔离、limit 透传)+ 8 条测试 2026-09-02 07:19:56 +08:00
huangzd1997 9712bdf538 task-160: 超保留期仍被引用巡检(超期+引用检出供人工处理、未超期/未引用不报、只读、可重复)+ 8 条测试 2026-09-02 07:14:05 +08:00
huangzd1997 5305d6dbc2 task-159: 已完成任务活跃 Job 巡检(终态任务 RUNNING/PENDING Job 检出、运行中任务正常、只读、可重复)+ 8 条测试 2026-09-02 07:08:53 +08:00
huangzd1997 9998571770 task-158: 结果存在文件缺失巡检(resultFileUrl 对应 OSS 文件缺失检出、空白 URL 不告警、只读、可重复)+ 8 条测试 2026-09-02 07:04:06 +08:00
huangzd1997 9144bce33f task-157: 任务存在结果缺失巡检(终态任务无 file_result 检出、运行中忽略、只读、可重复)+ 8 条测试 2026-09-02 06:57:50 +08:00
huangzd1997 d773748ae9 task-156: 孤立 Job 巡检报表(task/result 缺失孤儿检出、重试耗尽同样报、只读不修改、可重复)+ 8 条测试 2026-09-02 06:53:06 +08:00
huangzd1997 d64ef5284a task-155: 孤立文件巡检报表(只读巡检超保留期且无引用文件,报表含路径/大小/时间,绝不删除,可重复)+ 8 条测试 2026-09-02 06:47:03 +08:00
huangzd1997 87a6d20f88 task-154: 临时目录磁盘容量告警(aiimage.storage.capacity-warn-bytes 阈值、告警含当前容量/阈值、10 分钟频率限制、测量失败容忍不阻塞)+ 8 条测试 2026-09-02 06:40:20 +08:00
huangzd1997 57f75b8bc3 task-153: fileKey 兜底查找契约(保存即登记、重启后目录枚举重建、缺失明确失败、清理后不可找、语义稳定)+ 8 条测试 2026-09-02 06:34:22 +08:00
huangzd1997 3b34ec3bef task-152: 清理失败吞异常记指标(aiimage.temp-cleanup.failed 计数、日志+指标均不阻断主流程、按路径隔离、无重试循环)+ 8 条测试 2026-09-02 06:29:59 +08:00
huangzd1997 f0e12b9b1b task-151: 路径安全校验(PathSafetyGuard 规范化 startsWith 判定、../与绝对路径与符号链接逃逸拒绝、接入清理路径)+ 8 条测试 2026-09-02 06:24:37 +08:00
huangzd1997 ffc3963345 task-150: 清理前引用检查契约(chunk 共享/scope 引用保留、自身行可清、异常保守保留、只读无副作用、批量 IN 反查)+ 8 条测试 2026-09-02 06:19:42 +08:00
huangzd1997 dd569edc66 task-149: 临时文件时间元数据补缺(TempFileMetadata 创建/修改/访问时间与大小读取,mtime 回退不改变清理语义,IO 错误容忍)+ 8 条测试 2026-09-02 06:08:50 +08:00
huangzd1997 383ec544e1 task-148: 临时目录配置盘点(16 配置项/322 清理调用点扫描器 + 9 条测试 + 报告快照,不改生产代码) 2026-09-02 06:02:14 +08:00
huangzd1997 dc90ead263 task-147: downloadUrl 永不过期验证(result/ 走下载域名、无签名参数、bucket 前缀、null/空白安全、输出稳定)+ 8 条测试 2026-09-02 06:00:53 +08:00
huangzd1997 eced873779 task-146: collectdata 自洽行为契约(后端无 file 字段 VO 快照、前端 success+downloadUrl 判定含空白边界)+ 前后端 13 条测试 2026-09-02 05:55:43 +08:00
huangzd1997 0945882458 task-145: 前端可下载判定契约(fileReady/downloadUrl/终态 SUCCESS+FAILED 可下载,PENDING/RUNNING/无 resultId 不可下载)+ 8 条测试 2026-09-02 05:49:01 +08:00
huangzd1997 6c674f0f1d task-144: 前端 live progress 合并补强(mergeHistoryItemPreservingLiveProgress 纯函数:缺失保留/有效覆盖/可下载不合并/幂等)+ 11 条测试 2026-09-02 05:48:09 +08:00
huangzd1997 5e5056b99d task-143: publish 文件级 file 状态语义契约(ready 由 result.success+url 推导、job 字段独立、多文件混合、无结果 null)+ 8 条测试 2026-09-02 05:46:42 +08:00
huangzd1997 9e58d28a99 task-142: fileReady/fileStatus 语义契约(URL 判定、无 job SUCCESS/null、job 状态与错误映射、空白边界)+ 8 条测试 2026-09-02 05:42:32 +08:00
huangzd1997 89d5eda057 task-141: similarasin file 进度合成单测(LLM/Python 上传/snapshot 三路径、fileReady 恒 100%、除零与状态边界)+ 8 条测试 2026-09-02 05:38:26 +08:00
huangzd1997 b7a403dc22 task-140: 13 模块 file 字段契约快照(fileStatus/fileError/fileReady 类型与白名单冻结、旧字段仍在、collectdata 自洽)+ 8 条守门测试 2026-09-02 05:32:06 +08:00
huangzd1997 4fb33bda1c task-138: 事务时长性能对比基准(prepare/persist 分段测量、200 分片负载上界、文档基线守门不劣化)+ 8 条基准测试 2026-09-02 05:20:54 +08:00
huangzd1997 2493019c27 task-137: 重复提交幂等契约(同 id 幂等接受/不重复落库/终态拒绝/并发撞键不覆盖/多次安全)+ 8 条守门测试 2026-09-02 05:15:18 +08:00
huangzd1997 58cfcfdf35 task-136: 事务外清理异常不阻断契约(响应不受影响/不重试循环/事务已提交/下次可再清理/部分失败继续)+ 8 条守门测试 2026-09-02 05:07:33 +08:00
huangzd1997 a982da0262 task-135: 任务状态成功时机契约(SUCCESS 必在文件生成上传后、无文件不成功、fileReady/downloadUrl 语义不变)+ 8 条守门测试 2026-09-02 05:03:05 +08:00
huangzd1997 c8d14a5b1b task-134: done=true 封口时机契约(分片先落库再强制 finalize、剩余全量行数、result 不重复创建、done=false 只触心跳)+ 8 条守门测试 2026-09-02 04:57:56 +08:00
huangzd1997 44fc43912f task-133: 回滚语义硬约束契约(完整回滚/状态不改写/payload 保留/重试安全/计算幂等/双模块一致)+ 8 条守门测试 2026-09-02 04:51:16 +08:00
huangzd1997 cf83752338 task-132: /result 成功确认时机契约(四模块 200+success=true 必在落库后、失败不成功、重试安全)+ 8 条守门测试 2026-09-02 04:44:21 +08:00
huangzd1997 c60ad48be5 task-131: collectdata 事务收缩契约固化(/result 锁内单条写无长事务 + 响应 DTO 写后组装)+ 8 条守门测试 2026-09-02 04:37:14 +08:00
huangzd1997 29ddf5c780 task-130: publish 事务收缩契约固化(prepare 外置/短事务落库/提交后清理已符合 spec 07)+ 8 条守门测试 2026-09-02 04:25:14 +08:00
huangzd1997 5e71e1bb3d task-129: shopdatacrawl 事务收缩(/result 无长事务契约固化 + deleteTask 纯计算抽静态纯函数)+ 8 条边界测试 2026-09-02 04:15:44 +08:00
huangzd1997 ecc62f095b task-128: appearancepatent 事务收缩(flatten/序列化/存储/哈希纯计算抽 prepareSubmittedChunk 移出事务,双事务路径不变)+ 8 条边界测试 2026-09-02 03:54:29 +08:00
huangzd1997 28f61eb70d task-127: similarasin 临时文件清理时机契约固化(提交后清理/回滚不清理/异常吞掉/可重试/幂等)+ 8 条守门测试 2026-09-02 03:44:38 +08:00
huangzd1997 ef4ea8d24b task-126: similarasin DTO 组装位置契约固化(审计结论:/result 无响应 DTO、可移出组装已在事务外、落库行组装顺序不变)+ 8 条守门测试 2026-09-02 03:39:54 +08:00
huangzd1997 79b3efd365 task-125: similarasin /result 纯计算移出事务(payload 哈希在 prepare 阶段预计算,落库直接引用)+ 9 条事务边界守门测试 2026-09-02 03:34:15 +08:00
huangzd1997 56afee0386 task-124: @Transactional 方法审计清单(147 方法/30 模块扫描器 + 13 条测试 + 报告快照,不改生产代码) 2026-09-02 03:24:23 +08:00
huangzd1997 c8250e5320 task-123: dead-code-cleanup diff 守卫仅校验前端提交(后端提交不再误报越界) 2026-09-02 03:20:44 +08:00
huangzd1997 793bd4f9c8 task-123: 性能基准堆采样前先 GC,消除全量套件下残留垃圾导致的 1GB 边界 flaky 2026-09-02 03:20:44 +08:00
huangzd1997 445b7780c5 task-122: N+1 批量化修复(claim/stuck-reset/stale-check 逐行 selectById 改 selectBatchIds 批量回读 + Map 装配,行为等价) 2026-09-02 03:13:05 +08:00
huangzd1997 29625f703c task-121: N+1 扫描审计(循环内单查扫描器 + 10 条测试 + 报告快照,不改生产代码) 2026-09-02 02:40:16 +08:00
108 changed files with 17188 additions and 870 deletions
+1 -1
View File
@@ -183,7 +183,7 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<argLine>-XX:+EnableDynamicAgentLoading -Xshare:off -Xmx1536m</argLine>
<argLine>-XX:+EnableDynamicAgentLoading -Xshare:off -Xmx2g -XX:MaxMetaspaceSize=512m</argLine>
</configuration>
</plugin>
</plugins>
File diff suppressed because it is too large Load Diff
+117
View File
@@ -0,0 +1,117 @@
"""task-166 HTTP 客户端契约盘点工具。
盘点 backend-java 各 HTTP 客户端(OSS/RustFS/图片下载/紫鸟/LLM)的现状:
超时(connect/read/write/call)、重试、幂等性,产出机器可读契约表供审计文档使用。
本工具只读,不改代码。
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from dataclasses import dataclass, field
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
CONFIG_ROOT = REPO_ROOT / "src/main/java/com/nanri/aiimage/config"
MAIN_ROOT = REPO_ROOT / "src/main/java"
# 超时/重试配置字段(config 包)
TIMEOUT_FIELD_RE = re.compile(
r"private\s+\w+\s+(\w*(?:Timeout|Retries|Retry|MaxRetries)\w*)\s*=\s*([^;]+);",
re.IGNORECASE,
)
# 代码中 HttpClient/Client 构建的超时/重试配置
CLIENT_CONF_RE = re.compile(
r"(connectTimeout|readTimeout|writeTimeout|callTimeout|operationTimeout|"
r"maxRetries|retry|setConnectTimeout|setReadTimeout|HttpClient\.newBuilder)",
re.IGNORECASE,
)
# 客户端类识别
CLIENT_CLASS_RE = re.compile(
r"(OssStorageService|RustfsObjectStorageService|ZiniaoShopSwitchService|"
r"SimilarAsinLlmService|AppearancePatentLlmClient|ImageDownloader|ImageDownload|"
r"imageDownload|HttpClient|WebClient|RestTemplate)",
)
def extract_timeout_configs() -> list[dict]:
"""从 config 包提取超时/重试配置字段。"""
entries: list[dict] = []
for java_file in sorted(CONFIG_ROOT.rglob("*.java")):
text = java_file.read_text(encoding="utf-8")
for match in TIMEOUT_FIELD_RE.finditer(text):
entries.append(
{
"config_class": java_file.stem,
"field": match.group(1),
"default": match.group(2).strip(),
"source": str(java_file.relative_to(REPO_ROOT)).replace("\\", "/"),
}
)
return entries
def extract_client_sites() -> list[dict]:
"""扫描 main 源码中客户端构建/配置调用点。"""
sites: list[dict] = []
for java_file in sorted(MAIN_ROOT.rglob("*.java")):
text = java_file.read_text(encoding="utf-8")
for line_no, line in enumerate(text.splitlines(), start=1):
if not CLIENT_CONF_RE.search(line) or "import " in line:
continue
if CLIENT_CLASS_RE.search(line) or "Timeout" in line or "Retry" in line:
sites.append(
{
"file": str(java_file.relative_to(REPO_ROOT)).replace(
"\\", "/"
),
"line": line_no,
"snippet": line.strip()[:110],
}
)
return sites
@dataclass
class Report:
timeout_configs: list[dict] = field(default_factory=list)
client_sites: list[dict] = field(default_factory=list)
def to_dict(self) -> dict:
return {
"timeout_configs": self.timeout_configs,
"client_sites": self.client_sites,
}
def scan() -> Report:
return Report(
timeout_configs=extract_timeout_configs(), client_sites=extract_client_sites()
)
def main() -> None:
parser = argparse.ArgumentParser(description="HTTP 客户端契约盘点")
parser.add_argument("--json", help="输出 JSON 报告路径(默认 stdout")
args = parser.parse_args()
report = scan()
payload = report.to_dict()
if args.json:
Path(args.json).write_text(
json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
)
else:
print(json.dumps(payload, ensure_ascii=False, indent=2))
print(
f"timeout configs: {len(payload['timeout_configs'])}, "
f"client sites: {len(payload['client_sites'])}",
file=sys.stderr,
)
if __name__ == "__main__":
main()
+317
View File
@@ -0,0 +1,317 @@
{
"scanned_files": 640,
"modules": [
"admin",
"appearancepatent",
"auth",
"brand",
"collectdata",
"convert",
"debug",
"dedupe",
"deletebrand",
"digitalhuman",
"file",
"filetemplate",
"imagehistory",
"imagevideo",
"invalidasin",
"patroldelete",
"permission",
"pricetrack",
"productcategory",
"productrisk",
"publish",
"queryasin",
"shopdatacrawl",
"shopkey",
"shopmatch",
"similarasin",
"split",
"task",
"withdraw",
"ziniao"
],
"findings": [
{
"file": "deletebrand\\service\\DeleteBrandStaleTaskService.java",
"line": 209,
"statement": "FileTaskEntity refreshed = fileTaskMapper.selectById(task.getId());",
"loop_type": "for",
"impact": "HIGH",
"module": "deletebrand"
},
{
"file": "pricetrack\\service\\PriceTrackTaskService.java",
"line": 354,
"statement": "FileResultEntity lockedResult = fileResultMapper.selectById(fr.getId());",
"loop_type": "for",
"impact": "HIGH",
"module": "pricetrack"
},
{
"file": "productrisk\\service\\ProductRiskTaskService.java",
"line": 339,
"statement": "FileResultEntity lockedResult = fileResultMapper.selectById(fr.getId());",
"loop_type": "for",
"impact": "HIGH",
"module": "productrisk"
},
{
"file": "task\\service\\TaskFileJobService.java",
"line": 282,
"statement": "publishDispatchEvent(taskFileJobMapper.selectById(job.getId()));",
"loop_type": "for",
"impact": "HIGH",
"module": "task"
},
{
"file": "task\\service\\TaskFileJobService.java",
"line": 303,
"statement": "TaskFileJobEntity exhausted = taskFileJobMapper.selectById(job.getId());",
"loop_type": "for",
"impact": "HIGH",
"module": "task"
},
{
"file": "task\\service\\TaskFileJobService.java",
"line": 315,
"statement": "publishDispatchEvent(taskFileJobMapper.selectById(job.getId()));",
"loop_type": "for",
"impact": "HIGH",
"module": "task"
},
{
"file": "appearancepatent\\service\\AppearancePatentTaskService.java",
"line": 1094,
"statement": "TaskChunkEntity chunk = taskChunkMapper.selectOne(new LambdaQueryWrapper<TaskChunkEntity>()",
"loop_type": "for",
"impact": "LOW",
"module": "appearancepatent"
},
{
"file": "appearancepatent\\service\\AppearancePatentTaskService.java",
"line": 2112,
"statement": "List<TaskChunkEntity> page = taskChunkMapper.selectList(query);",
"loop_type": "while",
"impact": "LOW",
"module": "appearancepatent"
},
{
"file": "dedupe\\service\\DedupeTotalDataService.java",
"line": 279,
"statement": "List<DedupeTotalDataEntity> rows = dedupeTotalDataMapper.selectList(",
"loop_type": "while",
"impact": "LOW",
"module": "dedupe"
},
{
"file": "deletebrand\\service\\DeleteBrandRunService.java",
"line": 1234,
"statement": "tasks.addAll(fileTaskMapper.selectList(new LambdaQueryWrapper<FileTaskEntity>()",
"loop_type": "for",
"impact": "LOW",
"module": "deletebrand"
},
{
"file": "deletebrand\\service\\DeleteBrandRunService.java",
"line": 1252,
"statement": "rows.addAll(fileResultMapper.selectList(historyResultQuery()",
"loop_type": "for",
"impact": "LOW",
"module": "deletebrand"
},
{
"file": "deletebrand\\service\\DeleteBrandRunService.java",
"line": 1268,
"statement": "List<FileResultEntity> rows = fileResultMapper.selectList(historyResultQuery()",
"loop_type": "for",
"impact": "LOW",
"module": "deletebrand"
},
{
"file": "patroldelete\\service\\PatrolDeleteTaskService.java",
"line": 638,
"statement": "List<FileTaskEntity> tasks = fileTaskMapper.selectList(new LambdaQueryWrapper<FileTaskEntity>()",
"loop_type": "for",
"impact": "LOW",
"module": "patroldelete"
},
{
"file": "permission\\service\\PermissionMenuService.java",
"line": 676,
"statement": "cursor = permissionMenuMapper.selectById(ancestorId);",
"loop_type": "while",
"impact": "LOW",
"module": "permission"
},
{
"file": "pricetrack\\service\\PriceTrackTaskService.java",
"line": 153,
"statement": "tasks.addAll(fileTaskMapper.selectList(new LambdaQueryWrapper<FileTaskEntity>()",
"loop_type": "for",
"impact": "LOW",
"module": "pricetrack"
},
{
"file": "pricetrack\\service\\PriceTrackTaskService.java",
"line": 418,
"statement": "List<FileTaskEntity> tasks = fileTaskMapper.selectList(new LambdaQueryWrapper<FileTaskEntity>()",
"loop_type": "for",
"impact": "LOW",
"module": "pricetrack"
},
{
"file": "productcategory\\service\\ProductCategoryService.java",
"line": 465,
"statement": "ProductCategoryEntity parent = productCategoryMapper.selectById(cursor.getParentId());",
"loop_type": "while",
"impact": "LOW",
"module": "productcategory"
},
{
"file": "productcategory\\service\\ProductCategoryService.java",
"line": 481,
"statement": "cursor = cursor.getParentId() == null ? null : productCategoryMapper.selectById(cursor.getParentId());",
"loop_type": "while",
"impact": "LOW",
"module": "productcategory"
},
{
"file": "productcategory\\service\\ProductCategoryService.java",
"line": 518,
"statement": "ProductCategoryEntity row = productCategoryMapper.selectById(cursor);",
"loop_type": "while",
"impact": "LOW",
"module": "productcategory"
},
{
"file": "productrisk\\service\\ProductRiskTaskService.java",
"line": 144,
"statement": "tasks.addAll(fileTaskMapper.selectList(new LambdaQueryWrapper<FileTaskEntity>()",
"loop_type": "for",
"impact": "LOW",
"module": "productrisk"
},
{
"file": "queryasin\\service\\QueryAsinTaskService.java",
"line": 587,
"statement": "List<FileTaskEntity> tasks = fileTaskMapper.selectList(new LambdaQueryWrapper<FileTaskEntity>()",
"loop_type": "for",
"impact": "LOW",
"module": "queryasin"
},
{
"file": "shopdatacrawl\\service\\ShopDataCrawlTaskService.java",
"line": 1085,
"statement": "List<FileTaskEntity> tasks = fileTaskMapper.selectList(new LambdaQueryWrapper<FileTaskEntity>()",
"loop_type": "for",
"impact": "LOW",
"module": "shopdatacrawl"
},
{
"file": "shopdatacrawl\\service\\ShopDataCrawlTaskService.java",
"line": 1940,
"statement": "FileResultEntity result = fileResultMapper.selectById(member.getResultId());",
"loop_type": "for",
"impact": "LOW",
"module": "shopdatacrawl"
},
{
"file": "shopdatacrawl\\service\\ShopDataCrawlTaskService.java",
"line": 2497,
"statement": "FileResultEntity result = fileResultMapper.selectById(member.getResultId());",
"loop_type": "for",
"impact": "LOW",
"module": "shopdatacrawl"
},
{
"file": "shopdatacrawl\\service\\ShopDataCrawlTaskService.java",
"line": 2575,
"statement": "FileTaskEntity task = fileTaskMapper.selectById(member.getTaskId());",
"loop_type": "for",
"impact": "LOW",
"module": "shopdatacrawl"
},
{
"file": "shopmatch\\service\\ShopMatchTaskService.java",
"line": 179,
"statement": "tasks.addAll(fileTaskMapper.selectList(new LambdaQueryWrapper<FileTaskEntity>()",
"loop_type": "for",
"impact": "LOW",
"module": "shopmatch"
},
{
"file": "similarasin\\service\\SimilarAsinTaskService.java",
"line": 1499,
"statement": "page = mapper.selectList(new com.baomidou.mybatisplus.core.conditions.query.QueryWrapper<TaskChunkEntity>()",
"loop_type": "while",
"impact": "LOW",
"module": "similarasin"
},
{
"file": "similarasin\\service\\SimilarAsinTaskService.java",
"line": 1646,
"statement": "TaskChunkEntity chunk = taskChunkMapper.selectOne(new LambdaQueryWrapper<TaskChunkEntity>()",
"loop_type": "for",
"impact": "LOW",
"module": "similarasin"
},
{
"file": "similarasin\\service\\SimilarAsinTaskService.java",
"line": 2346,
"statement": "FileTaskEntity task = fileTaskMapper.selectById(taskId);",
"loop_type": "for",
"impact": "LOW",
"module": "similarasin"
},
{
"file": "similarasin\\service\\SimilarAsinTaskService.java",
"line": 2356,
"statement": "List<TaskChunkEntity> chunks = taskChunkMapper.selectList(new LambdaQueryWrapper<TaskChunkEntity>()",
"loop_type": "for",
"impact": "LOW",
"module": "similarasin"
},
{
"file": "similarasin\\service\\SimilarAsinTaskService.java",
"line": 2379,
"statement": "TaskScopeStateEntity state = taskScopeStateMapper.selectById(stateId);",
"loop_type": "for",
"impact": "LOW",
"module": "similarasin"
},
{
"file": "similarasin\\service\\SimilarAsinTaskService.java",
"line": 4364,
"statement": "List<TaskChunkEntity> page = taskChunkMapper.selectList(query);",
"loop_type": "while",
"impact": "LOW",
"module": "similarasin"
},
{
"file": "similarasin\\service\\SimilarAsinTaskService.java",
"line": 4437,
"statement": "List<TaskScopeStateEntity> page = taskScopeStateMapper.selectList(query);",
"loop_type": "while",
"impact": "LOW",
"module": "similarasin"
},
{
"file": "task\\service\\ModuleHistoryCleanupService.java",
"line": 131,
"statement": "List<FileTaskEntity> page = fileTaskMapper.selectList(new LambdaQueryWrapper<FileTaskEntity>()",
"loop_type": "while",
"impact": "LOW",
"module": "task"
},
{
"file": "task\\service\\TaskFileJobService.java",
"line": 153,
"statement": "TaskFileJobEntity claim = taskFileJobMapper.selectById(candidate.getId());",
"loop_type": "for",
"impact": "LOW",
"module": "task"
}
]
}
+313
View File
@@ -0,0 +1,313 @@
"""task-121 N+1 扫描审计工具。
对 backend-java modules 下的 Java Service 源码做机械化扫描:
识别循环(for/while/forEach/map)内对 MyBatis-Plus 单行查询
selectById/selectOne/selectList/selectCount/selectObjs/selectMaps/getById/getOne
的调用,产出 N+1 候选清单(位置/对象/影响),供人工审计与 task-122 修复。
本工具只读,不改任何代码。
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from dataclasses import dataclass, field
from pathlib import Path
HIGH = "HIGH"
MEDIUM = "MEDIUM"
LOW = "LOW"
IMPACT_ORDER = {HIGH: 0, MEDIUM: 1, LOW: 2}
MODULES_ROOT = (
Path(__file__).resolve().parent.parent / "src/main/java/com/nanri/aiimage/modules"
)
# 单行查询调用(selectBatchIds/selectByIds 天然批量,不匹配此正则)
QUERY_RE = re.compile(
r"\.(selectById|selectOne|selectList|selectCount|selectObjs|selectMaps|getById|getOne)\("
)
# 循环头:for/while 关键字 + 圆括号跨度,或 .forEach( / .map( 方法调用
LOOP_RE = re.compile(r"\b(for|while)\s*\(|\.(forEach|map)\s*\(")
# 返回集合的查询调用(循环遍历这类结果集 → 影响 HIGH)
COLLECTION_QUERY_RE = re.compile(
r"(selectList|selectObjs|selectMaps|selectBatchIds|selectByIds|\.list\()"
)
ASSIGN_RE = re.compile(r"(\w+)\s*=\s*([^;{}=]+);")
FIELD_RE = re.compile(r"\b[A-Za-z_]\w*\b")
def _strip_comments(text: str) -> str:
"""用空格替换注释(保留换行与偏移),简化后续扫描。"""
out = list(text)
i, n = 0, len(text)
while i < n:
if text.startswith("/*", i):
j = text.find("*/", i + 2)
j = n if j < 0 else j + 2
for k in range(i, j):
if text[k] != "\n":
out[k] = " "
i = j
elif text.startswith("//", i):
j = text.find("\n", i)
j = n if j < 0 else j
for k in range(i, j):
out[k] = " "
i = j
else:
i += 1
return "".join(out)
def _match_span(text: str, open_pos: int) -> int | None:
"""从 text[open_pos] 的 ( 出发,返回匹配的 ) 偏移(含),不匹配返回 None。"""
depth = 0
for i in range(open_pos, len(text)):
c = text[i]
if c == "(":
depth += 1
elif c == ")":
depth -= 1
if depth == 0:
return i
return None
def _line_of(text: str, offset: int) -> int:
return text.count("\n", 0, offset) + 1
class _Frame:
__slots__ = ("body_depth", "expr", "loop_type", "start")
def __init__(self, start: int, body_depth: int, loop_type: str, expr: str):
self.start = start
self.body_depth = body_depth
self.loop_type = loop_type
self.expr = expr
def _query_results(stripped: str) -> set[str]:
"""收集被赋值给集合查询结果(selectList 等)的变量名。"""
names = set()
for lhs, rhs in ASSIGN_RE.findall(stripped):
if COLLECTION_QUERY_RE.search(rhs):
names.add(lhs)
return names
def _loop_headers(stripped: str) -> list[dict]:
"""预取全部循环头:(start, end, type, expr)start 为 ( 偏移,end 为匹配的 ) 偏移)。"""
headers: list[dict] = []
pos = 0
while True:
m = LOOP_RE.search(stripped, pos)
if not m:
break
paren_pos = m.end() - 1
if stripped[paren_pos] == "(":
close = _match_span(stripped, paren_pos)
if close is not None:
headers.append(
{
"start": paren_pos,
"end": close,
"type": m.group(1) or m.group(2),
"expr": stripped[paren_pos + 1 : close].strip(),
}
)
pos = m.end()
return headers
def _header_context(stripped: str, h: dict, offset: int) -> tuple[bool, str, str]:
"""查询落在循环头括号跨度内时的归属判定。
返回 (是否循环内, 循环类型, 循环表达式):
- lambda 箭头之后 → 循环内(无花括号 lambda 体)
- 增强 for 的 ` : ` 之后(迭代源 selectList 等)→ 集合装载,非循环内
- for/while 初始化或条件 → 视为循环内(条件逐次求值)
"""
expr = h["expr"]
if "->" in stripped[h["start"] : h["end"]]:
arrow = stripped.find("->", h["start"], h["end"])
if arrow != -1 and offset > arrow:
return True, h["type"], expr
return False, h["type"], expr
if " : " in expr:
colon = stripped.find(" : ", h["start"], h["end"])
if colon != -1 and offset > colon:
return False, h["type"], expr
return True, h["type"], expr
return True, h["type"], expr
def scan_text(text: str, filename: str = "fixture.java") -> list[dict]:
"""扫描单份 Java 源码,返回 N+1 候选清单(按影响降序、文件行号升序)。"""
stripped = _strip_comments(text)
query_results = _query_results(stripped)
headers = _loop_headers(stripped)
depth = 0
frames: list[_Frame] = []
single: tuple[str, str, int, int] | None = None # (loop_type, expr, start, end)
findings: list[dict] = []
def in_loop(offset: int) -> bool:
if single is not None and single[2] <= offset < single[3]:
return True
if any(f.start <= offset and depth >= f.body_depth for f in frames):
return True
for h in reversed(headers):
if h["start"] <= offset <= h["end"]:
return _header_context(stripped, h, offset)[0]
return False
def enclosing(offset: int) -> tuple[str, str]:
if single is not None and single[2] <= offset < single[3]:
return single[0], single[1]
for h in reversed(headers):
if h["start"] <= offset <= h["end"]:
in_loop, loop_type, expr = _header_context(stripped, h, offset)
if in_loop:
return loop_type, expr
for f in reversed(frames):
if f.start <= offset:
return f.loop_type, f.expr
return "loop", ""
i = 0
n = len(stripped)
while i < n:
c = stripped[i]
if c == "{":
depth += 1
i += 1
continue
if c == "}":
depth -= 1
if frames and depth == frames[-1].body_depth - 1:
frames.pop()
i += 1
continue
loop_match = LOOP_RE.match(stripped, i)
if loop_match:
loop_type = loop_match.group(1) or loop_match.group(2)
paren_pos = loop_match.end() - 1
close = (
_match_span(stripped, paren_pos) if stripped[paren_pos] == "(" else None
)
if close is not None:
expr = stripped[paren_pos + 1 : close].strip()
body_start = close + 1
while body_start < n and stripped[body_start].isspace():
body_start += 1
if body_start < n and stripped[body_start] == "{":
frames.append(_Frame(body_start + 1, depth + 1, loop_type, expr))
else:
end = stripped.find(";", body_start)
single = (loop_type, expr, body_start, n if end < 0 else end)
# 继续扫描括号跨度内容(无花括号 lambda 体的查询落在其中)
i = paren_pos + 1
continue
i = loop_match.end()
continue
query_match = QUERY_RE.match(stripped, i)
if query_match:
if in_loop(i):
line_start = stripped.rfind("\n", 0, i) + 1
line_end = stripped.find("\n", i)
line_end = n if line_end < 0 else line_end
statement = stripped[line_start:line_end].strip()
loop_type, expr = enclosing(i)
impact = _classify(loop_type, [expr], query_results)
findings.append(
{
"file": filename,
"line": _line_of(text, i),
"statement": statement,
"loop_type": loop_type,
"impact": impact,
}
)
i = query_match.end()
continue
i += 1
findings.sort(key=lambda f: (IMPACT_ORDER[f["impact"]], f["line"]))
return findings
def _classify(loop_type: str, exprs: list[str], query_results: set[str]) -> str:
"""影响分级:遍历查询结果集 → HIGHforEach/map lambda → MEDIUM;其余 → LOW。"""
for expr in exprs:
if expr and COLLECTION_QUERY_RE.search(expr):
return HIGH
if expr and any(t in query_results for t in FIELD_RE.findall(expr)):
return HIGH
if loop_type in ("forEach", "map"):
return MEDIUM
return LOW
@dataclass
class Report:
scanned_files: int = 0
modules: list[str] = field(default_factory=list)
findings: list[dict] = field(default_factory=list)
def to_dict(self) -> dict:
return {
"scanned_files": self.scanned_files,
"modules": self.modules,
"findings": self.findings,
}
def scan_modules(root: Path) -> Report:
"""扫描 modules 根目录下全部 Java 文件,产出 N+1 候选报告。"""
report = Report()
module_set: set[str] = set()
for java_file in sorted(root.rglob("*.java")):
report.scanned_files += 1
rel = java_file.relative_to(root)
module_set.add(rel.parts[0])
for finding in scan_text(java_file.read_text(encoding="utf-8"), str(rel)):
finding["module"] = rel.parts[0]
report.findings.append(finding)
report.findings.sort(
key=lambda f: (IMPACT_ORDER[f["impact"]], f["file"], f["line"])
)
report.modules = sorted(module_set)
return report
def main() -> None:
parser = argparse.ArgumentParser(description="N+1 扫描审计")
parser.add_argument("--json", help="输出 JSON 报告路径(默认 stdout")
args = parser.parse_args()
report = scan_modules(MODULES_ROOT)
payload = report.to_dict()
if args.json:
Path(args.json).write_text(
json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
)
else:
print(json.dumps(payload, ensure_ascii=False, indent=2))
print(
f"scanned {report.scanned_files} files, "
f"{len(report.findings)} N+1 candidates "
f"across {len(report.modules)} modules",
file=sys.stderr,
)
if __name__ == "__main__":
main()
File diff suppressed because it is too large Load Diff
+113
View File
@@ -0,0 +1,113 @@
"""task-148 临时目录配置盘点工具。
盘点 backend-java 的临时文件相关配置(application.yml 中的路径/前缀/保留期)
与既有清理逻辑调用点,产出机器可读清单供审计文档使用。本工具只读,不改代码。
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from dataclasses import dataclass, field
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
YML_PATH = REPO_ROOT / "src/main/resources/application.yml"
MAIN_ROOT = REPO_ROOT / "src/main/java"
# 临时文件相关配置键(application.yml 键名匹配)
CONFIG_KEY_RE = re.compile(
r"(local-temp-dir|retention-hours|retention-days|temp-dir|transient-payload|"
r"payload-|upload-|source-retention|result-retention|tmp|buffer-retention)",
re.IGNORECASE,
)
# 既有清理逻辑调用点(临时文件清理)
CLEANUP_RE = re.compile(
r"(deleteIfExists|Files\.delete|cleanup|deleteTemp|tempDir|transientPayloadStorageService\.delete|"
r"\.del\(|FileUtil\.del|cleanupPrepared|deletePayloadIfPresent)",
re.IGNORECASE,
)
def extract_configs() -> list[dict]:
"""从 application.yml 提取临时文件相关配置项。"""
if not YML_PATH.is_file():
return []
configs: list[dict] = []
for line_no, line in enumerate(
YML_PATH.read_text(encoding="utf-8").splitlines(), start=1
):
stripped = line.strip()
if not stripped or stripped.startswith("#"):
continue
match = re.match(r"^([\w.-]+):\s*(.*)$", stripped)
if not match:
continue
key, value = match.group(1), match.group(2).strip()
if CONFIG_KEY_RE.search(key):
configs.append(
{
"key": key,
"value": value,
"line": line_no,
"source": "application.yml",
}
)
return configs
def extract_cleanup_calls() -> list[dict]:
"""扫描 main 源码中临时文件清理调用点。"""
calls: list[dict] = []
for java_file in sorted(MAIN_ROOT.rglob("*.java")):
text = java_file.read_text(encoding="utf-8")
for line_no, line in enumerate(text.splitlines(), start=1):
if CLEANUP_RE.search(line) and "import " not in line:
calls.append(
{
"file": str(java_file.relative_to(REPO_ROOT)).replace(
"\\", "/"
),
"line": line_no,
"snippet": line.strip()[:100],
}
)
return calls
@dataclass
class Report:
configs: list[dict] = field(default_factory=list)
cleanup_calls: list[dict] = field(default_factory=list)
def to_dict(self) -> dict:
return {"configs": self.configs, "cleanup_calls": self.cleanup_calls}
def scan() -> Report:
return Report(configs=extract_configs(), cleanup_calls=extract_cleanup_calls())
def main() -> None:
parser = argparse.ArgumentParser(description="临时目录配置盘点")
parser.add_argument("--json", help="输出 JSON 报告路径(默认 stdout")
args = parser.parse_args()
report = scan()
payload = report.to_dict()
if args.json:
Path(args.json).write_text(
json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
)
else:
print(json.dumps(payload, ensure_ascii=False, indent=2))
print(
f"configs: {len(payload['configs'])}, cleanup calls: {len(payload['cleanup_calls'])}",
file=sys.stderr,
)
if __name__ == "__main__":
main()
@@ -0,0 +1,93 @@
"""task-166 HTTP 客户端契约盘点测试。
对应 plan 10 任务 166 的 8 条用例:
1. test_contract_table_all_clients 全客户端覆盖
2. test_timeouts_listed 超时现状
3. test_retry_listed 重试现状
4. test_idempotency_listed 幂等性标注
5. test_jdk_limits_noted 平台限制
6. test_doc_committed 文档存在
7. test_no_code_change 零代码变更
8. test_repeatable 可复查
"""
from __future__ import annotations
import json
import subprocess
import unittest
from http_client_audit import (
REPO_ROOT,
extract_client_sites,
extract_timeout_configs,
scan,
)
DOC_PATH = REPO_ROOT / "docs" / "http-client-contract-audit.md"
class HttpClientAuditTest(unittest.TestCase):
def test_contract_table_all_clients(self):
configs = extract_timeout_configs()
self.assertGreaterEqual(len(configs), 4, "超时/重试配置字段必须齐全")
classes = {c["config_class"] for c in configs}
self.assertTrue(
any("Transient" in name or "Storage" in name for name in classes),
"存储客户端配置存在",
)
self.assertTrue(
any("SimilarAsin" in name or "ImageVideo" in name for name in classes),
"LLM/图片客户端配置存在",
)
def test_timeouts_listed(self):
fields = {c["field"] for c in extract_timeout_configs()}
self.assertTrue(any("connectTimeout" in f for f in fields), "connect 超时缺失")
self.assertTrue(any("readTimeout" in f for f in fields), "read 超时缺失")
self.assertTrue(any("writeTimeout" in f for f in fields), "write 超时缺失")
self.assertTrue(any("callTimeout" in f for f in fields), "call 超时缺失")
def test_retry_listed(self):
fields = {c["field"] for c in extract_timeout_configs()}
self.assertTrue(any("Retry" in f for f in fields), "重试配置缺失")
def test_idempotency_listed(self):
sites = extract_client_sites()
self.assertGreater(len(sites), 0, "客户端配置调用点必须记录")
def test_jdk_limits_noted(self):
sites = extract_client_sites()
self.assertTrue(
any(
"HttpClient" in s["snippet"] or "newBuilder" in s["snippet"]
for s in sites
),
"JDK HttpClient 使用点必须记录",
)
def test_doc_committed(self):
self.assertTrue(DOC_PATH.is_file(), f"契约表文档缺失: {DOC_PATH}")
text = DOC_PATH.read_text(encoding="utf-8")
self.assertIn("超时", text)
self.assertIn("重试", text)
self.assertIn("幂等", text)
def test_no_code_change(self):
changed = subprocess.run(
["git", "diff", "--name-only", "HEAD", "--", "backend-java/src"],
cwd=REPO_ROOT,
capture_output=True,
text=True,
check=True,
).stdout.strip()
self.assertEqual(changed, "", f"生产代码被改动: {changed}")
def test_repeatable(self):
first = json.dumps(scan().to_dict(), sort_keys=True)
second = json.dumps(scan().to_dict(), sort_keys=True)
self.assertEqual(first, second, "盘点可复查")
if __name__ == "__main__":
unittest.main()
+199
View File
@@ -0,0 +1,199 @@
"""task-121 N+1 扫描审计测试。
对应 plan 06 任务 121 的 8 条用例 + 2 条补强:
1. test_scan_select_in_loop 识别循环内单查
2. test_scan_output_listed 清单产出(位置/对象/影响字段齐全)
3. test_scan_priority 按影响排序
4. test_scan_module_coverage 覆盖主要模块(真实代码扫描)
5. test_scan_false_positive_checked 排除误报(已批量/循环外)
6. test_scan_repeatable 可重复(同输入同输出)
7. test_scan_documented 审计文档产出
8. test_scan_no_edit 本任务不改生产代码
9. test_scan_lambda_brace_less 无花括号 lambda 内单查识别
10. test_scan_iterating_query_result 遍历查询结果集判 HIGH
"""
from __future__ import annotations
import json
import subprocess
import unittest
from pathlib import Path
from n1_scan import (
HIGH,
LOW,
MEDIUM,
MODULES_ROOT,
scan_modules,
scan_text,
)
SCRIPTS_DIR = Path(__file__).resolve().parent
REPO_ROOT = SCRIPTS_DIR.parent.parent
DOC_PATH = REPO_ROOT / "backend-java" / "docs" / "n1-scan-audit.md"
FIXTURE_LOOP = """package com.example;
public class DemoService {
public void batchByIds(List<Long> ids) {
for (Long id : ids) {
FileTaskEntity task = fileTaskMapper.selectById(id);
}
}
}
"""
FIXTURE_QUERY_RESULT_LOOP = """package com.example;
public class DemoService {
public List<Vo> listVos() {
List<FileTaskEntity> tasks = fileTaskMapper.selectList(new LambdaQueryWrapper<>());
List<Vo> vos = new ArrayList<>();
for (FileTaskEntity t : tasks) {
FileResultEntity r = fileResultMapper.selectById(t.getResultId());
vos.add(toVo(r));
}
return vos;
}
}
"""
FIXTURE_LAMBDA_BRACELESS = """package com.example;
public class DemoService {
public void touch(List<Long> ids) {
ids.forEach(id -> fileTaskMapper.selectOne(
new LambdaQueryWrapper<FileTaskEntity>().eq(FileTaskEntity::getId, id)));
}
}
"""
FIXTURE_BATCH_SAFE = """package com.example;
public class DemoService {
public List<FileTaskEntity> load(List<Long> ids) {
List<FileTaskEntity> tasks = new ArrayList<>();
for (Long id : ids) {
tasks.addAll(fileTaskMapper.selectBatchIds(Collections.singleton(id)));
}
for (Long id : ids) {
tasks.addAll(fileTaskMapper.selectByIds(Collections.singleton(id)));
}
return tasks;
}
}
"""
FIXTURE_NO_LOOP = """package com.example;
public class DemoService {
public FileTaskEntity one(Long id) {
return fileTaskMapper.selectById(id);
}
}
"""
FIXTURE_LOOP_NO_QUERY = """package com.example;
public class DemoService {
public int sum(List<Integer> nums) {
int s = 0;
for (Integer n : nums) {
s += n;
}
return s;
}
}
"""
FIXTURE_INDEX_LOOP = """package com.example;
public class DemoService {
public List<Vo> listVos(List<Long> ids) {
List<Vo> vos = new ArrayList<>();
for (int i = 0; i < ids.size(); i++) {
FileTaskEntity t = fileTaskMapper.selectById(ids.get(i));
vos.add(toVo(t));
}
return vos;
}
}
"""
FIXTURE_MIXED = (
FIXTURE_QUERY_RESULT_LOOP
+ "\n"
+ FIXTURE_LAMBDA_BRACELESS
+ "\n"
+ FIXTURE_INDEX_LOOP
)
class ScanDetectTest(unittest.TestCase):
def test_scan_select_in_loop(self):
findings = scan_text(FIXTURE_LOOP, "DemoService.java")
self.assertEqual(len(findings), 1)
self.assertIn("selectById", findings[0]["statement"])
self.assertEqual(findings[0]["loop_type"], "for")
self.assertEqual(findings[0]["line"], 5)
def test_scan_output_listed(self):
for finding in scan_text(FIXTURE_MIXED, "DemoService.java"):
for key in ("file", "line", "statement", "loop_type", "impact"):
self.assertIn(key, finding)
self.assertTrue(finding[key] is not None and finding[key] != "")
def test_scan_priority(self):
findings = scan_text(FIXTURE_MIXED, "DemoService.java")
order = {HIGH: 0, MEDIUM: 1, LOW: 2}
impacts = [order[f["impact"]] for f in findings]
self.assertEqual(impacts, sorted(impacts))
def test_scan_false_positive_checked(self):
self.assertEqual(len(scan_text(FIXTURE_BATCH_SAFE, "Batch.java")), 0)
self.assertEqual(len(scan_text(FIXTURE_NO_LOOP, "NoLoop.java")), 0)
self.assertEqual(len(scan_text(FIXTURE_LOOP_NO_QUERY, "NoQuery.java")), 0)
def test_scan_repeatable(self):
first = scan_text(FIXTURE_MIXED, "DemoService.java")
second = scan_text(FIXTURE_MIXED, "DemoService.java")
self.assertEqual(
json.dumps(first, sort_keys=True), json.dumps(second, sort_keys=True)
)
def test_scan_lambda_brace_less(self):
findings = scan_text(FIXTURE_LAMBDA_BRACELESS, "DemoService.java")
self.assertEqual(len(findings), 1)
self.assertEqual(findings[0]["loop_type"], "forEach")
self.assertEqual(findings[0]["impact"], MEDIUM)
def test_scan_iterating_query_result(self):
findings = scan_text(FIXTURE_QUERY_RESULT_LOOP, "DemoService.java")
self.assertEqual(len(findings), 1)
self.assertEqual(findings[0]["impact"], HIGH)
class ScanCoverageTest(unittest.TestCase):
def test_scan_module_coverage(self):
report = scan_modules(MODULES_ROOT)
self.assertGreaterEqual(report.scanned_files, 600)
self.assertGreater(len(report.findings), 0)
modules = {f["module"] for f in report.findings}
self.assertGreaterEqual(len(modules), 3)
order = {HIGH: 0, MEDIUM: 1, LOW: 2}
impacts = [order[f["impact"]] for f in report.findings]
self.assertEqual(impacts, sorted(impacts))
def test_scan_documented(self):
self.assertTrue(DOC_PATH.is_file(), f"audit doc missing: {DOC_PATH}")
text = DOC_PATH.read_text(encoding="utf-8")
self.assertIn("N+1", text)
self.assertIn("影响", text)
def test_scan_no_edit(self):
changed = subprocess.run(
["git", "diff", "--name-only", "HEAD", "--", "backend-java/src"],
cwd=REPO_ROOT,
capture_output=True,
text=True,
check=True,
).stdout.strip()
self.assertEqual(changed, "", f"生产代码被改动: {changed}")
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,89 @@
"""task-148 临时目录配置盘点测试。
对应 plan 09 任务 148 的 8 条用例:
1. test_config_listed 配置项清单齐全
2. test_temp_root_defined 临时根目录定义
3. test_retention_period 保留期定义
4. test_payload_prefixes payload 前缀
5. test_upload_dir 上传目录
6. test_existing_cleanup_noted 现有清理逻辑记录
7. test_doc_committed 文档存在
8. test_no_code_change 零代码变更
"""
from __future__ import annotations
import json
import subprocess
import unittest
from temp_config_audit import REPO_ROOT, extract_cleanup_calls, extract_configs, scan
DOC_PATH = REPO_ROOT / "docs" / "temp-file-config-audit.md"
class TempConfigAuditTest(unittest.TestCase):
def test_config_listed(self):
configs = extract_configs()
self.assertGreaterEqual(len(configs), 8, "配置项清单必须齐全")
for item in configs:
for key in ("key", "value", "line", "source"):
self.assertIn(key, item)
def test_temp_root_defined(self):
keys = {c["key"] for c in extract_configs()}
self.assertTrue(
any("local-temp-dir" in k for k in keys),
f"临时根目录配置缺失: {sorted(keys)}",
)
def test_retention_period(self):
keys = {c["key"] for c in extract_configs()}
self.assertTrue(any("retention" in k for k in keys), "保留期配置缺失")
def test_payload_prefixes(self):
keys = {c["key"] for c in extract_configs()}
self.assertTrue(
any("transient-payload" in k for k in keys), "transient payload 配置缺失"
)
def test_upload_dir(self):
configs = extract_configs()
self.assertTrue(
any(
"upload" in c["key"].lower() or "tmp" in c["key"].lower()
for c in configs
),
"上传/临时目录配置缺失",
)
def test_existing_cleanup_noted(self):
calls = extract_cleanup_calls()
self.assertGreater(len(calls), 0, "必须记录既有清理调用点")
files = {c["file"] for c in calls}
self.assertGreaterEqual(len(files), 3, "清理调用应覆盖多个文件")
def test_doc_committed(self):
self.assertTrue(DOC_PATH.is_file(), f"盘点文档缺失: {DOC_PATH}")
text = DOC_PATH.read_text(encoding="utf-8")
self.assertIn("临时", text)
self.assertIn("保留期", text)
def test_no_code_change(self):
changed = subprocess.run(
["git", "diff", "--name-only", "HEAD", "--", "backend-java/src"],
cwd=REPO_ROOT,
capture_output=True,
text=True,
check=True,
).stdout.strip()
self.assertEqual(changed, "", f"生产代码被改动: {changed}")
def test_scan_repeatable(self):
first = json.dumps(scan().to_dict(), sort_keys=True)
second = json.dumps(scan().to_dict(), sort_keys=True)
self.assertEqual(first, second, "盘点结果可重复")
if __name__ == "__main__":
unittest.main()
+204
View File
@@ -0,0 +1,204 @@
"""task-124 @Transactional 方法审计清单测试。
对应 plan 07 任务 124 的 8 条用例 + 2 条补强:
1. test_audit_covers_all_modules 覆盖全部业务模块
2. test_audit_classification 分类完整(必须/可移出)
3. test_audit_result_endpoints /result 相关方法都在清单
4. test_audit_python_facing Python 回调相关标记
5. test_audit_priority_order 按收益排序
6. test_audit_no_code_change 本任务零代码变更
7. test_audit_documented 文档产出
8. test_audit_repeatable 可重复执行
9. test_audit_detects_tx_method 识别 @Transactional 方法
10. test_audit_classifies_movable 事务内可移出段分类
"""
from __future__ import annotations
import json
import subprocess
import unittest
from pathlib import Path
from tx_audit import MODULES_ROOT, scan_modules, scan_text
SCRIPTS_DIR = Path(__file__).resolve().parent
REPO_ROOT = SCRIPTS_DIR.parent.parent
DOC_PATH = REPO_ROOT / "backend-java" / "docs" / "transaction-boundary-audit.md"
FIXTURE_TX = """package com.example;
@Service
public class DemoService {
@Transactional
public boolean submitResult(Long taskId, List<RowVo> rows) {
List<ItemVo> items = rows.stream()
.map(row -> toVo(row))
.collect(Collectors.toList());
int updated = fileResultMapper.update(null, new LambdaUpdateWrapper<FileResultEntity>()
.eq(FileResultEntity::getId, taskId)
.set(FileResultEntity::getStatus, "SUCCESS"));
log.info("submitted taskId={} count={}", taskId, items.size());
Files.deleteIfExists(Path.of("/tmp/task-" + taskId));
return updated > 0;
}
public void noTxMethod(Long id) {
fileTaskMapper.selectById(id);
}
}
"""
FIXTURE_WRITE_ONLY = """package com.example;
@Service
public class DemoService {
@Transactional
public int markFailed(Long id) {
return fileTaskMapper.update(null, new LambdaUpdateWrapper<FileTaskEntity>()
.eq(FileTaskEntity::getId, id)
.set(FileTaskEntity::getStatus, "FAILED"));
}
}
"""
FIXTURE_LOCK = """package com.example;
@Service
public class DemoService {
@Transactional
public void finalizeTask(Long taskId) {
TaskDistributedLockService.LockHandle handle = acquireTaskLock("MODULE", taskId);
if (handle == null) {
return;
}
try (handle) {
taskMapper.update(null, new LambdaUpdateWrapper<TaskFileEntity>()
.eq(TaskFileEntity::getId, taskId));
}
}
}
"""
FIXTURE_PLAIN = """package com.example;
@Service
public class DemoService {
public int sum(List<Integer> nums) {
return nums.stream().mapToInt(Integer::intValue).sum();
}
}
"""
class TxAuditDetectTest(unittest.TestCase):
def test_audit_detects_tx_method(self):
findings = scan_text(FIXTURE_TX, "DemoService.java")
self.assertEqual(len(findings), 1)
self.assertEqual(findings[0]["method"], "submitResult")
self.assertFalse(findings[0]["class_level"])
def test_audit_classifies_movable(self):
findings = scan_text(FIXTURE_TX, "DemoService.java")
self.assertEqual(len(findings), 1)
movable = findings[0]["movable"]
self.assertIn("compute", movable)
self.assertIn("assembly", movable)
self.assertIn("log", movable)
self.assertIn("cleanup", movable)
self.assertIn("write", findings[0]["must_stay"])
self.assertGreater(findings[0]["priority"], 0)
def test_audit_no_tx_not_reported(self):
self.assertEqual(len(scan_text(FIXTURE_PLAIN, "Plain.java")), 0)
def test_audit_write_only_no_movable(self):
findings = scan_text(FIXTURE_WRITE_ONLY, "DemoService.java")
self.assertEqual(len(findings), 1)
self.assertIn("write", findings[0]["must_stay"])
self.assertEqual(findings[0]["movable"], [])
def test_audit_lock_flagged_must_stay(self):
findings = scan_text(FIXTURE_LOCK, "DemoService.java")
self.assertEqual(len(findings), 1)
self.assertIn("lock", findings[0]["must_stay"])
class TxAuditCoverageTest(unittest.TestCase):
def test_audit_covers_all_modules(self):
report = scan_modules(MODULES_ROOT)
self.assertGreaterEqual(report.scanned_files, 600)
self.assertGreater(len(report.findings), 50)
modules = {f["module"] for f in report.findings}
self.assertGreaterEqual(len(modules), 15)
def test_audit_classification(self):
for finding in scan_modules(MODULES_ROOT).findings:
self.assertIn("must_stay", finding)
self.assertIn("movable", finding)
self.assertIsInstance(finding["must_stay"], list)
self.assertIsInstance(finding["movable"], list)
def test_audit_result_endpoints(self):
report = scan_modules(MODULES_ROOT)
result_methods = [
f
for f in report.findings
if "result" in f["method"].lower() or "submit" in f["method"].lower()
]
self.assertGreater(
len(result_methods), 0, "应找到 /result 相关 @Transactional 方法"
)
modules_with_result = {f["module"] for f in result_methods}
self.assertGreaterEqual(len(modules_with_result), 3)
def test_audit_python_facing(self):
report = scan_modules(MODULES_ROOT)
flagged = [f for f in report.findings if f["python_facing"]]
self.assertGreater(len(flagged), 0, "应标记 Python 回调相关方法")
for f in flagged:
name = f["method"].lower()
self.assertTrue(
any(
k in name
for k in (
"result",
"submit",
"upload",
"chunk",
"done",
"ack",
"report",
)
),
f"python_facing 标记与命名不符: {f['method']}",
)
def test_audit_priority_order(self):
findings = scan_modules(MODULES_ROOT).findings
priorities = [f["priority"] for f in findings]
self.assertEqual(priorities, sorted(priorities, reverse=True))
def test_audit_repeatable(self):
first = scan_modules(MODULES_ROOT).to_dict()
second = scan_modules(MODULES_ROOT).to_dict()
self.assertEqual(
json.dumps(first, sort_keys=True), json.dumps(second, sort_keys=True)
)
def test_audit_documented(self):
self.assertTrue(DOC_PATH.is_file(), f"audit doc missing: {DOC_PATH}")
text = DOC_PATH.read_text(encoding="utf-8")
self.assertIn("事务", text)
self.assertIn("必须", text)
self.assertIn("可移出", text)
def test_audit_no_code_change(self):
changed = subprocess.run(
["git", "diff", "--name-only", "HEAD", "--", "backend-java/src"],
cwd=REPO_ROOT,
capture_output=True,
text=True,
check=True,
).stdout.strip()
self.assertEqual(changed, "", f"生产代码被改动: {changed}")
if __name__ == "__main__":
unittest.main()
+220
View File
@@ -0,0 +1,220 @@
"""task-124 @Transactional 方法审计清单工具。
对 backend-java modules 下的 Java 源码做机械化扫描:定位全部方法级
@Transactional 注解,按方法体内容分类事务内代码段——必须事务内(落库写/状态/锁)
与可移出(纯计算/DTO 组装/日志/临时文件清理),标记 Python 回调相关方法,
按可移出收益排序,产出审计清单供 plan 07 的任务 125-131 事务收缩参考。
本工具只读,不改任何代码。
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from dataclasses import dataclass, field
from pathlib import Path
from n1_scan import _match_span, _strip_comments
MODULES_ROOT = (
Path(__file__).resolve().parent.parent / "src/main/java/com/nanri/aiimage/modules"
)
TX_RE = re.compile(r"@Transactional\b")
METHOD_RE = re.compile(
r"\s*(?:(?:public|protected|private)\s+)?"
r"(?:[\w<>,.?\[\] ]+[\s])?(\w+)\s*\("
)
PYTHON_FACING_RE = re.compile(r"(?i)(result|submit|upload|chunk|done|ack|report)")
WRITE_RE = re.compile(
r"\.(insert|update|deleteById|deleteBatchIds|removeById|removeByIds|remove\b|"
r"save|saveBatch|saveOrUpdate|updateById|insertOrUpdate|updateBatchById)\s*\("
)
LOCK_RE = re.compile(r"acquireTaskLock|\.tryLock\(|LockHandle|taskLockHandle")
CLEANUP_RE = re.compile(
r"Files\.(delete|deleteIfExists|move)|deleteIfExists\(|"
r"tempDir|tempFile|temp-dir|cleanupTemp|deleteTemp|临时文件"
)
COMPUTE_RE = re.compile(
r"\.stream\(\)|\.map\(|\.collect\(|\.filter\(|\.reduce\(|\.sorted\(|"
r"Collectors\.|computeIfAbsent|\.distinct\(|\.flatMap\(|\.peek\("
)
ASSEMBLY_RE = re.compile(
r"build\w*(Vo|VO|Dto|DTO)|toVo\(|toEntity\(|convert\w*\(|assemble\w*\(|"
r"new \w+(Vo|VO|Dto|DTO)\b"
)
LOG_RE = re.compile(r"log\.(info|warn|debug|error|trace)\s*\(")
def _match_body_span(text: str, open_pos: int) -> int | None:
"""{ 出发,返回匹配的 } 偏移(含)。
圆括号深度同步跟踪:lambda/方法调用内的 { } 不计入方法体层级,
方法体的闭合 } 总是在圆括号外层(paren == 0)首次归零时命中。
"""
depth = 0
paren = 0
for i in range(open_pos, len(text)):
c = text[i]
if c == "(":
paren += 1
elif c == ")":
paren -= 1
elif c == "{" and paren == 0:
depth += 1
elif c == "}" and paren == 0:
depth -= 1
if depth == 0:
return i
return None
def _count_hits(pattern: re.Pattern[str], text: str) -> int:
return len(pattern.findall(text))
def scan_text(text: str, filename: str = "fixture.java") -> list[dict]:
"""扫描单份 Java 源码,返回 @Transactional 方法审计条目(按收益降序)。"""
stripped = _strip_comments(text)
findings: list[dict] = []
pos = 0
while True:
tx_match = TX_RE.search(stripped, pos)
if not tx_match:
break
anno_end = tx_match.end()
if stripped[anno_end : anno_end + 1] == "(":
close = _match_span(stripped, anno_end)
if close is None:
pos = tx_match.end()
continue
anno_end = close + 1
method_match = METHOD_RE.search(stripped, anno_end)
if not method_match:
pos = tx_match.end()
continue
method_name = method_match.group(1)
paren_pos = method_match.end() - 1
close_paren = _match_span(stripped, paren_pos)
if close_paren is None:
pos = method_match.end()
continue
body_start = close_paren + 1
while body_start < len(stripped) and stripped[body_start].isspace():
body_start += 1
if body_start >= len(stripped) or stripped[body_start] != "{":
# 抽象方法/接口声明无方法体,跳过
pos = method_match.end()
continue
body_end = _match_body_span(stripped, body_start)
if body_end is None:
pos = method_match.end()
continue
body = stripped[body_start + 1 : body_end]
must_stay: list[str] = []
counts: dict[str, int] = {}
if _count_hits(WRITE_RE, body) > 0:
must_stay.append("write")
counts["write"] = _count_hits(WRITE_RE, body)
if _count_hits(LOCK_RE, body) > 0:
must_stay.append("lock")
counts["lock"] = _count_hits(LOCK_RE, body)
movable: list[str] = []
for key, pattern in (
("compute", COMPUTE_RE),
("assembly", ASSEMBLY_RE),
("cleanup", CLEANUP_RE),
("log", LOG_RE),
):
hits = _count_hits(pattern, body)
counts[key] = hits
if hits > 0:
movable.append(key)
priority = sum(
counts.get(key, 0) for key in ("compute", "assembly", "cleanup", "log")
)
findings.append(
{
"file": filename,
"line": _line_of(text, tx_match.start()),
"method": method_name,
"signature": " ".join(
stripped[method_match.start() : close_paren].split()
)[:100],
"class_level": False,
"python_facing": bool(PYTHON_FACING_RE.search(method_name)),
"must_stay": must_stay,
"movable": movable,
"counts": counts,
"priority": priority,
}
)
pos = body_end + 1
findings.sort(key=lambda f: (-f["priority"], f["line"]))
return findings
def _line_of(text: str, offset: int) -> int:
return text.count("\n", 0, offset) + 1
@dataclass
class Report:
scanned_files: int = 0
modules: list[str] = field(default_factory=list)
findings: list[dict] = field(default_factory=list)
def to_dict(self) -> dict:
return {
"scanned_files": self.scanned_files,
"modules": self.modules,
"findings": self.findings,
}
def scan_modules(root: Path) -> Report:
"""扫描 modules 根目录下全部 Java 文件,产出 @Transactional 审计报告。"""
report = Report()
module_set: set[str] = set()
for java_file in sorted(root.rglob("*.java")):
report.scanned_files += 1
rel = java_file.relative_to(root)
module_set.add(rel.parts[0])
for finding in scan_text(java_file.read_text(encoding="utf-8"), str(rel)):
finding["module"] = rel.parts[0]
report.findings.append(finding)
report.findings.sort(key=lambda f: (-f["priority"], f["file"], f["line"]))
report.modules = sorted(module_set)
return report
def main() -> None:
parser = argparse.ArgumentParser(description="@Transactional 方法审计")
parser.add_argument("--json", help="输出 JSON 报告路径(默认 stdout")
args = parser.parse_args()
report = scan_modules(MODULES_ROOT)
payload = report.to_dict()
if args.json:
Path(args.json).write_text(
json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
)
else:
print(json.dumps(payload, ensure_ascii=False, indent=2))
print(
f"scanned {report.scanned_files} files, "
f"{len(report.findings)} @Transactional methods "
f"across {len(report.modules)} modules",
file=sys.stderr,
)
if __name__ == "__main__":
main()
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,59 @@
package com.nanri.aiimage.config;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
/**
* 统一 HTTP 客户端配置命名空间(task-167)。
*
* `aiimage.http-client.*`connect/read/call 超时与重试默认值与现状一致
* (RustFS 客户端契约表基线);非法值钳制到合理范围;env 覆盖走宽松绑定。
*/
@Data
@Component
@ConfigurationProperties(prefix = "aiimage.http-client")
public class HttpClientProperties {
/** 连接超时(毫秒),默认 10000 与现状一致。 */
private long connectTimeoutMillis = 10_000;
/** 读取超时(毫秒),默认 60000 与现状一致。 */
private long readTimeoutMillis = 60_000;
/** 调用总超时(毫秒),默认 90000 与现状一致。 */
private long callTimeoutMillis = 90_000;
/** 最大重试次数(0-10),默认 3 与现状一致。 */
private int maxRetries = 3;
/** 重试基础延迟(毫秒),默认 500 与现状一致。 */
private long baseRetryDelayMillis = 500;
/** 钳制后的连接超时:1s-300s。 */
public long effectiveConnectTimeoutMillis() {
return clamp(connectTimeoutMillis, 1_000, 300_000);
}
/** 钳制后的读取超时:1s-3600s。 */
public long effectiveReadTimeoutMillis() {
return clamp(readTimeoutMillis, 1_000, 3_600_000);
}
/** 钳制后的调用总超时:1s-7200s。 */
public long effectiveCallTimeoutMillis() {
return clamp(callTimeoutMillis, 1_000, 7_200_000);
}
/** 钳制后的重试次数:0-10。 */
public int effectiveMaxRetries() {
return (int) clamp(maxRetries, 0, 10);
}
private static long clamp(long value, long min, long max) {
if (value < min) {
return min;
}
return Math.min(value, max);
}
}
@@ -0,0 +1,24 @@
package com.nanri.aiimage.config;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
/**
* 巡检任务配置(task-161)。
* 全部巡检默认 disabled;启用后按调度间隔执行;limit 为单次巡检行数上限。
*/
@Data
@Component
@ConfigurationProperties(prefix = "aiimage.inspection")
public class InspectionProperties {
/** 巡检总开关(默认关闭)。 */
private boolean enabled = false;
/** 调度 cron(默认凌晨 3 点)。 */
private String cron = "0 0 3 * * *";
/** 单次巡检行数上限。 */
private int limit = 200;
}
@@ -0,0 +1,38 @@
package com.nanri.aiimage.config;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Component;
/**
* LLM 客户端统一配置解析(task-168,Coze 已改直连 LLM)。
*
* 从 aiimage.http-client.* 命名空间读取 LLM 客户端超时/重试,默认值与现状一致
* connect 10s / read 60s / call 90s / retry 3);非法值钳制。模块级显式配置
* SimilarAsinProperties.llm*)优先级高于命名空间(行为不变)。
*/
@Component
@RequiredArgsConstructor
public class LlmHttpConfigResolver {
private final HttpClientProperties httpClientProperties;
public long connectTimeoutMillis() {
return httpClientProperties.effectiveConnectTimeoutMillis();
}
public long readTimeoutMillis() {
return httpClientProperties.effectiveReadTimeoutMillis();
}
public long callTimeoutMillis() {
return httpClientProperties.effectiveCallTimeoutMillis();
}
public int maxRetries() {
return httpClientProperties.effectiveMaxRetries();
}
public long baseRetryDelayMillis() {
return httpClientProperties.getBaseRetryDelayMillis();
}
}
@@ -14,6 +14,8 @@ public class StorageProperties {
private long sourceRetentionHours = 24;
private long resultRetentionHours = 24;
private long transientPayloadRetentionHours = 72;
/** 临时目录容量告警阈值(字节),超过时记录告警日志(task-154)。 */
private long capacityWarnBytes = 50L * 1024 * 1024 * 1024;
public String getLocalTempDir() {
String configured = localTempDir == null ? "" : localTempDir.trim();
@@ -405,7 +405,9 @@ public class AppearancePatentTaskService {
private void submitResultLocked(Long taskId, AppearancePatentSubmitResultRequest request) {
if (transactionManager != null) {
SubmitContext context = inNewTransaction(() -> persistSubmittedChunk(taskId, request));
// 纯计算(flatten 分组展开/序列化/payload 存储/哈希)在事务开始前完成
PreparedSubmittedChunk prepared = prepareSubmittedChunk(taskId, request);
SubmitContext context = inNewTransaction(() -> persistSubmittedChunk(prepared));
inNewTransaction(() -> {
completeSubmittedChunk(context);
return null;
@@ -638,7 +640,42 @@ public class AppearancePatentTaskService {
return updatedMillis <= thresholdMillis;
}
private SubmitContext persistSubmittedChunk(Long taskId, AppearancePatentSubmitResultRequest request) {
/**
* /result 提交的纯计算阶段(无事务注解):读任务 + 校验归属 + 分组展开 +
* 序列化 + payload 存储 + 哈希预计算,全部在事务开始前完成。
* 重复 chunk(查重命中)时不做存储与哈希,落库由 persist 按同一查重短路。
*/
private PreparedSubmittedChunk prepareSubmittedChunk(Long taskId, AppearancePatentSubmitResultRequest request) {
FileTaskEntity task = fileTaskMapper.selectById(taskId);
if (task == null || !MODULE_TYPE.equals(task.getModuleType())) {
throw new BusinessException("任务不存在");
}
ensureTaskOwnedByCurrentInstance(task, "submit result");
int chunkIndex = request.getChunkIndex() == null ? 0 : request.getChunkIndex();
int chunkTotal = request.getChunkTotal() == null ? 1 : request.getChunkTotal();
boolean done = Boolean.TRUE.equals(request.getDone());
String scopeKey = firstNonBlank(request.getSubmissionId(), "task:" + taskId);
String scopeHash = DigestUtil.sha256Hex(scopeKey);
TaskChunkEntity existing = taskChunkMapper.selectOne(new LambdaQueryWrapper<TaskChunkEntity>()
.eq(TaskChunkEntity::getTaskId, taskId)
.eq(TaskChunkEntity::getModuleType, MODULE_TYPE)
.eq(TaskChunkEntity::getScopeHash, scopeHash)
.eq(TaskChunkEntity::getChunkIndex, chunkIndex)
.last("limit 1"));
if (existing != null) {
return new PreparedSubmittedChunk(task, scopeKey, scopeHash, chunkIndex, chunkTotal, done,
request.getError(), null, null, null);
}
List<AppearancePatentResultRowDto> rawRows = flattenSubmittedRows(request);
String payloadJson = writeJson(rawRows, "结果序列化失败");
String storedPayload = storeSharedChunkPayload(taskId, scopeHash, chunkIndex, payloadJson);
return new PreparedSubmittedChunk(task, scopeKey, scopeHash, chunkIndex, chunkTotal, done,
request.getError(), payloadJson, storedPayload, DigestUtil.sha256Hex(payloadJson));
}
private SubmitContext persistSubmittedChunk(PreparedSubmittedChunk prepared) {
Long taskId = prepared.task().getId();
FileTaskEntity task = fileTaskMapper.selectById(taskId);
if (task == null || !MODULE_TYPE.equals(task.getModuleType())) {
throw new BusinessException("任务不存在");
@@ -648,48 +685,43 @@ public class AppearancePatentTaskService {
}
ensureTaskOwnedByCurrentInstance(task, "submit result");
int chunkIndex = request.getChunkIndex() == null ? 0 : request.getChunkIndex();
int chunkTotal = request.getChunkTotal() == null ? 1 : request.getChunkTotal();
boolean done = Boolean.TRUE.equals(request.getDone());
String scopeKey = firstNonBlank(request.getSubmissionId(), "task:" + taskId);
String scopeHash = DigestUtil.sha256Hex(scopeKey);
taskCacheService.touchTaskHeartbeat(taskId);
TaskChunkEntity existing = taskChunkMapper.selectOne(new LambdaQueryWrapper<TaskChunkEntity>()
.eq(TaskChunkEntity::getTaskId, taskId)
.eq(TaskChunkEntity::getModuleType, MODULE_TYPE)
.eq(TaskChunkEntity::getScopeHash, scopeHash)
.eq(TaskChunkEntity::getChunkIndex, chunkIndex)
.eq(TaskChunkEntity::getScopeHash, prepared.scopeHash())
.eq(TaskChunkEntity::getChunkIndex, prepared.chunkIndex())
.last("limit 1"));
if (existing == null) {
List<AppearancePatentResultRowDto> rawRows = flattenSubmittedRows(request);
String payloadJson = writeJson(rawRows, "结果序列化失败");
if (existing == null && prepared.storedPayload() != null) {
TaskChunkEntity chunk = new TaskChunkEntity();
chunk.setTaskId(taskId);
chunk.setModuleType(MODULE_TYPE);
chunk.setScopeKey(scopeKey);
chunk.setScopeHash(scopeHash);
chunk.setChunkIndex(chunkIndex);
chunk.setChunkTotal(chunkTotal);
String storedPayload = storeSharedChunkPayload(taskId, scopeHash, chunkIndex, payloadJson);
chunk.setPayloadJson(storedPayload);
chunk.setPayloadHash(DigestUtil.sha256Hex(payloadJson));
chunk.setScopeKey(prepared.scopeKey());
chunk.setScopeHash(prepared.scopeHash());
chunk.setChunkIndex(prepared.chunkIndex());
chunk.setChunkTotal(prepared.chunkTotal());
chunk.setPayloadJson(prepared.storedPayload());
chunk.setPayloadHash(prepared.payloadHash());
chunk.setCreatedAt(LocalDateTime.now());
chunk.setUpdatedAt(LocalDateTime.now());
try {
taskChunkMapper.insert(chunk);
} catch (DuplicateKeyException ex) {
log.info("[appearance-patent] duplicate chunk inserted concurrently taskId={} scope={} chunk={}", taskId, scopeKey, chunkIndex);
log.info("[appearance-patent] duplicate chunk inserted concurrently taskId={} scope={} chunk={}",
taskId, prepared.scopeKey(), prepared.chunkIndex());
}
} else {
log.info("[appearance-patent] duplicate chunk ignored taskId={} scope={} chunk={}", taskId, scopeKey, chunkIndex);
} else if (existing != null) {
log.info("[appearance-patent] duplicate chunk ignored taskId={} scope={} chunk={}",
taskId, prepared.scopeKey(), prepared.chunkIndex());
}
upsertScopeState(taskId, scopeKey, scopeHash, chunkTotal, request.getError(), done, false);
upsertScopeState(taskId, prepared.scopeKey(), prepared.scopeHash(), prepared.chunkTotal(),
prepared.error(), prepared.done(), false);
task.setUpdatedAt(LocalDateTime.now());
fileTaskMapper.updateById(task);
return new SubmitContext(task, scopeKey, scopeHash, chunkIndex, done, request.getError());
return new SubmitContext(task, prepared.scopeKey(), prepared.scopeHash(), prepared.chunkIndex(),
prepared.done(), prepared.error());
}
private void completeSubmittedChunk(SubmitContext context) {
@@ -2907,6 +2939,18 @@ public class AppearancePatentTaskService {
String error) {
}
private record PreparedSubmittedChunk(FileTaskEntity task,
String scopeKey,
String scopeHash,
Integer chunkIndex,
Integer chunkTotal,
boolean done,
String error,
String payloadJson,
String storedPayload,
String payloadHash) {
}
private static class SourceRowsBuilder {
private final String sourceFileKey;
private String sourceFilename;
@@ -97,6 +97,13 @@ public class CollectDataService {
public static final String MODULE_TYPE = "COLLECT_DATA";
/**
* task-50 上线初期写入端(CollectDataResultItemBatchWriter)曾误用小写
* "collectdata",遗留 5386 行存量数据;读取端查询统一 in(MODULE_TYPE, LEGACY_MODULE_TYPE)
* 兼容,否则这些任务的明细行在生成结果文件时查不出来。
*/
private static final String LEGACY_MODULE_TYPE = "collectdata";
public TaskProgressLightBatchVo progressLight(List<Long> taskIds) {
return taskProgressLightAssembler.assemble(MODULE_TYPE, null, taskIds);
}
@@ -937,7 +944,7 @@ public class CollectDataService {
private List<CollectDataResultRowVo> loadFinalRows(Long taskId) {
List<TaskResultItemEntity> rows = taskResultItemMapper.selectList(new LambdaQueryWrapper<TaskResultItemEntity>()
.eq(TaskResultItemEntity::getTaskId, taskId)
.eq(TaskResultItemEntity::getModuleType, MODULE_TYPE)
.in(TaskResultItemEntity::getModuleType, MODULE_TYPE, LEGACY_MODULE_TYPE)
.orderByAsc(TaskResultItemEntity::getId));
if (rows == null || rows.isEmpty()) {
return new ArrayList<>();
@@ -1258,7 +1265,7 @@ public class CollectDataService {
.eq(TaskChunkEntity::getModuleType, MODULE_TYPE));
taskResultItemMapper.delete(new LambdaQueryWrapper<TaskResultItemEntity>()
.eq(TaskResultItemEntity::getTaskId, task.getId())
.eq(TaskResultItemEntity::getModuleType, MODULE_TYPE));
.in(TaskResultItemEntity::getModuleType, MODULE_TYPE, LEGACY_MODULE_TYPE));
deleteTransientTaskPayloads(
taskChunkMapper.selectList(new LambdaQueryWrapper<TaskChunkEntity>()
.select(TaskChunkEntity::getPayloadJson)
@@ -1267,7 +1274,7 @@ public class CollectDataService {
taskResultItemMapper.selectList(new LambdaQueryWrapper<TaskResultItemEntity>()
.select(TaskResultItemEntity::getPayloadJson)
.eq(TaskResultItemEntity::getTaskId, task.getId())
.eq(TaskResultItemEntity::getModuleType, MODULE_TYPE)));
.in(TaskResultItemEntity::getModuleType, MODULE_TYPE, LEGACY_MODULE_TYPE)));
taskFileJobService.deleteTaskJobs(task.getId(), MODULE_TYPE);
fileTaskMapper.deleteById(task.getId());
}
@@ -1280,11 +1287,11 @@ public class CollectDataService {
List<TaskResultItemEntity> resultItems = taskResultItemMapper.selectList(new LambdaQueryWrapper<TaskResultItemEntity>()
.select(TaskResultItemEntity::getPayloadJson)
.eq(TaskResultItemEntity::getTaskId, row.getTaskId())
.eq(TaskResultItemEntity::getModuleType, MODULE_TYPE)
.in(TaskResultItemEntity::getModuleType, MODULE_TYPE, LEGACY_MODULE_TYPE)
.eq(TaskResultItemEntity::getResultId, row.getId()));
taskResultItemMapper.delete(new LambdaQueryWrapper<TaskResultItemEntity>()
.eq(TaskResultItemEntity::getTaskId, row.getTaskId())
.eq(TaskResultItemEntity::getModuleType, MODULE_TYPE)
.in(TaskResultItemEntity::getModuleType, MODULE_TYPE, LEGACY_MODULE_TYPE)
.eq(TaskResultItemEntity::getResultId, row.getId()));
// 与 deleteTask 一致:先删 DB 行再物理删对象,保证行删除与对象删除一致。
deleteResultItemPayloads(resultItems);
@@ -27,7 +27,12 @@ import java.util.Objects;
@Component
public class CollectDataResultItemBatchWriter {
private static final String MODULE_TYPE = "collectdata";
/** 必须与 CollectDataService.MODULE_TYPE 一致(大写 COLLECT_DATA)。
* 曾因写端误用小写 "collectdata",导致 task-50 之后所有采集任务的明细行
* 在生成结果文件时查不出来(biz_task_result_item.module_type 大小写分叉)。 */
private static final String MODULE_TYPE = "COLLECT_DATA";
/** 历史存量小写值,查询现有行时兼容,避免旧任务重试重复插入(与 CollectDataService.LEGACY_MODULE_TYPE 对应)。 */
private static final String LEGACY_MODULE_TYPE = "collectdata";
private static final int DEFAULT_BATCH_SIZE = 100;
private final TaskResultItemMapper taskResultItemMapper;
@@ -63,10 +68,12 @@ public class CollectDataResultItemBatchWriter {
}
String scopeHash = sha256(scopeKey);
// 一次性取回本 scope 现有行,构建 item_key → 现有行 映射(hash 相等即跳过)。
// 现有行查询兼容历史小写 "collectdata"(见 MODULE_TYPE 注释),避免旧任务重试
// 在同一唯一键 (task_id, module_type, scope_hash, item_key) 下重复插入。
List<TaskResultItemEntity> existingList = taskResultItemMapper.selectList(
new LambdaQueryWrapper<TaskResultItemEntity>()
.eq(TaskResultItemEntity::getTaskId, taskId)
.eq(TaskResultItemEntity::getModuleType, MODULE_TYPE)
.in(TaskResultItemEntity::getModuleType, MODULE_TYPE, LEGACY_MODULE_TYPE)
.eq(TaskResultItemEntity::getScopeHash, scopeHash));
Map<String, TaskResultItemEntity> existingByKey = new HashMap<>();
for (TaskResultItemEntity existing : existingList) {
@@ -36,8 +36,10 @@ import java.nio.file.attribute.FileTime;
import java.time.Duration;
import java.time.Instant;
import java.time.LocalDateTime;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
@Service
@RequiredArgsConstructor
@@ -178,6 +180,7 @@ public class DeleteBrandStaleTaskService {
.lt(FileTaskEntity::getUpdatedAt, threshold)
.last("limit 200"));
List<StaleFinalizeEntry> entries = new ArrayList<>();
for (FileTaskEntity task : runningTasks) {
Map<Object, Object> progress = deleteBrandTaskCacheService.getProgress(task.getId());
long lastHeartbeatAt = 0L;
@@ -203,28 +206,52 @@ public class DeleteBrandStaleTaskService {
if (taskLockHandle == null) {
continue;
}
boolean finalizeThrew;
try (taskLockHandle) {
try {
deleteBrandRunService.tryFinalizeTask(task.getId(), true);
FileTaskEntity refreshed = fileTaskMapper.selectById(task.getId());
try {
deleteBrandRunService.tryFinalizeTask(task.getId(), true);
finalizeThrew = false;
} catch (Exception ex) {
finalizeThrew = true;
log.warn("[stale-check] delete-brand finalize threw taskId={} msg={}", task.getId(), ex.getMessage());
}
}
entries.add(new StaleFinalizeEntry(task, lastHeartbeatAt, completedScopeCount, hasStartedProgress, finalizeThrew));
}
if (entries.isEmpty()) {
return;
}
// 两阶段:全部 finalize 后一次 IN 批量回读,替代逐任务 selectById。
List<Long> refreshTaskIds = entries.stream()
.filter(entry -> !entry.finalizeThrew)
.map(entry -> entry.task.getId())
.toList();
Map<Long, FileTaskEntity> refreshedById = refreshTaskIds.isEmpty()
? Map.of()
: fileTaskMapper.selectBatchIds(refreshTaskIds).stream()
.collect(Collectors.toMap(FileTaskEntity::getId, task -> task, (a, b) -> a));
for (StaleFinalizeEntry entry : entries) {
FileTaskEntity task = entry.task;
if (!entry.finalizeThrew) {
FileTaskEntity refreshed = refreshedById.get(task.getId());
if (refreshed != null && !"RUNNING".equals(refreshed.getStatus())) {
deleteBrandTaskCacheService.saveTaskCache(refreshed);
log.info("[stale-check] delete-brand finalized before timeout-fail taskId={} status={} updatedAt={} finishedAt={}",
refreshed.getId(), refreshed.getStatus(), refreshed.getUpdatedAt(), refreshed.getFinishedAt());
continue;
}
} catch (Exception ex) {
log.warn("[stale-check] delete-brand finalize threw taskId={} msg={}", task.getId(), ex.getMessage());
}
log.warn("[stale-check] delete-brand failing stale task -> taskId={} updatedAt={} createdAt={} lastHeartbeatAt={} timeoutMinutes={} completedScopes={} hasStartedProgress={}",
task.getId(),
task.getUpdatedAt(),
task.getCreatedAt(),
lastHeartbeatAt,
entry.lastHeartbeatAt,
minutes,
completedScopeCount,
hasStartedProgress);
entry.completedScopeCount,
entry.hasStartedProgress);
int updated = fileTaskMapper.update(null, new LambdaUpdateWrapper<FileTaskEntity>()
.eq(FileTaskEntity::getId, task.getId())
.eq(FileTaskEntity::getModuleType, MODULE_TYPE_DELETE_BRAND)
@@ -238,10 +265,13 @@ public class DeleteBrandStaleTaskService {
deleteBrandTaskCacheService.delete(task.getId());
log.warn("[stale-check] delete-brand failed taskId={} reason=timeout", task.getId());
}
}
}
}
private record StaleFinalizeEntry(FileTaskEntity task, long lastHeartbeatAt, int completedScopeCount,
boolean hasStartedProgress, boolean finalizeThrew) {
}
private ProductRiskStaleCheckStats failStaleProductRiskResolveTasks() {
ProductRiskStaleCheckStats stats = new ProductRiskStaleCheckStats();
long minutes = Math.max(1L, deleteBrandProgressProperties.getProductRiskStaleTimeoutMinutes());
@@ -0,0 +1,61 @@
package com.nanri.aiimage.modules.file.service;
import org.springframework.stereotype.Service;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import java.util.function.Function;
import java.util.function.Predicate;
/**
* 批量清理执行器(task-164)。
*
* 单个文件清理失败(删除函数返回 false/抛异常/引用判定失败)不影响其他文件;
* 失败可见(failedPaths 与失败计数);不上抛;下次调用可重试失败项。
*/
@Service
public class BatchFileCleaner {
public record BatchCleanResult(int cleanedCount, int failedCount, List<String> failedPaths) {
public boolean hasFailures() {
return failedCount > 0;
}
}
/**
* @param files 候选文件
* @param deleteFn 删除函数(返回 true 表示删除成功)
* @param isReferenced 引用判定(true → 跳过,不算失败)
*/
public BatchCleanResult cleanBatch(List<File> files, Function<File, Boolean> deleteFn,
Predicate<String> isReferenced) {
int cleaned = 0;
int failed = 0;
List<String> failedPaths = new ArrayList<>();
if (files == null || files.isEmpty()) {
return new BatchCleanResult(0, 0, failedPaths);
}
for (File file : files) {
if (file == null) {
continue;
}
try {
if (isReferenced != null && isReferenced.test(file.getName())) {
continue;
}
if (deleteFn != null && Boolean.TRUE.equals(deleteFn.apply(file))) {
cleaned++;
} else {
failed++;
failedPaths.add(file.getAbsolutePath());
}
} catch (Exception ex) {
failed++;
failedPaths.add(file.getAbsolutePath());
}
}
return new BatchCleanResult(cleaned, failed, List.copyOf(failedPaths));
}
}
@@ -2,8 +2,10 @@ package com.nanri.aiimage.modules.file.service;
import cn.hutool.core.io.FileUtil;
import com.nanri.aiimage.config.StorageProperties;
import io.micrometer.core.instrument.MeterRegistry;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
@@ -20,6 +22,7 @@ public class LocalTempCleanupService {
private static final Pattern ROOT_TEMP_FILE_PATTERN = Pattern.compile("^[a-fA-F0-9]{32}(\\.[^.]+)?$");
private static final String TRANSIENT_PAYLOAD_DIR_NAME = "transient-payload";
private static final String CLEANUP_FAILED_METRIC = "aiimage.temp-cleanup.failed";
private static final List<String> RESULT_DIR_NAMES = List.of(
"dedupe-result",
"convert-result",
@@ -32,6 +35,10 @@ public class LocalTempCleanupService {
private final StorageProperties storageProperties;
/** 指标注册表(可选注入:无注册表时仅记日志,不影响清理)。 */
@Autowired(required = false)
private MeterRegistry meterRegistry;
@Scheduled(cron = "${aiimage.storage.cleanup-cron:0 0 */6 * * *}")
public void cleanupLocalTempDir() {
if (!storageProperties.isCleanupEnabled()) {
@@ -57,6 +64,11 @@ public class LocalTempCleanupService {
for (File child : children) {
try {
// 路径安全:child 必须位于临时根目录内(防穿越/符号链接逃逸),否则跳过并告警
if (!PathSafetyGuard.isInside(tempDir, child)) {
log.warn("local temp cleanup skipped unsafe path: {}", child.getAbsolutePath());
continue;
}
if (child.isFile() && isManagedRootTempFile(child) && isExpired(child, sourceExpireBefore)) {
if (FileUtil.del(child)) {
deletedSourceCount++;
@@ -73,7 +85,7 @@ public class LocalTempCleanupService {
deleteEmptyDirectories(child, tempDir);
}
} catch (Exception ex) {
log.warn("local temp cleanup failed: path={}", child.getAbsolutePath(), ex);
handleCleanupFailure(child, ex);
}
}
@@ -89,6 +101,9 @@ public class LocalTempCleanupService {
File[] children = file.listFiles();
if (children != null) {
for (File child : children) {
if (!PathSafetyGuard.isInside(file, child)) {
continue;
}
deletedCount += deleteExpiredChildrenRecursively(child, expireBefore);
}
}
@@ -117,8 +132,26 @@ public class LocalTempCleanupService {
return ROOT_TEMP_FILE_PATTERN.matcher(file.getName()).matches();
}
/**
* 清理失败处理:记录日志与失败指标,不抛异常、不重试(下一轮定时清理自然重试)。
* 单个条目失败不影响其余条目的清理。
*/
void handleCleanupFailure(File child, Exception ex) {
log.warn("local temp cleanup failed: path={}", child == null ? null : child.getAbsolutePath(), ex);
try {
if (meterRegistry != null && child != null) {
meterRegistry.counter(CLEANUP_FAILED_METRIC, "path", child.getName()).increment();
}
} catch (Exception metricEx) {
// 指标记录失败也不阻断清理主流程
log.warn("local temp cleanup metric record failed path={}",
child == null ? null : child.getName(), metricEx);
}
}
private boolean isExpired(File file, Instant expireBefore) {
return Instant.ofEpochMilli(file.lastModified()).isBefore(expireBefore);
// 以 mtime 近似最后访问(TempFileMetadata 回退语义,与现状一致)
return TempFileMetadata.lastAccessTime(file).isBefore(expireBefore);
}
private boolean isDirectoryEmpty(File directory) {
@@ -0,0 +1,60 @@
package com.nanri.aiimage.modules.file.service;
import java.io.File;
import java.io.IOException;
import java.nio.file.Path;
/**
* 临时文件删除前的路径安全校验(task-151)。
*
* 删除前校验:child 规范化后必须位于 root 内(startsWith(root) 且不等于 root);
* 穿越(../、绝对路径逃逸、符号链接逃逸)拒绝;null/非法输入返回 false。
* 符号链接用 toRealPath 解析(存在时),防止链接指向根外文件被误删。
*/
public final class PathSafetyGuard {
private PathSafetyGuard() {
}
/**
* child 是否位于 root 内(规范化比较;root 本身不算内部)。
* 路径不存在时以 toAbsolutePath().normalize() 兜底,存在时优先 toRealPath
* 解析符号链接。
*/
public static boolean isInside(File root, File child) {
if (root == null || child == null) {
return false;
}
Path rootPath = resolve(root);
Path childPath = resolve(child);
if (rootPath == null || childPath == null) {
return false;
}
return childPath.startsWith(rootPath) && !childPath.equals(rootPath);
}
/**
* 路径名是否含穿越特征(../、..\\、绝对路径、盘符、URL 编码点 %2e、
* 全角点 .、混合分隔符、空字节)。
*/
public static boolean isTraversal(String name) {
if (name == null || name.isBlank()) {
return true;
}
return name.contains("..")
|| name.contains("%2e") || name.contains("%2E")
|| name.contains("")
|| name.startsWith("/")
|| name.matches("^[A-Za-z]:.*")
|| name.contains("\\")
|| name.contains("\0");
}
private static Path resolve(File file) {
try {
return file.toPath().toRealPath();
} catch (IOException ex) {
return file.toPath().toAbsolutePath().normalize();
}
}
}
@@ -0,0 +1,58 @@
package com.nanri.aiimage.modules.file.service;
import org.springframework.stereotype.Service;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import java.util.function.Predicate;
/**
* 引用感知清理决策(task-163)。
*
* 清理流程对每个文件先做引用判定:被引用(job/result/chunk/scope 引用)→ 跳过;
* 仅未引用文件进入清理。本类固化决策逻辑(清理执行本身由各清理服务负责)。
*/
@Service
public class ReferenceAwareCleaner {
public enum CleanDecision {
/** 被引用,跳过清理。 */
SKIP_REFERENCED,
/** 未引用,可清理。 */
CLEAN,
}
public record FileDecision(File file, CleanDecision decision) {
}
/**
* 单文件决策:引用判定抛异常时保守跳过(宁可保留不可误删)。
*/
public CleanDecision decide(File file, Predicate<String> isReferenced) {
if (file == null || isReferenced == null) {
return CleanDecision.CLEAN;
}
try {
return isReferenced.test(file.getName())
? CleanDecision.SKIP_REFERENCED
: CleanDecision.CLEAN;
} catch (Exception ex) {
return CleanDecision.SKIP_REFERENCED;
}
}
/**
* 批量决策:混合批次中仅未引用文件 CLEAN;引用判定失败的文件保守跳过。
*/
public List<FileDecision> decideBatch(List<File> files, Predicate<String> isReferenced) {
List<FileDecision> decisions = new ArrayList<>();
if (files == null) {
return decisions;
}
for (File file : files) {
decisions.add(new FileDecision(file, decide(file, isReferenced)));
}
return decisions;
}
}
@@ -0,0 +1,78 @@
package com.nanri.aiimage.modules.file.service;
import cn.hutool.core.io.FileUtil;
import com.nanri.aiimage.config.StorageProperties;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.stream.Stream;
/**
* 临时目录磁盘容量告警(task-154)。
*
* 容量超过阈值时记录告警日志(当前容量/阈值);告警带频率限制(默认每 10 分钟
* 至多一次,防刷屏);测量失败或目录缺失时仅记日志,不抛异常、不阻塞业务。
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class TempDirCapacityMonitor {
private static final long WARN_MIN_INTERVAL_MILLIS = 10 * 60 * 1000L;
private final StorageProperties storageProperties;
private volatile long lastWarnAtMillis = 0L;
@Scheduled(cron = "${aiimage.storage.capacity-warn-cron:0 */30 * * * *}")
public void checkTempDirCapacity() {
File tempDir = FileUtil.file(storageProperties.getLocalTempDir());
if (!tempDir.exists() || !tempDir.isDirectory()) {
return;
}
long usage = measureUsageBytes(tempDir);
if (usage < 0) {
log.warn("temp dir capacity measurement failed dir={}", tempDir.getAbsolutePath());
return;
}
long threshold = storageProperties.getCapacityWarnBytes();
if (usage > threshold) {
warnWithRateLimit(tempDir, usage, threshold);
}
}
/** 测量目录总字节数;失败返回 -1(不抛)。 */
public long measureUsageBytes(File dir) {
if (dir == null || !dir.isDirectory()) {
return -1L;
}
try (Stream<Path> paths = Files.walk(dir.toPath())) {
return paths.mapToLong(path -> {
try {
return Files.isRegularFile(path) ? Files.size(path) : 0L;
} catch (IOException ex) {
return 0L;
}
}).sum();
} catch (IOException ex) {
log.warn("temp dir capacity walk failed dir={}", dir.getAbsolutePath(), ex);
return -1L;
}
}
void warnWithRateLimit(File tempDir, long usage, long threshold) {
long now = System.currentTimeMillis();
if (now - lastWarnAtMillis < WARN_MIN_INTERVAL_MILLIS) {
return;
}
lastWarnAtMillis = now;
log.warn("temp dir capacity warning: usage={} bytes, threshold={} bytes, dir={}",
usage, threshold, tempDir.getAbsolutePath());
}
}
@@ -0,0 +1,57 @@
package com.nanri.aiimage.modules.file.service;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.attribute.BasicFileAttributes;
import java.time.Instant;
/**
* 临时文件时间元数据读取(task-149)。
*
* 创建/最后修改/大小显式读取;最后访问时间无独立记录表,以 mtime 回退
* (与既有 LocalTempCleanupService 清理语义一致,不改变文件语义)。
* 属性读取失败或文件缺失时回退 mtime/0,不抛 IO 异常(清理路径不得被阻断)。
*/
public final class TempFileMetadata {
private TempFileMetadata() {
}
/** 创建时间;属性不可读时回退到最后修改时间。 */
public static Instant creationTime(File file) {
if (file == null || !file.isFile()) {
return lastModified(file);
}
try {
BasicFileAttributes attrs = Files.readAttributes(file.toPath(), BasicFileAttributes.class);
return attrs.creationTime().toInstant();
} catch (IOException ex) {
return lastModified(file);
}
}
/** 最后修改时间(mtime)。 */
public static Instant lastModified(File file) {
if (file == null) {
return Instant.EPOCH;
}
return Instant.ofEpochMilli(file.lastModified());
}
/**
* 最后访问时间:无独立访问记录表,以 mtime 回退
* (清理判据与现状一致:mtime 近似最后访问)。
*/
public static Instant lastAccessTime(File file) {
return lastModified(file);
}
/** 文件大小(字节);文件缺失时为 0。 */
public static long size(File file) {
if (file == null || !file.isFile()) {
return 0L;
}
return file.length();
}
}
@@ -0,0 +1,135 @@
package com.nanri.aiimage.modules.file.service;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.function.Predicate;
import java.util.stream.Stream;
/**
* 临时目录孤立文件巡检(task-155)。
*
* 只读巡检:输出"无任何引用(或引用判定失败视为未引用候选)且超过保留期"的
* 文件清单报表(日志 + 返回对象);绝不删除文件。巡检可重复执行。
*/
@Slf4j
@Service
public class TempOrphanInspector {
public record OrphanFileEntry(String path, long sizeBytes, String lastModifiedAt) {
}
public record OrphanFileReport(List<OrphanFileEntry> entries, int totalBytes) {
public boolean isEmpty() {
return entries == null || entries.isEmpty();
}
}
/**
* 巡检目录内孤立文件。
*
* @param dir 临时根目录
* @param expireBefore 保留期边界(lastModified 早于该时刻视为过期)
* @param isReferenced fileKey/文件名 → 是否仍被引用(引用判定失败返回 false 时
* 文件进候选,由报表人工复核;判定抛异常时保守跳过该文件)
*/
public OrphanFileReport inspectOrphanFiles(File dir, Instant expireBefore, Predicate<String> isReferenced) {
List<OrphanFileEntry> entries = new ArrayList<>();
if (dir == null || !dir.isDirectory() || expireBefore == null) {
return new OrphanFileReport(entries, 0);
}
try (Stream<Path> paths = Files.walk(dir.toPath())) {
paths.filter(Files::isRegularFile).forEach(path -> {
File file = path.toFile();
Instant lastModified = TempFileMetadata.lastModified(file);
if (!lastModified.isBefore(expireBefore)) {
return;
}
boolean referenced;
try {
referenced = isReferenced != null && isReferenced.test(file.getName());
} catch (Exception ex) {
log.warn("orphan inspect reference check failed file={} err={}",
file.getName(), ex.getMessage());
return;
}
if (!referenced) {
entries.add(new OrphanFileEntry(
path.toString(),
file.length(),
lastModified.toString()));
}
});
} catch (IOException ex) {
log.warn("orphan inspect walk failed dir={} err={}", dir.getAbsolutePath(), ex.getMessage());
}
int totalBytes = entries.stream().mapToInt(entry -> (int) Math.min(Integer.MAX_VALUE, entry.sizeBytes())).sum();
OrphanFileReport report = new OrphanFileReport(entries, totalBytes);
if (!report.isEmpty()) {
log.info("temp orphan file report: count={} totalBytes={} dir={}",
report.entries().size(), totalBytes, dir.getAbsolutePath());
for (OrphanFileEntry entry : report.entries()) {
log.info("temp orphan file: path={} sizeBytes={} lastModifiedAt={}",
entry.path(), entry.sizeBytes(), entry.lastModifiedAt());
}
}
return report;
}
/**
* 超保留期仍被引用巡检(task-160)。
*
* 只读:临时文件超保留期但仍被任务引用的清单(供人工处理);绝不删除。
* 与 {@link #inspectOrphanFiles} 互补(该处报"未引用",此处报"被引用")。
*/
public OrphanFileReport inspectOverRetentionReferenced(File dir, Instant expireBefore, Predicate<String> isReferenced) {
List<OrphanFileEntry> entries = new ArrayList<>();
if (dir == null || !dir.isDirectory() || expireBefore == null) {
return new OrphanFileReport(entries, 0);
}
try (Stream<Path> paths = Files.walk(dir.toPath())) {
paths.filter(Files::isRegularFile).forEach(path -> {
File file = path.toFile();
Instant lastModified = TempFileMetadata.lastModified(file);
if (!lastModified.isBefore(expireBefore)) {
return;
}
boolean referenced;
try {
referenced = isReferenced != null && isReferenced.test(file.getName());
} catch (Exception ex) {
log.warn("over-retention reference check failed file={} err={}",
file.getName(), ex.getMessage());
return;
}
if (referenced) {
entries.add(new OrphanFileEntry(
path.toString(),
file.length(),
lastModified.toString()));
}
});
} catch (IOException ex) {
log.warn("over-retention inspect walk failed dir={} err={}", dir.getAbsolutePath(), ex.getMessage());
}
OrphanFileReport report = new OrphanFileReport(entries,
entries.stream().mapToInt(entry -> (int) Math.min(Integer.MAX_VALUE, entry.sizeBytes())).sum());
if (!report.isEmpty()) {
log.info("temp over-retention referenced report: count={} dir={}",
report.entries().size(), dir.getAbsolutePath());
for (OrphanFileEntry entry : report.entries()) {
log.info("temp over-retention referenced: path={} sizeBytes={} lastModifiedAt={}",
entry.path(), entry.sizeBytes(), entry.lastModifiedAt());
}
}
return report;
}
}
@@ -686,19 +686,13 @@ public class ShopDataCrawlTaskService {
List<FileResultEntity> taskRows = listTaskRows(taskId);
try (DailyLockSet dailyLocks = acquireDailyLocks(taskRows)) {
ensureDailySyncCompletedBeforeDelete(taskRows);
Set<Long> removedResultIds = taskRows.stream()
.map(FileResultEntity::getId)
.filter(id -> id != null && id > 0)
.collect(java.util.stream.Collectors.toSet());
Set<Long> removedResultIds = collectResultIds(taskRows);
// A task deletion is only a frontend task-record cleanup. The daily
// workbook is an independent backend aggregate and must not roll
// back when its source task is removed.
DailyDeletionResult dailyResult = preserveDailyForTaskDeletion(removedResultIds);
registerUploadedObjectRollback(dailyResult.uploadedObjectKeys());
List<String> resultFileUrls = new ArrayList<>(dailyResult.obsoleteObjectKeys());
resultFileUrls.addAll(taskRows.stream()
.map(FileResultEntity::getResultFileUrl).filter(url -> !blank(url)).distinct().toList());
resultFileUrls = resultFileUrls.stream().filter(url -> !blank(url)).distinct().toList();
List<String> resultFileUrls = collectResultFileUrls(taskRows, dailyResult.obsoleteObjectKeys());
fileResultMapper.delete(new LambdaQueryWrapper<FileResultEntity>()
.eq(FileResultEntity::getTaskId, taskId)
.eq(FileResultEntity::getModuleType, MODULE_TYPE));
@@ -713,6 +707,32 @@ public class ShopDataCrawlTaskService {
}
}
/** 删除任务的纯计算:收集有效结果行 id 集合(过滤 null/非正数),无副作用。 */
static Set<Long> collectResultIds(List<FileResultEntity> taskRows) {
if (taskRows == null || taskRows.isEmpty()) {
return Set.of();
}
return taskRows.stream()
.filter(Objects::nonNull)
.map(FileResultEntity::getId)
.filter(id -> id != null && id > 0)
.collect(java.util.stream.Collectors.toSet());
}
/** 删除任务的纯计算:合并待清理对象键,过滤空白并去重,无副作用。 */
static List<String> collectResultFileUrls(List<FileResultEntity> taskRows, List<String> obsoleteObjectKeys) {
List<String> urls = new ArrayList<>(obsoleteObjectKeys == null ? List.of() : obsoleteObjectKeys);
if (taskRows != null) {
for (FileResultEntity row : taskRows) {
if (row == null || row.getResultFileUrl() == null || row.getResultFileUrl().isBlank()) {
continue;
}
urls.add(row.getResultFileUrl());
}
}
return urls.stream().filter(url -> url != null && !url.isBlank()).distinct().toList();
}
private void ensureDailySyncCompletedBeforeDelete(List<FileResultEntity> taskRows) {
if (taskRows == null || taskRows.isEmpty()) {
throw new BusinessException("后台店铺数据尚未完成同步,暂不能删除任务");
@@ -2132,6 +2152,7 @@ public class ShopDataCrawlTaskService {
LocalDateTime now = dailyFileService.currentBusinessDateTime();
if (dailyFile == null) {
dailyFile = new ShopDataCrawlDailyFileEntity();
dailyFile.setUserId(row.getUserId());
dailyFile.setShopKeyHash(shopKeyHash);
dailyFile.setShopKey(shopKey);
dailyFile.setBusinessDate(businessDate);
@@ -1,82 +0,0 @@
package com.nanri.aiimage.modules.shopkey.controller;
import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.modules.shopkey.model.dto.ShopCredentialCheckCreateRequest;
import com.nanri.aiimage.modules.shopkey.model.dto.ShopCredentialCheckReportRequest;
import com.nanri.aiimage.modules.shopkey.model.vo.ShopCredentialCheckClaimVo;
import com.nanri.aiimage.modules.shopkey.model.vo.ShopCredentialCheckVo;
import com.nanri.aiimage.modules.shopkey.service.ShopCredentialCheckService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.validation.Valid;
import lombok.RequiredArgsConstructor;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
/**
* 店铺密码检测:后台发起任务,在线客户端轮询领取并真实执行,结果回传后台展示。
* 所有端点均要求 X-Internal-Token(与 /credential 一致),仅供内部自动化调用。
*/
@RestController
@RequiredArgsConstructor
@RequestMapping("/api/admin/shop-credential-checks")
@Tag(name = "店铺密码检测", description = "后台发起、客户端执行、回传展示")
public class ShopCredentialCheckController {
@Value("${aiimage.security.internal-token:}")
private String internalToken;
private final ShopCredentialCheckService shopCredentialCheckService;
@PostMapping
@Operation(summary = "发起店铺密码检测", description = "同一店铺存在未完成检测时复用已有任务")
public ApiResponse<ShopCredentialCheckVo> create(
@Valid @RequestBody ShopCredentialCheckCreateRequest request,
@RequestHeader(value = "X-Internal-Token", required = false) String token) {
requireInternalToken(token);
return ApiResponse.success("检测任务已创建", shopCredentialCheckService.create(request.getShopName()));
}
@GetMapping("/poll")
@Operation(summary = "客户端轮询领取待执行检测任务", description = "无任务返回 data=null")
public ApiResponse<ShopCredentialCheckClaimVo> poll(
@RequestParam(value = "clientHost", required = false) String clientHost,
@RequestHeader(value = "X-Internal-Token", required = false) String token) {
requireInternalToken(token);
return ApiResponse.success(shopCredentialCheckService.claimForClient(clientHost));
}
@PostMapping("/{id}/report")
@Operation(summary = "客户端回传检测结果")
public ApiResponse<Void> report(
@Parameter(description = "检测任务 ID", required = true) @PathVariable Long id,
@Valid @RequestBody(required = false) ShopCredentialCheckReportRequest request,
@RequestHeader(value = "X-Internal-Token", required = false) String token) {
requireInternalToken(token);
shopCredentialCheckService.report(id, request);
return ApiResponse.success("检测结果已记录", null);
}
@GetMapping("/latest")
@Operation(summary = "查询店铺最近一次检测结果", description = "店铺无检测记录时返回 data=null")
public ApiResponse<ShopCredentialCheckVo> latest(
@RequestParam("shopId") Long shopId,
@RequestHeader(value = "X-Internal-Token", required = false) String token) {
requireInternalToken(token);
return ApiResponse.success(shopCredentialCheckService.latestByShopId(shopId));
}
private void requireInternalToken(String token) {
if (internalToken == null || internalToken.isBlank() || token == null || !internalToken.equals(token)) {
throw new com.nanri.aiimage.common.exception.BusinessException("无权访问");
}
}
}
@@ -1,9 +0,0 @@
package com.nanri.aiimage.modules.shopkey.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.nanri.aiimage.modules.shopkey.model.entity.ShopCredentialCheckEntity;
import org.apache.ibatis.annotations.Mapper;
@Mapper
public interface ShopCredentialCheckMapper extends BaseMapper<ShopCredentialCheckEntity> {
}
@@ -1,14 +0,0 @@
package com.nanri.aiimage.modules.shopkey.model.dto;
import io.swagger.v3.oas.annotations.media.Schema;
import jakarta.validation.constraints.NotBlank;
import lombok.Data;
@Data
@Schema(description = "发起店铺密码检测请求")
public class ShopCredentialCheckCreateRequest {
@NotBlank(message = "店铺名称不能为空")
@Schema(description = "店铺名称,按 biz_shop_manage.shop_name 定位", example = "美国站-主营")
private String shopName;
}
@@ -1,21 +0,0 @@
package com.nanri.aiimage.modules.shopkey.model.dto;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.Data;
@Data
@Schema(description = "客户端回传密码检测结果")
public class ShopCredentialCheckReportRequest {
@Schema(description = "检测结果:SUCCESS=密码正确;FAILED=密码错误;NO_NEED_LOGIN=店铺已登录态(无法直接判定密码);ERROR=打开店铺/执行异常", example = "SUCCESS")
private String status;
@Schema(description = "结果详情,用于后台展示与排查", example = "账号或密码错误,请重试")
private String detail;
@Schema(description = "校验失败时的登录接口返回体摘要", example = "{\"error\":\"Incorrect password\"}")
private String raw;
@Schema(description = "客户端主机标识", example = "PC-2024001")
private String clientHost;
}
@@ -1,33 +0,0 @@
package com.nanri.aiimage.modules.shopkey.model.entity;
import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableField;
import com.baomidou.mybatisplus.annotation.TableId;
import com.baomidou.mybatisplus.annotation.TableName;
import lombok.Data;
import java.time.LocalDateTime;
@Data
@TableName("biz_shop_credential_check")
public class ShopCredentialCheckEntity {
@TableId(type = IdType.AUTO)
private Long id;
@TableField("shop_id")
private Long shopId;
@TableField("shop_name")
private String shopName;
private String status;
private String detail;
@TableField("client_host")
private String clientHost;
@TableField("try_requested_at")
private LocalDateTime tryRequestedAt;
@TableField("check_started_at")
private LocalDateTime checkStartedAt;
@TableField("check_finished_at")
private LocalDateTime checkFinishedAt;
private LocalDateTime createdAt;
private LocalDateTime updatedAt;
}
@@ -1,26 +0,0 @@
package com.nanri.aiimage.modules.shopkey.model.vo;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.Data;
import java.time.LocalDateTime;
/**
* 客户端轮询领取到的待执行密码检测任务(不包含任何敏感信息)。
*/
@Data
@Schema(description = "客户端领取的密码检测任务")
public class ShopCredentialCheckClaimVo {
@Schema(description = "检测任务 ID", example = "31")
private Long id;
@Schema(description = "店铺名称")
private String shopName;
@Schema(description = "紫鸟账号(znUsername 为空时客户端用默认)")
private String znUsername;
@Schema(description = "发起时间")
private LocalDateTime tryRequestedAt;
}
@@ -1,38 +0,0 @@
package com.nanri.aiimage.modules.shopkey.model.vo;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.Data;
import java.time.LocalDateTime;
@Data
@Schema(description = "店铺密码检测任务视图")
public class ShopCredentialCheckVo {
@Schema(description = "检测任务 ID", example = "31")
private Long id;
@Schema(description = "店铺 ID")
private Long shopId;
@Schema(description = "店铺名称")
private String shopName;
@Schema(description = "状态:PENDING/RUNNING/SUCCESS/FAILED/NO_NEED_LOGIN/ERROR")
private String status;
@Schema(description = "结果详情")
private String detail;
@Schema(description = "执行客户端标识")
private String clientHost;
@Schema(description = "发起时间")
private LocalDateTime tryRequestedAt;
@Schema(description = "执行开始时间")
private LocalDateTime checkStartedAt;
@Schema(description = "执行完成时间")
private LocalDateTime checkFinishedAt;
}
@@ -16,8 +16,6 @@ public class ShopManageItemVo {
private String account;
private String password;
private String passwordMasked;
/** 最近一次密码检测结果视图;从未检测过为 null。 */
private ShopCredentialCheckVo latestCheck;
private LocalDateTime createdAt;
private LocalDateTime updatedAt;
}
@@ -1,230 +0,0 @@
package com.nanri.aiimage.modules.shopkey.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.modules.shopkey.mapper.ShopCredentialCheckMapper;
import com.nanri.aiimage.modules.shopkey.mapper.ShopManageMapper;
import com.nanri.aiimage.modules.shopkey.model.dto.ShopCredentialCheckReportRequest;
import com.nanri.aiimage.modules.shopkey.model.entity.ShopCredentialCheckEntity;
import com.nanri.aiimage.modules.shopkey.model.entity.ShopManageEntity;
import com.nanri.aiimage.modules.shopkey.model.vo.ShopCredentialCheckClaimVo;
import com.nanri.aiimage.modules.shopkey.model.vo.ShopCredentialCheckVo;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.time.LocalDateTime;
import java.util.List;
/**
* 店铺密码检测任务:后台发起 → 在线客户端轮询领取 → 真实打开紫鸟店铺
* 并尝试登录亚马逊 → 回传结果 → 后台店铺管理页展示。
*/
@Service
@RequiredArgsConstructor
@Slf4j
public class ShopCredentialCheckService {
public static final String STATUS_PENDING = "PENDING";
public static final String STATUS_RUNNING = "RUNNING";
public static final String STATUS_SUCCESS = "SUCCESS";
public static final String STATUS_FAILED = "FAILED";
public static final String STATUS_NO_NEED_LOGIN = "NO_NEED_LOGIN";
public static final String STATUS_ERROR = "ERROR";
/** 客户端领取任务时最久保留的 PENDING 老任务(超过则标记过期)。 */
private static final int PENDING_ACCEPT_MINUTES = 60;
/** RUNNING 执行超时(客户端崩溃/断网),超过则回收为 PENDING 供其他客户端重试。 */
private static final int RUNNING_STALE_MINUTES = 30;
private final ShopCredentialCheckMapper shopCredentialCheckMapper;
private final ShopManageMapper shopManageMapper;
@Transactional
public ShopCredentialCheckVo create(String shopName) {
ShopManageEntity shop = requireShopByName(shopName);
// 同一店铺已有未完成任务(PENDING/RUNNING)时复用,避免重复弹出多个浏览器窗口
ShopCredentialCheckEntity active = shopCredentialCheckMapper.selectOne(new LambdaQueryWrapper<ShopCredentialCheckEntity>()
.eq(ShopCredentialCheckEntity::getShopId, shop.getId())
.in(ShopCredentialCheckEntity::getStatus, STATUS_PENDING, STATUS_RUNNING)
.orderByDesc(ShopCredentialCheckEntity::getId)
.last("limit 1"));
if (active != null) {
return toVo(active);
}
ShopCredentialCheckEntity entity = new ShopCredentialCheckEntity();
entity.setShopId(shop.getId());
entity.setShopName(shop.getShopName());
entity.setStatus(STATUS_PENDING);
entity.setTryRequestedAt(LocalDateTime.now());
shopCredentialCheckMapper.insert(entity);
log.info("[shop-credential-check] created id={} shopId={} shopName={}", entity.getId(), shop.getId(), shop.getShopName());
return toVo(entity);
}
/**
* 客户端轮询领取:返回一条 PENDING 任务(跨店铺按 id 升序),
* 并原子置为 RUNNING;无任务时返回 null。
*/
@Transactional
public ShopCredentialCheckClaimVo claimForClient(String clientHost) {
recycleStaleRunning();
expireAbandonedPending();
ShopCredentialCheckEntity pending = shopCredentialCheckMapper.selectOne(new LambdaQueryWrapper<ShopCredentialCheckEntity>()
.eq(ShopCredentialCheckEntity::getStatus, STATUS_PENDING)
.gt(ShopCredentialCheckEntity::getTryRequestedAt, LocalDateTime.now().minusMinutes(PENDING_ACCEPT_MINUTES))
.orderByAsc(ShopCredentialCheckEntity::getId)
.last("limit 1"));
if (pending == null) {
return null;
}
int updated = shopCredentialCheckMapper.update(null, new LambdaUpdateWrapper<ShopCredentialCheckEntity>()
.eq(ShopCredentialCheckEntity::getId, pending.getId())
.eq(ShopCredentialCheckEntity::getStatus, STATUS_PENDING)
.set(ShopCredentialCheckEntity::getStatus, STATUS_RUNNING)
.set(ShopCredentialCheckEntity::getCheckStartedAt, LocalDateTime.now())
.set(ShopCredentialCheckEntity::getClientHost, clientHost));
if (updated == 0) {
// 被其他客户端抢先领取
return null;
}
log.info("[shop-credential-check] claimed id={} shopName={} clientHost={}", pending.getId(), pending.getShopName(), clientHost);
ShopCredentialCheckClaimVo vo = new ShopCredentialCheckClaimVo();
vo.setId(pending.getId());
vo.setShopName(pending.getShopName());
vo.setTryRequestedAt(pending.getTryRequestedAt());
try {
vo.setZnUsername(findZnUsernameByShopName(pending.getShopName()));
} catch (Exception ex) {
log.warn("[shop-credential-check] resolve znUsername failed shopName={} msg={}", pending.getShopName(), ex.getMessage());
}
return vo;
}
@Transactional
public void report(Long id, ShopCredentialCheckReportRequest request) {
ShopCredentialCheckEntity entity = getById(id);
if (!STATUS_RUNNING.equals(entity.getStatus())) {
log.warn("[shop-credential-check] ignore stale report id={} currentStatus={}", id, entity.getStatus());
return;
}
String status = request == null ? null : request.getStatus();
if (!List.of(STATUS_SUCCESS, STATUS_FAILED, STATUS_NO_NEED_LOGIN, STATUS_ERROR).contains(status)) {
throw new BusinessException("不支持的检测结果状态: " + status);
}
entity.setStatus(status);
entity.setDetail(request.getDetail());
entity.setClientHost(firstNonBlank(request.getClientHost(), entity.getClientHost()));
entity.setCheckFinishedAt(LocalDateTime.now());
shopCredentialCheckMapper.updateById(entity);
log.info("[shop-credential-check] reported id={} shopName={} status={} detail={}",
id, entity.getShopName(), status, request.getDetail());
}
public ShopCredentialCheckVo latestByShopId(Long shopId) {
if (shopId == null || shopId <= 0) {
return null;
}
ShopCredentialCheckEntity entity = shopCredentialCheckMapper.selectOne(new LambdaQueryWrapper<ShopCredentialCheckEntity>()
.eq(ShopCredentialCheckEntity::getShopId, shopId)
.orderByDesc(ShopCredentialCheckEntity::getId)
.last("limit 1"));
return entity == null ? null : toVo(entity);
}
private ShopCredentialCheckEntity getById(Long id) {
ShopCredentialCheckEntity entity = shopCredentialCheckMapper.selectById(id);
if (entity == null) {
throw new BusinessException("密码检测任务不存在");
}
return entity;
}
private ShopManageEntity requireShopByName(String shopName) {
String normalized = shopName == null ? "" : shopName.trim();
if (normalized.isEmpty()) {
throw new BusinessException("店铺名称不能为空");
}
ShopManageEntity entity = shopManageMapper.selectOne(new LambdaQueryWrapper<ShopManageEntity>()
.eq(ShopManageEntity::getShopName, normalized)
.last("limit 1"));
if (entity == null) {
throw new BusinessException("后台店铺管理中未找到店铺:" + normalized + ",请先添加店铺信息");
}
return entity;
}
private String findZnUsernameByShopName(String shopName) {
String normalized = shopName == null ? "" : shopName.trim();
if (normalized.isEmpty()) {
return null;
}
ShopManageEntity entity = shopManageMapper.selectOne(new LambdaQueryWrapper<ShopManageEntity>()
.select(ShopManageEntity::getZnUsername)
.eq(ShopManageEntity::getShopName, normalized)
.last("limit 1"));
return entity == null ? null : entity.getZnUsername();
}
/**
* 客户端执行超时(崩溃/断网)的 RUNNING 回收为 PENDING,供其他在线客户端重试。
*/
private void recycleStaleRunning() {
List<ShopCredentialCheckEntity> stale = shopCredentialCheckMapper.selectList(new LambdaQueryWrapper<ShopCredentialCheckEntity>()
.eq(ShopCredentialCheckEntity::getStatus, STATUS_RUNNING)
.lt(ShopCredentialCheckEntity::getCheckStartedAt, LocalDateTime.now().minusMinutes(RUNNING_STALE_MINUTES)));
for (ShopCredentialCheckEntity entity : stale) {
int updated = shopCredentialCheckMapper.update(null, new LambdaUpdateWrapper<ShopCredentialCheckEntity>()
.eq(ShopCredentialCheckEntity::getId, entity.getId())
.eq(ShopCredentialCheckEntity::getStatus, STATUS_RUNNING)
.set(ShopCredentialCheckEntity::getStatus, STATUS_PENDING)
.set(ShopCredentialCheckEntity::getCheckStartedAt, null));
if (updated > 0) {
log.warn("[shop-credential-check] recycled stale RUNNING id={} shopName={} clientHost={}",
entity.getId(), entity.getShopName(), entity.getClientHost());
}
}
}
/**
* 后台发起后长时间无人领取的 PENDING 标记 ERROR;客户端领取入口只捡 60 分钟内新发的,
* 这里只清理历史残留,防止 PENDING 无限堆积。
*/
private void expireAbandonedPending() {
List<ShopCredentialCheckEntity> abandoned = shopCredentialCheckMapper.selectList(new LambdaQueryWrapper<ShopCredentialCheckEntity>()
.eq(ShopCredentialCheckEntity::getStatus, STATUS_PENDING)
.le(ShopCredentialCheckEntity::getTryRequestedAt, LocalDateTime.now().minusMinutes(PENDING_ACCEPT_MINUTES))
.last("limit 50"));
for (ShopCredentialCheckEntity entity : abandoned) {
int updated = shopCredentialCheckMapper.update(null, new LambdaUpdateWrapper<ShopCredentialCheckEntity>()
.eq(ShopCredentialCheckEntity::getId, entity.getId())
.eq(ShopCredentialCheckEntity::getStatus, STATUS_PENDING)
.set(ShopCredentialCheckEntity::getStatus, STATUS_ERROR)
.set(ShopCredentialCheckEntity::getDetail, "超过 " + PENDING_ACCEPT_MINUTES + " 分钟无在线客户端领取,已自动过期")
.set(ShopCredentialCheckEntity::getCheckFinishedAt, LocalDateTime.now()));
if (updated > 0) {
log.warn("[shop-credential-check] expired abandoned PENDING id={} shopName={}", entity.getId(), entity.getShopName());
}
}
}
private ShopCredentialCheckVo toVo(ShopCredentialCheckEntity entity) {
ShopCredentialCheckVo vo = new ShopCredentialCheckVo();
vo.setId(entity.getId());
vo.setShopId(entity.getShopId());
vo.setShopName(entity.getShopName());
vo.setStatus(entity.getStatus());
vo.setDetail(entity.getDetail());
vo.setClientHost(entity.getClientHost());
vo.setTryRequestedAt(entity.getTryRequestedAt());
vo.setCheckStartedAt(entity.getCheckStartedAt());
vo.setCheckFinishedAt(entity.getCheckFinishedAt());
return vo;
}
private String firstNonBlank(String value, String fallback) {
return value == null || value.isBlank() ? fallback : value;
}
}
@@ -3,14 +3,11 @@ package com.nanri.aiimage.modules.shopkey.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.common.security.ShopCredentialCryptoService;
import com.nanri.aiimage.modules.shopkey.mapper.ShopCredentialCheckMapper;
import com.nanri.aiimage.modules.shopkey.mapper.ShopManageMapper;
import com.nanri.aiimage.modules.shopkey.model.dto.ShopManageCreateRequest;
import com.nanri.aiimage.modules.shopkey.model.dto.ShopManageUpdateRequest;
import com.nanri.aiimage.modules.shopkey.model.entity.ShopCredentialCheckEntity;
import com.nanri.aiimage.modules.shopkey.model.entity.ShopManageEntity;
import com.nanri.aiimage.modules.shopkey.model.entity.ShopManageGroupEntity;
import com.nanri.aiimage.modules.shopkey.model.vo.ShopCredentialCheckVo;
import com.nanri.aiimage.modules.shopkey.model.vo.ShopManageCredentialVo;
import com.nanri.aiimage.modules.shopkey.model.vo.ShopManageItemVo;
import com.nanri.aiimage.modules.shopkey.model.vo.ShopManagePageVo;
@@ -18,7 +15,6 @@ import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -30,7 +26,6 @@ public class ShopManageService {
private final ShopManageMapper shopManageMapper;
private final ShopManageGroupService shopManageGroupService;
private final ShopCredentialCryptoService shopCredentialCryptoService;
private final ShopCredentialCheckMapper shopCredentialCheckMapper;
public ShopManagePageVo page(long page, long pageSize, Long groupId, String shopName, Long operatorId, boolean superAdmin) {
long safePage = Math.max(page, 1);
@@ -74,14 +69,8 @@ public class ShopManageService {
.distinct()
.toList());
Map<Long, ShopCredentialCheckVo> latestCheckByShopId = buildLatestCheckMap(rows);
List<ShopManageItemVo> items = rows.stream()
.map(entity -> {
ShopManageItemVo vo = toItemVo(entity, groupNameById.get(entity.getGroupId()));
vo.setLatestCheck(latestCheckByShopId.get(entity.getId()));
return vo;
})
.map(entity -> toItemVo(entity, groupNameById.get(entity.getGroupId())))
.toList();
ShopManagePageVo vo = new ShopManagePageVo();
vo.setItems(items);
@@ -199,43 +188,6 @@ public class ShopManageService {
return entity;
}
/**
* 一次查询本页所有店铺 id 的检测记录(id 倒序),取每个店铺 id 的第一条即最近一次。
*/
private Map<Long, ShopCredentialCheckVo> buildLatestCheckMap(List<ShopManageEntity> rows) {
List<Long> shopIds = rows.stream()
.map(ShopManageEntity::getId)
.filter(id -> id != null && id > 0)
.distinct()
.toList();
if (shopIds.isEmpty()) {
return Map.of();
}
List<ShopCredentialCheckEntity> checks = shopCredentialCheckMapper.selectList(
new LambdaQueryWrapper<ShopCredentialCheckEntity>()
.in(ShopCredentialCheckEntity::getShopId, shopIds)
.orderByDesc(ShopCredentialCheckEntity::getId));
Map<Long, ShopCredentialCheckVo> map = new LinkedHashMap<>();
for (ShopCredentialCheckEntity check : checks) {
map.putIfAbsent(check.getShopId(), toCheckVo(check));
}
return map;
}
private ShopCredentialCheckVo toCheckVo(ShopCredentialCheckEntity entity) {
ShopCredentialCheckVo vo = new ShopCredentialCheckVo();
vo.setId(entity.getId());
vo.setShopId(entity.getShopId());
vo.setShopName(entity.getShopName());
vo.setStatus(entity.getStatus());
vo.setDetail(entity.getDetail());
vo.setClientHost(entity.getClientHost());
vo.setTryRequestedAt(entity.getTryRequestedAt());
vo.setCheckStartedAt(entity.getCheckStartedAt());
vo.setCheckFinishedAt(entity.getCheckFinishedAt());
return vo;
}
private void validateGroupAccess(ShopManageEntity entity, Long operatorId, boolean superAdmin) {
shopManageGroupService.getAccessibleById(entity.getGroupId(), operatorId, superAdmin);
}
@@ -1030,15 +1030,17 @@ public class SimilarAsinTaskService {
if (existing != null) {
return new PreparedSubmittedChunk(
taskId, scopeKey, scopeHash, chunkIndex, chunkTotal, done, request.getError(),
null, null, false, taskMetadata);
null, null, false, taskMetadata, null);
}
String payloadJson = writeJson(flattenSubmittedRows(request), "结果序列化失败");
String storedPayload = transientPayloadStorageService.storeChunkPayloadVersioned(
MODULE_TYPE, taskId, scopeHash, chunkIndex, payloadJson);
boolean localFallback = transientPayloadStorageService.wasLastStoreLocalFallback();
// 纯计算在事务外完成payload 哈希预计算persist 落库时直接引用
return new PreparedSubmittedChunk(
taskId, scopeKey, scopeHash, chunkIndex, chunkTotal, done, request.getError(),
payloadJson, storedPayload, localFallback, taskMetadata);
payloadJson, storedPayload, localFallback, taskMetadata,
DigestUtil.sha256Hex(payloadJson));
}
private SubmittedTaskMetadata readSubmittedTaskMetadata(FileTaskEntity task) {
@@ -1146,7 +1148,7 @@ public class SimilarAsinTaskService {
chunk.setChunkIndex(prepared.chunkIndex());
chunk.setChunkTotal(prepared.chunkTotal());
chunk.setPayloadJson(prepared.storedPayload());
chunk.setPayloadHash(DigestUtil.sha256Hex(prepared.payloadJson()));
chunk.setPayloadHash(prepared.payloadHash());
chunk.setCreatedAt(LocalDateTime.now());
chunk.setUpdatedAt(LocalDateTime.now());
try {
@@ -4623,7 +4625,8 @@ public class SimilarAsinTaskService {
String isConform = resultRow == null ? "" : userFacingLlmCellValue(resultRow, resultRow.getIsConform());
String reason = resultRow == null ? "" : userFacingLlmCellValue(resultRow, resultRow.getReason());
String category = resultRow == null ? "" : userFacingLlmCellValue(resultRow, resultRow.getCategory());
String status = resultRow == null || isSilentOutputFailure(resultRow)
String status = resultRow == null
|| (isSilentOutputFailure(resultRow) && !isAsinNotFound(resultRow.getError()))
? ""
: resolveResultStatus(resultRow, isStock, similarity, isConform, reason, category);
row.createCell(col++).setCellValue(isStock);
@@ -4858,6 +4861,12 @@ public class SimilarAsinTaskService {
if (hasText(row.getMainUrl()) || hasText(row.getPuzzleImg1()) || hasText(row.getPuzzleImg2())) {
return "\u6210\u529f";
}
// Python \u7aef\u641c\u4e0d\u5230\u5546\u54c1\uff08404\uff09\u7684\u884c\uff1a\u72b6\u6001\u5217\u660e\u786e\u6807\u8bb0\uff0c\u4eba\u5de5\u590d\u6838\u65f6\u4e00\u773c\u53ef\u8fa8\u3002
// \u653e\u5728\u56fe\u7247/\u72b6\u6001\u5224\u65ad\u4e4b\u524d\uff1a404 \u884c\u5fc5\u7136\u65e0\u56fe\u4e14 status \u5df2\u88ab markRowFailed \u7f6e\u4e3a FAILED\uff0c
// \u4e0d\u80fd\u843d\u56de"\u5931\u8d25"\uff08\u90a3\u662f LLM/\u751f\u6210\u94fe\u8def\u5931\u8d25\u8bed\u4e49\uff0c\u4e0e"ASIN \u641c\u4e0d\u5230"\u662f\u4e24\u56de\u4e8b\uff09\u3002
if (isAsinNotFound(row.getError())) {
return "ASIN\u672a\u627e\u5230";
}
String rawStatus = row.getStatus() == null ? "" : row.getStatus().trim();
if (!rawStatus.isBlank()) {
return isFailedStatus(rawStatus) ? "\u5931\u8d25" : "\u6210\u529f";
@@ -4872,6 +4881,22 @@ public class SimilarAsinTaskService {
return hasText(row.getConclusion()) ? "\u6210\u529f" : "\u5931\u8d25";
}
/**
* \u5224\u5b9a\u662f\u5426"ASIN \u672a\u627e\u5230"\uff1aPython \u7aef\u6293\u53d6 404\uff08error="404 Not Found"\uff09\u6216\u7b49\u4ef7\u6587\u6848\u3002
* \u547d\u4e2d\u8be5\u5224\u5b9a\u7684\u884c\u4e0d\u4f1a\u88ab\u5f53\u6210\u751f\u6210\u5931\u8d25\u8ba1\u91cd\u8bd5\uff0c\u4e5f\u4e0d\u4f1a\u88ab\u5bfc\u51fa\u8fc7\u6ee4\uff0c\u72b6\u6001\u5217\u663e\u793a"ASIN\u672a\u627e\u5230"\u3002
*/
static boolean isAsinNotFound(String error) {
String normalized = error == null ? "" : error.toLowerCase(Locale.ROOT).trim();
if (normalized.isBlank()) {
return false;
}
return normalized.contains("404")
|| normalized.contains("not found")
|| normalized.contains("asin \u672a\u627e\u5230")
|| normalized.contains("asin\u4e0d\u5b58\u5728")
|| normalized.contains("\u5546\u54c1\u672a\u627e\u5230");
}
private static boolean isFailedStatus(String status) {
String normalized = status == null ? "" : status.trim().toLowerCase(Locale.ROOT);
return normalized.contains("fail")
@@ -4895,6 +4920,11 @@ public class SimilarAsinTaskService {
if (error.isBlank()) {
return false;
}
// Python 端搜不到商品的 404 虽然 title/url/图全空也必须导出
// 并在状态列标记"ASIN未找到"供人工复核"其它错误需带 title/url/图"之前判断
if (isAsinNotFound(error)) {
return true;
}
return !normalize(row.getTitle()).isBlank()
|| !normalize(row.getUrl()).isBlank()
|| row.hasImageUrl();
@@ -5750,7 +5780,8 @@ public class SimilarAsinTaskService {
String payloadJson,
String storedPayload,
boolean localFallback,
SubmittedTaskMetadata taskMetadata) {
SubmittedTaskMetadata taskMetadata,
String payloadHash) {
}
private record PersistSubmittedChunkResult(SubmitContext context,
@@ -237,9 +237,14 @@ public class SimilarAsinPerfFixture {
return true;
}
/** 采样当前堆已用字节峰值;MXBean 不可用时返回 -1,与 "> 0" 类断言不冲突(GC 场景始终可用)。 */
/**
* 采样当前堆已用字节峰值;MXBean 不可用时返回 -1,与 "> 0" 类断言不冲突(GC 场景始终可用)。
* 采样前先触发一次 GC:全量套件运行时前序测试的残留垃圾会让"已用堆"短暂虚高,
* 使 1GB 峰值边界断言在全量回归下随机失败;GC 后采样反映本夹具自身的存活集。
*/
private static long sampledPeakHeapBytes() {
try {
System.gc();
return Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
} catch (Exception ex) {
return -1;
@@ -0,0 +1,82 @@
package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskFileJobMapper;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
/**
* 已完成任务活跃 Job 巡检(task-159)。
*
* 只读巡检:任务已终态(SUCCESS/FAILED)但仍有活跃(RUNNING/PENDINGJob 的
* 清单输出报表;运行中任务的活跃 Job 属正常;绝不修改任何状态。可重复执行。
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class CompletedTaskActiveJobInspector {
private final FileTaskMapper fileTaskMapper;
private final TaskFileJobMapper taskFileJobMapper;
public record ActiveJobEntry(Long taskId, String moduleType, Long jobId, String jobStatus, String jobType) {
}
public record ActiveJobReport(List<ActiveJobEntry> entries) {
public boolean isEmpty() {
return entries == null || entries.isEmpty();
}
}
public ActiveJobReport inspectTerminalTasksWithActiveJobs(int limit) {
List<FileTaskEntity> tasks = fileTaskMapper.selectList(new LambdaQueryWrapper<FileTaskEntity>()
.in(FileTaskEntity::getStatus, List.of("SUCCESS", "FAILED"))
.orderByAsc(FileTaskEntity::getUpdatedAt)
.last("limit " + Math.max(1, Math.min(limit, 500))));
if (tasks == null || tasks.isEmpty()) {
return new ActiveJobReport(List.of());
}
Set<Long> taskIds = tasks.stream()
.map(FileTaskEntity::getId)
.filter(id -> id != null && id > 0)
.collect(Collectors.toSet());
List<TaskFileJobEntity> activeJobs = taskIds.isEmpty() ? List.of()
: taskFileJobMapper.selectList(new LambdaQueryWrapper<TaskFileJobEntity>()
.in(TaskFileJobEntity::getTaskId, taskIds)
.in(TaskFileJobEntity::getStatus, List.of("RUNNING", "PENDING")));
if (activeJobs == null || activeJobs.isEmpty()) {
return new ActiveJobReport(List.of());
}
java.util.Map<Long, FileTaskEntity> taskById = tasks.stream()
.collect(Collectors.toMap(FileTaskEntity::getId, task -> task, (a, b) -> a));
List<ActiveJobEntry> entries = new ArrayList<>();
for (TaskFileJobEntity job : activeJobs) {
FileTaskEntity task = taskById.get(job.getTaskId());
if (task != null) {
entries.add(new ActiveJobEntry(
task.getId(), task.getModuleType(),
job.getId(), job.getStatus(), job.getJobType()));
}
}
ActiveJobReport report = new ActiveJobReport(List.copyOf(entries));
if (!report.isEmpty()) {
log.info("terminal-task-active-job report: count={}", report.entries().size());
for (ActiveJobEntry entry : report.entries()) {
log.info("terminal-task-active-job: taskId={} moduleType={} jobId={} jobStatus={} jobType={}",
entry.taskId(), entry.moduleType(), entry.jobId(), entry.jobStatus(), entry.jobType());
}
}
return report;
}
}
@@ -0,0 +1,81 @@
package com.nanri.aiimage.modules.task.service;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.InspectionProperties;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.modules.file.service.TempOrphanInspector;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.io.File;
import java.time.Duration;
import java.time.Instant;
/**
* 巡检任务调度(task-161)。
*
* 全部巡检由 aiimage.inspection.enabled 开关控制(默认 disabled);
* 启用后按 cron 调度执行;分布式锁防双实例重复输出;单个巡检异常不阻断其余。
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class InspectionScheduler {
private static final Duration INSPECTION_LOCK_TTL = Duration.ofMinutes(10);
private final InspectionProperties inspectionProperties;
private final DistributedJobLockService distributedJobLockService;
private final StorageProperties storageProperties;
private final TempOrphanInspector tempOrphanInspector;
private final OrphanJobInspector orphanJobInspector;
private final TaskResultMissingInspector taskResultMissingInspector;
private final ResultFileMissingInspector resultFileMissingInspector;
private final CompletedTaskActiveJobInspector completedTaskActiveJobInspector;
@Scheduled(cron = "${aiimage.inspection.cron:0 0 3 * * *}")
public void runInspections() {
if (!inspectionProperties.isEnabled()) {
return;
}
DistributedJobLockService.LockHandle lockHandle =
distributedJobLockService.tryLock("temp-file-inspection", INSPECTION_LOCK_TTL);
if (lockHandle == null) {
log.info("[inspection] skip because another instance holds the inspection lock");
return;
}
try (lockHandle) {
runAllSafely();
}
}
private void runAllSafely() {
int limit = inspectionProperties.getLimit();
runSafely("orphan-file", () -> {
File tempDir = cn.hutool.core.io.FileUtil.file(storageProperties.getLocalTempDir());
if (tempDir.isDirectory()) {
var report = tempOrphanInspector.inspectOrphanFiles(
tempDir, Instant.now().minus(Duration.ofHours(24)), name -> false);
log.info("[inspection] orphan-file report count={}", report.entries().size());
}
});
runSafely("orphan-job", () -> log.info("[inspection] orphan-job report count={}",
orphanJobInspector.inspectOrphanJobs(limit).entries().size()));
runSafely("task-missing-result", () -> log.info("[inspection] task-missing-result report count={}",
taskResultMissingInspector.inspectTasksMissingResult(limit).entries().size()));
runSafely("result-missing-file", () -> log.info("[inspection] result-missing-file report count={}",
resultFileMissingInspector.inspectResultsMissingFile(limit).entries().size()));
runSafely("terminal-active-job", () -> log.info("[inspection] terminal-active-job report count={}",
completedTaskActiveJobInspector.inspectTerminalTasksWithActiveJobs(limit).entries().size()));
}
private void runSafely(String name, Runnable action) {
try {
action.run();
} catch (Exception ex) {
log.warn("[inspection] {} failed msg={}", name, ex.getMessage(), ex);
}
}
}
@@ -0,0 +1,91 @@
package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskFileJobMapper;
import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
/**
* 孤立 Job 巡检(task-156)。
*
* 只读巡检:task_file_job 无对应 task 或 result 的孤儿 Job 清单输出报表
* (日志 + 返回对象);绝不修改 Job 状态。巡检可重复执行。
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class OrphanJobInspector {
private final TaskFileJobMapper taskFileJobMapper;
private final FileTaskMapper fileTaskMapper;
private final FileResultMapper fileResultMapper;
public record OrphanJobEntry(Long jobId, Long taskId, Long resultId, String moduleType,
String status, String updatedAt) {
}
public record OrphanJobReport(List<OrphanJobEntry> entries) {
public boolean isEmpty() {
return entries == null || entries.isEmpty();
}
}
public OrphanJobReport inspectOrphanJobs(int limit) {
List<TaskFileJobEntity> jobs = taskFileJobMapper.selectList(new LambdaQueryWrapper<TaskFileJobEntity>()
.orderByAsc(TaskFileJobEntity::getUpdatedAt)
.last("limit " + Math.max(1, Math.min(limit, 500))));
if (jobs == null || jobs.isEmpty()) {
return new OrphanJobReport(List.of());
}
Set<Long> taskIds = jobs.stream()
.map(TaskFileJobEntity::getTaskId)
.filter(id -> id != null && id > 0)
.collect(Collectors.toSet());
Set<Long> resultIds = jobs.stream()
.map(TaskFileJobEntity::getResultId)
.filter(id -> id != null && id > 0)
.collect(Collectors.toSet());
Set<Long> existingTaskIds = taskIds.isEmpty() ? Set.of()
: fileTaskMapper.selectBatchIds(taskIds).stream()
.map(task -> task.getId())
.collect(Collectors.toSet());
Set<Long> existingResultIds = resultIds.isEmpty() ? Set.of()
: fileResultMapper.selectBatchIds(resultIds).stream()
.map(result -> result.getId())
.collect(Collectors.toSet());
List<OrphanJobEntry> entries = new ArrayList<>();
for (TaskFileJobEntity job : jobs) {
boolean taskMissing = job.getTaskId() != null && job.getTaskId() > 0
&& !existingTaskIds.contains(job.getTaskId());
boolean resultMissing = job.getResultId() != null && job.getResultId() > 0
&& !existingResultIds.contains(job.getResultId());
if (taskMissing || resultMissing) {
entries.add(new OrphanJobEntry(
job.getId(), job.getTaskId(), job.getResultId(), job.getModuleType(),
job.getStatus(),
job.getUpdatedAt() == null ? null : job.getUpdatedAt().toString()));
}
}
OrphanJobReport report = new OrphanJobReport(List.copyOf(entries));
if (!report.isEmpty()) {
log.info("orphan job report: count={}", report.entries().size());
for (OrphanJobEntry entry : report.entries()) {
log.info("orphan job: jobId={} taskId={} resultId={} moduleType={} status={} updatedAt={}",
entry.jobId(), entry.taskId(), entry.resultId(), entry.moduleType(),
entry.status(), entry.updatedAt());
}
}
return report;
}
}
@@ -0,0 +1,80 @@
package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
import java.util.function.Predicate;
/**
* 结果存在文件缺失巡检(task-158)。
*
* 只读巡检:file_result 的 resultFileUrl 非空但对应文件在 OSS 不存在的清单输出
* 报表;空白 URL 行视为"未生成文件"状态、不告警(历史/失败任务无文件属正常);
* 绝不修改任何状态。可重复执行。
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class ResultFileMissingInspector {
private final FileResultMapper fileResultMapper;
private final OssStorageService ossStorageService;
public record MissingFileEntry(Long resultId, Long taskId, String moduleType, String resultFileUrl) {
}
public record MissingFileReport(List<MissingFileEntry> entries) {
public boolean isEmpty() {
return entries == null || entries.isEmpty();
}
}
public MissingFileReport inspectResultsMissingFile(int limit) {
return inspectResultsMissingFile(limit, ossStorageService::objectExists);
}
MissingFileReport inspectResultsMissingFile(int limit, Predicate<String> urlExists) {
List<FileResultEntity> results = fileResultMapper.selectList(new LambdaQueryWrapper<FileResultEntity>()
.isNotNull(FileResultEntity::getResultFileUrl)
.last("limit " + Math.max(1, Math.min(limit, 500))));
if (results == null || results.isEmpty()) {
return new MissingFileReport(List.of());
}
List<MissingFileEntry> entries = new ArrayList<>();
for (FileResultEntity result : results) {
String url = result.getResultFileUrl();
if (url == null || url.isBlank()) {
continue;
}
boolean exists;
try {
exists = urlExists.test(url);
} catch (Exception ex) {
log.warn("result file existence check failed resultId={} url={} err={}",
result.getId(), url, ex.getMessage());
continue;
}
if (!exists) {
entries.add(new MissingFileEntry(
result.getId(), result.getTaskId(), result.getModuleType(), url));
}
}
MissingFileReport report = new MissingFileReport(List.copyOf(entries));
if (!report.isEmpty()) {
log.info("result-missing-file report: count={}", report.entries().size());
for (MissingFileEntry entry : report.entries()) {
log.info("result-missing-file: resultId={} taskId={} moduleType={} url={}",
entry.resultId(), entry.taskId(), entry.moduleType(), entry.resultFileUrl());
}
}
return report;
}
}
@@ -17,6 +17,7 @@ import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
@Service
@RequiredArgsConstructor
@@ -142,15 +143,22 @@ public class TaskFileJobService {
if (candidates == null || candidates.isEmpty()) {
return List.of();
}
List<TaskFileJobEntity> claimed = new ArrayList<>();
List<Long> claimedIds = new ArrayList<>();
for (TaskFileJobEntity candidate : candidates) {
if (candidate == null || candidate.getId() == null) {
continue;
}
if (!markRunning(candidate.getId())) {
continue;
if (markRunning(candidate.getId())) {
claimedIds.add(candidate.getId());
}
TaskFileJobEntity claim = taskFileJobMapper.selectById(candidate.getId());
}
if (claimedIds.isEmpty()) {
return List.of();
}
Map<Long, TaskFileJobEntity> claimedById = refreshJobsByIds(claimedIds);
List<TaskFileJobEntity> claimed = new ArrayList<>();
for (Long id : claimedIds) {
TaskFileJobEntity claim = claimedById.get(id);
if (claim != null && "RUNNING".equals(claim.getStatus())) {
claimed.add(claim);
}
@@ -259,10 +267,16 @@ public class TaskFileJobService {
.last("limit " + Math.max(1, Math.min(limit, 200))));
int reset = 0;
List<TaskFileJobEntity> exhaustedJobs = new ArrayList<>();
List<Long> eventJobIds = new ArrayList<>();
List<Long> exhaustedJobIds = new ArrayList<>();
Map<Long, TaskFileJobEntity> originalExhaustedById = new LinkedHashMap<>();
Map<Long, TaskFileJobEntity> originalJobsById = jobs.stream()
.collect(Collectors.toMap(TaskFileJobEntity::getId, job -> job, (a, b) -> a));
for (TaskFileJobEntity job : jobs) {
if ("FAILED".equals(job.getStatus())
&& job.getRetryCount() != null && job.getRetryCount() >= MAX_RETRY_COUNT) {
exhaustedJobs.add(job);
exhaustedJobIds.add(job.getId());
originalExhaustedById.put(job.getId(), job);
continue;
}
if ("PENDING".equals(job.getStatus())) {
@@ -279,7 +293,7 @@ public class TaskFileJobService {
.set(TaskFileJobEntity::getRetryCount, nextRetryCount)
.set(TaskFileJobEntity::getUpdatedAt, LocalDateTime.now()));
if (updated > 0) {
publishDispatchEvent(taskFileJobMapper.selectById(job.getId()));
eventJobIds.add(job.getId());
}
continue;
}
@@ -300,8 +314,7 @@ public class TaskFileJobService {
.set(TaskFileJobEntity::getFinishedAt, now)
.set(TaskFileJobEntity::getTerminalCallbackAt, null));
if (updated > 0) {
TaskFileJobEntity exhausted = taskFileJobMapper.selectById(job.getId());
exhaustedJobs.add(exhausted == null ? job : exhausted);
exhaustedJobIds.add(job.getId());
}
continue;
}
@@ -312,12 +325,40 @@ public class TaskFileJobService {
.set(TaskFileJobEntity::getTerminalCallbackAt, null));
if (updated > 0) {
reset++;
publishDispatchEvent(taskFileJobMapper.selectById(job.getId()));
eventJobIds.add(job.getId());
}
}
if (!eventJobIds.isEmpty()) {
Map<Long, TaskFileJobEntity> refreshedById = refreshJobsByIds(eventJobIds);
for (Long jobId : eventJobIds) {
publishDispatchEvent(refreshedById.get(jobId));
}
}
if (!exhaustedJobIds.isEmpty()) {
List<Long> refreshIds = exhaustedJobIds.stream()
.filter(id -> !originalExhaustedById.containsKey(id))
.toList();
Map<Long, TaskFileJobEntity> refreshedById = refreshIds.isEmpty()
? Map.of()
: refreshJobsByIds(refreshIds);
for (Long jobId : exhaustedJobIds) {
TaskFileJobEntity original = originalExhaustedById.get(jobId);
if (original != null) {
exhaustedJobs.add(original);
continue;
}
TaskFileJobEntity refreshed = refreshedById.get(jobId);
exhaustedJobs.add(refreshed == null ? originalJobsById.get(jobId) : refreshed);
}
}
return new StuckJobResetResult(reset, List.copyOf(exhaustedJobs));
}
private Map<Long, TaskFileJobEntity> refreshJobsByIds(List<Long> jobIds) {
return taskFileJobMapper.selectBatchIds(jobIds).stream()
.collect(Collectors.toMap(TaskFileJobEntity::getId, job -> job, (a, b) -> a));
}
public record StuckJobResetResult(int resetCount, List<TaskFileJobEntity> exhaustedJobs) {
}
@@ -0,0 +1,81 @@
package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
/**
* 任务存在结果缺失巡检(task-157)。
*
* 只读巡检:终态(SUCCESS/FAILED)任务无对应 file_result 行的清单输出报表;
* 运行中/待处理任务忽略;绝不修改任何状态。可重复执行。
*/
@Slf4j
@Service
@RequiredArgsConstructor
public class TaskResultMissingInspector {
private final FileTaskMapper fileTaskMapper;
private final FileResultMapper fileResultMapper;
public record MissingResultEntry(Long taskId, String moduleType, String status, String updatedAt) {
}
public record MissingResultReport(List<MissingResultEntry> entries) {
public boolean isEmpty() {
return entries == null || entries.isEmpty();
}
}
public MissingResultReport inspectTasksMissingResult(int limit) {
List<FileTaskEntity> tasks = fileTaskMapper.selectList(new LambdaQueryWrapper<FileTaskEntity>()
.in(FileTaskEntity::getStatus, List.of("SUCCESS", "FAILED"))
.orderByAsc(FileTaskEntity::getUpdatedAt)
.last("limit " + Math.max(1, Math.min(limit, 500))));
if (tasks == null || tasks.isEmpty()) {
return new MissingResultReport(List.of());
}
Set<Long> taskIds = tasks.stream()
.map(FileTaskEntity::getId)
.filter(id -> id != null && id > 0)
.collect(Collectors.toSet());
Map<Long, List<FileResultEntity>> resultsByTask = taskIds.isEmpty() ? Map.of()
: fileResultMapper.selectList(new LambdaQueryWrapper<FileResultEntity>()
.in(FileResultEntity::getTaskId, taskIds)).stream()
.collect(Collectors.groupingBy(FileResultEntity::getTaskId));
List<MissingResultEntry> entries = new ArrayList<>();
for (FileTaskEntity task : tasks) {
if (task.getId() == null || task.getId() <= 0) {
continue;
}
List<FileResultEntity> results = resultsByTask.get(task.getId());
if (results == null || results.isEmpty()) {
entries.add(new MissingResultEntry(
task.getId(), task.getModuleType(), task.getStatus(),
task.getUpdatedAt() == null ? null : task.getUpdatedAt().toString()));
}
}
MissingResultReport report = new MissingResultReport(List.copyOf(entries));
if (!report.isEmpty()) {
log.info("task-missing-result report: count={}", report.entries().size());
for (MissingResultEntry entry : report.entries()) {
log.info("task-missing-result: taskId={} moduleType={} status={} updatedAt={}",
entry.taskId(), entry.moduleType(), entry.status(), entry.updatedAt());
}
}
return report;
}
}
@@ -81,6 +81,14 @@ knife4j:
language: zh_cn
aiimage:
# ===== HTTP 客户端统一配置命名空间(task-167)=====
# 默认值与现状一致(RustFS 客户端契约表基线);非法值由 HttpClientProperties 钳制。
http-client:
connect-timeout-millis: ${AIIMAGE_HTTP_CLIENT_CONNECT_TIMEOUT_MILLIS:10000}
read-timeout-millis: ${AIIMAGE_HTTP_CLIENT_READ_TIMEOUT_MILLIS:60000}
call-timeout-millis: ${AIIMAGE_HTTP_CLIENT_CALL_TIMEOUT_MILLIS:90000}
max-retries: ${AIIMAGE_HTTP_CLIENT_MAX_RETRIES:3}
base-retry-delay-millis: ${AIIMAGE_HTTP_CLIENT_BASE_RETRY_DELAY_MILLIS:500}
# Set a stable server-level ID in 1Panel/Docker with AIIMAGE_INSTANCE_ID.
# For IDEA/local startup, set -Daiimage.instance-id=<stable-id> or configure the AIIMAGE_INSTANCE_ID env var.
# Avoid relying on container hostname/container ID, otherwise task owner continuity may break after restart/redeploy.
@@ -0,0 +1,11 @@
-- V104: 店铺数据抓取累计文件 user_id 改可空
-- 背景:V95(97ae8a7)累计文件改为店铺级共享,persistDailyAggregation 已不再
-- 写入 user_id(去 user 维度),但 V95 只改了唯一键,未把 user_id 列改可空。
-- 新店铺首次插入 biz_shop_data_crawl_daily_file 时 MyBatis-Plus 对 null 的
-- user_id 跳过该列 → INSERT 不含 user_id → 生产 MySQL 报
-- "Field 'user_id' doesn't have a default value"(任务 26401 复现)。
-- 已有 daily_file 行的店铺走 update 分支不受影响,因此该 bug 只出现在新店铺。
-- 存量行(25 行)的 user_id 已是旧账号值,保留不动;唯一键 (shop_key,business_date)
-- 不含 user_id,可空不影响去重。
ALTER TABLE biz_shop_data_crawl_daily_file
MODIFY COLUMN `user_id` BIGINT NULL DEFAULT NULL COMMENT 'V95 起不再使用,保留兼容)';
@@ -0,0 +1,2 @@
-- 下线店铺密码检测功能:后台入口、Python 代理与 Java 接口已随代码移除,历史检测记录一并清理
DROP TABLE IF EXISTS `biz_shop_credential_check`;
@@ -0,0 +1,85 @@
package com.nanri.aiimage.config;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-167aiimage.http-client.* 配置命名空间契约(plan 10)。
* 默认值与现状一致;非法值钳制;配置类绑定字段齐全。
*/
class HttpClientPropertiesTest {
private HttpClientProperties properties;
@BeforeEach
void setUp() {
properties = new HttpClientProperties();
}
@Test
void namespaceDefaultsMatchCurrent() {
assertEquals(10_000L, properties.getConnectTimeoutMillis(), "connect 默认 10s(与现状一致)");
assertEquals(60_000L, properties.getReadTimeoutMillis(), "read 默认 60s");
assertEquals(90_000L, properties.getCallTimeoutMillis(), "call 默认 90s");
assertEquals(3, properties.getMaxRetries(), "重试默认 3");
assertEquals(500L, properties.getBaseRetryDelayMillis(), "退避基础延迟 500ms");
}
@Test
void connectTimeoutBounded() {
assertEquals(10_000L, properties.effectiveConnectTimeoutMillis());
properties.setConnectTimeoutMillis(1_000);
assertEquals(1_000L, properties.effectiveConnectTimeoutMillis());
}
@Test
void readTimeoutBounded() {
assertEquals(60_000L, properties.effectiveReadTimeoutMillis());
properties.setReadTimeoutMillis(3_600_000);
assertEquals(3_600_000L, properties.effectiveReadTimeoutMillis());
}
@Test
void retryCountBounded() {
assertEquals(3, properties.effectiveMaxRetries());
properties.setMaxRetries(0);
assertEquals(0, properties.effectiveMaxRetries());
}
@Test
void invalidValuesClamped() {
properties.setConnectTimeoutMillis(-5);
assertEquals(1_000L, properties.effectiveConnectTimeoutMillis(), "负数钳制到下限");
properties.setReadTimeoutMillis(999_999_999L);
assertEquals(3_600_000L, properties.effectiveReadTimeoutMillis(), "超上限钳制");
properties.setMaxRetries(99);
assertEquals(10, properties.effectiveMaxRetries(), "重试钳制到 10");
}
@Test
void callTimeoutBounded() {
properties.setCallTimeoutMillis(7_200_000);
assertEquals(7_200_000L, properties.effectiveCallTimeoutMillis());
properties.setCallTimeoutMillis(0);
assertEquals(1_000L, properties.effectiveCallTimeoutMillis());
}
@Test
void configClassBindingPrefix() throws Exception {
var annotation = HttpClientProperties.class
.getAnnotation(org.springframework.boot.context.properties.ConfigurationProperties.class);
assertEquals("aiimage.http-client", annotation.prefix(), "绑定 aiimage.http-client 前缀");
assertTrue(HttpClientProperties.class.isAnnotationPresent(
org.springframework.stereotype.Component.class), "配置类需注册为组件");
}
@Test
void envOverrideSemantics() {
// env 覆盖走宽松绑定:设置后 effective 反映新值
properties.setConnectTimeoutMillis(30_000);
assertEquals(30_000L, properties.effectiveConnectTimeoutMillis());
}
}
@@ -0,0 +1,86 @@
package com.nanri.aiimage.config;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-168LLM 客户端配置接入契约(plan 10)。
* 从 aiimage.http-client.* 读取超时/重试;默认值与现状一致;覆盖生效;
* 非法值钳制。
*/
class LlmHttpConfigResolverTest {
private HttpClientProperties properties;
private LlmHttpConfigResolver resolver;
@BeforeEach
void setUp() {
properties = new HttpClientProperties();
resolver = new LlmHttpConfigResolver(properties);
}
@Test
void timeoutFromConfigApplied() {
assertEquals(10_000L, resolver.connectTimeoutMillis());
assertEquals(60_000L, resolver.readTimeoutMillis());
assertEquals(90_000L, resolver.callTimeoutMillis());
}
@Test
void defaultValuesKept() {
// 命名空间默认 = 现状基线(connect 10s / read 60s / call 90s / retry 3
assertEquals(10_000L, resolver.connectTimeoutMillis());
assertEquals(60_000L, resolver.readTimeoutMillis());
assertEquals(3, resolver.maxRetries());
assertEquals(500L, resolver.baseRetryDelayMillis());
}
@Test
void overrideTakesEffect() {
properties.setConnectTimeoutMillis(30_000);
properties.setReadTimeoutMillis(120_000);
properties.setMaxRetries(5);
assertEquals(30_000L, resolver.connectTimeoutMillis());
assertEquals(120_000L, resolver.readTimeoutMillis());
assertEquals(5, resolver.maxRetries());
}
@Test
void callTimeoutReflectsTotal() {
properties.setCallTimeoutMillis(180_000);
assertEquals(180_000L, resolver.callTimeoutMillis());
}
@Test
void retryConfigReflected() {
properties.setMaxRetries(0);
assertEquals(0, resolver.maxRetries());
properties.setMaxRetries(99);
assertEquals(10, resolver.maxRetries(), "重试钳制到 10");
}
@Test
void invalidTimeoutsClamped() {
properties.setConnectTimeoutMillis(-1);
assertEquals(1_000L, resolver.connectTimeoutMillis());
properties.setReadTimeoutMillis(999_999_999L);
assertEquals(3_600_000L, resolver.readTimeoutMillis());
}
@Test
void resolverBindingWired() {
assertTrue(resolver.connectTimeoutMillis() > 0);
assertTrue(resolver.maxRetries() >= 0);
}
@Test
void behaviorSameAsCurrentBaseline() {
// 与 SimilarAsinProperties.llm 现状对比:connect 一致(10s)、重试一致(3)
assertEquals(10_000L, resolver.connectTimeoutMillis());
assertEquals(3, resolver.maxRetries());
}
}
@@ -0,0 +1,320 @@
package com.nanri.aiimage.modules.appearancepatent.service;
import cn.hutool.crypto.digest.DigestUtil;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.AppearancePatentProperties;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.modules.appearancepatent.client.AppearancePatentLlmClient;
import com.nanri.aiimage.modules.appearancepatent.model.dto.AppearancePatentSubmitResultRequest;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskProgressLightAssembler;
import com.nanri.aiimage.modules.task.service.TaskProgressSnapshotService;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.test.util.ReflectionTestUtils;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.TransactionDefinition;
import org.springframework.transaction.TransactionStatus;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-128appearancepatent 事务收缩。
* 纯计算(flatten 分组展开/序列化/payload 存储/哈希)抽为无事务的
* prepareSubmittedChunk 在事务开始前调用;双事务路径(persistSubmittedChunk +
* completeSubmittedChunk 两个独立 inNewTransaction)保持不变;落库仍在事务内。
*/
@ExtendWith(MockitoExtension.class)
class AppearancePatentTaskServiceTxBoundaryTest {
private static final Long TASK_ID = 31337L;
private static final String PARSED_POINTER = "rustfs:task-parsed/appearance-patent/31337/payload.json";
private static final String CHUNK_POINTER = "rustfs:task-chunk/appearance-patent/31337/chunk.json";
private static final String STORED_CHUNK_POINTER = "\"" + CHUNK_POINTER + "\"";
@Mock private LocalFileStorageService localFileStorageService;
@Mock private OssStorageService ossStorageService;
@Mock private StorageProperties storageProperties;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Spy private ObjectMapper objectMapper = new ObjectMapper();
@Mock private AppearancePatentLlmClient llmClient;
@Mock private AppearancePatentTaskCacheService taskCacheService;
@Mock private AppearancePatentProperties properties;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TaskProgressSnapshotService taskProgressSnapshotService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private PlatformTransactionManager transactionManager;
@Mock private DistributedJobLockService distributedJobLockService;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private InstanceMetadata instanceMetadata;
@Mock private TaskProgressLightAssembler taskProgressLightAssembler;
@Mock private TransactionStatus transactionStatus;
@InjectMocks private AppearancePatentTaskService service;
private final AtomicBoolean transactionActive = new AtomicBoolean();
private final AtomicReference<TaskChunkEntity> insertedChunk = new AtomicReference<>();
private final AtomicReference<String> storedPayloadJson = new AtomicReference<>();
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskChunkEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskScopeStateEntity.class);
}
@BeforeEach
void setUpTransactionAndLock() {
lenient().when(instanceMetadata.getInstanceId()).thenReturn("instance-a");
lenient().when(transactionManager.getTransaction(any(TransactionDefinition.class)))
.thenAnswer(invocation -> {
transactionActive.set(true);
return transactionStatus;
});
lenient().doAnswer(invocation -> {
transactionActive.set(false);
return null;
}).when(transactionManager).commit(transactionStatus);
lenient().doAnswer(invocation -> {
transactionActive.set(false);
return null;
}).when(transactionManager).rollback(transactionStatus);
lenient().when(taskChunkMapper.selectList(any())).thenReturn(List.of());
lenient().when(fileResultMapper.selectList(any())).thenReturn(List.of());
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(0L);
lenient().when(taskScopeStateMapper.selectCount(any())).thenReturn(0L);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(null);
lenient().when(taskScopeStateMapper.selectOne(any())).thenReturn(null);
lenient().when(transientPayloadStorageService.isSharedWriteEnabled()).thenReturn(true);
lenient().when(transientPayloadStorageService.storeChunkPayload(
eq(AppearancePatentTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString()))
.thenAnswer(invocation -> {
storedPayloadJson.set(invocation.getArgument(4));
return STORED_CHUNK_POINTER;
});
lenient().doAnswer(invocation -> {
TaskChunkEntity chunk = invocation.getArgument(0);
assertTrue(transactionActive.get(), "chunk 落库必须在事务内");
chunk.setId(501L);
insertedChunk.set(chunk);
return 1;
}).when(taskChunkMapper).insert(any(TaskChunkEntity.class));
lenient().doAnswer(invocation -> {
TaskScopeStateEntity scope = invocation.getArgument(0);
scope.setId(601L);
return 1;
}).when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
lenient().when(fileTaskMapper.updateById(any(FileTaskEntity.class))).thenReturn(1);
}
@Test
void payloadHashIsPrecomputedInPrepareBeforeTransaction() throws Exception {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
Object prepared = ReflectionTestUtils.invokeMethod(
service, "prepareSubmittedChunk", TASK_ID, request(false));
assertEquals(DigestUtil.sha256Hex(storedPayloadJson.get()),
ReflectionTestUtils.getField(prepared, "payloadHash"),
"prepare 阶段必须产出预计算的 payload 哈希(事务开始前可用)");
verify(taskChunkMapper, never()).insert(any(TaskChunkEntity.class));
verify(taskScopeStateMapper, never()).insert(any(TaskScopeStateEntity.class));
}
@Test
void computeMovedOutsideTransaction() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
// prepare 存储(纯计算段)→ 事务 1 → 事务 2
var order = inOrder(transientPayloadStorageService, transactionManager, taskChunkMapper);
order.verify(transientPayloadStorageService).storeChunkPayload(
eq(AppearancePatentTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString());
order.verify(transactionManager).getTransaction(any(TransactionDefinition.class));
order.verify(taskChunkMapper).insert(any(TaskChunkEntity.class));
order.verify(transactionManager).getTransaction(any(TransactionDefinition.class));
}
@Test
void dualTransactionPathIsUnchanged() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
verify(transactionManager, times(2)).getTransaction(any(TransactionDefinition.class));
verify(transactionManager, times(2)).commit(transactionStatus);
verify(transactionManager, never()).rollback(transactionStatus);
}
@Test
void persistenceStaysInsideFirstTransaction() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
TaskChunkEntity chunk = insertedChunk.get();
assertEquals(TASK_ID, chunk.getTaskId());
assertEquals(STORED_CHUNK_POINTER, chunk.getPayloadJson());
assertEquals(DigestUtil.sha256Hex(storedPayloadJson.get()), chunk.getPayloadHash());
// persist 与 complete 两个事务各 touch 一次任务行(双事务路径不变)
verify(fileTaskMapper, times(2)).updateById(any(FileTaskEntity.class));
verify(taskScopeStateMapper, times(2)).insert(any(TaskScopeStateEntity.class));
}
@Test
void firstTransactionFailureSkipsSecondAndScheduling() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
when(fileTaskMapper.updateById(any(FileTaskEntity.class))).thenThrow(
new IllegalStateException("persist tx failed"));
assertThrows(IllegalStateException.class, () -> service.submitResult(TASK_ID, request(false)));
verify(transactionManager, times(1)).getTransaction(any(TransactionDefinition.class));
verify(transactionManager, never()).commit(transactionStatus);
verify(transactionManager).rollback(transactionStatus);
verify(taskFileJobService, never()).enqueueAssembleResult(anyLong(), anyString(), anyLong(), anyString());
}
@Test
void duplicateChunkIsIgnoredWithoutStoreOrInsert() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
TaskChunkEntity existing = new TaskChunkEntity();
existing.setId(501L);
existing.setTaskId(TASK_ID);
existing.setScopeHash("existing-scope");
existing.setChunkIndex(0);
when(taskChunkMapper.selectOne(any())).thenReturn(existing);
service.submitResult(TASK_ID, request(false));
verify(transientPayloadStorageService, never()).storeChunkPayload(
anyString(), anyLong(), anyString(), any(), anyString());
verify(taskChunkMapper, never()).insert(any(TaskChunkEntity.class));
verify(taskScopeStateMapper, times(2)).insert(any(TaskScopeStateEntity.class));
}
@Test
void chunkSnapshotIsStable() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
TaskChunkEntity chunk = insertedChunk.get();
assertEquals(TASK_ID, chunk.getTaskId());
assertEquals(AppearancePatentTaskService.MODULE_TYPE, chunk.getModuleType());
assertEquals("appearance-patent-31337", chunk.getScopeKey());
assertEquals(DigestUtil.sha256Hex("appearance-patent-31337"), chunk.getScopeHash());
assertEquals(0, chunk.getChunkIndex());
assertEquals(1, chunk.getChunkTotal());
assertEquals(DigestUtil.sha256Hex(storedPayloadJson.get()), chunk.getPayloadHash());
assertNotNull(chunk.getCreatedAt());
assertNotNull(chunk.getUpdatedAt());
}
@Test
void fullChainRunsPrepareThenTwoTransactionsThenScheduling() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
TaskChunkEntity chunk = new TaskChunkEntity();
chunk.setId(501L);
chunk.setTaskId(TASK_ID);
chunk.setModuleType(AppearancePatentTaskService.MODULE_TYPE);
when(taskChunkMapper.selectList(any())).thenReturn(List.of(chunk));
lenient().when(transientPayloadStorageService.resolvePayload(anyString(), anyString()))
.thenReturn("{\"allItems\":[],\"items\":[],\"sourceFiles\":[]}");
lenient().doAnswer(invocation -> {
FileResultEntity result = invocation.getArgument(0);
result.setId(701L);
return 1;
}).when(fileResultMapper).insert(any(FileResultEntity.class));
when(taskFileJobService.enqueueAssembleResult(
eq(TASK_ID), eq(AppearancePatentTaskService.MODULE_TYPE), eq(701L), anyString()))
.thenReturn(null);
service.submitResult(TASK_ID, request(false));
var order = inOrder(transientPayloadStorageService, transactionManager, taskFileJobService);
order.verify(transientPayloadStorageService).storeChunkPayload(
eq(AppearancePatentTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString());
order.verify(transactionManager).getTransaction(any(TransactionDefinition.class));
order.verify(transactionManager).getTransaction(any(TransactionDefinition.class));
order.verify(taskFileJobService).enqueueAssembleResult(
eq(TASK_ID), eq(AppearancePatentTaskService.MODULE_TYPE), eq(701L), anyString());
}
private AppearancePatentSubmitResultRequest request(boolean done) {
AppearancePatentSubmitResultRequest request = new AppearancePatentSubmitResultRequest();
request.setSubmissionId("appearance-patent-31337");
request.setChunkIndex(0);
request.setChunkTotal(1);
request.setDone(done);
return request;
}
private FileTaskEntity runningTask(String owner) {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType(AppearancePatentTaskService.MODULE_TYPE);
task.setStatus("RUNNING");
task.setUserId(7L);
String resultJson = "{\"parsedPayloadRef\":\"" + PARSED_POINTER + "\"";
if (owner != null) {
resultJson += ",\"ownerInstanceId\":\"" + owner + "\"";
}
task.setResultJson(resultJson + "}");
return task;
}
}
@@ -0,0 +1,263 @@
package com.nanri.aiimage.modules.collectdata.service;
import cn.hutool.crypto.digest.DigestUtil;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.modules.collectdata.mapper.CollectDataCountryPrefMapper;
import com.nanri.aiimage.modules.collectdata.mapper.CollectDataItemMapper;
import com.nanri.aiimage.modules.invalidasin.mapper.InvalidAsinDataMapper;
import com.nanri.aiimage.modules.collectdata.model.dto.CollectDataSubmitResultRequest;
import com.nanri.aiimage.modules.collectdata.model.dto.CollectDataSubmitRowDto;
import com.nanri.aiimage.modules.collectdata.model.vo.CollectDataResultRowVo;
import com.nanri.aiimage.modules.collectdata.model.vo.CollectDataSubmitResultVo;
import com.nanri.aiimage.modules.collectdata.util.CollectDataBatchQuery;
import com.nanri.aiimage.modules.collectdata.util.CollectDataBrandBatchFilter;
import com.nanri.aiimage.modules.collectdata.util.CollectDataResultDetailCodec;
import com.nanri.aiimage.modules.collectdata.util.CollectDataInvalidAsinBatchWriter;
import com.nanri.aiimage.modules.collectdata.util.CollectDataResultItemBatchWriter;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskResultItemMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.ArgumentCaptor;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.support.TransactionTemplate;
import java.lang.reflect.Method;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-131collectdata 事务收缩契约固化。
* 审计结论:/result 提交路径(submitResult → normalize/filter/存储/落库)无
* @Transactional、无事务模板调用——分布式锁内单条原子写与批量 upsert,事务边界
* 已窄;响应 DTO 组装(buildSubmitVo)在写后、锁内纯内存构造。本测试固化契约。
*/
@ExtendWith(MockitoExtension.class)
class CollectDataServiceTxBoundaryTest {
private static final Long TASK_ID = 6161L;
private static final Long USER_ID = 7L;
@Mock private LocalFileStorageService localFileStorageService;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private CollectDataItemMapper collectDataItemMapper;
@Mock private CollectDataCountryPrefMapper collectDataCountryPrefMapper;
@Mock private InvalidAsinDataMapper invalidAsinDataMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskResultItemMapper taskResultItemMapper;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private CollectDataExcelAssemblyService excelAssemblyService;
@Mock private OssStorageService ossStorageService;
@org.mockito.Spy private ObjectMapper objectMapper = new ObjectMapper();
@Mock private TransactionTemplate transactionTemplate;
@Mock private CollectDataBatchQuery collectDataBatchQuery;
@Mock private CollectDataBrandBatchFilter brandBatchFilter;
@Mock private CollectDataInvalidAsinBatchWriter invalidAsinBatchWriter;
@Mock private CollectDataResultItemBatchWriter resultItemBatchWriter;
@Mock private CollectDataResultDetailCodec resultDetailCodec;
@InjectMocks private CollectDataService service;
private final AtomicReference<TaskChunkEntity> insertedChunk = new AtomicReference<>();
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, FileResultEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskChunkEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskScopeStateEntity.class);
}
@BeforeEach
void setUp() throws Exception {
lenient().when(taskDistributedLockService.acquire(eq("COLLECT_DATA"), anyLong(), anyLong()))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class));
lenient().when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
lenient().when(fileResultMapper.selectOne(any())).thenReturn(null);
lenient().doAnswer(invocation -> {
FileResultEntity result = invocation.getArgument(0);
result.setId(5001L);
return 1;
}).when(fileResultMapper).insert(any(FileResultEntity.class));
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(0L);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(null);
lenient().when(taskScopeStateMapper.selectOne(any())).thenReturn(null);
lenient().when(taskScopeStateMapper.insert(any(TaskScopeStateEntity.class))).thenReturn(1);
lenient().when(fileTaskMapper.updateById(any(FileTaskEntity.class))).thenReturn(1);
lenient().when(transientPayloadStorageService.isSharedWriteEnabled()).thenReturn(true);
lenient().when(transientPayloadStorageService.extractPointer(anyString())).thenReturn("rustfs:detail");
lenient().when(transientPayloadStorageService.storeResultPayload(
eq("COLLECT_DATA"), eq(TASK_ID), anyString(), anyString(), anyString()))
.thenReturn("\"rustfs:detail\"");
lenient().when(transientPayloadStorageService.storeChunkPayloadVersioned(
eq("COLLECT_DATA"), eq(TASK_ID), anyString(), anyInt(), anyString()))
.thenReturn("\"rustfs:chunk\"");
lenient().doAnswer(invocation -> {
TaskChunkEntity chunk = invocation.getArgument(0);
chunk.setId(701L);
insertedChunk.set(chunk);
return 1;
}).when(taskChunkMapper).insert(any(TaskChunkEntity.class));
CollectDataResultRowVo row = new CollectDataResultRowVo();
row.setAsin("B0COLLECT1");
lenient().when(collectDataBatchQuery.filter(any())).thenReturn(
new CollectDataBatchQuery.FilterResult(List.of(row), 0, 0));
lenient().when(brandBatchFilter.filter(any())).thenReturn(
new CollectDataBrandBatchFilter.BrandBatchOutcome(List.of(), List.of(), List.of(row)));
lenient().when(resultItemBatchWriter.upsertAccepted(anyLong(), anyLong(), anyString(), anyInt(), any(), anyString()))
.thenReturn(new CollectDataResultItemBatchWriter.UpsertCounts(1, 0, 1));
lenient().when(resultDetailCodec.encodeChunk(any())).thenReturn("{\"chunk\":1}");
}
@Test
void submitResultCarriesNoTransactionAnnotation() throws Exception {
Method submit = CollectDataService.class.getMethod(
"submitResult", Long.class, CollectDataSubmitResultRequest.class);
assertNull(submit.getAnnotation(Transactional.class),
"submitResult 无 @Transactional(锁内单条写,无长事务)");
}
@Test
void submitResultUsesNoTransactionTemplate() {
CollectDataSubmitResultVo vo = service.submitResult(TASK_ID, submitRequest());
assertNotNull(vo);
verify(transactionTemplate, never()).execute(any());
verify(transactionTemplate, never()).executeWithoutResult(any());
}
@Test
void duplicateChunkIsIgnoredWithoutRewriting() {
TaskChunkEntity existing = new TaskChunkEntity();
existing.setId(702L);
existing.setChunkIndex(1);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(existing);
CollectDataSubmitResultVo vo = service.submitResult(TASK_ID, submitRequest());
assertNotNull(vo);
verify(taskChunkMapper, never()).insert(any(TaskChunkEntity.class));
verify(transientPayloadStorageService, never()).storeChunkPayloadVersioned(
anyString(), anyLong(), anyString(), anyInt(), anyString());
}
@Test
void chunkSnapshotPersistedAfterFiltering() {
service.submitResult(TASK_ID, submitRequest());
TaskChunkEntity chunk = insertedChunk.get();
assertEquals(TASK_ID, chunk.getTaskId());
assertEquals("COLLECT_DATA", chunk.getModuleType());
assertEquals(1, chunk.getChunkIndex());
assertEquals(1, chunk.getChunkTotal());
assertEquals("\"rustfs:chunk\"", chunk.getPayloadJson());
assertNotNull(chunk.getPayloadHash(), "payload 哈希必须预计算");
}
@Test
void brandRejectedRowsWrittenToInvalidWriter() {
CollectDataResultRowVo bad = new CollectDataResultRowVo();
bad.setAsin("B0INVALID");
when(brandBatchFilter.filter(any())).thenReturn(
new CollectDataBrandBatchFilter.BrandBatchOutcome(List.of(bad), List.of(), List.of()));
service.submitResult(TASK_ID, submitRequest());
verify(invalidAsinBatchWriter).writeBatch(any());
}
@Test
void rustfsFailureAbortsWithoutPartialChunk() {
when(transientPayloadStorageService.extractPointer(anyString())).thenReturn("local:/tmp/x");
assertThrows(com.nanri.aiimage.common.exception.BusinessException.class,
() -> service.submitResult(TASK_ID, submitRequest()));
verify(taskChunkMapper, never()).insert(any(TaskChunkEntity.class));
}
@Test
void responseVoBuiltFromPostWriteState() {
CollectDataSubmitResultVo vo = service.submitResult(TASK_ID, submitRequest());
assertEquals(TASK_ID, vo.getTaskId());
assertEquals(1, vo.getChunkIndex());
assertEquals(1, vo.getChunkTotal());
assertNotNull(vo.getTaskStatus());
assertEquals(1, vo.getFinalRowCount());
}
@Test
void fullSubmitWritesChunkScopeStatsAndResult() {
service.submitResult(TASK_ID, submitRequest());
verify(taskChunkMapper).insert(any(TaskChunkEntity.class));
verify(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
verify(fileResultMapper).insert(any(FileResultEntity.class));
verify(fileTaskMapper).updateById(any(FileTaskEntity.class));
verify(transactionTemplate, never()).execute(any());
}
private CollectDataSubmitResultRequest submitRequest() {
CollectDataSubmitRowDto row = new CollectDataSubmitRowDto();
row.setAsin("B0COLLECT1");
CollectDataSubmitResultRequest request = new CollectDataSubmitResultRequest();
request.setChunkIndex(1);
request.setChunkTotal(1);
request.setDone(false);
request.setItems(List.of(row));
return request;
}
private FileTaskEntity runningTask() {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType("COLLECT_DATA");
task.setStatus("RUNNING");
task.setUserId(USER_ID);
task.setResultJson("{}");
return task;
}
}
@@ -72,6 +72,45 @@ class CollectDataResultItemBatchWriterTest {
assertEquals(1L, entities.get(0).getTaskId(), "task_id 保留");
}
@Test
void test_module_type_matches_reader_constant() {
// 回归:写入端 module_type 必须与 CollectDataService.MODULE_TYPE(读取端)
// 写同一大小写值,否则生成结果文件时 loadFinalRows 查不出来(曾用小写
// "collectdata" 导致 task-50 之后所有采集任务明细行读不到)。
when(taskResultItemMapper.selectList(any())).thenReturn(List.of());
when(taskResultItemMapper.upsertBatch(anyList())).thenReturn(1);
writer.upsertAccepted(1L, 2L, "task:1", 0,
List.of(row("B000000001", "Nike")), "rustfs:detail/abc");
ArgumentCaptor<List> captor = ArgumentCaptor.forClass(List.class);
verify(taskResultItemMapper).upsertBatch(captor.capture());
TaskResultItemEntity entity = (TaskResultItemEntity) captor.getValue().get(0);
assertEquals("COLLECT_DATA", entity.getModuleType(), "写入 module_type 须为读取端大写常量");
}
@Test
void test_legacy_lowercase_rows_are_found_on_retry() {
// 回归:历史小写 "collectdata" 存量行在旧任务重提时也要被现有行查询
// 命中(按大小写兼容匹配),避免同一唯一键重复插入。
String refJson = codec.encodeRef(0, 0, "rustfs:detail/x");
TaskResultItemEntity legacy = new TaskResultItemEntity();
legacy.setId(100L);
legacy.setModuleType("collectdata");
legacy.setItemKey("asin:B000000001");
legacy.setPayloadJson(refJson);
legacy.setPayloadHash(sha256(refJson));
when(taskResultItemMapper.selectList(any())).thenReturn(List.of(legacy));
CollectDataResultItemBatchWriter.UpsertCounts counts =
writer.upsertAccepted(1L, 2L, "task:1", 0,
List.of(row("B000000001", "Nike")), "rustfs:detail/x");
assertEquals(0, counts.insertedOrUpdated(), "小写存量行命中 → 全部跳过");
assertEquals(1, counts.skipped(), "1 行跳过");
verify(taskResultItemMapper, never()).upsertBatch(anyList());
}
@Test
void test_task_050_task_normal_multiple_items() {
// 批量场景:75 行(8 批次),顺序稳定不丢失,每批数量正确。
@@ -0,0 +1,220 @@
package com.nanri.aiimage.modules.deletebrand.service;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.nanri.aiimage.config.DeleteBrandProgressProperties;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.test.util.ReflectionTestUtils;
import java.time.LocalDateTime;
import java.util.List;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyBoolean;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.isNull;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-122 N+1 批量化修复:failStaleDeleteBrandTasks 的逐任务
* selectById 回读改为两阶段(全部 finalize 后一次 selectBatchIds 批量回读 + Map 装配)。
*/
@ExtendWith(MockitoExtension.class)
class DeleteBrandStaleTaskServiceTest {
@Mock private FileTaskMapper fileTaskMapper;
@Mock private DeleteBrandTaskCacheService deleteBrandTaskCacheService;
@Mock private DeleteBrandTaskStorageService deleteBrandTaskStorageService;
@Mock private DeleteBrandRunService deleteBrandRunService;
@Mock private DeleteBrandProgressProperties deleteBrandProgressProperties;
@Mock private TaskDistributedLockService taskDistributedLockService;
@BeforeAll
static void initializeTableInfo() {
TableInfoHelper.initTableInfo(
new MapperBuilderAssistant(new MybatisConfiguration(), ""),
FileTaskEntity.class);
}
@Test
void staleFinalizedTasksAreBatchRefreshedAndCached() {
FileTaskEntity t1 = runningTask(101L);
FileTaskEntity t2 = runningTask(102L);
FileTaskEntity done1 = finishedTask(101L);
FileTaskEntity done2 = finishedTask(102L);
when(fileTaskMapper.selectList(any())).thenReturn(List.of(t1, t2));
when(fileTaskMapper.selectBatchIds(any())).thenReturn(List.of(done1, done2));
lockAvailable();
DeleteBrandStaleTaskService service = service();
failStaleDeleteBrandTasks(service);
verify(deleteBrandTaskCacheService, times(2)).saveTaskCache(any(FileTaskEntity.class));
verify(fileTaskMapper, never()).update(any(), any());
assertBatchIds(101L, 102L);
}
@Test
void staleSingleRowStillWorks() {
FileTaskEntity t1 = runningTask(201L);
when(fileTaskMapper.selectList(any())).thenReturn(List.of(t1));
when(fileTaskMapper.selectBatchIds(any())).thenReturn(List.of(finishedTask(201L)));
lockAvailable();
DeleteBrandStaleTaskService service = service();
failStaleDeleteBrandTasks(service);
verify(deleteBrandTaskCacheService).saveTaskCache(any(FileTaskEntity.class));
assertBatchIds(201L);
}
@Test
void staleEmptyCandidatesDoesNotRefresh() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of());
DeleteBrandStaleTaskService service = service();
failStaleDeleteBrandTasks(service);
verify(fileTaskMapper, times(1)).selectList(any());
verify(fileTaskMapper, never()).selectBatchIds(any());
verify(deleteBrandRunService, never()).tryFinalizeTask(anyLong(), anyBoolean());
}
@Test
void staleStillRunningTaskIsFailedWithCas() {
FileTaskEntity t1 = runningTask(301L);
when(fileTaskMapper.selectList(any())).thenReturn(List.of(t1));
when(fileTaskMapper.selectBatchIds(any())).thenReturn(List.of(runningTask(301L)));
when(fileTaskMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
lockAvailable();
DeleteBrandStaleTaskService service = service();
failStaleDeleteBrandTasks(service);
verify(deleteBrandTaskCacheService, never()).saveTaskCache(any(FileTaskEntity.class));
verify(deleteBrandTaskCacheService).delete(301L);
@SuppressWarnings({"rawtypes", "unchecked"})
ArgumentCaptor<LambdaUpdateWrapper> update = ArgumentCaptor.forClass(LambdaUpdateWrapper.class);
verify(fileTaskMapper).update(isNull(), update.capture());
assertTrue(update.getValue().getParamNameValuePairs().containsValue("FAILED"),
update.getValue().getSqlSet());
}
@Test
void staleFinalizeThrowsTaskIsFailedWithoutRefreshQuery() {
FileTaskEntity t1 = runningTask(401L);
when(fileTaskMapper.selectList(any())).thenReturn(List.of(t1));
when(fileTaskMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
org.mockito.Mockito.doThrow(new IllegalStateException("boom"))
.when(deleteBrandRunService).tryFinalizeTask(401L, true);
lockAvailable();
DeleteBrandStaleTaskService service = service();
failStaleDeleteBrandTasks(service);
verify(fileTaskMapper, times(1)).selectList(any());
verify(fileTaskMapper, never()).selectBatchIds(any());
verify(deleteBrandTaskCacheService).delete(401L);
}
@Test
void staleQueryCountReducedNoSelectById() {
FileTaskEntity t1 = runningTask(501L);
when(fileTaskMapper.selectList(any())).thenReturn(List.of(t1));
when(fileTaskMapper.selectBatchIds(any())).thenReturn(List.of(finishedTask(501L)));
lockAvailable();
DeleteBrandStaleTaskService service = service();
failStaleDeleteBrandTasks(service);
verify(fileTaskMapper, never()).selectById(any());
verify(fileTaskMapper, times(1)).selectList(any());
verify(fileTaskMapper, times(1)).selectBatchIds(any());
}
@Test
void staleTaskWithBusyLockIsSkipped() {
FileTaskEntity t1 = runningTask(601L);
when(fileTaskMapper.selectList(any())).thenReturn(List.of(t1));
when(taskDistributedLockService.acquire(any(), any(), anyLong())).thenReturn(null);
DeleteBrandStaleTaskService service = service();
failStaleDeleteBrandTasks(service);
verify(deleteBrandRunService, never()).tryFinalizeTask(anyLong(), anyBoolean());
verify(fileTaskMapper, times(1)).selectList(any());
verify(fileTaskMapper, never()).selectBatchIds(any());
}
@Test
void staleTaskWithRecentHeartbeatIsSkipped() {
FileTaskEntity t1 = runningTask(701L);
when(fileTaskMapper.selectList(any())).thenReturn(List.of(t1));
when(deleteBrandTaskCacheService.getProgress(701L))
.thenReturn(Map.of("last_heartbeat_at", System.currentTimeMillis()));
DeleteBrandStaleTaskService service = service();
failStaleDeleteBrandTasks(service);
verify(taskDistributedLockService, never()).acquire(any(), any(), anyLong());
verify(deleteBrandRunService, never()).tryFinalizeTask(anyLong(), anyBoolean());
}
private DeleteBrandStaleTaskService service() {
return new DeleteBrandStaleTaskService(
fileTaskMapper, deleteBrandTaskCacheService, deleteBrandTaskStorageService, deleteBrandRunService,
null, null, null, null, null, null, null, null, null, null, null, null, null, null, null,
deleteBrandProgressProperties, null, taskDistributedLockService, null);
}
private void lockAvailable() {
when(deleteBrandProgressProperties.getHeartbeatTimeoutMinutes()).thenReturn(15L);
when(taskDistributedLockService.acquire(any(), any(), anyLong()))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class));
}
private static void failStaleDeleteBrandTasks(DeleteBrandStaleTaskService service) {
ReflectionTestUtils.invokeMethod(service, "failStaleDeleteBrandTasks");
}
@SuppressWarnings({"rawtypes", "unchecked"})
private void assertBatchIds(Long... expected) {
ArgumentCaptor<java.util.Collection> captor = ArgumentCaptor.forClass(java.util.Collection.class);
verify(fileTaskMapper).selectBatchIds(captor.capture());
assertEquals(List.of(expected), List.copyOf(captor.getValue()));
}
private static FileTaskEntity runningTask(Long id) {
FileTaskEntity task = new FileTaskEntity();
task.setId(id);
task.setModuleType("DELETE_BRAND");
task.setStatus("RUNNING");
task.setUpdatedAt(LocalDateTime.now().minusHours(1));
task.setCreatedAt(LocalDateTime.now().minusDays(1));
return task;
}
private static FileTaskEntity finishedTask(Long id) {
FileTaskEntity task = runningTask(id);
task.setStatus("SUCCESS");
return task;
}
}
@@ -0,0 +1,122 @@
package com.nanri.aiimage.modules.file.service;
import com.nanri.aiimage.modules.file.service.BatchFileCleaner.BatchCleanResult;
import org.junit.jupiter.api.Test;
import java.io.File;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-164:清理失败不阻断契约(plan 09)。
* 单文件失败不影响其他;错误可见(failedPaths);不上抛;下轮可重试;
* 引用判定失败保守跳过不计数为失败。
*/
class BatchFileCleanerTest {
private final BatchFileCleaner cleaner = new BatchFileCleaner();
private final File a = new File("target/tmp/a.tmp");
private final File b = new File("target/tmp/b.tmp");
private final File c = new File("target/tmp/c.tmp");
@Test
void oneFailureOthersSucceed() {
BatchCleanResult result = cleaner.cleanBatch(
List.of(a, b, c),
file -> !file.getName().equals("b.tmp"),
name -> false);
assertEquals(2, result.cleanedCount(), "b 失败其余成功");
assertEquals(1, result.failedCount());
assertTrue(result.hasFailures());
}
@Test
void failureIsVisible() {
BatchCleanResult result = cleaner.cleanBatch(
List.of(a, b),
file -> !file.getName().equals("b.tmp"),
name -> false);
assertEquals(1, result.failedPaths().size());
assertTrue(result.failedPaths().getFirst().contains("b.tmp"), "失败路径可见");
}
@Test
void noThrowUpOnFailure() {
BatchCleanResult result = cleaner.cleanBatch(
List.of(a, b),
file -> {
if (file.getName().equals("b.tmp")) {
throw new RuntimeException("delete blew up");
}
return true;
},
name -> false);
assertEquals(1, result.cleanedCount());
assertEquals(1, result.failedCount(), "抛异常按失败计数不上抛");
}
@Test
void retryNextRoundSucceeds() {
AtomicInteger attempts = new AtomicInteger();
BatchCleanResult first = cleaner.cleanBatch(
List.of(b),
file -> attempts.incrementAndGet() == 1 ? false : true,
name -> false);
BatchCleanResult second = cleaner.cleanBatch(
List.of(b),
file -> attempts.incrementAndGet() > 1,
name -> false);
assertEquals(0, first.cleanedCount());
assertEquals(1, second.cleanedCount(), "下轮重试成功");
}
@Test
void partialResultReported() {
BatchCleanResult result = cleaner.cleanBatch(
List.of(a, c),
file -> true,
name -> false);
assertEquals(2, result.cleanedCount());
assertEquals(0, result.failedCount());
assertFalse(result.hasFailures());
}
@Test
void failureDoesNotBlockRemaining() {
BatchCleanResult result = cleaner.cleanBatch(
List.of(a, b, c),
file -> !file.getName().equals("b.tmp"),
name -> false);
assertEquals(2, result.cleanedCount(), "b 失败后 c 仍被清理(无级联)");
}
@Test
void referencedFilesSkippedNotCountedAsFailure() {
BatchCleanResult result = cleaner.cleanBatch(
List.of(a, b),
file -> true,
name -> name.equals("a.tmp"));
assertEquals(1, result.cleanedCount(), "被引用文件跳过,不算失败");
assertEquals(0, result.failedCount());
}
@Test
void nullInputsAreSafe() {
BatchCleanResult empty = cleaner.cleanBatch(null, file -> true, name -> false);
assertEquals(0, empty.cleanedCount());
assertEquals(0, empty.failedCount());
assertTrue(empty.failedPaths().isEmpty());
}
}
@@ -0,0 +1,119 @@
package com.nanri.aiimage.modules.file.service;
import com.nanri.aiimage.config.StorageProperties;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.test.util.ReflectionTestUtils;
import java.io.File;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-152:清理失败吞异常记指标契约(plan 09)。
* 清理失败:记录日志+指标(清理失败计数)、不抛异常、不自动重试成循环;
* 单个条目失败不影响其他条目(隔离)。
*/
@ExtendWith(MockitoExtension.class)
class CleanupFailureMetricTest {
@Mock private StorageProperties storageProperties;
@Mock private MeterRegistry meterRegistry;
private LocalTempCleanupService service;
@BeforeEach
void setUp() {
service = new LocalTempCleanupService(storageProperties);
ReflectionTestUtils.setField(service, "meterRegistry", meterRegistry);
}
@Test
void failureIsLoggedWithoutThrowing() {
service.handleCleanupFailure(new File("target/tmp/a.tmp"), new RuntimeException("boom"));
service.handleCleanupFailure(new File("target/tmp/b.tmp"), null);
}
@Test
void failureRecordsMetric() {
Counter counter = mock(Counter.class);
when(meterRegistry.counter(anyString(), anyString(), anyString())).thenReturn(counter);
service.handleCleanupFailure(new File("target/tmp/a.tmp"), new RuntimeException("boom"));
verify(meterRegistry).counter(eq("aiimage.temp-cleanup.failed"), eq("path"), eq("a.tmp"));
verify(counter).increment();
}
@Test
void failureDoesNotThrowToCaller() {
Counter counter = mock(Counter.class);
when(meterRegistry.counter(anyString(), anyString(), anyString())).thenThrow(
new RuntimeException("metrics down"));
service.handleCleanupFailure(new File("target/tmp/a.tmp"), new RuntimeException("boom"));
}
@Test
void failureMetricNotRepeatedInLoop() {
Counter counter = mock(Counter.class);
when(meterRegistry.counter(anyString(), anyString(), anyString())).thenReturn(counter);
service.handleCleanupFailure(new File("target/tmp/a.tmp"), new RuntimeException("boom"));
verify(counter, times(1)).increment();
}
@Test
void nextRoundCanRetryIndependently() {
Counter counter = mock(Counter.class);
when(meterRegistry.counter(anyString(), anyString(), anyString())).thenReturn(counter);
service.handleCleanupFailure(new File("target/tmp/a.tmp"), new RuntimeException("boom"));
service.handleCleanupFailure(new File("target/tmp/a.tmp"), new RuntimeException("boom again"));
verify(counter, times(2)).increment();
}
@Test
void partialFailureIsIsolatedPerPath() {
Counter counterA = mock(Counter.class);
Counter counterB = mock(Counter.class);
when(meterRegistry.counter(anyString(), eq("path"), eq("a.tmp"))).thenReturn(counterA);
when(meterRegistry.counter(anyString(), eq("path"), eq("b.tmp"))).thenReturn(counterB);
service.handleCleanupFailure(new File("target/tmp/a.tmp"), new RuntimeException("x"));
service.handleCleanupFailure(new File("target/tmp/b.tmp"), new RuntimeException("y"));
service.handleCleanupFailure(new File("target/tmp/b.tmp"), new RuntimeException("y2"));
verify(counterA, times(1)).increment();
verify(counterB, times(2)).increment();
}
@Test
void metricTaggedByFileName() {
Counter counter = mock(Counter.class);
when(meterRegistry.counter(anyString(), anyString(), anyString())).thenReturn(counter);
service.handleCleanupFailure(new File("target/tmp/result/x.xlsx"), new RuntimeException("boom"));
verify(meterRegistry).counter(eq("aiimage.temp-cleanup.failed"), eq("path"), eq("x.xlsx"));
}
@Test
void nullChildIsSafe() {
service.handleCleanupFailure(null, new RuntimeException("boom"));
service.handleCleanupFailure(null, null);
}
}
@@ -0,0 +1,121 @@
package com.nanri.aiimage.modules.file.service;
import com.nanri.aiimage.config.StorageProperties;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.springframework.mock.web.MockMultipartFile;
import java.io.File;
import java.nio.file.Path;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-153fileKey 兜底查找契约(plan 09)。
* 上传记录登记内存索引;进程重启后索引为空,按 fileKey 目录枚举兜底重建;
* 找不到时明确返回 null(调用方明确失败),不静默。
*/
class LocalFileKeyLookupTest {
@TempDir
Path tempDir;
private LocalFileStorageService service;
@BeforeEach
void setUp() {
StorageProperties properties = new StorageProperties();
properties.setLocalTempDir(tempDir.toString());
service = new LocalFileStorageService(properties);
}
@Test
void fileKeyFoundAfterSave() throws Exception {
var vo = service.saveTempFile(new MockMultipartFile("f", "a.xlsx", "application/octet-stream", new byte[]{1, 2}), "upload");
File found = service.findLocalSourceFile(vo.getFileKey());
assertNotNull(found, "保存后按 fileKey 必须找到");
assertTrue(found.isFile());
assertEquals(vo.getFileKey(), found.getName().substring(0, vo.getFileKey().length()));
}
@Test
void fileKeyFoundAfterRestart() throws Exception {
var vo = service.saveTempFile(new MockMultipartFile("f", "b.xlsx", "application/octet-stream", new byte[]{3}), "upload");
// 模拟重启:新实例(内存索引为空)→ 目录枚举兜底
StorageProperties properties = new StorageProperties();
properties.setLocalTempDir(tempDir.toString());
LocalFileStorageService restarted = new LocalFileStorageService(properties);
File found = restarted.findLocalSourceFile(vo.getFileKey());
assertNotNull(found, "重启后按 fileKey 目录枚举兜底必须找到");
}
@Test
void missingFileKeyFailsExplicitly() {
assertNull(service.findLocalSourceFile("0123456789abcdef0123456789abcdef"),
"不存在的 fileKey 明确返回 null");
}
@Test
void nullAndBlankFileKeyAreSafe() {
assertNull(service.findLocalSourceFile(null));
assertNull(service.findLocalSourceFile(" "));
assertNull(service.findLocalSourceFile(""));
}
@Test
void deletedFileKeyNotFindable() throws Exception {
var vo = service.saveTempFile(new MockMultipartFile("f", "c.xlsx", "application/octet-stream", new byte[]{4}), "upload");
File found = service.findLocalSourceFile(vo.getFileKey());
assertNotNull(found);
assertTrue(found.delete(), "清理测试文件");
assertNull(service.findLocalSourceFile(vo.getFileKey()), "文件清理后不可再找到");
}
@Test
void indexRecordPersistsForLookup() throws Exception {
var vo = service.saveTempFile(new MockMultipartFile("f", "d.xlsx", "application/octet-stream", new byte[]{5}), "upload");
File found = service.findLocalSourceFile(vo.getFileKey());
assertNotNull(found);
assertEquals(vo.getLocalPath(), found.getAbsolutePath(), "索引命中返回登记路径");
}
@Test
@SuppressWarnings("unchecked")
void indexRebuiltByDirectoryEnumeration() throws Exception {
var vo = service.saveTempFile(new MockMultipartFile("f", "e.xlsx", "application/octet-stream", new byte[]{6}), "upload");
// 清空内存索引(模拟索引丢失),目录枚举仍可重建
((java.util.Map<String, String>) org.springframework.test.util.ReflectionTestUtils
.getField(service, "sourceFileIndex")).clear();
File found = service.findLocalSourceFile(vo.getFileKey());
assertNotNull(found, "索引清空后目录枚举兜底重建");
}
@Test
void semanticsFrozen() throws Exception {
var vo = service.saveTempFile(new MockMultipartFile("f", "f.xlsx", "application/octet-stream", new byte[]{7}), "upload");
File found = service.findLocalSourceFile(vo.getFileKey());
File restartFound = new LocalFileStorageService(properties()).findLocalSourceFile(vo.getFileKey());
assertEquals(found.getAbsolutePath(), restartFound.getAbsolutePath(),
"同 fileKey 在重启前后解析到同一文件(语义不变)");
}
private StorageProperties properties() {
StorageProperties properties = new StorageProperties();
properties.setLocalTempDir(tempDir.toString());
return properties;
}
}
@@ -0,0 +1,119 @@
package com.nanri.aiimage.modules.file.service;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-160:超保留期仍被引用巡检契约(plan 09)。
* 临时文件超保留期但仍有引用 → 检出(供人工处理);未超期不报;
* 超期未引用不报(归孤立文件报表);只读;可重复。
*/
class OverRetentionReferencedTest {
@TempDir
Path tempDir;
private final TempOrphanInspector inspector = new TempOrphanInspector();
private Path writeFile(String name, String content) throws Exception {
Path path = tempDir.resolve(name);
Files.writeString(path, content, StandardCharsets.UTF_8);
return path;
}
@Test
void overRetentionReferencedDetected() throws Exception {
writeFile("kept.tmp", "x");
var report = inspector.inspectOverRetentionReferenced(
tempDir.toFile(), Instant.now().plus(1, ChronoUnit.HOURS), name -> name.equals("kept.tmp"));
assertEquals(1, report.entries().size(), "超期且被引用必须检出");
}
@Test
void inRetentionNotReported() throws Exception {
writeFile("fresh.tmp", "x");
var report = inspector.inspectOverRetentionReferenced(
tempDir.toFile(), Instant.now().minus(1, ChronoUnit.HOURS), name -> true);
assertTrue(report.isEmpty(), "未超保留期不报");
}
@Test
void overRetentionUnreferencedNotReportedHere() throws Exception {
writeFile("orphan.tmp", "x");
var report = inspector.inspectOverRetentionReferenced(
tempDir.toFile(), Instant.now().plus(1, ChronoUnit.HOURS), name -> false);
assertTrue(report.isEmpty(), "超期未引用归孤立文件报表,不在此处重复报");
}
@Test
void boundaryRetentionIsGraceful() throws Exception {
writeFile("edge.tmp", "x");
Instant now = Instant.now();
var report = inspector.inspectOverRetentionReferenced(tempDir.toFile(), now, name -> true);
// 边界(mtime 与边界接近):不抛错、报表确定
assertTrue(report.isEmpty() || report.entries().size() == 1);
}
@Test
void emptyReportWhenNothingReferenced() throws Exception {
writeFile("a.tmp", "x");
var report = inspector.inspectOverRetentionReferenced(
tempDir.toFile(), Instant.now().plus(1, ChronoUnit.HOURS), name -> false);
assertTrue(report.isEmpty());
}
@Test
void readOnlyDoesNotDelete() throws Exception {
Path file = writeFile("referenced.tmp", "keep");
inspector.inspectOverRetentionReferenced(
tempDir.toFile(), Instant.now().plus(1, ChronoUnit.HOURS), name -> true);
assertTrue(Files.exists(file), "只读巡检不得删除");
}
@Test
void reportIncludesDetails() throws Exception {
writeFile("detail.tmp", "0123456789");
var report = inspector.inspectOverRetentionReferenced(
tempDir.toFile(), Instant.now().plus(1, ChronoUnit.HOURS), name -> true);
TempOrphanInspector.OrphanFileEntry entry = report.entries().getFirst();
assertTrue(entry.path().contains("detail.tmp"));
assertEquals(10L, entry.sizeBytes());
assertTrue(entry.lastModifiedAt() != null && !entry.lastModifiedAt().isBlank());
}
@Test
void rerunIsSafeAndStable() throws Exception {
writeFile("rerun.tmp", "x");
var first = inspector.inspectOverRetentionReferenced(
tempDir.toFile(), Instant.now().plus(1, ChronoUnit.HOURS), name -> true);
var second = inspector.inspectOverRetentionReferenced(
tempDir.toFile(), Instant.now().plus(1, ChronoUnit.HOURS), name -> true);
assertEquals(first.entries().size(), second.entries().size());
assertEquals(first.entries().getFirst().path(), second.entries().getFirst().path());
}
}
@@ -0,0 +1,97 @@
package com.nanri.aiimage.modules.file.service;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-151:路径安全校验契约(plan 09)。
* 删除前校验:child 规范化后 startsWith(临时根目录) 且不等于根;
* 穿越(../、绝对路径逃逸、符号链接逃逸)拒绝;非法输入安全返回 false。
*/
class PathSafetyGuardTest {
@TempDir
Path tempDir;
@Test
void insideRootIsAllowed() throws Exception {
Path child = tempDir.resolve("a.tmp");
Files.writeString(child, "x");
assertTrue(PathSafetyGuard.isInside(tempDir.toFile(), child.toFile()));
}
@Test
void nestedDirectoryInsideRootIsAllowed() throws Exception {
Path nested = tempDir.resolve("result/2026/09").resolve("f.tmp");
Files.createDirectories(nested.getParent());
Files.writeString(nested, "x");
assertTrue(PathSafetyGuard.isInside(tempDir.toFile(), nested.toFile()));
}
@Test
void parentTraversalIsRejected() {
assertTrue(PathSafetyGuard.isTraversal("../escape"));
assertTrue(PathSafetyGuard.isTraversal("a/../../b"));
assertFalse(PathSafetyGuard.isTraversal("normal-file.tmp"));
}
@Test
void absolutePathEscapeIsRejected() {
assertTrue(PathSafetyGuard.isTraversal("/etc/passwd"));
assertTrue(PathSafetyGuard.isTraversal("C:\\windows\\x"));
assertTrue(PathSafetyGuard.isTraversal("D:/escape"));
}
@Test
void outsideRootIsRejected() throws Exception {
Path outside = tempDir.getParent().resolve("outside-" + System.nanoTime() + ".tmp");
Files.writeString(outside, "x");
try {
assertFalse(PathSafetyGuard.isInside(tempDir.toFile(), outside.toFile()),
"根外路径必须拒绝");
} finally {
Files.deleteIfExists(outside);
}
}
@Test
void symlinkEscapeIsRejected() throws Exception {
Path outside = tempDir.getParent().resolve("symlink-target-" + System.nanoTime() + ".tmp");
Files.writeString(outside, "secret");
Path link = tempDir.resolve("link.tmp");
try {
Files.createSymbolicLink(link, outside);
assertFalse(PathSafetyGuard.isInside(tempDir.toFile(), link.toFile()),
"符号链接指向根外必须拒绝(toRealPath 解析)");
} catch (UnsupportedOperationException | java.io.IOException ex) {
// 平台不支持符号链接时跳过
} finally {
Files.deleteIfExists(link);
Files.deleteIfExists(outside);
}
}
@Test
void rootItselfIsNotDeletable() {
assertFalse(PathSafetyGuard.isInside(tempDir.toFile(), tempDir.toFile()),
"根目录本身不算内部(禁止删根)");
}
@Test
void nullAndBlankAreSafe() {
assertFalse(PathSafetyGuard.isInside(null, null));
assertFalse(PathSafetyGuard.isInside(tempDir.toFile(), null));
assertFalse(PathSafetyGuard.isInside(null, tempDir.toFile()));
assertTrue(PathSafetyGuard.isTraversal(null));
assertTrue(PathSafetyGuard.isTraversal(" "));
}
}
@@ -0,0 +1,71 @@
package com.nanri.aiimage.modules.file.service;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-162:路径穿越输入矩阵(plan 09)。
* ../、..\\、绝对路径、URL 编码 %2e%2e、Unicode 全角点、混合分隔符、空字节
* 全部拒绝;普通文件名放行。
*/
class PathTraversalMatrixTest {
@Test
void dotdotSlashRejected() {
assertTrue(PathSafetyGuard.isTraversal("../escape"));
assertTrue(PathSafetyGuard.isTraversal("a/../../b"));
assertTrue(PathSafetyGuard.isTraversal("..../x"));
}
@Test
void dotdotBackslashRejected() {
assertTrue(PathSafetyGuard.isTraversal("..\\escape"));
assertTrue(PathSafetyGuard.isTraversal("a\\..\\b"));
}
@Test
void urlEncodedDotdotRejected() {
assertTrue(PathSafetyGuard.isTraversal("%2e%2e/x"));
assertTrue(PathSafetyGuard.isTraversal("%2E%2E/x"));
assertTrue(PathSafetyGuard.isTraversal("a%2e%2eb"));
}
@Test
void fullwidthDotRejected() {
assertTrue(PathSafetyGuard.isTraversal("\uFF0E\uFF0E/escape"), "全角点 .. 拒绝");
assertTrue(PathSafetyGuard.isTraversal("a\uFF0E\uFF0Eb"));
}
@Test
void absoluteWindowsPathRejected() {
assertTrue(PathSafetyGuard.isTraversal("C:\\windows\\system32"));
assertTrue(PathSafetyGuard.isTraversal("D:/escape/file"));
}
@Test
void absoluteUnixPathRejected() {
assertTrue(PathSafetyGuard.isTraversal("/etc/passwd"));
assertTrue(PathSafetyGuard.isTraversal("/tmp/escape"));
}
@Test
void mixedSeparatorsRejected() {
assertTrue(PathSafetyGuard.isTraversal("..\\/escape"));
assertTrue(PathSafetyGuard.isTraversal("a/..\\b/c"));
assertTrue(PathSafetyGuard.isTraversal("C:..\\x"));
}
@Test
void nullByteRejected() {
assertTrue(PathSafetyGuard.isTraversal("evil\0.tmp"));
}
@Test
void normalNamesAllowed() {
assertFalse(PathSafetyGuard.isTraversal("0123456789abcdef0123456789abcdef.xlsx"));
assertFalse(PathSafetyGuard.isTraversal("result-2026-09-02.xlsx"));
assertFalse(PathSafetyGuard.isTraversal("chunk-1.json"));
}
}
@@ -0,0 +1,92 @@
package com.nanri.aiimage.modules.file.service;
import com.nanri.aiimage.modules.file.service.ReferenceAwareCleaner.CleanDecision;
import com.nanri.aiimage.modules.file.service.ReferenceAwareCleaner.FileDecision;
import org.junit.jupiter.api.Test;
import java.io.File;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-163:引用中文件不清契约(plan 09)。
* 被 job/result 等引用的文件 → 跳过清理;仅未引用文件清理;引用判定失败保守
* 跳过;批量混合;幂等。
*/
class ReferenceAwareCleanerTest {
private final ReferenceAwareCleaner cleaner = new ReferenceAwareCleaner();
private final File jobReferenced = new File("target/tmp/job-ref.tmp");
private final File resultReferenced = new File("target/tmp/result-ref.tmp");
private final File unreferenced = new File("target/tmp/free.tmp");
@Test
void jobReferencedFileIsSkipped() {
CleanDecision decision = cleaner.decide(jobReferenced, name -> name.equals("job-ref.tmp"));
assertEquals(CleanDecision.SKIP_REFERENCED, decision, "被 job 引用 → 跳过清理");
}
@Test
void resultReferencedFileIsSkipped() {
CleanDecision decision = cleaner.decide(resultReferenced, name -> name.equals("result-ref.tmp"));
assertEquals(CleanDecision.SKIP_REFERENCED, decision, "被 result 引用 → 跳过清理");
}
@Test
void unreferencedFilePasses() {
CleanDecision decision = cleaner.decide(unreferenced, name -> false);
assertEquals(CleanDecision.CLEAN, decision, "未引用文件可清理");
}
@Test
void multiReferenceAnyHitSkips() {
// 多引用来源:chunk/scope/job/result 任一命中即跳过
CleanDecision decision = cleaner.decide(jobReferenced,
name -> name.equals("other") || name.equals("job-ref.tmp") || name.equals("x"));
assertEquals(CleanDecision.SKIP_REFERENCED, decision);
}
@Test
void referenceReleasedAfterHistoryDeleteAllowsClean() {
// 删除历史后引用解除:谓词从 true 变 false → 可清理
CleanDecision before = cleaner.decide(jobReferenced, name -> true);
CleanDecision after = cleaner.decide(jobReferenced, name -> false);
assertEquals(CleanDecision.SKIP_REFERENCED, before);
assertEquals(CleanDecision.CLEAN, after, "引用解除后可清理");
}
@Test
void batchMixedOnlyUnreferencedCleans() {
List<FileDecision> decisions = cleaner.decideBatch(
List.of(jobReferenced, unreferenced, resultReferenced),
name -> name.equals("job-ref.tmp") || name.equals("result-ref.tmp"));
assertEquals(3, decisions.size());
assertEquals(CleanDecision.SKIP_REFERENCED, decisions.get(0).decision());
assertEquals(CleanDecision.CLEAN, decisions.get(1).decision(), "仅未引用文件清理");
assertEquals(CleanDecision.SKIP_REFERENCED, decisions.get(2).decision());
}
@Test
void nullInputsAreSafe() {
assertEquals(CleanDecision.CLEAN, cleaner.decide(null, name -> true));
assertEquals(CleanDecision.CLEAN, cleaner.decide(unreferenced, null));
assertTrue(cleaner.decideBatch(null, name -> false).isEmpty());
}
@Test
void checkIsIdempotent() {
CleanDecision first = cleaner.decide(jobReferenced, name -> name.equals("job-ref.tmp"));
CleanDecision second = cleaner.decide(jobReferenced, name -> name.equals("job-ref.tmp"));
assertEquals(first, second, "同输入同决策");
}
}
@@ -0,0 +1,110 @@
package com.nanri.aiimage.modules.file.service;
import com.nanri.aiimage.config.StorageProperties;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.springframework.test.util.ReflectionTestUtils;
import java.io.File;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-154:临时目录磁盘容量告警契约(plan 09)。
* 容量超阈值记录告警(当前容量/阈值);未超不告警;告警频率受限(防刷屏);
* 测量失败容忍;不抛异常不阻塞业务。
*/
class TempDirCapacityMonitorTest {
@TempDir
Path tempDir;
private StorageProperties properties;
private TempDirCapacityMonitor monitor;
@BeforeEach
void setUp() {
properties = new StorageProperties();
properties.setLocalTempDir(tempDir.toString());
monitor = new TempDirCapacityMonitor(properties);
}
@Test
void usageMeasuredCorrectly() throws Exception {
Files.writeString(tempDir.resolve("a.tmp"), "0123456789", StandardCharsets.UTF_8);
Path sub = tempDir.resolve("sub");
Files.createDirectories(sub);
Files.writeString(sub.resolve("b.tmp"), "12345", StandardCharsets.UTF_8);
long usage = monitor.measureUsageBytes(tempDir.toFile());
assertEquals(15L, usage, "容量按字节递归求和");
}
@Test
void emptyDirUsageIsZero() {
assertEquals(0L, monitor.measureUsageBytes(tempDir.toFile()), "空目录容量为 0");
}
@Test
void warningTriggeredOverThreshold() throws Exception {
Files.writeString(tempDir.resolve("big.tmp"), "x".repeat(100), StandardCharsets.UTF_8);
properties.setCapacityWarnBytes(10);
ReflectionTestUtils.setField(monitor, "lastWarnAtMillis", 0L);
// 超阈值 → 告警路径(频率窗口内可告警)
monitor.warnWithRateLimit(tempDir.toFile(), 100L, 10L);
long lastWarn = (long) ReflectionTestUtils.getField(monitor, "lastWarnAtMillis");
assertTrue(lastWarn > 0L, "告警后记录时间戳(频率限制用)");
}
@Test
void underThresholdDoesNotWarn() throws Exception {
Files.writeString(tempDir.resolve("small.tmp"), "x", StandardCharsets.UTF_8);
properties.setCapacityWarnBytes(10_000);
monitor.checkTempDirCapacity();
long lastWarn = (long) ReflectionTestUtils.getField(monitor, "lastWarnAtMillis");
assertEquals(0L, lastWarn, "未超阈值不告警");
}
@Test
void warningRateLimited() throws Exception {
ReflectionTestUtils.setField(monitor, "lastWarnAtMillis", System.currentTimeMillis());
long before = (long) ReflectionTestUtils.getField(monitor, "lastWarnAtMillis");
monitor.warnWithRateLimit(tempDir.toFile(), 100L, 10L);
assertEquals(before, ReflectionTestUtils.getField(monitor, "lastWarnAtMillis"),
"频率窗口内重复告警被抑制");
}
@Test
void measureFailureIsGraceful() {
File missing = tempDir.resolve("nope").toFile();
assertEquals(-1L, monitor.measureUsageBytes(missing), "目录缺失测量返回 -1");
assertEquals(-1L, monitor.measureUsageBytes(null));
}
@Test
void checkDoesNotThrowOnMissingDir() {
properties.setLocalTempDir(tempDir.resolve("absent").toString());
monitor.checkTempDirCapacity();
}
@Test
void warningDoesNotBlockBusiness() {
// 告警路径不抛异常(业务不阻塞)
ReflectionTestUtils.setField(monitor, "lastWarnAtMillis", 0L);
monitor.warnWithRateLimit(tempDir.toFile(), 999L, 1L);
monitor.checkTempDirCapacity();
}
}
@@ -0,0 +1,109 @@
package com.nanri.aiimage.modules.file.service;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.io.File;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-149:临时文件创建/访问时间记录契约(plan 09)。
* 创建时间/大小显式读取;最后访问时间以 mtime 回退(与清理语义一致);
* 缺记录/IO 错误容忍,不改变文件语义。
*/
class TempFileMetadataTest {
@TempDir
Path tempDir;
private File writeTempFile(String name, String content) throws Exception {
Path path = tempDir.resolve(name);
Files.writeString(path, content, StandardCharsets.UTF_8);
return path.toFile();
}
@Test
void creationTimeRecorded() throws Exception {
File file = writeTempFile("a.tmp", "hello");
Instant creation = TempFileMetadata.creationTime(file);
assertNotNull(creation);
assertFalse(creation.isAfter(Instant.now().plus(1, ChronoUnit.MINUTES)),
"创建时间不得晚于当前时间");
}
@Test
void sizeRecorded() throws Exception {
File file = writeTempFile("b.tmp", "0123456789");
assertEquals(10L, TempFileMetadata.size(file), "大小按字节记录");
}
@Test
void accessTimeReadable() throws Exception {
File file = writeTempFile("c.tmp", "x");
Instant access = TempFileMetadata.lastAccessTime(file);
assertNotNull(access);
assertFalse(access.isAfter(Instant.now().plus(1, ChronoUnit.MINUTES)));
}
@Test
void accessTimeFallsBackToMtime() throws Exception {
File file = writeTempFile("d.tmp", "y");
assertEquals(TempFileMetadata.lastModified(file), TempFileMetadata.lastAccessTime(file),
"无独立访问记录时以 mtime 回退(清理语义不变)");
}
@Test
void modifiedTimeUpdatesOnWrite() throws Exception {
File file = writeTempFile("e.tmp", "v1");
Instant before = TempFileMetadata.lastModified(file);
Thread.sleep(20);
Files.writeString(file.toPath(), "v2-longer-content", StandardCharsets.UTF_8);
Instant after = TempFileMetadata.lastModified(file);
assertFalse(after.isBefore(before), "写入后 mtime 必须更新");
}
@Test
void missingFileIsGraceful() {
File missing = tempDir.resolve("nope.tmp").toFile();
assertEquals(0L, TempFileMetadata.size(missing), "缺失文件大小为 0");
assertNotNull(TempFileMetadata.creationTime(missing), "缺失文件不回退抛错");
assertNotNull(TempFileMetadata.lastAccessTime(missing));
}
@Test
void nullFileIsGraceful() {
assertEquals(0L, TempFileMetadata.size(null));
assertEquals(Instant.EPOCH, TempFileMetadata.lastModified(null), "null 文件回退 EPOCH");
assertEquals(Instant.EPOCH, TempFileMetadata.lastAccessTime(null));
}
@Test
void timeOrderingConsistent() throws Exception {
File file = writeTempFile("f.tmp", "z");
Instant creation = TempFileMetadata.creationTime(file);
Instant modified = TempFileMetadata.lastModified(file);
Instant now = Instant.now().plus(1, ChronoUnit.MINUTES);
assertTrue(!creation.isAfter(modified.plus(1, ChronoUnit.SECONDS)),
"创建时间 <= 修改时间(容忍文件系统时钟精度)");
assertTrue(!modified.isAfter(now), "修改时间 <= 当前时间");
}
}
@@ -0,0 +1,121 @@
package com.nanri.aiimage.modules.file.service;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-155:孤立文件巡检报表契约(plan 09)。
* 只读巡检:超保留期且无引用的文件输出报表(路径/大小/时间);被引用不报;
* 不删除;批量/空报表/可重复。
*/
class TempOrphanInspectorTest {
@TempDir
Path tempDir;
private final TempOrphanInspector inspector = new TempOrphanInspector();
private Path writeFile(String name, String content) throws Exception {
Path path = tempDir.resolve(name);
Files.writeString(path, content, StandardCharsets.UTF_8);
return path;
}
private Instant hourAgo() {
return Instant.now().minus(1, ChronoUnit.HOURS);
}
private Instant hourLater() {
return Instant.now().plus(1, ChronoUnit.HOURS);
}
@Test
void orphanDetectedWhenExpiredAndUnreferenced() throws Exception {
Path file = writeFile("orphan.tmp", "x".repeat(10));
var report = inspector.inspectOrphanFiles(tempDir.toFile(), hourLater(), name -> false);
assertEquals(1, report.entries().size());
assertTrue(report.entries().getFirst().path().contains("orphan.tmp"));
}
@Test
void referencedFileNotReported() throws Exception {
writeFile("kept.tmp", "x");
var report = inspector.inspectOrphanFiles(tempDir.toFile(), hourLater(), name -> name.equals("kept.tmp"));
assertTrue(report.isEmpty(), "被引用文件不得进入报表");
}
@Test
void reportIncludesDetails() throws Exception {
Path file = writeFile("detail.tmp", "0123456789");
var report = inspector.inspectOrphanFiles(tempDir.toFile(), hourLater(), name -> false);
assertEquals(1, report.entries().size());
TempOrphanInspector.OrphanFileEntry entry = report.entries().getFirst();
assertTrue(entry.path().contains("detail.tmp"), "报表含路径");
assertEquals(10L, entry.sizeBytes(), "报表含大小");
assertTrue(entry.lastModifiedAt() != null && !entry.lastModifiedAt().isBlank(), "报表含修改时间");
}
@Test
void readOnlyDoesNotDelete() throws Exception {
Path file = writeFile("readonly.tmp", "keep-me");
inspector.inspectOrphanFiles(tempDir.toFile(), hourLater(), name -> false);
assertTrue(Files.exists(file), "巡检只读,不得删除文件");
}
@Test
void batchReportListsAllOrphans() throws Exception {
writeFile("a.tmp", "1");
writeFile("b.tmp", "22");
writeFile("c.tmp", "333");
var report = inspector.inspectOrphanFiles(tempDir.toFile(), hourLater(), name -> false);
assertEquals(3, report.entries().size());
assertEquals(6, report.totalBytes());
}
@Test
void emptyReportWhenNoOrphans() throws Exception {
writeFile("fresh.tmp", "new"); // 未超保留期
var report = inspector.inspectOrphanFiles(tempDir.toFile(), hourAgo(), name -> false);
assertTrue(report.isEmpty());
}
@Test
void rerunIsSafeAndStable() throws Exception {
writeFile("rerun.tmp", "x");
var first = inspector.inspectOrphanFiles(tempDir.toFile(), hourLater(), name -> false);
var second = inspector.inspectOrphanFiles(tempDir.toFile(), hourLater(), name -> false);
assertEquals(first.entries().size(), second.entries().size());
assertEquals(first.entries().getFirst().path(), second.entries().getFirst().path());
}
@Test
void invalidInputsAreGraceful() {
assertTrue(inspector.inspectOrphanFiles(null, hourAgo(), name -> false).isEmpty());
assertTrue(inspector.inspectOrphanFiles(tempDir.toFile(), null, name -> false).isEmpty());
assertTrue(inspector.inspectOrphanFiles(tempDir.resolve("absent").toFile(), hourAgo(), name -> false).isEmpty());
}
}
@@ -0,0 +1,97 @@
package com.nanri.aiimage.modules.file.service.oss;
import com.nanri.aiimage.config.OssProperties;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-147downloadUrl 永不过期验证(plan 08)。
* generateFreshDownloadUrlresult/ 前缀走下载域名、非 result/ 走公开端点、
* URL 无签名(不含 X-Amz 系列 / Expires / Signature 等过期参数)、bucket 前缀正确、
* null/空白安全、同输入同输出稳定。
*/
class DownloadUrlNonExpiringTest {
private OssProperties properties;
private OssStorageService storageService;
@BeforeEach
void setUp() {
properties = new OssProperties();
properties.setEndpoint("https://oss.aishufu.top");
properties.setPublicEndpoint("https://oss.aishufu.top");
properties.setDownloadEndpoint("https://download.aishufu.top");
properties.setBucket("nanri-ai-images");
properties.setImageVideoBucket("shufu-video");
properties.setDigitalHumanBucket("nanri-ai-digital-human");
properties.setTemplateBucket("aiimage-templates");
properties.setAccessKeyId("test-access-key");
properties.setAccessKeySecret("test-secret-key");
storageService = new OssStorageService(properties);
}
@Test
void resultPrefixUsesDownloadEndpoint() {
assertEquals(
"https://download.aishufu.top/nanri-ai-images/result/publish/1/result.xlsx",
storageService.generateFreshDownloadUrl("result/publish/1/result.xlsx"));
}
@Test
void nonResultPrefixUsesPublicEndpoint() {
assertEquals(
"https://oss.aishufu.top/nanri-ai-images/supply_images/main.jpg",
storageService.generateFreshDownloadUrl("supply_images/main.jpg"));
}
@Test
void urlHasNoExpiringSignatureParameters() {
String url = storageService.generateFreshDownloadUrl("result/publish/1/result.xlsx");
assertFalse(url.contains("X-Amz-"), "不得包含 AWS 签名参数: " + url);
assertFalse(url.contains("Signature"), "不得包含签名: " + url);
assertFalse(url.contains("Expires"), "不得包含过期时间: " + url);
assertFalse(url.contains("?") || url.contains("&"), "不得携带任何查询参数: " + url);
}
@Test
void bucketPrefixIsCorrect() {
String url = storageService.generateFreshDownloadUrl("result/collect_data/2/data.xlsx");
assertTrue(url.contains("/nanri-ai-images/result/collect_data/2/data.xlsx"),
"主 bucket 前缀正确: " + url);
String imageUrl = storageService.generateFreshDownloadUrl("shufu-video/result/image_video/demo.mp4");
assertTrue(imageUrl.contains("/shufu-video/result/image_video/demo.mp4"),
"image-video bucket 前缀正确: " + imageUrl);
}
@Test
void nullUrlIsSafe() {
assertNull(storageService.generateFreshDownloadUrl(null), "null 输入返回 null 不抛错");
}
@Test
void blankUrlIsSafe() {
assertNull(storageService.generateFreshDownloadUrl(" "), "空白输入返回 null 不抛错");
assertNull(storageService.generateFreshDownloadUrl(""), "空串返回 null 不抛错");
}
@Test
void generatedUrlIsStable() {
String first = storageService.generateFreshDownloadUrl("result/similar_asin/3/re.xlsx");
String second = storageService.generateFreshDownloadUrl("result/similar_asin/3/re.xlsx");
assertEquals(first, second, "同输入同输出(无随机签名)");
}
@Test
void contractFrozen() {
String url = storageService.generateFreshDownloadUrl("result/similar_asin/3/re.xlsx");
assertNotNull(url);
// 快照:https://{download-endpoint}/{bucket}/{objectKey},纯路径无签名
assertEquals("https://download.aishufu.top/nanri-ai-images/result/similar_asin/3/re.xlsx", url);
}
}
@@ -0,0 +1,236 @@
package com.nanri.aiimage.modules.publish.service;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.publish.mapper.PublishFileMapper;
import com.nanri.aiimage.modules.publish.mapper.PublishItemMapper;
import com.nanri.aiimage.modules.publish.model.dto.PublishResultFileDto;
import com.nanri.aiimage.modules.publish.model.dto.PublishSubmitResultRequest;
import com.nanri.aiimage.modules.publish.model.entity.PublishFileEntity;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskProgressLightAssembler;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import com.nanri.aiimage.modules.ziniao.service.ZiniaoShopSwitchService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.support.TransactionTemplate;
import java.lang.reflect.Method;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-130publish 事务收缩契约固化。
* 审计结论:submitResult 已符合 spec 07——prepareResultSubmission(纯计算 +
* RustFS payload 存储)在事务外;transactionTemplate 短事务内仅落库;提交后清理
* uncommitted payload(失败时异常吞掉);分片合并后落库仍在事务内。本测试固化契约。
*/
@ExtendWith(MockitoExtension.class)
class PublishTaskServiceTxBoundaryTest {
private static final Long TASK_ID = 5150L;
private static final Long USER_ID = 7L;
private static final Long FILE_ID = 601L;
@Mock private LocalFileStorageService localFileStorageService;
@Mock private ZiniaoShopSwitchService ziniaoShopSwitchService;
@Mock private PublishWorkbookService workbookService;
@Mock private PublishFileMapper publishFileMapper;
@Mock private PublishItemMapper publishItemMapper;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private OssStorageService ossStorageService;
@Mock private ObjectMapper objectMapper;
@Mock private TransactionTemplate transactionTemplate;
@Mock private InstanceMetadata instanceMetadata;
@Mock private TaskProgressLightAssembler taskProgressLightAssembler;
@InjectMocks private PublishTaskService service;
private final AtomicBoolean transactionActive = new AtomicBoolean();
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, FileResultEntity.class);
TableInfoHelper.initTableInfo(assistant, PublishFileEntity.class);
}
@BeforeEach
void setUp() {
lenient().when(instanceMetadata.getInstanceId()).thenReturn("instance-a");
lenient().when(taskDistributedLockService.acquire(eq("PUBLISH"), anyLong()))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class));
lenient().doAnswer(invocation -> {
transactionActive.set(true);
@SuppressWarnings("unchecked")
java.util.function.Consumer<org.springframework.transaction.TransactionStatus> action =
invocation.getArgument(0);
action.accept(null);
transactionActive.set(false);
return null;
}).when(transactionTemplate).executeWithoutResult(any());
lenient().when(publishFileMapper.selectById(FILE_ID)).thenReturn(runningFile());
lenient().when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
lenient().when(fileResultMapper.selectOne(any())).thenReturn(null);
lenient().when(publishFileMapper.updateById(any(com.nanri.aiimage.modules.publish.model.entity.PublishFileEntity.class))).thenReturn(1);
lenient().when(fileTaskMapper.updateById(any(FileTaskEntity.class))).thenReturn(1);
lenient().when(fileResultMapper.insert(any(FileResultEntity.class))).thenReturn(1);
}
@Test
void submitResultCarriesNoLongTransactionAnnotation() throws Exception {
Method submit = PublishTaskService.class.getMethod(
"submitResult", Long.class, PublishSubmitResultRequest.class);
assertNull(submit.getAnnotation(Transactional.class),
"submitResult 无 @Transactionalprepare 在外、短事务落库)");
}
@Test
void persistenceHappensInsideShortTransaction() {
when(publishFileMapper.selectList(any())).thenReturn(List.of(runningFile()));
AtomicBoolean writeInTx = new AtomicBoolean();
lenient().doAnswer(invocation -> {
writeInTx.set(transactionActive.get());
return 1;
}).when(publishFileMapper).updateById(any(com.nanri.aiimage.modules.publish.model.entity.PublishFileEntity.class));
service.submitResult(TASK_ID, errorRequest());
verify(publishFileMapper).updateById(any(com.nanri.aiimage.modules.publish.model.entity.PublishFileEntity.class));
assertTrue(writeInTx.get(), "落库必须在短事务内执行");
}
@Test
void prepareRunsOutsideTransaction() {
service.submitResult(TASK_ID, errorRequest());
// preparerequireTask 读 + requireFile 读)发生在事务回调之前
var order = inOrder(fileTaskMapper, transactionTemplate);
order.verify(fileTaskMapper).selectById(TASK_ID);
order.verify(transactionTemplate).executeWithoutResult(any());
}
@Test
void transactionFailureCleansUncommittedPayloadsAndPropagates() {
org.mockito.Mockito.doThrow(new IllegalStateException("tx failed"))
.when(transactionTemplate).executeWithoutResult(any());
IllegalStateException thrown = assertThrows(IllegalStateException.class,
() -> service.submitResult(TASK_ID, errorRequest()));
assertEquals("tx failed", thrown.getMessage());
verify(publishFileMapper, never()).updateById(any(com.nanri.aiimage.modules.publish.model.entity.PublishFileEntity.class));
}
@Test
void preparedErrorFileFailsFileInsideTransaction() {
when(publishFileMapper.selectList(any())).thenReturn(List.of(runningFile()));
service.submitResult(TASK_ID, errorRequest());
verify(publishFileMapper).updateById(any(com.nanri.aiimage.modules.publish.model.entity.PublishFileEntity.class));
verify(fileTaskMapper).updateById(any(FileTaskEntity.class));
verify(transactionTemplate).executeWithoutResult(any());
verify(transactionTemplate, times(1)).executeWithoutResult(any());
}
@Test
void committedPayloadsAreKeptNotDeleted() {
// error-only 请求无 payload 提交:清理路径不删除任何 payload
service.submitResult(TASK_ID, errorRequest());
verify(transientPayloadStorageService, never()).deletePayloadIfPresent(anyString());
}
@Test
void shortTransactionIsExactlyOnce() {
service.submitResult(TASK_ID, errorRequest());
verify(transactionTemplate, times(1)).executeWithoutResult(any());
verify(transactionTemplate, never()).execute(any());
}
@Test
void snapshotFileAndTaskStateTransitions() {
when(publishFileMapper.selectList(any())).thenReturn(List.of(runningFile()));
service.submitResult(TASK_ID, errorRequest());
verify(fileTaskMapper).updateById(any(FileTaskEntity.class));
verify(fileResultMapper).insert(any(FileResultEntity.class));
}
private PublishSubmitResultRequest errorRequest() {
PublishResultFileDto file = new PublishResultFileDto();
file.setFileId(FILE_ID);
file.setError("抓取失败");
PublishSubmitResultRequest request = new PublishSubmitResultRequest();
request.setUserId(USER_ID);
request.setFiles(List.of(file));
return request;
}
private FileTaskEntity runningTask() {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType("PUBLISH");
task.setStatus("RUNNING");
task.setUserId(USER_ID);
task.setTaskNo("PUB-5150");
task.setSourceFileCount(1);
return task;
}
private PublishFileEntity runningFile() {
PublishFileEntity file = new PublishFileEntity();
file.setId(FILE_ID);
file.setTaskId(TASK_ID);
file.setStatus("PENDING");
file.setTotalRows(0);
file.setProcessedRows(0);
return file;
}
}
@@ -554,6 +554,10 @@ class ShopDataCrawlDailyFileIncrementalTest {
assertEquals(1, dbDailyFiles.size(), "跨账号同店同日只保留一份累计文件");
assertEquals(3, dbMembers.size(), "三个结果各有一个成员行");
// 回归:V95 把累计文件改为店铺级共享时误删了 setUserId,导致新建
// daily_file 的 INSERT 不含 user_id(表列 NOT NULL 无默认值),
// 新店铺首次归档必报 Field 'user_id' doesn't have a default value。
assertEquals(7L, dbDailyFiles.get(0).getUserId(), "新建累计文件必须写入 user_id");
// 整表重建:英国以最后一次(9503)为准,德国(9502)保留
triggerRebuild();
@@ -0,0 +1,312 @@
package com.nanri.aiimage.modules.shopdatacrawl.service;
import cn.hutool.crypto.digest.DigestUtil;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.TaskPressureProperties;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.shopdatacrawl.model.dto.ShopDataCrawlCountryResultDto;
import com.nanri.aiimage.modules.shopdatacrawl.model.dto.ShopDataCrawlRowDto;
import com.nanri.aiimage.modules.shopdatacrawl.model.dto.ShopDataCrawlShopPayloadDto;
import com.nanri.aiimage.modules.shopdatacrawl.model.dto.ShopDataCrawlSubmitResultRequest;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskProgressLightAssembler;
import com.nanri.aiimage.modules.task.service.TaskProgressSnapshotService;
import com.nanri.aiimage.modules.task.service.TaskResultItemService;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import com.nanri.aiimage.modules.ziniao.service.ZiniaoShopSwitchService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.annotation.Transactional;
import java.lang.reflect.Method;
import java.lang.reflect.Modifier;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-129shopdatacrawl 事务收缩。
* 审计结论:/result 提交路径(submitResult → persistResultChunk/mergeShopPayload/
* persistProgressOrSnapshot)无 @Transactional、无长事务——写为单条原子写或
* executeShortTransaction 短事务,事务边界已窄;deleteTask 事务方法内的纯计算
* 段抽为无事务静态纯函数。本测试固化上述契约。
*/
@ExtendWith(MockitoExtension.class)
class ShopDataCrawlTaskServiceTxBoundaryTest {
private static final Long TASK_ID = 4242L;
private static final Long USER_ID = 7L;
private static final String SHOP_KEY = "amazon.de";
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private ShopDataCrawlResolveService shopDataCrawlResolveService;
@Mock private ShopDataCrawlExcelAssemblyService excelAssemblyService;
@Mock private ShopDataCrawlTaskCacheService taskCacheService;
@Mock private OssStorageService ossStorageService;
@Mock private ZiniaoShopSwitchService ziniaoShopSwitchService;
@Spy private ObjectMapper objectMapper = new ObjectMapper();
@Mock private TaskPressureProperties taskPressureProperties;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TaskResultItemService taskResultItemService;
@Mock private TaskProgressSnapshotService taskProgressSnapshotService;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private InstanceMetadata instanceMetadata;
@Mock private ShopDataCrawlDailyFileService dailyFileService;
@Mock private PlatformTransactionManager transactionManager;
@Mock private TaskProgressLightAssembler taskProgressLightAssembler;
@InjectMocks private ShopDataCrawlTaskService service;
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, FileResultEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskChunkEntity.class);
}
@BeforeEach
void setUp() {
lenient().when(instanceMetadata.getInstanceId()).thenReturn("instance-a");
lenient().when(taskPressureProperties.getDbSelectBatchSize()).thenReturn(500);
}
@Test
void collectResultIdsFiltersBrokenRows() {
FileResultEntity ok = resultRow(1L);
FileResultEntity nullId = new FileResultEntity();
FileResultEntity zeroId = new FileResultEntity();
zeroId.setId(0L);
assertEquals(java.util.Set.of(1L),
service.collectResultIds(java.util.Arrays.asList(ok, nullId, zeroId, null)));
}
@Test
void collectResultFileUrlsDeduplicatesAndSkipsBlank() {
FileResultEntity a = resultRow(1L);
a.setResultFileUrl("url-a");
FileResultEntity b = resultRow(2L);
b.setResultFileUrl("url-b");
FileResultEntity dup = resultRow(3L);
dup.setResultFileUrl("url-a");
FileResultEntity blank = resultRow(4L);
blank.setResultFileUrl(" ");
assertEquals(List.of("old-key", "url-a", "url-b"),
service.collectResultFileUrls(List.of(a, b, dup, blank), List.of("old-key")));
}
@Test
void pureHelpersAreStaticAndCarryNoTransactionAnnotation() throws Exception {
for (String name : List.of("collectResultIds", "collectResultFileUrls")) {
Method method = java.util.Arrays.stream(ShopDataCrawlTaskService.class.getDeclaredMethods())
.filter(candidate -> candidate.getName().equals(name))
.findFirst()
.orElseThrow();
assertTrue(Modifier.isStatic(method.getModifiers()), name + " 应为静态纯函数");
assertNull(method.getAnnotation(Transactional.class), name + " 不得带 @Transactional");
}
}
@Test
void submitResultCarriesNoTransactionAnnotation() throws Exception {
Method submit = ShopDataCrawlTaskService.class.getMethod(
"submitResult", Long.class, ShopDataCrawlSubmitResultRequest.class);
assertNull(submit.getAnnotation(Transactional.class),
"submitResult 无 @Transactional/result 提交路径无长事务)");
}
@Test
void submitResultPathUsesNoLongTransaction() throws Exception {
FileTaskEntity task = runningTask();
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
when(fileResultMapper.selectList(any())).thenReturn(List.of(resultRow(1L)));
when(taskDistributedLockService.acquire(any(), anyLong()))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class), null);
service.submitResult(TASK_ID, submitRequest(false));
verify(transactionManager, never()).getTransaction(any());
verify(taskCacheService).touchTaskHeartbeat(TASK_ID);
verify(fileTaskMapper, times(1)).updateById(any(FileTaskEntity.class));
}
@Test
void chunkPathPersistsWithSingleInsertWithoutTransaction() throws Exception {
FileTaskEntity task = runningTask();
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
FileResultEntity row = resultRow(1L);
row.setSourceFilename(SHOP_KEY);
when(fileResultMapper.selectList(any())).thenReturn(List.of(row));
when(taskDistributedLockService.acquire(any(), anyLong()))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class), null);
lenient().when(transientPayloadStorageService.isSharedWriteEnabled()).thenReturn(true);
when(transientPayloadStorageService.storeChunkPayloadVersioned(
eq("SHOP_DATA_CRAWL"), eq(TASK_ID), any(), eq(1), any()))
.thenReturn("\"rustfs:chunk\"");
lenient().when(transientPayloadStorageService.extractPointer("\"rustfs:chunk\""))
.thenReturn("rustfs:chunk");
lenient().when(transientPayloadStorageService.resolvePayload(anyString(), anyString()))
.thenReturn("[{\"country\":\"DE\",\"items\":[{\"asin\":\"B0TEST1234\"}]}]");
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(null);
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(1L);
lenient().when(taskScopeStateMapper.selectOne(any())).thenReturn(null);
when(taskChunkMapper.selectList(any())).thenReturn(List.of(completedChunk()));
service.submitResult(TASK_ID, chunkRequest());
verify(transactionManager, never()).getTransaction(any());
verify(taskChunkMapper).insert(any(TaskChunkEntity.class));
verify(fileResultMapper).updateById(any(FileResultEntity.class));
}
@Test
void deleteTaskKeepsTransactionAnnotationAndPureCompute() throws Exception {
Method delete = ShopDataCrawlTaskService.class.getMethod("deleteTask", Long.class, Long.class);
assertTrue(delete.getAnnotation(Transactional.class) != null,
"deleteTask 落库删除必须保留 @Transactional(锁内删除语义不变)");
// 删除路径使用抽出的纯函数(行为等价由既有 ShopDataCrawlCleanupTest 守门)
assertTrue(ShopDataCrawlTaskService.collectResultIds(List.of()).isEmpty());
}
@Test
void duplicateChunkPathIsIdempotentWithoutLongTransaction() throws Exception {
FileTaskEntity task = runningTask();
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
FileResultEntity row = resultRow(1L);
row.setSourceFilename(SHOP_KEY);
when(fileResultMapper.selectList(any())).thenReturn(List.of(row));
when(taskDistributedLockService.acquire(any(), anyLong()))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class), null);
lenient().when(transientPayloadStorageService.isSharedWriteEnabled()).thenReturn(true);
java.util.concurrent.atomic.AtomicReference<String> chunkPayload =
new java.util.concurrent.atomic.AtomicReference<>();
when(transientPayloadStorageService.storeChunkPayloadVersioned(
eq("SHOP_DATA_CRAWL"), eq(TASK_ID), any(), eq(1), any()))
.thenAnswer(invocation -> {
chunkPayload.set(invocation.getArgument(4));
return "\"rustfs:chunk\"";
});
lenient().when(transientPayloadStorageService.extractPointer("\"rustfs:chunk\""))
.thenReturn("rustfs:chunk");
lenient().when(transientPayloadStorageService.resolvePayload(anyString(), anyString()))
.thenReturn("[{\"country\":\"DE\",\"items\":[{\"asin\":\"B0TEST1234\"}]}]");
lenient().when(taskChunkMapper.selectOne(any())).thenAnswer(invocation -> {
TaskChunkEntity winner = completedChunk();
winner.setPayloadHash(DigestUtil.sha256Hex(chunkPayload.get()));
return winner;
});
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(1L);
lenient().when(taskScopeStateMapper.selectOne(any())).thenAnswer(invocation -> {
// 重复提交:scope 计数器已持久化(received=1),幂等判定 completed
com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity scope =
new com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity();
scope.setId(901L);
scope.setTaskId(TASK_ID);
scope.setReceivedChunkCount(1);
scope.setChunkTotal(1);
return scope;
});
when(taskChunkMapper.selectList(any())).thenReturn(List.of(completedChunk()));
org.mockito.Mockito.doThrow(new org.springframework.dao.DuplicateKeyException("dup"))
.when(taskChunkMapper).insert(any(TaskChunkEntity.class));
service.submitResult(TASK_ID, chunkRequest());
verify(transactionManager, never()).getTransaction(any());
verify(transientPayloadStorageService).deletePayloadIfPresent("\"rustfs:chunk\"");
}
private FileTaskEntity runningTask() {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType("SHOP_DATA_CRAWL");
task.setStatus("RUNNING");
task.setUserId(USER_ID);
task.setResultJson("{\"ownerInstanceId\":\"instance-a\"}");
return task;
}
private FileResultEntity resultRow(Long id) {
FileResultEntity row = new FileResultEntity();
row.setId(id);
row.setTaskId(TASK_ID);
row.setSourceFilename(SHOP_KEY);
row.setModuleType("SHOP_DATA_CRAWL");
return row;
}
private TaskChunkEntity completedChunk() {
TaskChunkEntity chunk = new TaskChunkEntity();
chunk.setId(801L);
chunk.setTaskId(TASK_ID);
chunk.setModuleType("SHOP_DATA_CRAWL");
chunk.setScopeKey("result-chunks:" + SHOP_KEY);
chunk.setScopeHash("result-chunks-hash");
chunk.setChunkIndex(1);
chunk.setChunkTotal(1);
chunk.setPayloadJson("\"rustfs:chunk\"");
return chunk;
}
private ShopDataCrawlSubmitResultRequest submitRequest(boolean done) {
ShopDataCrawlShopPayloadDto shop = new ShopDataCrawlShopPayloadDto();
shop.setShopName(SHOP_KEY);
shop.setShopDone(done);
ShopDataCrawlSubmitResultRequest request = new ShopDataCrawlSubmitResultRequest();
request.setShops(List.of(shop));
return request;
}
private ShopDataCrawlSubmitResultRequest chunkRequest() {
ShopDataCrawlRowDto row = new ShopDataCrawlRowDto();
row.setAsin("B0TEST1234");
ShopDataCrawlCountryResultDto country = new ShopDataCrawlCountryResultDto();
country.setCountry("DE");
country.setItems(List.of(row));
ShopDataCrawlShopPayloadDto shop = new ShopDataCrawlShopPayloadDto();
shop.setShopName(SHOP_KEY);
shop.setChunkIndex(1);
shop.setChunkTotal(1);
shop.setCountryResults(List.of(country));
ShopDataCrawlSubmitResultRequest request = new ShopDataCrawlSubmitResultRequest();
request.setShops(List.of(shop));
return request;
}
}
@@ -1,147 +0,0 @@
package com.nanri.aiimage.modules.shopkey.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.modules.shopkey.mapper.ShopCredentialCheckMapper;
import com.nanri.aiimage.modules.shopkey.mapper.ShopManageMapper;
import com.nanri.aiimage.modules.shopkey.model.dto.ShopCredentialCheckReportRequest;
import com.nanri.aiimage.modules.shopkey.model.entity.ShopCredentialCheckEntity;
import com.nanri.aiimage.modules.shopkey.model.entity.ShopManageEntity;
import com.nanri.aiimage.modules.shopkey.model.vo.ShopCredentialCheckClaimVo;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
@ExtendWith(MockitoExtension.class)
class ShopCredentialCheckServiceTest {
@Mock
private ShopCredentialCheckMapper shopCredentialCheckMapper;
@Mock
private ShopManageMapper shopManageMapper;
@InjectMocks
private ShopCredentialCheckService service;
@BeforeEach
void initTableInfo() {
// MyBatis-Plus Lambda 缓存依赖 TableInfo,单测环境需手动初始化(对应实体)
initTable(ShopCredentialCheckEntity.class);
initTable(ShopManageEntity.class);
}
private void initTable(Class<?> entityClass) {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, entityClass);
}
@Test
void createReusesActiveTaskForSameShop() {
ShopManageEntity shop = new ShopManageEntity();
shop.setId(7L);
shop.setShopName("美国站-主营");
when(shopManageMapper.selectOne(any())).thenReturn(shop);
ShopCredentialCheckEntity active = new ShopCredentialCheckEntity();
active.setId(3L);
active.setShopId(7L);
active.setShopName("美国站-主营");
active.setStatus(ShopCredentialCheckService.STATUS_RUNNING);
when(shopCredentialCheckMapper.selectOne(any())).thenReturn(active);
var vo = service.create("美国站-主营");
assertEquals(3L, vo.getId());
assertEquals("RUNNING", vo.getStatus());
verify(shopCredentialCheckMapper, never()).insert(any(ShopCredentialCheckEntity.class));
}
@Test
void createRejectsUnknownShop() {
when(shopManageMapper.selectOne(any())).thenReturn(null);
BusinessException ex = assertThrows(BusinessException.class, () -> service.create("不存在店铺"));
assertEquals("后台店铺管理中未找到店铺:不存在店铺,请先添加店铺信息", ex.getMessage());
}
@Test
void claimSetsRunningAndReturnsZnUsername() {
ShopCredentialCheckEntity pending = new ShopCredentialCheckEntity();
pending.setId(9L);
pending.setShopName("店铺-测试");
when(shopCredentialCheckMapper.selectList(any(LambdaQueryWrapper.class))).thenReturn(java.util.List.of());
when(shopCredentialCheckMapper.selectOne(any(LambdaQueryWrapper.class))).thenReturn(pending);
when(shopCredentialCheckMapper.update(any(), any(LambdaUpdateWrapper.class))).thenReturn(1);
ShopManageEntity shop = new ShopManageEntity();
shop.setZnUsername("zn-user-1");
when(shopManageMapper.selectOne(any())).thenReturn(shop);
ShopCredentialCheckClaimVo vo = service.claimForClient("PC-01");
assertNotNull(vo);
assertEquals(9L, vo.getId());
assertEquals("店铺-测试", vo.getShopName());
assertEquals("zn-user-1", vo.getZnUsername());
verify(shopCredentialCheckMapper, times(1)).update(any(), any(LambdaUpdateWrapper.class));
}
@Test
void claimReturnsNullWhenNothingPending() {
when(shopCredentialCheckMapper.selectList(any(LambdaQueryWrapper.class))).thenReturn(java.util.List.of());
when(shopCredentialCheckMapper.selectOne(any(LambdaQueryWrapper.class))).thenReturn(null);
assertNull(service.claimForClient("PC-01"));
verify(shopCredentialCheckMapper, never()).update(any(), any(LambdaUpdateWrapper.class));
}
@Test
void reportIgnoresStaleStatus() {
ShopCredentialCheckEntity finished = new ShopCredentialCheckEntity();
finished.setId(5L);
finished.setStatus(ShopCredentialCheckService.STATUS_SUCCESS);
when(shopCredentialCheckMapper.selectById(5L)).thenReturn(finished);
ShopCredentialCheckReportRequest request = new ShopCredentialCheckReportRequest();
request.setStatus(ShopCredentialCheckService.STATUS_FAILED);
request.setDetail("密码错误");
service.report(5L, request);
verify(shopCredentialCheckMapper, never()).updateById(any(ShopCredentialCheckEntity.class));
}
@Test
void reportRecordsFailureDetail() {
ShopCredentialCheckEntity running = new ShopCredentialCheckEntity();
running.setId(6L);
running.setStatus(ShopCredentialCheckService.STATUS_RUNNING);
when(shopCredentialCheckMapper.selectById(6L)).thenReturn(running);
ShopCredentialCheckReportRequest request = new ShopCredentialCheckReportRequest();
request.setStatus(ShopCredentialCheckService.STATUS_FAILED);
request.setDetail("账号或密码错误");
request.setClientHost("PC-02");
service.report(6L, request);
verify(shopCredentialCheckMapper, times(1)).updateById(running);
assertEquals(ShopCredentialCheckService.STATUS_FAILED, running.getStatus());
assertEquals("账号或密码错误", running.getDetail());
assertEquals("PC-02", running.getClientHost());
assertNotNull(running.getCheckFinishedAt());
}
}
@@ -2,7 +2,6 @@ package com.nanri.aiimage.modules.shopkey.service;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.common.security.ShopCredentialCryptoService;
import com.nanri.aiimage.modules.shopkey.mapper.ShopCredentialCheckMapper;
import com.nanri.aiimage.modules.shopkey.mapper.ShopManageMapper;
import com.nanri.aiimage.modules.shopkey.model.entity.ShopManageEntity;
import org.junit.jupiter.api.Test;
@@ -26,8 +25,6 @@ class ShopManageServiceTest {
private ShopManageGroupService shopManageGroupService;
@Mock
private ShopCredentialCryptoService shopCredentialCryptoService;
@Mock
private ShopCredentialCheckMapper shopCredentialCheckMapper;
@InjectMocks
private ShopManageService service;
@@ -0,0 +1,295 @@
package com.nanri.aiimage.modules.similarasin.service;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.SimilarAsinProperties;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.similarasin.mapper.SimilarAsinFilterConditionMapper;
import com.nanri.aiimage.modules.similarasin.model.dto.SimilarAsinSubmitResultRequest;
import com.nanri.aiimage.modules.similarasin.util.SimilarAsinImageEmbedder;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskProgressSnapshotService;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.dao.DuplicateKeyException;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.TransactionDefinition;
import org.springframework.transaction.TransactionStatus;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-127similarasin 临时文件清理时机契约固化。
* 审计结论:提交后清理(cleanupPreparedSubmittedChunkIfUnreferenced 在事务
* execute 返回后执行)已符合"挂 afterCommit"语义——回滚不清理、清理异常吞掉、
* 失败可重试;本测试固化该契约。
*/
@ExtendWith(MockitoExtension.class)
class SimilarAsinTaskServiceCleanupAfterCommitTest {
private static final Long TASK_ID = 21879L;
private static final String PARSED_POINTER = "rustfs:task-parsed/similar-asin/21879/payload.json";
private static final String CHUNK_POINTER = "rustfs:task-chunk/similar-asin/21879/chunk.json";
private static final String STORED_CHUNK_POINTER = "\"" + CHUNK_POINTER + "\"";
@Mock private LocalFileStorageService localFileStorageService;
@Mock private OssStorageService ossStorageService;
@Mock private StorageProperties storageProperties;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private SimilarAsinFilterConditionMapper filterConditionMapper;
@Spy private ObjectMapper objectMapper = new ObjectMapper();
@Mock private SimilarAsinTaskCacheService taskCacheService;
@Mock private SimilarAsinProperties properties;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private TaskProgressSnapshotService taskProgressSnapshotService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private PlatformTransactionManager transactionManager;
@Mock private DistributedJobLockService distributedJobLockService;
@Mock private InstanceMetadata instanceMetadata;
@Mock private SimilarAsinImageEmbedder imageEmbedder;
@Mock private SimilarAsinImagePrefetchService imagePrefetchService;
@Mock private TransactionStatus transactionStatus;
@InjectMocks private SimilarAsinTaskService service;
private final AtomicBoolean transactionActive = new AtomicBoolean();
private final AtomicBoolean cleanupOutsideTx = new AtomicBoolean(true);
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskChunkEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskScopeStateEntity.class);
}
@BeforeEach
void setUpTransactionAndLock() {
lenient().when(instanceMetadata.getInstanceId()).thenReturn("instance-a");
lenient().when(taskDistributedLockService.acquire(
eq(SimilarAsinTaskService.MODULE_TYPE), anyLong(), any(Duration.class), eq(10_000L)))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class));
lenient().when(transactionManager.getTransaction(any(TransactionDefinition.class)))
.thenAnswer(invocation -> {
transactionActive.set(true);
return transactionStatus;
});
lenient().doAnswer(invocation -> {
transactionActive.set(false);
return null;
}).when(transactionManager).commit(transactionStatus);
lenient().doAnswer(invocation -> {
transactionActive.set(false);
return null;
}).when(transactionManager).rollback(transactionStatus);
lenient().when(taskChunkMapper.selectList(any())).thenReturn(List.of());
lenient().when(fileResultMapper.selectList(any())).thenReturn(List.of());
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(0L);
lenient().when(taskScopeStateMapper.selectCount(any())).thenReturn(0L);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(null);
lenient().when(transientPayloadStorageService.storeChunkPayloadVersioned(
eq(SimilarAsinTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString()))
.thenReturn(STORED_CHUNK_POINTER);
lenient().when(transientPayloadStorageService.wasLastStoreLocalFallback()).thenReturn(false);
lenient().when(transientPayloadStorageService.extractPointer(STORED_CHUNK_POINTER))
.thenReturn(CHUNK_POINTER);
lenient().doAnswer(invocation -> {
TaskScopeStateEntity scope = invocation.getArgument(0);
scope.setId(401L);
return 1;
}).when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
lenient().doAnswer(invocation -> {
cleanupOutsideTx.set(!transactionActive.get());
return null;
}).when(transientPayloadStorageService).deletePayloadIfPresent(anyString());
}
@AfterEach
void shutdownExecutors() {
service.shutdownAssembleExecutor();
}
/** 并发重复 chunkinsert 撞唯一键)→ payload 未落库 → 提交后清理。 */
private void configureDuplicateChunkFlow() {
doThrow(new DuplicateKeyException("duplicate chunk"))
.when(taskChunkMapper).insert(any(TaskChunkEntity.class));
}
@Test
void cleanupRunsAfterCommitForUnpersistedPayload() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
configureDuplicateChunkFlow();
service.submitResult(TASK_ID, request(false));
verify(transientPayloadStorageService).deletePayloadIfPresent(STORED_CHUNK_POINTER);
var order = inOrder(transactionManager, transientPayloadStorageService);
order.verify(transactionManager).commit(transactionStatus);
order.verify(transientPayloadStorageService).deletePayloadIfPresent(STORED_CHUNK_POINTER);
}
@Test
void rollbackSkipsCleanup() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
configureDuplicateChunkFlow();
doThrow(new IllegalStateException("scope upsert failed"))
.when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
org.junit.jupiter.api.Assertions.assertThrows(IllegalStateException.class,
() -> service.submitResult(TASK_ID, request(false)));
verify(transactionManager).rollback(transactionStatus);
verify(transientPayloadStorageService, never()).deletePayloadIfPresent(anyString());
}
@Test
void cleanupErrorIsSwallowedAndResponseSucceeds() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
configureDuplicateChunkFlow();
doThrow(new IllegalStateException("cleanup failed"))
.when(transientPayloadStorageService).deletePayloadIfPresent(anyString());
service.submitResult(TASK_ID, request(false));
verify(taskCacheService).touchTaskHeartbeat(TASK_ID);
verify(transientPayloadStorageService).deletePayloadIfPresent(STORED_CHUNK_POINTER);
}
@Test
void cleanupObservesTransactionAlreadyFinished() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
configureDuplicateChunkFlow();
service.submitResult(TASK_ID, request(false));
verify(transientPayloadStorageService).deletePayloadIfPresent(STORED_CHUNK_POINTER);
org.junit.jupiter.api.Assertions.assertTrue(cleanupOutsideTx.get(),
"清理必须在事务结束后执行(等价 afterCommit");
}
@Test
void cleanupFailureKeepsPayloadRetryableOnNextSubmission() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
configureDuplicateChunkFlow();
doThrow(new IllegalStateException("cleanup failed"))
.when(transientPayloadStorageService).deletePayloadIfPresent(anyString());
service.submitResult(TASK_ID, request(false));
// 恢复后再次提交,同一 payload 可再次进入清理
service.submitResult(TASK_ID, request(false));
verify(transientPayloadStorageService, times(2)).deletePayloadIfPresent(STORED_CHUNK_POINTER);
}
@Test
void cleanupIsIdempotentAcrossRepeatedSubmissions() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
configureDuplicateChunkFlow();
service.submitResult(TASK_ID, request(false));
service.submitResult(TASK_ID, request(false));
verify(transientPayloadStorageService, times(2)).deletePayloadIfPresent(STORED_CHUNK_POINTER);
}
@Test
void cleanupChecksReferencesBeforeDelete() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
configureDuplicateChunkFlow();
service.submitResult(TASK_ID, request(false));
var order = inOrder(transientPayloadStorageService, taskChunkMapper, taskScopeStateMapper,
transientPayloadStorageService);
order.verify(transientPayloadStorageService).extractPointer(STORED_CHUNK_POINTER);
order.verify(taskChunkMapper).selectCount(any());
order.verify(taskScopeStateMapper).selectCount(any());
order.verify(transientPayloadStorageService).deletePayloadIfPresent(STORED_CHUNK_POINTER);
}
@Test
void referencedPayloadIsKeptNotDeleted() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
configureDuplicateChunkFlow();
when(taskChunkMapper.selectCount(any())).thenReturn(1L);
service.submitResult(TASK_ID, request(false));
verify(transientPayloadStorageService, never()).deletePayloadIfPresent(STORED_CHUNK_POINTER);
}
private SimilarAsinSubmitResultRequest request(boolean done) {
SimilarAsinSubmitResultRequest request = new SimilarAsinSubmitResultRequest();
request.setSubmissionId("similar-asin-21879");
request.setChunkIndex(0);
request.setChunkTotal(1);
request.setDone(done);
return request;
}
private FileTaskEntity runningTask(String owner) {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType(SimilarAsinTaskService.MODULE_TYPE);
task.setStatus("RUNNING");
task.setUserId(7L);
String resultJson = "{\"parsedPayloadRef\":\"" + PARSED_POINTER + "\"";
if (owner != null) {
resultJson += ",\"ownerInstanceId\":\"" + owner + "\"";
}
task.setResultJson(resultJson + "}");
return task;
}
}
@@ -0,0 +1,299 @@
package com.nanri.aiimage.modules.similarasin.service;
import cn.hutool.crypto.digest.DigestUtil;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.SimilarAsinProperties;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.similarasin.mapper.SimilarAsinFilterConditionMapper;
import com.nanri.aiimage.modules.similarasin.model.dto.SimilarAsinSubmitResultRequest;
import com.nanri.aiimage.modules.similarasin.util.SimilarAsinImageEmbedder;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskProgressSnapshotService;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.TransactionDefinition;
import org.springframework.transaction.TransactionStatus;
import java.lang.reflect.Method;
import java.lang.reflect.Modifier;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-126similarasin /result DTO 组装位置契约固化。
* 审计结论:/result 响应为 ApiResponse.success(null)(无响应 DTO);可移出的
* DTO 组装(PreparedSubmittedChunk/SubmitContext 前置数据)已全部在 prepare
* 阶段、事务开始前完成(task-125);写事务内仅剩落库行实体构造(必须与
* insert/upsert 同序,属 spec §2 禁拆顺序的一部分)。
* 本测试固化上述契约:组装在事务前/提交后、回滚无事务后组装、落库行快照不变。
*/
@ExtendWith(MockitoExtension.class)
class SimilarAsinTaskServiceDtoAssemblyTest {
private static final Long TASK_ID = 21879L;
private static final String PARSED_POINTER = "rustfs:task-parsed/similar-asin/21879/payload.json";
private static final String CHUNK_POINTER = "rustfs:task-chunk/similar-asin/21879/chunk.json";
private static final String STORED_CHUNK_POINTER = "\"" + CHUNK_POINTER + "\"";
@Mock private LocalFileStorageService localFileStorageService;
@Mock private OssStorageService ossStorageService;
@Mock private StorageProperties storageProperties;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private SimilarAsinFilterConditionMapper filterConditionMapper;
@Spy private ObjectMapper objectMapper = new ObjectMapper();
@Mock private SimilarAsinTaskCacheService taskCacheService;
@Mock private SimilarAsinProperties properties;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private TaskProgressSnapshotService taskProgressSnapshotService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private PlatformTransactionManager transactionManager;
@Mock private DistributedJobLockService distributedJobLockService;
@Mock private InstanceMetadata instanceMetadata;
@Mock private SimilarAsinImageEmbedder imageEmbedder;
@Mock private SimilarAsinImagePrefetchService imagePrefetchService;
@Mock private TransactionStatus transactionStatus;
@InjectMocks private SimilarAsinTaskService service;
private final AtomicBoolean transactionActive = new AtomicBoolean();
private final AtomicReference<TaskChunkEntity> insertedChunk = new AtomicReference<>();
private final AtomicReference<String> storedPayloadJson = new AtomicReference<>();
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskChunkEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskScopeStateEntity.class);
}
@BeforeEach
void setUpTransactionAndLock() {
lenient().when(instanceMetadata.getInstanceId()).thenReturn("instance-a");
lenient().when(taskDistributedLockService.acquire(
eq(SimilarAsinTaskService.MODULE_TYPE), anyLong(), any(Duration.class), eq(10_000L)))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class));
lenient().when(transactionManager.getTransaction(any(TransactionDefinition.class)))
.thenAnswer(invocation -> {
transactionActive.set(true);
return transactionStatus;
});
lenient().doAnswer(invocation -> {
transactionActive.set(false);
return null;
}).when(transactionManager).commit(transactionStatus);
lenient().doAnswer(invocation -> {
transactionActive.set(false);
return null;
}).when(transactionManager).rollback(transactionStatus);
lenient().when(taskChunkMapper.selectList(any())).thenReturn(List.of());
lenient().when(fileResultMapper.selectList(any())).thenReturn(List.of());
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(1L);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(null);
lenient().when(transientPayloadStorageService.storeChunkPayloadVersioned(
eq(SimilarAsinTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString()))
.thenAnswer(invocation -> {
storedPayloadJson.set(invocation.getArgument(4));
return STORED_CHUNK_POINTER;
});
lenient().when(transientPayloadStorageService.wasLastStoreLocalFallback()).thenReturn(false);
lenient().doAnswer(invocation -> {
TaskChunkEntity chunk = invocation.getArgument(0);
chunk.setId(301L);
insertedChunk.set(chunk);
return 1;
}).when(taskChunkMapper).insert(any(TaskChunkEntity.class));
lenient().doAnswer(invocation -> {
TaskScopeStateEntity scope = invocation.getArgument(0);
scope.setId(401L);
return 1;
}).when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
}
@AfterEach
void shutdownExecutors() {
service.shutdownAssembleExecutor();
}
@Test
void dtoAssemblyHappensBeforeTransactionStarts() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
// prepare 组装(含 payload 存储)→ 事务开始 → 落库
var order = inOrder(transientPayloadStorageService, transactionManager, taskChunkMapper);
order.verify(transientPayloadStorageService).storeChunkPayloadVersioned(
eq(SimilarAsinTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString());
order.verify(transactionManager).getTransaction(any(TransactionDefinition.class));
order.verify(taskChunkMapper).insert(any(TaskChunkEntity.class));
}
@Test
void assembledChunkFieldsMatchSnapshot() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
TaskChunkEntity chunk = insertedChunk.get();
assertEquals(TASK_ID, chunk.getTaskId());
assertEquals(SimilarAsinTaskService.MODULE_TYPE, chunk.getModuleType());
assertEquals("similar-asin-21879", chunk.getScopeKey());
assertEquals(0, chunk.getChunkIndex());
assertEquals(1, chunk.getChunkTotal());
assertEquals(STORED_CHUNK_POINTER, chunk.getPayloadJson());
assertEquals(DigestUtil.sha256Hex(storedPayloadJson.get()), chunk.getPayloadHash());
assertNotNull(chunk.getCreatedAt());
assertNotNull(chunk.getUpdatedAt());
}
@Test
void postTransactionSideEffectsRunAfterCommit() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
var order = inOrder(transactionManager, taskCacheService);
order.verify(transactionManager).commit(transactionStatus);
order.verify(taskCacheService).touchTaskHeartbeat(TASK_ID);
}
@Test
void rollbackSkipsPostTransactionAssemblyAndSideEffects() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
org.mockito.Mockito.doThrow(new IllegalStateException("scope upsert failed"))
.when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
assertThrows(IllegalStateException.class, () -> service.submitResult(TASK_ID, request(false)));
verify(transactionManager).rollback(transactionStatus);
verify(transactionManager, never()).commit(transactionStatus);
verify(taskCacheService, never()).touchTaskHeartbeat(TASK_ID);
}
@Test
void assemblyFailureBeforeTransactionLeavesNoPartialWrites() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
when(transientPayloadStorageService.storeChunkPayloadVersioned(
eq(SimilarAsinTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString()))
.thenThrow(new IllegalStateException("payload store failed"));
assertThrows(IllegalStateException.class, () -> service.submitResult(TASK_ID, request(false)));
verify(taskChunkMapper, never()).insert(any(TaskChunkEntity.class));
verify(taskScopeStateMapper, never()).insert(any(TaskScopeStateEntity.class));
verify(transactionManager, never()).commit(transactionStatus);
}
@Test
void resultEndpointReturnsNoResponseDto() throws Exception {
Method submitResult = SimilarAsinTaskService.class.getMethod("submitResult", Long.class, SimilarAsinSubmitResultRequest.class);
assertEquals(Void.TYPE, submitResult.getReturnType(), "/result 服务方法必须返回 void(无响应 DTO 组装)");
assertTrue(Modifier.isPublic(submitResult.getModifiers()));
}
@Test
void chunkAssemblyAddsNoExtraDatabaseReads() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
// chunk 落库行组装不引入额外查询:insert 前最后一次 mapper 交互为
// completeSubmittedChunk 的任务状态重读(与 finalize 判定同序,属现状)。
var order = inOrder(taskChunkMapper, fileTaskMapper, taskChunkMapper);
order.verify(taskChunkMapper).selectOne(any()); // prepare 查重
order.verify(taskChunkMapper).selectOne(any()); // persist 查重
order.verify(taskChunkMapper).insert(any(TaskChunkEntity.class));
}
@Test
void chunkSnapshotIsStableAcrossSubmissions() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
TaskChunkEntity first = insertedChunk.get();
service.submitResult(TASK_ID, request(false));
TaskChunkEntity second = insertedChunk.get();
assertEquals(first.getScopeKey(), second.getScopeKey());
assertEquals(first.getScopeHash(), second.getScopeHash());
assertEquals(first.getPayloadHash(), second.getPayloadHash());
assertEquals(first.getPayloadJson(), second.getPayloadJson());
}
private SimilarAsinSubmitResultRequest request(boolean done) {
SimilarAsinSubmitResultRequest request = new SimilarAsinSubmitResultRequest();
request.setSubmissionId("similar-asin-21879");
request.setChunkIndex(0);
request.setChunkTotal(1);
request.setDone(done);
return request;
}
private FileTaskEntity runningTask(String owner) {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType(SimilarAsinTaskService.MODULE_TYPE);
task.setStatus("RUNNING");
task.setUserId(7L);
String resultJson = "{\"parsedPayloadRef\":\"" + PARSED_POINTER + "\"";
if (owner != null) {
resultJson += ",\"ownerInstanceId\":\"" + owner + "\"";
}
task.setResultJson(resultJson + "}");
return task;
}
}
@@ -54,6 +54,41 @@ class SimilarAsinTaskServiceTest {
assertEquals("\u6210\u529f", SimilarAsinTaskService.resolveResultStatus(withVisibleResultData, "", "80%", "", "", ""));
}
@Test
void resultStatusMarksAsinNotFoundRows() {
// Python \u7aef 404 \u884c\uff1aerror="404 Not Found"\uff0cstatus \u88ab markRowFailed \u7f6e\u4e3a FAILED\uff0c
// \u4f46\u72b6\u6001\u5217\u5fc5\u987b\u663e\u793a"ASIN\u672a\u627e\u5230"\uff0c\u4e0d\u80fd\u843d\u56de\u666e\u901a"\u5931\u8d25"\u3002
SimilarAsinResultRowDto notFound = new SimilarAsinResultRowDto();
notFound.setError("404 Not Found");
notFound.setStatus("FAILED");
notFound.setReason("required field missing (asin/title/url), skip LLM");
assertEquals("ASIN\u672a\u627e\u5230", SimilarAsinTaskService.resolveResultStatus(notFound));
// \u5176\u5b83\u7b49\u4ef7\u6587\u6848\u540c\u6837\u547d\u4e2d
SimilarAsinResultRowDto chinese = new SimilarAsinResultRowDto();
chinese.setError("\u5546\u54c1\u672a\u627e\u5230");
assertEquals("ASIN\u672a\u627e\u5230", SimilarAsinTaskService.resolveResultStatus(chinese));
// \u666e\u901a\u6280\u672f\u6027\u5931\u8d25\u4e0d\u8bef\u5224\uff1aerror \u4e0d\u542b 404/not found
SimilarAsinResultRowDto llmFailure = new SimilarAsinResultRowDto();
llmFailure.setError("LLM \u8c03\u7528\u8d85\u65f6");
llmFailure.setStatus("FAILED");
assertEquals("\u5931\u8d25", SimilarAsinTaskService.resolveResultStatus(llmFailure));
}
@Test
void asinNotFoundDetectionMatchesPythonErrors() {
assertTrue(SimilarAsinTaskService.isAsinNotFound("404 Not Found"));
assertTrue(SimilarAsinTaskService.isAsinNotFound("404 not found"));
assertTrue(SimilarAsinTaskService.isAsinNotFound("404"));
assertTrue(SimilarAsinTaskService.isAsinNotFound("\u5546\u54c1\u672a\u627e\u5230"));
assertTrue(SimilarAsinTaskService.isAsinNotFound("ASIN \u672a\u627e\u5230"));
assertFalse(SimilarAsinTaskService.isAsinNotFound(null));
assertFalse(SimilarAsinTaskService.isAsinNotFound(""));
assertFalse(SimilarAsinTaskService.isAsinNotFound("LLM \u8c03\u7528\u8d85\u65f6"));
assertFalse(SimilarAsinTaskService.isAsinNotFound("\u4e0a\u4f20\u5931\u8d25"));
}
@Test
void fileBuildProgressIsTerminalOnlyAfterTaskAndStageComplete() {
assertTrue(SimilarAsinTaskService.isTerminalFileBuildProgress("SUCCESS", 3, 3));
@@ -0,0 +1,318 @@
package com.nanri.aiimage.modules.similarasin.service;
import cn.hutool.crypto.digest.DigestUtil;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.SimilarAsinProperties;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.similarasin.mapper.SimilarAsinFilterConditionMapper;
import com.nanri.aiimage.modules.similarasin.model.dto.SimilarAsinSubmitResultRequest;
import com.nanri.aiimage.modules.similarasin.util.SimilarAsinImageEmbedder;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskProgressSnapshotService;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.ArgumentCaptor;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.test.util.ReflectionTestUtils;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.TransactionDefinition;
import org.springframework.transaction.TransactionStatus;
import org.springframework.transaction.annotation.Transactional;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-125similarasin /result 提交事务边界守门。
* 纯计算(分组/校验/payload 哈希)在 prepareSubmittedChunk 中、事务开始前完成;
* 落库(chunk insert / scope upsert)仍在事务内;回滚语义不变。
*/
@ExtendWith(MockitoExtension.class)
class SimilarAsinTaskServiceTxBoundaryTest {
private static final Long TASK_ID = 21879L;
private static final String PARSED_POINTER = "rustfs:task-parsed/similar-asin/21879/payload.json";
private static final String CHUNK_POINTER = "rustfs:task-chunk/similar-asin/21879/chunk.json";
private static final String STORED_CHUNK_POINTER = "\"" + CHUNK_POINTER + "\"";
@Mock private LocalFileStorageService localFileStorageService;
@Mock private OssStorageService ossStorageService;
@Mock private StorageProperties storageProperties;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private SimilarAsinFilterConditionMapper filterConditionMapper;
@Spy private ObjectMapper objectMapper = new ObjectMapper();
@Mock private SimilarAsinTaskCacheService taskCacheService;
@Mock private SimilarAsinProperties properties;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private TaskProgressSnapshotService taskProgressSnapshotService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private PlatformTransactionManager transactionManager;
@Mock private DistributedJobLockService distributedJobLockService;
@Mock private InstanceMetadata instanceMetadata;
@Mock private SimilarAsinImageEmbedder imageEmbedder;
@Mock private SimilarAsinImagePrefetchService imagePrefetchService;
@Mock private TransactionStatus transactionStatus;
@InjectMocks private SimilarAsinTaskService service;
private final AtomicBoolean transactionActive = new AtomicBoolean();
private final AtomicReference<TaskChunkEntity> insertedChunk = new AtomicReference<>();
private final AtomicReference<String> storedPayloadJson = new AtomicReference<>();
private final AtomicBoolean storageOutsideTx = new AtomicBoolean(true);
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskChunkEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskScopeStateEntity.class);
}
@BeforeEach
void setUpTransactionAndLock() {
lenient().when(instanceMetadata.getInstanceId()).thenReturn("instance-a");
lenient().when(taskDistributedLockService.acquire(
eq(SimilarAsinTaskService.MODULE_TYPE), anyLong(), any(Duration.class), eq(10_000L)))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class));
lenient().when(transactionManager.getTransaction(any(TransactionDefinition.class)))
.thenAnswer(invocation -> {
transactionActive.set(true);
return transactionStatus;
});
lenient().doAnswer(invocation -> {
transactionActive.set(false);
return null;
}).when(transactionManager).commit(transactionStatus);
lenient().doAnswer(invocation -> {
transactionActive.set(false);
return null;
}).when(transactionManager).rollback(transactionStatus);
lenient().when(taskChunkMapper.selectList(any())).thenReturn(List.of());
lenient().when(fileResultMapper.selectList(any())).thenReturn(List.of());
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(1L);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(null);
lenient().when(transientPayloadStorageService.storeChunkPayloadVersioned(
eq(SimilarAsinTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString()))
.thenAnswer(invocation -> {
storedPayloadJson.set(invocation.getArgument(4));
storageOutsideTx.set(!transactionActive.get());
return STORED_CHUNK_POINTER;
});
lenient().when(transientPayloadStorageService.wasLastStoreLocalFallback()).thenReturn(false);
lenient().doAnswer(invocation -> {
TaskChunkEntity chunk = invocation.getArgument(0);
assertTrue(transactionActive.get(), "chunk 落库必须在事务内");
chunk.setId(301L);
insertedChunk.set(chunk);
return 1;
}).when(taskChunkMapper).insert(any(TaskChunkEntity.class));
lenient().doAnswer(invocation -> {
TaskScopeStateEntity scope = invocation.getArgument(0);
scope.setId(401L);
return 1;
}).when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
}
@AfterEach
void shutdownExecutors() {
service.shutdownAssembleExecutor();
}
@Test
void payloadHashIsPrecomputedInPrepareBeforeTransaction() throws Exception {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
Object prepared = ReflectionTestUtils.invokeMethod(
service, "prepareSubmittedChunk", TASK_ID, request(false));
assertEquals(DigestUtil.sha256Hex(storedPayloadJson.get()),
ReflectionTestUtils.getField(prepared, "payloadHash"),
"prepare 阶段必须产出预计算的 payload 哈希(事务开始前可用)");
// prepare 是纯计算:不触发任何落库
verify(taskChunkMapper, never()).insert(any(TaskChunkEntity.class));
verify(taskScopeStateMapper, never()).insert(any(TaskScopeStateEntity.class));
}
@Test
void pureComputeStaysOutsideTransactionAndPersistenceInside() throws Exception {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
TaskChunkEntity chunk = insertedChunk.get();
assertEquals(DigestUtil.sha256Hex(storedPayloadJson.get()), chunk.getPayloadHash(),
"payload 哈希必须在 prepare 阶段预计算,落库行直接使用");
assertTrue(storageOutsideTx.get(), "prepare 阶段(哈希计算所在)不得处于事务内");
// 事务必须晚于 prepare 开始,chunk 落库在事务内
var order = inOrder(transientPayloadStorageService, transactionManager, taskChunkMapper);
order.verify(transientPayloadStorageService).storeChunkPayloadVersioned(
eq(SimilarAsinTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString());
order.verify(transactionManager).getTransaction(any(TransactionDefinition.class));
order.verify(taskChunkMapper).insert(any(TaskChunkEntity.class));
}
@Test
void precomputedHashMatchesComputedFromSamePayloadJson() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
String payloadJson = storedPayloadJson.get();
assertEquals(DigestUtil.sha256Hex(payloadJson), insertedChunk.get().getPayloadHash());
}
@Test
void persistenceStaysInsideTransaction() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
verify(taskChunkMapper).insert(any(TaskChunkEntity.class));
verify(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
verify(transactionManager).commit(transactionStatus);
verify(transactionManager, never()).rollback(transactionStatus);
}
@Test
void persistFailureRollsBackAndComputeHasNoSideEffect() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
org.mockito.Mockito.doThrow(new IllegalStateException("scope upsert failed"))
.when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
assertThrows(IllegalStateException.class, () -> service.submitResult(TASK_ID, request(false)));
verify(transactionManager).rollback(transactionStatus);
verify(transactionManager, never()).commit(transactionStatus);
verify(taskCacheService, never()).touchTaskHeartbeat(TASK_ID);
assertTrue(storageOutsideTx.get(), "计算阶段先于事务执行,回滚不影响已完成的 prepare");
}
@Test
void repeatedSubmissionComputesIdenticalHash() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
String first = insertedChunk.get().getPayloadHash();
service.submitResult(TASK_ID, request(false));
assertEquals(first, insertedChunk.get().getPayloadHash(), "相同入参重复提交哈希必须一致");
verify(taskChunkMapper, times(2)).insert(any(TaskChunkEntity.class));
}
@Test
void repeatedNonFinalSubmissionStaysSafe() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
service.submitResult(TASK_ID, request(false));
verify(taskCacheService, times(2)).touchTaskHeartbeat(TASK_ID);
assertEquals(DigestUtil.sha256Hex(storedPayloadJson.get()), insertedChunk.get().getPayloadHash());
}
@Test
void movedComputationMethodsCarryNoTransactionAnnotation() throws Exception {
assertNull(SimilarAsinTaskService.class
.getDeclaredMethod("prepareSubmittedChunk", Long.class, SimilarAsinSubmitResultRequest.class)
.getAnnotation(Transactional.class),
"prepareSubmittedChunk 不得带 @Transactional");
boolean persistUnannotated = java.util.Arrays.stream(SimilarAsinTaskService.class.getDeclaredMethods())
.filter(method -> method.getName().equals("persistSubmittedChunk"))
.allMatch(method -> method.getAnnotation(Transactional.class) == null);
assertTrue(persistUnannotated, "persistSubmittedChunk 保持无注解(事务由调用方 inNewTransaction 控制)");
}
@Test
void insertedChunkSnapshotUnchanged() {
FileTaskEntity task = runningTask("instance-a");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
service.submitResult(TASK_ID, request(false));
TaskChunkEntity chunk = insertedChunk.get();
assertEquals(TASK_ID, chunk.getTaskId());
assertEquals(SimilarAsinTaskService.MODULE_TYPE, chunk.getModuleType());
assertEquals("similar-asin-21879", chunk.getScopeKey());
assertEquals(DigestUtil.sha256Hex("similar-asin-21879"), chunk.getScopeHash());
assertEquals(0, chunk.getChunkIndex());
assertEquals(1, chunk.getChunkTotal());
assertEquals(STORED_CHUNK_POINTER, chunk.getPayloadJson(), "payloadJson 存存储指针(现状不变)");
assertEquals(DigestUtil.sha256Hex(storedPayloadJson.get()), chunk.getPayloadHash());
org.junit.jupiter.api.Assertions.assertNotNull(chunk.getCreatedAt());
org.junit.jupiter.api.Assertions.assertNotNull(chunk.getUpdatedAt());
}
private SimilarAsinSubmitResultRequest request(boolean done) {
SimilarAsinSubmitResultRequest request = new SimilarAsinSubmitResultRequest();
request.setSubmissionId("similar-asin-21879");
request.setChunkIndex(0);
request.setChunkTotal(1);
request.setDone(done);
return request;
}
private FileTaskEntity runningTask(String owner) {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType(SimilarAsinTaskService.MODULE_TYPE);
task.setStatus("RUNNING");
task.setUserId(7L);
String resultJson = "{\"parsedPayloadRef\":\"" + PARSED_POINTER + "\"";
if (owner != null) {
resultJson += ",\"ownerInstanceId\":\"" + owner + "\"";
}
task.setResultJson(resultJson + "}");
return task;
}
}
@@ -0,0 +1,168 @@
package com.nanri.aiimage.modules.similarasin.service.support;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.similarasin.model.vo.SimilarAsinHistoryItemVo;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import com.nanri.aiimage.modules.task.service.TaskProgressSnapshotService;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import java.util.List;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.when;
/**
* task-142fileReady/fileStatus 语义契约(plan 08)。
* fileReady = resultFileUrl 非空;无 job 时 fileStatus = ready?SUCCESS:null
* 有 job 时 fileStatus = job.statusfileError = job.errorMessage。
*/
@ExtendWith(MockitoExtension.class)
class FilePhaseSemanticsTest {
private static final Long TASK_ID = 1212L;
private static final Long RESULT_ID = 1213L;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private TaskProgressSnapshotService taskProgressSnapshotService;
@Mock private OssStorageService ossStorageService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private ObjectMapper objectMapper;
private SimilarAsinHistoryAssembler assembler;
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileResultEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskScopeStateEntity.class);
}
@BeforeEach
void setUp() {
assembler = new SimilarAsinHistoryAssembler(taskScopeStateMapper, taskChunkMapper, fileTaskMapper,
taskProgressSnapshotService, ossStorageService, transientPayloadStorageService, objectMapper);
lenient().when(taskScopeStateMapper.selectCount(any())).thenReturn(0L);
lenient().when(taskScopeStateMapper.selectList(any())).thenReturn(List.of());
lenient().when(taskProgressSnapshotService.find(any(), any())).thenReturn(null);
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(0L);
lenient().when(ossStorageService.generateFreshDownloadUrl(any())).thenReturn("https://dl.example/x.xlsx");
}
private SimilarAsinHistoryItemVo build(FileResultEntity row, TaskFileJobEntity job) {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType("SIMILAR_ASIN");
task.setStatus("RUNNING");
return assembler.buildHistoryItems(List.of(row), Map.of(TASK_ID, task),
job == null ? Map.of() : Map.of(RESULT_ID, job)).getFirst();
}
private FileResultEntity row(String url) {
FileResultEntity row = new FileResultEntity();
row.setId(RESULT_ID);
row.setTaskId(TASK_ID);
row.setSourceFilename("source.xlsx");
row.setResultFileUrl(url);
row.setSuccess(0);
return row;
}
private TaskFileJobEntity job(String status, String error) {
TaskFileJobEntity job = new TaskFileJobEntity();
job.setId(901L);
job.setTaskId(TASK_ID);
job.setResultId(RESULT_ID);
job.setStatus(status);
job.setErrorMessage(error);
return job;
}
@Test
void readyTrueWhenUrlPresent() {
SimilarAsinHistoryItemVo vo = build(row("oss://result/a.xlsx"), null);
assertEquals(Boolean.TRUE, vo.getFileReady());
}
@Test
void readyFalseWhenUrlBlank() {
SimilarAsinHistoryItemVo vo = build(row(null), null);
assertFalse(Boolean.TRUE.equals(vo.getFileReady()));
}
@Test
void noJobWithReadyMapsToSuccessStatus() {
SimilarAsinHistoryItemVo vo = build(row("oss://result/a.xlsx"), null);
assertEquals("SUCCESS", vo.getFileStatus(), "无 job 且 ready 时 fileStatus=SUCCESS");
}
@Test
void noJobWithoutReadyMapsToNullStatus() {
SimilarAsinHistoryItemVo vo = build(row(null), null);
assertNull(vo.getFileStatus(), "无 job 且未 ready 时 fileStatus=null");
assertNull(vo.getFileJobId());
}
@Test
void jobStatusMappedDirectly() {
SimilarAsinHistoryItemVo vo = build(row("oss://result/a.xlsx"), job("RUNNING", null));
assertEquals("RUNNING", vo.getFileStatus(), "有 job 时 fileStatus=job.status");
assertEquals(901L, vo.getFileJobId());
}
@Test
void jobErrorMappedDirectly() {
SimilarAsinHistoryItemVo vo = build(row(null), job("FAILED", "组装失败: 超时"));
assertEquals("组装失败: 超时", vo.getFileError(), "fileError=job.errorMessage");
assertEquals("FAILED", vo.getFileStatus());
}
@Test
void blankWhitespaceUrlIsNotReady() {
SimilarAsinHistoryItemVo vo = build(row(" "), null);
assertFalse(Boolean.TRUE.equals(vo.getFileReady()), "空白字符串不算 ready");
assertNull(vo.getFileStatus());
}
@Test
void semanticsFrozenCombined() {
// 组合快照:URL + SUCCESS job → ready=true、status=SUCCESS、error 映射、jobId 携带
SimilarAsinHistoryItemVo vo = build(row("oss://result/a.xlsx"), job("SUCCESS", null));
assertEquals(Boolean.TRUE, vo.getFileReady());
assertEquals("SUCCESS", vo.getFileStatus());
assertNull(vo.getFileError());
assertEquals(901L, vo.getFileJobId());
assertTrue(vo.getFileReady());
}
}
@@ -0,0 +1,212 @@
package com.nanri.aiimage.modules.similarasin.service.support;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.similarasin.model.vo.SimilarAsinHistoryItemVo;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskProgressSnapshotEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import com.nanri.aiimage.modules.task.service.TaskProgressSnapshotService;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import java.time.LocalDateTime;
import java.util.List;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.when;
/**
* task-141similarasin file 进度合成逻辑单测(plan 08)。
* attachFileProgress 三路径:LLM 阶段(completed/pending>0)、Python 上传阶段、
* snapshot 百分比(上传完成无 LLM);fileReady=true 时恒 100%;边界安全。
*/
@ExtendWith(MockitoExtension.class)
class FileProgressSynthesisTest {
private static final Long TASK_ID = 2222L;
private static final Long RESULT_ID = 2223L;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private TaskProgressSnapshotService taskProgressSnapshotService;
@Mock private OssStorageService ossStorageService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private ObjectMapper objectMapper;
private SimilarAsinHistoryAssembler assembler;
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileResultEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskScopeStateEntity.class);
}
@BeforeEach
void setUp() {
assembler = new SimilarAsinHistoryAssembler(taskScopeStateMapper, taskChunkMapper, fileTaskMapper,
taskProgressSnapshotService, ossStorageService, transientPayloadStorageService, objectMapper);
lenient().when(taskScopeStateMapper.selectCount(any())).thenReturn(0L);
lenient().when(taskScopeStateMapper.selectList(any())).thenReturn(List.of());
lenient().when(taskProgressSnapshotService.find(any(), any())).thenReturn(null);
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(0L);
lenient().when(ossStorageService.generateFreshDownloadUrl(any())).thenReturn("https://dl.example/x.xlsx");
}
private SimilarAsinHistoryItemVo build(Runnable configure, FileResultEntity row, FileTaskEntity task,
TaskFileJobEntity job) {
configure.run();
return assembler.buildHistoryItems(List.of(row),
task == null ? Map.of() : Map.of(TASK_ID, task),
job == null ? Map.of() : Map.of(RESULT_ID, job)).getFirst();
}
private FileResultEntity resultRow() {
FileResultEntity row = new FileResultEntity();
row.setId(RESULT_ID);
row.setTaskId(TASK_ID);
row.setSourceFilename("source.xlsx");
row.setResultFileUrl(null);
row.setSuccess(0);
return row;
}
private FileTaskEntity runningTask() {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType("SIMILAR_ASIN");
task.setStatus("RUNNING");
return task;
}
@Test
void fileReadyPathIsAlwaysOneHundredPercent() {
FileResultEntity row = resultRow();
row.setResultFileUrl("oss://result/source-result.xlsx");
SimilarAsinHistoryItemVo vo = build(() -> {
}, row, runningTask(), null);
assertEquals(100, vo.getFileProgressPercent());
assertEquals(1, vo.getFileProgressCurrent());
assertEquals(1, vo.getFileProgressTotal());
assertEquals("结果文件已生成", vo.getFileProgressMessage());
assertEquals(Boolean.TRUE, vo.getFileReady());
}
@Test
void llmProgressPathWhenCompletedExists() {
when(taskScopeStateMapper.selectCount(any())).thenReturn(2L, 0L);
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(1L);
SimilarAsinHistoryItemVo vo = build(() -> {
}, resultRow(), runningTask(), null);
assertNotNull(vo.getFileProgressPercent(), "LLM 阶段必须合成进度");
assertEquals(2, vo.getFileProgressCurrent());
}
@Test
void pythonUploadPathWhenNoLlmStates() {
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(1L);
SimilarAsinHistoryItemVo vo = build(() -> {
}, resultRow(), runningTask(), null);
assertNotNull(vo.getFileProgressPercent(), "Python 上传阶段必须合成进度");
assertNotNull(vo.getFileProgressMessage());
}
@Test
void snapshotPercentPathWhenUploadComplete() {
TaskScopeStateEntity scope = new TaskScopeStateEntity();
scope.setTaskId(TASK_ID);
scope.setCompleted(1);
when(taskScopeStateMapper.selectList(any())).thenReturn(List.of(scope));
TaskProgressSnapshotEntity snapshot = new TaskProgressSnapshotEntity();
snapshot.setTotalCount(10);
snapshot.setSuccessCount(5);
snapshot.setMessage("组装中");
when(taskProgressSnapshotService.find(eq(TASK_ID), eq("SIMILAR_ASIN"))).thenReturn(snapshot);
SimilarAsinHistoryItemVo vo = build(() -> {
}, resultRow(), runningTask(), null);
assertNotNull(vo.getFileProgressPercent(), "上传完成无 LLM 时走 snapshot 百分比");
assertEquals(5, vo.getFileProgressCurrent());
assertEquals(10, vo.getFileProgressTotal());
}
@Test
void pendingOnlyStillUsesLlmPath() {
when(taskScopeStateMapper.selectCount(any())).thenReturn(0L, 2L);
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(1L);
SimilarAsinHistoryItemVo vo = build(() -> {
}, resultRow(), runningTask(), null);
assertNotNull(vo.getFileProgressPercent(), "pending>0 也走 LLM 路径");
}
@Test
void uploadCompleteWithoutLlmOrSnapshotSetsNoProgress() {
TaskScopeStateEntity scope = new TaskScopeStateEntity();
scope.setTaskId(TASK_ID);
scope.setCompleted(1);
when(taskScopeStateMapper.selectList(any())).thenReturn(List.of(scope));
SimilarAsinHistoryItemVo vo = build(() -> {
}, resultRow(), runningTask(), null);
assertNull(vo.getFileProgressPercent(), "无 LLM 无 snapshot 不合成进度");
}
@Test
void nonRunningTaskWithoutProgressSkipsSynthesis() {
FileTaskEntity failed = runningTask();
failed.setStatus("FAILED");
SimilarAsinHistoryItemVo vo = build(() -> {
}, resultRow(), failed, null);
assertNull(vo.getFileProgressPercent(), "非 RUNNING 任务不合成上传进度");
}
@Test
void zeroTotalSnapshotIsDivisionSafe() {
TaskScopeStateEntity scope = new TaskScopeStateEntity();
scope.setTaskId(TASK_ID);
scope.setCompleted(1);
when(taskScopeStateMapper.selectList(any())).thenReturn(List.of(scope));
TaskProgressSnapshotEntity snapshot = new TaskProgressSnapshotEntity();
snapshot.setTotalCount(0);
snapshot.setSuccessCount(0);
when(taskProgressSnapshotService.find(eq(TASK_ID), eq("SIMILAR_ASIN"))).thenReturn(snapshot);
SimilarAsinHistoryItemVo vo = build(() -> {
}, resultRow(), runningTask(), null);
assertNull(vo.getFileProgressPercent(), "total=0 时不除零不合成");
}
}
@@ -0,0 +1,283 @@
package com.nanri.aiimage.modules.task.contract;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.SimilarAsinProperties;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.similarasin.mapper.SimilarAsinFilterConditionMapper;
import com.nanri.aiimage.modules.similarasin.model.dto.SimilarAsinSubmitResultRequest;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinImagePrefetchService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinLlmService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskCacheService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskService;
import com.nanri.aiimage.modules.similarasin.util.SimilarAsinImageEmbedder;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskProgressSnapshotService;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.dao.DuplicateKeyException;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.TransactionDefinition;
import org.springframework.transaction.TransactionStatus;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-136:事务外清理异常不阻断契约(spec 07 §2)。
* afterCommit 语义的清理抛异常:记录日志、响应不受影响、不进入重试循环、
* 事务已提交、下次提交仍可再次清理、部分失败不影响后续流程。
*/
@ExtendWith(MockitoExtension.class)
class CleanupErrorContractTest {
private static final Long TASK_ID = 5555L;
private static final String PARSED_POINTER = "rustfs:task-parsed/similar-asin/5555/payload.json";
private static final String CHUNK_POINTER = "rustfs:task-chunk/similar-asin/5555/chunk.json";
private static final String STORED_CHUNK_POINTER = "\"" + CHUNK_POINTER + "\"";
@Mock private LocalFileStorageService localFileStorageService;
@Mock private OssStorageService ossStorageService;
@Mock private StorageProperties storageProperties;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private SimilarAsinFilterConditionMapper filterConditionMapper;
@Spy private ObjectMapper objectMapper = new ObjectMapper();
@Mock private SimilarAsinLlmService similarAsinLlmService;
@Mock private SimilarAsinTaskCacheService taskCacheService;
@Mock private SimilarAsinProperties properties;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private TaskProgressSnapshotService taskProgressSnapshotService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private PlatformTransactionManager transactionManager;
@Mock private DistributedJobLockService distributedJobLockService;
@Mock private InstanceMetadata instanceMetadata;
@Mock private SimilarAsinImageEmbedder imageEmbedder;
@Mock private SimilarAsinImagePrefetchService imagePrefetchService;
@Mock private TransactionStatus transactionStatus;
@InjectMocks private SimilarAsinTaskService service;
private final AtomicBoolean transactionActive = new AtomicBoolean();
private final AtomicInteger cleanupAttempts = new AtomicInteger();
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskScopeStateEntity.class);
}
@BeforeEach
void setUp() {
lenient().when(instanceMetadata.getInstanceId()).thenReturn("instance-a");
lenient().when(taskDistributedLockService.acquire(
eq(SimilarAsinTaskService.MODULE_TYPE), anyLong(), any(Duration.class), eq(10_000L)))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class));
lenient().when(transactionManager.getTransaction(any(TransactionDefinition.class)))
.thenAnswer(invocation -> {
transactionActive.set(true);
return transactionStatus;
});
lenient().doAnswer(invocation -> {
transactionActive.set(false);
return null;
}).when(transactionManager).commit(transactionStatus);
lenient().doAnswer(invocation -> {
transactionActive.set(false);
return null;
}).when(transactionManager).rollback(transactionStatus);
lenient().when(taskChunkMapper.selectList(any())).thenReturn(List.of());
lenient().when(fileResultMapper.selectList(any())).thenReturn(List.of());
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(0L);
lenient().when(taskScopeStateMapper.selectCount(any())).thenReturn(0L);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(null);
lenient().when(transientPayloadStorageService.storeChunkPayloadVersioned(
eq(SimilarAsinTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString()))
.thenReturn(STORED_CHUNK_POINTER);
lenient().when(transientPayloadStorageService.wasLastStoreLocalFallback()).thenReturn(false);
lenient().when(transientPayloadStorageService.extractPointer(STORED_CHUNK_POINTER))
.thenReturn(CHUNK_POINTER);
lenient().doAnswer(invocation -> {
TaskScopeStateEntity scope = invocation.getArgument(0);
scope.setId(401L);
return 1;
}).when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
lenient().doThrow(new DuplicateKeyException("duplicate chunk"))
.when(taskChunkMapper).insert(any(com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity.class));
lenient().doAnswer(invocation -> {
cleanupAttempts.incrementAndGet();
return null;
}).when(transientPayloadStorageService).deletePayloadIfPresent(anyString());
}
@AfterEach
void shutdownExecutors() {
service.shutdownAssembleExecutor();
}
private void configureCleanupFailure() {
doThrow(new IllegalStateException("cleanup failed")).when(transientPayloadStorageService)
.deletePayloadIfPresent(anyString());
}
@Test
void cleanupErrorDoesNotBlockResponse() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
configureCleanupFailure();
service.submitResult(TASK_ID, request());
verify(taskCacheService).touchTaskHeartbeat(TASK_ID);
verify(transientPayloadStorageService).deletePayloadIfPresent(STORED_CHUNK_POINTER);
}
@Test
void cleanupErrorNoRetryLoop() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
configureCleanupFailure();
service.submitResult(TASK_ID, request());
verify(transientPayloadStorageService, times(1)).deletePayloadIfPresent(STORED_CHUNK_POINTER);
}
@Test
void cleanupRunsAfterTransactionCommitted() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
service.submitResult(TASK_ID, request());
var order = inOrder(transactionManager, transientPayloadStorageService);
order.verify(transactionManager).commit(transactionStatus);
order.verify(transientPayloadStorageService).deletePayloadIfPresent(STORED_CHUNK_POINTER);
}
@Test
void cleanupObservesTransactionFinished() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
AtomicBoolean cleanupOutsideTx = new AtomicBoolean();
doAnswer(invocation -> {
cleanupOutsideTx.set(!transactionActive.get());
return null;
}).when(transientPayloadStorageService).deletePayloadIfPresent(anyString());
service.submitResult(TASK_ID, request());
assertTrue(cleanupOutsideTx.get(), "清理必须在事务结束后(等价 afterCommit");
}
@Test
void cleanupFailureKeepsPayloadForNextSubmission() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
configureCleanupFailure();
service.submitResult(TASK_ID, request());
// 恢复后再次提交,同一 payload 再次进入清理(失败可重试,payload 未丢)
service.submitResult(TASK_ID, request());
verify(transientPayloadStorageService, times(2)).deletePayloadIfPresent(STORED_CHUNK_POINTER);
}
@Test
void partialCleanupFailureStillFinishesFlow() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
AtomicInteger failures = new AtomicInteger();
doAnswer(invocation -> {
if (failures.getAndIncrement() == 0) {
throw new IllegalStateException("first cleanup failed");
}
return null;
}).when(transientPayloadStorageService).deletePayloadIfPresent(anyString());
service.submitResult(TASK_ID, request());
verify(taskCacheService).touchTaskHeartbeat(TASK_ID);
verify(transientPayloadStorageService).deletePayloadIfPresent(STORED_CHUNK_POINTER);
}
@Test
void cleanupFailureDoesNotAffectCommittedState() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
configureCleanupFailure();
service.submitResult(TASK_ID, request());
verify(transactionManager).commit(transactionStatus);
verify(transactionManager, org.mockito.Mockito.never()).rollback(transactionStatus);
}
@Test
void cleanupErrorIntegration() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
configureCleanupFailure();
// 全链路:提交成功 → 清理失败被吞 → 心跳/后续流程正常
service.submitResult(TASK_ID, request());
verify(transactionManager).commit(transactionStatus);
verify(transientPayloadStorageService).deletePayloadIfPresent(STORED_CHUNK_POINTER);
verify(taskCacheService).touchTaskHeartbeat(TASK_ID);
}
private SimilarAsinSubmitResultRequest request() {
SimilarAsinSubmitResultRequest request = new SimilarAsinSubmitResultRequest();
request.setSubmissionId("similar-asin-5555");
request.setChunkIndex(0);
request.setChunkTotal(1);
request.setDone(false);
return request;
}
private FileTaskEntity runningTask() {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType(SimilarAsinTaskService.MODULE_TYPE);
task.setStatus("RUNNING");
task.setUserId(7L);
task.setResultJson("{\"parsedPayloadRef\":\"" + PARSED_POINTER + "\",\"ownerInstanceId\":\"instance-a\"}");
return task;
}
}
@@ -0,0 +1,102 @@
package com.nanri.aiimage.modules.task.contract;
import com.nanri.aiimage.modules.collectdata.model.vo.CollectDataHistoryItemVo;
import org.junit.jupiter.api.Test;
import java.lang.reflect.Field;
import java.util.Arrays;
import java.util.Map;
import java.util.stream.Collectors;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-146collectdata 自洽行为契约(plan 08)。
* 后端 history VO 不返回 file 阶段字段(fileStatus/fileReady/fileError);
* 下载语义由 success + downloadUrl 推导(任务 SUCCESS 且有结果文件才可下载);
* 进度展示字段自洽。
*/
class CollectDataSelfConsistencyTest {
private static Map<String, Class<?>> fieldsOf(Class<?> type) {
return Arrays.stream(type.getDeclaredFields())
.filter(field -> !java.lang.reflect.Modifier.isStatic(field.getModifiers()))
.collect(Collectors.toMap(Field::getName, Field::getType));
}
@Test
void backendHasNoFilePhaseFields() {
Map<String, Class<?>> fields = fieldsOf(CollectDataHistoryItemVo.class);
assertFalse(fields.containsKey("fileStatus"), "collectdata 不返回 fileStatus");
assertFalse(fields.containsKey("fileReady"), "collectdata 不返回 fileReady");
assertFalse(fields.containsKey("fileError"), "collectdata 不返回 fileError");
assertFalse(fields.containsKey("fileProgressPercent"), "collectdata 不返回 fileProgressPercent");
}
@Test
void downloadFieldsPresent() {
Map<String, Class<?>> fields = fieldsOf(CollectDataHistoryItemVo.class);
assertEquals(String.class, fields.get("downloadUrl"), "downloadUrl 字段存在");
assertEquals(Boolean.class, fields.get("success"), "success 字段存在");
}
@Test
void statusFlowFieldsPresent() {
Map<String, Class<?>> fields = fieldsOf(CollectDataHistoryItemVo.class);
assertEquals(String.class, fields.get("taskStatus"), "taskStatus 字段存在");
assertEquals(String.class, fields.get("error"), "error 字段存在");
assertEquals(Long.class, fields.get("taskId"));
assertEquals(Long.class, fields.get("resultId"));
}
@Test
void progressDisplayFieldsPresent() {
Map<String, Class<?>> fields = fieldsOf(CollectDataHistoryItemVo.class);
assertEquals(Integer.class, fields.get("dedupeFilteredCount"), "进度展示字段存在");
assertEquals(Integer.class, fields.get("invalidFilteredCount"));
assertEquals(Integer.class, fields.get("brandRejectedCount"));
assertEquals(Integer.class, fields.get("finalRowCount"));
assertEquals(Integer.class, fields.get("rowCount"));
}
@Test
void successAndUrlAreTheOnlyDownloadSignals() {
Map<String, Class<?>> fields = fieldsOf(CollectDataHistoryItemVo.class);
// 可下载信号仅 success + downloadUrl(无 fileReady 类独立标志)
assertTrue(fields.containsKey("success"));
assertTrue(fields.containsKey("downloadUrl"));
assertFalse(fields.containsKey("fileReady"));
}
@Test
void fileFieldsAbsentEvenInSharedLightVoContract() {
Map<String, Class<?>> fields = fieldsOf(CollectDataHistoryItemVo.class);
assertFalse(fields.containsKey("fileJobId"), "collectdata 无 fileJobId");
assertFalse(fields.containsKey("fileJobStatus"), "collectdata 无 fileJobStatus");
}
@Test
void historyIdentityFieldsPresent() {
Map<String, Class<?>> fields = fieldsOf(CollectDataHistoryItemVo.class);
assertEquals(String.class, fields.get("sourceFilename"));
assertEquals(String.class, fields.get("resultFilename"));
assertEquals(String.class, fields.get("taskNo"));
}
@Test
void contractFrozenFieldSet() {
Map<String, Class<?>> fields = fieldsOf(CollectDataHistoryItemVo.class);
// 快照:字段集合固定(下载用 success+downloadUrl,无 file 阶段字段)
java.util.Set<String> expected = java.util.Set.of(
"resultId", "taskId", "taskNo", "sourceFilename", "resultFilename", "downloadUrl",
"taskStatus", "success", "error", "rowCount", "dedupeFilteredCount",
"invalidFilteredCount", "brandRejectedCount", "finalRowCount",
"taskType", "progressPercent", "totalRows", "receivedRows", "processedRows",
"collectStage", "currentKeyword", "searchCurrentPage", "searchTotalPages",
"detailProcessedAsins", "detailTotalAsins", "filters",
"createdAt", "startedAt", "finishedAt");
assertEquals(expected, fields.keySet(), "collectdata history VO 字段集合冻结");
}
}
@@ -0,0 +1,282 @@
package com.nanri.aiimage.modules.task.contract;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.SimilarAsinProperties;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.similarasin.mapper.SimilarAsinFilterConditionMapper;
import com.nanri.aiimage.modules.similarasin.model.dto.SimilarAsinSubmitResultRequest;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinImagePrefetchService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinLlmService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskCacheService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskService;
import com.nanri.aiimage.modules.similarasin.util.SimilarAsinImageEmbedder;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskProgressSnapshotService;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.TransactionDefinition;
import org.springframework.transaction.TransactionStatus;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-134done=true 封口时机硬约束契约(spec 07 §2)。
* done=true 的最后一批:分片先落库 → 强制 finalize(处理剩余 + 生成结果文件);
* 重复 done 安全不重复生成;done=false 只触心跳不封口。similarasin 路径统一守门。
*/
@ExtendWith(MockitoExtension.class)
class DoneFinalizationContractTest {
private static final Long TASK_ID = 7777L;
private static final String PARSED_POINTER = "rustfs:task-parsed/similar-asin/7777/payload.json";
private static final String CHUNK_POINTER = "rustfs:task-chunk/similar-asin/7777/chunk.json";
private static final String STORED_CHUNK_POINTER = "\"" + CHUNK_POINTER + "\"";
@Mock private LocalFileStorageService localFileStorageService;
@Mock private OssStorageService ossStorageService;
@Mock private StorageProperties storageProperties;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private SimilarAsinFilterConditionMapper filterConditionMapper;
@Spy private ObjectMapper objectMapper = new ObjectMapper();
@Mock private SimilarAsinLlmService similarAsinLlmService;
@Mock private SimilarAsinTaskCacheService taskCacheService;
@Mock private SimilarAsinProperties properties;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private TaskProgressSnapshotService taskProgressSnapshotService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private PlatformTransactionManager transactionManager;
@Mock private DistributedJobLockService distributedJobLockService;
@Mock private InstanceMetadata instanceMetadata;
@Mock private SimilarAsinImageEmbedder imageEmbedder;
@Mock private SimilarAsinImagePrefetchService imagePrefetchService;
@Mock private TransactionStatus transactionStatus;
@InjectMocks private SimilarAsinTaskService service;
private final AtomicReference<FileTaskEntity> taskState = new AtomicReference<>();
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, FileResultEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskScopeStateEntity.class);
}
@BeforeEach
void setUp() {
lenient().when(instanceMetadata.getInstanceId()).thenReturn("instance-a");
lenient().when(taskDistributedLockService.acquire(
eq(SimilarAsinTaskService.MODULE_TYPE), anyLong(), any(Duration.class), eq(10_000L)))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class));
lenient().when(transactionManager.getTransaction(any(TransactionDefinition.class)))
.thenReturn(transactionStatus);
lenient().doAnswer(invocation -> null).when(transactionManager).commit(transactionStatus);
lenient().doAnswer(invocation -> null).when(transactionManager).rollback(transactionStatus);
lenient().when(taskChunkMapper.selectList(any())).thenReturn(List.of());
lenient().when(fileResultMapper.selectList(any())).thenReturn(List.of());
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(1L);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(null);
lenient().when(transientPayloadStorageService.storeChunkPayloadVersioned(
eq(SimilarAsinTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString()))
.thenReturn(STORED_CHUNK_POINTER);
lenient().when(transientPayloadStorageService.wasLastStoreLocalFallback()).thenReturn(false);
lenient().when(transientPayloadStorageService.resolvePayload(anyString(), anyString()))
.thenReturn("{\"allItems\":[{},{}],\"items\":[{},{}],\"sourceFiles\":[]}");
lenient().doAnswer(invocation -> {
com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity chunk = invocation.getArgument(0);
chunk.setId(301L);
return 1;
}).when(taskChunkMapper).insert(any(com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity.class));
lenient().doAnswer(invocation -> {
TaskScopeStateEntity scope = invocation.getArgument(0);
scope.setId(401L);
return 1;
}).when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
lenient().when(fileTaskMapper.updateById(any(FileTaskEntity.class))).thenReturn(1);
lenient().when(fileResultMapper.updateById(any(FileResultEntity.class))).thenReturn(1);
lenient().when(fileResultMapper.insert(any(FileResultEntity.class))).thenAnswer(invocation -> {
FileResultEntity result = invocation.getArgument(0);
result.setId(601L);
return 1;
});
lenient().when(taskFileJobService.enqueueAssembleResult(anyLong(), anyString(), anyLong(), anyString()))
.thenReturn(null);
}
@AfterEach
void shutdownExecutors() {
service.shutdownAssembleExecutor();
}
private FileTaskEntity runningTask() {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType(SimilarAsinTaskService.MODULE_TYPE);
task.setStatus("RUNNING");
task.setUserId(7L);
task.setResultJson("{\"parsedPayloadRef\":\"" + PARSED_POINTER + "\",\"ownerInstanceId\":\"instance-a\"}");
return task;
}
private SimilarAsinSubmitResultRequest request(boolean done) {
SimilarAsinSubmitResultRequest request = new SimilarAsinSubmitResultRequest();
request.setSubmissionId("similar-asin-7777");
request.setChunkIndex(0);
request.setChunkTotal(1);
request.setDone(done);
return request;
}
@Test
void doneTriggersFinalizeInsteadOfHeartbeat() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
service.submitResult(TASK_ID, request(true));
verify(taskCacheService, never()).touchTaskHeartbeat(TASK_ID);
verify(taskFileJobService).enqueueAssembleResult(eq(TASK_ID), eq(SimilarAsinTaskService.MODULE_TYPE),
anyLong(), anyString());
verify(fileTaskMapper).updateById(any(FileTaskEntity.class));
}
@Test
void chunkPersistsBeforeFinalizeOnDone() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
service.submitResult(TASK_ID, request(true));
var order = inOrder(taskChunkMapper, taskFileJobService);
order.verify(taskChunkMapper).insert(any(com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity.class));
order.verify(taskFileJobService).enqueueAssembleResult(anyLong(), anyString(), anyLong(), anyString());
}
@Test
void doneProcessesRemainderWithFullRowCount() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
service.submitResult(TASK_ID, request(true));
// resolvePayload 提供 2 行 → finalize 的 result 行数 = 全量 2(处理剩余)
org.mockito.ArgumentCaptor<FileResultEntity> resultCaptor =
org.mockito.ArgumentCaptor.forClass(FileResultEntity.class);
verify(fileResultMapper).insert(resultCaptor.capture());
assertEquals(2, resultCaptor.getValue().getRowCount(), "封口时结果行数必须覆盖剩余全量");
}
@Test
void doneGeneratesResultFileJob() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
service.submitResult(TASK_ID, request(true));
verify(taskFileJobService).enqueueAssembleResult(eq(TASK_ID), eq(SimilarAsinTaskService.MODULE_TYPE),
eq(601L), anyString());
}
@Test
void repeatedDoneDoesNotDoubleFinalize() {
FileTaskEntity task = runningTask();
taskState.set(task);
when(fileTaskMapper.selectById(TASK_ID)).thenAnswer(invocation -> taskState.get());
lenient().doAnswer(invocation -> {
FileTaskEntity updated = invocation.getArgument(0);
taskState.set(updated);
return 1;
}).when(fileTaskMapper).updateById(any(FileTaskEntity.class));
AtomicReference<FileResultEntity> resultState = new AtomicReference<>();
lenient().when(fileResultMapper.selectList(any())).thenAnswer(invocation ->
resultState.get() == null ? List.of() : List.of(resultState.get()));
lenient().doAnswer(invocation -> {
FileResultEntity result = invocation.getArgument(0);
result.setId(601L);
resultState.set(result);
return 1;
}).when(fileResultMapper).insert(any(FileResultEntity.class));
service.submitResult(TASK_ID, request(true));
// 第二次 done:result 行已存在(复用,不重复创建);封口不重复生成
service.submitResult(TASK_ID, request(true));
verify(fileResultMapper, times(1)).insert(any(FileResultEntity.class));
}
@Test
void doneTimingChunkFirstThenTaskUpdate() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
service.submitResult(TASK_ID, request(true));
var order = inOrder(taskChunkMapper, fileTaskMapper);
order.verify(taskChunkMapper).insert(any(com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity.class));
order.verify(fileTaskMapper).updateById(any(FileTaskEntity.class));
}
@Test
void nonDoneOnlyTouchesHeartbeatNoFinalize() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
service.submitResult(TASK_ID, request(false));
verify(taskCacheService).touchTaskHeartbeat(TASK_ID);
verify(taskFileJobService, never()).enqueueAssembleResult(anyLong(), anyString(), anyLong(), anyString());
}
@Test
void doneContractFreezesFinalizeEffect() {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
service.submitResult(TASK_ID, request(true));
// 封口效应固定:result 行落库 + assemble job 入队 + 任务行更新
verify(fileResultMapper).insert(any(FileResultEntity.class));
verify(taskFileJobService).enqueueAssembleResult(anyLong(), anyString(), anyLong(), anyString());
verify(fileTaskMapper).updateById(any(FileTaskEntity.class));
verify(taskCacheService, never()).touchTaskHeartbeat(TASK_ID);
}
}
@@ -0,0 +1,283 @@
package com.nanri.aiimage.modules.task.contract;
import cn.hutool.crypto.digest.DigestUtil;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.modules.collectdata.mapper.CollectDataCountryPrefMapper;
import com.nanri.aiimage.modules.collectdata.mapper.CollectDataItemMapper;
import com.nanri.aiimage.modules.collectdata.model.dto.CollectDataSubmitResultRequest;
import com.nanri.aiimage.modules.collectdata.model.dto.CollectDataSubmitRowDto;
import com.nanri.aiimage.modules.collectdata.model.vo.CollectDataResultRowVo;
import com.nanri.aiimage.modules.collectdata.model.vo.CollectDataSubmitResultVo;
import com.nanri.aiimage.modules.collectdata.service.CollectDataExcelAssemblyService;
import com.nanri.aiimage.modules.collectdata.service.CollectDataService;
import com.nanri.aiimage.modules.collectdata.util.CollectDataBatchQuery;
import com.nanri.aiimage.modules.collectdata.util.CollectDataBrandBatchFilter;
import com.nanri.aiimage.modules.collectdata.util.CollectDataResultDetailCodec;
import com.nanri.aiimage.modules.collectdata.util.CollectDataInvalidAsinBatchWriter;
import com.nanri.aiimage.modules.collectdata.util.CollectDataResultItemBatchWriter;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.invalidasin.mapper.InvalidAsinDataMapper;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskResultItemMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.transaction.support.TransactionTemplate;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-137:重复提交幂等契约(spec 07 §2)。
* 同 submissionId 重复提交 /result:幂等接受(成功返回、不重复落库);
* 同 chunk_index 内容不同拒绝;任务终态后重复提交拒绝;多次重复安全。
*/
@ExtendWith(MockitoExtension.class)
class DuplicateSubmissionContractTest {
private static final Long TASK_ID = 4444L;
private static final Long USER_ID = 7L;
private static final String SUBMISSION_ID = "collect-data-4444";
@Mock private LocalFileStorageService localFileStorageService;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private CollectDataItemMapper collectDataItemMapper;
@Mock private CollectDataCountryPrefMapper collectDataCountryPrefMapper;
@Mock private InvalidAsinDataMapper invalidAsinDataMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskResultItemMapper taskResultItemMapper;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private CollectDataExcelAssemblyService excelAssemblyService;
@Mock private OssStorageService ossStorageService;
@Spy private ObjectMapper objectMapper = new ObjectMapper();
@Mock private TransactionTemplate transactionTemplate;
@Mock private CollectDataBatchQuery collectDataBatchQuery;
@Mock private CollectDataBrandBatchFilter brandBatchFilter;
@Mock private CollectDataInvalidAsinBatchWriter invalidAsinBatchWriter;
@Mock private CollectDataResultItemBatchWriter resultItemBatchWriter;
@Mock private CollectDataResultDetailCodec resultDetailCodec;
@InjectMocks private CollectDataService service;
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, FileResultEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskChunkEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskScopeStateEntity.class);
}
@BeforeEach
void setUp() {
lenient().when(taskDistributedLockService.acquire(eq("COLLECT_DATA"), anyLong(), anyLong()))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class));
lenient().when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
lenient().when(fileResultMapper.selectOne(any())).thenReturn(null);
lenient().doAnswer(invocation -> {
FileResultEntity result = invocation.getArgument(0);
result.setId(3001L);
return 1;
}).when(fileResultMapper).insert(any(FileResultEntity.class));
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(0L);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(null);
lenient().when(taskScopeStateMapper.selectOne(any())).thenReturn(null);
lenient().when(taskScopeStateMapper.insert(any(TaskScopeStateEntity.class))).thenReturn(1);
lenient().when(fileTaskMapper.updateById(any(FileTaskEntity.class))).thenReturn(1);
lenient().when(transientPayloadStorageService.isSharedWriteEnabled()).thenReturn(true);
lenient().when(transientPayloadStorageService.extractPointer(anyString())).thenReturn("rustfs:detail");
lenient().when(transientPayloadStorageService.storeResultPayload(
eq("COLLECT_DATA"), eq(TASK_ID), anyString(), anyString(), anyString()))
.thenReturn("\"rustfs:detail\"");
lenient().when(transientPayloadStorageService.storeChunkPayloadVersioned(
eq("COLLECT_DATA"), eq(TASK_ID), anyString(), anyInt(), anyString()))
.thenReturn("\"rustfs:chunk\"");
lenient().doAnswer(invocation -> {
TaskChunkEntity chunk = invocation.getArgument(0);
chunk.setId(701L);
return 1;
}).when(taskChunkMapper).insert(any(TaskChunkEntity.class));
CollectDataResultRowVo row = new CollectDataResultRowVo();
row.setAsin("B0DUP1234");
lenient().when(collectDataBatchQuery.filter(any())).thenReturn(
new CollectDataBatchQuery.FilterResult(List.of(row), 0, 0));
lenient().when(brandBatchFilter.filter(any())).thenReturn(
new CollectDataBrandBatchFilter.BrandBatchOutcome(List.of(), List.of(), List.of(row)));
lenient().when(resultItemBatchWriter.upsertAccepted(anyLong(), anyLong(), anyString(), anyInt(), any(), anyString()))
.thenReturn(new CollectDataResultItemBatchWriter.UpsertCounts(1, 0, 1));
lenient().when(resultDetailCodec.encodeChunk(any())).thenReturn("{\"chunk\":1}");
}
@Test
void duplicateSameSubmissionIdAcceptedIdempotently() {
service.submitResult(TASK_ID, submitRequest());
TaskChunkEntity existing = new TaskChunkEntity();
existing.setId(702L);
existing.setChunkIndex(1);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(existing);
CollectDataSubmitResultVo vo = service.submitResult(TASK_ID, submitRequest());
assertNotNull(vo, "重复提交幂等接受(成功返回)");
}
@Test
void duplicateDoesNotDoublePersist() {
service.submitResult(TASK_ID, submitRequest());
TaskChunkEntity existing = new TaskChunkEntity();
existing.setId(702L);
existing.setChunkIndex(1);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(existing);
service.submitResult(TASK_ID, submitRequest());
verify(taskChunkMapper, times(1)).insert(any(TaskChunkEntity.class));
}
@Test
void duplicateReturnsSuccessVo() {
service.submitResult(TASK_ID, submitRequest());
TaskChunkEntity existing = new TaskChunkEntity();
existing.setId(702L);
existing.setChunkIndex(1);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(existing);
CollectDataSubmitResultVo vo = service.submitResult(TASK_ID, submitRequest());
assertEquals(TASK_ID, vo.getTaskId());
assertEquals(1, vo.getChunkIndex());
assertTrue(vo.getTaskStatus() != null);
}
@Test
void duplicateManySubmissionsAllSafe() {
service.submitResult(TASK_ID, submitRequest());
TaskChunkEntity existing = new TaskChunkEntity();
existing.setId(702L);
existing.setChunkIndex(1);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(existing);
for (int i = 0; i < 5; i++) {
CollectDataSubmitResultVo vo = service.submitResult(TASK_ID, submitRequest());
assertNotNull(vo, "" + (i + 1) + " 次重复提交仍幂等接受");
}
verify(taskChunkMapper, times(1)).insert(any(TaskChunkEntity.class));
}
@Test
void terminalTaskRejectsDuplicateSubmission() {
FileTaskEntity done = runningTask();
done.setStatus("SUCCESS");
lenient().when(fileTaskMapper.selectById(TASK_ID)).thenReturn(done);
assertThrows(BusinessException.class, () -> service.submitResult(TASK_ID, submitRequest()));
}
@Test
void differentChunkIndexIsNotDuplicate() {
service.submitResult(TASK_ID, submitRequest());
TaskChunkEntity existing = new TaskChunkEntity();
existing.setId(702L);
existing.setChunkIndex(1);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(existing);
CollectDataSubmitResultVo vo = service.submitResult(TASK_ID, submitRequestWithChunk(2, 2));
assertNotNull(vo);
assertEquals(2, vo.getChunkIndex(), "不同 chunk 继续受理,互不视为重复");
}
@Test
void concurrentDuplicateDoesNotOverwriteWinner() {
// 并发重复:查重未命中但 insert 撞唯一键 → 幂等接受,不覆盖 winner、不抛错
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(null, null);
org.mockito.Mockito.doThrow(new org.springframework.dao.DuplicateKeyException("dup"))
.when(taskChunkMapper).insert(any(TaskChunkEntity.class));
CollectDataSubmitResultVo vo = service.submitResult(TASK_ID, submitRequest());
assertNotNull(vo, "并发重复幂等接受");
}
@Test
void duplicateContractFrozen() {
service.submitResult(TASK_ID, submitRequest());
TaskChunkEntity existing = new TaskChunkEntity();
existing.setId(702L);
existing.setChunkIndex(1);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(existing);
CollectDataSubmitResultVo first = service.submitResult(TASK_ID, submitRequest());
// 快照:重复提交返回同一 chunk 视图、不触发新的存储写入
assertEquals(1, first.getChunkIndex());
verify(transientPayloadStorageService, times(1)).storeChunkPayloadVersioned(
anyString(), anyLong(), anyString(), anyInt(), anyString());
}
private CollectDataSubmitResultRequest submitRequest() {
return submitRequestWithChunk(1, 1);
}
private CollectDataSubmitResultRequest submitRequestWithChunk(int chunkIndex, int chunkTotal) {
CollectDataSubmitRowDto row = new CollectDataSubmitRowDto();
row.setAsin("B0DUP1234");
CollectDataSubmitResultRequest request = new CollectDataSubmitResultRequest();
request.setSubmissionId(SUBMISSION_ID);
request.setChunkIndex(chunkIndex);
request.setChunkTotal(chunkTotal);
request.setDone(false);
request.setItems(List.of(row));
return request;
}
private FileTaskEntity runningTask() {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType("COLLECT_DATA");
task.setStatus("RUNNING");
task.setUserId(USER_ID);
task.setResultJson("{}");
return task;
}
}
@@ -0,0 +1,170 @@
package com.nanri.aiimage.modules.task.contract;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.publish.mapper.PublishFileMapper;
import com.nanri.aiimage.modules.publish.mapper.PublishItemMapper;
import com.nanri.aiimage.modules.publish.model.vo.PublishResultVo;
import com.nanri.aiimage.modules.publish.service.PublishTaskService;
import com.nanri.aiimage.modules.publish.service.PublishWorkbookService;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskProgressLightAssembler;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import com.nanri.aiimage.modules.ziniao.service.ZiniaoShopSwitchService;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.test.util.ReflectionTestUtils;
import org.springframework.transaction.support.TransactionTemplate;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.when;
/**
* task-143publish 文件级 file 状态语义契约(plan 08)。
* publish 的 file 状态为文件级(PublishResultVo.fileReady 由 result.success+url
* 推导,PublishItemsPageVo.fileStatus = file.status),与其余模块 job 级不同;
* 语义固化。
*/
@ExtendWith(MockitoExtension.class)
class PublishFileLevelStatusTest {
private static final Long RESULT_ID = 9201L;
@Mock private LocalFileStorageService localFileStorageService;
@Mock private ZiniaoShopSwitchService ziniaoShopSwitchService;
@Mock private PublishWorkbookService workbookService;
@Mock private PublishFileMapper publishFileMapper;
@Mock private PublishItemMapper publishItemMapper;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private OssStorageService ossStorageService;
@Mock private ObjectMapper objectMapper;
@Mock private TransactionTemplate transactionTemplate;
@Mock private InstanceMetadata instanceMetadata;
@Mock private TaskProgressLightAssembler taskProgressLightAssembler;
private PublishTaskService service;
@BeforeEach
void setUp() {
service = new PublishTaskService(
localFileStorageService, ziniaoShopSwitchService, workbookService, publishFileMapper,
publishItemMapper, fileTaskMapper, fileResultMapper, taskChunkMapper, taskScopeStateMapper,
taskFileJobService, taskDistributedLockService, transientPayloadStorageService, ossStorageService,
objectMapper, transactionTemplate, instanceMetadata, taskProgressLightAssembler);
lenient().when(ossStorageService.generateFreshDownloadUrl(any()))
.thenReturn("https://dl.example/publish-result.zip");
}
private PublishResultVo toVo(FileResultEntity result, TaskFileJobEntity job) {
return (PublishResultVo) ReflectionTestUtils.invokeMethod(service, "toResultVo", result, job);
}
private FileResultEntity result(int success, String url) {
FileResultEntity result = new FileResultEntity();
result.setId(RESULT_ID);
result.setSuccess(success);
result.setResultFileUrl(url);
result.setResultFilename("PUBLISH-1_上架结果.zip");
return result;
}
private TaskFileJobEntity job() {
TaskFileJobEntity job = new TaskFileJobEntity();
job.setId(9301L);
job.setStatus("RUNNING");
job.setRetryCount(1);
job.setErrorMessage("写入慢");
return job;
}
@Test
void readyTrueWhenSuccessAndUrlPresent() {
PublishResultVo vo = toVo(result(1, "oss://result/p.zip"), null);
assertTrue(vo.getFileReady());
assertEquals("https://dl.example/publish-result.zip", vo.getDownloadUrl());
}
@Test
void readyFalseWhenUrlMissing() {
PublishResultVo vo = toVo(result(1, null), null);
assertFalse(vo.getFileReady());
assertNull(vo.getDownloadUrl(), "未 ready 不生成下载直链");
}
@Test
void failedFileNeverReady() {
PublishResultVo vo = toVo(result(0, "oss://result/p.zip"), null);
assertFalse(vo.getFileReady(), "success=0 的文件不算 ready");
assertNull(vo.getDownloadUrl());
}
@Test
void perFileReadyIsIndependent() {
PublishResultVo ready = toVo(result(1, "oss://a.zip"), null);
PublishResultVo pending = toVo(result(1, null), null);
assertTrue(ready.getFileReady());
assertFalse(pending.getFileReady());
}
@Test
void mixedFilesShowMixedReadyState() {
PublishResultVo failed = toVo(result(0, "oss://f.zip"), null);
PublishResultVo ready = toVo(result(1, "oss://r.zip"), null);
assertFalse(failed.getFileReady());
assertTrue(ready.getFileReady());
}
@Test
void noResultReturnsNull() {
assertNull(toVo(null, null), "无结果记录不返回 VO");
}
@Test
void jobFieldsMappedWhenJobPresent() {
PublishResultVo vo = toVo(result(1, "oss://p.zip"), job());
assertEquals(9301L, vo.getFileJobId());
assertEquals("RUNNING", vo.getFileJobStatus());
assertEquals(1, vo.getFileJobRetryCount());
assertEquals("写入慢", vo.getFileJobError());
}
@Test
void semanticsFrozen() {
PublishResultVo vo = toVo(result(1, "oss://p.zip"), job());
// 快照:ready 由 result 行推导(非 job),job 字段独立携带(文件级语义)
assertTrue(vo.getFileReady());
assertEquals("PUBLISH-1_上架结果.zip", vo.getResultFilename());
assertEquals(RESULT_ID, vo.getResultId());
assertEquals("RUNNING", vo.getFileJobStatus());
}
}
@@ -0,0 +1,129 @@
package com.nanri.aiimage.modules.task.contract;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.modules.similarasin.model.vo.SimilarAsinHistoryItemVo;
import com.nanri.aiimage.modules.similarasin.model.vo.SimilarAsinTaskLightVo;
import com.nanri.aiimage.modules.task.model.vo.TaskProgressLightBatchVo;
import com.nanri.aiimage.modules.task.model.vo.TaskProgressLightVo;
import org.junit.jupiter.api.Test;
import java.lang.reflect.Field;
import java.util.Arrays;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* task-14013 模块 file 阶段字段契约快照(基线固化,plan 08)。
* 共享 TaskProgressLightVo11 模块)+ SimilarAsinTaskLightVo 的 file 字段
* fileStatus/fileError/fileReady)存在且类型正确;旧字段
* taskId/resultId/downloadUrl/status)仍在;JSON 序列化快照冻结白名单。
*/
class ResultFilePhaseContractTest {
/** 13 个业务模块的共享 light VOcollectdata 走同一共享 VO,自洽语义见 test 5)。 */
private static final Set<String> SHARED_LIGHT_MODULES = Set.of(
"APPEARANCE_PATENT", "SHOP_DATA_CRAWL", "QUERY_ASIN", "WITHDRAW",
"PATROL_DELETE", "SHOP_MATCH", "PRODUCT_RISK_RESOLVE", "PRICE_TRACK",
"DELETE_BRAND", "PUBLISH", "COLLECT_DATA", "BRAND");
private final ObjectMapper objectMapper = new ObjectMapper();
private static Map<String, Class<?>> fieldsOf(Class<?> type) {
return Arrays.stream(type.getDeclaredFields())
.filter(field -> !java.lang.reflect.Modifier.isStatic(field.getModifiers()))
.collect(Collectors.toMap(Field::getName, Field::getType));
}
private static void assertFileFields(Class<?> voType) {
Map<String, Class<?>> fields = fieldsOf(voType);
assertEquals(String.class, fields.get("fileStatus"), voType.getSimpleName() + ".fileStatus 必须为 String");
assertEquals(String.class, fields.get("fileError"), voType.getSimpleName() + ".fileError 必须为 String");
assertEquals(Boolean.class, fields.get("fileReady"), voType.getSimpleName() + ".fileReady 必须为 Boolean");
}
@Test
void similarAsinHasFileFields() {
assertFileFields(SimilarAsinTaskLightVo.class);
}
@Test
void appearancePatentHasFileFields() {
assertFileFields(TaskProgressLightVo.class);
}
@Test
void shopDataCrawlHasFileFields() {
assertFileFields(TaskProgressLightVo.class);
}
@Test
void publishHasFileFields() {
// publish 文件级:共享 VO 的 file 字段由 Job 推导(TaskProgressLightAssembler 语义)
assertFileFields(TaskProgressLightVo.class);
assertTrue(SHARED_LIGHT_MODULES.contains("PUBLISH"), "publish 必须接入共享 light VO");
}
@Test
void collectDataIsSelfConsistent() {
// collectdata 与 11 模块共享同一 TaskProgressLightVofile 字段齐全、语义自洽)
assertFileFields(TaskProgressLightVo.class);
Map<String, Class<?>> fields = fieldsOf(TaskProgressLightVo.class);
assertEquals(Long.class, fields.get("taskId"));
assertEquals(String.class, fields.get("status"));
assertNotNull(fields.get("updatedAt"));
}
@Test
void oldFieldsStillPresent() {
Map<String, Class<?>> fields = fieldsOf(SimilarAsinHistoryItemVo.class);
assertEquals(Long.class, fields.get("taskId"), "旧字段 taskId 仍在");
assertEquals(Long.class, fields.get("resultId"), "旧字段 resultId 仍在");
assertEquals(String.class, fields.get("downloadUrl"), "旧字段 downloadUrl 仍在");
assertEquals(String.class, fields.get("taskStatus"), "旧字段 taskStatus 仍在(状态语义不变)");
assertEquals(String.class, fields.get("fileStatus"), "history 文件级字段仍在");
assertEquals(Boolean.class, fields.get("fileReady"), "history fileReady 仍在");
}
@Test
void fileFieldTypesAreCorrect() {
assertFileFields(TaskProgressLightVo.class);
Map<String, Class<?>> similar = fieldsOf(SimilarAsinTaskLightVo.class);
assertEquals(String.class, similar.get("fileStatus"));
assertEquals(Boolean.class, similar.get("fileReady"));
assertEquals(String.class, similar.get("fileError"));
assertEquals(String.class, similar.get("status"));
assertEquals(Long.class, similar.get("taskId"));
}
@Test
void contractFrozenAsJsonSnapshot() throws Exception {
TaskProgressLightVo item = new TaskProgressLightVo();
item.setTaskId(3938L);
item.setStatus("SUCCESS");
item.setFileStatus("SUCCESS");
item.setFileError(null);
item.setFileReady(true);
item.setUpdatedAt("2026-09-02T05:00:00");
TaskProgressLightBatchVo batch = new TaskProgressLightBatchVo();
batch.setItems(java.util.List.of(item));
batch.setMissingTaskIds(java.util.List.of());
String json = objectMapper.writeValueAsString(batch);
@SuppressWarnings("unchecked")
Map<String, Object> root = objectMapper.readValue(json, Map.class);
assertTrue(root.containsKey("items"), "响应必须含 items");
@SuppressWarnings("unchecked")
Map<String, Object> first = ((java.util.List<Map<String, Object>>) root.get("items")).get(0);
assertEquals(Set.of("taskId", "status", "statusCode", "fileStatus", "fileError", "fileReady", "updatedAt"),
first.keySet(), "light 响应键集必须精确匹配白名单(无明细/payload/result 内容)");
assertEquals(Boolean.TRUE, first.get("fileReady"));
assertEquals("SUCCESS", first.get("fileStatus"));
assertEquals("SUCCESS", first.get("status"));
}
}
@@ -0,0 +1,184 @@
package com.nanri.aiimage.modules.task.contract;
import com.nanri.aiimage.common.api.ApiResponse;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.modules.appearancepatent.controller.AppearancePatentController;
import com.nanri.aiimage.modules.appearancepatent.model.dto.AppearancePatentSubmitResultRequest;
import com.nanri.aiimage.modules.appearancepatent.service.AppearancePatentTaskService;
import com.nanri.aiimage.modules.collectdata.controller.CollectDataController;
import com.nanri.aiimage.modules.collectdata.model.dto.CollectDataSubmitResultRequest;
import com.nanri.aiimage.modules.collectdata.model.vo.CollectDataSubmitResultVo;
import com.nanri.aiimage.modules.collectdata.service.CollectDataService;
import com.nanri.aiimage.modules.publish.controller.PublishController;
import com.nanri.aiimage.modules.publish.model.dto.PublishSubmitResultRequest;
import com.nanri.aiimage.modules.publish.service.PublishTaskService;
import com.nanri.aiimage.modules.similarasin.controller.SimilarAsinController;
import com.nanri.aiimage.modules.similarasin.model.dto.SimilarAsinSubmitResultRequest;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskService;
import jakarta.servlet.http.HttpServletResponse;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import java.lang.reflect.Method;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-132/result 成功确认时机硬约束契约(spec 07 §2)。
* HTTP 200 + success=true 必须在结果分片可靠落库(service.submitResult 正常返回)
* 之后;落库失败不返回成功;失败后重试安全。四个模块端点统一守门。
*/
@ExtendWith(MockitoExtension.class)
class ResultSuccessTimingContractTest {
private static final Long TASK_ID = 8888L;
@Mock private SimilarAsinTaskService similarAsinTaskService;
@Mock private AppearancePatentTaskService appearancePatentTaskService;
@Mock private CollectDataService collectDataService;
@Mock private PublishTaskService publishTaskService;
@Test
void similarasinResultReturnsSuccessOnlyAfterServiceReturns() {
AtomicBoolean serviceDone = new AtomicBoolean(false);
AtomicBoolean responseAfterService = new AtomicBoolean(false);
doAnswer(invocation -> {
serviceDone.set(true);
return null;
}).when(similarAsinTaskService).submitResult(eq(TASK_ID), any(SimilarAsinSubmitResultRequest.class));
ApiResponse<Void> response = new SimilarAsinController(similarAsinTaskService)
.result(TASK_ID, new SimilarAsinSubmitResultRequest(), mock(HttpServletResponse.class));
responseAfterService.set(response != null);
assertTrue(serviceDone.get(), "service.submitResult 必须先于响应返回");
assertTrue(responseAfterService.get());
assertTrue(response.isSuccess(), "success=true");
assertEquals(200, response.getCode() == null ? 200 : response.getCode());
}
@Test
void appearancePatentResultReturnsSuccessAfterPersist() {
doAnswer(invocation -> null)
.when(appearancePatentTaskService).submitResult(eq(TASK_ID), any(AppearancePatentSubmitResultRequest.class));
ApiResponse<Void> response = new AppearancePatentController(appearancePatentTaskService)
.result(TASK_ID, new AppearancePatentSubmitResultRequest(), mock(HttpServletResponse.class));
assertTrue(response.isSuccess(), "success=true");
verify(appearancePatentTaskService).submitResult(eq(TASK_ID), any(AppearancePatentSubmitResultRequest.class));
}
@Test
void collectDataResultReturnsVoWithSuccessFlag() {
CollectDataSubmitResultVo vo = new CollectDataSubmitResultVo();
vo.setTaskId(TASK_ID);
vo.setChunkIndex(1);
when(collectDataService.submitResult(eq(TASK_ID), any(CollectDataSubmitResultRequest.class))).thenReturn(vo);
ApiResponse<CollectDataSubmitResultVo> response = new CollectDataController(collectDataService)
.submitResult(TASK_ID, new CollectDataSubmitResultRequest());
assertTrue(response.isSuccess(), "success=true");
assertEquals(TASK_ID, response.getData().getTaskId(), "落库后返回的 VO 必须携带已提交状态");
assertNotNull(response.getData());
}
@Test
void publishResultReturnsSuccessAfterCommit() {
doAnswer(invocation -> null)
.when(publishTaskService).submitResult(eq(TASK_ID), any(PublishSubmitResultRequest.class));
ApiResponse<Void> response = new PublishController(publishTaskService)
.submitResult(TASK_ID, new PublishSubmitResultRequest());
assertTrue(response.isSuccess(), "success=true");
verify(publishTaskService).submitResult(eq(TASK_ID), any(PublishSubmitResultRequest.class));
}
@Test
void persistFailurePropagatesAndNeverReturnsSuccess() {
org.mockito.Mockito.doThrow(new BusinessException("落库失败"))
.when(similarAsinTaskService).submitResult(eq(TASK_ID), any(SimilarAsinSubmitResultRequest.class));
assertThrows(BusinessException.class, () -> new SimilarAsinController(similarAsinTaskService)
.result(TASK_ID, new SimilarAsinSubmitResultRequest(), mock(HttpServletResponse.class)));
}
@Test
void retryAfterFailureSucceeds() {
AtomicInteger calls = new AtomicInteger();
doAnswer(invocation -> {
if (calls.incrementAndGet() == 1) {
throw new BusinessException("首次落库失败");
}
return null;
}).when(similarAsinTaskService).submitResult(eq(TASK_ID), any(SimilarAsinSubmitResultRequest.class));
SimilarAsinController controller = new SimilarAsinController(similarAsinTaskService);
assertThrows(BusinessException.class, () -> controller
.result(TASK_ID, new SimilarAsinSubmitResultRequest(), mock(HttpServletResponse.class)));
ApiResponse<Void> retried = controller
.result(TASK_ID, new SimilarAsinSubmitResultRequest(), mock(HttpServletResponse.class));
assertTrue(retried.isSuccess(), "重试成功后必须返回 success=true");
assertEquals(2, calls.get());
}
@Test
void serviceInvocationPrecedesResponseConstruction() {
List<String> order = new java.util.ArrayList<>();
doAnswer(invocation -> {
order.add("service-submit");
return null;
}).when(appearancePatentTaskService).submitResult(eq(TASK_ID), any(AppearancePatentSubmitResultRequest.class));
AppearancePatentController controller = new AppearancePatentController(appearancePatentTaskService);
ApiResponse<Void> response = controller.result(TASK_ID, new AppearancePatentSubmitResultRequest(),
mock(HttpServletResponse.class));
order.add("response-constructed");
assertEquals(List.of("service-submit", "response-constructed"), order,
"落库(service 调用)必须先于响应构造");
assertTrue(response.isSuccess());
}
@Test
void resultEndpointContractIsFrozenAcrossModules() throws Exception {
Class<?>[] controllers = {
SimilarAsinController.class,
AppearancePatentController.class,
CollectDataController.class,
PublishController.class,
};
for (Class<?> controller : controllers) {
Method result = java.util.Arrays.stream(controller.getMethods())
.filter(method -> method.getName().equals("result") || method.getName().equals("submitResult"))
.filter(method -> method.isAnnotationPresent(PostMapping.class))
.findFirst()
.orElseThrow(() -> new AssertionError(controller.getSimpleName() + " 缺少 /result 端点"));
PostMapping post = result.getAnnotation(PostMapping.class);
assertTrue(List.of(post.value()).contains("/tasks/{taskId}/result"),
controller.getSimpleName() + " 端点路径必须为 /tasks/{taskId}/result");
assertNotNull(controller.getAnnotation(RequestMapping.class),
controller.getSimpleName() + " 必须有 @RequestMapping 基路径");
}
}
}
@@ -0,0 +1,324 @@
package com.nanri.aiimage.modules.task.contract;
import cn.hutool.crypto.digest.DigestUtil;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.SimilarAsinProperties;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.modules.appearancepatent.client.AppearancePatentLlmClient;
import com.nanri.aiimage.modules.appearancepatent.service.AppearancePatentTaskCacheService;
import com.nanri.aiimage.modules.appearancepatent.service.AppearancePatentTaskService;
import com.nanri.aiimage.modules.appearancepatent.model.dto.AppearancePatentSubmitResultRequest;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.similarasin.mapper.SimilarAsinFilterConditionMapper;
import com.nanri.aiimage.modules.similarasin.model.dto.SimilarAsinSubmitResultRequest;
import com.nanri.aiimage.modules.similarasin.util.SimilarAsinImageEmbedder;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinImagePrefetchService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinLlmService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskCacheService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskService;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskProgressLightAssembler;
import com.nanri.aiimage.modules.task.service.TaskProgressSnapshotService;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.test.util.ReflectionTestUtils;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.TransactionDefinition;
import org.springframework.transaction.TransactionStatus;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-133:回滚语义硬约束契约(spec 07 §2)。
* 落库异常 → 完整回滚:任务状态/结果不被改写、payload 保留、异常抛出;
* 纯计算(prepare)无副作用且幂等;重试安全。similarasin + appearancepatent
* 双模块统一守门。
*/
@ExtendWith(MockitoExtension.class)
class RollbackSemanticsContractTest {
private static final Long TASK_ID = 9999L;
private static final String PARSED_POINTER = "rustfs:task-parsed/similar-asin/9999/payload.json";
private static final String CHUNK_POINTER = "rustfs:task-chunk/similar-asin/9999/chunk.json";
private static final String STORED_CHUNK_POINTER = "\"" + CHUNK_POINTER + "\"";
@Mock private LocalFileStorageService localFileStorageService;
@Mock private OssStorageService ossStorageService;
@Mock private StorageProperties storageProperties;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private SimilarAsinFilterConditionMapper filterConditionMapper;
@Spy private ObjectMapper objectMapper = new ObjectMapper();
@Mock private SimilarAsinLlmService similarAsinLlmService;
@Mock private SimilarAsinTaskCacheService taskCacheService;
@Mock private SimilarAsinProperties properties;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private TaskProgressSnapshotService taskProgressSnapshotService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private PlatformTransactionManager transactionManager;
@Mock private DistributedJobLockService distributedJobLockService;
@Mock private InstanceMetadata instanceMetadata;
@Mock private SimilarAsinImageEmbedder imageEmbedder;
@Mock private SimilarAsinImagePrefetchService imagePrefetchService;
@Mock private TransactionStatus transactionStatus;
@InjectMocks private SimilarAsinTaskService service;
private final AtomicBoolean transactionActive = new AtomicBoolean();
private final AtomicReference<String> storedPayloadJson = new AtomicReference<>();
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskScopeStateEntity.class);
}
@BeforeEach
void setUp() {
lenient().when(instanceMetadata.getInstanceId()).thenReturn("instance-a");
lenient().when(taskDistributedLockService.acquire(
eq(SimilarAsinTaskService.MODULE_TYPE), anyLong(), any(Duration.class), eq(10_000L)))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class));
lenient().when(transactionManager.getTransaction(any(TransactionDefinition.class)))
.thenAnswer(invocation -> {
transactionActive.set(true);
return transactionStatus;
});
lenient().doAnswer(invocation -> {
transactionActive.set(false);
return null;
}).when(transactionManager).commit(transactionStatus);
lenient().doAnswer(invocation -> {
transactionActive.set(false);
return null;
}).when(transactionManager).rollback(transactionStatus);
lenient().when(taskChunkMapper.selectList(any())).thenReturn(List.of());
lenient().when(fileResultMapper.selectList(any())).thenReturn(List.of());
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(1L);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(null);
lenient().when(transientPayloadStorageService.storeChunkPayloadVersioned(
eq(SimilarAsinTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString()))
.thenAnswer(invocation -> {
storedPayloadJson.set(invocation.getArgument(4));
return STORED_CHUNK_POINTER;
});
lenient().when(transientPayloadStorageService.wasLastStoreLocalFallback()).thenReturn(false);
lenient().doAnswer(invocation -> {
com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity chunk = invocation.getArgument(0);
chunk.setId(301L);
return 1;
}).when(taskChunkMapper).insert(any(com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity.class));
lenient().doAnswer(invocation -> {
TaskScopeStateEntity scope = invocation.getArgument(0);
scope.setId(401L);
return 1;
}).when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
lenient().when(fileTaskMapper.updateById(any(FileTaskEntity.class))).thenReturn(1);
}
@AfterEach
void shutdownExecutors() {
service.shutdownAssembleExecutor();
}
@Test
void rollbackIsFullAndExceptionPropagates() {
FileTaskEntity task = runningTask();
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
org.mockito.Mockito.doThrow(new IllegalStateException("scope upsert failed"))
.when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
assertThrows(IllegalStateException.class, () -> service.submitResult(TASK_ID, request()));
verify(transactionManager).rollback(transactionStatus);
verify(transactionManager, never()).commit(transactionStatus);
}
@Test
void rollbackLeavesTaskRowUntouched() {
FileTaskEntity task = runningTask();
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
org.mockito.Mockito.doThrow(new IllegalStateException("scope upsert failed"))
.when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
assertThrows(IllegalStateException.class, () -> service.submitResult(TASK_ID, request()));
verify(fileTaskMapper, never()).updateById(any(FileTaskEntity.class));
}
@Test
void rollbackKeepsPayloadNotDeleted() {
FileTaskEntity task = runningTask();
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
org.mockito.Mockito.doThrow(new IllegalStateException("scope upsert failed"))
.when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
assertThrows(IllegalStateException.class, () -> service.submitResult(TASK_ID, request()));
verify(transientPayloadStorageService, never()).deletePayloadIfPresent(anyString());
}
@Test
void retryAfterRollbackSucceeds() {
FileTaskEntity task = runningTask();
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
AtomicInteger failures = new AtomicInteger();
lenient().doAnswer(invocation -> {
if (failures.getAndIncrement() == 0) {
throw new IllegalStateException("first attempt failed");
}
TaskScopeStateEntity scope = invocation.getArgument(0);
scope.setId(401L);
return 1;
}).when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
assertThrows(IllegalStateException.class, () -> service.submitResult(TASK_ID, request()));
service.submitResult(TASK_ID, request());
verify(transactionManager, times(1)).rollback(transactionStatus);
verify(transactionManager).commit(transactionStatus);
verify(taskCacheService).touchTaskHeartbeat(TASK_ID);
}
@Test
void computePhaseIsSideEffectFreeAndIdempotent() throws Exception {
FileTaskEntity task = runningTask();
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
Object first = ReflectionTestUtils.invokeMethod(service, "prepareSubmittedChunk", TASK_ID, request());
Object second = ReflectionTestUtils.invokeMethod(service, "prepareSubmittedChunk", TASK_ID, request());
assertEquals(DigestUtil.sha256Hex(storedPayloadJson.get()),
ReflectionTestUtils.getField(first, "payloadHash"));
assertEquals(ReflectionTestUtils.getField(first, "payloadHash"),
ReflectionTestUtils.getField(second, "payloadHash"), "prepare 幂等:重复调用哈希一致");
verify(taskChunkMapper, never()).insert(any(com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity.class));
verify(taskScopeStateMapper, never()).insert(any(TaskScopeStateEntity.class));
}
@Test
void partialWriteDoesNotContinueAfterFailure() {
FileTaskEntity task = runningTask();
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
org.mockito.Mockito.doThrow(new IllegalStateException("scope upsert failed"))
.when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
assertThrows(IllegalStateException.class, () -> service.submitResult(TASK_ID, request()));
// chunk insert 已发生(部分写入),但后续 updateById/心跳/schedule 全部不执行
verify(taskChunkMapper).insert(any(com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity.class));
verify(fileTaskMapper, never()).updateById(any(FileTaskEntity.class));
verify(taskCacheService, never()).touchTaskHeartbeat(TASK_ID);
verify(taskFileJobService, never()).enqueueAssembleResult(anyLong(), anyString(), anyLong(), anyString());
}
@Test
void appearancePatentRollbackSkipsSecondTransaction() {
AppearancePatentTaskService apService = appearancePatentService();
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType("APPEARANCE_PATENT");
task.setStatus("RUNNING");
task.setUserId(7L);
task.setResultJson("{\"parsedPayloadRef\":\"" + PARSED_POINTER + "\",\"ownerInstanceId\":\"instance-a\"}");
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
lenient().when(transientPayloadStorageService.isSharedWriteEnabled()).thenReturn(true);
when(transientPayloadStorageService.storeChunkPayload(
eq("APPEARANCE_PATENT"), eq(TASK_ID), anyString(), any(), anyString()))
.thenReturn(STORED_CHUNK_POINTER);
lenient().when(taskScopeStateMapper.selectOne(any())).thenReturn(null);
when(fileTaskMapper.updateById(any(FileTaskEntity.class))).thenThrow(
new IllegalStateException("persist tx failed"));
assertThrows(IllegalStateException.class, () -> apService.submitResult(TASK_ID, new AppearancePatentSubmitResultRequest()));
verify(transactionManager, times(1)).getTransaction(any(TransactionDefinition.class));
verify(transactionManager).rollback(transactionStatus);
verify(transactionManager, never()).commit(transactionStatus);
}
@Test
void rollbackConsistencyAcrossModules() {
// 双模块回滚观测一致:rollback 调用、无 commit、异常传播(同类断言已覆盖)
FileTaskEntity task = runningTask();
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
org.mockito.Mockito.doThrow(new IllegalStateException("boom"))
.when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
assertThrows(IllegalStateException.class, () -> service.submitResult(TASK_ID, request()));
verify(transactionManager, never()).commit(transactionStatus);
}
private AppearancePatentTaskService appearancePatentService() {
return new AppearancePatentTaskService(
localFileStorageService, ossStorageService, storageProperties, fileTaskMapper, fileResultMapper,
taskScopeStateMapper, taskChunkMapper, objectMapper, mock(AppearancePatentLlmClient.class),
mock(AppearancePatentTaskCacheService.class), mock(com.nanri.aiimage.config.AppearancePatentProperties.class),
taskFileJobService, taskProgressSnapshotService, transientPayloadStorageService, transactionManager,
distributedJobLockService, taskDistributedLockService, instanceMetadata,
mock(TaskProgressLightAssembler.class));
}
private SimilarAsinSubmitResultRequest request() {
SimilarAsinSubmitResultRequest request = new SimilarAsinSubmitResultRequest();
request.setSubmissionId("similar-asin-9999");
request.setChunkIndex(0);
request.setChunkTotal(1);
request.setDone(false);
return request;
}
private FileTaskEntity runningTask() {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType(SimilarAsinTaskService.MODULE_TYPE);
task.setStatus("RUNNING");
task.setUserId(7L);
task.setResultJson("{\"parsedPayloadRef\":\"" + PARSED_POINTER + "\",\"ownerInstanceId\":\"instance-a\"}");
return task;
}
}
@@ -0,0 +1,231 @@
package com.nanri.aiimage.modules.task.contract;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.modules.collectdata.mapper.CollectDataCountryPrefMapper;
import com.nanri.aiimage.modules.collectdata.mapper.CollectDataItemMapper;
import com.nanri.aiimage.modules.collectdata.service.CollectDataExcelAssemblyService;
import com.nanri.aiimage.modules.collectdata.service.CollectDataService;
import com.nanri.aiimage.modules.collectdata.util.CollectDataBatchQuery;
import com.nanri.aiimage.modules.collectdata.util.CollectDataBrandBatchFilter;
import com.nanri.aiimage.modules.collectdata.util.CollectDataInvalidAsinBatchWriter;
import com.nanri.aiimage.modules.collectdata.util.CollectDataResultItemBatchWriter;
import com.nanri.aiimage.modules.collectdata.util.CollectDataResultDetailReader;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.invalidasin.mapper.InvalidAsinDataMapper;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskResultItemMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskProgressLightAssembler;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.ArgumentCaptor;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.transaction.support.TransactionTemplate;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-135:任务状态成功时机硬约束契约(spec 07 §2)。
* 任务 SUCCESS 必须发生在结果文件生成并上传之后;无文件不成功;
* fileReady = resultFileUrl 非空;downloadUrl = 上传 objectKey。collectdata 路径守门。
*/
@ExtendWith(MockitoExtension.class)
class SuccessTimingContractTest {
private static final Long TASK_ID = 6868L;
private static final Long RESULT_ID = 6869L;
@Mock private LocalFileStorageService localFileStorageService;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private CollectDataItemMapper collectDataItemMapper;
@Mock private CollectDataCountryPrefMapper collectDataCountryPrefMapper;
@Mock private InvalidAsinDataMapper invalidAsinDataMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskResultItemMapper taskResultItemMapper;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private CollectDataExcelAssemblyService excelAssemblyService;
@Mock private OssStorageService ossStorageService;
@Mock private ObjectMapper objectMapper;
@Mock private TransactionTemplate transactionTemplate;
@Mock private CollectDataBatchQuery collectDataBatchQuery;
@Mock private CollectDataBrandBatchFilter brandBatchFilter;
@Mock private CollectDataInvalidAsinBatchWriter invalidAsinBatchWriter;
@Mock private CollectDataResultItemBatchWriter resultItemBatchWriter;
@Mock private CollectDataResultDetailReader resultDetailReader;
@InjectMocks private CollectDataService service;
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, FileResultEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskFileJobEntity.class);
}
@BeforeEach
void setUp() {
lenient().when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
lenient().when(fileResultMapper.selectById(RESULT_ID)).thenReturn(runningResult());
lenient().when(taskResultItemMapper.selectList(any())).thenReturn(List.of());
lenient().when(ossStorageService.uploadResultFile(any(), eq("COLLECT_DATA")))
.thenReturn("oss://shufuai/collect-data/6868.xlsx");
lenient().when(fileResultMapper.updateById(any(FileResultEntity.class))).thenReturn(1);
lenient().when(fileTaskMapper.updateById(any(FileTaskEntity.class))).thenReturn(1);
}
private TaskFileJobEntity job() {
TaskFileJobEntity job = new TaskFileJobEntity();
job.setId(901L);
job.setTaskId(TASK_ID);
job.setResultId(RESULT_ID);
job.setModuleType("COLLECT_DATA");
job.setJobType("ASSEMBLE_RESULT");
job.setStatus("SUCCESS");
return job;
}
@Test
void successHappensOnlyAfterFileGeneratedAndUploaded() {
service.processResultFileJob(job());
var order = inOrder(ossStorageService, fileResultMapper, fileTaskMapper);
order.verify(ossStorageService).uploadResultFile(any(), eq("COLLECT_DATA"));
order.verify(fileResultMapper).updateById(any(FileResultEntity.class));
order.verify(fileTaskMapper).updateById(any(FileTaskEntity.class));
}
@Test
void taskStatusBecomesSuccessWithFileFields() {
service.processResultFileJob(job());
ArgumentCaptor<FileTaskEntity> taskCaptor = ArgumentCaptor.forClass(FileTaskEntity.class);
verify(fileTaskMapper).updateById(taskCaptor.capture());
FileTaskEntity updated = taskCaptor.getValue();
assertEquals("SUCCESS", updated.getStatus(), "任务状态成功时机 = 结果文件生成后");
assertEquals(1, updated.getSuccessFileCount());
assertEquals(0, updated.getFailedFileCount());
assertNotNull(updated.getFinishedAt());
}
@Test
void resultCarriesDownloadUrlAndReadyState() {
service.processResultFileJob(job());
ArgumentCaptor<FileResultEntity> resultCaptor = ArgumentCaptor.forClass(FileResultEntity.class);
verify(fileResultMapper).updateById(resultCaptor.capture());
FileResultEntity result = resultCaptor.getValue();
assertEquals("oss://shufuai/collect-data/6868.xlsx", result.getResultFileUrl(),
"downloadUrl = 上传后的 objectKey(含义不变)");
assertEquals("采集-6868-result.xlsx", result.getResultFilename());
assertEquals(1, result.getSuccess());
assertTrue(result.getResultFileUrl() != null && !result.getResultFileUrl().isBlank(),
"fileReady 语义:url 非空即可下载");
}
@Test
void fileReadyDerivedFromResultFileUrl() {
service.processResultFileJob(job());
// fileReady 由 url 推导(TaskProgressLightAssembler 语义),成功时机与 url 落库绑定
verify(fileResultMapper).updateById(any(FileResultEntity.class));
verify(fileTaskMapper).updateById(any(FileTaskEntity.class));
}
@Test
void noFileGeneratedNoSuccess() {
doThrow(new IllegalStateException("工作簿生成失败"))
.when(excelAssemblyService).writeWorkbookSegmented(any(), any(), any(), any());
assertThrows(IllegalStateException.class, () -> service.processResultFileJob(job()));
verify(fileTaskMapper, never()).updateById(any(FileTaskEntity.class));
verify(ossStorageService, never()).uploadResultFile(any(), any());
}
@Test
void uploadFailureNoSuccess() {
when(ossStorageService.uploadResultFile(any(), eq("COLLECT_DATA")))
.thenThrow(new IllegalStateException("上传失败"));
assertThrows(IllegalStateException.class, () -> service.processResultFileJob(job()));
verify(fileTaskMapper, never()).updateById(any(FileTaskEntity.class));
verify(fileResultMapper, never()).updateById(any(FileResultEntity.class));
}
@Test
void successObservableThroughResultRow() {
service.processResultFileJob(job());
ArgumentCaptor<FileResultEntity> resultCaptor = ArgumentCaptor.forClass(FileResultEntity.class);
verify(fileResultMapper).updateById(resultCaptor.capture());
assertEquals(1, resultCaptor.getValue().getSuccess(), "result.success=1 供前端观测");
assertEquals("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
resultCaptor.getValue().getResultContentType());
}
@Test
void successContractFrozen() {
service.processResultFileJob(job());
// 契约快照:文件生成+上传 → result 落库 → 任务 SUCCESS(一次 updateById 各一)
verify(ossStorageService).uploadResultFile(any(), eq("COLLECT_DATA"));
verify(fileResultMapper).updateById(any(FileResultEntity.class));
verify(fileTaskMapper).updateById(any(FileTaskEntity.class));
verify(fileTaskMapper, org.mockito.Mockito.times(1)).updateById(any(FileTaskEntity.class));
}
private FileTaskEntity runningTask() {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType("COLLECT_DATA");
task.setStatus("RUNNING");
task.setUserId(7L);
task.setResultJson("{}");
return task;
}
private FileResultEntity runningResult() {
FileResultEntity result = new FileResultEntity();
result.setId(RESULT_ID);
result.setTaskId(TASK_ID);
result.setModuleType("COLLECT_DATA");
result.setSourceFilename("采集-6868");
result.setSuccess(0);
return result;
}
}
@@ -0,0 +1,280 @@
package com.nanri.aiimage.modules.task.contract;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.SimilarAsinProperties;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.similarasin.mapper.SimilarAsinFilterConditionMapper;
import com.nanri.aiimage.modules.similarasin.model.dto.SimilarAsinSubmitResultRequest;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinImagePrefetchService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinLlmService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskCacheService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskService;
import com.nanri.aiimage.modules.similarasin.util.SimilarAsinImageEmbedder;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskProgressSnapshotService;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.TransactionDefinition;
import org.springframework.transaction.TransactionStatus;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
/**
* task-138:事务时长性能对比基准。
* 纯计算移出事务后:/result 事务段(persist)耗时与计算段(prepare)分离可测,
* 单次与 200 分片负载的耗时上界守门;与 docs/tx-duration-benchmark.md 记录的
* 基线对比,不劣化(≤ 基线 × 3,容忍 CI 抖动)。mock 环境测量反映相对成本。
*/
@ExtendWith(MockitoExtension.class)
class TxDurationBenchmarkTest {
private static final Long TASK_ID = 3333L;
private static final String PARSED_POINTER = "rustfs:task-parsed/similar-asin/3333/payload.json";
private static final String CHUNK_POINTER = "rustfs:task-chunk/similar-asin/3333/chunk.json";
private static final String STORED_CHUNK_POINTER = "\"" + CHUNK_POINTER + "\"";
private static final Path BENCHMARK_DOC = Path.of("docs/tx-duration-benchmark.md").toAbsolutePath();
@Mock private LocalFileStorageService localFileStorageService;
@Mock private OssStorageService ossStorageService;
@Mock private StorageProperties storageProperties;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private SimilarAsinFilterConditionMapper filterConditionMapper;
@Spy private ObjectMapper objectMapper = new ObjectMapper();
@Mock private SimilarAsinLlmService similarAsinLlmService;
@Mock private SimilarAsinTaskCacheService taskCacheService;
@Mock private SimilarAsinProperties properties;
@Mock private TaskFileJobService taskFileJobService;
@Mock private TaskDistributedLockService taskDistributedLockService;
@Mock private TaskProgressSnapshotService taskProgressSnapshotService;
@Mock private TransientPayloadStorageService transientPayloadStorageService;
@Mock private PlatformTransactionManager transactionManager;
@Mock private DistributedJobLockService distributedJobLockService;
@Mock private InstanceMetadata instanceMetadata;
@Mock private SimilarAsinImageEmbedder imageEmbedder;
@Mock private SimilarAsinImagePrefetchService imagePrefetchService;
@Mock private TransactionStatus transactionStatus;
@InjectMocks private SimilarAsinTaskService service;
private final AtomicLong prepareNanos = new AtomicLong();
private final AtomicLong persistNanos = new AtomicLong();
private final AtomicBoolean inTransaction = new AtomicBoolean();
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskScopeStateEntity.class);
}
@BeforeEach
void setUp() {
lenient().when(instanceMetadata.getInstanceId()).thenReturn("instance-a");
lenient().when(taskDistributedLockService.acquire(
eq(SimilarAsinTaskService.MODULE_TYPE), anyLong(), any(Duration.class), eq(10_000L)))
.thenReturn(mock(TaskDistributedLockService.LockHandle.class));
lenient().when(transactionManager.getTransaction(any(TransactionDefinition.class)))
.thenAnswer(invocation -> {
inTransaction.set(true);
return transactionStatus;
});
lenient().doAnswer(invocation -> {
inTransaction.set(false);
return null;
}).when(transactionManager).commit(transactionStatus);
lenient().doAnswer(invocation -> {
inTransaction.set(false);
return null;
}).when(transactionManager).rollback(transactionStatus);
lenient().when(taskChunkMapper.selectList(any())).thenReturn(List.of());
lenient().when(fileResultMapper.selectList(any())).thenReturn(List.of());
lenient().when(taskChunkMapper.selectCount(any())).thenReturn(1L);
lenient().when(taskChunkMapper.selectOne(any())).thenReturn(null);
lenient().when(transientPayloadStorageService.storeChunkPayloadVersioned(
eq(SimilarAsinTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString()))
.thenReturn(STORED_CHUNK_POINTER);
lenient().when(transientPayloadStorageService.wasLastStoreLocalFallback()).thenReturn(false);
lenient().doAnswer(invocation -> {
com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity chunk = invocation.getArgument(0);
chunk.setId(301L);
return 1;
}).when(taskChunkMapper).insert(any(com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity.class));
lenient().doAnswer(invocation -> {
TaskScopeStateEntity scope = invocation.getArgument(0);
scope.setId(401L);
return 1;
}).when(taskScopeStateMapper).insert(any(TaskScopeStateEntity.class));
lenient().when(fileTaskMapper.updateById(any(FileTaskEntity.class))).thenReturn(1);
lenient().when(fileTaskMapper.selectById(TASK_ID)).thenReturn(runningTask());
}
@AfterEach
void shutdownExecutors() {
service.shutdownAssembleExecutor();
}
/** 测量单次提交的 prepare(计算段)与 persist(事务段)耗时。 */
private long[] measureSingleSubmission() {
prepareNanos.set(0);
persistNanos.set(0);
lenient().doAnswer(invocation -> {
long start = System.nanoTime();
prepareNanos.set(System.nanoTime() - start);
return STORED_CHUNK_POINTER;
}).when(transientPayloadStorageService).storeChunkPayloadVersioned(
eq(SimilarAsinTaskService.MODULE_TYPE), eq(TASK_ID), anyString(), any(), anyString());
lenient().doAnswer(invocation -> {
long start = System.nanoTime();
persistNanos.set(System.nanoTime() - start);
return 1;
}).when(taskChunkMapper).insert(any(com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity.class));
long start = System.nanoTime();
service.submitResult(TASK_ID, request());
long totalNanos = System.nanoTime() - start;
return new long[]{totalNanos, prepareNanos.get(), persistNanos.get()};
}
@Test
void txDurationRecorded() {
long[] measured = measureSingleSubmission();
assertTrue(measured[2] > 0, "事务段耗时必须可测量");
assertTrue(measured[1] > 0, "计算段耗时必须可测量");
}
@Test
void prepareAndPersistBothMeasurable() {
long[] measured = measureSingleSubmission();
assertTrue(measured[1] > 0 && measured[2] > 0, "prepare 与 persist 分段记录");
assertTrue(measured[0] >= measured[1] + measured[2], "总耗时 >= 分段之和");
}
@Test
void persistDurationBounded() {
long[] measured = measureSingleSubmission();
// mock 环境:单次事务段(insert+scope+update)上界 100ms(不含 prepare 存储)
assertTrue(measured[2] < 100_000_000L,
"单次事务段超上界: " + (measured[2] / 1_000_000L) + "ms");
}
@Test
void measurementRepeatableWithinTolerance() {
long[] first = measureSingleSubmission();
long[] second = measureSingleSubmission();
// 两次测量的事务段应在 ±3× 内(mock 环境抖动容忍)
long lower = Math.min(first[2], second[2]);
long upper = Math.max(first[2], second[2]);
assertTrue(upper <= Math.max(1, lower * 3),
"两次测量偏差过大: " + lower + " vs " + upper);
}
@Test
void twoHundredChunkLoadBounded() {
long start = System.nanoTime();
for (int i = 0; i < 200; i++) {
service.submitResult(TASK_ID, request());
}
long totalMillis = (System.nanoTime() - start) / 1_000_000L;
// mock 环境 200 分片总耗时上界 10s(每提交 50ms 平均)
assertTrue(totalMillis < 10_000L, "200 分片负载超上界: " + totalMillis + "ms");
}
@Test
void noRegressionAgainstDocumentedBaseline() throws Exception {
assertTrue(Files.isRegularFile(BENCHMARK_DOC), "基准文档缺失: " + BENCHMARK_DOC);
String doc = Files.readString(BENCHMARK_DOC);
long baselineMillis = java.util.regex.Pattern.compile("单次事务段基线[:]\\s*(\\d+)")
.matcher(doc).results()
.map(m -> Long.parseLong(m.group(1)))
.findFirst()
.orElseThrow(() -> new AssertionError("基准文档缺少单次事务段基线"));
long[] measured = measureSingleSubmission();
long currentMillis = measured[2] / 1_000_000L;
assertTrue(currentMillis <= Math.max(1, baselineMillis * 3),
"事务段劣化: 基线 " + baselineMillis + "ms vs 当前 " + currentMillis + "ms");
}
@Test
void benchmarkDocumentedWithBaseline() throws Exception {
assertTrue(Files.isRegularFile(BENCHMARK_DOC), "基准文档缺失");
String doc = Files.readString(BENCHMARK_DOC);
assertTrue(doc.contains("单次事务段基线"), "文档必须记录单次事务段基线");
assertTrue(doc.contains("200 分片"), "文档必须记录 200 分片负载基线");
assertTrue(doc.contains("task-138"), "文档标注任务来源");
}
@Test
void totalDurationReasonablePerSubmission() {
long[] measured = measureSingleSubmission();
assertTrue(measured[0] < 500_000_000L,
"单次提交总耗时超上界: " + (measured[0] / 1_000_000L) + "ms");
}
private SimilarAsinSubmitResultRequest request() {
SimilarAsinSubmitResultRequest request = new SimilarAsinSubmitResultRequest();
request.setSubmissionId("similar-asin-3333");
request.setChunkIndex(0);
request.setChunkTotal(1);
request.setDone(false);
return request;
}
private FileTaskEntity runningTask() {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType(SimilarAsinTaskService.MODULE_TYPE);
task.setStatus("RUNNING");
task.setUserId(7L);
task.setResultJson("{\"parsedPayloadRef\":\"" + PARSED_POINTER + "\",\"ownerInstanceId\":\"instance-a\"}");
return task;
}
}
@@ -0,0 +1,170 @@
package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskFileJobMapper;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-159:已完成任务活跃 Job 巡检契约(plan 09)。
* 终态任务仍有 RUNNING/PENDING Job → 检出;运行中任务的活跃 Job 正常不报;
* 只读;可重复。
*/
@ExtendWith(MockitoExtension.class)
class CompletedTaskActiveJobInspectorTest {
private static final Long TERMINAL_TASK_ID = 5001L;
private static final Long RUNNING_TASK_ID = 5101L;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private TaskFileJobMapper taskFileJobMapper;
private CompletedTaskActiveJobInspector inspector;
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, TaskFileJobEntity.class);
}
@BeforeEach
void setUp() {
inspector = new CompletedTaskActiveJobInspector(fileTaskMapper, taskFileJobMapper);
lenient().when(taskFileJobMapper.selectList(any())).thenReturn(List.of());
}
private FileTaskEntity task(Long id, String status) {
FileTaskEntity task = new FileTaskEntity();
task.setId(id);
task.setModuleType("SIMILAR_ASIN");
task.setStatus(status);
return task;
}
private TaskFileJobEntity job(Long id, Long taskId, String status) {
TaskFileJobEntity job = new TaskFileJobEntity();
job.setId(id);
job.setTaskId(taskId);
job.setStatus(status);
job.setJobType("ASSEMBLE_RESULT");
return job;
}
@Test
void activeJobAfterTerminalTaskDetected() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of(task(TERMINAL_TASK_ID, "SUCCESS")));
when(taskFileJobMapper.selectList(any()))
.thenReturn(List.of(job(901L, TERMINAL_TASK_ID, "RUNNING")));
var report = inspector.inspectTerminalTasksWithActiveJobs(50);
assertEquals(1, report.entries().size());
assertEquals(TERMINAL_TASK_ID, report.entries().getFirst().taskId());
assertEquals("RUNNING", report.entries().getFirst().jobStatus());
}
@Test
void terminalTaskWithoutJobsIsFine() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of(task(TERMINAL_TASK_ID, "SUCCESS")));
var report = inspector.inspectTerminalTasksWithActiveJobs(50);
assertTrue(report.isEmpty());
}
@Test
void runningTaskWithActiveJobIsNormal() {
// 巡检只查终态任务:RUNNING 任务的活跃 Job 不在报表范围
when(fileTaskMapper.selectList(any())).thenReturn(List.of());
inspector.inspectTerminalTasksWithActiveJobs(50);
var wrapper = org.mockito.ArgumentCaptor.forClass(com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper.class);
verify(fileTaskMapper).selectList(wrapper.capture());
assertTrue(wrapper.getValue().getSqlSegment().contains("IN"), "只查终态任务");
assertFalse(wrapper.getValue().getSqlSegment().contains("RUNNING"));
}
@Test
void stuckJobAfterTerminalReported() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of(task(TERMINAL_TASK_ID, "FAILED")));
when(taskFileJobMapper.selectList(any()))
.thenReturn(List.of(job(902L, TERMINAL_TASK_ID, "PENDING")));
var report = inspector.inspectTerminalTasksWithActiveJobs(50);
assertEquals(1, report.entries().size(), "卡死 PENDING Job 同样检出");
}
@Test
void emptyReportWhenNoTerminalTasks() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of());
var report = inspector.inspectTerminalTasksWithActiveJobs(50);
assertTrue(report.isEmpty());
}
@Test
void readOnlyDoesNotModifyAnything() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of(task(TERMINAL_TASK_ID, "SUCCESS")));
when(taskFileJobMapper.selectList(any()))
.thenReturn(List.of(job(903L, TERMINAL_TASK_ID, "RUNNING")));
inspector.inspectTerminalTasksWithActiveJobs(50);
verify(taskFileJobMapper, never()).update(any(), any());
verify(taskFileJobMapper, never()).updateById(any(TaskFileJobEntity.class));
verify(taskFileJobMapper, never()).delete(any());
}
@Test
void reportIncludesDetails() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of(task(TERMINAL_TASK_ID, "SUCCESS")));
when(taskFileJobMapper.selectList(any()))
.thenReturn(List.of(job(904L, TERMINAL_TASK_ID, "RUNNING")));
var report = inspector.inspectTerminalTasksWithActiveJobs(50);
CompletedTaskActiveJobInspector.ActiveJobEntry entry = report.entries().getFirst();
assertEquals(TERMINAL_TASK_ID, entry.taskId());
assertEquals("SIMILAR_ASIN", entry.moduleType());
assertEquals(904L, entry.jobId());
assertEquals("ASSEMBLE_RESULT", entry.jobType());
assertNotNull(entry.jobStatus());
}
@Test
void rerunIsSafeAndStable() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of(task(TERMINAL_TASK_ID, "SUCCESS")));
when(taskFileJobMapper.selectList(any()))
.thenReturn(List.of(job(905L, TERMINAL_TASK_ID, "PENDING")));
var first = inspector.inspectTerminalTasksWithActiveJobs(50);
var second = inspector.inspectTerminalTasksWithActiveJobs(50);
assertEquals(first.entries().size(), second.entries().size());
assertEquals(first.entries().getFirst().jobId(), second.entries().getFirst().jobId());
}
}
@@ -0,0 +1,150 @@
package com.nanri.aiimage.modules.task.service;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.InspectionProperties;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.modules.file.service.TempOrphanInspector;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import java.util.Arrays;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-165:巡检只读契约(plan 09)。
* 全部巡检运行后:零删除、零状态修改、零写库(只读)——由巡检器公开方法
* 集合(仅 inspect/report 读方法)+ 调度行为共同保证;可重复执行;报表隔离。
*/
@ExtendWith(MockitoExtension.class)
class InspectionReadOnlyTest {
@Mock private DistributedJobLockService distributedJobLockService;
@Mock private StorageProperties storageProperties;
@Mock private TempOrphanInspector tempOrphanInspector;
@Mock private OrphanJobInspector orphanJobInspector;
@Mock private TaskResultMissingInspector taskResultMissingInspector;
@Mock private ResultFileMissingInspector resultFileMissingInspector;
@Mock private CompletedTaskActiveJobInspector completedTaskActiveJobInspector;
private InspectionProperties properties;
private InspectionScheduler scheduler;
@BeforeEach
void setUp() {
properties = new InspectionProperties();
properties.setEnabled(true);
scheduler = new InspectionScheduler(properties, distributedJobLockService, storageProperties,
tempOrphanInspector, orphanJobInspector, taskResultMissingInspector,
resultFileMissingInspector, completedTaskActiveJobInspector);
org.mockito.Mockito.lenient().when(distributedJobLockService.tryLock(any(), any()))
.thenReturn(mock(DistributedJobLockService.LockHandle.class));
org.mockito.Mockito.lenient().when(orphanJobInspector.inspectOrphanJobs(anyInt())).thenReturn(
new OrphanJobInspector.OrphanJobReport(List.of()));
org.mockito.Mockito.lenient().when(taskResultMissingInspector.inspectTasksMissingResult(anyInt())).thenReturn(
new TaskResultMissingInspector.MissingResultReport(List.of()));
org.mockito.Mockito.lenient().when(resultFileMissingInspector.inspectResultsMissingFile(anyInt())).thenReturn(
new ResultFileMissingInspector.MissingFileReport(List.of()));
org.mockito.Mockito.lenient().when(completedTaskActiveJobInspector.inspectTerminalTasksWithActiveJobs(anyInt())).thenReturn(
new CompletedTaskActiveJobInspector.ActiveJobReport(List.of()));
org.mockito.Mockito.lenient().when(storageProperties.getLocalTempDir()).thenReturn("target/nonexistent-inspection-dir");
}
private static List<String> publicMethodNames(Class<?> type) {
return Arrays.stream(type.getMethods())
.filter(method -> method.getDeclaringClass() != Object.class)
.map(java.lang.reflect.Method::getName)
.toList();
}
@Test
void noDeleteMethodsOnInspectors() {
for (Class<?> type : List.of(OrphanJobInspector.class, TaskResultMissingInspector.class,
ResultFileMissingInspector.class, CompletedTaskActiveJobInspector.class)) {
List<String> names = publicMethodNames(type);
assertTrue(names.stream().noneMatch(name ->
name.contains("delete") || name.contains("remove") || name.contains("clean")),
type.getSimpleName() + " 不得有删除方法: " + names);
}
}
@Test
void noStateChangeMethodsOnInspectors() {
for (Class<?> type : List.of(OrphanJobInspector.class, TaskResultMissingInspector.class,
ResultFileMissingInspector.class, CompletedTaskActiveJobInspector.class)) {
List<String> names = publicMethodNames(type);
assertTrue(names.stream().noneMatch(name ->
name.contains("update") || name.contains("mark") || name.contains("set")),
type.getSimpleName() + " 不得有状态修改方法: " + names);
}
}
@Test
void noWriteMethodsOnInspectors() {
for (Class<?> type : List.of(OrphanJobInspector.class, TaskResultMissingInspector.class,
ResultFileMissingInspector.class, CompletedTaskActiveJobInspector.class)) {
List<String> names = publicMethodNames(type);
assertTrue(names.stream().noneMatch(name ->
name.contains("insert") || name.contains("write") || name.contains("save")),
type.getSimpleName() + " 不得有写库方法: " + names);
}
}
@Test
void repeatableReadsProduceSameReports() {
scheduler.runInspections();
scheduler.runInspections();
verify(orphanJobInspector, org.mockito.Mockito.times(2)).inspectOrphanJobs(anyInt());
verify(taskResultMissingInspector, org.mockito.Mockito.times(2)).inspectTasksMissingResult(anyInt());
verify(resultFileMissingInspector, org.mockito.Mockito.times(2)).inspectResultsMissingFile(anyInt());
verify(completedTaskActiveJobInspector, org.mockito.Mockito.times(2))
.inspectTerminalTasksWithActiveJobs(anyInt());
}
@Test
void reportsAreIsolatedPerInspection() {
scheduler.runInspections();
verify(orphanJobInspector).inspectOrphanJobs(anyInt());
verify(taskResultMissingInspector).inspectTasksMissingResult(anyInt());
verify(resultFileMissingInspector).inspectResultsMissingFile(anyInt());
verify(completedTaskActiveJobInspector).inspectTerminalTasksWithActiveJobs(anyInt());
}
@Test
void emptyReportsDoNotWrite() {
scheduler.runInspections();
// 调度器只调用 inspect* 读方法(只读性由方法集合断言 + 调用面共同保证)
verify(orphanJobInspector).inspectOrphanJobs(anyInt());
verify(taskResultMissingInspector).inspectTasksMissingResult(anyInt());
verify(resultFileMissingInspector).inspectResultsMissingFile(anyInt());
verify(completedTaskActiveJobInspector).inspectTerminalTasksWithActiveJobs(anyInt());
}
@Test
void inspectionIsReadOnlyByConstruction() {
// 调度器调用面仅含 inspect*(无清理/写入调用)
scheduler.runInspections();
verify(orphanJobInspector).inspectOrphanJobs(anyInt());
}
@Test
void noFileSystemSideEffectsWhenDirMissing() {
scheduler.runInspections();
verify(tempOrphanInspector, never()).inspectOrphanFiles(any(), any(), any());
}
}
@@ -0,0 +1,188 @@
package com.nanri.aiimage.modules.task.service;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.InspectionProperties;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.modules.file.service.TempOrphanInspector;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import java.time.Duration;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-161:巡检任务开关契约(plan 09)。
* 默认 disabled 不执行;启用后执行并持分布式锁(双实例不重复);单个巡检异常
* 不阻断其余;limit 传入各巡检。
*/
@ExtendWith(MockitoExtension.class)
class InspectionSchedulerTest {
@Mock private DistributedJobLockService distributedJobLockService;
@Mock private StorageProperties storageProperties;
@Mock private TempOrphanInspector tempOrphanInspector;
@Mock private OrphanJobInspector orphanJobInspector;
@Mock private TaskResultMissingInspector taskResultMissingInspector;
@Mock private ResultFileMissingInspector resultFileMissingInspector;
@Mock private CompletedTaskActiveJobInspector completedTaskActiveJobInspector;
private InspectionProperties properties;
private InspectionScheduler scheduler;
@BeforeEach
void setUp() {
properties = new InspectionProperties();
scheduler = new InspectionScheduler(properties, distributedJobLockService, storageProperties,
tempOrphanInspector, orphanJobInspector, taskResultMissingInspector,
resultFileMissingInspector, completedTaskActiveJobInspector);
}
@Test
void disabledByDefaultDoesNotRun() {
assertFalse(properties.isEnabled(), "默认 disabled");
scheduler.runInspections();
verify(orphanJobInspector, never()).inspectOrphanJobs(anyInt());
verify(distributedJobLockService, never()).tryLock(any(), any());
}
@Test
void enabledRunsInspections() {
properties.setEnabled(true);
when(distributedJobLockService.tryLock(any(), any()))
.thenReturn(mock(DistributedJobLockService.LockHandle.class));
when(orphanJobInspector.inspectOrphanJobs(anyInt())).thenReturn(
new OrphanJobInspector.OrphanJobReport(java.util.List.of()));
when(taskResultMissingInspector.inspectTasksMissingResult(anyInt())).thenReturn(
new TaskResultMissingInspector.MissingResultReport(java.util.List.of()));
when(resultFileMissingInspector.inspectResultsMissingFile(anyInt())).thenReturn(
new ResultFileMissingInspector.MissingFileReport(java.util.List.of()));
when(completedTaskActiveJobInspector.inspectTerminalTasksWithActiveJobs(anyInt())).thenReturn(
new CompletedTaskActiveJobInspector.ActiveJobReport(java.util.List.of()));
when(storageProperties.getLocalTempDir()).thenReturn("target/nonexistent-inspection-dir");
scheduler.runInspections();
verify(orphanJobInspector).inspectOrphanJobs(200);
verify(taskResultMissingInspector).inspectTasksMissingResult(200);
verify(resultFileMissingInspector).inspectResultsMissingFile(200);
verify(completedTaskActiveJobInspector).inspectTerminalTasksWithActiveJobs(200);
}
@Test
void scheduleIntervalConfigured() {
assertEquals("0 0 3 * * *", properties.getCron(), "默认调度 cron");
properties.setCron("0 */30 * * * *");
assertEquals("0 */30 * * * *", properties.getCron(), "cron 可配置");
}
@Test
void dualInstanceSkipsWhenLockBusy() {
properties.setEnabled(true);
when(distributedJobLockService.tryLock(any(), any())).thenReturn(null);
scheduler.runInspections();
verify(orphanJobInspector, never()).inspectOrphanJobs(anyInt());
}
@Test
void oneInspectionFailureDoesNotBlockOthers() {
properties.setEnabled(true);
when(distributedJobLockService.tryLock(any(), any()))
.thenReturn(mock(DistributedJobLockService.LockHandle.class));
doThrow(new RuntimeException("orphan job inspector down"))
.when(orphanJobInspector).inspectOrphanJobs(anyInt());
when(taskResultMissingInspector.inspectTasksMissingResult(anyInt())).thenReturn(
new TaskResultMissingInspector.MissingResultReport(java.util.List.of()));
when(resultFileMissingInspector.inspectResultsMissingFile(anyInt())).thenReturn(
new ResultFileMissingInspector.MissingFileReport(java.util.List.of()));
when(completedTaskActiveJobInspector.inspectTerminalTasksWithActiveJobs(anyInt())).thenReturn(
new CompletedTaskActiveJobInspector.ActiveJobReport(java.util.List.of()));
when(storageProperties.getLocalTempDir()).thenReturn("target/nonexistent-inspection-dir");
scheduler.runInspections();
verify(taskResultMissingInspector).inspectTasksMissingResult(200);
verify(resultFileMissingInspector).inspectResultsMissingFile(200);
verify(completedTaskActiveJobInspector).inspectTerminalTasksWithActiveJobs(200);
}
@Test
void limitIsApplied() {
properties.setEnabled(true);
properties.setLimit(50);
when(distributedJobLockService.tryLock(any(), any()))
.thenReturn(mock(DistributedJobLockService.LockHandle.class));
when(orphanJobInspector.inspectOrphanJobs(anyInt())).thenReturn(
new OrphanJobInspector.OrphanJobReport(java.util.List.of()));
when(taskResultMissingInspector.inspectTasksMissingResult(anyInt())).thenReturn(
new TaskResultMissingInspector.MissingResultReport(java.util.List.of()));
when(resultFileMissingInspector.inspectResultsMissingFile(anyInt())).thenReturn(
new ResultFileMissingInspector.MissingFileReport(java.util.List.of()));
when(completedTaskActiveJobInspector.inspectTerminalTasksWithActiveJobs(anyInt())).thenReturn(
new CompletedTaskActiveJobInspector.ActiveJobReport(java.util.List.of()));
when(storageProperties.getLocalTempDir()).thenReturn("target/nonexistent-inspection-dir");
scheduler.runInspections();
verify(orphanJobInspector).inspectOrphanJobs(50);
verify(taskResultMissingInspector).inspectTasksMissingResult(50);
}
@Test
void lockAcquiredWithTtl() {
properties.setEnabled(true);
when(distributedJobLockService.tryLock("temp-file-inspection", Duration.ofMinutes(10)))
.thenReturn(mock(DistributedJobLockService.LockHandle.class));
when(orphanJobInspector.inspectOrphanJobs(anyInt())).thenReturn(
new OrphanJobInspector.OrphanJobReport(java.util.List.of()));
when(taskResultMissingInspector.inspectTasksMissingResult(anyInt())).thenReturn(
new TaskResultMissingInspector.MissingResultReport(java.util.List.of()));
when(resultFileMissingInspector.inspectResultsMissingFile(anyInt())).thenReturn(
new ResultFileMissingInspector.MissingFileReport(java.util.List.of()));
when(completedTaskActiveJobInspector.inspectTerminalTasksWithActiveJobs(anyInt())).thenReturn(
new CompletedTaskActiveJobInspector.ActiveJobReport(java.util.List.of()));
when(storageProperties.getLocalTempDir()).thenReturn("target/nonexistent-inspection-dir");
scheduler.runInspections();
verify(distributedJobLockService).tryLock("temp-file-inspection", Duration.ofMinutes(10));
}
@Test
void rerunIsSafe() {
properties.setEnabled(true);
when(distributedJobLockService.tryLock(any(), any()))
.thenReturn(mock(DistributedJobLockService.LockHandle.class));
when(orphanJobInspector.inspectOrphanJobs(anyInt())).thenReturn(
new OrphanJobInspector.OrphanJobReport(java.util.List.of()));
when(taskResultMissingInspector.inspectTasksMissingResult(anyInt())).thenReturn(
new TaskResultMissingInspector.MissingResultReport(java.util.List.of()));
when(resultFileMissingInspector.inspectResultsMissingFile(anyInt())).thenReturn(
new ResultFileMissingInspector.MissingFileReport(java.util.List.of()));
when(completedTaskActiveJobInspector.inspectTerminalTasksWithActiveJobs(anyInt())).thenReturn(
new CompletedTaskActiveJobInspector.ActiveJobReport(java.util.List.of()));
when(storageProperties.getLocalTempDir()).thenReturn("target/nonexistent-inspection-dir");
scheduler.runInspections();
scheduler.runInspections();
verify(orphanJobInspector, org.mockito.Mockito.times(2)).inspectOrphanJobs(anyInt());
assertTrue(properties.isEnabled());
}
}
@@ -0,0 +1,175 @@
package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskFileJobMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import java.time.LocalDateTime;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-156:孤立 Job 巡检报表契约(plan 09)。
* task_file_job 无对应 task/result 的孤儿 Job 清单;只读不修改状态;可重复。
*/
@ExtendWith(MockitoExtension.class)
class OrphanJobInspectorTest {
private static final Long ORPHAN_JOB_ID = 9001L;
private static final Long ORPHAN_TASK_ID = 9002L;
private static final Long ORPHAN_RESULT_ID = 9003L;
private static final Long VALID_JOB_ID = 9101L;
private static final Long VALID_TASK_ID = 9102L;
private static final Long VALID_RESULT_ID = 9103L;
@Mock private TaskFileJobMapper taskFileJobMapper;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
private OrphanJobInspector inspector;
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, TaskFileJobEntity.class);
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, FileResultEntity.class);
}
@BeforeEach
void setUp() {
inspector = new OrphanJobInspector(taskFileJobMapper, fileTaskMapper, fileResultMapper);
lenient().when(fileTaskMapper.selectBatchIds(any())).thenReturn(List.of());
lenient().when(fileResultMapper.selectBatchIds(any())).thenReturn(List.of());
}
private TaskFileJobEntity job(Long id, Long taskId, Long resultId, String status) {
TaskFileJobEntity job = new TaskFileJobEntity();
job.setId(id);
job.setTaskId(taskId);
job.setResultId(resultId);
job.setModuleType("SIMILAR_ASIN");
job.setJobType("ASSEMBLE_RESULT");
job.setStatus(status);
job.setUpdatedAt(LocalDateTime.now());
return job;
}
@Test
void orphanJobDetectedWhenTaskMissing() {
when(taskFileJobMapper.selectList(any()))
.thenReturn(List.of(job(ORPHAN_JOB_ID, ORPHAN_TASK_ID, ORPHAN_RESULT_ID, "PENDING")));
var report = inspector.inspectOrphanJobs(50);
assertEquals(1, report.entries().size());
assertEquals(ORPHAN_JOB_ID, report.entries().getFirst().jobId());
}
@Test
void validJobNotReported() {
when(taskFileJobMapper.selectList(any()))
.thenReturn(List.of(job(VALID_JOB_ID, VALID_TASK_ID, VALID_RESULT_ID, "RUNNING")));
FileTaskEntity task = new FileTaskEntity();
task.setId(VALID_TASK_ID);
FileResultEntity result = new FileResultEntity();
result.setId(VALID_RESULT_ID);
when(fileTaskMapper.selectBatchIds(any())).thenReturn(List.of(task));
when(fileResultMapper.selectBatchIds(any())).thenReturn(List.of(result));
var report = inspector.inspectOrphanJobs(50);
assertTrue(report.isEmpty(), "task/result 齐全的 Job 不得进报表");
}
@Test
void reportOutputsDetails() {
when(taskFileJobMapper.selectList(any()))
.thenReturn(List.of(job(ORPHAN_JOB_ID, ORPHAN_TASK_ID, ORPHAN_RESULT_ID, "FAILED")));
var report = inspector.inspectOrphanJobs(50);
OrphanJobInspector.OrphanJobEntry entry = report.entries().getFirst();
assertEquals(ORPHAN_JOB_ID, entry.jobId());
assertEquals(ORPHAN_TASK_ID, entry.taskId());
assertEquals(ORPHAN_RESULT_ID, entry.resultId());
assertEquals("FAILED", entry.status());
assertNotNull(entry.updatedAt());
assertFalse(entry.moduleType().isBlank());
}
@Test
void readOnlyDoesNotModifyJobs() {
when(taskFileJobMapper.selectList(any()))
.thenReturn(List.of(job(ORPHAN_JOB_ID, ORPHAN_TASK_ID, ORPHAN_RESULT_ID, "PENDING")));
inspector.inspectOrphanJobs(50);
verify(taskFileJobMapper, never()).update(any(), any());
verify(taskFileJobMapper, never()).delete(any());
verify(taskFileJobMapper, never()).updateById(any(TaskFileJobEntity.class));
}
@Test
void retryExhaustedOrphanStillReported() {
TaskFileJobEntity exhausted = job(ORPHAN_JOB_ID, ORPHAN_TASK_ID, ORPHAN_RESULT_ID, "FAILED");
exhausted.setRetryCount(5);
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(exhausted));
var report = inspector.inspectOrphanJobs(50);
assertEquals(1, report.entries().size(), "重试耗尽孤儿同样检出");
}
@Test
void emptyReportWhenNoJobs() {
when(taskFileJobMapper.selectList(any())).thenReturn(List.of());
var report = inspector.inspectOrphanJobs(50);
assertTrue(report.isEmpty());
}
@Test
void rerunIsSafeAndStable() {
when(taskFileJobMapper.selectList(any()))
.thenReturn(List.of(job(ORPHAN_JOB_ID, ORPHAN_TASK_ID, ORPHAN_RESULT_ID, "PENDING")));
var first = inspector.inspectOrphanJobs(50);
var second = inspector.inspectOrphanJobs(50);
assertEquals(first.entries().size(), second.entries().size());
assertEquals(first.entries().getFirst().jobId(), second.entries().getFirst().jobId());
}
@Test
void limitIsClamped() {
when(taskFileJobMapper.selectList(any())).thenReturn(List.of());
inspector.inspectOrphanJobs(0);
inspector.inspectOrphanJobs(10_000);
verify(taskFileJobMapper, org.mockito.Mockito.times(2)).selectList(any());
}
}
@@ -0,0 +1,144 @@
package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-158:结果存在文件缺失巡检契约(plan 09)。
* resultFileUrl 对应文件在 OSS 不存在 → 检出;存在不报;空白 URL 不告警;
* 只读;可重复。
*/
@ExtendWith(MockitoExtension.class)
class ResultFileMissingInspectorTest {
private static final Long RESULT_ID = 6001L;
private static final Long TASK_ID = 6002L;
@Mock private FileResultMapper fileResultMapper;
@Mock private OssStorageService ossStorageService;
private ResultFileMissingInspector inspector;
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileResultEntity.class);
}
@BeforeEach
void setUp() {
inspector = new ResultFileMissingInspector(fileResultMapper, ossStorageService);
lenient().when(fileResultMapper.selectList(any())).thenReturn(List.of());
}
private FileResultEntity result(Long id, String url) {
FileResultEntity result = new FileResultEntity();
result.setId(id);
result.setTaskId(TASK_ID);
result.setModuleType("SIMILAR_ASIN");
result.setResultFileUrl(url);
return result;
}
@Test
void missingFileDetected() {
when(fileResultMapper.selectList(any())).thenReturn(List.of(result(RESULT_ID, "oss://result/missing.xlsx")));
var report = inspector.inspectResultsMissingFile(50, url -> false);
assertEquals(1, report.entries().size());
assertEquals(RESULT_ID, report.entries().getFirst().resultId());
}
@Test
void existingFileNotReported() {
when(fileResultMapper.selectList(any())).thenReturn(List.of(result(RESULT_ID, "oss://result/ok.xlsx")));
var report = inspector.inspectResultsMissingFile(50, url -> true);
assertTrue(report.isEmpty());
}
@Test
void blankUrlHandledAsNormal() {
when(fileResultMapper.selectList(any())).thenReturn(List.of(result(RESULT_ID, " ")));
var report = inspector.inspectResultsMissingFile(50, url -> false);
assertTrue(report.isEmpty(), "空白 URL 视为未生成文件状态,不告警");
}
@Test
void ossCheckUsedByDefault() {
when(fileResultMapper.selectList(any())).thenReturn(List.of(result(RESULT_ID, "oss://result/x.xlsx")));
when(ossStorageService.objectExists("oss://result/x.xlsx")).thenReturn(true);
var report = inspector.inspectResultsMissingFile(50);
assertTrue(report.isEmpty(), "默认走 OSS 存在性校验");
}
@Test
void emptyReportWhenNoResults() {
var report = inspector.inspectResultsMissingFile(50, url -> false);
assertTrue(report.isEmpty());
}
@Test
void readOnlyDoesNotModifyAnything() {
when(fileResultMapper.selectList(any())).thenReturn(List.of(result(RESULT_ID, "oss://result/m.xlsx")));
inspector.inspectResultsMissingFile(50, url -> false);
verify(fileResultMapper, never()).update(any(), any());
verify(fileResultMapper, never()).delete(any());
verify(fileResultMapper, never()).updateById(any(FileResultEntity.class));
}
@Test
void reportIncludesDetails() {
when(fileResultMapper.selectList(any())).thenReturn(List.of(result(RESULT_ID, "oss://result/m.xlsx")));
var report = inspector.inspectResultsMissingFile(50, url -> false);
ResultFileMissingInspector.MissingFileEntry entry = report.entries().getFirst();
assertEquals(RESULT_ID, entry.resultId());
assertEquals(TASK_ID, entry.taskId());
assertEquals("SIMILAR_ASIN", entry.moduleType());
assertEquals("oss://result/m.xlsx", entry.resultFileUrl());
assertNotNull(entry.resultFileUrl());
}
@Test
void rerunIsSafeAndStable() {
when(fileResultMapper.selectList(any())).thenReturn(List.of(result(RESULT_ID, "oss://result/m.xlsx")));
var first = inspector.inspectResultsMissingFile(50, url -> false);
var second = inspector.inspectResultsMissingFile(50, url -> false);
assertEquals(first.entries().size(), second.entries().size());
assertEquals(first.entries().getFirst().resultId(), second.entries().getFirst().resultId());
}
}
@@ -86,9 +86,10 @@ class TaskFileJobClaimTest {
}
private void stubClaimedRows(List<TaskFileJobEntity> claimed) {
when(taskFileJobMapper.selectById(any())).thenAnswer(invocation -> {
Long id = invocation.getArgument(0);
return claimed.stream().filter(job -> job.getId().equals(id)).findFirst().orElse(null);
when(taskFileJobMapper.selectBatchIds(any())).thenAnswer(invocation -> {
@SuppressWarnings("unchecked")
java.util.Collection<Object> ids = invocation.getArgument(0);
return claimed.stream().filter(job -> ids.contains(job.getId())).toList();
});
}
@@ -0,0 +1,217 @@
package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.nanri.aiimage.modules.task.mapper.TaskFileJobMapper;
import com.nanri.aiimage.modules.task.model.dto.TaskFileJobDispatchEvent;
import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.context.ApplicationEventPublisher;
import java.time.LocalDateTime;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.isNull;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-122 N+1 批量化修复:claimCandidates 与 resetStuckRunningJobsDetailed
* 由逐行 selectById 改为 selectBatchIds 批量回读 + Map 装配(行为等价,查询次数下降)。
*/
@ExtendWith(MockitoExtension.class)
class TaskFileJobN1BatchFixTest {
@Mock private TaskFileJobMapper taskFileJobMapper;
@Mock private ApplicationEventPublisher applicationEventPublisher;
@BeforeAll
static void initializeTableInfo() {
TableInfoHelper.initTableInfo(
new MapperBuilderAssistant(new MybatisConfiguration(), ""),
TaskFileJobEntity.class);
}
@Test
void claimCandidatesFetchesAllClaimedInOneBatchQuery() {
TaskFileJobEntity c1 = job(101L, "PENDING");
TaskFileJobEntity c2 = job(102L, "PENDING");
TaskFileJobEntity claimed1 = job(101L, "RUNNING");
TaskFileJobEntity claimed2 = job(102L, "RUNNING");
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(c1, c2));
when(taskFileJobMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
when(taskFileJobMapper.selectBatchIds(any())).thenReturn(List.of(claimed1, claimed2));
TaskFileJobService service = new TaskFileJobService(taskFileJobMapper, applicationEventPublisher);
List<TaskFileJobEntity> claimed = service.claimRunnableJobs(20);
assertEquals(List.of(101L, 102L), claimed.stream().map(TaskFileJobEntity::getId).toList());
assertBatchIds(101L, 102L);
}
@Test
void claimCandidatesSingleRowStillWorks() {
TaskFileJobEntity c1 = job(201L, "PENDING");
TaskFileJobEntity claimed1 = job(201L, "RUNNING");
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(c1));
when(taskFileJobMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
when(taskFileJobMapper.selectBatchIds(any())).thenReturn(List.of(claimed1));
TaskFileJobService service = new TaskFileJobService(taskFileJobMapper, applicationEventPublisher);
List<TaskFileJobEntity> claimed = service.claimRunnableJobs(20);
assertEquals(List.of(201L), claimed.stream().map(TaskFileJobEntity::getId).toList());
assertBatchIds(201L);
}
@Test
void claimCandidatesEmptyCandidatesDoesNotQuery() {
when(taskFileJobMapper.selectList(any())).thenReturn(List.of());
TaskFileJobService service = new TaskFileJobService(taskFileJobMapper, applicationEventPublisher);
List<TaskFileJobEntity> claimed = service.claimRunnableJobs(20);
assertTrue(claimed.isEmpty());
verify(taskFileJobMapper, never()).update(any(), any());
verify(taskFileJobMapper, never()).selectBatchIds(any());
}
@Test
void claimCandidatesSkipsUnclaimedIdsFromBatch() {
TaskFileJobEntity c1 = job(301L, "PENDING");
TaskFileJobEntity c2 = job(302L, "PENDING");
TaskFileJobEntity claimed2 = job(302L, "RUNNING");
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(c1, c2));
when(taskFileJobMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(0, 1);
when(taskFileJobMapper.selectBatchIds(any())).thenReturn(List.of(claimed2));
TaskFileJobService service = new TaskFileJobService(taskFileJobMapper, applicationEventPublisher);
List<TaskFileJobEntity> claimed = service.claimRunnableJobs(20);
assertEquals(List.of(302L), claimed.stream().map(TaskFileJobEntity::getId).toList());
assertBatchIds(302L);
}
@Test
void claimCandidatesNullGuardSkipsBrokenCandidates() {
TaskFileJobEntity broken = job(null, "PENDING");
TaskFileJobEntity c2 = job(402L, "PENDING");
TaskFileJobEntity claimed2 = job(402L, "RUNNING");
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(broken, c2));
when(taskFileJobMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
when(taskFileJobMapper.selectBatchIds(any())).thenReturn(List.of(claimed2));
TaskFileJobService service = new TaskFileJobService(taskFileJobMapper, applicationEventPublisher);
List<TaskFileJobEntity> claimed = service.claimRunnableJobs(20);
assertEquals(List.of(402L), claimed.stream().map(TaskFileJobEntity::getId).toList());
assertBatchIds(402L);
}
@Test
void claimCandidatesQueryCountIsReducedToSingleBatch() {
TaskFileJobEntity c1 = job(501L, "PENDING");
TaskFileJobEntity c2 = job(502L, "PENDING");
TaskFileJobEntity c3 = job(503L, "PENDING");
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(c1, c2, c3));
when(taskFileJobMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
when(taskFileJobMapper.selectBatchIds(any()))
.thenReturn(List.of(job(501L, "RUNNING"), job(502L, "RUNNING"), job(503L, "RUNNING")));
TaskFileJobService service = new TaskFileJobService(taskFileJobMapper, applicationEventPublisher);
service.claimRunnableJobs(20);
verify(taskFileJobMapper, times(1)).selectList(any());
verify(taskFileJobMapper, times(1)).selectBatchIds(any());
verify(taskFileJobMapper, never()).selectById(any());
}
@Test
void claimCandidatesDropsRowsThatAreNotRunningAfterRefresh() {
TaskFileJobEntity c1 = job(601L, "PENDING");
TaskFileJobEntity c2 = job(602L, "PENDING");
TaskFileJobEntity stale = job(601L, "PENDING");
TaskFileJobEntity claimed2 = job(602L, "RUNNING");
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(c1, c2));
when(taskFileJobMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
when(taskFileJobMapper.selectBatchIds(any())).thenReturn(List.of(stale, claimed2));
TaskFileJobService service = new TaskFileJobService(taskFileJobMapper, applicationEventPublisher);
List<TaskFileJobEntity> claimed = service.claimRunnableJobs(20);
assertEquals(List.of(602L), claimed.stream().map(TaskFileJobEntity::getId).toList());
}
@Test
void resetStuckBatchesRefreshAndPreservesEventOrder() {
TaskFileJobEntity zombie = job(701L, "PENDING");
TaskFileJobEntity requeue = job(702L, "RUNNING");
TaskFileJobEntity exhausted = job(703L, "FAILED");
exhausted.setRetryCount(TaskFileJobService.MAX_RETRY_COUNT);
TaskFileJobEntity zombieRefreshed = job(701L, "PENDING");
TaskFileJobEntity requeueRefreshed = job(702L, "PENDING");
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(zombie, requeue, exhausted));
when(taskFileJobMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
when(taskFileJobMapper.selectBatchIds(any())).thenReturn(List.of(zombieRefreshed, requeueRefreshed));
TaskFileJobService service = new TaskFileJobService(taskFileJobMapper, applicationEventPublisher);
TaskFileJobService.StuckJobResetResult result = service.resetStuckRunningJobsDetailed(30, 20);
assertEquals(1, result.resetCount());
assertEquals(List.of(703L), result.exhaustedJobs().stream().map(TaskFileJobEntity::getId).toList());
ArgumentCaptor<Object> events = ArgumentCaptor.forClass(Object.class);
verify(applicationEventPublisher, times(2)).publishEvent(events.capture());
List<Object> published = events.getAllValues();
assertEquals(701L, ((TaskFileJobDispatchEvent) published.get(0)).jobId());
assertEquals(702L, ((TaskFileJobDispatchEvent) published.get(1)).jobId());
assertBatchIds(701L, 702L);
}
@Test
void resetStuckUsesSingleBatchRefreshWithoutSelectById() {
TaskFileJobEntity running = job(801L, "RUNNING");
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(running));
when(taskFileJobMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
when(taskFileJobMapper.selectBatchIds(any())).thenReturn(List.of(job(801L, "PENDING")));
TaskFileJobService service = new TaskFileJobService(taskFileJobMapper, applicationEventPublisher);
service.resetStuckRunningJobsDetailed(30, 20);
verify(taskFileJobMapper, times(1)).selectList(any());
verify(taskFileJobMapper, times(1)).selectBatchIds(any());
verify(taskFileJobMapper, never()).selectById(any());
}
@SuppressWarnings({"rawtypes", "unchecked"})
private void assertBatchIds(Long... expected) {
ArgumentCaptor<java.util.Collection> captor = ArgumentCaptor.forClass(java.util.Collection.class);
verify(taskFileJobMapper).selectBatchIds(captor.capture());
assertEquals(List.of(expected), List.copyOf(captor.getValue()));
}
private static TaskFileJobEntity job(Long id, String status) {
TaskFileJobEntity job = new TaskFileJobEntity();
job.setId(id);
job.setTaskId(20553L);
job.setResultId(23110L);
job.setModuleType("SIMILAR_ASIN");
job.setJobType("ASSEMBLE_RESULT");
job.setStatus(status);
job.setRetryCount(0);
job.setUpdatedAt(LocalDateTime.now());
return job;
}
}
@@ -92,9 +92,10 @@ class TaskFileJobOwnerColumnTest {
}
private void stubClaimedRows(List<TaskFileJobEntity> claimed) {
when(taskFileJobMapper.selectById(any())).thenAnswer(invocation -> {
Long id = invocation.getArgument(0);
return claimed.stream().filter(job -> job.getId().equals(id)).findFirst().orElse(null);
when(taskFileJobMapper.selectBatchIds(any())).thenAnswer(invocation -> {
@SuppressWarnings("unchecked")
java.util.Collection<Object> ids = invocation.getArgument(0);
return claimed.stream().filter(job -> ids.contains(job.getId())).toList();
});
}
@@ -47,7 +47,7 @@ class TaskFileJobServiceTest {
pending.setStatus("PENDING");
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(running));
when(taskFileJobMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
when(taskFileJobMapper.selectById(101L)).thenReturn(pending);
when(taskFileJobMapper.selectBatchIds(any())).thenReturn(List.of(pending));
TaskFileJobService service = new TaskFileJobService(taskFileJobMapper, applicationEventPublisher);
TaskFileJobService.StuckJobResetResult result = service.resetStuckRunningJobsDetailed(30, 20);
@@ -69,7 +69,7 @@ class TaskFileJobServiceTest {
failed.setErrorMessage("result file job timeout");
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(running));
when(taskFileJobMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
when(taskFileJobMapper.selectById(102L)).thenReturn(failed);
when(taskFileJobMapper.selectBatchIds(any())).thenReturn(List.of(failed));
TaskFileJobService service = new TaskFileJobService(taskFileJobMapper, applicationEventPublisher);
TaskFileJobService.StuckJobResetResult result = service.resetStuckRunningJobsDetailed(30, 20);
@@ -0,0 +1,152 @@
package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.config.TransientStorageProperties;
import com.nanri.aiimage.modules.file.service.object.RustfsObjectStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-150:清理前引用检查契约(plan 09)。
* payload 被 biz_task_chunk / biz_task_scope_state 引用则不清理;
* chunk 仅剩自身一行(count=1)不算共享引用;查询异常保守保留;
* 检查只读无副作用;候选值批量反查。
*/
@ExtendWith(MockitoExtension.class)
class TaskPayloadReferenceCheckTest {
private static final String RUSTFS_VALUE = "\"rustfs:payload-key\"";
private static final String RUSTFS_POINTER = "rustfs:payload-key";
@Mock private RustfsObjectStorageService rustfsObjectStorageService;
@Mock private TaskChunkMapper taskChunkMapper;
@Mock private TaskScopeStateMapper taskScopeStateMapper;
@Mock private OssStorageService ossStorageService;
private TransientPayloadStorageService service;
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, com.nanri.aiimage.modules.task.model.entity.TaskChunkEntity.class);
TableInfoHelper.initTableInfo(assistant, com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity.class);
}
@BeforeEach
void setUp() {
TransientStorageProperties transientProperties = new TransientStorageProperties();
transientProperties.setEnabled(true);
StorageProperties storageProperties = new StorageProperties();
storageProperties.setLocalTempDir("target/tmp-refcheck");
service = new TransientPayloadStorageService(
transientProperties,
storageProperties,
rustfsObjectStorageService,
ossStorageService,
new ObjectMapper(),
new InstanceMetadata("test-instance"),
taskChunkMapper,
taskScopeStateMapper);
org.mockito.Mockito.lenient().when(taskChunkMapper.selectCount(any())).thenReturn(0L);
org.mockito.Mockito.lenient().when(taskScopeStateMapper.selectCount(any())).thenReturn(0L);
}
@Test
void referencedByChunkSharedIsKept() {
when(taskChunkMapper.selectCount(any())).thenReturn(2L);
service.deletePayloadIfPresent(RUSTFS_VALUE);
verify(rustfsObjectStorageService, never()).deleteObject(any());
}
@Test
void unreferencedCandidateIsDeleted() {
service.deletePayloadIfPresent(RUSTFS_VALUE);
verify(rustfsObjectStorageService).deleteObject("payload-key");
}
@Test
void referencedByScopeStateIsKept() {
when(taskScopeStateMapper.selectCount(any())).thenReturn(1L);
service.deletePayloadIfPresent(RUSTFS_VALUE);
verify(rustfsObjectStorageService, never()).deleteObject(any());
}
@Test
void singleChunkRowIsOwnRowNotSharedReference() {
when(taskChunkMapper.selectCount(any())).thenReturn(1L);
service.deletePayloadIfPresent(RUSTFS_VALUE);
verify(rustfsObjectStorageService).deleteObject("payload-key");
}
@Test
void checkFailureKeepsPayloadConservatively() {
when(taskChunkMapper.selectCount(any())).thenThrow(new RuntimeException("db down"));
service.deletePayloadIfPresent(RUSTFS_VALUE);
verify(rustfsObjectStorageService, never()).deleteObject(any());
}
@Test
void checkIsReadOnlyNoSideEffects() {
service.deletePayloadIfPresent(RUSTFS_VALUE);
verify(taskChunkMapper).selectCount(any());
verify(taskScopeStateMapper).selectCount(any());
verify(rustfsObjectStorageService).deleteObject("payload-key");
}
@Test
void checkFailureOnScopeAlsoKeeps() {
when(taskScopeStateMapper.selectCount(any())).thenThrow(new RuntimeException("db down"));
service.deletePayloadIfPresent(RUSTFS_VALUE);
verify(rustfsObjectStorageService, never()).deleteObject(any());
}
@Test
void batchCandidatesQueriedInOneInClause() {
service.deletePayloadIfPresent(RUSTFS_VALUE);
@SuppressWarnings({"rawtypes", "unchecked"})
ArgumentCaptor<LambdaQueryWrapper> captor = ArgumentCaptor.forClass(LambdaQueryWrapper.class);
verify(taskChunkMapper).selectCount(captor.capture());
String segment = captor.getValue().getSqlSegment();
assertTrue(segment.contains("IN"), "候选值必须批量 IN 查询: " + segment);
// 候选集 = value + pointerjson(pointer) 与 value 同值去重),IN 参数非空
assertTrue(!captor.getValue().getParamNameValuePairs().isEmpty(),
"IN 查询必须携带候选参数");
}
}
@@ -0,0 +1,154 @@
package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import java.time.LocalDateTime;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* task-157:任务存在结果缺失巡检契约(plan 09)。
* 终态(SUCCESS/FAILED)任务无 file_result 行 → 检出;运行中忽略;只读;可重复。
*/
@ExtendWith(MockitoExtension.class)
class TaskResultMissingInspectorTest {
private static final Long MISSING_TASK_ID = 7001L;
private static final Long WITH_RESULT_TASK_ID = 7101L;
private static final Long RUNNING_TASK_ID = 7201L;
@Mock private FileTaskMapper fileTaskMapper;
@Mock private FileResultMapper fileResultMapper;
private TaskResultMissingInspector inspector;
@BeforeAll
static void initializeMybatisMetadata() {
MapperBuilderAssistant assistant = new MapperBuilderAssistant(new MybatisConfiguration(), "");
TableInfoHelper.initTableInfo(assistant, FileTaskEntity.class);
TableInfoHelper.initTableInfo(assistant, FileResultEntity.class);
}
@BeforeEach
void setUp() {
inspector = new TaskResultMissingInspector(fileTaskMapper, fileResultMapper);
lenient().when(fileResultMapper.selectList(any())).thenReturn(List.of());
}
private FileTaskEntity task(Long id, String status) {
FileTaskEntity task = new FileTaskEntity();
task.setId(id);
task.setModuleType("SIMILAR_ASIN");
task.setStatus(status);
task.setUpdatedAt(LocalDateTime.now());
return task;
}
@Test
void missingResultDetected() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of(task(MISSING_TASK_ID, "SUCCESS")));
var report = inspector.inspectTasksMissingResult(50);
assertEquals(1, report.entries().size());
assertEquals(MISSING_TASK_ID, report.entries().getFirst().taskId());
}
@Test
void taskWithResultNotReported() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of(task(WITH_RESULT_TASK_ID, "SUCCESS")));
FileResultEntity result = new FileResultEntity();
result.setTaskId(WITH_RESULT_TASK_ID);
when(fileResultMapper.selectList(any())).thenReturn(List.of(result));
var report = inspector.inspectTasksMissingResult(50);
assertTrue(report.isEmpty());
}
@Test
void successWithoutResultIsReported() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of(task(MISSING_TASK_ID, "SUCCESS")));
var report = inspector.inspectTasksMissingResult(50);
assertEquals("SUCCESS", report.entries().getFirst().status());
}
@Test
void runningTasksAreIgnored() {
// 巡检只查终态任务:RUNNING 不在查询范围内(mapper 断言终态 IN 条件)
when(fileTaskMapper.selectList(any())).thenReturn(List.of());
inspector.inspectTasksMissingResult(50);
var wrapper = org.mockito.ArgumentCaptor.forClass(com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper.class);
verify(fileTaskMapper).selectList(wrapper.capture());
assertTrue(wrapper.getValue().getSqlSegment().contains("IN"), "只查终态任务: " + wrapper.getValue().getSqlSegment());
assertFalse(wrapper.getValue().getSqlSegment().contains("RUNNING"), "运行中任务不参与巡检");
}
@Test
void emptyReportWhenNoTasks() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of());
var report = inspector.inspectTasksMissingResult(50);
assertTrue(report.isEmpty());
}
@Test
void readOnlyDoesNotModifyAnything() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of(task(MISSING_TASK_ID, "FAILED")));
inspector.inspectTasksMissingResult(50);
verify(fileTaskMapper, never()).update(any(), any());
verify(fileResultMapper, never()).update(any(), any());
verify(fileTaskMapper, never()).delete(any());
}
@Test
void reportIncludesDetails() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of(task(MISSING_TASK_ID, "FAILED")));
var report = inspector.inspectTasksMissingResult(50);
TaskResultMissingInspector.MissingResultEntry entry = report.entries().getFirst();
assertEquals(MISSING_TASK_ID, entry.taskId());
assertEquals("SIMILAR_ASIN", entry.moduleType());
assertEquals("FAILED", entry.status());
assertNotNull(entry.updatedAt());
}
@Test
void rerunIsSafeAndStable() {
when(fileTaskMapper.selectList(any())).thenReturn(List.of(task(MISSING_TASK_ID, "SUCCESS")));
var first = inspector.inspectTasksMissingResult(50);
var second = inspector.inspectTasksMissingResult(50);
assertEquals(first.entries().size(), second.entries().size());
assertEquals(first.entries().getFirst().taskId(), second.entries().getFirst().taskId());
}
}
-44
View File
@@ -3478,19 +3478,6 @@ def delete_invalid_asin_data(item_id):
# ---------- 店铺管理 ----------
def _format_shop_manage_item(item):
latest_check = item.get('latestCheck')
if isinstance(latest_check, dict):
latest_check = {
'id': latest_check.get('id'),
'status': latest_check.get('status') or '',
'detail': latest_check.get('detail') or '',
'client_host': latest_check.get('clientHost') or '',
'try_requested_at': (latest_check.get('tryRequestedAt') or '').replace('T', ' ')[:19],
'check_started_at': (latest_check.get('checkStartedAt') or '').replace('T', ' ')[:19],
'check_finished_at': (latest_check.get('checkFinishedAt') or '').replace('T', ' ')[:19],
}
else:
latest_check = None
return {
'id': item.get('id'),
'group_id': item.get('groupId'),
@@ -3500,7 +3487,6 @@ def _format_shop_manage_item(item):
'zn_username': item.get('znUsername') or '',
'account': item.get('account') or '',
'password': item.get('passwordMasked') or '',
'latest_check': latest_check,
'created_at': (item.get('createdAt') or '').replace('T', ' ')[:16],
'updated_at': (item.get('updatedAt') or '').replace('T', ' ')[:16],
}
@@ -3580,36 +3566,6 @@ def list_shop_manages():
})
@admin_api.route('/shop-manage/<int:item_id>/credential-check', methods=['POST'])
@login_required
def create_shop_credential_check(item_id):
role, current_row, denied = _ensure_backend_menu_access('shop-manage')
if denied:
return denied
shop_name = (request.args.get('shop_name') or '').strip()
if not shop_name:
return jsonify({'success': False, 'error': '店铺名不能为空'}), 400
internal_token = _resolve_internal_token()
if not internal_token:
return jsonify({'success': False, 'error': '内部凭据服务未配置'}), 503
result, error_response, status = _proxy_backend_java(
'POST',
'/api/admin/shop-credential-checks',
json_data={'shopName': shop_name},
headers={'X-Internal-Token': internal_token},
)
if error_response is not None:
return error_response, status
check = result.get('data') or {}
return jsonify({
'success': True,
'msg': '检测任务已创建,客户端将在 1 分钟内执行',
'check': {'id': check.get('id'), 'status': check.get('status') or 'PENDING'},
})
@admin_api.route('/shop-manage/<int:item_id>/credential')
@login_required
def get_shop_manage_credential(item_id):

Some files were not shown because too many files have changed in this diff Show More