refactor+perf+fix: G5 模块边界 SPI 化、C5 流式解析、C6/C7 查询优化、D13 队列持久化、A3/A4/A7 鉴权

模块边界(G5 / G7)
- 新增 task/spi/TaskModuleHeartbeatSpi + 13 个模块实现:TaskHeartbeatService 不再 import 任何
  业务模块(原先注入 12 个 CacheService 并用 switch 分发);启动校验重复注册
- 新增 task/spi/BrandTaskHeartbeatSpi(品牌任务心跳/中断)、BrandTaskStaleRepairSpi(陈旧修复)、
  CollectDataItemCleanupSpi(历史清理):跨模块 Mapper 操作收回业务模块
- G7:10 个被跨模块借用的 productrisk VO 迁至 common/model/vo
- 架构棘轮收紧:TASK_TO_BUSINESS_BASELINE 119 → 6(实测)
- 新增 TaskModuleHeartbeatSpiCoverageTest(moduleType 覆盖与拼写)

性能与容量(C5/C6/C7/D13)
- C5 流式解析:SkipPriceAsinService(含两行表头语义)、AppearancePatentExcelParser、
  BrandTaskService、DeleteBrandRunService、LocalFileStorageService.getExcelInfo 改 ExcelStreamReader;
  行数上限改为迭代中生效
- C6 去重总数据列表:关键字改前缀匹配(命中 uk_data_value);V124 删除永不生效的 idx_country
- C7 撞款扫描:按店逐批取数(索引前缀),不再全表 GROUP BY + JOIN + 全量拉内存
- D13 待删对象本地日志 PendingDeleteJournal(启动回放 + 收敛重写),RustfsDeleteRetryService 与
  TransientPayloadDeleteOrchestrator 接入;异步删除失败对象写回日志
- V123 删除 biz_file_result 两个被复合索引覆盖的单列索引

安全(A3/A4/A7 + 守卫名单)
- A4 数字人版本写操作要求管理员;A7 视频密钥按登录身份(超管例外)
- A3 上传接口加危险扩展名黑名单(可配置)
- AdminApiGuardFilter 用户态名单补 /api/ziniao(controller 已 requireAdmin,此处为开关打开后的第二层)

容量(明细表保留期)
- 新增 ShopDataCrawlItemRetentionService:biz_shop_data_crawl_item 按保留期(默认 30 天)分批清理
  (该表此前无任何清理策略,是增长最快的表),job 锁 + 单轮批次上限
This commit is contained in:
2026-09-14 06:46:27 +08:00
parent d87d76398e
commit f4ee7e16f0
86 changed files with 3191 additions and 952 deletions
@@ -0,0 +1,90 @@
package com.nanri.aiimage.common.util;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* 待删对象日志:入队追加、启动回放、收敛重写。对应审查项 D13(删除补偿队列重启即丢)。
*/
class PendingDeleteJournalTest {
@TempDir
Path tempDir;
@Test
void recordThenReadAllReturnsAdmittedKeysInOrder() {
PendingDeleteJournal journal = new PendingDeleteJournal(tempDir.resolve("nested/queue.log"), 100);
journal.record("a/1.bin");
journal.record("b/2.bin");
journal.record("a/1.bin");
assertEquals(List.of("a/1.bin", "b/2.bin"), journal.readAll());
}
@Test
void rewriteKeepsOnlyRemainingAndDropsFileWhenEmpty() throws Exception {
Path path = tempDir.resolve("queue.log");
PendingDeleteJournal journal = new PendingDeleteJournal(path, 100);
journal.record("keep.bin");
journal.record("done.bin");
journal.rewrite(List.of("keep.bin"));
assertEquals(List.of("keep.bin"), journal.readAll());
journal.rewrite(List.of());
assertFalse(Files.exists(path), "队列收敛为空后日志文件应删除");
assertEquals(List.of(), journal.readAll());
}
@Test
void missingFileYieldsEmptyInsteadOfThrowing() {
PendingDeleteJournal journal = new PendingDeleteJournal(tempDir.resolve("not-created.log"), 10);
assertEquals(List.of(), journal.readAll());
}
@Test
void blankAndMultilineRecordsAreSanitizedOrIgnored() throws Exception {
Path path = tempDir.resolve("queue.log");
PendingDeleteJournal journal = new PendingDeleteJournal(path, 10);
journal.record(null);
journal.record(" ");
journal.record("bad\nkey.bin");
journal.record(" good.bin ");
assertEquals(List.of("bad key.bin", "good.bin"), journal.readAll());
// 单行一条记录:换行被替换,不会破坏行结构
assertEquals(2, Files.readAllLines(path, StandardCharsets.UTF_8).size());
}
@Test
void corruptedLineIsIgnoredWhileValidOnesSurvive() throws Exception {
Path path = tempDir.resolve("queue.log");
Files.createDirectories(tempDir);
Files.writeString(path, "\n \nvalid.bin\n", StandardCharsets.UTF_8);
PendingDeleteJournal journal = new PendingDeleteJournal(path, 10);
assertEquals(List.of("valid.bin"), journal.readAll());
}
@Test
void rewriteRespectsMaxEntriesCap() {
PendingDeleteJournal journal = new PendingDeleteJournal(tempDir.resolve("queue.log"), 2);
journal.rewrite(List.of("k1.bin", "k2.bin", "k3.bin"));
assertEquals(List.of("k1.bin", "k2.bin"), journal.readAll());
assertTrue(journal.readAll().size() <= 2);
}
}