refactor+perf+fix: G5 模块边界 SPI 化、C5 流式解析、C6/C7 查询优化、D13 队列持久化、A3/A4/A7 鉴权

模块边界(G5 / G7)
- 新增 task/spi/TaskModuleHeartbeatSpi + 13 个模块实现:TaskHeartbeatService 不再 import 任何
  业务模块(原先注入 12 个 CacheService 并用 switch 分发);启动校验重复注册
- 新增 task/spi/BrandTaskHeartbeatSpi(品牌任务心跳/中断)、BrandTaskStaleRepairSpi(陈旧修复)、
  CollectDataItemCleanupSpi(历史清理):跨模块 Mapper 操作收回业务模块
- G7:10 个被跨模块借用的 productrisk VO 迁至 common/model/vo
- 架构棘轮收紧:TASK_TO_BUSINESS_BASELINE 119 → 6(实测)
- 新增 TaskModuleHeartbeatSpiCoverageTest(moduleType 覆盖与拼写)

性能与容量(C5/C6/C7/D13)
- C5 流式解析:SkipPriceAsinService(含两行表头语义)、AppearancePatentExcelParser、
  BrandTaskService、DeleteBrandRunService、LocalFileStorageService.getExcelInfo 改 ExcelStreamReader;
  行数上限改为迭代中生效
- C6 去重总数据列表:关键字改前缀匹配(命中 uk_data_value);V124 删除永不生效的 idx_country
- C7 撞款扫描:按店逐批取数(索引前缀),不再全表 GROUP BY + JOIN + 全量拉内存
- D13 待删对象本地日志 PendingDeleteJournal(启动回放 + 收敛重写),RustfsDeleteRetryService 与
  TransientPayloadDeleteOrchestrator 接入;异步删除失败对象写回日志
- V123 删除 biz_file_result 两个被复合索引覆盖的单列索引

安全(A3/A4/A7 + 守卫名单)
- A4 数字人版本写操作要求管理员;A7 视频密钥按登录身份(超管例外)
- A3 上传接口加危险扩展名黑名单(可配置)
- AdminApiGuardFilter 用户态名单补 /api/ziniao(controller 已 requireAdmin,此处为开关打开后的第二层)

容量(明细表保留期)
- 新增 ShopDataCrawlItemRetentionService:biz_shop_data_crawl_item 按保留期(默认 30 天)分批清理
  (该表此前无任何清理策略,是增长最快的表),job 锁 + 单轮批次上限
This commit is contained in:
2026-09-14 06:46:27 +08:00
parent d87d76398e
commit f4ee7e16f0
86 changed files with 3191 additions and 952 deletions
@@ -34,7 +34,9 @@ class ArchitectureBoundaryTest {
* 本次以实测值 119 为准恢复告警能力;经 git diff 核验,当日的 task 模块改动
* (引用计数按 task 收敛、快照 upsert 缓存、心跳条件更新、锁续期重试等)未新增任何业务依赖。
*/
private static final int TASK_TO_BUSINESS_BASELINE = 119;
// 2026-09-14 心跳/陈旧修复/历史清理 SPI 化后实测 6(G5:task 只依赖 task.spi 下接口,
// 业务模块各自实现);继续按棘轮收敛,不允许回升
private static final int TASK_TO_BUSINESS_BASELINE = 6;
private static volatile JavaClasses cached;
@@ -0,0 +1,90 @@
package com.nanri.aiimage.common.util;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* 待删对象日志:入队追加、启动回放、收敛重写。对应审查项 D13(删除补偿队列重启即丢)。
*/
class PendingDeleteJournalTest {
@TempDir
Path tempDir;
@Test
void recordThenReadAllReturnsAdmittedKeysInOrder() {
PendingDeleteJournal journal = new PendingDeleteJournal(tempDir.resolve("nested/queue.log"), 100);
journal.record("a/1.bin");
journal.record("b/2.bin");
journal.record("a/1.bin");
assertEquals(List.of("a/1.bin", "b/2.bin"), journal.readAll());
}
@Test
void rewriteKeepsOnlyRemainingAndDropsFileWhenEmpty() throws Exception {
Path path = tempDir.resolve("queue.log");
PendingDeleteJournal journal = new PendingDeleteJournal(path, 100);
journal.record("keep.bin");
journal.record("done.bin");
journal.rewrite(List.of("keep.bin"));
assertEquals(List.of("keep.bin"), journal.readAll());
journal.rewrite(List.of());
assertFalse(Files.exists(path), "队列收敛为空后日志文件应删除");
assertEquals(List.of(), journal.readAll());
}
@Test
void missingFileYieldsEmptyInsteadOfThrowing() {
PendingDeleteJournal journal = new PendingDeleteJournal(tempDir.resolve("not-created.log"), 10);
assertEquals(List.of(), journal.readAll());
}
@Test
void blankAndMultilineRecordsAreSanitizedOrIgnored() throws Exception {
Path path = tempDir.resolve("queue.log");
PendingDeleteJournal journal = new PendingDeleteJournal(path, 10);
journal.record(null);
journal.record(" ");
journal.record("bad\nkey.bin");
journal.record(" good.bin ");
assertEquals(List.of("bad key.bin", "good.bin"), journal.readAll());
// 单行一条记录:换行被替换,不会破坏行结构
assertEquals(2, Files.readAllLines(path, StandardCharsets.UTF_8).size());
}
@Test
void corruptedLineIsIgnoredWhileValidOnesSurvive() throws Exception {
Path path = tempDir.resolve("queue.log");
Files.createDirectories(tempDir);
Files.writeString(path, "\n \nvalid.bin\n", StandardCharsets.UTF_8);
PendingDeleteJournal journal = new PendingDeleteJournal(path, 10);
assertEquals(List.of("valid.bin"), journal.readAll());
}
@Test
void rewriteRespectsMaxEntriesCap() {
PendingDeleteJournal journal = new PendingDeleteJournal(tempDir.resolve("queue.log"), 2);
journal.rewrite(List.of("k1.bin", "k2.bin", "k3.bin"));
assertEquals(List.of("k1.bin", "k2.bin"), journal.readAll());
assertTrue(journal.readAll().size() <= 2);
}
}
@@ -0,0 +1,112 @@
package com.nanri.aiimage.modules.deletebrand.service;
import com.nanri.aiimage.common.module.TaskModuleRegistry;
import com.nanri.aiimage.modules.appearancepatent.service.AppearancePatentTaskService;
import com.nanri.aiimage.modules.brand.service.BrandTaskService;
import com.nanri.aiimage.modules.shopdatacrawl.service.ShopDataCrawlTaskService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskService;
import com.nanri.aiimage.config.ModuleCleanupProperties;
import com.nanri.aiimage.config.TaskFileJobConfig;
import org.junit.jupiter.api.Test;
import java.util.List;
import java.util.Map;
import java.util.Set;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.mock;
/**
* 模块清单覆盖面自检(2026-09 全维度审查 G6)。
*
* 新增工具模块此前要同时改 5 处枚举,其中 4 处没有自检,漏改表现为"功能静默不生效"。
* 本测试把「注册表 = 单一来源」这件事钉住:数据驱动的三处必须与注册表完全一致,
* 委派式判死巡检线必须为注册表标记的每个模块登记动作。
* 新增模块时若漏改任一处,这里会红。
*/
class TaskModuleCoverageTest {
/** 当前线上模块清单(新增模块时同时更新:注册表 + 本清单 + 对应 Handler) */
private static final Set<String> EXPECTED_MODULES = Set.of(
"PUBLISH", "DEDUPE", "SPLIT", "CONVERT", "DELETE_BRAND", "PRODUCT_RISK_RESOLVE",
"PRICE_TRACK", "SHOP_MATCH", "PATROL_DELETE", "QUERY_ASIN", "WITHDRAW",
"APPEARANCE_PATENT", "SIMILAR_ASIN", "COLLECT_DATA", "SHOP_DATA_CRAWL", "BRAND");
@Test
void registryCoversEveryKnownModuleWithLabel() {
assertEquals(EXPECTED_MODULES, TaskModuleRegistry.moduleTypes());
Map<String, String> labels = TaskModuleRegistry.labels();
assertEquals(EXPECTED_MODULES.size(), labels.size());
for (String moduleType : EXPECTED_MODULES) {
String label = labels.get(moduleType);
assertNotNull(label, "模块缺少中文名: " + moduleType);
assertTrue(!label.isBlank(), "模块中文名不能为空: " + moduleType);
}
assertEquals("跟价", TaskModuleRegistry.labelOf("PRICE_TRACK"));
assertEquals("UNKNOWN_MODULE", TaskModuleRegistry.labelOf("UNKNOWN_MODULE"), "未知模块回退原值");
}
@Test
void resultFileJobWhitelistMatchesRegistry() {
assertEquals(TaskModuleRegistry.resultFileJobModuleTypes(), TaskFileJobConfig.RESULT_FILE_JOB_MODULE_TYPES);
// 纯表格处理模块不产出结果文件作业
assertTrue(!TaskFileJobConfig.RESULT_FILE_JOB_MODULE_TYPES.contains("DEDUPE"));
assertTrue(!TaskFileJobConfig.RESULT_FILE_JOB_MODULE_TYPES.contains("SPLIT"));
assertTrue(!TaskFileJobConfig.RESULT_FILE_JOB_MODULE_TYPES.contains("CONVERT"));
assertTrue(TaskFileJobConfig.RESULT_FILE_JOB_MODULE_TYPES.contains("SHOP_DATA_CRAWL"));
}
@Test
void ageCleanupListMatchesRegistry() {
ModuleCleanupProperties properties = new ModuleCleanupProperties();
assertEquals(TaskModuleRegistry.ageCleanupModuleTypes(), Set.copyOf(properties.getModuleTypes()));
// 店铺数据采集靠每日工作簿自身保留期,不参与按天清理
assertTrue(!properties.getModuleTypes().contains("SHOP_DATA_CRAWL"));
assertTrue(properties.getModuleTypes().contains("DEDUPE"));
}
/**
* 只装配 delegatedStaleChecks 需要的四个业务服务,其余依赖留空——
* 这样测试聚焦"巡检线登记了哪些模块",不牵扯任务锁/缓存等无关依赖。
*/
private static DeleteBrandStaleTaskService serviceForDelegatedChecks() {
return new DeleteBrandStaleTaskService(
null, null, null, null, null, null, null, null, null, null, null, null, null, null, null,
null,
mock(BrandTaskService.class),
mock(AppearancePatentTaskService.class),
mock(SimilarAsinTaskService.class),
null, null, null, null,
mock(ShopDataCrawlTaskService.class));
}
@Test
void delegatedStaleCheckHandlersCoverRegistryFlag() {
DeleteBrandStaleTaskService service = serviceForDelegatedChecks();
Map<String, Runnable> handlers = service.delegatedStaleChecks();
assertEquals(TaskModuleRegistry.delegatedStaleCheckModuleTypes(), handlers.keySet(),
"注册表标记了 delegatedStaleCheck 的模块必须都在巡检线登记动作(反之亦然)");
for (Runnable action : handlers.values()) {
assertNotNull(action);
}
}
@Test
void delegatedStaleCheckIncludesModulesThatHadOwnScheduler() {
DeleteBrandStaleTaskService service = serviceForDelegatedChecks();
List<String> keys = List.copyOf(service.delegatedStaleChecks().keySet());
// 商品管理采集 2026-09 从自带 @Scheduled 并入巡检线(P1-8 双实例判死盲区),不得回退
assertTrue(keys.contains("SHOP_DATA_CRAWL"), "商品管理采集必须留在 stale-check 巡检线内: " + keys);
assertTrue(keys.contains("SIMILAR_ASIN"));
assertTrue(keys.contains("APPEARANCE_PATENT"));
assertTrue(keys.contains("BRAND"));
}
}
@@ -0,0 +1,84 @@
package com.nanri.aiimage.modules.shopdatacrawl.service;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.modules.shopdatacrawl.mapper.ShopDataCrawlItemMapper;
import org.junit.jupiter.api.Test;
import java.time.LocalDate;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* 采集明细表保留期清理(2026-09 全维度审查:该表此前无任何清理策略)。
*/
class ShopDataCrawlItemRetentionServiceTest {
private final ShopDataCrawlItemMapper itemMapper = mock(ShopDataCrawlItemMapper.class);
private final DistributedJobLockService jobLockService = mock(DistributedJobLockService.class);
private ShopDataCrawlItemRetentionService service() {
return new ShopDataCrawlItemRetentionService(itemMapper, jobLockService);
}
private void lockAvailable() {
when(jobLockService.tryLock(anyString(), any()))
.thenReturn(mock(DistributedJobLockService.LockHandle.class));
}
@Test
void deletesInBatchesUntilBatchNotFull() {
lockAvailable();
when(itemMapper.deleteOlderThanBatch(any(LocalDate.class), anyInt()))
.thenReturn(5000, 5000, 120);
service().purgeExpiredItems();
verify(itemMapper, times(3)).deleteOlderThanBatch(any(LocalDate.class), eq(5000));
}
@Test
void stopsAtBatchLimitPerRun() {
lockAvailable();
when(itemMapper.deleteOlderThanBatch(any(LocalDate.class), anyInt())).thenReturn(5000);
service().purgeExpiredItems();
verify(itemMapper, times(ShopDataCrawlItemRetentionService.MAX_BATCHES_PER_RUN))
.deleteOlderThanBatch(any(LocalDate.class), anyInt());
}
@Test
void skipsWhenAnotherInstanceHoldsLock() {
when(jobLockService.tryLock(anyString(), any())).thenReturn(null);
service().purgeExpiredItems();
verify(itemMapper, never()).deleteOlderThanBatch(any(LocalDate.class), anyInt());
}
@Test
void deleteFailureIsLoggedAndDoesNotThrow() {
lockAvailable();
when(itemMapper.deleteOlderThanBatch(any(LocalDate.class), anyInt()))
.thenThrow(new IllegalStateException("db down"));
service().purgeExpiredItems();
verify(itemMapper).deleteOlderThanBatch(any(LocalDate.class), anyInt());
}
@Test
void retentionDaysHasLowerBound() {
ShopDataCrawlItemRetentionService service = service();
assertEquals(30, service.effectiveRetentionDays(), "默认保留 30 天");
}
}
@@ -9,7 +9,7 @@ import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.ModuleCleanupProperties;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.config.TransientStorageProperties;
import com.nanri.aiimage.modules.collectdata.mapper.CollectDataItemMapper;
import com.nanri.aiimage.modules.task.spi.CollectDataItemCleanupSpi;
import com.nanri.aiimage.modules.collectdata.model.entity.CollectDataItemEntity;
import com.nanri.aiimage.modules.file.service.object.RustfsObjectStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
@@ -90,7 +90,7 @@ class ModuleHistoryCleanupKeysetPaginationTest {
private TaskResultPayloadMapper taskResultPayloadMapper;
private TaskScopeStateMapper taskScopeStateMapper;
private TaskChunkMapper taskChunkMapper;
private CollectDataItemMapper collectDataItemMapper;
private CollectDataItemCleanupSpi collectDataItemMapper;
private DistributedJobLockService lockService;
private TransientPayloadStorageService storage;
private RustfsObjectStorageService rustfs;
@@ -129,7 +129,7 @@ class ModuleHistoryCleanupKeysetPaginationTest {
taskResultPayloadMapper = mock(TaskResultPayloadMapper.class);
taskScopeStateMapper = mock(TaskScopeStateMapper.class);
taskChunkMapper = mock(TaskChunkMapper.class);
collectDataItemMapper = mock(CollectDataItemMapper.class);
collectDataItemMapper = mock(CollectDataItemCleanupSpi.class);
StringRedisTemplate redisTemplate = mock(StringRedisTemplate.class);
ValueOperations<String, String> valueOperations = mock(ValueOperations.class);
when(redisTemplate.opsForValue()).thenReturn(valueOperations);
@@ -13,7 +13,7 @@ import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.ModuleCleanupProperties;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.config.TransientStorageProperties;
import com.nanri.aiimage.modules.collectdata.mapper.CollectDataItemMapper;
import com.nanri.aiimage.modules.task.spi.CollectDataItemCleanupSpi;
import com.nanri.aiimage.modules.collectdata.model.entity.CollectDataItemEntity;
import com.nanri.aiimage.modules.file.service.object.RustfsObjectStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
@@ -92,7 +92,7 @@ class ModuleHistoryCleanupLoggingTest {
private TaskResultPayloadMapper taskResultPayloadMapper;
private TaskScopeStateMapper taskScopeStateMapper;
private TaskChunkMapper taskChunkMapper;
private CollectDataItemMapper collectDataItemMapper;
private CollectDataItemCleanupSpi collectDataItemMapper;
private DistributedJobLockService lockService;
private TransientPayloadStorageService storage;
private RustfsObjectStorageService rustfs;
@@ -128,7 +128,7 @@ class ModuleHistoryCleanupLoggingTest {
taskResultPayloadMapper = mock(TaskResultPayloadMapper.class);
taskScopeStateMapper = mock(TaskScopeStateMapper.class);
taskChunkMapper = mock(TaskChunkMapper.class);
collectDataItemMapper = mock(CollectDataItemMapper.class);
collectDataItemMapper = mock(CollectDataItemCleanupSpi.class);
StringRedisTemplate redisTemplate = mock(StringRedisTemplate.class);
ValueOperations<String, String> valueOperations = mock(ValueOperations.class);
when(redisTemplate.opsForValue()).thenReturn(valueOperations);
@@ -9,7 +9,7 @@ import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.ModuleCleanupProperties;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.config.TransientStorageProperties;
import com.nanri.aiimage.modules.collectdata.mapper.CollectDataItemMapper;
import com.nanri.aiimage.modules.task.spi.CollectDataItemCleanupSpi;
import com.nanri.aiimage.modules.collectdata.model.entity.CollectDataItemEntity;
import com.nanri.aiimage.modules.file.service.object.RustfsObjectStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
@@ -86,7 +86,7 @@ class ModuleHistoryCleanupPayloadCleanupTest {
private TaskResultPayloadMapper taskResultPayloadMapper;
private TaskScopeStateMapper taskScopeStateMapper;
private TaskChunkMapper taskChunkMapper;
private CollectDataItemMapper collectDataItemMapper;
private CollectDataItemCleanupSpi collectDataItemMapper;
private DistributedJobLockService lockService;
private TransientPayloadStorageService storage;
private RustfsObjectStorageService rustfs;
@@ -122,7 +122,7 @@ class ModuleHistoryCleanupPayloadCleanupTest {
taskResultPayloadMapper = mock(TaskResultPayloadMapper.class);
taskScopeStateMapper = mock(TaskScopeStateMapper.class);
taskChunkMapper = mock(TaskChunkMapper.class);
collectDataItemMapper = mock(CollectDataItemMapper.class);
collectDataItemMapper = mock(CollectDataItemCleanupSpi.class);
StringRedisTemplate redisTemplate = mock(StringRedisTemplate.class);
ValueOperations<String, String> valueOperations = mock(ValueOperations.class);
when(redisTemplate.opsForValue()).thenReturn(valueOperations);
@@ -1,50 +1,48 @@
package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.baomidou.mybatisplus.core.MybatisConfiguration;
import com.baomidou.mybatisplus.core.metadata.TableInfoHelper;
import com.nanri.aiimage.config.SimilarAsinProperties;
import com.nanri.aiimage.modules.appearancepatent.service.AppearancePatentTaskCacheService;
import com.nanri.aiimage.modules.brand.mapper.BrandCrawlTaskMapper;
import com.nanri.aiimage.modules.brand.service.BrandTaskProgressCacheService;
import com.nanri.aiimage.modules.collectdata.service.CollectDataService;
import com.nanri.aiimage.modules.deletebrand.service.DeleteBrandTaskCacheService;
import com.nanri.aiimage.modules.patroldelete.service.PatrolDeleteTaskCacheService;
import com.nanri.aiimage.modules.pricetrack.service.PriceTrackTaskCacheService;
import com.nanri.aiimage.modules.productrisk.service.ProductRiskTaskCacheService;
import com.nanri.aiimage.modules.collectdata.service.CollectDataTaskHeartbeatSpi;
import com.nanri.aiimage.modules.publish.service.PublishTaskHeartbeatSpi;
import com.nanri.aiimage.modules.publish.service.PublishTaskService;
import com.nanri.aiimage.modules.queryasin.service.QueryAsinTaskCacheService;
import com.nanri.aiimage.modules.shopdatacrawl.service.ShopDataCrawlTaskCacheService;
import com.nanri.aiimage.modules.shopdatacrawl.service.ShopDataCrawlTaskHeartbeatSpi;
import com.nanri.aiimage.modules.shopdatacrawl.service.ShopDataCrawlTaskService;
import com.nanri.aiimage.modules.shopmatch.service.ShopMatchTaskCacheService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskCacheService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskHeartbeatSpi;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.model.dto.TaskHeartbeatRequest;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.vo.TaskHeartbeatVo;
import com.nanri.aiimage.modules.withdraw.service.WithdrawTaskCacheService;
import com.nanri.aiimage.modules.task.spi.BrandTaskHeartbeatSpi;
import com.nanri.aiimage.modules.task.spi.TaskModuleHeartbeatSpi;
import org.apache.ibatis.builder.MapperBuilderAssistant;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.InOrder;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.junit.jupiter.api.Test;
import java.time.LocalDateTime;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.isNull;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoInteractions;
import static org.mockito.Mockito.when;
@ExtendWith(MockitoExtension.class)
/**
* 心跳分发契约(G5 SPI 化后):task 侧只按 moduleType 路由到 {@link TaskModuleHeartbeatSpi},
* 各模块的行为由各自的 SPI 实现负责,因此这里同时覆盖"路由正确"与"实现行为正确"。
*/
class TaskHeartbeatServiceTest {
@BeforeAll
@@ -54,35 +52,30 @@ class TaskHeartbeatServiceTest {
FileTaskEntity.class);
}
@Mock private FileTaskMapper fileTaskMapper;
@Mock private BrandCrawlTaskMapper brandCrawlTaskMapper;
@Mock private ProductRiskTaskCacheService productRiskTaskCacheService;
@Mock private PublishTaskService publishTaskService;
@Mock private PriceTrackTaskCacheService priceTrackTaskCacheService;
@Mock private ShopMatchTaskCacheService shopMatchTaskCacheService;
@Mock private PatrolDeleteTaskCacheService patrolDeleteTaskCacheService;
@Mock private QueryAsinTaskCacheService queryAsinTaskCacheService;
@Mock private ShopDataCrawlTaskCacheService shopDataCrawlTaskCacheService;
@Mock private ShopDataCrawlTaskService shopDataCrawlTaskService;
@Mock private WithdrawTaskCacheService withdrawTaskCacheService;
@Mock private AppearancePatentTaskCacheService appearancePatentTaskCacheService;
@Mock private SimilarAsinTaskCacheService similarAsinTaskCacheService;
@Mock private SimilarAsinProperties similarAsinProperties;
@Mock private TaskFileJobService taskFileJobService;
@Mock private DeleteBrandTaskCacheService deleteBrandTaskCacheService;
@Mock private BrandTaskProgressCacheService brandTaskProgressCacheService;
@Mock private CollectDataService collectDataService;
private final FileTaskMapper fileTaskMapper = mock(FileTaskMapper.class);
private final TaskProgressSnapshotService taskProgressSnapshotService = mock(TaskProgressSnapshotService.class);
private final BrandTaskHeartbeatSpi brandTaskHeartbeatSpi = mock(BrandTaskHeartbeatSpi.class);
@InjectMocks private TaskHeartbeatService service;
private TaskHeartbeatService serviceWith(TaskModuleHeartbeatSpi... handlers) {
return new TaskHeartbeatService(fileTaskMapper, taskProgressSnapshotService, brandTaskHeartbeatSpi,
List.of(handlers));
}
private static FileTaskEntity runningTask(long taskId, String moduleType) {
FileTaskEntity task = new FileTaskEntity();
task.setId(taskId);
task.setModuleType(moduleType);
task.setStatus("RUNNING");
return task;
}
@Test
@SuppressWarnings("unchecked")
void publishHeartbeatTouchesGenericTaskAndModuleProgress() {
long taskId = 20142L;
FileTaskEntity task = new FileTaskEntity();
task.setId(taskId);
task.setModuleType(PublishTaskService.MODULE_TYPE);
task.setStatus("RUNNING");
FileTaskEntity task = runningTask(taskId, PublishTaskService.MODULE_TYPE);
PublishTaskService publishTaskService = mock(PublishTaskService.class);
TaskHeartbeatService service = serviceWith(new PublishTaskHeartbeatSpi(publishTaskService));
TaskHeartbeatRequest request = new TaskHeartbeatRequest();
request.setPhase("dispatching");
@@ -90,7 +83,6 @@ class TaskHeartbeatServiceTest {
request.setTotal(100);
when(fileTaskMapper.selectOne(any(LambdaQueryWrapper.class))).thenReturn(task);
when(brandCrawlTaskMapper.selectOne(any(LambdaQueryWrapper.class))).thenReturn(null);
when(fileTaskMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
TaskHeartbeatVo result = service.heartbeat(taskId, request);
@@ -99,7 +91,7 @@ class TaskHeartbeatServiceTest {
assertEquals(PublishTaskService.MODULE_TYPE, result.getModuleType());
InOrder routingBeforeUpdate = inOrder(publishTaskService, fileTaskMapper);
routingBeforeUpdate.verify(publishTaskService)
.ensureTaskOwnedByCurrentInstance(task, "publish task heartbeat");
.ensureTaskOwnedByCurrentInstance(task, PublishTaskService.MODULE_TYPE + " task heartbeat");
routingBeforeUpdate.verify(fileTaskMapper).update(isNull(), any(LambdaUpdateWrapper.class));
verify(publishTaskService).touchHeartbeat(taskId, request);
}
@@ -108,10 +100,12 @@ class TaskHeartbeatServiceTest {
@SuppressWarnings("unchecked")
void shopDataCrawlHeartbeatChecksOwnerBeforeDatabaseUpdate() {
long taskId = 20143L;
FileTaskEntity task = new FileTaskEntity();
task.setId(taskId);
task.setModuleType("SHOP_DATA_CRAWL");
task.setStatus("RUNNING");
FileTaskEntity task = runningTask(taskId, "SHOP_DATA_CRAWL");
ShopDataCrawlTaskService shopDataCrawlTaskService = mock(ShopDataCrawlTaskService.class);
ShopDataCrawlTaskCacheService cacheService = mock(ShopDataCrawlTaskCacheService.class);
TaskHeartbeatService service = serviceWith(
new ShopDataCrawlTaskHeartbeatSpi(cacheService, shopDataCrawlTaskService));
TaskHeartbeatRequest request = new TaskHeartbeatRequest();
request.setModuleType("SHOP_DATA_CRAWL");
when(fileTaskMapper.selectOne(any(LambdaQueryWrapper.class))).thenReturn(task);
@@ -122,27 +116,27 @@ class TaskHeartbeatServiceTest {
assertTrue(result.isAlive());
InOrder order = inOrder(shopDataCrawlTaskService, fileTaskMapper);
order.verify(shopDataCrawlTaskService)
.ensureTaskOwnedByCurrentInstance(task, "shop data crawl task heartbeat");
.ensureTaskOwnedByCurrentInstance(task, "SHOP_DATA_CRAWL task heartbeat");
order.verify(fileTaskMapper).update(isNull(), any(LambdaUpdateWrapper.class));
verify(shopDataCrawlTaskCacheService).touchTaskHeartbeat(taskId);
verify(shopDataCrawlTaskCacheService).saveTaskCache(task);
verify(brandCrawlTaskMapper, never()).selectOne(any(LambdaQueryWrapper.class));
verify(cacheService).touchTaskHeartbeat(taskId);
verify(cacheService).saveTaskCache(task);
// SHOP_DATA_CRAWL 不需要查品牌任务表(历史行为:跳过品牌查找)
verifyNoInteractions(brandTaskHeartbeatSpi);
}
@Test
@SuppressWarnings("unchecked")
void collectDataHeartbeatForwardsProcessedKeywordProgress() {
long taskId = 21016L;
FileTaskEntity task = new FileTaskEntity();
task.setId(taskId);
task.setModuleType("COLLECT_DATA");
task.setStatus("RUNNING");
FileTaskEntity task = runningTask(taskId, "COLLECT_DATA");
CollectDataService collectDataService = mock(CollectDataService.class);
TaskHeartbeatService service = serviceWith(new CollectDataTaskHeartbeatSpi(collectDataService));
TaskHeartbeatRequest request = new TaskHeartbeatRequest();
request.setCurrent(4);
request.setTotal(19);
when(fileTaskMapper.selectOne(any(LambdaQueryWrapper.class))).thenReturn(task);
when(brandCrawlTaskMapper.selectOne(any(LambdaQueryWrapper.class))).thenReturn(null);
when(fileTaskMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
TaskHeartbeatVo result = service.heartbeat(taskId, request);
@@ -155,21 +149,39 @@ class TaskHeartbeatServiceTest {
@SuppressWarnings("unchecked")
void similarAsinHeartbeatUsesRedisWithoutRefreshingRecentDatabaseCheckpoint() {
long taskId = 20581L;
FileTaskEntity task = new FileTaskEntity();
task.setId(taskId);
task.setModuleType("SIMILAR_ASIN");
task.setStatus("RUNNING");
FileTaskEntity task = runningTask(taskId, "SIMILAR_ASIN");
task.setUpdatedAt(LocalDateTime.now());
SimilarAsinTaskCacheService cacheService = mock(SimilarAsinTaskCacheService.class);
TaskFileJobService taskFileJobService = mock(TaskFileJobService.class);
SimilarAsinProperties properties = mock(SimilarAsinProperties.class);
TaskHeartbeatService service = serviceWith(
new SimilarAsinTaskHeartbeatSpi(cacheService, taskFileJobService, properties));
when(fileTaskMapper.selectOne(any(LambdaQueryWrapper.class))).thenReturn(task);
when(brandCrawlTaskMapper.selectOne(any(LambdaQueryWrapper.class))).thenReturn(null);
when(similarAsinProperties.getDbTaskTouchIntervalMillis()).thenReturn(120000L);
when(properties.getDbTaskTouchIntervalMillis()).thenReturn(120000L);
TaskHeartbeatVo result = service.heartbeat(taskId, new TaskHeartbeatRequest());
assertTrue(result.isAlive());
assertEquals("SIMILAR_ASIN", result.getModuleType());
verify(similarAsinTaskCacheService).touchTaskHeartbeat(taskId);
verify(cacheService).touchTaskHeartbeat(taskId);
verify(taskFileJobService).touchRunningAssembleJobsIfStale(taskId, "SIMILAR_ASIN", 120000L);
// 检查点未到期:不写 biz_file_task.updated_at
verify(fileTaskMapper, never()).update(isNull(), any(LambdaUpdateWrapper.class));
}
@Test
void unknownModuleStillHeartbeatsFileTaskWithoutModuleHandler() {
long taskId = 20150L;
FileTaskEntity task = runningTask(taskId, "UNREGISTERED_MODULE");
TaskHeartbeatService service = serviceWith();
when(fileTaskMapper.selectOne(any(LambdaQueryWrapper.class))).thenReturn(task);
when(fileTaskMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
TaskHeartbeatVo result = service.heartbeat(taskId, new TaskHeartbeatRequest());
assertTrue(result.isAlive(), "未注册 SPI 的模块不应影响通用心跳(保持向后兼容)");
assertEquals("UNREGISTERED_MODULE", result.getModuleType());
}
}
@@ -0,0 +1,96 @@
package com.nanri.aiimage.modules.task.service;
import com.nanri.aiimage.config.SimilarAsinProperties;
import com.nanri.aiimage.modules.appearancepatent.service.AppearancePatentTaskCacheService;
import com.nanri.aiimage.modules.appearancepatent.service.AppearancePatentTaskHeartbeatSpi;
import com.nanri.aiimage.modules.collectdata.service.CollectDataService;
import com.nanri.aiimage.modules.collectdata.service.CollectDataTaskHeartbeatSpi;
import com.nanri.aiimage.modules.deletebrand.service.DeleteBrandTaskCacheService;
import com.nanri.aiimage.modules.deletebrand.service.DeleteBrandTaskHeartbeatSpi;
import com.nanri.aiimage.modules.patroldelete.service.PatrolDeleteTaskCacheService;
import com.nanri.aiimage.modules.patroldelete.service.PatrolDeleteTaskHeartbeatSpi;
import com.nanri.aiimage.modules.pricetrack.service.PriceTrackTaskCacheService;
import com.nanri.aiimage.modules.pricetrack.service.PriceTrackTaskHeartbeatSpi;
import com.nanri.aiimage.modules.productrisk.service.ProductRiskTaskCacheService;
import com.nanri.aiimage.modules.productrisk.service.ProductRiskTaskHeartbeatSpi;
import com.nanri.aiimage.modules.publish.service.PublishTaskHeartbeatSpi;
import com.nanri.aiimage.modules.publish.service.PublishTaskService;
import com.nanri.aiimage.modules.queryasin.service.QueryAsinTaskCacheService;
import com.nanri.aiimage.modules.queryasin.service.QueryAsinTaskHeartbeatSpi;
import com.nanri.aiimage.modules.shopdatacrawl.service.ShopDataCrawlTaskCacheService;
import com.nanri.aiimage.modules.shopdatacrawl.service.ShopDataCrawlTaskHeartbeatSpi;
import com.nanri.aiimage.modules.shopdatacrawl.service.ShopDataCrawlTaskService;
import com.nanri.aiimage.modules.shopmatch.service.ShopMatchTaskCacheService;
import com.nanri.aiimage.modules.shopmatch.service.ShopMatchTaskHeartbeatSpi;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskCacheService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskHeartbeatSpi;
import com.nanri.aiimage.modules.task.spi.TaskModuleHeartbeatSpi;
import com.nanri.aiimage.modules.withdraw.service.WithdrawTaskCacheService;
import com.nanri.aiimage.modules.withdraw.service.WithdrawTaskHeartbeatSpi;
import org.junit.jupiter.api.Test;
import java.util.LinkedHashMap;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.Mockito.mock;
/**
* 模块心跳 SPI 的覆盖面自检(2026-09 全维度审查 G5)。
*
* <p>心跳按 moduleType 路由,moduleType 字符串写错会让该模块的心跳静默失效
* (任务被当成无心跳 → 30 分钟后误判失败)。这里把"哪个实现负责哪个 moduleType"
* 钉死:新增模块漏实现或写错字符串都会红测试。
*/
class TaskModuleHeartbeatSpiCoverageTest {
private static Map<String, TaskModuleHeartbeatSpi> implementationsByModuleType() {
Map<String, TaskModuleHeartbeatSpi> impls = new LinkedHashMap<>();
register(impls, new AppearancePatentTaskHeartbeatSpi(mock(AppearancePatentTaskCacheService.class)));
register(impls, new CollectDataTaskHeartbeatSpi(mock(CollectDataService.class)));
register(impls, new DeleteBrandTaskHeartbeatSpi(mock(DeleteBrandTaskCacheService.class)));
register(impls, new PatrolDeleteTaskHeartbeatSpi(mock(PatrolDeleteTaskCacheService.class)));
register(impls, new PriceTrackTaskHeartbeatSpi(mock(PriceTrackTaskCacheService.class)));
register(impls, new ProductRiskTaskHeartbeatSpi(mock(ProductRiskTaskCacheService.class)));
register(impls, new PublishTaskHeartbeatSpi(mock(PublishTaskService.class)));
register(impls, new QueryAsinTaskHeartbeatSpi(mock(QueryAsinTaskCacheService.class)));
register(impls, new ShopDataCrawlTaskHeartbeatSpi(
mock(ShopDataCrawlTaskCacheService.class), mock(ShopDataCrawlTaskService.class)));
register(impls, new ShopMatchTaskHeartbeatSpi(mock(ShopMatchTaskCacheService.class)));
register(impls, new SimilarAsinTaskHeartbeatSpi(
mock(SimilarAsinTaskCacheService.class), mock(TaskFileJobService.class),
mock(SimilarAsinProperties.class)));
register(impls, new WithdrawTaskHeartbeatSpi(mock(WithdrawTaskCacheService.class)));
return impls;
}
private static void register(Map<String, TaskModuleHeartbeatSpi> impls, TaskModuleHeartbeatSpi impl) {
TaskModuleHeartbeatSpi exists = impls.put(impl.moduleType(), impl);
if (exists != null) {
throw new IllegalStateException("moduleType=" + impl.moduleType() + " 重复实现: "
+ exists.getClass().getName() + " / " + impl.getClass().getName());
}
}
@Test
void everyHeartbeatModuleHasImplementationWithExactModuleType() {
Map<String, TaskModuleHeartbeatSpi> impls = implementationsByModuleType();
assertEquals(12, impls.size(), "有任务心跳的模块都应有 SPI 实现,实际 " + impls.keySet());
for (String moduleType : new String[]{
"APPEARANCE_PATENT", "COLLECT_DATA", "DELETE_BRAND", "PATROL_DELETE", "PRICE_TRACK",
"PRODUCT_RISK_RESOLVE", "PUBLISH", "QUERY_ASIN", "SHOP_DATA_CRAWL", "SHOP_MATCH",
"SIMILAR_ASIN", "WITHDRAW"}) {
assertEquals(true, impls.containsKey(moduleType), "缺少心跳实现: " + moduleType);
}
}
@Test
void moduleTypesAreUppercaseAndUnique() {
for (TaskModuleHeartbeatSpi impl : implementationsByModuleType().values()) {
String moduleType = impl.moduleType();
assertEquals(moduleType.toUpperCase(), moduleType,
"moduleType 必须与 biz_file_task.module_type 一致(全大写): " + moduleType);
}
}
}