refactor+perf+fix: G5 模块边界 SPI 化、C5 流式解析、C6/C7 查询优化、D13 队列持久化、A3/A4/A7 鉴权

模块边界(G5 / G7)
- 新增 task/spi/TaskModuleHeartbeatSpi + 13 个模块实现:TaskHeartbeatService 不再 import 任何
  业务模块(原先注入 12 个 CacheService 并用 switch 分发);启动校验重复注册
- 新增 task/spi/BrandTaskHeartbeatSpi(品牌任务心跳/中断)、BrandTaskStaleRepairSpi(陈旧修复)、
  CollectDataItemCleanupSpi(历史清理):跨模块 Mapper 操作收回业务模块
- G7:10 个被跨模块借用的 productrisk VO 迁至 common/model/vo
- 架构棘轮收紧:TASK_TO_BUSINESS_BASELINE 119 → 6(实测)
- 新增 TaskModuleHeartbeatSpiCoverageTest(moduleType 覆盖与拼写)

性能与容量(C5/C6/C7/D13)
- C5 流式解析:SkipPriceAsinService(含两行表头语义)、AppearancePatentExcelParser、
  BrandTaskService、DeleteBrandRunService、LocalFileStorageService.getExcelInfo 改 ExcelStreamReader;
  行数上限改为迭代中生效
- C6 去重总数据列表:关键字改前缀匹配(命中 uk_data_value);V124 删除永不生效的 idx_country
- C7 撞款扫描:按店逐批取数(索引前缀),不再全表 GROUP BY + JOIN + 全量拉内存
- D13 待删对象本地日志 PendingDeleteJournal(启动回放 + 收敛重写),RustfsDeleteRetryService 与
  TransientPayloadDeleteOrchestrator 接入;异步删除失败对象写回日志
- V123 删除 biz_file_result 两个被复合索引覆盖的单列索引

安全(A3/A4/A7 + 守卫名单)
- A4 数字人版本写操作要求管理员;A7 视频密钥按登录身份(超管例外)
- A3 上传接口加危险扩展名黑名单(可配置)
- AdminApiGuardFilter 用户态名单补 /api/ziniao(controller 已 requireAdmin,此处为开关打开后的第二层)

容量(明细表保留期)
- 新增 ShopDataCrawlItemRetentionService:biz_shop_data_crawl_item 按保留期(默认 30 天)分批清理
  (该表此前无任何清理策略,是增长最快的表),job 锁 + 单轮批次上限
This commit is contained in:
2026-09-14 06:46:27 +08:00
parent d87d76398e
commit f4ee7e16f0
86 changed files with 3191 additions and 952 deletions
@@ -4,8 +4,7 @@ import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.ModuleCleanupProperties;
import com.nanri.aiimage.modules.collectdata.mapper.CollectDataItemMapper;
import com.nanri.aiimage.modules.collectdata.model.entity.CollectDataItemEntity;
import com.nanri.aiimage.modules.task.spi.CollectDataItemCleanupSpi;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
@@ -55,7 +54,7 @@ public class ModuleHistoryCleanupService {
private final TaskResultPayloadMapper taskResultPayloadMapper;
private final TaskScopeStateMapper taskScopeStateMapper;
private final TaskChunkMapper taskChunkMapper;
private final CollectDataItemMapper collectDataItemMapper;
private final CollectDataItemCleanupSpi collectDataItemCleanupSpi;
private final DistributedJobLockService distributedJobLockService;
private final TransientPayloadDeleteOrchestrator transientPayloadDeleteOrchestrator;
private final TransactionTemplate transactionTemplate;
@@ -76,7 +75,7 @@ public class ModuleHistoryCleanupService {
TaskResultPayloadMapper taskResultPayloadMapper,
TaskScopeStateMapper taskScopeStateMapper,
TaskChunkMapper taskChunkMapper,
CollectDataItemMapper collectDataItemMapper,
CollectDataItemCleanupSpi collectDataItemCleanupSpi,
DistributedJobLockService distributedJobLockService,
TransientPayloadDeleteOrchestrator transientPayloadDeleteOrchestrator,
PlatformTransactionManager platformTransactionManager) {
@@ -89,7 +88,7 @@ public class ModuleHistoryCleanupService {
this.taskResultPayloadMapper = taskResultPayloadMapper;
this.taskScopeStateMapper = taskScopeStateMapper;
this.taskChunkMapper = taskChunkMapper;
this.collectDataItemMapper = collectDataItemMapper;
this.collectDataItemCleanupSpi = collectDataItemCleanupSpi;
this.distributedJobLockService = distributedJobLockService;
this.transientPayloadDeleteOrchestrator = transientPayloadDeleteOrchestrator;
this.transactionTemplate = new TransactionTemplate(platformTransactionManager);
@@ -218,8 +217,8 @@ public class ModuleHistoryCleanupService {
.in(TaskChunkEntity::getTaskId, cleanupTaskIds));
if (!collectDataTaskIds.isEmpty()) {
collectDataItemMapper.delete(new LambdaQueryWrapper<CollectDataItemEntity>()
.in(CollectDataItemEntity::getTaskId, collectDataTaskIds));
// 采集明细属于 collectdata 模块(G5:消除 task → 业务依赖,走 SPI)
collectDataItemCleanupSpi.deleteItemsByTaskIds(collectDataTaskIds);
}
fileResultMapper.delete(new LambdaQueryWrapper<FileResultEntity>()
@@ -3,8 +3,7 @@ package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.modules.brand.mapper.BrandCrawlTaskMapper;
import com.nanri.aiimage.modules.brand.model.entity.BrandCrawlTaskEntity;
import com.nanri.aiimage.modules.task.spi.BrandTaskStaleRepairSpi;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import lombok.RequiredArgsConstructor;
@@ -32,16 +31,15 @@ public class StaleTaskRepairService {
private static final String STATUS_RUNNING = "RUNNING";
private static final String STATUS_FAILED = "FAILED";
private static final String STATUS_BRAND_PENDING = "pending";
private static final String STATUS_BRAND_RUNNING = "running";
private static final String STATUS_BRAND_FAILED = "failed";
private static final String STATUS_BRAND_CANCELLED = "cancelled";
/** 中间态(无人接单)或 RUNNING(心跳残留)超过该时长未更新即判死。真活任务心跳会不断刷新 updatedAt,不会被误杀。 */
private static final long STALE_IDLE_MINUTES = 120;
/** 单轮品牌陈旧任务处理上限(原实现内联 limit 500)。 */
private static final int MAX_BRAND_STALE_BATCH = 500;
private final FileTaskMapper fileTaskMapper;
private final BrandCrawlTaskMapper brandCrawlTaskMapper;
private final BrandTaskStaleRepairSpi brandTaskStaleRepairSpi;
private final DistributedJobLockService distributedJobLockService;
/**
@@ -117,26 +115,15 @@ public class StaleTaskRepairService {
}
}
/** brand_crawl_tasks 的 pending/running 陈旧 → failed。 */
/**
* brand_crawl_tasks 的 pending/running 陈旧 → failed。
* 品牌表与状态字面量属于 brand 模块,实现放在 BrandTaskStaleRepairSpiImpl(G5:消除 task → 业务依赖)。
*/
private void repairBrandStale() {
LocalDateTime cutoff = LocalDateTime.now().minusMinutes(STALE_IDLE_MINUTES);
List<BrandCrawlTaskEntity> stale = brandCrawlTaskMapper.selectList(new LambdaQueryWrapper<BrandCrawlTaskEntity>()
.select(BrandCrawlTaskEntity::getId)
.in(BrandCrawlTaskEntity::getStatus, STATUS_BRAND_PENDING, STATUS_BRAND_RUNNING)
.lt(BrandCrawlTaskEntity::getUpdatedAt, cutoff)
.last("limit 500"));
if (stale.isEmpty()) {
return;
}
int updated = brandCrawlTaskMapper.update(null, new LambdaUpdateWrapper<BrandCrawlTaskEntity>()
.in(BrandCrawlTaskEntity::getId, stale.stream().map(BrandCrawlTaskEntity::getId).toList())
.in(BrandCrawlTaskEntity::getStatus, STATUS_BRAND_PENDING, STATUS_BRAND_RUNNING)
.lt(BrandCrawlTaskEntity::getUpdatedAt, cutoff)
.set(BrandCrawlTaskEntity::getStatus, STATUS_BRAND_FAILED)
.set(BrandCrawlTaskEntity::getErrorMessage, "任务长期无心跳,已自动失败"));
if (updated > 0) {
log.warn("[stale-task-repair] brand 陈旧任务已标失败 count={} ids={}",
updated, stale.stream().map(BrandCrawlTaskEntity::getId).toList());
List<Long> failedIds = brandTaskStaleRepairSpi.failStaleBrandTasks(cutoff, MAX_BRAND_STALE_BATCH);
if (!failedIds.isEmpty()) {
log.warn("[stale-task-repair] brand 陈旧任务已标失败 count={} ids={}", failedIds.size(), failedIds);
}
}
}
@@ -2,76 +2,66 @@ package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
import com.nanri.aiimage.config.SimilarAsinProperties;
import com.nanri.aiimage.modules.appearancepatent.service.AppearancePatentTaskCacheService;
import com.nanri.aiimage.modules.brand.mapper.BrandCrawlTaskMapper;
import com.nanri.aiimage.modules.brand.model.entity.BrandCrawlTaskEntity;
import com.nanri.aiimage.modules.brand.service.BrandTaskProgressCacheService;
import com.nanri.aiimage.modules.collectdata.service.CollectDataService;
import com.nanri.aiimage.modules.deletebrand.service.DeleteBrandTaskCacheService;
import com.nanri.aiimage.modules.patroldelete.service.PatrolDeleteTaskCacheService;
import com.nanri.aiimage.modules.pricetrack.service.PriceTrackTaskCacheService;
import com.nanri.aiimage.modules.productrisk.service.ProductRiskTaskCacheService;
import com.nanri.aiimage.modules.publish.service.PublishTaskService;
import com.nanri.aiimage.modules.queryasin.service.QueryAsinTaskCacheService;
import com.nanri.aiimage.modules.shopdatacrawl.service.ShopDataCrawlTaskCacheService;
import com.nanri.aiimage.modules.shopdatacrawl.service.ShopDataCrawlTaskService;
import com.nanri.aiimage.modules.shopmatch.service.ShopMatchTaskCacheService;
import com.nanri.aiimage.modules.similarasin.service.SimilarAsinTaskCacheService;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.model.dto.TaskHeartbeatRequest;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.vo.TaskHeartbeatVo;
import com.nanri.aiimage.modules.withdraw.service.WithdrawTaskCacheService;
import lombok.RequiredArgsConstructor;
import com.nanri.aiimage.modules.task.spi.BrandTaskHeartbeatSpi;
import com.nanri.aiimage.modules.task.spi.TaskModuleHeartbeatSpi;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.time.Duration;
import java.time.LocalDateTime;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
/**
* 任务心跳入口。
*
* <p>2026-09 全维度审查 G5:本类此前直接注入 12 个业务模块的 CacheService/Service 并用 switch 分发,
* 与"业务模块实现 task 的 Handler SPI"形成 task ↔ 业务的循环依赖。现改为依赖
* {@link TaskModuleHeartbeatSpi}(各业务模块自己实现)与 {@link BrandTaskHeartbeatSpi}(品牌任务,
* 独立于 biz_file_task),task 侧不再 import 任何业务模块。
*/
@Service
@RequiredArgsConstructor
@Slf4j
public class TaskHeartbeatService {
private static final String STATUS_RUNNING = "RUNNING";
private static final String BRAND_STATUS_RUNNING = "running";
private static final String MODULE_DELETE_BRAND = "DELETE_BRAND";
private static final String MODULE_PRODUCT_RISK = "PRODUCT_RISK_RESOLVE";
private static final String MODULE_PRICE_TRACK = "PRICE_TRACK";
private static final String MODULE_SHOP_MATCH = "SHOP_MATCH";
private static final String MODULE_PATROL_DELETE = "PATROL_DELETE";
private static final String MODULE_QUERY_ASIN = "QUERY_ASIN";
private static final String MODULE_SHOP_DATA_CRAWL = "SHOP_DATA_CRAWL";
private static final String MODULE_WITHDRAW = "WITHDRAW";
private static final String MODULE_APPEARANCE_PATENT = "APPEARANCE_PATENT";
private static final String MODULE_SIMILAR_ASIN = "SIMILAR_ASIN";
private static final String MODULE_COLLECT_DATA = "COLLECT_DATA";
private static final String MODULE_BRAND = "BRAND";
private static final String MODULE_SHOP_DATA_CRAWL = "SHOP_DATA_CRAWL";
private final FileTaskMapper fileTaskMapper;
private final TaskProgressSnapshotService taskProgressSnapshotService;
private final BrandCrawlTaskMapper brandCrawlTaskMapper;
private final ProductRiskTaskCacheService productRiskTaskCacheService;
private final PublishTaskService publishTaskService;
private final PriceTrackTaskCacheService priceTrackTaskCacheService;
private final ShopMatchTaskCacheService shopMatchTaskCacheService;
private final PatrolDeleteTaskCacheService patrolDeleteTaskCacheService;
private final QueryAsinTaskCacheService queryAsinTaskCacheService;
private final ShopDataCrawlTaskCacheService shopDataCrawlTaskCacheService;
private final ShopDataCrawlTaskService shopDataCrawlTaskService;
private final WithdrawTaskCacheService withdrawTaskCacheService;
private final AppearancePatentTaskCacheService appearancePatentTaskCacheService;
private final SimilarAsinTaskCacheService similarAsinTaskCacheService;
private final SimilarAsinProperties similarAsinProperties;
private final TaskFileJobService taskFileJobService;
private final DeleteBrandTaskCacheService deleteBrandTaskCacheService;
private final BrandTaskProgressCacheService brandTaskProgressCacheService;
private final CollectDataService collectDataService;
private final BrandTaskHeartbeatSpi brandTaskHeartbeatSpi;
/** moduleType → 模块心跳实现(启动时建索引并校验重复注册) */
private final Map<String, TaskModuleHeartbeatSpi> moduleHandlers;
public TaskHeartbeatService(FileTaskMapper fileTaskMapper,
TaskProgressSnapshotService taskProgressSnapshotService,
BrandTaskHeartbeatSpi brandTaskHeartbeatSpi,
List<TaskModuleHeartbeatSpi> moduleHeartbeatHandlers) {
this.fileTaskMapper = fileTaskMapper;
this.taskProgressSnapshotService = taskProgressSnapshotService;
this.brandTaskHeartbeatSpi = brandTaskHeartbeatSpi;
Map<String, TaskModuleHeartbeatSpi> handlers = new LinkedHashMap<>();
for (TaskModuleHeartbeatSpi handler : moduleHeartbeatHandlers == null ? List.<TaskModuleHeartbeatSpi>of() : moduleHeartbeatHandlers) {
String moduleType = handler.moduleType();
if (moduleType == null || moduleType.isBlank()) {
throw new IllegalStateException("TaskModuleHeartbeatSpi 未声明 moduleType: "
+ handler.getClass().getName());
}
TaskModuleHeartbeatSpi exists = handlers.put(moduleType, handler);
if (exists != null) {
throw new IllegalStateException("moduleType=" + moduleType + " 注册了多个心跳实现: "
+ exists.getClass().getName() + " / " + handler.getClass().getName());
}
}
this.moduleHandlers = Map.copyOf(handlers);
log.info("[task-heartbeat-spi] 模块心跳实现注册完成 count={} modules={}", handlers.size(), handlers.keySet());
}
public TaskHeartbeatVo heartbeat(Long taskId, TaskHeartbeatRequest request) {
if (taskId == null || taskId <= 0) {
@@ -82,15 +72,12 @@ public class TaskHeartbeatService {
String requestedModuleType = request == null || request.getModuleType() == null
? null
: request.getModuleType().trim();
FileTaskEntity fileTask = MODULE_BRAND.equalsIgnoreCase(requestedModuleType)
? null
: selectFileTask(taskId);
BrandCrawlTaskEntity brandTask = MODULE_SHOP_DATA_CRAWL.equalsIgnoreCase(requestedModuleType)
? null
: selectBrandTask(taskId);
// 品牌任务在独立表里:客户端声明是 BRAND 时不必再查 biz_file_task(历史行为保持一致)
boolean brandOnly = MODULE_BRAND.equalsIgnoreCase(requestedModuleType);
boolean skipBrandLookup = MODULE_SHOP_DATA_CRAWL.equalsIgnoreCase(requestedModuleType);
TaskHeartbeatVo fileResult = touchFileTaskIfRunning(fileTask, request);
TaskHeartbeatVo brandResult = touchBrandTaskIfRunning(brandTask, request);
TaskHeartbeatVo fileResult = brandOnly ? null : touchFileTaskIfRunning(selectFileTask(taskId), request);
TaskHeartbeatVo brandResult = skipBrandLookup ? null : brandTaskHeartbeatSpi.heartbeat(taskId, request);
if (fileResult != null && fileResult.isAlive() && brandResult != null && brandResult.isAlive()) {
return TaskHeartbeatVo.alive(fileResult.getModuleType() + "," + MODULE_BRAND, STATUS_RUNNING);
@@ -158,204 +145,72 @@ public class TaskHeartbeatService {
syncTerminalState(fileTask, status, fileTask.getErrorMessage());
return TaskHeartbeatVo.notAlive(fileTask.getModuleType(), status, "task is not running");
}
BrandCrawlTaskEntity brandTask = selectBrandTask(taskId);
if (brandTask != null) {
String status = brandTask.getStatus();
if ("running".equalsIgnoreCase(status) || "pending".equalsIgnoreCase(status)) {
// 与 file 分支统一改为条件更新:整行 updateById 会拿读取快照覆盖并发写入的字段
// (客户端重启上报与品牌任务自身状态流转同时发生时)
int updated = brandCrawlTaskMapper.update(null, new LambdaUpdateWrapper<BrandCrawlTaskEntity>()
.eq(BrandCrawlTaskEntity::getId, brandTask.getId())
.in(BrandCrawlTaskEntity::getStatus, "running", "pending", "RUNNING", "PENDING")
.set(BrandCrawlTaskEntity::getStatus, "cancelled")
.set(BrandCrawlTaskEntity::getErrorMessage, safeReason)
.set(BrandCrawlTaskEntity::getUpdatedAt, LocalDateTime.now()));
if (updated > 0) {
log.warn("[task-interrupted] brand task marked cancelled by client restart taskId={} reason={}",
taskId, safeReason);
return TaskHeartbeatVo.notAlive(MODULE_BRAND, "cancelled", "marked cancelled");
}
}
log.info("[task-interrupted] brand task not in running/pending, skipped taskId={} status={}", taskId, status);
return TaskHeartbeatVo.notAlive(MODULE_BRAND, status, "task is not running");
TaskHeartbeatVo brandResult = brandTaskHeartbeatSpi.markInterrupted(taskId, safeReason);
if (brandResult != null) {
return brandResult;
}
log.debug("[task-interrupted] task not found (normal polling) taskId={}", taskId);
return TaskHeartbeatVo.notAlive(null, null, "task not found");
}
private BrandCrawlTaskEntity selectBrandTask(Long taskId) {
LambdaQueryWrapper<BrandCrawlTaskEntity> brandQuery = new LambdaQueryWrapper<BrandCrawlTaskEntity>()
.eq(BrandCrawlTaskEntity::getId, taskId)
.last("limit 1");
return brandCrawlTaskMapper.selectOne(brandQuery);
}
private TaskHeartbeatVo touchFileTaskIfRunning(FileTaskEntity task, TaskHeartbeatRequest request) {
if (task == null) {
return null;
}
String moduleType = task.getModuleType() == null ? "" : task.getModuleType();
if (PublishTaskService.MODULE_TYPE.equals(moduleType)) {
publishTaskService.ensureTaskOwnedByCurrentInstance(task, "publish task heartbeat");
}
if (MODULE_SHOP_DATA_CRAWL.equals(moduleType)) {
shopDataCrawlTaskService.ensureTaskOwnedByCurrentInstance(task, "shop data crawl task heartbeat");
TaskModuleHeartbeatSpi handler = moduleHandlers.get(moduleType);
if (handler != null) {
handler.ensureOwnership(task, moduleType + " task heartbeat");
}
String status = task.getStatus();
if (!STATUS_RUNNING.equals(status)) {
log.debug("[task-heartbeat] file task is not running (normal polling) taskId={ actualUserId={} moduleType={} status={}",
log.debug("[task-heartbeat] file task is not running (normal polling) taskId={} actualUserId={} moduleType={} status={}",
task.getId(), task.getUserId(), moduleType, status);
return TaskHeartbeatVo.notAlive(moduleType, status, "task is not running");
}
if (MODULE_SIMILAR_ASIN.equals(moduleType)) {
return touchSimilarAsinHeartbeat(task);
// updated_at 检查点:默认每次心跳刷新;实现声明了节流间隔的(similar-asin)按间隔刷新,
// 避免客户端密集心跳把库写满
Duration interval = handler == null ? null : handler.checkpointInterval();
if (isCheckpointDue(task, interval)) {
LocalDateTime now = LocalDateTime.now();
int updated = fileTaskMapper.update(null, new LambdaUpdateWrapper<FileTaskEntity>()
.eq(FileTaskEntity::getId, task.getId())
.eq(FileTaskEntity::getStatus, STATUS_RUNNING)
.set(FileTaskEntity::getUpdatedAt, now));
if (updated <= 0) {
FileTaskEntity latest = fileTaskMapper.selectById(task.getId());
if (interval != null && latest != null && STATUS_RUNNING.equals(latest.getStatus())) {
// 节流下的条件更新未命中:并发心跳刚刷新过,按最新实体继续(不判死)
task = latest;
} else {
log.warn("[task-heartbeat] file task heartbeat update missed taskId={} actualUserId={} moduleType={} status={} latestStatus={}",
task.getId(), task.getUserId(), moduleType, status,
latest == null ? null : latest.getStatus());
return TaskHeartbeatVo.notAlive(moduleType, latest == null ? status : latest.getStatus(),
"task is not running");
}
} else {
task.setUpdatedAt(now);
}
}
LocalDateTime now = LocalDateTime.now();
int updated = fileTaskMapper.update(null, new LambdaUpdateWrapper<FileTaskEntity>()
.eq(FileTaskEntity::getId, task.getId())
.eq(FileTaskEntity::getStatus, STATUS_RUNNING)
.set(FileTaskEntity::getUpdatedAt, now));
if (updated <= 0) {
FileTaskEntity latest = fileTaskMapper.selectById(task.getId());
log.warn("[task-heartbeat] file task heartbeat update missed taskId={} actualUserId={} moduleType={} status={} latestStatus={}",
task.getId(), task.getUserId(), moduleType, status,
latest == null ? null : latest.getStatus());
return TaskHeartbeatVo.notAlive(moduleType, latest == null ? status : latest.getStatus(), "task is not running");
if (handler != null) {
handler.touchHeartbeat(task.getId(), request);
handler.onHeartbeat(task);
handler.saveTaskCache(task);
}
touchModuleHeartbeat(moduleType, task.getId(), request);
task.setUpdatedAt(now);
saveFileTaskCache(moduleType, task);
return TaskHeartbeatVo.alive(moduleType, STATUS_RUNNING);
}
private TaskHeartbeatVo touchSimilarAsinHeartbeat(FileTaskEntity task) {
LocalDateTime now = LocalDateTime.now();
long intervalMillis = Math.max(1_000L, similarAsinProperties.getDbTaskTouchIntervalMillis());
LocalDateTime cutoff = now.minus(Duration.ofMillis(intervalMillis));
boolean checkpointDue = task.getUpdatedAt() == null || !task.getUpdatedAt().isAfter(cutoff);
similarAsinTaskCacheService.touchTaskHeartbeat(task.getId());
taskFileJobService.touchRunningAssembleJobsIfStale(task.getId(), MODULE_SIMILAR_ASIN, intervalMillis);
if (!checkpointDue) {
saveFileTaskCache(MODULE_SIMILAR_ASIN, task);
return TaskHeartbeatVo.alive(MODULE_SIMILAR_ASIN, STATUS_RUNNING);
}
int updated = fileTaskMapper.update(null, new LambdaUpdateWrapper<FileTaskEntity>()
.eq(FileTaskEntity::getId, task.getId())
.eq(FileTaskEntity::getStatus, STATUS_RUNNING)
.and(wrapper -> wrapper.isNull(FileTaskEntity::getUpdatedAt)
.or()
.le(FileTaskEntity::getUpdatedAt, cutoff))
.set(FileTaskEntity::getUpdatedAt, now));
if (updated <= 0) {
FileTaskEntity latest = fileTaskMapper.selectById(task.getId());
if (latest == null || !STATUS_RUNNING.equals(latest.getStatus())) {
return TaskHeartbeatVo.notAlive(MODULE_SIMILAR_ASIN,
latest == null ? task.getStatus() : latest.getStatus(), "task is not running");
}
task = latest;
} else {
task.setUpdatedAt(now);
}
saveFileTaskCache(MODULE_SIMILAR_ASIN, task);
return TaskHeartbeatVo.alive(MODULE_SIMILAR_ASIN, STATUS_RUNNING);
}
private TaskHeartbeatVo touchBrandTaskIfRunning(BrandCrawlTaskEntity task, TaskHeartbeatRequest request) {
if (task == null) {
return null;
}
String status = task.getStatus();
if (!BRAND_STATUS_RUNNING.equals(status)) {
log.warn("[task-heartbeat] brand task is not running taskId={} actualUserId={} status={}",
task.getId(), task.getUserId(), status);
return TaskHeartbeatVo.notAlive(MODULE_BRAND, status, "task is not running");
}
int updated = brandCrawlTaskMapper.update(null, new LambdaUpdateWrapper<BrandCrawlTaskEntity>()
.eq(BrandCrawlTaskEntity::getId, task.getId())
.eq(BrandCrawlTaskEntity::getStatus, BRAND_STATUS_RUNNING)
.set(BrandCrawlTaskEntity::getUpdatedAt, LocalDateTime.now()));
if (updated <= 0) {
BrandCrawlTaskEntity latest = brandCrawlTaskMapper.selectById(task.getId());
log.warn("[task-heartbeat] brand task heartbeat update missed taskId={} actualUserId={} status={} latestStatus={}",
task.getId(), task.getUserId(), status,
latest == null ? null : latest.getStatus());
return TaskHeartbeatVo.notAlive(MODULE_BRAND, latest == null ? status : latest.getStatus(), "task is not running");
}
brandTaskProgressCacheService.touchHeartbeat(
task.getId(),
request == null ? null : request.getPhase(),
request == null ? null : request.getCurrent(),
request == null ? null : request.getTotal());
return TaskHeartbeatVo.alive(MODULE_BRAND, BRAND_STATUS_RUNNING);
}
private void touchModuleHeartbeat(String moduleType, Long taskId, TaskHeartbeatRequest request) {
switch (moduleType) {
case MODULE_PRODUCT_RISK -> {
productRiskTaskCacheService.touchTaskHeartbeat(taskId);
}
case PublishTaskService.MODULE_TYPE -> {
publishTaskService.touchHeartbeat(taskId, request);
}
case MODULE_PRICE_TRACK -> {
priceTrackTaskCacheService.touchTaskHeartbeat(taskId);
}
case MODULE_SHOP_MATCH -> {
shopMatchTaskCacheService.touchTaskHeartbeat(taskId);
}
case MODULE_PATROL_DELETE -> {
patrolDeleteTaskCacheService.touchTaskHeartbeat(taskId);
}
case MODULE_QUERY_ASIN -> {
queryAsinTaskCacheService.touchTaskHeartbeat(taskId);
}
case MODULE_SHOP_DATA_CRAWL -> shopDataCrawlTaskCacheService.touchTaskHeartbeat(taskId);
case MODULE_WITHDRAW -> {
withdrawTaskCacheService.touchTaskHeartbeat(taskId);
}
case MODULE_APPEARANCE_PATENT -> {
appearancePatentTaskCacheService.touchTaskHeartbeat(taskId);
}
case MODULE_SIMILAR_ASIN -> {
similarAsinTaskCacheService.touchTaskHeartbeat(taskId);
}
case MODULE_DELETE_BRAND -> {
deleteBrandTaskCacheService.saveProgress(taskId, buildDeleteBrandHeartbeatProgress(request), true);
}
case MODULE_COLLECT_DATA -> collectDataService.updateProgress(taskId, request);
default -> {
}
}
}
private Map<String, String> buildDeleteBrandHeartbeatProgress(TaskHeartbeatRequest request) {
String now = String.valueOf(System.currentTimeMillis());
Map<String, String> values = new LinkedHashMap<>();
values.put("last_heartbeat_at", now);
values.put("updated_at", now);
if (request != null) {
putIfPresent(values, "phase", request.getPhase());
putIfPresent(values, "current", request.getCurrent());
putIfPresent(values, "total", request.getTotal());
}
return values;
}
private void saveFileTaskCache(String moduleType, FileTaskEntity task) {
switch (moduleType) {
case MODULE_PRODUCT_RISK -> productRiskTaskCacheService.saveTaskCache(task);
case MODULE_PRICE_TRACK -> priceTrackTaskCacheService.saveTaskCache(task);
case MODULE_SHOP_MATCH -> shopMatchTaskCacheService.saveTaskCache(task);
case MODULE_PATROL_DELETE -> patrolDeleteTaskCacheService.saveTaskCache(task);
case MODULE_QUERY_ASIN -> queryAsinTaskCacheService.saveTaskCache(task);
case MODULE_SHOP_DATA_CRAWL -> shopDataCrawlTaskCacheService.saveTaskCache(task);
case MODULE_WITHDRAW -> withdrawTaskCacheService.saveTaskCache(task);
case MODULE_DELETE_BRAND -> deleteBrandTaskCacheService.saveTaskCache(task);
default -> {
}
/** 无节流间隔时永远认为需要刷新(历史行为);有间隔时按 updated_at 距上次刷新的时长判断。 */
private boolean isCheckpointDue(FileTaskEntity task, Duration interval) {
if (interval == null) {
return true;
}
long intervalMillis = Math.max(1_000L, interval.toMillis());
LocalDateTime cutoff = LocalDateTime.now().minus(Duration.ofMillis(intervalMillis));
return task.getUpdatedAt() == null || !task.getUpdatedAt().isAfter(cutoff);
}
/** 把终态同步到模块缓存与进度快照;失败只告警,不影响中断接口本身的成功语义。 */
@@ -372,10 +227,13 @@ public class TaskHeartbeatService {
}
}
private void putIfPresent(Map<String, String> values, String key, Object value) {
if (value != null) {
values.put(key, String.valueOf(value));
private void saveFileTaskCache(String moduleType, FileTaskEntity task) {
if (moduleType == null) {
return;
}
TaskModuleHeartbeatSpi handler = moduleHandlers.get(moduleType);
if (handler != null) {
handler.saveTaskCache(task);
}
}
}
@@ -2,6 +2,7 @@ package com.nanri.aiimage.modules.task.service;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.util.PendingDeleteJournal;
import com.nanri.aiimage.modules.file.service.object.RustfsObjectStorageService;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
@@ -10,8 +11,11 @@ import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Service;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
@@ -49,6 +53,12 @@ public class TransientPayloadDeleteOrchestrator {
@Value("${aiimage.transient-storage.max-pending-deletes:1000}")
private long maxPendingDeletes = 1000;
/**
* 待删对象的本地落盘日志:pending 集合重启即丢,未被引用的对象会一直残留在桶里。
* 入队追加、flush 后按「仍在 pending + 异步删除失败」收敛重写、启动时回放后重新走引用检查。
*/
private volatile PendingDeleteJournal journal;
public TransientPayloadDeleteOrchestrator(TransientPayloadStorageService transientPayloadStorageService,
RustfsObjectStorageService rustfsObjectStorageService,
TaskChunkMapper taskChunkMapper,
@@ -84,6 +94,7 @@ public class TransientPayloadDeleteOrchestrator {
}
if (pendingPointers.add(pointer)) {
accepted++;
journal().record(pointer);
}
}
return accepted;
@@ -164,6 +175,7 @@ public class TransientPayloadDeleteOrchestrator {
}
private void deleteObjects(List<String> pointers) {
List<String> failed = new ArrayList<>();
for (String pointer : pointers) {
if (pointer == null || !pointer.startsWith(RUSTFS_POINTER_PREFIX)) {
// 本地/OSS 指针有实例归属与其它清理路径,不在此批量删除范围。
@@ -173,10 +185,50 @@ public class TransientPayloadDeleteOrchestrator {
try {
rustfsObjectStorageService.deleteObject(objectKey);
} catch (Exception ex) {
// 失败对象写回日志:进程重启后由启动回放重新提交,不再随进程消失
failed.add(pointer);
log.warn("[transient-payload] async delete failed objectKey={} err={}",
objectKey, ex.getMessage());
}
}
synchronized (pendingPointers) {
Set<String> keep = new LinkedHashSet<>(pendingPointers);
keep.addAll(failed);
journal().rewrite(keep);
}
if (!failed.isEmpty()) {
log.warn("[transient-payload] async delete failed, kept in journal count={}", failed.size());
}
}
/**
* 启动回放:把上次进程残留的待删对象重新入队,等待下一轮 flush 做引用检查与物理删除
* (引用检查幂等,重复提交同一对象只入队一次)。
*/
@EventListener(ApplicationReadyEvent.class)
public void replayJournalOnStartup() {
List<String> pointers = journal().readAll();
if (pointers.isEmpty()) {
return;
}
int accepted = submitDeletes(pointers);
log.info("[transient-payload] 启动回放待删对象 journalSize={} accepted={} pending={}",
pointers.size(), accepted, pendingCount());
}
private PendingDeleteJournal journal() {
PendingDeleteJournal local = journal;
if (local != null) {
return local;
}
synchronized (pendingPointers) {
if (journal == null) {
Path path = Path.of(System.getProperty("java.io.tmpdir"),
"aiimage-delete-journal", "transient-payload-pending.log");
journal = new PendingDeleteJournal(path, pendingLimit());
}
return journal;
}
}
/** 批量 IN 反查两张引用表,返回仍被引用的指针集合(查询异常抛给调用方)。 */
@@ -0,0 +1,25 @@
package com.nanri.aiimage.modules.task.spi;
import com.nanri.aiimage.modules.task.model.dto.TaskHeartbeatRequest;
import com.nanri.aiimage.modules.task.model.vo.TaskHeartbeatVo;
/**
* 品牌检测任务(brand_crawl_tasks,独立于 biz_file_task)的心跳/中断扩展点。
*
* <p>2026-09 全维度审查 G5:品牌任务有自己的表与进度缓存,原先由 {@code TaskHeartbeatService}
* 直接注入 brand 模块的 Mapper 与缓存服务,构成 task → 业务的依赖。这里把品牌侧逻辑整体
* 交回 brand 模块实现,task 侧只按接口调用。
*/
public interface BrandTaskHeartbeatSpi {
/**
* 品牌任务心跳:任务不存在返回 null;非 running 返回 notAlive;running 则条件更新
* updated_at 并刷新进度缓存。
*/
TaskHeartbeatVo heartbeat(Long taskId, TaskHeartbeatRequest request);
/**
* 客户端重启中断:running/pending → cancelled(条件更新),返回心跳结果;任务不存在返回 null。
*/
TaskHeartbeatVo markInterrupted(Long taskId, String reason);
}
@@ -0,0 +1,22 @@
package com.nanri.aiimage.modules.task.spi;
import java.time.LocalDateTime;
import java.util.List;
/**
* 品牌检测任务的陈旧修复扩展点(2026-09 全维度审查 G5)。
*
* <p>brand_crawl_tasks 有独立的表与状态字面量(pending/running/failed),原先由
* {@code StaleTaskRepairService} 直接操作品牌模块的 Mapper,构成 task → 业务的依赖。
* 现由品牌模块实现本接口,task 侧只按接口调用。
*/
public interface BrandTaskStaleRepairSpi {
/**
* 把 pending/running 且 updated_at 早于 cutoff 的品牌任务置为失败(条件更新,幂等)。
*
* @param limit 单轮处理上限
* @return 实际标记失败的任务 id(空表示本轮无陈旧任务)
*/
List<Long> failStaleBrandTasks(LocalDateTime cutoff, int limit);
}
@@ -0,0 +1,20 @@
package com.nanri.aiimage.modules.task.spi;
import java.util.List;
/**
* 采集明细行的历史清理扩展点(2026-09 全维度审查 G5)。
*
* <p>任务历史清理时除了 task 自己的表,还要连带删除 biz_collect_data_item 的明细行;
* 该表属于采集模块,原先由 {@code ModuleHistoryCleanupService} 直接使用其 Mapper,
* 构成 task → 业务的依赖。现由采集模块实现本接口。
*/
public interface CollectDataItemCleanupSpi {
/**
* 删除这些任务名下的采集明细行。
*
* @return 删除行数
*/
int deleteItemsByTaskIds(List<Long> taskIds);
}
@@ -0,0 +1,57 @@
package com.nanri.aiimage.modules.task.spi;
import com.nanri.aiimage.modules.task.model.dto.TaskHeartbeatRequest;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import java.time.Duration;
/**
* 任务心跳的模块侧扩展点(2026-09 全维度审查 G5:消除 task ↔ 业务模块的双向依赖)。
*
* <p>背景:{@code TaskHeartbeatService} 原先直接注入 12 个业务模块的 CacheService 并用 switch 分发,
* 与"业务模块实现 task 的 Handler SPI"形成真实循环依赖。现改为:task 侧只依赖本接口,
* 各业务模块各自实现一个 Bean(Spring 注入 List 后在 task 侧建索引)。
*
* <p>新增模块时:实现本接口并在注册表中登记模块类型;心跳契约测试会校验覆盖。
*/
public interface TaskModuleHeartbeatSpi {
/** 本实现负责的 moduleType(与 biz_file_task.module_type 一致)。 */
String moduleType();
/**
* 刷新模块自己的进度缓存/心跳(非 DB 写入)。
* 多数模块实现为 {@code cacheService.touchTaskHeartbeat(taskId)}。
*/
void touchHeartbeat(Long taskId, TaskHeartbeatRequest request);
/**
* 心跳时回写模块的任务缓存(部分模块没有独立缓存,保持默认空实现)。
*/
default void saveTaskCache(FileTaskEntity task) {
}
/**
* 心跳前的任务归属校验:publish / shop-data-crawl 在非本实例拥有时拒绝心跳,
* 其余模块默认放行。
*/
default void ensureOwnership(FileTaskEntity task, String operation) {
}
/**
* 心跳时的额外模块侧动作(如 similar-asin 刷新结果文件组装作业的租约)。
*/
default void onHeartbeat(FileTaskEntity task) {
}
/**
* biz_file_task.updated_at 检查点节流间隔。
*
* <p>返回非空时,DB 的 updated_at 只在距上次刷新超过该间隔后才更新
* (客户端心跳间隔远小于 stale 判定阈值时避免每次心跳都写库);
* 默认 null 表示每次心跳都刷新(与历史行为一致)。
*/
default Duration checkpointInterval() {
return null;
}
}