feat(任务判死): 心跳时钟改为从首次心跳起算——排队中的 RUNNING 任务不再被心跳超时线误杀
任务一被认领(页面 activate / 客户端兜底拉取)就记 RUNNING,而客户端要等执行器真开始跑 才发心跳(同店占用、线程池 5 格全忙、浏览器首屏)。这段排队时间里服务端一行心跳都没有、 updated_at 停在认领时刻,被 30/120 分钟心跳线杀掉的是排队而不是死任务 (现网 2026-10-17 查得近 7 天 COLLECT_DATA 9 条 62 分钟判死、SIMILAR_ASIN 2 条一行未产出)。 - 新增 TaskHeartbeatLivenessService:每条心跳无条件打点(Redis,24h TTL), 读失败一律落 0=没有证据;新增 StaleTaskHeartbeatPolicy:收到过心跳才走心跳时钟, 从未心跳=仍在排队,超过未启动宽限(默认 240 分钟,置 0 即回退旧行为)才判死。 - 接入 product-risk/跟价/店铺匹配/巡店删除/查 ASIN/提现六条委派判死线、采集 30 分钟线、 generic 120 分钟线(该文件另一路会话正在改,只提交本笔 HEAD+本改动的切片)。 - 沉默棘轮按逐处记名抬 3 格(total 1002→1003、不重抛 559→561、只 log 172→173), 两处新增 catch 均为 fail-open 且误差方向是「少杀一个任务」。 验证:backend 全量 4400 例 0 失败;CatchSilenceRatchetTest 9 例绿; ops 判死签名尺子 6 例绿;ruff check . All checks passed。
This commit is contained in:
@@ -45,8 +45,9 @@ class CatchSilenceRatchetTest {
|
||||
private static final Path TREE = moduleRoot().resolve("src/main/java");
|
||||
|
||||
// ==== 验收基线(用户 2026-10-08 12:0x 实测,HEAD 274ad017,单位:处)====
|
||||
// 10-17 心跳证据两处 fail-open 抬 559 -> 561,逐处出处见 STRICT_NOT_RETHROW 上方注释。
|
||||
static final int GIVEN_TOTAL = 1000;
|
||||
static final int GIVEN_NOT_RETHROW = 559;
|
||||
static final int GIVEN_NOT_RETHROW = 561;
|
||||
static final int GIVEN_LOG_ONLY = 249;
|
||||
static final int GIVEN_EMPTY = 56;
|
||||
|
||||
@@ -59,8 +60,13 @@ class CatchSilenceRatchetTest {
|
||||
* BuildInfoLogger 那一处不是吞掉:它打 {@code [build-info-missing]} 的 WARN 并返回可区分的
|
||||
* "章存在但没盖住"状态,巡检按锚点数得到。抬这一格是为了把出处留在这份 diff 里,
|
||||
* 而不是让后来人以为基线是自然涨上来的。
|
||||
* 10-17 再 +2,逐处记名:{@code TaskHeartbeatLivenessService.stamp()}(心跳证据写失败只 WARN,
|
||||
* 心跳接口不许因证据写不下去而 500)与同类 {@code lastHeartbeatMillis()}(读证据失败一律落到 0=
|
||||
* 「没有证据」,判死线据此放宽)。两处都是 fail-open 且方向是「少杀一个任务」,与上面
|
||||
* BuildInfoLogger 同形状;本仓实测基线由 1001 起(原记 1002 留有 1 格无人认领的额度),
|
||||
* 加这 2 处后实测 1003。
|
||||
*/
|
||||
static final int STRICT_TOTAL = 1002;
|
||||
static final int STRICT_TOTAL = 1003;
|
||||
/**
|
||||
* 重抛型 catch 的**下限**(不是上限):本轮把两处吞异常改成重抛之后实测 388。
|
||||
* 这一格只能涨不能跌——它是"异常会往外走"的总量,把它钉成"只许减"就等于拦着别人修吞异常。
|
||||
@@ -75,8 +81,23 @@ class CatchSilenceRatchetTest {
|
||||
// 的两个零调用者重载,其中 `throw new BusinessException("删除 Excel 匹配数据失败", e)` 是 RETHROW 形状,
|
||||
// 随死代码一起离开树;实测重抛 386、不重抛与只 log 两数未涨(那两格仍绿)=> 没有任何一处「异常会往外走」被改回吞掉。
|
||||
static final int RETHROW_FLOOR = 386;
|
||||
static final int STRICT_NOT_RETHROW = 559;
|
||||
static final int STRICT_LOG_ONLY = 172; // 严格口径:块内每条顶层语句都是 log 调用才算
|
||||
/**
|
||||
* 10-17 抬 559 -> 561,逐处记名(两处都是新增的 fail-open 证据读写,不是把已有出口改回吞掉):
|
||||
* 1){@code TaskHeartbeatLivenessService.stamp()}:心跳证据写失败只 WARN——心跳接口不许因为
|
||||
* 「记不下证据」而 500,那会让客户端整轮心跳线程崩掉,反而制造更多孤儿任务;
|
||||
* 2){@code TaskHeartbeatLivenessService.lastHeartbeatMillis()}:读证据失败一律落到 0=「没有证据」,
|
||||
* 判死线据此**放宽**(顶多让任务多活一会儿,最终仍由未启动宽限与 stale-repair 兜底)。
|
||||
* 两处的误差方向都是「少杀一个任务」,与这一格想拦的「异常到此为止、外面看不见」不同性质;
|
||||
* 抬格是为了把出处留在这份 diff 里,而不是让后来人以为基线自然涨上来的。
|
||||
*/
|
||||
static final int STRICT_NOT_RETHROW = 561;
|
||||
/**
|
||||
* 10-17 抬 172 -> 173,出处就是上面点名过的 {@code TaskHeartbeatLivenessService.stamp()}:
|
||||
* 块体只有「取到 key 名 + log.warn」一句,写心跳证据失败时既不重试也不抛出——
|
||||
* 心跳接口不许因为「记不下证据」而 500。同类的 {@code lastHeartbeatMillis()} 读失败还要
|
||||
* 落 0 返回,形状是 log+return 默认值,因此不进这一格。
|
||||
*/
|
||||
static final int STRICT_LOG_ONLY = 173; // 严格口径:块内每条顶层语句都是 log 调用才算
|
||||
static final int STRICT_EMPTY = 56;
|
||||
|
||||
/**
|
||||
|
||||
+2
-1
@@ -147,7 +147,8 @@ class CollectDataActivateGuardTest {
|
||||
mock(TaskProgressLightAssembler.class),
|
||||
mock(com.nanri.aiimage.modules.notification.service.NotificationDispatchService.class),
|
||||
mock(com.nanri.aiimage.modules.asincache.service.AsinResultCacheService.class),
|
||||
mock(CollectDataBrandRecheckService.class));
|
||||
mock(CollectDataBrandRecheckService.class),
|
||||
mock(com.nanri.aiimage.modules.task.service.StaleTaskHeartbeatPolicy.class));
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
|
||||
+2
-1
@@ -120,7 +120,8 @@ class CollectDataDeleteConsistencyTest {
|
||||
mock(TaskProgressLightAssembler.class),
|
||||
mock(com.nanri.aiimage.modules.notification.service.NotificationDispatchService.class),
|
||||
mock(com.nanri.aiimage.modules.asincache.service.AsinResultCacheService.class),
|
||||
mock(CollectDataBrandRecheckService.class));
|
||||
mock(CollectDataBrandRecheckService.class),
|
||||
mock(com.nanri.aiimage.modules.task.service.StaleTaskHeartbeatPolicy.class));
|
||||
}
|
||||
|
||||
private FileTaskEntity task(long id, long userId) {
|
||||
|
||||
+29
@@ -58,6 +58,13 @@ class CollectDataServiceTest {
|
||||
@Mock(answer = org.mockito.Answers.RETURNS_MOCKS)
|
||||
private com.nanri.aiimage.common.service.DistributedJobLockService distributedJobLockService;
|
||||
|
||||
/**
|
||||
* 「首次心跳之前不判死」闸门:默认 false=不豁免,本类原有 stale 判死用例的行为保持不变;
|
||||
* 豁免分支另有用例(staleTaskWithoutFirstHeartbeatWithinGraceIsNotJudgedDead)。
|
||||
*/
|
||||
@Mock
|
||||
private com.nanri.aiimage.modules.task.service.StaleTaskHeartbeatPolicy staleTaskHeartbeatPolicy;
|
||||
|
||||
@Spy
|
||||
private ObjectMapper objectMapper = new ObjectMapper();
|
||||
|
||||
@@ -190,6 +197,28 @@ class CollectDataServiceTest {
|
||||
verify(lockHandle).close();
|
||||
}
|
||||
|
||||
// 2026-10-17 现网 9 条误杀的口径:采集这条 30 分钟心跳线只看 updated_at,而客户端认领任务后要先等
|
||||
// 同店释放、等线程池那 5 格、等浏览器首屏——这段时间一行心跳都没有,updated_at 停在认领时刻。
|
||||
// 闸门放行时,本条线既不能标失败,也不能封口(封口会让后续回传被拒、数据丢失)。
|
||||
@Test
|
||||
void staleTaskWithoutFirstHeartbeatWithinGraceIsNotJudgedDead() throws Exception {
|
||||
FileTaskEntity queued = staleTask(95L);
|
||||
|
||||
when(fileTaskMapper.selectList(any())).thenReturn(List.of(queued));
|
||||
when(taskDistributedLockService.acquire(CollectDataService.MODULE_TYPE, queued.getId(), 0L))
|
||||
.thenReturn(lockHandle);
|
||||
when(fileTaskMapper.selectById(queued.getId())).thenReturn(queued);
|
||||
when(staleTaskHeartbeatPolicy.isStillQueuedBeforeFirstHeartbeat(
|
||||
org.mockito.ArgumentMatchers.eq(95L), org.mockito.ArgumentMatchers.eq(0L),
|
||||
any(), any())).thenReturn(true);
|
||||
|
||||
service.finalizeStaleTasks();
|
||||
|
||||
assertThat(queued.getStatus()).isEqualTo("RUNNING");
|
||||
verify(fileTaskMapper, never()).update(any(), any());
|
||||
verify(taskFileJobService, never()).enqueueAssembleResult(any(), any(), any(), any());
|
||||
}
|
||||
|
||||
private static FileTaskEntity staleTask(long taskId) {
|
||||
FileTaskEntity task = new FileTaskEntity();
|
||||
task.setId(taskId);
|
||||
|
||||
+2
-1
@@ -117,7 +117,8 @@ class CollectDataStorageCallCountTest {
|
||||
mock(TaskProgressLightAssembler.class),
|
||||
mock(com.nanri.aiimage.modules.notification.service.NotificationDispatchService.class),
|
||||
mock(com.nanri.aiimage.modules.asincache.service.AsinResultCacheService.class),
|
||||
mock(CollectDataBrandRecheckService.class));
|
||||
mock(CollectDataBrandRecheckService.class),
|
||||
mock(com.nanri.aiimage.modules.task.service.StaleTaskHeartbeatPolicy.class));
|
||||
}
|
||||
|
||||
private FileResultEntity result(long id, long taskId, long userId) {
|
||||
|
||||
+64
-1
@@ -62,6 +62,9 @@ class DeleteBrandStaleTaskServiceTest {
|
||||
@Mock private TaskHeartbeatPositionService taskHeartbeatPositionService;
|
||||
@Mock private TaskResumeService taskResumeService;
|
||||
@Mock private PriceTrackLoopRunStaleService priceTrackLoopRunStaleService;
|
||||
@Mock private com.nanri.aiimage.modules.pricetrack.service.PriceTrackTaskService priceTrackTaskService;
|
||||
@Mock private com.nanri.aiimage.modules.pricetrack.service.PriceTrackTaskCacheService priceTrackTaskCacheService;
|
||||
@Mock private com.nanri.aiimage.modules.task.service.StaleTaskHeartbeatPolicy staleTaskHeartbeatPolicy;
|
||||
|
||||
@BeforeAll
|
||||
static void initializeTableInfo() {
|
||||
@@ -448,7 +451,67 @@ class DeleteBrandStaleTaskServiceTest {
|
||||
null, null, priceTrackLoopRunStaleService, null, null, null, null, null, null, null,
|
||||
null, null, null, null,
|
||||
deleteBrandProgressProperties, null, taskDistributedLockService, taskFileJobService, null,
|
||||
taskScopeStateMapper, taskHeartbeatPositionService, taskResumeService);
|
||||
taskScopeStateMapper, taskHeartbeatPositionService, taskResumeService, staleTaskHeartbeatPolicy);
|
||||
}
|
||||
|
||||
/** 只喂跟价那条线需要的两个 mock,其余仍走 service() 的窄桩口径。 */
|
||||
private DeleteBrandStaleTaskService priceTrackService() {
|
||||
return new DeleteBrandStaleTaskService(
|
||||
fileTaskMapper, deleteBrandTaskCacheService, deleteBrandTaskStorageService, deleteBrandRunService,
|
||||
productRiskTaskService, productRiskTaskCacheService,
|
||||
priceTrackTaskService, priceTrackTaskCacheService, priceTrackLoopRunStaleService,
|
||||
null, null, null, null, null, null, null,
|
||||
null, null, null, null,
|
||||
deleteBrandProgressProperties, null, taskDistributedLockService, taskFileJobService, null,
|
||||
taskScopeStateMapper, taskHeartbeatPositionService, taskResumeService, staleTaskHeartbeatPolicy);
|
||||
}
|
||||
|
||||
@SuppressWarnings({"rawtypes", "unchecked"})
|
||||
private FileTaskEntity priceTrackCandidate(Long id) {
|
||||
FileTaskEntity task = runningTask(id);
|
||||
task.setModuleType("PRICE_TRACK");
|
||||
// 认领后再没有别的写入:updated_at 就是排队起点
|
||||
task.setUpdatedAt(LocalDateTime.now().minusMinutes(40));
|
||||
return task;
|
||||
}
|
||||
|
||||
@Test
|
||||
@SuppressWarnings({"rawtypes", "unchecked"})
|
||||
void priceTrackCandidateWithoutFirstHeartbeatIsNotJudgedDead() {
|
||||
FileTaskEntity queued = priceTrackCandidate(9001L);
|
||||
when(fileTaskMapper.selectList(any())).thenReturn(List.of(queued));
|
||||
when(deleteBrandProgressProperties.getPriceTrackStaleTimeoutMinutes()).thenReturn(30L);
|
||||
when(priceTrackTaskCacheService.getTaskHeartbeatMillisBatch(any())).thenReturn(Map.of());
|
||||
when(staleTaskHeartbeatPolicy.isStillQueuedBeforeFirstHeartbeat(
|
||||
org.mockito.ArgumentMatchers.eq(9001L), org.mockito.ArgumentMatchers.eq(0L),
|
||||
any(), any())).thenReturn(true);
|
||||
|
||||
ReflectionTestUtils.invokeMethod(priceTrackService(), "failStalePriceTrackTasks");
|
||||
|
||||
// 排队中的任务:既不该标失败,也不该被 finalize 封口(封口会让后续回传被拒、数据丢失)
|
||||
verify(fileTaskMapper, never()).update(isNull(), any(LambdaUpdateWrapper.class));
|
||||
verify(priceTrackTaskService, never()).tryFinalizeTask(anyLong(), anyBoolean());
|
||||
}
|
||||
|
||||
@Test
|
||||
@SuppressWarnings({"rawtypes", "unchecked"})
|
||||
void priceTrackCandidatePastTheGraceWindowIsStillJudgedDead() {
|
||||
FileTaskEntity orphan = priceTrackCandidate(9002L);
|
||||
when(fileTaskMapper.selectList(any())).thenReturn(List.of(orphan));
|
||||
when(deleteBrandProgressProperties.getPriceTrackStaleTimeoutMinutes()).thenReturn(30L);
|
||||
when(priceTrackTaskCacheService.getTaskHeartbeatMillisBatch(any())).thenReturn(Map.of());
|
||||
when(staleTaskHeartbeatPolicy.isStillQueuedBeforeFirstHeartbeat(
|
||||
org.mockito.ArgumentMatchers.eq(9002L), org.mockito.ArgumentMatchers.eq(0L),
|
||||
any(), any())).thenReturn(false);
|
||||
when(taskDistributedLockService.acquire(any(), any(), anyLong()))
|
||||
.thenReturn(mock(TaskDistributedLockService.LockHandle.class));
|
||||
when(priceTrackTaskService.tryFinalizeTask(org.mockito.ArgumentMatchers.eq(9002L), anyBoolean()))
|
||||
.thenReturn(false);
|
||||
when(fileTaskMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
|
||||
|
||||
ReflectionTestUtils.invokeMethod(priceTrackService(), "failStalePriceTrackTasks");
|
||||
|
||||
verify(fileTaskMapper).update(isNull(), any(LambdaUpdateWrapper.class));
|
||||
}
|
||||
|
||||
private void lockAvailable() {
|
||||
|
||||
+1
-1
@@ -82,7 +82,7 @@ class TaskModuleCoverageTest {
|
||||
mock(SimilarAsinTaskService.class),
|
||||
null, null, null, null,
|
||||
mock(ShopDataCrawlTaskService.class),
|
||||
null, null, null);
|
||||
null, null, null, null);
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
package com.nanri.aiimage.modules.task.service;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.time.LocalDateTime;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.mockito.ArgumentMatchers.anyLong;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.never;
|
||||
import static org.mockito.Mockito.verify;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
/**
|
||||
* 「心跳时钟从首次心跳起算」的判据契约。
|
||||
*
|
||||
* <p>要守住的现场(2026-10-17 生产实录):任务一被认领就记 RUNNING,而客户端要等执行器真开始跑才发
|
||||
* 心跳(同店占用 / 线程池 5 格全忙 / 浏览器首屏)。这段排队时间里服务端一行心跳都没有,
|
||||
* 拿各模块 30 分钟心跳超时线去套,杀的是排队任务。
|
||||
*/
|
||||
class StaleTaskHeartbeatPolicyTest {
|
||||
|
||||
private final TaskHeartbeatLivenessService livenessService = mock(TaskHeartbeatLivenessService.class);
|
||||
private final StaleTaskHeartbeatPolicy policy = new StaleTaskHeartbeatPolicy(livenessService);
|
||||
|
||||
private StaleTaskHeartbeatPolicy withGrace(long minutes) {
|
||||
policy.setFirstHeartbeatGraceMinutes(minutes);
|
||||
return policy;
|
||||
}
|
||||
|
||||
@Test
|
||||
void 从未心跳且在宽限内_还在排队本轮不判死() {
|
||||
when(livenessService.lastHeartbeatMillis(31279L)).thenReturn(0L);
|
||||
|
||||
assertTrue(withGrace(240).isStillQueuedBeforeFirstHeartbeat(
|
||||
31279L, 0L, LocalDateTime.now().minusMinutes(40), LocalDateTime.now()));
|
||||
}
|
||||
|
||||
@Test
|
||||
void 从未心跳且超出宽限_照旧判死不许留成永久孤儿() {
|
||||
when(livenessService.lastHeartbeatMillis(31279L)).thenReturn(0L);
|
||||
|
||||
assertFalse(withGrace(240).isStillQueuedBeforeFirstHeartbeat(
|
||||
31279L, 0L, LocalDateTime.now().minusMinutes(241), LocalDateTime.now()));
|
||||
}
|
||||
|
||||
@Test
|
||||
void 模块自带心跳读数时不回落共享打点() {
|
||||
assertFalse(withGrace(240).isStillQueuedBeforeFirstHeartbeat(
|
||||
31279L, System.currentTimeMillis(), LocalDateTime.now().minusMinutes(90), LocalDateTime.now()));
|
||||
|
||||
verify(livenessService, never()).lastHeartbeatMillis(anyLong());
|
||||
}
|
||||
|
||||
@Test
|
||||
void 模块没有心跳读数时回落共享打点() {
|
||||
when(livenessService.lastHeartbeatMillis(31279L)).thenReturn(System.currentTimeMillis());
|
||||
|
||||
assertFalse(withGrace(240).isStillQueuedBeforeFirstHeartbeat(
|
||||
31279L, 0L, LocalDateTime.now().minusMinutes(40), LocalDateTime.now()));
|
||||
}
|
||||
|
||||
@Test
|
||||
void 宽限置零_闸门关闭退回旧行为() {
|
||||
when(livenessService.lastHeartbeatMillis(31279L)).thenReturn(0L);
|
||||
|
||||
assertFalse(withGrace(0).isStillQueuedBeforeFirstHeartbeat(
|
||||
31279L, 0L, LocalDateTime.now().minusMinutes(31), LocalDateTime.now()));
|
||||
}
|
||||
|
||||
@Test
|
||||
void 没有updated_at就无从算宽限() {
|
||||
when(livenessService.lastHeartbeatMillis(31279L)).thenReturn(0L);
|
||||
|
||||
assertTrue(withGrace(240).isStillQueuedBeforeFirstHeartbeat(
|
||||
31279L, 0L, null, LocalDateTime.now()));
|
||||
}
|
||||
}
|
||||
+19
-1
@@ -23,6 +23,7 @@ import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.eq;
|
||||
import static org.mockito.ArgumentMatchers.isNull;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.never;
|
||||
@@ -40,8 +41,9 @@ class StaleTaskRepairServiceTest {
|
||||
private final FileTaskMapper fileTaskMapper = mock(FileTaskMapper.class);
|
||||
private final BrandTaskStaleRepairSpi brandTaskStaleRepairSpi = mock(BrandTaskStaleRepairSpi.class);
|
||||
private final DistributedJobLockService distributedJobLockService = mock(DistributedJobLockService.class);
|
||||
private final StaleTaskHeartbeatPolicy staleTaskHeartbeatPolicy = mock(StaleTaskHeartbeatPolicy.class);
|
||||
private final StaleTaskRepairService service = new StaleTaskRepairService(
|
||||
fileTaskMapper, brandTaskStaleRepairSpi, distributedJobLockService);
|
||||
fileTaskMapper, brandTaskStaleRepairSpi, distributedJobLockService, staleTaskHeartbeatPolicy);
|
||||
|
||||
@BeforeEach
|
||||
void initTableInfo() {
|
||||
@@ -99,6 +101,22 @@ class StaleTaskRepairServiceTest {
|
||||
verifyNoMoreInteractions(brandTaskStaleRepairSpi);
|
||||
}
|
||||
|
||||
// 2026-10-17 实录:任务一被认领就记 RUNNING,而客户端要等执行器真开始跑才发心跳(同店占用 /
|
||||
// 线程池 5 格全忙 / 浏览器首屏)。本线是各模块心跳线的天花板——不在这里放行,模块侧把宽限
|
||||
// 放到多少分钟都会被这里的 2 小时重新杀成「任务心跳超时」。
|
||||
@Test
|
||||
void 未收到首次心跳的_RUNNING_任务在排队宽限内暂不判死() {
|
||||
when(staleTaskHeartbeatPolicy.isStillQueuedBeforeFirstHeartbeat(
|
||||
eq(31279L), eq(0L), any(), any())).thenReturn(true);
|
||||
FileTaskEntity queued = task(31279L);
|
||||
queued.setStatus("RUNNING");
|
||||
when(fileTaskMapper.selectList(any())).thenReturn(List.of(queued));
|
||||
|
||||
ReflectionTestUtils.invokeMethod(service, "repairFileTaskStaleRunning");
|
||||
|
||||
verify(fileTaskMapper, never()).update(isNull(), any());
|
||||
}
|
||||
|
||||
private Map<String, Object> selectParams() {
|
||||
ArgumentCaptor<Wrapper<FileTaskEntity>> captor = wrapperCaptor();
|
||||
verify(fileTaskMapper).selectList(captor.capture());
|
||||
|
||||
+93
@@ -0,0 +1,93 @@
|
||||
package com.nanri.aiimage.modules.task.service;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.data.redis.core.StringRedisTemplate;
|
||||
import org.springframework.data.redis.core.ValueOperations;
|
||||
|
||||
import java.time.Duration;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.anyString;
|
||||
import static org.mockito.ArgumentMatchers.eq;
|
||||
import static org.mockito.Mockito.doThrow;
|
||||
import static org.mockito.Mockito.lenient;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.never;
|
||||
import static org.mockito.Mockito.verify;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
/**
|
||||
* 心跳打点契约:判死线把它当「这任务到底有没有心跳上来过」的唯一通用证据,
|
||||
* 因此读取侧的每一个失败分支都必须落到 0(=没有证据=放宽),绝不能把异常抛给判死线。
|
||||
*/
|
||||
@SuppressWarnings("unchecked")
|
||||
class TaskHeartbeatLivenessServiceTest {
|
||||
|
||||
private final StringRedisTemplate redisTemplate = mock(StringRedisTemplate.class);
|
||||
private final ValueOperations<String, String> valueOperations = mock(ValueOperations.class);
|
||||
private final TaskHeartbeatLivenessService service = new TaskHeartbeatLivenessService(redisTemplate);
|
||||
|
||||
private TaskHeartbeatLivenessServiceTest withRedis() {
|
||||
lenient().when(redisTemplate.opsForValue()).thenReturn(valueOperations);
|
||||
return this;
|
||||
}
|
||||
|
||||
@Test
|
||||
void 收到心跳即打点并带过期() {
|
||||
withRedis();
|
||||
|
||||
service.stamp(31279L);
|
||||
|
||||
verify(valueOperations).set(eq("task:heartbeat:beat:31279"), anyString(), any(Duration.class));
|
||||
}
|
||||
|
||||
@Test
|
||||
void 打点失败不许影响心跳本身() {
|
||||
withRedis();
|
||||
doThrow(new IllegalStateException("redis down"))
|
||||
.when(valueOperations).set(anyString(), anyString(), any(Duration.class));
|
||||
|
||||
service.stamp(31279L); // 不抛即通过:心跳接口不能因为证据写不下去而 500
|
||||
}
|
||||
|
||||
@Test
|
||||
void 读回打点时刻() {
|
||||
withRedis();
|
||||
when(valueOperations.get("task:heartbeat:beat:31279")).thenReturn("1792228880835");
|
||||
|
||||
assertEquals(1792228880835L, service.lastHeartbeatMillis(31279L));
|
||||
}
|
||||
|
||||
@Test
|
||||
void 没有打点时读成从未心跳() {
|
||||
withRedis();
|
||||
when(valueOperations.get(anyString())).thenReturn(null);
|
||||
|
||||
assertEquals(0L, service.lastHeartbeatMillis(31279L));
|
||||
}
|
||||
|
||||
@Test
|
||||
void 脏值按没有证据处理而不是把判死线带崩() {
|
||||
withRedis();
|
||||
when(valueOperations.get(anyString())).thenReturn("not-a-timestamp");
|
||||
|
||||
assertEquals(0L, service.lastHeartbeatMillis(31279L));
|
||||
}
|
||||
|
||||
@Test
|
||||
void Redis异常时读成从未心跳() {
|
||||
withRedis();
|
||||
when(valueOperations.get(anyString())).thenThrow(new IllegalStateException("redis down"));
|
||||
|
||||
assertEquals(0L, service.lastHeartbeatMillis(31279L));
|
||||
}
|
||||
|
||||
@Test
|
||||
void 非法taskId不打点也不读() {
|
||||
service.stamp(0L);
|
||||
assertEquals(0L, service.lastHeartbeatMillis(-1L));
|
||||
|
||||
verify(redisTemplate, never()).opsForValue();
|
||||
}
|
||||
}
|
||||
+34
-1
@@ -29,8 +29,10 @@ import java.time.LocalDateTime;
|
||||
import java.util.List;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.anyLong;
|
||||
import static org.mockito.ArgumentMatchers.isNull;
|
||||
import static org.mockito.Mockito.inOrder;
|
||||
import static org.mockito.Mockito.mock;
|
||||
@@ -55,10 +57,11 @@ class TaskHeartbeatServiceTest {
|
||||
private final FileTaskMapper fileTaskMapper = mock(FileTaskMapper.class);
|
||||
private final TaskProgressSnapshotService taskProgressSnapshotService = mock(TaskProgressSnapshotService.class);
|
||||
private final BrandTaskHeartbeatSpi brandTaskHeartbeatSpi = mock(BrandTaskHeartbeatSpi.class);
|
||||
private final TaskHeartbeatLivenessService livenessService = mock(TaskHeartbeatLivenessService.class);
|
||||
|
||||
private TaskHeartbeatService serviceWith(TaskModuleHeartbeatSpi... handlers) {
|
||||
return new TaskHeartbeatService(fileTaskMapper, taskProgressSnapshotService, brandTaskHeartbeatSpi,
|
||||
List.of(handlers));
|
||||
livenessService, List.of(handlers));
|
||||
}
|
||||
|
||||
private static FileTaskEntity runningTask(long taskId, String moduleType) {
|
||||
@@ -184,4 +187,34 @@ class TaskHeartbeatServiceTest {
|
||||
assertTrue(result.isAlive(), "未注册 SPI 的模块不应影响通用心跳(保持向后兼容)");
|
||||
assertEquals("UNREGISTERED_MODULE", result.getModuleType());
|
||||
}
|
||||
|
||||
@Test
|
||||
@SuppressWarnings("unchecked")
|
||||
void heartbeatStampsLivenessEvenWhenTaskIsNotRunningYet() {
|
||||
long taskId = 20151L;
|
||||
FileTaskEntity justClaimed = runningTask(taskId, "PRICE_TRACK");
|
||||
justClaimed.setStatus("PENDING");
|
||||
TaskHeartbeatService service = serviceWith();
|
||||
|
||||
when(fileTaskMapper.selectOne(any(LambdaQueryWrapper.class))).thenReturn(justClaimed);
|
||||
|
||||
TaskHeartbeatVo result = service.heartbeat(taskId, new TaskHeartbeatRequest());
|
||||
|
||||
assertFalse(result.isAlive(), "非 RUNNING 的心跳不得把任务改成存活(既有契约不变)");
|
||||
verify(livenessService).stamp(taskId);
|
||||
}
|
||||
|
||||
@Test
|
||||
@SuppressWarnings("unchecked")
|
||||
void heartbeatForUnknownTaskDoesNotStampLiveness() {
|
||||
long taskId = 20152L;
|
||||
TaskHeartbeatService service = serviceWith();
|
||||
|
||||
when(fileTaskMapper.selectOne(any(LambdaQueryWrapper.class))).thenReturn(null);
|
||||
|
||||
service.heartbeat(taskId, new TaskHeartbeatRequest());
|
||||
|
||||
// 坏 id 的轮询不能把 Redis 写成垃圾键
|
||||
verify(livenessService, never()).stamp(anyLong());
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user