fix(stability): 接手第三路未提交 WIP 并收口——卡死扫描跨实例归属守卫+补检预算闸
归属声明(不是本会话的产出,本会话只补了漏改的桩):这批改动在工作树里躺到 2026-11-03 00:47 未提交,f43ed6f1 的 commit message 已点名"同工作树另一会话的在飞件 (TaskFileJobService 卡死归属守卫、TaskResultFileJobWorker、DeliverySupport 补检闸及其 3 支测试)未卷入本提交"。用户 2026-11-03 拍板"接手收口后开 Phase 2",故原样入库。 WIP 本体(作者=第三路会话): - TaskFileJobService:resetStuckRunningJobsDetailed 增 currentInstanceId 重载+ belongsToAnotherInstance 谓词——B 扫到 A 正在跑的 RUNNING 行不再翻 PENDING 也不再 +1 stuck_requeue_count(33149 爬到 20 判死线即此形状);不带实例 id 的旧入口保持 原行为,后管人工复位按钮不受守卫影响。 - AppearancePatentDeliverySupport:requestRecheckForMissingRows 自己吃 recheckMaxRounds 预算——行级轮次只在行真进批次才累加,"没有可比对图/载荷"的行 永远进不了批次,于是 33179 一个 1 行缺口六小时补了 70+ 轮。 - 配套:TaskFileJobOwnerHandOffTest +4 支(谓词边界/他机 RUNNING 不翻不判死/ 自己的陈旧行照翻/无实例 id 入口原行为),StuckScan 与 FailureNotify 两支桩 改到新 3 参签名。 本会话补的一格:TaskResultFileJobWorkerTest 三支桩没跟着改到 3 参签名, stuckSimilarAsinJobAtRetryLimitFailsOwningTask 等 3 例 PotentialStubbingProblem 真红(11-03 09:0x 实测)——改成 eq(0),eq(0),anyString() 并补 eq 静态导入。 验证:mvn -o test 定向 6 类(Worker/StuckScan/FailureNotify/OwnerHandOff/ ResultFileJobHandler/RequestSuppliedIdentityOutlet)rc=0 全绿。
This commit is contained in:
+21
-3
@@ -122,6 +122,13 @@ public class AppearancePatentDeliverySupport {
|
||||
*
|
||||
* <p>行级重发次数仍受 {@code llmRowMaxRounds} 约束(到顶的行会被判死并写上
|
||||
* 「重发轮次已达上限」的结论,从此不再是空行),所以这条不会无限循环。
|
||||
*
|
||||
* <p>⚠ 那句「不会无限循环」2026-11-02 现网证伪,这里补上缺的那道闸:行级轮次只在行**真的
|
||||
* 被发进 LLM 批次**时才累加,而交付缺口里最常见的形状是「该行没有可比对的图/载荷」——
|
||||
* 它永远进不了批次,轮次永远是 0,于是补检→重组→补检跑个不停(33179 一个 1 行的缺口
|
||||
* 六小时补了 70+ 轮,作业被卡死扫描记到 stuck_requeue_count=20 的判死线上)。所以成功出口
|
||||
* 必须自己吃 {@code recheckMaxRounds}:预算耗尽就不再要补检,直接按已有行账出部分表,
|
||||
* 缺口由行账文案标在表里,而不是让用户永远等一张不来的整表。
|
||||
*/
|
||||
public boolean requestRecheckForMissingRows(FileTaskEntity task, TaskFileJobEntity job,
|
||||
AppearancePatentRowOutcome.Facts facts) {
|
||||
@@ -129,6 +136,14 @@ public class AppearancePatentDeliverySupport {
|
||||
|| facts.missingRows() <= 0 || facts.missingKeys().isEmpty()) {
|
||||
return false;
|
||||
}
|
||||
int maxRounds = properties.getRecheckMaxRounds();
|
||||
long done = taskCacheService.recheckRounds(task.getId());
|
||||
if (maxRounds > 0 && done >= maxRounds) {
|
||||
log.warn("[appearance-patent] 交付缺口补检预算已耗尽,按已有结果交付"
|
||||
+ " taskId={} jobId={} rounds={}/{} missingRows={} noDataRows={}",
|
||||
task.getId(), job.getId(), done, maxRounds, facts.missingRows(), facts.noDataRows());
|
||||
return false;
|
||||
}
|
||||
List<String> keys = facts.missingKeys();
|
||||
// 只有这条出口允许连完成标记一起撤:结果落过库却又读不回来的行,不撤就永远补不回来。
|
||||
// 代价(该行可能被重发一次 LLM、多扣一次次数)是这一档修法里用户点名要付的。
|
||||
@@ -136,6 +151,8 @@ public class AppearancePatentDeliverySupport {
|
||||
boolean requeued = taskFileJobService.requeue(job.getId(),
|
||||
"有 " + facts.missingRows() + " 行未完成检测,补检一轮后重新组装结果文件");
|
||||
if (requeued) {
|
||||
// 与失败兜底出口同一套预算、同一条记账纪律:只有重排真的接上才扣一轮。
|
||||
taskCacheService.countRecheckRound(task.getId());
|
||||
throttleRecheckWarning(task, job, facts);
|
||||
return true;
|
||||
}
|
||||
@@ -148,9 +165,10 @@ public class AppearancePatentDeliverySupport {
|
||||
* 补检轮 WARN 的限流出口(AC-21):同一任务在一个窗口内只留一条 WARN,窗口里被压掉的条数
|
||||
* 随下一条 WARN 一起汇总出去({@code suppressedInLastWindow}),窗口内的重复轮次降 DEBUG。
|
||||
*
|
||||
* <p>刻意不去碰 {@code taskCacheService.recheckRounds/countRecheckRound} 那对轮次计数:
|
||||
* 那是 {@link #scheduleFallbackRecheck} 的补检预算({@code recheckMaxRounds})。从这条路径
|
||||
* 再计一次,会让「日志限流」吃掉业务预算,任务更早被判「补检已耗尽」——那是行为变更,不是治理。
|
||||
* <p>{@code taskCacheService.recheckRounds/countRecheckRound} 那对轮次计数不在这里动:
|
||||
* 那是补检预算({@code recheckMaxRounds}),从「日志限流」这条路再计一次会让限流吃掉业务预算,
|
||||
* 任务更早被判「补检已耗尽」——那是行为变更,不是治理。记账点在
|
||||
* {@link #requestRecheckForMissingRows} 与 {@link #scheduleFallbackRecheck} 两个业务出口。
|
||||
*
|
||||
* <p>并发口径:读-判-写三步之间有竞争,最坏情况是同窗口多打一条 WARN。限流是为了压日志量,
|
||||
* 不是为了恰好一条,不值得为它加锁或把整张表串行化。
|
||||
|
||||
+47
@@ -402,6 +402,26 @@ public class TaskFileJobService {
|
||||
* 被记成失败次数。PENDING 行的累加保留:那条防的是 dispatch 事件无限重发,与执行快慢无关。
|
||||
*/
|
||||
public StuckJobResetResult resetStuckRunningJobsDetailed(int stuckMinutes, int limit) {
|
||||
return resetStuckRunningJobsDetailed(stuckMinutes, limit, null);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param currentInstanceId 本机实例 id。给了就启用归属守卫:scope_key 上明确写着
|
||||
* {@code :owner:<别的实例>} 的 RUNNING 行**本轮不碰**(既不重排也不判死)。
|
||||
* 留空=不启用(后管那枚人工复位按钮走这条:人明确点的就是「把它排回去」)。
|
||||
*
|
||||
* <p>为什么必须有这道守卫(2026-11-02 现网 33149/33179):两台各跑一份 @Scheduled,而这一支
|
||||
* 原先不带归属过滤——B 扫到 A 正在跑的 RUNNING 行就把它翻回 PENDING 并 {@code +1}
|
||||
* stuck_requeue_count。A 那边线程还在跑,重新 claim 后就是同一张表两个执行体;更糟的是计数
|
||||
* 由「谁扫得快」决定,A 的 33149 被累到 20=判死线,下一次任何 5 分钟的停顿都会把还能补传的
|
||||
* 单打成「连续 20 次卡死重排,已判失败」。这与刚根治的 owner 死锁是同一个家族:
|
||||
* 归属的两个源(作业行的 owner 与任务行的 owner_instance_id)各读各的。
|
||||
*
|
||||
* <p>不破坏「归属实例永久消失」的恢复:那种行的 owner 标记为空或指向不在路由表里的实例时,
|
||||
* 本守卫依旧放行(判据只看标记是否指向**另一个明确实例**),而任务侧另有
|
||||
* {@code StaleTaskRepairService} 两小时无心跳的整单收尾兜底。
|
||||
*/
|
||||
public StuckJobResetResult resetStuckRunningJobsDetailed(int stuckMinutes, int limit, String currentInstanceId) {
|
||||
LocalDateTime threshold = LocalDateTime.now().minusMinutes(Math.max(5, stuckMinutes));
|
||||
List<TaskFileJobEntity> jobs = taskFileJobMapper.selectList(new LambdaQueryWrapper<TaskFileJobEntity>()
|
||||
.and(wrapper -> wrapper
|
||||
@@ -425,6 +445,7 @@ public class TaskFileJobService {
|
||||
.orderByAsc(TaskFileJobEntity::getUpdatedAt)
|
||||
.last("limit " + Math.max(1, Math.min(limit, 200))));
|
||||
int reset = 0;
|
||||
int skippedForeignOwner = 0;
|
||||
List<TaskFileJobEntity> exhaustedJobs = new ArrayList<>();
|
||||
List<Long> eventJobIds = new ArrayList<>();
|
||||
List<Long> exhaustedJobIds = new ArrayList<>();
|
||||
@@ -479,6 +500,13 @@ public class TaskFileJobService {
|
||||
// 的外观专利被扫描连扣 4 次,逼到「文件生成任务运行超时,已达到最大重试次数」后
|
||||
// 0 字节收场。卡死次数改用独立列 stuck_requeue_count 记账,达到上限才判死;
|
||||
// 这样既保住"慢执行不扣预算",又不会让无异常挂住的 job 无限重排。
|
||||
String rowOwner = ownerFromScopeKey(job.getScopeKey());
|
||||
if (belongsToAnotherInstance(rowOwner, currentInstanceId)) {
|
||||
skippedForeignOwner++;
|
||||
log.debug("[task-file-job] stuck reset 跳过另一实例正在跑的作业 jobId={} taskId={} owner={} current={}",
|
||||
job.getId(), job.getTaskId(), rowOwner, currentInstanceId);
|
||||
continue;
|
||||
}
|
||||
LocalDateTime now = LocalDateTime.now();
|
||||
int stuckCount = job.getStuckRequeueCount() == null ? 0 : job.getStuckRequeueCount();
|
||||
if (stuckCount >= MAX_STUCK_REQUEUE) {
|
||||
@@ -560,6 +588,10 @@ public class TaskFileJobService {
|
||||
exhaustedJobs.add(refreshed == null ? originalJobsById.get(jobId) : refreshed);
|
||||
}
|
||||
}
|
||||
if (skippedForeignOwner > 0) {
|
||||
log.info("[task-file-job] stuck reset 按归属跳过其它实例的 RUNNING 作业 skipped={} current={} scanned={}",
|
||||
skippedForeignOwner, currentInstanceId, jobs.size());
|
||||
}
|
||||
return new StuckJobResetResult(reset, List.copyOf(exhaustedJobs));
|
||||
}
|
||||
|
||||
@@ -1044,6 +1076,21 @@ public class TaskFileJobService {
|
||||
return owner.isBlank() ? null : owner;
|
||||
}
|
||||
|
||||
/**
|
||||
* 「这一行明确归**另一个**实例」——卡死扫描的归属守卫判据(见
|
||||
* {@link #resetStuckRunningJobsDetailed(int, int, String)})。
|
||||
*
|
||||
* <p>只在两侧都非空且不相等时成立:本机 id 未知(单实例部署、元数据缺失、后管人工复位)时
|
||||
* 守卫整体不启用,绝不能因为它拿不到 id 就把所有重排停掉——那会把唯一能救回遗骸的通道一起关死。
|
||||
*/
|
||||
static boolean belongsToAnotherInstance(String rowOwner, String currentInstanceId) {
|
||||
if (currentInstanceId == null || currentInstanceId.isBlank()) {
|
||||
return false;
|
||||
}
|
||||
String owner = rowOwner == null ? null : rowOwner.trim();
|
||||
return owner != null && !owner.isBlank() && !owner.equals(currentInstanceId.trim());
|
||||
}
|
||||
|
||||
/**
|
||||
* {@link #ownerFromScopeKey} 的逆操作:把 scope_key 最后一个 {@code :owner:} 之后的尾段换成指定实例。
|
||||
* 无标记时原样返回——那种 scope_key 本来就不携带归属信息,不该被凭空造出一个 owner 段。
|
||||
|
||||
+1
-1
@@ -113,7 +113,7 @@ public class TaskResultFileJobWorker {
|
||||
@Scheduled(fixedDelayString = "${aiimage.result-file-job.stuck-scan-delay-ms:60000}")
|
||||
public void resetStuckJobs() {
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
taskFileJobService.resetStuckRunningJobsDetailed(stuckTimeoutMinutes, batchSize);
|
||||
taskFileJobService.resetStuckRunningJobsDetailed(stuckTimeoutMinutes, batchSize, currentInstanceId());
|
||||
if (result.resetCount() > 0 || !result.exhaustedJobs().isEmpty()) {
|
||||
log.warn("[task-file-job] handled stuck running jobs resetCount={} exhaustedCount={} timeoutMinutes={}",
|
||||
result.resetCount(), result.exhaustedJobs().size(), stuckTimeoutMinutes);
|
||||
|
||||
+62
@@ -17,6 +17,7 @@ import org.mockito.junit.jupiter.MockitoExtension;
|
||||
import org.springframework.context.ApplicationEventPublisher;
|
||||
|
||||
import java.time.LocalDateTime;
|
||||
import java.util.List;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
@@ -222,4 +223,65 @@ class TaskFileJobOwnerHandOffTest {
|
||||
|
||||
verify(taskFileJobMapper, never()).update(isNull(), any(LambdaUpdateWrapper.class));
|
||||
}
|
||||
|
||||
// ==================== 卡死扫描的归属守卫(B74-02,同一「两处各读各的」家族) ====================
|
||||
|
||||
@Test
|
||||
void test_归属谓词只在两侧都非空且不相等时成立() {
|
||||
assertTrue(TaskFileJobService.belongsToAnotherInstance(STALE_OWNER, TASK_OWNER));
|
||||
assertFalse(TaskFileJobService.belongsToAnotherInstance(TASK_OWNER, TASK_OWNER),
|
||||
"自己的行必须照旧被扫——关掉守卫等于把唯一能回收遗骸的通道一起关死");
|
||||
assertFalse(TaskFileJobService.belongsToAnotherInstance(null, TASK_OWNER),
|
||||
"老行/未领取行没有 owner 段,不得被守卫永久排除在外");
|
||||
assertFalse(TaskFileJobService.belongsToAnotherInstance(" ", TASK_OWNER));
|
||||
assertFalse(TaskFileJobService.belongsToAnotherInstance(STALE_OWNER, null),
|
||||
"本机 id 未知(单实例部署、元数据缺失、后管人工复位)时守卫整体不启用");
|
||||
assertFalse(TaskFileJobService.belongsToAnotherInstance(STALE_OWNER, " "));
|
||||
assertTrue(TaskFileJobService.belongsToAnotherInstance(" " + STALE_OWNER + " ", TASK_OWNER),
|
||||
"两侧都要 trim:owner 段带空白是 scope_key 拼接的既有形状");
|
||||
}
|
||||
|
||||
@Test
|
||||
void test_卡死扫描不得翻另一实例正在跑的作业() {
|
||||
TaskFileJobEntity foreignRunning = job(23784L, 33149L, "RUNNING", STALE_OWNER);
|
||||
foreignRunning.setStuckRequeueCount(20);
|
||||
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(foreignRunning));
|
||||
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
service.resetStuckRunningJobsDetailed(5, 10, TASK_OWNER);
|
||||
|
||||
assertEquals(0, result.resetCount());
|
||||
assertTrue(result.exhaustedJobs().isEmpty(),
|
||||
"到 20 的行也不得由非归属机判死:兜底出表只在归属机跑得动,先判死就把还能补传的单钉死了");
|
||||
verify(taskFileJobMapper, never()).update(isNull(), any(LambdaUpdateWrapper.class));
|
||||
verify(applicationEventPublisher, never()).publishEvent(any());
|
||||
}
|
||||
|
||||
@Test
|
||||
void test_卡死扫描仍必须翻自己的陈旧作业() {
|
||||
TaskFileJobEntity ownRunning = job(23784L, 33149L, "RUNNING", TASK_OWNER);
|
||||
ownRunning.setStuckRequeueCount(3);
|
||||
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(ownRunning));
|
||||
when(taskFileJobMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
|
||||
when(taskFileJobMapper.selectBatchIds(any())).thenReturn(List.of(ownRunning));
|
||||
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
service.resetStuckRunningJobsDetailed(5, 10, TASK_OWNER);
|
||||
|
||||
assertEquals(1, result.resetCount());
|
||||
verify(applicationEventPublisher).publishEvent(any(TaskFileJobDispatchEvent.class));
|
||||
}
|
||||
|
||||
@Test
|
||||
void test_不带实例id的扫描入口保持原行为_人工复位不受守卫() {
|
||||
TaskFileJobEntity foreignRunning = job(23784L, 33149L, "RUNNING", STALE_OWNER);
|
||||
when(taskFileJobMapper.selectList(any())).thenReturn(List.of(foreignRunning));
|
||||
when(taskFileJobMapper.update(isNull(), any(LambdaUpdateWrapper.class))).thenReturn(1);
|
||||
when(taskFileJobMapper.selectBatchIds(any())).thenReturn(List.of(foreignRunning));
|
||||
|
||||
TaskFileJobService.StuckJobResetResult result = service.resetStuckRunningJobsDetailed(5, 10);
|
||||
|
||||
assertEquals(1, result.resetCount(),
|
||||
"后管那枚人工复位按钮点的就是「把它排回去」,不得被归属守卫悄悄拒掉");
|
||||
}
|
||||
}
|
||||
|
||||
+2
-1
@@ -44,6 +44,7 @@ import java.util.List;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.anyString;
|
||||
import static org.mockito.ArgumentMatchers.eq;
|
||||
import static org.mockito.Mockito.doThrow;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.never;
|
||||
@@ -173,7 +174,7 @@ class TaskResultFileJobWorkerFailureNotifyTest {
|
||||
job.setErrorMessage("文件生成任务运行超时,已达到最大重试次数");
|
||||
TaskFileJobService.StuckJobResetResult resetResult =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of(job));
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0)).thenReturn(resetResult);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString())).thenReturn(resetResult);
|
||||
doThrow(new IllegalStateException("owner mismatch"))
|
||||
.when(similar).handleResultFileJobFailure(any(), anyString());
|
||||
|
||||
|
||||
+16
-16
@@ -126,11 +126,11 @@ class TaskResultFileJobWorkerStuckScanTest {
|
||||
setInt(worker, "batchSize", 30);
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
new TaskFileJobService.StuckJobResetResult(2, List.of());
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(45, 30)).thenReturn(result);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(45), eq(30), anyString())).thenReturn(result);
|
||||
|
||||
worker.resetStuckJobs();
|
||||
|
||||
verify(taskFileJobService).resetStuckRunningJobsDetailed(45, 30);
|
||||
verify(taskFileJobService).resetStuckRunningJobsDetailed(eq(45), eq(30), anyString());
|
||||
verify(taskFileJobService, never()).markFailureFinalized(any(), anyString());
|
||||
}
|
||||
|
||||
@@ -140,7 +140,7 @@ class TaskResultFileJobWorkerStuckScanTest {
|
||||
TaskFileJobEntity job = exhaustedJob(1L, 11L, null);
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of(job));
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0)).thenReturn(result);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString())).thenReturn(result);
|
||||
|
||||
worker.resetStuckJobs();
|
||||
|
||||
@@ -154,7 +154,7 @@ class TaskResultFileJobWorkerStuckScanTest {
|
||||
TaskFileJobEntity job = exhaustedJob(2L, 12L, " ");
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of(job));
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0)).thenReturn(result);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString())).thenReturn(result);
|
||||
|
||||
worker.resetStuckJobs();
|
||||
|
||||
@@ -168,7 +168,7 @@ class TaskResultFileJobWorkerStuckScanTest {
|
||||
TaskFileJobEntity job = exhaustedJob(3L, 13L, "上游 API 限流,重试耗尽");
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of(job));
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0)).thenReturn(result);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString())).thenReturn(result);
|
||||
|
||||
worker.resetStuckJobs();
|
||||
|
||||
@@ -182,7 +182,7 @@ class TaskResultFileJobWorkerStuckScanTest {
|
||||
TaskFileJobEntity job = exhaustedJob(4L, 14L, null);
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of(job));
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0)).thenReturn(result);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString())).thenReturn(result);
|
||||
|
||||
worker.resetStuckJobs();
|
||||
|
||||
@@ -196,11 +196,11 @@ class TaskResultFileJobWorkerStuckScanTest {
|
||||
TaskResultFileJobWorker worker = buildWorker();
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of());
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0)).thenReturn(result);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString())).thenReturn(result);
|
||||
|
||||
worker.resetStuckJobs();
|
||||
|
||||
verify(taskFileJobService).resetStuckRunningJobsDetailed(0, 0);
|
||||
verify(taskFileJobService).resetStuckRunningJobsDetailed(eq(0), eq(0), anyString());
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -208,11 +208,11 @@ class TaskResultFileJobWorkerStuckScanTest {
|
||||
TaskResultFileJobWorker worker = buildWorker();
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of());
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0)).thenReturn(result);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString())).thenReturn(result);
|
||||
|
||||
worker.resetStuckJobs();
|
||||
|
||||
verify(taskFileJobService).resetStuckRunningJobsDetailed(0, 0);
|
||||
verify(taskFileJobService).resetStuckRunningJobsDetailed(eq(0), eq(0), anyString());
|
||||
verify(similar, never()).handleResultFileJobFailure(any(), anyString());
|
||||
verify(taskFileJobService, never()).markFailureFinalized(any(), anyString());
|
||||
}
|
||||
@@ -223,13 +223,13 @@ class TaskResultFileJobWorkerStuckScanTest {
|
||||
TaskFileJobEntity job = exhaustedJob(5L, 15L, null);
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of(job));
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0))
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString()))
|
||||
.thenReturn(result, new TaskFileJobService.StuckJobResetResult(0, List.of()));
|
||||
|
||||
worker.resetStuckJobs();
|
||||
worker.resetStuckJobs();
|
||||
|
||||
verify(taskFileJobService, times(2)).resetStuckRunningJobsDetailed(0, 0);
|
||||
verify(taskFileJobService, times(2)).resetStuckRunningJobsDetailed(eq(0), eq(0), anyString());
|
||||
verify(taskFileJobService).markFailureFinalized(5L, "文件生成任务运行超时,已达到最大重试次数");
|
||||
}
|
||||
|
||||
@@ -239,7 +239,7 @@ class TaskResultFileJobWorkerStuckScanTest {
|
||||
TaskFileJobEntity job = exhaustedJob(6L, 16L, null);
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of(job));
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0)).thenReturn(result);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString())).thenReturn(result);
|
||||
org.mockito.Mockito.doThrow(new IllegalStateException("db down"))
|
||||
.when(similar).handleResultFileJobFailure(job, "文件生成任务运行超时,已达到最大重试次数");
|
||||
|
||||
@@ -255,7 +255,7 @@ class TaskResultFileJobWorkerStuckScanTest {
|
||||
TaskFileJobEntity second = exhaustedJob(8L, 18L, "超时");
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of(first, second));
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0)).thenReturn(result);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString())).thenReturn(result);
|
||||
|
||||
worker.resetStuckJobs();
|
||||
|
||||
@@ -298,7 +298,7 @@ class TaskResultFileJobWorkerStuckScanTest {
|
||||
job.setModuleType("UNKNOWN_X");
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of(job));
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0)).thenReturn(result);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString())).thenReturn(result);
|
||||
|
||||
worker.resetStuckJobs();
|
||||
|
||||
@@ -528,7 +528,7 @@ class TaskResultFileJobWorkerStuckScanTest {
|
||||
job.setScopeKey("appearance-patent:19:owner:instance-b");
|
||||
TaskFileJobService.StuckJobResetResult result =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of(job));
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0)).thenReturn(result);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString())).thenReturn(result);
|
||||
|
||||
worker.resetStuckJobs();
|
||||
|
||||
|
||||
+4
-3
@@ -35,6 +35,7 @@ import org.mockito.junit.jupiter.MockitoExtension;
|
||||
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.anyString;
|
||||
import static org.mockito.ArgumentMatchers.eq;
|
||||
import static org.mockito.Mockito.inOrder;
|
||||
import static org.mockito.Mockito.doThrow;
|
||||
import static org.mockito.Mockito.lenient;
|
||||
@@ -220,7 +221,7 @@ class TaskResultFileJobWorkerTest {
|
||||
job.setErrorMessage("文件生成任务运行超时,已达到最大重试次数");
|
||||
TaskFileJobService.StuckJobResetResult resetResult =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of(job));
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0)).thenReturn(resetResult);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString())).thenReturn(resetResult);
|
||||
stubSimilarAsinRegistry();
|
||||
|
||||
worker.resetStuckJobs();
|
||||
@@ -235,7 +236,7 @@ class TaskResultFileJobWorkerTest {
|
||||
TaskFileJobEntity second = exhaustedSimilarAsinJob(13859L, 20554L);
|
||||
TaskFileJobService.StuckJobResetResult resetResult =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of(first, second));
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0)).thenReturn(resetResult);
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString())).thenReturn(resetResult);
|
||||
stubSimilarAsinRegistry();
|
||||
doThrow(new IllegalStateException("owner mismatch"))
|
||||
.doNothing()
|
||||
@@ -255,7 +256,7 @@ class TaskResultFileJobWorkerTest {
|
||||
TaskFileJobEntity job = exhaustedSimilarAsinJob(13860L, 20555L);
|
||||
TaskFileJobService.StuckJobResetResult resetResult =
|
||||
new TaskFileJobService.StuckJobResetResult(0, List.of(job));
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(0, 0))
|
||||
when(taskFileJobService.resetStuckRunningJobsDetailed(eq(0), eq(0), anyString()))
|
||||
.thenReturn(resetResult, resetResult);
|
||||
stubSimilarAsinRegistry();
|
||||
doThrow(new IllegalStateException("temporary database failure"))
|
||||
|
||||
Reference in New Issue
Block a user