fix(appearance-patent): 补检改吃独立预算,缺口不再以"排不上队"的名义交付半张表

现网读数(2026-10-19 近 24h 日志):交付缺口触发过补检,但从没补齐过大面积缺口——
"补检排不上队,按已有结果交付" 四例 missingRows=112/455/1471/2110,作业全部 retry_count=5。
根因是一条自我锁死:补检重排走 requeue,而 requeue 的 WHERE 带 retry_count<MAX_RETRY_COUNT,
把这条预算抛满的恰恰是"还没检完"本身(覆盖率抛、空结果抛)。越慢 ⇒ 抛得越多 ⇒ 越补不上。

改法是把两条预算拆开,且只用底座现成件(modules/task/** 是共享文件,且
TaskFileJobService 在册 929/实 928,只剩 1 行棘轮余量,加方法必判红):

- recheck-max-rounds(默认 2):补检自己的轮次上限,与行级 llm-row-max-rounds 分家。
- completeLlmFileJob:补检决定挪到 assembleResultWorkbook 之前(新增 measureDeliveryFacts,
  只量行账不出表)——那一跑完就把成品传上 OSS 并挂好 resultFileUrl,那时才发现缺口,
  半张表已经在用户的下载列表里了。
- requestRecheckForMissingRows:requeue 排不上队时改吃抛,不再退化成按已有结果交付;
  让作业走到终态失败,由判死出口按补检预算再判一次。
- fallbackAssembleResultFile:出表前先试 scheduleFallbackRecheck,复用现成的
  resetTerminalFailedForRecovery(终态失败+retry 到顶 ⇒ 清零、重回 PENDING、重派发),
  补传恢复那条路一直在用它,只有交付缺口这条路没接上。轮次只在重排真接上之后才记,
  否则一轮没接成的重排会自己把预算吃光。预算耗尽仍照常出表——
  "起码失败了也可以下载"那条保证不变。

门禁:后端整量 mvn -o test 4549 例/0 红/0 错/16 skipped,BUILD SUCCESS;
定向 43 例(交付闸门 14+兜底出表 5+行标记 15+CatchSilence 9)全绿;
ops 面 ruff check . All checks passed、readability 棘轮 9 passed(AP 3048→3121,基线 3458)。

整量第一次跑抓到的是本笔自己的账:新增两枚 Redis 降级方法各带一处不重抛的 catch,
把 CatchSilenceRatchetTest 的"不重抛"从 562 顶到 564。没放宽那格判据,改成去掉这两处
新 catch——补检轮次直接复用既有的行级轮次计数(同 24h TTL、同降级口径),借哨兵键
__recheck__;行键恒为 id::ASIN::国家、必带 "::",撞不上。
This commit is contained in:
2026-10-20 00:41:29 +08:00
parent 773c41254f
commit 3c662389c0
5 changed files with 212 additions and 15 deletions
@@ -98,4 +98,15 @@ public class AppearancePatentProperties {
* 分母口径已改为「会写进结果文件的行集」,判据见 AppearancePatentRowOutcome。
*/
private double llmCoverageMinRatio = 0.95D;
/**
* 交付缺口最多允许再补检几轮(<=0 关闭补检)。
*
* <p>它存在的理由是「补检不能借用作业重试那条预算」:组装作业重排走 {@code requeue},
* 而 {@code requeue} 的 WHERE 带 {@code retry_count < MAX_RETRY_COUNT}——恰恰是"还没检完"
* 这件事把 retry_count 抛满的(覆盖率抛、空结果抛),于是越慢越补不上。
* 2026-10-19 现网读数:{@code 补检排不上队,按已有结果交付} 四例 missingRows=112/455/1471/2110,
* 作业全部 retry_count=5。
*/
private int recheckMaxRounds = 2;
}
@@ -28,6 +28,9 @@ public class AppearancePatentTaskCacheService extends AbstractPendingRowTaskCach
private static final long TTL_HOURS = 24;
/** 补检轮次借用的那枚哨兵键,见 {@link #recheckRounds(Long)}。 */
private static final String RECHECK_ROUND_KEY = "__recheck__";
public AppearancePatentTaskCacheService(StringRedisTemplate stringRedisTemplate,
ObjectMapper objectMapper) {
super("appearance-patent", AppearancePatentResultRowDto.class, stringRedisTemplate, objectMapper);
@@ -177,6 +180,25 @@ public class AppearancePatentTaskCacheService extends AbstractPendingRowTaskCach
}
}
/**
* 交付缺口补检已经排过几轮。
*
* <p>刻意复用行级轮次那套计数(同 24h TTL、同「Redis 降级按 0 计」口径)而不是另抄一遍:
* 另抄就是两处新的「异常到此为止、外面看不见」,正是 {@code CatchSilenceRatchetTest}
* 只许减的那本账。哨兵键撞不上真行——行键恒为 {@code id::ASIN::国家},必带 {@code "::"}。
*/
public long recheckRounds(Long taskId) {
return rowInspectionRounds(taskId, RECHECK_ROUND_KEY);
}
/** 排上一轮补检时记一次。 */
public void countRecheckRound(Long taskId) {
if (taskId == null || taskId <= 0) {
return;
}
countRowInspectionRound(taskId, List.of(RECHECK_ROUND_KEY));
}
private String rowRoundsKey(Long taskId, String rowKey) {
return "appearance-patent:task:row-rounds:" + taskId + ":" + rowKey;
}
@@ -1685,9 +1685,15 @@ public class AppearancePatentTaskService extends AbstractFileTaskPipeline<Appear
* 有行没跑完就先别把半张表当成品发:撤销这些行的检测标记(完成态与处理中都要撤,
* 否则 {@code beginRowInspection} 下一轮仍然申领不到它们),把作业排回队列补检一轮。
*
* <p>不会无限补检:作业重排受 {@code TaskFileJobService} 的 MAX_RETRY_COUNT 约束(requeue 返回
* false 时这里就放行出文件),行级重发次数受 {@code llmRowMaxRounds} 约束(到顶的行会被判死并
* 写上「重发轮次已达上限」的结论,从此不再是空行)。
* <p>排不上队时**不许**退化成"按已有结果交付"。现网(2026-10-19)四例
* {@code 交付缺口补检排不上队,按已有结果交付} 的 missingRows 是 112/455/1471/2110,
* 作业全部 {@code retry_count=5}——把预算抛满的恰恰是"还没检完"(覆盖率抛、空结果抛),
* 于是 {@code requeue} 那条 {@code retry_count < MAX} 的闸自我锁死:越慢越补不上。
* 这里改吃抛:让作业走到终态失败,由 {@link #fallbackAssembleResultFile} 按**补检自己的**
* 预算({@code recheckMaxRounds})决定重排还是放行,两条预算不再互相占用。
*
* <p>行级重发次数仍受 {@code llmRowMaxRounds} 约束(到顶的行会被判死并写上
* 「重发轮次已达上限」的结论,从此不再是空行),所以这条不会无限循环。
*/
private boolean requestRecheckForMissingRows(FileTaskEntity task, TaskFileJobEntity job,
AppearancePatentRowOutcome.Facts facts) {
@@ -1701,13 +1707,75 @@ public class AppearancePatentTaskService extends AbstractFileTaskPipeline<Appear
taskCacheService.releaseRowInspection(task.getId(), keys, true);
boolean requeued = taskFileJobService.requeue(job.getId(),
"有 " + facts.missingRows() + " 行未完成检测,补检一轮后重新组装结果文件");
if (!requeued) {
log.warn("[appearance-patent] 交付缺口补检排不上队,按已有结果交付 taskId={} jobId={} missingRows={}",
task.getId(), job.getId(), facts.missingRows());
if (requeued) {
log.warn("[appearance-patent] 交付缺口触发补检轮 taskId={} jobId={} missingRows={} keys={} noDataRows={}",
task.getId(), job.getId(), facts.missingRows(), keys.size(), facts.noDataRows());
return true;
}
log.warn("[appearance-patent] 交付缺口补检排不上队,转判死出口按补检预算处置 taskId={} jobId={} missingRows={}",
task.getId(), job.getId(), facts.missingRows());
throw new BusinessException("有 " + facts.missingRows() + " 行未完成检测,等待补检轮");
}
/**
* 只量行账、不出表。补检的决定必须做在 {@code assembleResultWorkbook} 之前,而那一跑完就把
* 成品传上 OSS 并挂好了 {@code resultFileUrl}。判据与出表共用
* {@link AppearancePatentRowOutcome#summarize},两边不许各判一套。
*/
private AppearancePatentRowOutcome.Facts measureDeliveryFacts(FileTaskEntity task, FileResultEntity result) {
AppearancePatentParsedPayloadDto parsed = readParsedPayload(task);
List<AppearancePatentParsedRowVo> parsedRows = parsed.getAllItems() == null
? List.of() : parsed.getAllItems();
AppearancePatentRowOutcome.Facts facts = AppearancePatentRowOutcome.Facts.EMPTY;
for (AppearancePatentResultPackager.SourceRows item
: splitRowsBySourceFile(parsed, parsedRows, result.getSourceFilename())) {
Map<String, AppearancePatentResultRowDto> resultMap =
loadPersistedResultRowsForRowsWithRetry(task.getId(), item.rows());
facts = facts.plus(AppearancePatentRowOutcome.summarize(
item.rows(), row -> findResultRow(row, resultMap)));
}
return facts;
}
/**
* 判死出口上的第二条补检预算。
*
* <p>{@link #fallbackAssembleResultFile} 的注释原来断言「作业已判死,缺口不再有补齐窗口」,
* 现网读数证明它是假的:底座早就有 {@code resetTerminalFailedForRecovery}(终态失败且 retry
* 到顶 ⇒ retryCount 清零、重回 PENDING、重派发),补传恢复那条路一直在用它,只有交付缺口
* 这条路没接上。轮次上限走 Redis 的 {@code recheckMaxRounds},与作业重试预算彻底分开——
* 「因为慢而抛满重试」不再等于「永远补不了」。
*
* <p>预算耗尽、或重排没接上时仍按已有结果出表:「起码失败了也可以下载」那条保证不变。
*/
private boolean scheduleFallbackRecheck(FileTaskEntity task, FileResultEntity result,
AppearancePatentRowOutcome.Facts facts) {
int maxRounds = properties.getRecheckMaxRounds();
if (task == null || task.getId() == null || result == null || result.getId() == null
|| facts == null || maxRounds <= 0) {
return false;
}
log.warn("[appearance-patent] 交付缺口触发补检轮 taskId={} jobId={} missingRows={} keys={} noDataRows={}",
task.getId(), job.getId(), facts.missingRows(), keys.size(), facts.noDataRows());
long done = taskCacheService.recheckRounds(task.getId());
if (done >= maxRounds) {
log.warn("[appearance-patent] 补检预算已耗尽,按已有结果交付 taskId={} rounds={}/{}",
task.getId(), done, maxRounds);
return false;
}
if (facts.missingRows() <= 0 || facts.missingKeys().isEmpty()) {
return false;
}
taskCacheService.releaseRowInspection(task.getId(), facts.missingKeys(), true);
TaskFileJobEntity reset = taskFileJobService.resetTerminalFailedForRecovery(
task.getId(), MODULE_TYPE, result.getId());
if (reset == null || !"PENDING".equals(reset.getStatus())) {
log.warn("[appearance-patent] 判死出口补检重排未接上,按已有结果交付 taskId={} resultId={} jobStatus={}",
task.getId(), result.getId(), reset == null ? "ABSENT" : reset.getStatus());
return false;
}
// 轮次只在重排真的接上之后才记:否则"没接上"也要扣一轮,预算会被失败的重排自己吃光
taskCacheService.countRecheckRound(task.getId());
log.warn("[appearance-patent] 判死出口触发补检轮 taskId={} jobId={} round={}/{} missingRows={} keys={}",
task.getId(), reset.getId(), done + 1, maxRounds, facts.missingRows(), facts.missingKeys().size());
return true;
}
@@ -1717,12 +1785,14 @@ public class AppearancePatentTaskService extends AbstractFileTaskPipeline<Appear
int totalProgressUnits) {
int assembleProgress = Math.max(1, totalProgressUnits - 2);
saveFileBuildProgress(task, job, totalProgressUnits, assembleProgress, "正在组装 xlsx");
AppearancePatentRowOutcome.Facts deliveryFacts = assembleResultWorkbook(task, result);
if (requestRecheckForMissingRows(task, job, deliveryFacts)) {
// 补检决定必须做在出表之前:assembleResultWorkbook 一跑完就把 xlsx 传上 OSS 并挂好
// resultFileUrl,那时才发现缺口,半张表已经在用户的下载列表里了。
if (requestRecheckForMissingRows(task, job, measureDeliveryFacts(task, result))) {
// 半张表先不发出去:撤销未完成行的检测标记、把作业排回队列补检一轮,
// 上限由行级轮次闸(llmRowMaxRounds)与作业重试上限共同兜住。
// 上限由行级轮次闸(llmRowMaxRounds)与补检轮次闸(recheckMaxRounds)共同兜住。
return false;
}
AppearancePatentRowOutcome.Facts deliveryFacts = assembleResultWorkbook(task, result);
saveFileBuildProgress(task, job, totalProgressUnits, totalProgressUnits - 1, "正在上传结果文件");
fileResultMapper.updateById(result);
boolean taskAlreadyFailed = STATUS_FAILED.equals(task.getStatus())
@@ -1788,6 +1858,9 @@ public class AppearancePatentTaskService extends AbstractFileTaskPipeline<Appear
job.getTaskId(), job.getResultId());
return false;
}
if (scheduleFallbackRecheck(task, result, measureDeliveryFacts(task, result))) {
return false;
}
AppearancePatentRowOutcome.Facts facts = assembleResultWorkbook(task, result, true);
fileResultMapper.updateById(result);
log.warn("[appearance-patent] fallback assemble produced partial result taskId={} resultId={} jobStatus={} expectedRows={} judgedRows={} missingRows={} noDataRows={}",
@@ -2059,7 +2132,8 @@ public class AppearancePatentTaskService extends AbstractFileTaskPipeline<Appear
/**
* @param allowPartial 供失败兜底路径({@link #fallbackAssembleResultFile})使用:
* 覆盖率闸门按「允许部分结果」放行——作业已判死,缺口不再有补齐窗口,
* 覆盖率闸门按「允许部分结果」放行——那条路在调用本方法前已经先问过
* {@link #scheduleFallbackRecheck},缺口能补的都补过了、补检预算也见底了,
* 此刻拦下成品只会让用户连半张表都拿不到;缺口仍由行账锚点记数。
*/
private AppearancePatentRowOutcome.Facts assembleResultWorkbook(FileTaskEntity task, FileResultEntity result,
@@ -393,6 +393,10 @@ aiimage:
# 单行最多被重发几轮(<=0 关闭)。失败行会被释放、下一轮重新申领,没有这道闸时
# 上游劣化窗口内会无限重发并每轮扣用户 LLM 次数——deferRunning 不累加 retryCount,任务层那道永不触发。
llm-row-max-rounds: ${AIIMAGE_APPEARANCE_PATENT_LLM_ROW_MAX_ROUNDS:3}
# 交付缺口补检轮数上限:与上一行是两条独立预算。上一行管"同一行被重发几次",
# 这一行管"整单还允许再补检几轮"——补检重排走 requeue,而它带 retry_count<5 的闸,
# 用作业重试预算当补检预算会自我锁死(2026-10-19 现网四例 112/455/1471/2110 行)。
recheck-max-rounds: ${AIIMAGE_APPEARANCE_PATENT_RECHECK_MAX_ROUNDS:2}
flush-pending-minutes: ${AIIMAGE_APPEARANCE_PATENT_FLUSH_PENDING_MINUTES:1}
stale-timeout-minutes: ${AIIMAGE_APPEARANCE_PATENT_STALE_TIMEOUT_MINUTES:30}
stale-finalize-cron: ${AIIMAGE_APPEARANCE_PATENT_STALE_FINALIZE_CRON:0 */2 * * * *}
@@ -15,6 +15,7 @@ import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
@@ -84,6 +85,7 @@ class AppearancePatentDeliveryGateTest {
@BeforeEach
void setUp() {
when(properties.getLlmCoverageMinRatio()).thenReturn(0.95D);
when(properties.getRecheckMaxRounds()).thenReturn(2);
service = new AppearancePatentTaskService(
localFileStorageService, null,
mock(com.nanri.aiimage.common.service.ResultDownloadResolver.class),
@@ -205,7 +207,38 @@ class AppearancePatentDeliveryGateTest {
"requestRecheckForMissingRows", FileTaskEntity.class, TaskFileJobEntity.class,
AppearancePatentRowOutcome.Facts.class);
method.setAccessible(true);
return (boolean) method.invoke(service, task, job, facts);
try {
return (boolean) method.invoke(service, task, job, facts);
} catch (InvocationTargetException ex) {
// 与 validate/validatePartial 同一口径:不剥壳,assertThrows 就只能断到反射壳本身
if (ex.getCause() instanceof RuntimeException runtime) {
throw runtime;
}
throw ex;
}
}
private static FileResultEntity result(long id) {
FileResultEntity result = new FileResultEntity();
result.setId(id);
result.setTaskId(TASK_ID);
result.setModuleType("APPEARANCE_PATENT");
return result;
}
private static TaskFileJobEntity pendingJob(Long id) {
TaskFileJobEntity job = job(id);
job.setStatus("PENDING");
return job;
}
private boolean scheduleFallbackRecheck(FileTaskEntity task, FileResultEntity result,
AppearancePatentRowOutcome.Facts facts) throws Exception {
Method method = AppearancePatentTaskService.class.getDeclaredMethod(
"scheduleFallbackRecheck", FileTaskEntity.class, FileResultEntity.class,
AppearancePatentRowOutcome.Facts.class);
method.setAccessible(true);
return (boolean) method.invoke(service, task, result, facts);
}
private String appendGap(String existingError, AppearancePatentRowOutcome.Facts facts) throws Exception {
@@ -306,12 +339,16 @@ class AppearancePatentDeliveryGateTest {
}
@Test
void recheckRefusalFallsBackToDeliveringWhatWeHave() throws Exception {
void recheckRefusalDoesNotShipTheHalfTable() throws Exception {
// 旧口径是"补检排不上队就按已有结果交付",现网据此发出过 missingRows=112/455/1471/2110
// 四张半表。改成吃抛:作业走到终态失败,由判死出口按补检自己的预算再判一次。
List<AppearancePatentParsedRowVo> rows = sheetRows(2);
AppearancePatentRowOutcome.Facts facts = summarize(rows, submittedRows(rows, 1, true));
when(taskFileJobService.requeue(anyLong(), any())).thenReturn(false);
assertFalse(requestRecheck(task("SUCCESS"), job(500L), facts));
BusinessException ex = assertThrows(BusinessException.class,
() -> requestRecheck(task("SUCCESS"), job(500L), facts));
assertTrue(ex.getMessage().contains("1 行未完成检测"), ex.getMessage());
verify(taskCacheService).releaseRowInspection(eq(TASK_ID), anyList(), eq(true));
}
@@ -324,6 +361,55 @@ class AppearancePatentDeliveryGateTest {
verify(taskFileJobService, never()).requeue(anyLong(), any());
}
// ---------- 判死出口的补检预算 ----------
@Test
void deadJobWithGapIsRequeuedOnItsOwnBudgetInsteadOfShipping() throws Exception {
List<AppearancePatentParsedRowVo> rows = sheetRows(2);
AppearancePatentRowOutcome.Facts facts = summarize(rows, submittedRows(rows, 1, true));
when(taskCacheService.recheckRounds(TASK_ID)).thenReturn(0L);
when(taskFileJobService.resetTerminalFailedForRecovery(TASK_ID, "APPEARANCE_PATENT", 700L))
.thenReturn(pendingJob(500L));
assertTrue(scheduleFallbackRecheck(task("FAILED"), result(700L), facts));
verify(taskCacheService).releaseRowInspection(eq(TASK_ID), anyList(), eq(true));
verify(taskCacheService).countRecheckRound(TASK_ID);
}
@Test
void exhaustedRecheckBudgetShipsWhatWeHave() throws Exception {
List<AppearancePatentParsedRowVo> rows = sheetRows(2);
AppearancePatentRowOutcome.Facts facts = summarize(rows, submittedRows(rows, 1, true));
when(taskCacheService.recheckRounds(TASK_ID)).thenReturn(2L);
assertFalse(scheduleFallbackRecheck(task("FAILED"), result(700L), facts));
verify(taskFileJobService, never()).resetTerminalFailedForRecovery(anyLong(), any(), anyLong());
}
@Test
void deadJobWithoutGapShipsImmediately() throws Exception {
List<AppearancePatentParsedRowVo> rows = sheetRows(2);
when(taskCacheService.recheckRounds(TASK_ID)).thenReturn(0L);
assertFalse(scheduleFallbackRecheck(task("FAILED"), result(700L),
summarize(rows, submittedRows(rows, 2, true))));
verify(taskFileJobService, never()).resetTerminalFailedForRecovery(anyLong(), any(), anyLong());
}
@Test
void failedRequeueDoesNotBurnARecheckRound() throws Exception {
// 重排没接上(作业不在"FAILED 且 retry 到顶"那格)时不能扣轮次:
// 否则一轮没补成的重排会把预算自己吃光,缺口行连一次补齐机会都没有就出表
List<AppearancePatentParsedRowVo> rows = sheetRows(2);
AppearancePatentRowOutcome.Facts facts = summarize(rows, submittedRows(rows, 1, true));
when(taskCacheService.recheckRounds(TASK_ID)).thenReturn(0L);
when(taskFileJobService.resetTerminalFailedForRecovery(TASK_ID, "APPEARANCE_PATENT", 700L))
.thenReturn(job(500L));
assertFalse(scheduleFallbackRecheck(task("FAILED"), result(700L), facts));
verify(taskCacheService, never()).countRecheckRound(anyLong());
}
// ---------- 任务消息 ----------
@Test