fix(tasks): 批量入库与 folder 去重逐条容错,发布文案不说谎

B49/B50 收尾三处当时被标"非 ★"跳过的同类缺陷。判据同一条:一颗坏项不许让
其余项一起不干活,也不许把"根本没拿到数据"说成"数据正常"。

1) shopkey/service/QueryAsinService.java(B49 #6,flushInsert:843-844)
   复验为真:整张表共用一条读行循环,某一批 insertBatch 抛出即冒到
   runImportTask 的 catch,后面的批次一个都跑不到、内存里的余行一起丢。
   审计里"报数虚高"那条已被复验否掉(原状下坏批计数确实不外发),本笔不按
   那个理由改;但逐批容错之后坏批计数会随后续进度外发,所以必须按"没落库"
   回退:insertedCount 扣回、skippedCount 记上、坏批点名进日志与 errorMessage。
   收尾新增 failIfBatchesLost():有坏批就让导入落"失败"态并报出缺口行数——
   admin 侧只在 failed 分支显示 errorMessage(QueryAsinPage.vue:260-271),
   留在 success 上等于把没入库的行说成导入完成。重传整表是安全的:已入库行
   会被 rowSignature 认成重复跳过。
   processImportFile 由 private 收到包级作测试接缝(形状同 tryAcquireFileSlot)。
   该 importer 此前零夹具,新建 3 例(真 xlsx + 真 ExcelStreamReader)。

2) dedupe/service/DedupeRunService.java(B49 #4 的 folder 分支)
   只在 folder 模式成立:逐文件清洗结果只落本地临时目录,唯一交付物是那个
   ZIP。两颗坏项会带走整批——① 收尾的 aborted 判定排在打包之前,一个 worker
   线程级异常就让打包整步跳过;② 打包循环里一个条目读不到/条目名相撞就整包
   作废。现改为 folder 分支先打包再判 aborted,打包逐条目容错并点名跳过;
   一个都没写进去时拒绝交空包(那才是"根本没拿到数据")。aborted 仍照旧把
   终态落 FAILED,不拿部分交付冒充全量成功;缺口写在交付条目与日志上,
   不写 file_task.error_message。非 folder 分支一个字节没动。
   DedupeArchiveEntry 与打包方法收到包级作测试接缝。新建 3 例,含一支用
   同线程执行器让第 2 个 worker 当场线程级异常的真链路用例。

3) publish/service/PublishTaskService.java(B49 #7/B50 #4,607-609)
   按上一轮复验结论只改文案,不动状态语义:逐文件 status/VO 本来就交付了。
   会说谎的地方是 publishResultFile(..., preserveFailure=false) 把结果行
   error_message 洗成 null,"N 个文件没进这张表"在文案层等于全部成功。
   缺口写在 biz_file_result.error_message(本仓既有"部分交付"载体:
   ShopMatch:825/PriceTrack:905/ProductRisk:827 都按它判 partialFailure;
   publish 侧只有 toResultVo:1216 读它,纯展示,fileReady 由 success+url 推导
   不受影响)。SUCCESS 的 file_task.error_message 继续留空并就地写明理由:
   TaskRetryEligibility.forceMajeureCategory 与 SimilarAsinStaleReopenSupport
   .isStaleFailure 都按那段文字做字面量分类,写进去会喂错 stale/重开/续跑判据。
   文案两个方向都不说谎:说清"只含成功处理的 X 个、另有 N 个未纳入(点名)",
   并明写"不是表格格式问题、可只重新提交这些文件",不把可恢复失败推给表格。
   新建 2 例(含反向钉:全成功时不得凭空造缺口文案)。

实测:mvn -o test-compile rc=0;新增 8 例改前 6 红(QueryAsin 2 红=原始
RuntimeException 直接冒出、批次被提前收走;Dedupe 3 红="Wanted but not
invoked: fileResultMapper.insert"+整包作废;Publish 1 红=errorMessage 恒
null),改后 8/8 绿;相邻既有 9 支类 97 例全绿(PublishTaskServiceTest 33、
DedupeRunProgressResumeTest 14、PublishFileLevelStatusTest 8、
PublishResultFileJobHandlerTest 8、DedupeRunServiceCleanRulesTest 9、
DedupeRunServiceFileSlotTest 2、QueryAsinResolveServiceTest 13 等)。

CatchSilenceRatchetTest 按预期判红(本笔新增 2 处 catch,均在被禁改的共享台账
里,合并时由主线程统一重钉):实测 total=1028(基线 1026)、
notRethrow(incl logOnly)=579(基线 577)、logOnly=181(不变)、
rethrow=393(不变)、empty=56(不变)。
This commit is contained in:
2026-10-27 16:02:11 +08:00
parent e08b5369d7
commit 88d737a7f0
6 changed files with 811 additions and 16 deletions
@@ -0,0 +1,262 @@
package com.nanri.aiimage.modules.dedupe.service;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.config.StorageProperties;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.dedupe.model.dto.DedupeRunRequest;
import com.nanri.aiimage.modules.dedupe.model.dto.DedupeSourceFileDto;
import com.nanri.aiimage.modules.dedupe.model.vo.DedupeRunProgressVo;
import com.nanri.aiimage.modules.dedupe.service.DedupeRunService.DedupeArchiveEntry;
import com.nanri.aiimage.modules.dedupe.service.DedupeRunService.FolderPackage;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import org.apache.poi.ss.usermodel.Row;
import org.apache.poi.ss.usermodel.Sheet;
import org.apache.poi.xssf.usermodel.XSSFWorkbook;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.api.io.TempDir;
import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.test.util.ReflectionTestUtils;
import java.io.File;
import java.io.FileOutputStream;
import java.nio.file.Path;
import java.util.List;
import java.util.Set;
import java.util.concurrent.AbstractExecutorService;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.atLeastOnce;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* 文件夹去重的打包容错(B49 #4 的 folder 分支)。
*
* <p>folder 模式的逐文件清洗结果只落在本地临时目录,<b>唯一交付物就是那个 ZIP</b>。原形状有两颗
* 会把整批带走的坏项:① 打包循环里一个条目读不到(临时文件被清掉/写失败)就整包作废;
* ② 收尾时一个 worker 线程级异常先判 {@code aborted} 再打包,于是打包整步被跳过,
* 已清洗完的 N-1 个文件一起不交付。两条口径都是「一颗坏项不许让其余项一起不干活」,
* 同时终态仍必须如实说「没全跑完」。
*/
@ExtendWith(MockitoExtension.class)
class DedupeRunServiceFolderPackageTest {
private static final String[] HEADERS = {"id", "ASIN", "国家", "价格", "发货类型", "品牌", "状态", "创建时间"};
@Mock
private FileTaskMapper fileTaskMapper;
@Mock
private FileResultMapper fileResultMapper;
@Mock
private StorageProperties storageProperties;
@Mock
private OssStorageService ossStorageService;
@Mock
private DedupeTotalDataService dedupeTotalDataService;
private final ObjectMapper objectMapper = new ObjectMapper();
private DedupeRunService newService() {
return new DedupeRunService(fileTaskMapper, fileResultMapper, storageProperties, ossStorageService,
dedupeTotalDataService, null, null, objectMapper, new InstanceMetadata("test-instance"));
}
/** 逐条目容错:坏条目只带走自己,其余结果文件照常进包,并被点名报出来。 */
@Test
@DisplayName("一个坏条目只带走自己:其余结果文件照常进包,坏条目点名")
void badEntryDoesNotDiscardTheWholeFolderPackage(@TempDir Path tempDir) throws Exception {
when(storageProperties.getLocalTempDir()).thenReturn(tempDir.toString());
File goodFirst = tempDir.resolve("a.xlsx").toFile();
File goodLast = tempDir.resolve("b.xlsx").toFile();
assertTrue(goodFirst.createNewFile());
assertTrue(goodLast.createNewFile());
// 现网真实形状:清洗结果写完后临时文件被清掉/读不到
File vanished = tempDir.resolve("c.xlsx").toFile();
FolderPackage packaged = newService().packageFolderDedupeResultsAsZip("批次", List.of(
new DedupeArchiveEntry("dir/a.xlsx", "a.xlsx", goodFirst),
new DedupeArchiveEntry("dir/c.xlsx", "c.xlsx", vanished),
new DedupeArchiveEntry("dir/b.xlsx", "b.xlsx", goodLast)));
assertEquals(2, packaged.packedCount(), "坏条目只认赔自己,其余两个必须进包");
assertEquals(List.of("c.xlsx"), packaged.skippedEntries(), "坏条目必须点名");
try (ZipFile zip = new ZipFile(packaged.file())) {
List<String> names = zip.stream().map(ZipEntry::getName).toList();
assertEquals(List.of("dir/a.xlsx", "dir/b.xlsx"), names,
"包里就是其余可读条目,坏项不得留半截文件");
}
}
/** 反向钉:一个都没写进去时不许交空包冒充「打包成功」。 */
@Test
@DisplayName("一个都没写进去时不许交空包冒充成功")
void refusesToShipPackageWhenNothingCouldBePacked(@TempDir Path tempDir) {
when(storageProperties.getLocalTempDir()).thenReturn(tempDir.toString());
List<DedupeArchiveEntry> allMissing = List.of(
new DedupeArchiveEntry("dir/a.xlsx", "a.xlsx", tempDir.resolve("a.xlsx").toFile()),
new DedupeArchiveEntry("dir/b.xlsx", "b.xlsx", tempDir.resolve("b.xlsx").toFile()));
BusinessException thrown = assertThrows(BusinessException.class,
() -> newService().packageFolderDedupeResultsAsZip("批次", allMissing));
assertTrue(thrown.getMessage().contains(String.valueOf(allMissing.size())),
"文案要说清有几个文件没交付,实际=" + thrown.getMessage());
}
/**
* 一颗坏 worker 不许带走整个 ZIP:先交付已清洗完的文件,终态仍如实落失败。
* 改前形状=打包排在 {@code aborted} 判定之后,一次结果行 insert 都不会发生。
*/
@Test
@DisplayName("坏 worker 不许带走整个 ZIP:已清洗完的文件仍交付,终态仍落失败")
void abortedWorkerStillDeliversCleansedFiles(@TempDir Path tempDir) throws Exception {
when(storageProperties.getLocalTempDir()).thenReturn(tempDir.toString());
writeSourceFile(tempDir.resolve("keyA.xlsx").toFile());
writeSourceFile(tempDir.resolve("keyB.xlsx").toFile());
when(dedupeTotalDataService.findExistingComparableValues(any(), any())).thenReturn(Set.of());
when(dedupeTotalDataService.normalizeComparableValueOrBlank(any()))
.thenAnswer(invocation -> {
Object value = invocation.getArgument(0);
return value == null ? "" : String.valueOf(value).trim().toUpperCase();
});
when(ossStorageService.uploadResultFileWithFreshDownloadUrl(any(), eq("DEDUPE")))
.thenReturn(new OssStorageService.UploadedResult("oss://dedupe/批次.zip", "https://dl/批次.zip"));
DedupeRunService service = newService();
ReflectionTestUtils.setField(service, "fileExecutor", secondWorkerDiesOnContact());
DedupeRunRequest request = new DedupeRunRequest();
request.setUserId(1L);
request.setArchiveName("批次");
request.setSelectedColumns(List.of("id", "ASIN", "国家"));
request.setFiles(List.of(sourceFile("keyA", "a.xlsx", "dir/a.xlsx"),
sourceFile("keyB", "b.xlsx", "dir/b.xlsx")));
FileTaskEntity task = new FileTaskEntity();
task.setId(5001L);
task.setTaskNo("DEDUPE-5001");
task.setStatus("RUNNING");
DedupeRunProgressVo progress = newProgress("DEDUPE-5001");
ReflectionTestUtils.invokeMethod(service, "runAsync", "DEDUPE-5001", task, request, progress, 1L);
ArgumentCaptor<FileResultEntity> rows = ArgumentCaptor.forClass(FileResultEntity.class);
verify(fileResultMapper, atLeastOnce()).insert(rows.capture());
assertTrue(rows.getAllValues().stream()
.anyMatch(row -> "application/zip".equals(row.getResultContentType())
&& row.getResultFileUrl() != null),
"已清洗完的 A 必须打进 ZIP 交付出去,不能被坏 worker 一起带走;实际结果行="
+ rows.getAllValues().stream().map(FileResultEntity::getResultContentType).toList());
// 诚实面:部分交付不等于全量成功,终态照旧落失败
assertEquals("FAILED", task.getStatus(), "有 worker 线程级异常结束时不得标 SUCCESS");
assertEquals("failed", progress.getStatus());
assertNotNull(progress.getError(), "缺口必须有一句可看的原因");
}
/** 第 2 个提交的 worker 当场线程级异常(不执行任务),其余同线程串行跑完。 */
private ExecutorService secondWorkerDiesOnContact() {
AtomicInteger submissions = new AtomicInteger();
return new AbstractExecutorService() {
@Override
public void execute(Runnable command) {
command.run();
}
@Override
public Future<?> submit(Runnable task) {
if (submissions.incrementAndGet() == 2) {
CompletableFuture<?> dead = new CompletableFuture<>();
dead.completeExceptionally(new StackOverflowError("模拟 worker 线程级异常"));
return dead;
}
task.run();
return CompletableFuture.completedFuture(null);
}
@Override
public void shutdown() {
// 测试用同线程执行器,没有常驻线程要收
}
@Override
public List<Runnable> shutdownNow() {
return List.of();
}
@Override
public boolean isShutdown() {
return false;
}
@Override
public boolean isTerminated() {
return false;
}
@Override
public boolean awaitTermination(long timeout, TimeUnit unit) {
return true;
}
};
}
private DedupeRunProgressVo newProgress(String runId) {
DedupeRunProgressVo progress = new DedupeRunProgressVo();
progress.setRunId(runId);
progress.setStatus("running");
progress.setTotal(2);
progress.setProcessedCount(0);
progress.setSuccessCount(0);
progress.setFailedCount(0);
progress.setFinished(false);
return progress;
}
private DedupeSourceFileDto sourceFile(String fileKey, String filename, String relativePath) {
DedupeSourceFileDto source = new DedupeSourceFileDto();
source.setFileKey(fileKey);
source.setOriginalFilename(filename);
source.setRelativePath(relativePath);
return source;
}
private void writeSourceFile(File file) throws Exception {
try (XSSFWorkbook workbook = new XSSFWorkbook()) {
Sheet sheet = workbook.createSheet("variant_collection");
Row headerRow = sheet.createRow(0);
for (int i = 0; i < HEADERS.length; i++) {
headerRow.createCell(i).setCellValue(HEADERS[i]);
}
String[] values = {"1", "B0AAAAAA01", "英国", "10.00", "FBM", "品牌A", "在售", "2026-09-01"};
Row row = sheet.createRow(1);
for (int c = 0; c < values.length; c++) {
row.createCell(c).setCellValue(values[c]);
}
try (FileOutputStream fos = new FileOutputStream(file)) {
workbook.write(fos);
}
}
}
}
@@ -0,0 +1,209 @@
package com.nanri.aiimage.modules.publish.service;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.service.DistributedJobLockService;
import com.nanri.aiimage.config.InstanceMetadata;
import com.nanri.aiimage.modules.file.service.LocalFileStorageService;
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
import com.nanri.aiimage.modules.publish.mapper.PublishFileMapper;
import com.nanri.aiimage.modules.publish.mapper.PublishItemMapper;
import com.nanri.aiimage.modules.publish.model.entity.PublishFileEntity;
import com.nanri.aiimage.modules.publish.model.vo.PublishResultVo;
import com.nanri.aiimage.modules.task.mapper.FileResultMapper;
import com.nanri.aiimage.modules.task.mapper.FileTaskMapper;
import com.nanri.aiimage.modules.task.mapper.TaskChunkMapper;
import com.nanri.aiimage.modules.task.mapper.TaskScopeStateMapper;
import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
import com.nanri.aiimage.modules.task.service.TaskHeartbeatPositionService;
import com.nanri.aiimage.modules.task.service.TaskProgressLightAssembler;
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
import com.nanri.aiimage.modules.ziniao.service.ZiniaoShopSwitchService;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.api.io.TempDir;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.mockito.junit.jupiter.MockitoSettings;
import org.mockito.quality.Strictness;
import org.springframework.test.util.ReflectionTestUtils;
import org.springframework.transaction.support.TransactionTemplate;
import java.io.File;
import java.nio.file.Path;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
/**
* 上架结果组装的「部分交付」文案(B49 #7/B50 #4,上一轮复验认定"整批丢弃"表述夸大:
* 逐文件 status 与 VO 都已交付,真问题只剩文案会说谎)。
*
* <p>原形状:任务里有一个文件没处理完时,结果行仍被 {@code publishResultFile(..., false)}
* 洗成「success=1 + error_message=null」,用户端只剩 {@code failedFileCount} 一个数字,
* 文案层面等同于"全部成功"。这里只把文案改准——
* <b>不动状态语义</b>:文件确实交付了,success/下载直链一改就把能用的结果也扣掉;
* 也<b>不给 SUCCESS 的 file_task 写 error_message</b>:{@code TaskRetryEligibility} 与
* {@code SimilarAsinStaleReopenSupport} 都按那段文字做字面量分类,写进去会喂错判据。
* 缺口落在 {@code biz_file_result.error_message}(本仓既有的「部分交付」载体,
* publish 侧只有 {@code toResultVo} 读它,纯展示)。
*/
@ExtendWith(MockitoExtension.class)
@MockitoSettings(strictness = Strictness.LENIENT)
class PublishTaskServicePartialDeliveryNoteTest {
private static final Long TASK_ID = 9001L;
private static final Long RESULT_ID = 9201L;
private static final String FAILED_FILENAME = "店铺B-上架表.xlsx";
@Mock
private PublishWorkbookService workbookService;
@Mock
private PublishFileMapper publishFileMapper;
@Mock
private PublishItemMapper publishItemMapper;
@Mock
private FileTaskMapper fileTaskMapper;
@Mock
private FileResultMapper fileResultMapper;
@Mock
private OssStorageService ossStorageService;
@Mock
private TaskFileJobService taskFileJobService;
private final ObjectMapper objectMapper = new ObjectMapper();
private PublishTaskService newService() {
return new PublishTaskService(
mock(LocalFileStorageService.class), mock(ZiniaoShopSwitchService.class), workbookService,
publishFileMapper, publishItemMapper, fileTaskMapper, fileResultMapper,
mock(TaskChunkMapper.class), mock(TaskScopeStateMapper.class), taskFileJobService,
mock(TaskDistributedLockService.class), mock(TransientPayloadStorageService.class),
ossStorageService, objectMapper, mock(TransactionTemplate.class),
mock(DistributedJobLockService.class, org.mockito.Answers.RETURNS_MOCKS),
mock(InstanceMetadata.class), mock(TaskProgressLightAssembler.class),
mock(TaskHeartbeatPositionService.class));
}
private FileTaskEntity task() {
FileTaskEntity task = new FileTaskEntity();
task.setId(TASK_ID);
task.setModuleType("PUBLISH");
task.setTaskNo("PUBLISH-9001");
task.setSourceFileCount(2);
task.setStatus("RUNNING");
return task;
}
private FileResultEntity resultRow() {
FileResultEntity result = new FileResultEntity();
result.setId(RESULT_ID);
result.setTaskId(TASK_ID);
result.setModuleType("PUBLISH");
result.setSuccess(0);
result.setErrorMessage("上一轮回执:等待组装");
return result;
}
private PublishFileEntity file(long id, String filename, String status) {
PublishFileEntity file = new PublishFileEntity();
file.setId(id);
file.setTaskId(TASK_ID);
file.setSourceFilename(filename);
file.setShopName("店铺A");
file.setStatus(status);
return file;
}
private TaskFileJobEntity job() {
TaskFileJobEntity job = new TaskFileJobEntity();
job.setId(9301L);
job.setTaskId(TASK_ID);
job.setResultId(RESULT_ID);
job.setModuleType("PUBLISH");
job.setStatus("RUNNING");
job.setRetryCount(0);
return job;
}
private void stubAssembly(FileTaskEntity task, FileResultEntity result, File packagedFile,
List<PublishFileEntity> files) {
when(fileTaskMapper.selectById(TASK_ID)).thenReturn(task);
when(fileResultMapper.selectById(RESULT_ID)).thenReturn(result);
when(publishFileMapper.selectList(any())).thenReturn(files);
when(workbookService.packageTaskResult(any(), any(), anyInt(), any()))
.thenReturn(new PublishWorkbookService.PackagedResult(
packagedFile, packagedFile.getName(), "application/zip"));
when(ossStorageService.uploadResultFile(any(), any())).thenReturn("oss://publish/result.zip");
when(ossStorageService.generateFreshDownloadUrl(any())).thenReturn("https://dl/publish-result.zip");
}
/** 有文件没处理完:文案必须说清缺口,且不许动 SUCCESS 语义与 file_task.error_message。 */
@Test
@DisplayName("部分文件未纳入结果:文案点名缺口,状态语义与 file_task 文案列不动")
void partialDeliveryIsStatedHonestlyWithoutTouchingStatus(@TempDir Path tempDir) {
FileTaskEntity task = task();
FileResultEntity result = resultRow();
File packaged = tempDir.resolve("PUBLISH-9001_上架结果.zip").toFile();
stubAssembly(task, result, packaged, List.of(
file(1L, "店铺A-上架表.xlsx", "SUCCESS"),
file(2L, FAILED_FILENAME, "FAILED")));
newService().processResultFileJob(job());
// 状态语义不动:任务仍 SUCCESS、计数照旧、file_task.error_message 留空
assertEquals("SUCCESS", task.getStatus());
assertEquals(1, task.getSuccessFileCount());
assertEquals(1, task.getFailedFileCount());
assertNull(task.getErrorMessage(),
"SUCCESS 的 file_task 不许写 error_message:TaskRetryEligibility/SimilarAsinStaleReopenSupport 按它分类");
// 结果行仍算交付(success=1+对象 key 在),但文案不再说谎
assertEquals(1, result.getSuccess(), "文件确实交付了,不能改成 success=0 把可用结果扣掉");
assertEquals("oss://publish/result.zip", result.getResultFileUrl());
String note = result.getErrorMessage();
assertNotNull(note, "有文件未纳入结果时,交付面必须有一句缺口说明");
assertTrue(note.contains(FAILED_FILENAME), "缺口要点名到具体文件,实际=" + note);
assertTrue(note.contains("未纳入"), "不许把部分项没处理说成全部完成,实际=" + note);
assertTrue(note.contains("不是表格"), "可恢复的处理失败不许说成表格问题,实际=" + note);
// 文案要能到达用户面:PublishResultVo 仍既有直链又有缺口说明(ready 由 success+url 推导,不受影响)
PublishResultVo vo = (PublishResultVo) ReflectionTestUtils.invokeMethod(
newService(), "toResultVo", result, null);
assertNotNull(vo);
assertTrue(Boolean.TRUE.equals(vo.getFileReady()), "写了缺口说明也不许让结果变得不可下载");
assertEquals("https://dl/publish-result.zip", vo.getDownloadUrl());
assertEquals(note, vo.getErrorMessage());
}
/** 反向钉:全部文件都成功时不许凭空造出缺口说明。 */
@Test
@DisplayName("全部文件都成功:不制造缺口文案")
void fullSuccessStaysWithoutAnyGapNote(@TempDir Path tempDir) {
FileTaskEntity task = task();
FileResultEntity result = resultRow();
File packaged = tempDir.resolve("PUBLISH-9001_上架结果.zip").toFile();
stubAssembly(task, result, packaged, List.of(
file(1L, "店铺A-上架表.xlsx", "SUCCESS"),
file(2L, "店铺B-上架表.xlsx", "SUCCESS")));
newService().processResultFileJob(job());
assertEquals("SUCCESS", task.getStatus());
assertEquals(2, task.getSuccessFileCount());
assertEquals(0, task.getFailedFileCount());
assertNull(task.getErrorMessage());
assertNull(result.getErrorMessage(), "一个文件都没落下时不许留任何缺口文案");
}
}
@@ -0,0 +1,164 @@
package com.nanri.aiimage.modules.shopkey.service;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.nanri.aiimage.common.exception.BusinessException;
import com.nanri.aiimage.config.TaskPressureProperties;
import com.nanri.aiimage.modules.shopkey.mapper.QueryAsinMapper;
import com.nanri.aiimage.modules.shopkey.model.entity.QueryAsinEntity;
import com.nanri.aiimage.modules.shopkey.model.vo.QueryAsinImportProgressVo;
import org.apache.poi.ss.usermodel.Row;
import org.apache.poi.ss.usermodel.Sheet;
import org.apache.poi.xssf.usermodel.XSSFWorkbook;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.api.io.TempDir;
import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.data.redis.core.StringRedisTemplate;
import java.io.File;
import java.io.FileOutputStream;
import java.nio.file.Path;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.anyList;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
* ASIN 导入的逐批入库容错(B49 #6)。
*
* <p>原形状:整张表共用一条读行循环,某一批 {@code insertBatch} 抛出就直接冒到
* {@code runImportTask} 的 catch——<b>后面的批次一个都跑不到</b>,已在内存里的余行一起丢,
* 用户看到的是「导入失败」而实际上一行都没进去(或只进去半张表,无从判断)。
* 现口径:一颗坏批只认赔自己那几行,其余批照常入库;同时不许把没落库的行说成「导入完成」。
*
* <p>审计里「报数虚高」那条已被复验否掉(原状下坏批的计数确实不外发),本类不是按那个理由改,
* 而是因为<b>逐批容错之后</b>坏批的计数会随后续进度外发,所以必须按「没落库」回退。
*/
@ExtendWith(MockitoExtension.class)
class QueryAsinServiceImportBatchToleranceTest {
/** 批大小取 2,5 行 ⇒ 三批:坏掉第一批仍能跑到第二、三批。 */
private static final int BATCH_SIZE = 2;
private static final int ROW_COUNT = 5;
private static final Long FALLBACK_GROUP_ID = 7L;
@Mock
private QueryAsinMapper queryAsinMapper;
@Mock
private ShopManageGroupService shopManageGroupService;
@Mock
private TaskPressureProperties taskPressureProperties;
@Mock
private StringRedisTemplate stringRedisTemplate;
private final ObjectMapper objectMapper = new ObjectMapper();
private QueryAsinService newService() {
when(taskPressureProperties.getDbSelectBatchSize()).thenReturn(BATCH_SIZE);
return new QueryAsinService(queryAsinMapper, shopManageGroupService, taskPressureProperties,
stringRedisTemplate, objectMapper);
}
/** 坏批不许带走其余批:其余批照常入库,坏批点名到行,并报数不说谎。 */
@Test
@DisplayName("一批坏:其余批照常入库,坏批点名,缺口如实报错")
void lostBatchDoesNotAbortRemainingBatches(@TempDir Path tempDir) throws Exception {
when(queryAsinMapper.insertBatch(anyList()))
.thenThrow(new RuntimeException("模拟数据库批量插入失败:Deadlock found"))
.thenReturn(2)
.thenReturn(1);
QueryAsinImportProgressVo progress = new QueryAsinImportProgressVo();
BusinessException thrown = assertThrows(BusinessException.class,
() -> newService().processImportFile(writeImportExcel(tempDir), "asin.xlsx",
FALLBACK_GROUP_ID, 1L, true, false, progress));
ArgumentCaptor<List<QueryAsinEntity>> batches = ArgumentCaptor.forClass(List.class);
verify(queryAsinMapper, times(3)).insertBatch(batches.capture());
List<String> attemptedShops = batches.getAllValues().stream()
.flatMap(batch -> batch.stream())
.map(QueryAsinEntity::getShopName)
.toList();
assertEquals(ROW_COUNT, attemptedShops.size(), "三批都必须被真的试过,不能被一颗坏批提前收走");
List<String> persistedShops = batches.getAllValues().subList(1, batches.getAllValues().size()).stream()
.flatMap(batch -> batch.stream())
.map(QueryAsinEntity::getShopName)
.toList();
assertTrue(persistedShops.containsAll(List.of("店铺3", "店铺4", "店铺5")),
"坏批之后的行必须真的入库,实际入库=" + persistedShops);
// 报数只算真落库的行:坏批按"没落库"回退
assertEquals(ROW_COUNT - BATCH_SIZE, progress.getInsertedCount(),
"未落库的行不许算进已导入");
assertEquals(BATCH_SIZE, progress.getSkippedCount(), "坏批的行数要有交代");
// 不许把缺口说成导入完成:状态由调用方按异常落 failed,文案点名坏批与真因
assertTrue(thrown.getMessage().contains("店铺1"), "要点名到没入库的行,实际=" + thrown.getMessage());
assertTrue(thrown.getMessage().contains(String.valueOf(BATCH_SIZE)),
"要说清丢了几行,实际=" + thrown.getMessage());
}
/** 反向钉:全部批次都落库时不许凭空改口报失败。 */
@Test
@DisplayName("全部落库时照常报成功,不得凭空制造缺口")
void fullImportStillHasNoGap(@TempDir Path tempDir) throws Exception {
QueryAsinImportProgressVo progress = new QueryAsinImportProgressVo();
newService().processImportFile(writeImportExcel(tempDir), "asin.xlsx",
FALLBACK_GROUP_ID, 1L, true, false, progress);
verify(queryAsinMapper, times(3)).insertBatch(anyList());
assertEquals(ROW_COUNT, progress.getInsertedCount());
assertEquals(0, progress.getSkippedCount());
}
/** 一行都没落库时不许报「导入完成」——那才是"根本没拿到数据说成数据正常"。 */
@Test
@DisplayName("整表都没入库时如实失败,并说清是数据库不是表格")
void everyBatchLostIsReportedAsNothingImported(@TempDir Path tempDir) throws Exception {
when(queryAsinMapper.insertBatch(anyList()))
.thenThrow(new RuntimeException("模拟数据库连接不可用"));
QueryAsinImportProgressVo progress = new QueryAsinImportProgressVo();
BusinessException thrown = assertThrows(BusinessException.class,
() -> newService().processImportFile(writeImportExcel(tempDir), "asin.xlsx",
FALLBACK_GROUP_ID, 1L, true, false, progress));
verify(queryAsinMapper, times(3)).insertBatch(anyList());
assertEquals(0, progress.getInsertedCount(), "一行都没落库就不许有已导入计数");
assertTrue(thrown.getMessage().contains(String.valueOf(ROW_COUNT)),
"要说清整表都没进去,实际=" + thrown.getMessage());
}
/**
* 真实形状的导入表:表头「店铺名 + DE」,逐行给不同店铺与 ASIN。
* 分组列故意留空,走页面的 fallbackGroupId(与现网「先选分组再传表」一致)。
*/
private File writeImportExcel(Path tempDir) throws Exception {
File file = tempDir.resolve("asin.xlsx").toFile();
try (XSSFWorkbook workbook = new XSSFWorkbook()) {
Sheet sheet = workbook.createSheet("asin");
Row header = sheet.createRow(0);
header.createCell(0).setCellValue("店铺名");
header.createCell(1).setCellValue("DE");
for (int r = 0; r < ROW_COUNT; r++) {
Row row = sheet.createRow(r + 1);
row.createCell(0).setCellValue("店铺" + (r + 1));
row.createCell(1).setCellValue("B0TEST000" + r);
}
try (FileOutputStream fos = new FileOutputStream(file)) {
workbook.write(fos);
}
}
return file;
}
}