refactor(phase-3): ShopDataCrawl 接入底座,家族层按证据不接——三支接得上、两处接不上写死
七支巨石里第三支接进 AbstractFileTaskPipeline。它接的是**底座**、不是 shop-payload 家族层: 家族模板 submitShopScopedResults 要求「按店名自建索引→逐行→changed 闸门→重算」,本支三条都不成立 (索引自建且带「回传店铺与任务不匹配」这道拒、单店合同 resultRows.size()!=1、尾部无 changed 闸门 而是无条件 persistProgressOrSnapshot+tryFinalizeTask)。照家族层套就是替它改行为。 本支复用底座的六件:newTaskHeader/newScopeRow(带显式时刻那一版——建单时刻一枚复用给头行与每一行, 默认时区逐行各取那一版会把这格改掉)、insertTask、listScopeRows(本地 listTaskRows 与孪生两支逐字同形,删)、 requireUserId(本地 validateUserId 逐字同形,删)、requireTaskId、writeJson(第四份同名件并进来, 捕获账净减:RETHROW 386→385 按该尺自述的「逐条点名改基线」处置,出处与授权逐字写进 CatchSilenceRatchetTest 注释;total 1002、不重抛/只 log/空体三格未涨)。 两处「按证据不做」写进代码而不是只留在注释里: 1)准入步序现网先归属后终态,与底座默认序相反 ⇒ 为它把 admitSubmission 拆成 requireSubmissionTask/rejectTerminalSubmission/touchSubmissionHeartbeat 三格、按本支的序复用; 孪生两支走的默认序一个字没动,它们的 golden 就是这条等值的证明。 2)收尾刷结果列走 LambdaUpdateWrapper 批量条件更新(只刷 success=1 的行、不写 success/errorMessage、 objectKey 传 null)⇒ publishResultFile 仍不适用,本地写法保留。 代价要说在白处:显式构造器是继承的结构性成本,本支 23 枚参数(孪生两支 17/15), ShopDataCrawlTaskService 3196→3215 行——这一格在 readability 棘轮上是**涨**, 与 Phase 3「只许降」那条准出标准相冲,不自行改基线判据、留给用户拍板。 对撞证据:本支两条新 golden(建单 14 例/回传 20 例)+孪生两支四条 golden 迁移前后逐字节相等, git status -- src/test/resources 一字未动;后端全量 mvn -o test 绿。
This commit is contained in:
+108
-88
@@ -10,7 +10,6 @@ import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.nanri.aiimage.common.exception.BusinessCodes;
|
||||
import com.nanri.aiimage.common.exception.BusinessException;
|
||||
import com.nanri.aiimage.common.exception.TaskOwnerMismatchException;
|
||||
import com.nanri.aiimage.common.security.AdminAuthSupport;
|
||||
import com.nanri.aiimage.config.InstanceMetadata;
|
||||
import com.nanri.aiimage.config.TaskPressureProperties;
|
||||
import com.nanri.aiimage.modules.file.service.oss.OssStorageService;
|
||||
@@ -40,6 +39,7 @@ import com.nanri.aiimage.modules.task.model.entity.FileResultEntity;
|
||||
import com.nanri.aiimage.modules.task.model.entity.FileTaskEntity;
|
||||
import com.nanri.aiimage.modules.task.model.entity.TaskFileJobEntity;
|
||||
import com.nanri.aiimage.modules.task.model.entity.TaskScopeStateEntity;
|
||||
import com.nanri.aiimage.modules.task.service.AbstractModuleTaskCacheService;
|
||||
import com.nanri.aiimage.modules.task.service.TaskDistributedLockService;
|
||||
import com.nanri.aiimage.modules.task.service.TaskFileJobService;
|
||||
import com.nanri.aiimage.modules.task.service.TaskFinalizeSupport;
|
||||
@@ -47,9 +47,9 @@ import com.nanri.aiimage.modules.task.service.TaskHeartbeatPositionService;
|
||||
import com.nanri.aiimage.modules.task.service.TaskProgressSnapshotService;
|
||||
import com.nanri.aiimage.modules.task.service.TaskResultItemService;
|
||||
import com.nanri.aiimage.modules.task.service.TransientPayloadStorageService;
|
||||
import com.nanri.aiimage.modules.task.service.pipeline.AbstractFileTaskPipeline;
|
||||
import com.nanri.aiimage.modules.ziniao.service.ZiniaoShopSwitchService;
|
||||
import com.nanri.aiimage.modules.ziniao.model.vo.ZiniaoShopMatchResultVo;
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.springframework.transaction.PlatformTransactionManager;
|
||||
import org.springframework.transaction.TransactionDefinition;
|
||||
@@ -82,9 +82,8 @@ import com.nanri.aiimage.modules.task.model.vo.TaskProgressLightBatchVo;
|
||||
import com.nanri.aiimage.modules.task.service.TaskProgressLightAssembler;
|
||||
|
||||
@Service
|
||||
@RequiredArgsConstructor
|
||||
@Slf4j
|
||||
public class ShopDataCrawlTaskService {
|
||||
public class ShopDataCrawlTaskService extends AbstractFileTaskPipeline<ShopDataCrawlShopPayloadDto> {
|
||||
|
||||
private static final String MODULE_TYPE = "SHOP_DATA_CRAWL";
|
||||
|
||||
@@ -104,20 +103,17 @@ public class ShopDataCrawlTaskService {
|
||||
private static final String INTERRUPTED_MESSAGE = "Python 在该店铺结果提交完成前中断";
|
||||
private static final String PARTIAL_RESULT_MESSAGE = "Python 中断,已保留已回传的部分数据";
|
||||
private static final String RESULT_CHUNK_SCOPE_PREFIX = "result-chunks:";
|
||||
/** 锁竞争日志标签:历史值是小写短名,巡检按这句文本数过,不随 {@code MODULE_TYPE} 的大写枚举形走。 */
|
||||
private static final String LOCK_LOG_TAG = "shop-data-crawl";
|
||||
|
||||
private final FileTaskMapper fileTaskMapper;
|
||||
private final FileResultMapper fileResultMapper;
|
||||
private final ShopDataCrawlResolveService shopDataCrawlResolveService;
|
||||
private final ShopDataCrawlExcelAssemblyService excelAssemblyService;
|
||||
private final ShopDataCrawlTaskCacheService taskCacheService;
|
||||
private final OssStorageService ossStorageService;
|
||||
private final ZiniaoShopSwitchService ziniaoShopSwitchService;
|
||||
private final ObjectMapper objectMapper;
|
||||
private final TaskPressureProperties taskPressureProperties;
|
||||
private final TaskFileJobService taskFileJobService;
|
||||
private final TaskResultItemService taskResultItemService;
|
||||
private final TaskProgressSnapshotService taskProgressSnapshotService;
|
||||
private final TaskDistributedLockService taskDistributedLockService;
|
||||
private final TaskChunkMapper taskChunkMapper;
|
||||
private final TaskScopeStateMapper taskScopeStateMapper;
|
||||
private final TransientPayloadStorageService transientPayloadStorageService;
|
||||
@@ -129,6 +125,69 @@ public class ShopDataCrawlTaskService {
|
||||
private final TaskProgressLightAssembler taskProgressLightAssembler;
|
||||
private final TaskHeartbeatPositionService taskHeartbeatPositionService;
|
||||
|
||||
/** 参数表顺序=原先 {@code @RequiredArgsConstructor} 按字段声明序生成的那一枚(测试夹具按位置 new)。 */
|
||||
public ShopDataCrawlTaskService(FileTaskMapper fileTaskMapper,
|
||||
FileResultMapper fileResultMapper,
|
||||
ShopDataCrawlResolveService shopDataCrawlResolveService,
|
||||
ShopDataCrawlExcelAssemblyService excelAssemblyService,
|
||||
ShopDataCrawlTaskCacheService taskCacheService,
|
||||
OssStorageService ossStorageService,
|
||||
ZiniaoShopSwitchService ziniaoShopSwitchService,
|
||||
ObjectMapper objectMapper,
|
||||
TaskPressureProperties taskPressureProperties,
|
||||
TaskFileJobService taskFileJobService,
|
||||
TaskResultItemService taskResultItemService,
|
||||
TaskProgressSnapshotService taskProgressSnapshotService,
|
||||
TaskDistributedLockService taskDistributedLockService,
|
||||
TaskChunkMapper taskChunkMapper,
|
||||
TaskScopeStateMapper taskScopeStateMapper,
|
||||
TransientPayloadStorageService transientPayloadStorageService,
|
||||
InstanceMetadata instanceMetadata,
|
||||
ShopDataCrawlDailyFileService dailyFileService,
|
||||
ShopDataCrawlItemStoreService shopDataCrawlItemStoreService,
|
||||
DuplicateCheckRefreshPort duplicateCheckRefreshPort,
|
||||
PlatformTransactionManager transactionManager,
|
||||
TaskProgressLightAssembler taskProgressLightAssembler,
|
||||
TaskHeartbeatPositionService taskHeartbeatPositionService) {
|
||||
super(fileTaskMapper, fileResultMapper, taskDistributedLockService, taskFileJobService, ossStorageService);
|
||||
this.shopDataCrawlResolveService = shopDataCrawlResolveService;
|
||||
this.excelAssemblyService = excelAssemblyService;
|
||||
this.taskCacheService = taskCacheService;
|
||||
this.ziniaoShopSwitchService = ziniaoShopSwitchService;
|
||||
this.objectMapper = objectMapper;
|
||||
this.taskPressureProperties = taskPressureProperties;
|
||||
this.taskResultItemService = taskResultItemService;
|
||||
this.taskProgressSnapshotService = taskProgressSnapshotService;
|
||||
this.taskChunkMapper = taskChunkMapper;
|
||||
this.taskScopeStateMapper = taskScopeStateMapper;
|
||||
this.transientPayloadStorageService = transientPayloadStorageService;
|
||||
this.instanceMetadata = instanceMetadata;
|
||||
this.dailyFileService = dailyFileService;
|
||||
this.shopDataCrawlItemStoreService = shopDataCrawlItemStoreService;
|
||||
this.duplicateCheckRefreshPort = duplicateCheckRefreshPort;
|
||||
this.transactionManager = transactionManager;
|
||||
this.taskProgressLightAssembler = taskProgressLightAssembler;
|
||||
this.taskHeartbeatPositionService = taskHeartbeatPositionService;
|
||||
}
|
||||
|
||||
// 管线钩子(Phase 3 的 3B 迁移):形状在底座,模块语义留在本类
|
||||
|
||||
@Override
|
||||
protected String moduleType() {
|
||||
return MODULE_TYPE;
|
||||
}
|
||||
|
||||
/** 本支是缓存优先派({@code TaskFinalizeSupport.loadTaskForExecution}),不在底座里替它选。 */
|
||||
@Override
|
||||
protected FileTaskEntity loadTaskForSubmission(Long taskId) {
|
||||
return loadTaskForExecution(taskId);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected AbstractModuleTaskCacheService<ShopDataCrawlShopPayloadDto> taskCacheService() {
|
||||
return taskCacheService;
|
||||
}
|
||||
|
||||
@Value("${aiimage.shop-data-crawl.stale-timeout-minutes:30}")
|
||||
private long staleTimeoutMinutes;
|
||||
|
||||
@@ -341,7 +400,7 @@ public class ShopDataCrawlTaskService {
|
||||
}
|
||||
|
||||
public ProductRiskDashboardVo dashboard(Long userId) {
|
||||
validateUserId(userId);
|
||||
requireUserId(userId);
|
||||
ProductRiskDashboardVo vo = new ProductRiskDashboardVo();
|
||||
vo.setCandidateCount(shopDataCrawlResolveService.countCandidates(userId));
|
||||
vo.setProcessedTaskCount(countTasks(userId,
|
||||
@@ -361,7 +420,7 @@ public class ShopDataCrawlTaskService {
|
||||
*/
|
||||
public ShopDataCrawlHistoryVo listHistory(Long userId, int page, int limit) {
|
||||
long startedAt = System.nanoTime();
|
||||
validateUserId(userId);
|
||||
requireUserId(userId);
|
||||
if (page < 1) {
|
||||
throw new BusinessException("page 必须大于等于 1");
|
||||
}
|
||||
@@ -493,14 +552,14 @@ public class ShopDataCrawlTaskService {
|
||||
}
|
||||
|
||||
public String resolveResultDownloadUrl(Long resultId, Long userId) {
|
||||
validateUserId(userId);
|
||||
requireUserId(userId);
|
||||
FileResultEntity entity = requireResultEntity(resultId);
|
||||
ensureResultOwner(entity, userId);
|
||||
return resolveDownloadUrl(entity);
|
||||
}
|
||||
|
||||
public String resolveResultDownloadFilename(Long resultId, Long userId) {
|
||||
validateUserId(userId);
|
||||
requireUserId(userId);
|
||||
FileResultEntity entity = requireResultEntity(resultId);
|
||||
ensureResultOwner(entity, userId);
|
||||
return resolveDownloadFilename(entity);
|
||||
@@ -548,7 +607,7 @@ public class ShopDataCrawlTaskService {
|
||||
|
||||
@Transactional
|
||||
public ShopDataCrawlCreateTaskVo createTask(ShopDataCrawlCreateTaskRequest request, String deviceId) {
|
||||
validateUserId(request.getUserId());
|
||||
requireUserId(request.getUserId());
|
||||
if (request.getItems() == null || request.getItems().size() != 1) {
|
||||
throw new BusinessException("items 必须且只能包含一个店铺");
|
||||
}
|
||||
@@ -572,20 +631,11 @@ public class ShopDataCrawlTaskService {
|
||||
}
|
||||
|
||||
LocalDateTime now = LocalDateTime.now();
|
||||
FileTaskEntity task = new FileTaskEntity();
|
||||
task.setTaskNo(MODULE_TYPE + "-" + IdUtil.getSnowflakeNextIdStr());
|
||||
task.setModuleType(MODULE_TYPE);
|
||||
task.setTaskMode("PYTHON_QUEUE");
|
||||
task.setStatus("RUNNING");
|
||||
task.setSourceFileCount(uniqueItems.size());
|
||||
FileTaskEntity task = newTaskHeader("RUNNING", "PYTHON_QUEUE", request.getUserId(), deviceId,
|
||||
uniqueItems.size(), now);
|
||||
task.setSuccessFileCount(0);
|
||||
task.setFailedFileCount(0);
|
||||
task.setCreatedBy("user:" + request.getUserId());
|
||||
task.setUserId(request.getUserId());
|
||||
task.setDeviceId(AdminAuthSupport.normalizeDeviceId(deviceId));
|
||||
task.setCreatedAt(now);
|
||||
task.setUpdatedAt(now);
|
||||
fileTaskMapper.insert(task);
|
||||
insertTask(task);
|
||||
taskCacheService.saveTaskCache(task);
|
||||
taskCacheService.touchTaskHeartbeat(task.getId());
|
||||
|
||||
@@ -595,14 +645,9 @@ public class ShopDataCrawlTaskService {
|
||||
if (blank(normalizedShopName)) {
|
||||
throw new BusinessException("任务数据已失效,请刷新后重试");
|
||||
}
|
||||
FileResultEntity result = new FileResultEntity();
|
||||
result.setTaskId(task.getId());
|
||||
result.setModuleType(MODULE_TYPE);
|
||||
result.setSourceFilename(normalizedShopName);
|
||||
result.setSourceFileUrl(item.getShopId());
|
||||
result.setUserId(request.getUserId());
|
||||
result.setSuccess(RESULT_PENDING);
|
||||
result.setCreatedAt(now);
|
||||
// rowCount 留 null、success 用本支的 RESULT_PENDING=-1、时刻复用头行那一枚 now(golden 的 result.insert 钉着)
|
||||
FileResultEntity result = newScopeRow(task.getId(), request.getUserId(), normalizedShopName,
|
||||
item.getShopId(), null, RESULT_PENDING, now);
|
||||
fileResultMapper.insert(result);
|
||||
snapshots.add(toSnapshotVo(result, item, task.getStatus(), null));
|
||||
}
|
||||
@@ -616,25 +661,21 @@ public class ShopDataCrawlTaskService {
|
||||
}
|
||||
|
||||
public void submitResult(Long taskId, ShopDataCrawlSubmitResultRequest request) {
|
||||
if (taskId == null || taskId <= 0) {
|
||||
throw new BusinessException("taskId 不合法");
|
||||
}
|
||||
requireTaskId(taskId);
|
||||
if (request == null || request.getShops() == null || request.getShops().size() != 1) {
|
||||
throw new BusinessException("shops 必须且只能包含一个店铺结果");
|
||||
}
|
||||
|
||||
try (TaskDistributedLockService.LockHandle ignored = acquireTaskLockOrThrow(taskId)) {
|
||||
FileTaskEntity task = loadTaskForExecution(taskId);
|
||||
if (task == null || !MODULE_TYPE.equals(task.getModuleType())) {
|
||||
throw new BusinessException("任务不存在");
|
||||
}
|
||||
try (TaskDistributedLockService.LockHandle ignored =
|
||||
acquireSubmissionLockOrThrow(taskId, LOCK_LOG_TAG)) {
|
||||
// 准入步序现网是「先归属、后终态」,与底座 admitSubmission 的默认序相反 ⇒ 不整段套它,
|
||||
// 只复用拆开的三格、序一个字不动。对撞证据:golden 的 ownershipCheckedBeforeTerminal(本支得
|
||||
// 40903,默认序得「任务已结束」)与 terminalTaskRejectedWithoutCode(本支终态拒绝不带 40901)。
|
||||
FileTaskEntity task = requireSubmissionTask(taskId);
|
||||
ensureTaskOwnedByCurrentInstance(task, "submit shop data crawl result");
|
||||
if ("SUCCESS".equals(task.getStatus()) || "FAILED".equals(task.getStatus())) {
|
||||
throw new BusinessException("任务已结束,拒绝重复提交");
|
||||
}
|
||||
|
||||
taskCacheService.touchTaskHeartbeat(taskId);
|
||||
List<FileResultEntity> resultRows = listTaskRows(taskId);
|
||||
rejectTerminalSubmission(task.getStatus(), TerminalRejection.ERROR);
|
||||
touchSubmissionHeartbeat(taskId);
|
||||
List<FileResultEntity> resultRows = listScopeRows(taskId);
|
||||
List<ShopDataCrawlResultItemVo> snapshots = buildSnapshotFromDb(task, resultRows);
|
||||
Map<Long, ShopDataCrawlResultItemVo> snapshotByResultId = indexSnapshotByResultId(snapshots);
|
||||
if (resultRows.size() != 1) throw new BusinessException("任务店铺数据异常");
|
||||
@@ -715,7 +756,7 @@ public class ShopDataCrawlTaskService {
|
||||
return true;
|
||||
}
|
||||
|
||||
List<FileResultEntity> rows = listTaskRows(taskId);
|
||||
List<FileResultEntity> rows = listScopeRows(taskId);
|
||||
if (rows.isEmpty()) {
|
||||
return false;
|
||||
}
|
||||
@@ -791,7 +832,7 @@ public class ShopDataCrawlTaskService {
|
||||
}
|
||||
}
|
||||
|
||||
List<FileResultEntity> latestRows = listTaskRows(taskId);
|
||||
List<FileResultEntity> latestRows = listScopeRows(taskId);
|
||||
updateTaskStatusFromRows(task, latestRows);
|
||||
if (latestRows.stream().allMatch(this::isResultFinished)) {
|
||||
if (changed || !snapshotJsonHasCountryRows(task)) {
|
||||
@@ -813,15 +854,15 @@ public class ShopDataCrawlTaskService {
|
||||
* DB 段(含每日文件行锁与三条删除链路)仍在同一个短事务内,远端对象清理在提交后进行。
|
||||
*/
|
||||
public void deleteTask(Long taskId, Long userId) {
|
||||
validateUserId(userId);
|
||||
requireUserId(userId);
|
||||
FileTaskEntity task = loadTaskForExecution(taskId);
|
||||
if (task == null || !MODULE_TYPE.equals(task.getModuleType()) || !userId.equals(task.getUserId())) {
|
||||
throw new BusinessException("任务不存在");
|
||||
}
|
||||
ensureTaskOwnedByCurrentInstance(task, "delete shop data crawl task");
|
||||
try (TaskDistributedLockService.LockHandle ignored = acquireTaskLockOrThrow(taskId)) {
|
||||
try (TaskDistributedLockService.LockHandle ignored = acquireSubmissionLockOrThrow(taskId, LOCK_LOG_TAG)) {
|
||||
executeShortTransaction(() -> {
|
||||
List<FileResultEntity> taskRows = listTaskRows(taskId);
|
||||
List<FileResultEntity> taskRows = listScopeRows(taskId);
|
||||
try (DailyLockSet dailyLocks = acquireDailyLocks(taskRows)) {
|
||||
ensureDailySyncCompletedBeforeDelete(taskRows);
|
||||
Set<Long> removedResultIds = collectResultIds(taskRows);
|
||||
@@ -898,7 +939,7 @@ public class ShopDataCrawlTaskService {
|
||||
* 远端对象清理已由 deleteResultObjectIfUnreferenced 在提交后执行。
|
||||
*/
|
||||
public void deleteHistory(Long resultId, Long userId) {
|
||||
validateUserId(userId);
|
||||
requireUserId(userId);
|
||||
FileResultEntity entity = requireResultEntity(resultId);
|
||||
ensureResultOwner(entity, userId);
|
||||
Long taskId = entity.getTaskId();
|
||||
@@ -910,7 +951,7 @@ public class ShopDataCrawlTaskService {
|
||||
resultId, taskId, task.getStatus());
|
||||
throw new BusinessException(BusinessCodes.TASK_BUSY, "任务仍在处理中,不能删除");
|
||||
}
|
||||
try (TaskDistributedLockService.LockHandle ignored = acquireTaskLockOrThrow(taskId)) {
|
||||
try (TaskDistributedLockService.LockHandle ignored = acquireSubmissionLockOrThrow(taskId, LOCK_LOG_TAG)) {
|
||||
executeShortTransaction(() -> {
|
||||
FileResultEntity latestEntity = fileResultMapper.selectById(resultId);
|
||||
if (latestEntity == null || !MODULE_TYPE.equals(latestEntity.getModuleType()) || !userId.equals(latestEntity.getUserId())) {
|
||||
@@ -1163,7 +1204,7 @@ public class ShopDataCrawlTaskService {
|
||||
if (task == null || !MODULE_TYPE.equals(task.getModuleType())) {
|
||||
return;
|
||||
}
|
||||
List<FileResultEntity> rows = listTaskRows(taskId);
|
||||
List<FileResultEntity> rows = listScopeRows(taskId);
|
||||
if (rows.isEmpty()) {
|
||||
taskFileJobService.deleteTaskJobs(taskId, MODULE_TYPE);
|
||||
taskResultItemService.deleteTaskItems(taskId, MODULE_TYPE);
|
||||
@@ -1191,13 +1232,6 @@ public class ShopDataCrawlTaskService {
|
||||
return count == null ? 0L : count;
|
||||
}
|
||||
|
||||
private List<FileResultEntity> listTaskRows(Long taskId) {
|
||||
return fileResultMapper.selectList(new LambdaQueryWrapper<FileResultEntity>()
|
||||
.eq(FileResultEntity::getTaskId, taskId)
|
||||
.eq(FileResultEntity::getModuleType, MODULE_TYPE)
|
||||
.orderByAsc(FileResultEntity::getId));
|
||||
}
|
||||
|
||||
Map<String, ShopDataCrawlShopPayloadDto> normalizePayloadByShop(
|
||||
List<ShopDataCrawlShopPayloadDto> shops, String expectedShopName) {
|
||||
Map<String, ShopDataCrawlShopPayloadDto> payloadByShop = new LinkedHashMap<>();
|
||||
@@ -1469,7 +1503,7 @@ public class ShopDataCrawlTaskService {
|
||||
if (!hasProcessableChunkData(incoming.getCountryResults())) {
|
||||
throw new BusinessException("店铺数据抓取结果分片内容为空,拒绝接收");
|
||||
}
|
||||
String payloadJson = writeJson(countryResults, "序列化店铺数据抓取结果分片失败");
|
||||
String payloadJson = writeJson(objectMapper, countryResults, "序列化店铺数据抓取结果分片失败");
|
||||
String payloadHash = DigestUtil.sha256Hex(payloadJson);
|
||||
|
||||
String scopeKey = resultChunkScopeKey(shopKey);
|
||||
@@ -1757,14 +1791,6 @@ public class ShopDataCrawlTaskService {
|
||||
return RESULT_CHUNK_SCOPE_PREFIX + trim(shopKey);
|
||||
}
|
||||
|
||||
private String writeJson(Object value, String message) {
|
||||
try {
|
||||
return objectMapper.writeValueAsString(value);
|
||||
} catch (Exception ex) {
|
||||
throw new BusinessException(message);
|
||||
}
|
||||
}
|
||||
|
||||
private String safeMessage(Exception ex) {
|
||||
return ex == null || blank(ex.getMessage()) ? ex == null ? "未知错误" : ex.getClass().getSimpleName() : ex.getMessage();
|
||||
}
|
||||
@@ -1875,7 +1901,7 @@ public class ShopDataCrawlTaskService {
|
||||
* 一条 UPDATE 落库结果文件列。
|
||||
*
|
||||
* <p>原先对每个成功行 {@code updateById}:N 行 = N 次网络往返 + N 次提交(本方法非事务、自动提交),
|
||||
* 且写 binlog 被同步放大;N=500 时白花约 0.5~1s。条件与 listTaskRows 的查询口径一致。
|
||||
* 且写 binlog 被同步放大;N=500 时白花约 0.5~1s。条件与 {@code listScopeRows} 的查询口径一致。
|
||||
*/
|
||||
private void updateResultFileColumns(List<Long> rowIds, String filename, String objectKey,
|
||||
long fileSize, int rowCount) {
|
||||
@@ -1940,7 +1966,7 @@ public class ShopDataCrawlTaskService {
|
||||
throw new BusinessException("任务不存在");
|
||||
}
|
||||
ensureTaskOwnedByCurrentInstance(task, "assemble shop data crawl result");
|
||||
List<FileResultEntity> rows = listTaskRows(job.getTaskId());
|
||||
List<FileResultEntity> rows = listScopeRows(job.getTaskId());
|
||||
List<ShopDataCrawlResultItemVo> snapshots = taskResultItemService.listResultSnapshots(job.getTaskId(), MODULE_TYPE, ShopDataCrawlResultItemVo.class);
|
||||
if (snapshots.isEmpty()) {
|
||||
snapshots = buildSnapshotFromDb(task, rows);
|
||||
@@ -2626,7 +2652,7 @@ public class ShopDataCrawlTaskService {
|
||||
if (task == null || !MODULE_TYPE.equals(task.getModuleType())) {
|
||||
return;
|
||||
}
|
||||
List<FileResultEntity> rows = listTaskRows(taskId);
|
||||
List<FileResultEntity> rows = listScopeRows(taskId);
|
||||
if (rows.isEmpty()) {
|
||||
return;
|
||||
}
|
||||
@@ -2853,10 +2879,7 @@ public class ShopDataCrawlTaskService {
|
||||
return job != null && !"SUCCESS".equals(job.getStatus());
|
||||
}
|
||||
|
||||
private TaskDistributedLockService.LockHandle acquireTaskLockOrThrow(Long taskId) {
|
||||
return TaskFinalizeSupport.acquireTaskLockOrThrow(taskDistributedLockService, MODULE_TYPE, taskId, "shop-data-crawl");
|
||||
}
|
||||
|
||||
/** 非阻塞/带等待的取锁两枚:拿不到就静默跳过(stale 巡检与判死用),语义与「抢不到即拒提交」不同,故不并入底座。 */
|
||||
private TaskDistributedLockService.LockHandle acquireTaskLock(Long taskId) {
|
||||
return TaskFinalizeSupport.acquireTaskLock(taskDistributedLockService, MODULE_TYPE, taskId);
|
||||
}
|
||||
@@ -3045,13 +3068,14 @@ public class ShopDataCrawlTaskService {
|
||||
|
||||
public void handleResultFileJobFailure(TaskFileJobEntity job, String message) {
|
||||
if (job == null || job.getTaskId() == null) return;
|
||||
try (TaskDistributedLockService.LockHandle ignored = acquireTaskLockOrThrow(job.getTaskId())) {
|
||||
try (TaskDistributedLockService.LockHandle ignored =
|
||||
acquireSubmissionLockOrThrow(job.getTaskId(), LOCK_LOG_TAG)) {
|
||||
FileTaskEntity task = fileTaskMapper.selectById(job.getTaskId());
|
||||
if (task == null || !MODULE_TYPE.equals(task.getModuleType())) return;
|
||||
ensureTaskOwnedByCurrentInstance(task, "fail shop data crawl result file job");
|
||||
FileResultEntity row = job.getResultId() == null ? null : fileResultMapper.selectById(job.getResultId());
|
||||
if (row != null && MODULE_TYPE.equals(row.getModuleType())) markResultFailed(row, message);
|
||||
List<FileResultEntity> rows = listTaskRows(job.getTaskId());
|
||||
List<FileResultEntity> rows = listScopeRows(job.getTaskId());
|
||||
List<ShopDataCrawlResultItemVo> snapshots = buildSnapshotFromDb(task, rows);
|
||||
for (ShopDataCrawlResultItemVo snapshot : snapshots) {
|
||||
if (Objects.equals(snapshot.getResultId(), job.getResultId())) applyFailureToSnapshot(snapshot, message);
|
||||
@@ -3111,6 +3135,8 @@ public class ShopDataCrawlTaskService {
|
||||
return (references == null || references == 0L) && dailyReferences == 0L;
|
||||
}
|
||||
|
||||
/** 归属实例校验:底座钩子默认放过,本支是多实例转发派,覆写成现网语义(外部转发链也调它,故保持 public)。 */
|
||||
@Override
|
||||
public void ensureTaskOwnedByCurrentInstance(FileTaskEntity task, String operation) {
|
||||
String owner = ownerFromTask(task);
|
||||
if (blank(owner) || Objects.equals(owner, currentInstanceId())) return;
|
||||
@@ -3162,7 +3188,7 @@ public class ShopDataCrawlTaskService {
|
||||
|
||||
private String buildTaskWorkbookFilename(FileTaskEntity task) {
|
||||
Long taskId = task == null ? null : task.getId();
|
||||
List<FileResultEntity> rows = taskId == null ? List.of() : listTaskRows(taskId);
|
||||
List<FileResultEntity> rows = taskId == null ? List.of() : listScopeRows(taskId);
|
||||
String shopName = rows.isEmpty() ? "result" : rows.get(0).getSourceFilename();
|
||||
return safeFileStem("店铺数据抓取-" + shopName + "-" + (taskId == null ? "result" : taskId)) + ".xlsx";
|
||||
}
|
||||
@@ -3185,11 +3211,5 @@ public class ShopDataCrawlTaskService {
|
||||
return ShopDataCrawlRowNormalizer.blankToNull(value);
|
||||
}
|
||||
|
||||
private void validateUserId(Long userId) {
|
||||
if (userId == null || userId <= 0) {
|
||||
throw new BusinessException("user_id 不合法");
|
||||
}
|
||||
}
|
||||
|
||||
private record ResultChunkReceipt(String scopeHash, int chunkTotal, boolean completed) {}
|
||||
}
|
||||
|
||||
+52
-19
@@ -48,8 +48,9 @@ import java.util.List;
|
||||
* {@code writeJson(Object, String)} 就是带文案参数的形状。
|
||||
* ⚠ task 模块内部那三份逐字同名件({@code TaskProgressSnapshotService:186}/
|
||||
* {@code TaskResultItemService:405}/{@code TaskResultPayloadService:113})<b>本轮仍未归并</b>:
|
||||
* 归并会把 RETHROW 那一格跌破下限(它是下限不是上限,实测首迁后正好落在 386),
|
||||
* 那属于「调整门禁判据」动作,与 Phase 0 那格「三格守卫」同批做,不在迁移笔里顺手干。</li>
|
||||
* 归并会把 RETHROW 那一格跌破下限(它是下限不是上限;首迁后落在 386,3B 并掉 ShopDataCrawl
|
||||
* 那份私有同名件后按「逐条点名改基线」调到 385,出处写在 {@code CatchSilenceRatchetTest}),
|
||||
* 再往下降就属于「调整门禁判据」动作,与 Phase 0 那格「三格守卫」同批做,不在迁移笔里顺手干。</li>
|
||||
* <li><b>跑批</b>:服务端<b>没有属于这七支的可沉件</b>——机制早已收在
|
||||
* {@link TaskFileJobService}(enqueue/claim/markSuccess/markFailed)+
|
||||
* {@code TaskResultFileJobWorker}(取锁、重排、重试)+
|
||||
@@ -357,9 +358,11 @@ public abstract class AbstractFileTaskPipeline<T> {
|
||||
*
|
||||
* <p><b>步序按 ShopMatch/PriceTrack 定:终态判定在归属校验之前。</b>ShopDataCrawl 现网是反的
|
||||
* ({@code :631} 先查归属、{@code :632} 再判终态),一支既被别人持有又已终态的任务,
|
||||
* 走它会得 {@code TaskOwnerMismatchException}、走本方法是终态拒绝。所以 ShopDataCrawl 迁移时
|
||||
* 要么保留自己的步序(传 {@code null} 后自查归属),要么把这条差异单独立账,
|
||||
* 不许悄悄套用本默认序就算迁完。
|
||||
* 走它会得 {@code TaskOwnerMismatchException}、走本方法是终态拒绝。3B 迁移那一支的处置是
|
||||
* <b>保留本地步序</b>:它按自己的序复用 {@link #requireSubmissionTask}/
|
||||
* {@link #rejectTerminalSubmission}/{@link #touchSubmissionHeartbeat} 三格,
|
||||
* 不整段套本方法——本方法的默认序因此一个字也没替它改,这条差异也不再只是记在注释里的口头账
|
||||
* (golden 的 {@code ownershipCheckedBeforeTerminal} 那例钉着)。
|
||||
*
|
||||
* <p><b>本方法不打日志</b>:七支里只有 PriceTrack{@code :602} 在终态拒绝前打一行 warn,
|
||||
* 其余六支沉默。日志不是公共形状,所以那一行留在它自己的调用点,别当成已沉掉后把它删了。
|
||||
@@ -368,30 +371,60 @@ public abstract class AbstractFileTaskPipeline<T> {
|
||||
TerminalRejection terminalRejection,
|
||||
String ownershipOperation) {
|
||||
requireTaskId(taskId);
|
||||
FileTaskEntity task = loadTaskForSubmission(taskId);
|
||||
if (task == null || !moduleType().equals(task.getModuleType())) {
|
||||
throw new BusinessException("任务不存在");
|
||||
}
|
||||
String status = task.getStatus();
|
||||
if ("SUCCESS".equals(status) || "FAILED".equals(status)) {
|
||||
if (terminalRejection == TerminalRejection.IDEMPOTENT) {
|
||||
throw new BusinessException(BusinessCodes.TASK_ALREADY_FINISHED, "任务已结束,拒绝重复提交");
|
||||
}
|
||||
throw new BusinessException("任务已结束,拒绝重复提交");
|
||||
}
|
||||
FileTaskEntity task = requireSubmissionTask(taskId);
|
||||
rejectTerminalSubmission(task.getStatus(), terminalRejection);
|
||||
if (ownershipOperation != null) {
|
||||
ensureTaskOwnedByCurrentInstance(task, ownershipOperation);
|
||||
}
|
||||
AbstractModuleTaskCacheService<T> cacheService = taskCacheService();
|
||||
if (cacheService != null) {
|
||||
cacheService.touchTaskHeartbeat(taskId);
|
||||
touchSubmissionHeartbeat(taskId);
|
||||
return task;
|
||||
}
|
||||
|
||||
/**
|
||||
* 准入的前两行:加载+模块校验,不符即「任务不存在」。
|
||||
* 出处:ShopMatch{@code :647-650}/PriceTrack{@code :597-600}/ShopDataCrawl{@code :627-630}(三支逐字同形)。
|
||||
*
|
||||
* <p><b>为什么把它从 {@link #admitSubmission} 里拆出来。</b>3B 实测缺件:ShopDataCrawl 的准入
|
||||
* 步序与本类默认序相反(先归属后终态),整段套 {@code admitSubmission} 就会改掉那格的可见结果
|
||||
* (见方法注释里立着的账)。拆开后它按自己的序复用这三格,而 ShopMatch/PriceTrack 走的
|
||||
* 默认序一个字没动——四格合起来与拆前逐句等值,两支的 golden 就是这条等值的证明。
|
||||
*/
|
||||
protected final FileTaskEntity requireSubmissionTask(Long taskId) {
|
||||
FileTaskEntity task = loadTaskForSubmission(taskId);
|
||||
if (task == null || !moduleType().equals(task.getModuleType())) {
|
||||
throw new BusinessException("任务不存在");
|
||||
}
|
||||
return task;
|
||||
}
|
||||
|
||||
/**
|
||||
* 终态那格:怎么拒由 {@code terminalRejection} 决定(两种线协议,见 {@link TerminalRejection})。
|
||||
* 与 {@link #requireSubmissionTask} 同一笔拆开的理由——步序不同的支要能单独复用这一格。
|
||||
*/
|
||||
protected final void rejectTerminalSubmission(String status, TerminalRejection terminalRejection) {
|
||||
if (!"SUCCESS".equals(status) && !"FAILED".equals(status)) {
|
||||
return;
|
||||
}
|
||||
if (terminalRejection == TerminalRejection.IDEMPOTENT) {
|
||||
throw new BusinessException(BusinessCodes.TASK_ALREADY_FINISHED, "任务已结束,拒绝重复提交");
|
||||
}
|
||||
throw new BusinessException("任务已结束,拒绝重复提交");
|
||||
}
|
||||
|
||||
/** 有模块缓存服务的支按现网形状打一次心跳;没有的支什么都不做(见 {@link #taskCacheService()})。 */
|
||||
protected final void touchSubmissionHeartbeat(Long taskId) {
|
||||
AbstractModuleTaskCacheService<T> cacheService = taskCacheService();
|
||||
if (cacheService != null) {
|
||||
cacheService.touchTaskHeartbeat(taskId);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 该任务的全部结果行,按主键升序——回传侧的作用域匹配与收尾判定都依赖这个顺序。
|
||||
* 出处:ShopMatch{@code :656-659}/PriceTrack{@code :606-610}/DeleteBrand 同款 selectList。
|
||||
* 3B 又并进来一支:ShopDataCrawl 本地那枚 {@code listTaskRows}(原 {@code :1194-1199})与这两家
|
||||
* 逐字同形、已删 ⇒ 真正调本件的支数现在是三(ShopMatch/PriceTrack/ShopDataCrawl),
|
||||
* 其余四支尚未迁移,别说成「七支都用它」。
|
||||
*/
|
||||
protected final List<FileResultEntity> listScopeRows(Long taskId) {
|
||||
return fileResultMapper.selectList(new LambdaQueryWrapper<FileResultEntity>()
|
||||
|
||||
Reference in New Issue
Block a user