huangzd1997
cbe0ec8bb2
task-12: mark done in progress
2026-08-29 16:46:38 +08:00
huangzd1997
4d91146256
task-12: extend coze result buffering beyond poll path
...
- bufferCozeRowsOrMerge: unified DONE result persist entry point; buffers to
transient storage when toggle on, falls back to immediate chunk merge on
failure (result never lost), no-op on empty input
- submit/retry synchronous immediate DONE results now buffer too (previously
always merged immediately); submit path persists a DONE scope state row to
carry the buffer pointer (persistImmediateCozeDoneState)
- poll path drops the batchTotal>1 gate: single-batch tasks also buffer
- CozeCandidate/CozeBatchContext records package-private for test access
2026-08-29 16:46:29 +08:00
huangzd1997
59ee3c8154
progress: task-11 done (completed=11, rounds=11)
2026-08-29 16:17:27 +08:00
huangzd1997
3d085c1605
task-11: Coze 结果合并前按稳定 rowKey HashSet 去重，消除 O(n²) 重复处理
2026-08-29 16:17:12 +08:00
huangzd1997
d64ba3a827
progress: task-10 done (completed=10, rounds=10)
2026-08-29 16:07:49 +08:00
huangzd1997
1ccf4f74ef
task-10: chunk 结果建立 rowKey 批量索引，消除跨 chunk 线性扫描
...
mergeCozeRowsIntoChunk 先 indexRowsByChunkKey 建立 rowKey→chunkKey
索引，assignCozeRowsToChunks 按 O(1) 查找分配行归属，保留原有
命中/fallback/orphan 语义与顺序稳定性，每 chunk 只读一次 payload。
2026-08-29 16:07:45 +08:00
huangzd1997
17a3292c78
progress: task-9 done (completed=9, rounds=9)
2026-08-29 15:52:57 +08:00
huangzd1997
99c0b7dd75
task-9: chunk 查询改为批量 keyset 分页，保持低内存读取
...
loadChunksKeyset 按 id 递增分批拉取（每批 pageSize），最后按
chunkIndex 升序合并，避免超大任务一次 selectList 全量载入 chunk
元数据。surefire fork 堆显式限制 1536m 防止与主 JVM 叠加挤爆内存。
2026-08-29 15:52:46 +08:00
huangzd1997
3b051bbafe
task-8: WorkbookFactory 输入解析改为受控读取，验证超大 Excel 失败提示
...
解析前用 ZipFile 中央目录探测 xlsx(zip) 条目数与解压总字节（不读压缩
内容），超过 maxWorkbookZipEntries(20000)/maxWorkbookUncompressedBytes
(512MB) 立即拒绝并给出可识别提示；非 zip/损坏文件跳过探测交给
WorkbookFactory 兜底转业务异常。0/负值配置回退默认。新增 8 个测试覆盖
正常路径、多文件、幂等、空输入、单行、解压超限、条目超限、损坏文件、
依赖失败恢复。
2026-08-29 15:11:53 +08:00
huangzd1997
68b7d9a4a4
task-7: 限制单文件大小、最大行数和最大字段长度，防止解析任务无界增长
...
新增 maxSourceFileBytes(50MB)/maxParseRows(50000)/maxFieldLength(2000)
三个配置项：文件超限在解析入口拒绝（异常消息可识别）、行数超限拒绝、
超长单元格字段截断。0/负值配置回退默认。新增 8 个测试覆盖正常路径、
多文件、幂等、空文件、单行、边界（恰好=上限通过、超限拒绝）、
非法输入（文件超限/字段截断）、依赖失败恢复。
2026-08-29 15:08:14 +08:00
huangzd1997
db6e8f0ce8
task-6: 分组数据改为索引/范围引用，避免 groups 嵌套复制完整行对象
...
buildParsedGroups 只写 [startIndex, endIndex) 半开区间引用（同 baseId 行在
items 中天然连续），行对象仅存在于 items 一次；resolveAllRows 优先按引用
展开并安全跳过越界/非法区间；响应组仍内嵌预览行兼容前端。旧 payload 内嵌
items 格式继续可读。新增 8 个测试覆盖引用写入、批量、幂等、空输入、单组、
越界 clamp、非法区间、依赖失败恢复。
2026-08-29 15:02:15 +08:00
huangzd1997
16bec1c0fa
task-5: 预览行数量改为配置驱动并做边界校验
...
解析响应预览上限从硬编码常量改为 SimilarAsinProperties.parseResponsePreviewLimit，
0/负值回退默认 100，超过 1000 clamp 到 1000 防止误配导致响应体无界膨胀。
新增 8 个测试覆盖默认路径、自定义值、幂等、空文件、单行、超限 clamp、非法配置回退、依赖失败恢复。
2026-08-29 14:48:01 +08:00
huangzd1997
026aa4e0c3
task-4: 解析接口只返回固定数量预览行，完整行仅保存在后端任务载荷
...
parseAndCreateTask 响应 items/groups 改为 buildResponsePreviewRows/Groups 裁剪
到 PARSE_RESPONSE_PREVIEW_LIMIT(100) 行：大文件(5000行)响应体不再携带全量行，
前端/队列只依赖统计字段与 taskId，Python 通过 parsed-payload 接口拉全量。
groups 预览副本保留 itemCount 与行级字段，字段完整且顺序稳定。
8 个测试覆盖 150/5000/1/100/101 行、空文件、非法输入、存储失败恢复与临时文件清理。
2026-08-29 14:43:08 +08:00
huangzd1997
9f8bdecc9e
task-3: 统一解析载荷全量行恢复逻辑，新旧结构均可恢复
...
新增 SimilarAsinTaskService.resolveAllRows(payload)：优先 items（新规范结构）、
其次 allItems（旧结构）、最后 groups 展开（最旧结构），并收敛 rowCount/
loadAllRowsByBaseId/hydrateParsedPayloadRows/两条 assemble 路径的重复 fallback 链。
旧 payload（含 allItems 或仅 groups）仍可恢复全量行，行数、顺序不丢失。
9 个测试覆盖默认/批量/幂等/空/单行/超限/新旧 JSON 反序列化/组展开顺序。
2026-08-29 14:36:15 +08:00
huangzd1997
8c84ff3394
task-2: 解析载荷改为单一规范行集合，消除 allItems 重复序列化
...
SimilarAsinParsedPayloadDto.allItems 标记 @JsonInclude(NON_EMPTY) 并仅作反序列化
兼容字段（新写入不再输出）；buildParsedPayloadJson 只写 items 一份全量行，
消除每任务双倍行序列化的内存与 payload 体积。读取侧 getAllItems() 消费点保留
以兼容旧载荷（task-3 再统一收敛）。9 个测试覆盖默认/批量/幂等/空/单行/超限/
旧格式吸收/依赖失败恢复。
2026-08-29 14:32:11 +08:00
huangzd1997
2039acdfe6
task-1: 建立 Similar ASIN 性能基线夹具 (1000/5000行、图片开关、chunk与payload采样)
...
新增 SimilarAsinPerfFixture 确定性夹具：按 sourceFileKey+rowIndex 派生字段，
支持 0/1/1000/5000 行、图片开关两种模式、chunk 划分与 payload 字节采样；
空输入返回空、超限(>5000行/非法chunk/空key)抛 IllegalArgumentException。
含 10 个测试覆盖默认路径/批量/幂等/边界/非法输入/依赖失败恢复。
2026-08-29 14:22:07 +08:00
huangzd1997
322905607a
修复店铺任务数据覆盖bug
Build Backend JAR / build (push) Has been cancelled
2026-08-29 00:16:47 +08:00
huangzd1997
79596a970c
采集数据进度条改为按关键词计算：processedRows/totalRows，标签改为关键词进度
Build Backend JAR / build (push) Has been cancelled
2026-08-28 11:43:51 +08:00
huangzd1997
ea0a319c27
品牌页前端工具菜单新增「ASIN变体采集」入口
Build Backend JAR / build (push) Has been cancelled
2026-08-27 05:10:51 -04:00
huangzd1997
d643ae156a
后台用户管理菜单权限目录：加载增加自动重试与失败提示/重新加载，创建编辑失败时自动恢复
Build Backend JAR / build (push) Has been cancelled
2026-08-27 05:09:40 -04:00
huangzd1997
75f23d4568
统一结果文件下载 URL 生成：历史记录/上传回包等漏网点改用 generateFreshDownloadUrl，result/ 前缀对象走 download 域名
Build Backend JAR / build (push) Has been cancelled
2026-08-26 03:58:29 -04:00
huangzd1997
18db20099a
OSS 结果文件下载分流：新增 aiimage.oss.download-endpoint 配置，result/ 前缀对象生成独立下载域名直链（默认未配置时回退原行为）
Build Backend JAR / build (push) Has been cancelled
2026-08-26 13:37:11 +08:00
huangzd1997
7b238522ed
去重导出列固定顺序为 id/ASIN/国家/价格/品牌；QueryAsin 按店铺加载数据并修正前端回退逻辑
Build Backend JAR / build (push) Has been cancelled
2026-08-25 22:38:38 +08:00
huangzd1997
c287fe01a8
品牌去重默认勾选列调整为 id/ASIN/国家/价格/品牌
Build Backend JAR / build (push) Has been cancelled
2026-08-25 22:05:35 +08:00
huangzd1997
7b81b7aa01
品牌去重默认勾选列调整为 ASIN/国家/价格/SKU ID/品牌
Build Backend JAR / build (push) Has been cancelled
2026-08-25 22:00:33 +08:00
huangzd1997
427a8cb79f
修复：JWT 密钥未配置时使用内置默认密钥，与桌面客户端保持一致
2026-08-25 22:00:30 +08:00
huangzd1997
5f519f9cd5
性能优化：N+1查询改批量、串行远程调用并发化、紫鸟索引巡检批量回写
Build Backend JAR / build (push) Has been cancelled
2026-08-25 19:42:43 +08:00
huangzd1997
338493ecaa
安全加固：CORS白名单、JWT密钥自动生成、SSRF防护、路径遍历与错误信息泄露修复
Build Backend JAR / build (push) Has been cancelled
2026-08-25 18:05:28 +08:00
huangzd1997
c8fb93ff29
Redis 改为哨兵模式，双节点接入 A/B 机哨兵集群（mymaster）
Build Backend JAR / build (push) Has been cancelled
2026-08-25 15:33:50 +08:00
huangzd1997
a11f44664f
还原ASIN变体采集入口，仅保留变体分析移除
Build Backend JAR / build (push) Has been cancelled
2026-08-24 23:21:53 +08:00
huangzd1997
9ba7181212
导航变体分析改为ASIN变体采集入口，修复AiAssetDropzone类型
Build Backend JAR / build (push) Has been cancelled
2026-08-24 23:20:54 +08:00
huangzd1997
5acf3c79ed
移除品牌页导航栏的变体分析入口
Build Backend JAR / build (push) Has been cancelled
2026-08-24 22:48:03 +08:00
huangzd1997
205b0b71b4
数据去重默认勾选列增加品牌/SKU ID/发货类型
Build Backend JAR / build (push) Has been cancelled
2026-08-24 20:13:14 +08:00
huangzd1997
bc8870f957
移除
Build Backend JAR / build (push) Has been cancelled
2026-08-24 00:08:13 +08:00
huangzd1997
a659132469
去重总数据国家限定五国并优化筛选布局
...
- 国家仅保留德国/英国/法国/意大利/西班牙，移除美国及US映射
- 保留服务器已有 V90 迁移，新增 V91 添加 country 列及索引
- ASIN列表筛选区改为两行布局：查询条件+按钮在上，日期+导出在下
2026-08-24 00:03:28 +08:00
huangzd1997
8712df2645
后台去重数据新增国家
2026-08-23 22:50:58 +08:00
huangzd1997
5a750be066
Merge branch 'master' of https://git.aishufu.top/super/crawler-plugin
2026-08-23 22:06:11 +08:00
159d52194c
更新任务存储、权限与货源查询流程
Build Backend JAR / build (push) Has been cancelled
2026-08-22 21:23:01 +08:00
huangzd1997
b00ea71c1b
换设备重新绑定
2026-08-22 16:06:05 +08:00
7a7f1dfa21
完善任务存储、权限及货源查询流程
Build Backend JAR / build (push) Has been cancelled
2026-08-14 22:49:15 +08:00
5b1ccad40e
后台更新 服务优化
Build Backend JAR / build (push) Failing after 16m54s
2026-08-11 14:50:21 +08:00
38216a1885
处理后台新需求
Build Backend JAR / build (push) Failing after 16m58s
2026-08-09 20:08:39 +08:00
f7e26ee266
Merge branch 'master' of https://git.aishufu.top/super/crawler-plugin
Build Backend JAR / build (push) Failing after 4s
2026-08-06 01:11:58 +08:00
28e7fce11c
新需求更新 同步更新
2026-08-06 01:11:54 +08:00
123buzhidao56
a821d306ac
上传
Build Backend JAR / build (push) Failing after 1m31s
2026-08-03 09:26:00 +08:00
123buzhidao56
18db4f80bb
Merge branch 'master' of https://git.aishufu.top/super/crawler-plugin
Build Backend JAR / build (push) Failing after 3s
2026-08-03 09:18:15 +08:00
db1f7f2238
Add Gitea Actions workflow
Build Backend JAR / build (push) Failing after 59s
2026-08-03 02:46:56 +08:00
123buzhidao56
4775748447
modified: backend/ali_oss.py
...
modified: backend/config.py
2026-07-31 14:36:08 +08:00
9048bbb7f8
处理数据量binlog问题
2026-07-28 15:49:41 +08:00
7c3c9e53a0
菜单修改优化
2026-07-28 13:40:19 +08:00
huangzd1997
supernijia
123buzhidao56