super/crawler-plugin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Commit Graph

  • a9a72c7bdf progress 37 huangzd1997 2026-08-29 22:36:15 +08:00
  • 95e314cef4 task-37 huangzd1997 2026-08-29 22:36:11 +08:00
  • 6f6d7be344 progress 36 huangzd1997 2026-08-29 22:05:36 +08:00
  • 93ccb27b40 task-36 huangzd1997 2026-08-29 22:05:30 +08:00
  • f2546fc152 progress 35 huangzd1997 2026-08-29 21:28:32 +08:00
  • 2f89f795f6 task-35 huangzd1997 2026-08-29 21:28:28 +08:00
  • add8b057d5 progress 34 huangzd1997 2026-08-29 20:44:25 +08:00
  • 11e19ef4f1 task-34: 将 ownerInstanceId 从 JSON 查询迁移到显式列并补充索引 huangzd1997 2026-08-29 20:44:21 +08:00
  • 790267560a progress 33 huangzd1997 2026-08-29 20:19:17 +08:00
  • 0e4a1c1c35 task-33: 将店铺源文件 key 映射改为确定路径，取消临时目录递归扫描 huangzd1997 2026-08-29 20:18:57 +08:00
  • 76d9558851 progress 32 huangzd1997 2026-08-29 20:11:20 +08:00
  • 2cd5e07248 task-32: 将 task entity 本地缓存替换为有容量和过期回收的实现 huangzd1997 2026-08-29 20:11:16 +08:00
  • 5f7457a07b progress 31 huangzd1997 2026-08-29 20:04:26 +08:00
  • cb4e10c87f task-31: 将任务快照改为轻量进度字段，避免每次写入完整结果 JSON huangzd1997 2026-08-29 20:04:22 +08:00
  • 76a51ac290 progress 30 huangzd1997 2026-08-29 19:52:14 +08:00
  • 199345fd11 task-30: 为国家结果行建立稳定去重键，替换线性重复扫描 huangzd1997 2026-08-29 19:52:08 +08:00
  • 6e3b43d591 progress 29 huangzd1997 2026-08-29 19:36:34 +08:00
  • ec28216984 task-29: 合并 scope 状态查询与更新，减少单 chunk 数据库往返 huangzd1997 2026-08-29 19:36:30 +08:00
  • 24ea02f150 progress 28 huangzd1997 2026-08-29 19:26:03 +08:00
  • f1db31f813 task-28: 以 scope 计数器替代每个 chunk 的 COUNT(*) 完整统计 huangzd1997 2026-08-29 19:25:59 +08:00
  • 7f49ef596e progress 27 huangzd1997 2026-08-29 19:11:17 +08:00
  • 23eddde770 task-27: chunk 接收改为原子插入/幂等 upsert，去掉先查后插 huangzd1997 2026-08-29 19:11:10 +08:00
  • 0092092dbd progress: task-26 done (completed=26) huangzd1997 2026-08-29 18:55:05 +08:00
  • 8ea4d2f502 task-26: 模板 workbook 大行数下的样式、图片和工作表兼容测试 huangzd1997 2026-08-29 18:55:01 +08:00
  • 6594e1f5d7 progress: task-25 done (completed=25) huangzd1997 2026-08-29 18:46:44 +08:00
  • 4e4af7911a task-25: 店铺结果 workbook 的 SXSSF 流式写入路径 huangzd1997 2026-08-29 18:46:40 +08:00
  • f816fed4f5 progress: task-24 done (completed=24) huangzd1997 2026-08-29 18:38:41 +08:00
  • 7eccbc016a task-24: 店铺图片预取增加任务级数量、字节和超时上限 huangzd1997 2026-08-29 18:38:37 +08:00
  • a9cbd18246 progress: task-23 done (completed=23) huangzd1997 2026-08-29 18:34:05 +08:00
  • 70d724382d task-23: 图片嵌入成功后立即释放外部缩略图字节副本 huangzd1997 2026-08-29 18:34:02 +08:00
  • 4a11fd612e progress: task-22 done (completed=22) huangzd1997 2026-08-29 18:29:29 +08:00
  • 6d488515ca task-22: 店铺 Excel 图片缓存替换为有界字节缓存 huangzd1997 2026-08-29 18:29:25 +08:00
  • 064e13c132 progress: task-21 done (completed=21) huangzd1997 2026-08-29 18:24:14 +08:00
  • 81da7a182e task-21: 建立店铺抓取性能基线夹具（1k/5k 行、多国家、图片成功/失败） huangzd1997 2026-08-29 18:24:10 +08:00
  • 3d5fd42bfa progress: task-20 done (completed=20) huangzd1997 2026-08-29 18:18:37 +08:00
  • 29b6353d7e task-20: Similar ASIN 端到端压测、GC 分析与结果文件兼容回归 huangzd1997 2026-08-29 18:18:24 +08:00
  • 9b5873bb84 progress: task-19 done (completed=19) huangzd1997 2026-08-29 18:11:28 +08:00
  • 77ae8b2823 task-19: Coze 请求/响应与 Python 回传日志改采样、截断、DEBUG huangzd1997 2026-08-29 18:11:24 +08:00
  • 07e99a5fa8 progress: task-18 done (completed=18) huangzd1997 2026-08-29 18:02:54 +08:00
  • 92858acb4d task-18: 统一图片 spool 生命周期，超时/取消/异常路径删除临时文件 huangzd1997 2026-08-29 18:02:50 +08:00
  • e1217c9c85 progress: task-17 done (completed=17) huangzd1997 2026-08-29 17:55:06 +08:00
  • 795dd1084c task-17: 解码采样推广全格式、子采样后像素上限、JPEG 质量估算搜索 huangzd1997 2026-08-29 17:55:00 +08:00
  • 423666ef99 progress: task-16 done (completed=16) huangzd1997 2026-08-29 17:37:25 +08:00
  • 0e3dbe450f task-16: 图片预取改短预算 best-effort，超时取消在途并回退 URL huangzd1997 2026-08-29 17:37:11 +08:00
  • f517b7585f progress: task-15 done (completed=15, rounds=15) huangzd1997 2026-08-29 17:24:31 +08:00
  • dc69b002db task-15: async batched last_used_at touch refresh huangzd1997 2026-08-29 17:24:25 +08:00
  • fb10077557 progress: task-14 done (completed=14, rounds=14) huangzd1997 2026-08-29 17:16:47 +08:00
  • d1ea8feb6f task-14: batch read image db cache and touch only actual hits huangzd1997 2026-08-29 17:16:40 +08:00
  • 4030897d49 progress: task-13 done (completed=13, rounds=13) huangzd1997 2026-08-29 17:05:59 +08:00
  • e201b13301 task-13: cap chunk merge row count and payload bytes with orphan fallback huangzd1997 2026-08-29 17:05:49 +08:00
  • cbe0ec8bb2 task-12: mark done in progress huangzd1997 2026-08-29 16:46:38 +08:00
  • 4d91146256 task-12: extend coze result buffering beyond poll path huangzd1997 2026-08-29 16:46:29 +08:00
  • 59ee3c8154 progress: task-11 done (completed=11, rounds=11) huangzd1997 2026-08-29 16:17:27 +08:00
  • 3d085c1605 task-11: Coze 结果合并前按稳定 rowKey HashSet 去重，消除 O(n²) 重复处理 huangzd1997 2026-08-29 16:17:12 +08:00
  • d64ba3a827 progress: task-10 done (completed=10, rounds=10) huangzd1997 2026-08-29 16:07:49 +08:00
  • 1ccf4f74ef task-10: chunk 结果建立 rowKey 批量索引，消除跨 chunk 线性扫描 huangzd1997 2026-08-29 16:07:45 +08:00
  • 17a3292c78 progress: task-9 done (completed=9, rounds=9) huangzd1997 2026-08-29 15:52:57 +08:00
  • 99c0b7dd75 task-9: chunk 查询改为批量 keyset 分页，保持低内存读取 huangzd1997 2026-08-29 15:52:46 +08:00
  • 3b051bbafe task-8: WorkbookFactory 输入解析改为受控读取，验证超大 Excel 失败提示 huangzd1997 2026-08-29 15:11:53 +08:00
  • 68b7d9a4a4 task-7: 限制单文件大小、最大行数和最大字段长度，防止解析任务无界增长 huangzd1997 2026-08-29 15:08:14 +08:00
  • db6e8f0ce8 task-6: 分组数据改为索引/范围引用，避免 groups 嵌套复制完整行对象 huangzd1997 2026-08-29 15:02:15 +08:00
  • 16bec1c0fa task-5: 预览行数量改为配置驱动并做边界校验 huangzd1997 2026-08-29 14:48:01 +08:00
  • 026aa4e0c3 task-4: 解析接口只返回固定数量预览行，完整行仅保存在后端任务载荷 huangzd1997 2026-08-29 14:43:08 +08:00
  • 9f8bdecc9e task-3: 统一解析载荷全量行恢复逻辑，新旧结构均可恢复 huangzd1997 2026-08-29 14:36:15 +08:00
  • 8c84ff3394 task-2: 解析载荷改为单一规范行集合，消除 allItems 重复序列化 huangzd1997 2026-08-29 14:32:11 +08:00
  • 2039acdfe6 task-1: 建立 Similar ASIN 性能基线夹具 (1000/5000行、图片开关、chunk与payload采样) huangzd1997 2026-08-29 14:22:07 +08:00
  • 322905607a 修复店铺任务数据覆盖bug huangzd1997 2026-08-29 00:16:47 +08:00
  • 79596a970c 采集数据进度条改为按关键词计算：processedRows/totalRows，标签改为关键词进度 huangzd1997 2026-08-28 11:43:51 +08:00
  • ea0a319c27 品牌页前端工具菜单新增「ASIN变体采集」入口 huangzd1997 2026-08-27 05:10:51 -04:00
  • d643ae156a 后台用户管理菜单权限目录：加载增加自动重试与失败提示/重新加载，创建编辑失败时自动恢复 huangzd1997 2026-08-27 05:09:40 -04:00
  • 75f23d4568 统一结果文件下载 URL 生成：历史记录/上传回包等漏网点改用 generateFreshDownloadUrl，result/ 前缀对象走 download 域名 huangzd1997 2026-08-26 03:58:29 -04:00
  • 18db20099a OSS 结果文件下载分流：新增 aiimage.oss.download-endpoint 配置，result/ 前缀对象生成独立下载域名直链（默认未配置时回退原行为） huangzd1997 2026-08-26 13:37:11 +08:00
  • 7b238522ed 去重导出列固定顺序为 id/ASIN/国家/价格/品牌；QueryAsin 按店铺加载数据并修正前端回退逻辑 huangzd1997 2026-08-25 22:38:38 +08:00
  • c287fe01a8 品牌去重默认勾选列调整为 id/ASIN/国家/价格/品牌 huangzd1997 2026-08-25 22:05:35 +08:00
  • 7b81b7aa01 品牌去重默认勾选列调整为 ASIN/国家/价格/SKU ID/品牌 huangzd1997 2026-08-25 22:00:33 +08:00
  • 427a8cb79f 修复：JWT 密钥未配置时使用内置默认密钥，与桌面客户端保持一致 huangzd1997 2026-08-25 22:00:30 +08:00
  • 5f519f9cd5 性能优化：N+1查询改批量、串行远程调用并发化、紫鸟索引巡检批量回写 huangzd1997 2026-08-25 19:42:43 +08:00
  • 338493ecaa 安全加固：CORS白名单、JWT密钥自动生成、SSRF防护、路径遍历与错误信息泄露修复 huangzd1997 2026-08-25 18:05:28 +08:00
  • c8fb93ff29 Redis 改为哨兵模式，双节点接入 A/B 机哨兵集群（mymaster） huangzd1997 2026-08-25 15:33:50 +08:00
  • a11f44664f 还原ASIN变体采集入口，仅保留变体分析移除 huangzd1997 2026-08-24 23:21:53 +08:00
  • 9ba7181212 导航变体分析改为ASIN变体采集入口，修复AiAssetDropzone类型 huangzd1997 2026-08-24 23:20:54 +08:00
  • 5acf3c79ed 移除品牌页导航栏的变体分析入口 huangzd1997 2026-08-24 22:48:03 +08:00
  • 205b0b71b4 数据去重默认勾选列增加品牌/SKU ID/发货类型 huangzd1997 2026-08-24 20:13:14 +08:00
  • bc8870f957 移除 huangzd1997 2026-08-24 00:08:13 +08:00
  • a659132469 去重总数据国家限定五国并优化筛选布局 huangzd1997 2026-08-24 00:03:28 +08:00
  • 8712df2645 后台去重数据新增国家 huangzd1997 2026-08-23 22:50:58 +08:00
  • 5a750be066 Merge branch 'master' of https://git.aishufu.top/super/crawler-plugin huangzd1997 2026-08-23 22:06:11 +08:00
  • 159d52194c 更新任务存储、权限与货源查询流程 supernijia 2026-08-22 21:23:01 +08:00
  • b00ea71c1b 换设备重新绑定 huangzd1997 2026-08-22 16:06:05 +08:00
  • 7a7f1dfa21 完善任务存储、权限及货源查询流程 supernijia 2026-08-14 22:49:15 +08:00
  • 5b1ccad40e 后台更新 服务优化 supernijia 2026-08-11 14:50:21 +08:00
  • 38216a1885 处理后台新需求 supernijia 2026-08-09 20:08:39 +08:00
  • f7e26ee266 Merge branch 'master' of https://git.aishufu.top/super/crawler-plugin supernijia 2026-08-06 01:11:58 +08:00
  • 28e7fce11c 新需求更新 同步更新 supernijia 2026-08-06 01:11:54 +08:00
  • a821d306ac 上传 123buzhidao56 2026-08-03 09:26:00 +08:00
  • 18db4f80bb Merge branch 'master' of https://git.aishufu.top/super/crawler-plugin 123buzhidao56 2026-08-03 09:18:15 +08:00
  • db1f7f2238 Add Gitea Actions workflow DevOps 2026-08-03 02:46:56 +08:00
  • 4775748447 modified: backend/ali_oss.py modified: backend/config.py 123buzhidao56 2026-07-31 14:36:08 +08:00
  • 9048bbb7f8 处理数据量binlog问题 supernijia 2026-07-28 15:49:41 +08:00
  • 7c3c9e53a0 菜单修改优化 supernijia 2026-07-28 13:40:19 +08:00