feat: publish click gap 20261014-20261020

.gitignore

@@ -1,10 +1,12 @@
 .DS_Store
 .env
 .env.*
+credentials.env
 *.db
 *.sqlite
 *.sqlite3
 *.log
+*.tmp
 __pycache__/
 *.py[cod]
 .cache/

README.md

@@ -1,71 +1,45 @@
-# Click Gap 数据看板
+https://ops.roasbloom.site/af-click-gap/
 
-运行目标是 **mac2**。另一个独立的外部 AppsFlyer 任务每天把汇总结果写入 SQLite；本项目不调用该任务、不访问 AppsFlyer、Superset 或其它提取 API，只从固定数据库只读查询并发布静态 JSON。
+# af-click-gap
 
-固定数据库：`/Users/ads-center/Documents/database/af_click_gap.db`。数据库必须预先存在 `click_gap` 表，脚本不建表、不迁移、不写入数据库。表要求包含：
+生成 AppsFlyer 点击差异看板的数据。上游 AppsFlyer 脚本负责采集并写入 SQLite；本项目流水线只读数据库，生成 JSON 并发布到 Gitea。
 
-```text
-pid TEXT NOT NULL, dt TEXT NOT NULL, attr_platform TEXT NOT NULL,
-pkg_name TEXT NOT NULL, shareit_clicks REAL NOT NULL, af_clicks REAL NOT NULL,
-installs_ua REAL NOT NULL, clicks_diff REAL NOT NULL,
-clicks_diff_rate REAL, updated_at TEXT NOT NULL
-```
+## 数据与输出
 
-主键必须为 `(pid, dt, pkg_name)`。脚本只读取 `pid='shareit'`，并校验字段、日期、数值、重复业务键和有限数值。
-
-## 输出协议
-
-- `data/YYYYMMDD.json`：直接 JSON 数组；最近 7 天每天生成一个文件，即使当天无数据也生成 `[]`。
-- `data/index.json`：包含 `schema_version`、`generated_at` 和最近 7 天的 `data/YYYYMMDD.json` 清单。
-- 每行字段为 `dt`（`YYYYMMDD`）、`attr_platform`、`pkg_name`、`shareit_clicks`、`af_clicks`、`installs_ua`、`clicks_diff`、`clicks_diff_rate`；最后一项可以为 `null`。
-
-默认日期边界为运行日往前 6 天至运行日，首尾包含，共 7 天。手工补数使用 `--start-date` 和 `--end-date`，两者必须同时提供且最多 7 天。
-
-## 手工运行
-
-```sh
-cd /Users/shareit./Library/CloudStorage/OneDrive-个人/ubuntu/Gitea/af-click-gap
-python3 -m py_compile scripts/af_click_gap_pipeline.py
-python3 scripts/af_click_gap_pipeline.py --help
-python3 scripts/af_click_gap_pipeline.py --dry-run
-python3 scripts/af_click_gap_pipeline.py --push       # 写 JSON，commit 并 push
-python3 scripts/af_click_gap_pipeline.py --no-push    # 写 JSON 和 commit，不 push
-python3 scripts/af_click_gap_pipeline.py --start-date 2026-10-10 --end-date 2026-10-16 --no-push
-```
-
-`--dry-run` 只读取数据库并在仓库外临时目录生成、校验 JSON，不写 `data` 或 Git。正式运行先原子替换 `data/` 输出目录中的数据文件，再执行 Git；数据库查询或校验失败不会发布文件。Git 安全检查只允许本次生成的 `data/*.json`（含 `data/index.json`），工作区存在其它未提交改动时拒绝继续。push 失败会保留本地 commit，不会 force push。
+- 数据库：`/Users/ads-center/Documents/database/af_click_gap.db`。
+- 默认生成最近 7 天（含当天）的 `data/YYYYMMDD.json`；使用 `--all` 时生成数据库中所有实际有数据日期的文件。
+- `data/index.json` 保存生成时间、版本和 JSON 文件清单。
+- 每个日期文件是 JSON 数组，记录包含 `dt`、`attr_platform`、`pkg_name`、`shareit_clicks`、`af_clicks`、`installs_ua`、`clicks_diff`、`clicks_diff_rate`。
+- SQLite 以只读方式打开；不会建表、迁移或写入数据库。
 
 ## 看板
 
-`index.html` 是无构建依赖的静态页面，读取并校验 `data/index.json` 及其清单，支持日期范围、包名筛选和表头排序。直接打开文件或执行 `python3 -m http.server` 后访问即可。
+`index.html` 是无构建依赖的静态页面，读取 `data/index.json` 和日期 JSON。开始日期、结束日期默认都为昨天，支持日期筛选、`pkg_name` 多选、表头排序、每页 200 行和下载原始数据。
 
-## mac2 launchd
+## mac2 任务
 
-模板文件为 `launchd/com.roasbloom.af-click-gap.plist.template`，必须在 mac2 使用实际 Python 路径和项目路径渲染后安装：
+两个任务按以下顺序执行：先由上游 AppsFlyer 脚本写数据库，再运行本项目流水线读取数据库并生成、推送 JSON：
 
 ```sh
-cd /Users/shareit./Library/CloudStorage/OneDrive-个人/ubuntu/Gitea/af-click-gap
-mkdir -p "$HOME/Library/Logs/af-click-gap" "$HOME/Library/LaunchAgents"
-PY=$(command -v python3)
-sed -e "s#__MAC2_PYTHON3__#$PY#" -e "s#__PROJECT_DIR__#$PWD#g" -e "s#__LOG_DIR__#$HOME/Library/Logs/af-click-gap#g" launchd/com.roasbloom.af-click-gap.plist.template > /tmp/com.roasbloom.af-click-gap.plist
-plutil -lint /tmp/com.roasbloom.af-click-gap.plist
-cp /tmp/com.roasbloom.af-click-gap.plist "$HOME/Library/LaunchAgents/"
-launchctl bootstrap "gui/$(id -u)" "$HOME/Library/LaunchAgents/com.roasbloom.af-click-gap.plist"
+/Users/ads-center/py314env/bin/python /Users/ads-center/ubuntu/appsflyer/af_click_gap.py
+/Users/ads-center/ubuntu/Gitea/af-click-gap/scripts/run_pipeline.sh
 ```
 
-卸载：`launchctl bootout "gui/$(id -u)/com.roasbloom.af-click-gap"` 后删除 LaunchAgents 中的 plist。日志写在 mac2 用户日志目录；数据库、`.env`、密钥、日志和缓存均不会提交。
+流水线脚本从用户目录的 `~/.config/gitea/credentials.env` 读取共享 Gitea 凭证（变量名为 `GITEA_USER` 和 `GITEA_TOKEN`），文件权限必须为 `600`，并要求账号为 `liuji`。凭据只通过临时 `GIT_ASKPASS` 提供给 Git，不写入 remote URL、命令行、日志或仓库。不要把该文件提交到仓库。业务配置仍可使用项目自己的 `.env`。
 
-## Git 创建和推送
-
-远程仓库约定为 Gitea `https://git.ops.roasbloom.site/liuji/af-click-gap.git`。创建前应检查仓库是否已存在，确认不存在后使用 macOS Keychain 中已有的 Git 凭证创建，再在本地配置 `origin` 并推送当前 `main` 分支。认证信息只通过现有 credential helper 使用，不写入本项目或命令输出。
-
-## 验证和安全
+## 手动运行
 
 ```sh
-python3 -m py_compile scripts/af_click_gap_pipeline.py
-python3 scripts/af_click_gap_pipeline.py --help
-plutil -lint launchd/com.roasbloom.af-click-gap.plist.template
-python3 -m json.tool data/index.json
+cd /Users/ads-center/ubuntu/Gitea/af-click-gap
+./scripts/run_pipeline.sh                         # 最近 7 天并推送
+./scripts/run_pipeline.sh --start-date 2026-10-10 --end-date 2026-10-16
+./scripts/run_pipeline.sh --all                   # 全量日期并推送
+python3 scripts/af_click_gap_pipeline.py --dry-run
+python3 scripts/af_click_gap_pipeline.py --all --dry-run
 ```
 
-禁止提交数据库文件、`.env*`、密钥、日志、浏览器状态、缓存或临时文件；`.gitignore` 已覆盖这些类型。
+`--all` 不能与日期范围同时使用；全量运行可能提交大量历史 JSON。`--dry-run` 只检查数据并在临时目录生成 JSON，不修改 `data/` 或 Git。
+
+## 安全
+
+不要提交 `af_click_gap.db`、数据库、`.env`、token、密钥、日志、浏览器状态或缓存。发布前确认工作区没有无关改动；不再使用 launchd 目录或 plist 配置。

data/20261020.json

@@ -0,0 +1 @@
+[]

data/index.json

@@ -1,7 +1,23 @@
 {
   "schema_version": 1,
-  "generated_at": "2026-10-16T15:13:23+08:00",
+  "generated_at": "2026-10-20T18:08:43+08:00",
   "files": [
+    {
+      "date": "20261020",
+      "path": "data/20261020.json"
+    },
+    {
+      "date": "20261019",
+      "path": "data/20261019.json"
+    },
+    {
+      "date": "20261018",
+      "path": "data/20261018.json"
+    },
+    {
+      "date": "20261017",
+      "path": "data/20261017.json"
+    },
     {
       "date": "20261016",
       "path": "data/20261016.json"
@@ -13,22 +29,6 @@
     {
       "date": "20261014",
       "path": "data/20261014.json"
-    },
-    {
-      "date": "20261013",
-      "path": "data/20261013.json"
-    },
-    {
-      "date": "20261012",
-      "path": "data/20261012.json"
-    },
-    {
-      "date": "20261011",
-      "path": "data/20261011.json"
-    },
-    {
-      "date": "20261010",
-      "path": "data/20261010.json"
     }
   ]
 }

index.html

@@ -2,19 +2,57 @@
 <html lang="zh-CN">
 <head>
 <meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>Click Gap 数据看板</title>
-<style>body{font:14px system-ui,sans-serif;margin:2rem;color:#223}label{margin-right:1rem}input,select,button{padding:.35rem}table{border-collapse:collapse;width:100%;margin-top:1rem}th,td{border:1px solid #ccd;padding:.45rem;text-align:right}th{cursor:pointer;background:#eef}th:first-child,td:first-child,th:nth-child(2),td:nth-child(2),th:nth-child(3),td:nth-child(3){text-align:left}.status{padding:.7rem;background:#f5f5f5}.error{color:#a00}.sort{font-size:.8em}</style>
+<link rel="icon" type="image/png" href="icon.png">
+<style>
+body{font:14px system-ui,sans-serif;margin:2rem;color:#223}label{margin-right:1rem}input,button{padding:.35rem}button{cursor:pointer}table{border-collapse:collapse;width:100%;margin-top:1rem}th,td{border:1px solid #ccd;padding:.45rem;text-align:right}th{cursor:pointer;background:#eef}th:first-child,td:first-child,th:nth-child(2),td:nth-child(2),th:nth-child(3),td:nth-child(3){text-align:left}.status{padding:.7rem;background:#f5f5f5}.error{color:#a00}.sort{font-size:.8em}.controls{display:flex;flex-wrap:wrap;gap:.7rem 1rem;align-items:center}.pagination{display:flex;align-items:center;gap:.7rem;margin-top:1rem}.pagination button:disabled{cursor:not-allowed;opacity:.5}.rate-high{color:#c00;font-weight:700}.hint{color:#667;font-size:.9em}.pkg-filter{position:relative;display:inline-block;vertical-align:middle}.pkg-trigger{min-width:18rem;text-align:left;background:#fff;border:1px solid #aaa}.pkg-trigger::after{content:"▾";float:right;margin-left:1rem}.pkg-panel{position:absolute;z-index:2;top:calc(100% + .25rem);left:0;width:19rem;padding:.6rem;background:#fff;border:1px solid #bbc;box-shadow:0 3px 10px #0002}.pkg-panel[hidden]{display:none}.pkg-search{box-sizing:border-box;width:100%;margin-bottom:.45rem}.pkg-options{max-height:12rem;overflow-y:auto;border:1px solid #ddd;padding:.2rem}.pkg-option{display:block;padding:.25rem}.pkg-option input{margin-right:.45rem}.pkg-actions{display:flex;justify-content:space-between;align-items:center;margin-top:.5rem}.pkg-actions button{padding:.25rem .5rem}.pkg-confirm{background:#3578c4;color:#fff;border:1px solid #28609c}
+</style>
 </head>
-<body><h1>Click Gap 数据看板</h1><div><label>开始 <input id="start" type="date"></label><label>结束 <input id="end" type="date"></label><label>pkg_name <select id="pkg"><option value="">全部</option></select></label><button id="apply">查询</button><button id="reset">重置</button></div><p id="status" class="status">加载中…</p><table><thead><tr id="head"></tr></thead><tbody id="body"></tbody></table>
+<body>
+<h1>Click Gap 数据看板</h1>
+<div class="controls">
+<label>开始 <input id="start" type="date"></label><label>结束 <input id="end" type="date"></label>
+<div class="pkg-filter" id="pkgFilter">
+<button id="pkgTrigger" class="pkg-trigger" type="button" aria-haspopup="dialog" aria-expanded="false">App（全部）</button>
+<div id="pkgPanel" class="pkg-panel" role="dialog" aria-label="选择 pkg_name" hidden>
+<input id="pkgSearch" class="pkg-search" type="search" placeholder="搜索..." aria-label="搜索包名">
+<div id="pkgOptions" class="pkg-options"></div>
+<div class="pkg-actions"><span><button id="selectAllPkg" type="button">全选</button><button id="clearPkg" type="button">清空</button></span><button id="confirmPkg" class="pkg-confirm" type="button">确定</button></div>
+</div></div>
+<button id="apply" type="button">查询</button><button id="reset" type="button">重置</button><button id="download" type="button">下载原始数据</button>
+</div>
+<p class="hint">选择包名后点击“确定”应用筛选；搜索只过滤候选项。</p>
+<p id="status" class="status">加载中…</p>
+<table><thead><tr id="head"></tr></thead><tbody id="body"></tbody></table>
+<div class="pagination" aria-label="分页"><button id="previous" type="button">上一页</button><span id="pageInfo">第 0 / 0 页</span><button id="next" type="button">下一页</button></div>
 <script>
 (() => {"use strict";
-const columns=["dt","attr_platform","pkg_name","shareit_clicks","af_clicks","installs_ua","clicks_diff","clicks_diff_rate"];let rows=[], failed=[];let sort={key:"dt",direction:-1};
-const $=id=>document.getElementById(id), status=$('status');
+const columns=["dt","attr_platform","pkg_name","shareit_clicks","af_clicks","installs_ua","clicks_diff","clicks_diff_rate"];
+const pageSize=200,maxConcurrent=6;let rows=[],failed=[],page=1,sort={key:"dt",direction:-1},files=[],loadVersion=0;
+const dataCache=new Map();
+const $=id=>document.getElementById(id),status=$("status");
 function text(value){return value===null||value===undefined?"—":String(value)}
-function validDate(v){return /^\d{4}-\d{2}-\d{2}$/.test(v)&&!Number.isNaN(Date.parse(v))}
-function date8(v){return v.replaceAll('-','')}
-function show(){const start=$('start').value,end=$('end').value,pkg=$('pkg').value;if(!validDate(start)||!validDate(end)||start>end){status.textContent='请输入有效日期范围，且开始日期不得晚于结束日期。';status.className='status error';return}const filtered=rows.filter(r=>r.dt>=date8(start)&&r.dt<=date8(end)&&(!pkg||r.pkg_name===pkg));filtered.sort((a,b)=>{const av=a[sort.key],bv=b[sort.key];if(av===bv)return 0;if(av===null||av===undefined)return 1;if(bv===null||bv===undefined)return -1;const n=typeof av==='number'&&typeof bv==='number'?av-bv:String(av).localeCompare(String(bv));return n*sort.direction});$('body').replaceChildren();filtered.forEach(r=>{const tr=document.createElement('tr');columns.forEach(c=>{const td=document.createElement('td');td.textContent=text(r[c]);tr.append(td)});$('body').append(tr)});status.textContent=(failed.length?`数据不完整：跳过 ${failed.length} 个文件。`:'')+`显示 ${filtered.length} 行。`;status.className='status'+(failed.length?' error':'')}
-function setupHeaders(){const h=$('head');columns.forEach(c=>{const th=document.createElement('th');th.dataset.key=c;th.textContent=c;th.addEventListener('click',()=>{if(sort.key===c)sort.direction*=-1;else{sort.key=c;sort.direction=-1}setupHeaders();show()});h.append(th)});}
-async function load(){setupHeaders();try{const response=await fetch('data/index.json',{cache:'no-store'});if(!response.ok)throw Error('index.json 加载失败');const index=await response.json();if(!Array.isArray(index.files))throw Error('数据清单格式错误');const safe=index.files.filter(f=>/^\d{8}$/.test(f.date)&&f.path===`data/${f.date}.json`);const bad=index.files.length-safe.length;failed=bad?['清单路径']:[];const loaded=await Promise.all(safe.map(async f=>{try{const r=await fetch(f.path,{cache:'no-store'});if(!r.ok)throw Error('HTTP');const data=await r.json();if(!Array.isArray(data)||data.some(x=>!x||typeof x!=='object'||x.dt!==f.date))throw Error('格式');return data}catch(e){failed.push(f.date);return []}}));rows=loaded.flat();const dates=rows.map(r=>r.dt).sort();if(!dates.length){$('start').value=$('end').value='';status.textContent='暂无数据。';return}const end=dates.at(-1), endDate=new Date(`${end.slice(0,4)}-${end.slice(4,6)}-${end.slice(6)}T00:00:00`);endDate.setDate(endDate.getDate()-29);$('end').value=`${end.slice(0,4)}-${end.slice(4,6)}-${end.slice(6)}`;$('start').value=endDate.toISOString().slice(0,10);[...new Set(rows.map(r=>r.pkg_name).filter(Boolean))].sort().forEach(p=>{const o=document.createElement('option');o.value=p;o.textContent=p;$('pkg').append(o)});show()}catch(e){status.textContent=`无法加载数据：${e.message}`;status.className='status error'}}
-$('apply').addEventListener('click',show);$('reset').addEventListener('click',()=>{if(rows.length){$('pkg').value='';const d=rows.map(r=>r.dt).sort().at(-1);$('end').value=`${d.slice(0,4)}-${d.slice(4,6)}-${d.slice(6)}`;const x=new Date($('end').value);x.setDate(x.getDate()-29);$('start').value=x.toISOString().slice(0,10);show()}});load();
+function validDate(value){return /^\d{4}-\d{2}-\d{2}$/.test(value)&&!Number.isNaN(Date.parse(value))}
+function date8(value){return value.replaceAll("-","")}
+function localDate(value){return `${value.getFullYear()}-${String(value.getMonth()+1).padStart(2,"0")}-${String(value.getDate()).padStart(2,"0")}`}
+let appliedPackages=new Set(),draftPackages=new Set();
+function selectedPackages(){return [...appliedPackages]}
+function packageLabel(){return appliedPackages.size?`App（${appliedPackages.size} 个已选）`:"App（全部）"}
+function renderPackageOptions(){const query=$("pkgSearch").value.trim().toLowerCase(),container=$("pkgOptions");container.replaceChildren();[...new Set(rows.map(row=>row.pkg_name).filter(Boolean))].sort().filter(pkg=>pkg.toLowerCase().includes(query)).forEach(pkg=>{const label=document.createElement("label");label.className="pkg-option";const checkbox=document.createElement("input");checkbox.type="checkbox";checkbox.value=pkg;checkbox.checked=draftPackages.has(pkg);checkbox.addEventListener("change",()=>{if(checkbox.checked)draftPackages.add(pkg);else draftPackages.delete(pkg)});label.append(checkbox,pkg);container.append(label)})}
+function openPackages(){draftPackages=new Set(appliedPackages);renderPackageOptions();$("pkgPanel").hidden=false;$("pkgTrigger").setAttribute("aria-expanded","true");$("pkgSearch").focus()}
+function closePackages(){$("pkgPanel").hidden=true;$("pkgTrigger").setAttribute("aria-expanded","false")}
+function filteredRows(){const start=$("start").value,end=$("end").value,packages=selectedPackages();return rows.filter(row=>row.dt>=date8(start)&&row.dt<=date8(end)&&(!packages.length||packages.includes(row.pkg_name)))}
+function rateValue(value){if(typeof value!=="number"||!Number.isFinite(value))return null;return Math.abs(value)<=1?value*100:value}
+function formatRate(value){const rate=rateValue(value);return rate===null?"—":`${rate.toFixed(2)}%`}
+function compare(a,b){const av=a[sort.key],bv=b[sort.key];if(av===bv)return 0;if(av===null||av===undefined)return 1;if(bv===null||bv===undefined)return -1;const an=sort.key==="clicks_diff_rate"?rateValue(av):av,bn=sort.key==="clicks_diff_rate"?rateValue(bv):bv;const result=typeof an==="number"&&typeof bn==="number"?an-bn:String(an).localeCompare(String(bn));return result*sort.direction}
+function show(){const start=$("start").value,end=$("end").value;if(!validDate(start)||!validDate(end)||start>end){status.textContent="请输入有效日期范围，且开始日期不得晚于结束日期。";status.className="status error";$("body").replaceChildren();updatePagination(0);return}const filtered=filteredRows().sort(compare),totalPages=Math.max(1,Math.ceil(filtered.length/pageSize));page=Math.min(page,totalPages);const visible=filtered.slice((page-1)*pageSize,page*pageSize);$("body").replaceChildren();visible.forEach(row=>{const tr=document.createElement("tr");columns.forEach(column=>{const td=document.createElement("td");td.textContent=column==="clicks_diff_rate"?formatRate(row[column]):text(row[column]);if(column==="clicks_diff_rate"&&rateValue(row[column])!==null&&rateValue(row[column])>10)td.className="rate-high";tr.append(td)});$("body").append(tr)});updatePagination(filtered.length);const selected=selectedPackages();status.textContent=(failed.length?`数据不完整：跳过 ${failed.length} 个文件。 `:"")+`显示 ${filtered.length} 行，第 ${page} / ${totalPages} 页（每页 ${pageSize} 行）。`+(selected.length?`已选择 ${selected.length} 个包名。`:"未筛选包名。");status.className="status"+(failed.length?" error":"")}
+function updatePagination(total){const totalPages=Math.max(1,Math.ceil(total/pageSize));$("pageInfo").textContent=`第 ${total?`${page} / ${totalPages}`:"0 / 0"} 页`;$('previous').disabled=!total||page<=1;$('next').disabled=!total||page>=totalPages}
+function setupHeaders(){const header=$("head");header.replaceChildren();columns.forEach(column=>{const th=document.createElement("th");th.dataset.key=column;th.textContent=column;th.addEventListener("click",()=>{if(sort.key===column)sort.direction*=-1;else{sort.key=column;sort.direction=-1}page=1;setupHeaders();show()});header.append(th)})}
+function resetPage(){page=1;show()}
+function setDefaultDates(){const yesterday=new Date();yesterday.setDate(yesterday.getDate()-1);const date=localDate(yesterday);$("start").value=date;$("end").value=date}
+function downloadRaw(){const blob=new Blob([JSON.stringify(rows,null,2)],{type:"application/json;charset=utf-8"}),url=URL.createObjectURL(blob),link=document.createElement("a"),stamp=new Date().toISOString().replaceAll(/[:.]/g,"-");link.href=url;link.download=`af-click-gap-raw-${stamp}.json`;link.click();URL.revokeObjectURL(url)}
+async function fetchDate(file){if(dataCache.has(file.path))return {date:file.date,data:dataCache.get(file.path)};try{const response=await fetch(file.path,{cache:"no-store"});if(!response.ok)throw Error("HTTP");const data=await response.json();if(!Array.isArray(data)||data.some(row=>!row||typeof row!=="object"||row.dt!==file.date))throw Error("格式");dataCache.set(file.path,data);return {date:file.date,data}}catch(error){return {date:file.date,data:null}}}
+async function loadRange(){const version=++loadVersion,start=$("start").value,end=$("end").value;if(!validDate(start)||!validDate(end)||start>end){show();return}const selected=files.filter(file=>file.date>=date8(start)&&file.date<=date8(end));rows=[];failed=[];page=1;status.textContent=selected.length?`正在加载 ${selected.length} 个日期文件…`:`所选日期范围暂无数据文件。`;status.className="status";if(!selected.length){renderPackageOptions();show();return}for(let offset=0;offset<selected.length;offset+=maxConcurrent){const batch=selected.slice(offset,offset+maxConcurrent),loaded=await Promise.all(batch.map(fetchDate));if(version!==loadVersion)return;loaded.forEach(item=>{if(item.data)rows.push(...item.data);else failed.push(item.date)});renderPackageOptions();$("pkgTrigger").textContent=packageLabel();show();if(offset+batch.length<selected.length)status.textContent=`已加载 ${Math.min(offset+batch.length,selected.length)} / ${selected.length} 个日期文件…`}if(failed.length)status.textContent=`数据不完整：跳过 ${failed.length} 个文件。 ${status.textContent}`}
+async function load(){setupHeaders();setDefaultDates();try{const response=await fetch("data/index.json",{cache:"no-store"});if(!response.ok)throw Error("index.json 加载失败");const index=await response.json();if(!Array.isArray(index.files))throw Error("数据清单格式错误");files=index.files.filter(file=>/^\d{8}$/.test(file.date)&&file.path===`data/${file.date}.json`);await loadRange()}catch(error){status.textContent=`无法加载数据：${error.message}`;status.className="status error"}}
+$("pkgTrigger").addEventListener("click",()=>$("pkgPanel").hidden?openPackages():closePackages());$("pkgSearch").addEventListener("input",renderPackageOptions);$("selectAllPkg").addEventListener("click",()=>{[...new Set(rows.map(row=>row.pkg_name).filter(Boolean))].forEach(pkg=>draftPackages.add(pkg));renderPackageOptions()});$("clearPkg").addEventListener("click",()=>{draftPackages.clear();renderPackageOptions()});$("confirmPkg").addEventListener("click",()=>{appliedPackages=new Set(draftPackages);$("pkgTrigger").textContent=packageLabel();closePackages();resetPage()});document.addEventListener("click",event=>{if(!$("pkgFilter").contains(event.target)&&!$("pkgPanel").hidden)closePackages()});document.addEventListener("keydown",event=>{if(event.key==="Escape"&&!$("pkgPanel").hidden)closePackages()});$("apply").addEventListener("click",loadRange);$("start").addEventListener("change",loadRange);$("end").addEventListener("change",loadRange);$("reset").addEventListener("click",()=>{appliedPackages.clear();$("pkgTrigger").textContent=packageLabel();setDefaultDates();loadRange()});$("previous").addEventListener("click",()=>{if(page>1){page--;show()}});$("next").addEventListener("click",()=>{if(page<Math.ceil(filteredRows().length/pageSize)){page++;show()}});$("download").addEventListener("click",downloadRaw);load();
 })();
 </script></body></html>

launchd/com.roasbloom.af-click-gap.plist.template

@@ -1,11 +0,0 @@
-<!-- TEMPLATE_ONLY: render and validate on mac2 before installing. -->
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
-<plist version="1.0"><dict>
-<key>Label</key><string>com.roasbloom.af-click-gap</string>
-<key>ProgramArguments</key><array><string>__MAC2_PYTHON3__</string><string>__PROJECT_DIR__/scripts/af_click_gap_pipeline.py</string><string>--push</string></array>
-<key>WorkingDirectory</key><string>__PROJECT_DIR__</string>
-<key>StartCalendarInterval</key><dict><key>Hour</key><integer>2</integer><key>Minute</key><integer>0</integer></dict>
-<key>StandardOutPath</key><string>__LOG_DIR__/pipeline.stdout.log</string><key>StandardErrorPath</key><string>__LOG_DIR__/pipeline.stderr.log</string>
-<key>RunAtLoad</key><false/>
-</dict></plist>

scripts/af_click_gap_pipeline.py

@@ -103,21 +103,33 @@ def validate_existing_schema(path: str) -> None:
         raise RuntimeError("click_gap 主键必须为 (pid, dt, pkg_name)")
 
 
-def read_rows(path: str, start: str, end: str, pid: str = PID) -> list[dict[str, Any]]:
+def read_rows(path: str, start: str | None, end: str | None, pid: str = PID) -> list[dict[str, Any]]:
     validate_existing_schema(path)
     try:
         with sqlite3.connect(f"file:{path}?mode=ro", uri=True) as conn:
             conn.row_factory = sqlite3.Row
-            result = conn.execute(
+            query = (
                 "SELECT dt, attr_platform, pkg_name, shareit_clicks, af_clicks, installs_ua, clicks_diff, clicks_diff_rate "
-                "FROM click_gap WHERE pid=? AND dt BETWEEN ? AND ? ORDER BY dt DESC, shareit_clicks DESC, pkg_name",
-                (pid, start, end),
+                "FROM click_gap WHERE pid=?"
             )
+            parameters: list[str] = [pid]
+            if start is not None and end is not None:
+                query += " AND dt BETWEEN ? AND ?"
+                parameters.extend((start, end))
+            query += " ORDER BY dt DESC, shareit_clicks DESC, pkg_name"
+            result = conn.execute(query, parameters)
             return [dict(row) for row in result]
     except sqlite3.Error as exc:
         raise RuntimeError(f"SQLite 查询失败: {exc}") from exc
 
 
+def available_dates(rows: list[dict[str, Any]]) -> list[str]:
+    dates = set()
+    for row in rows:
+        dates.add(date8(str(row["dt"])))
+    return sorted(dates, reverse=True)
+
+
 def validate_rows(rows: list[dict[str, Any]], start: str, end: str) -> list[dict[str, Any]]:
     normalized = []
     for row in rows:
@@ -213,6 +225,7 @@ def parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser(description="Read SQLite click-gap data and publish JSON")
     parser.add_argument("--start-date", help="读取开始日期 YYYY-MM-DD")
     parser.add_argument("--end-date", help="读取结束日期 YYYY-MM-DD")
+    parser.add_argument("--all", action="store_true", help="读取数据库中所有实际有数据的日期")
     parser.add_argument("--dry-run", action="store_true", help="只读取、校验并生成临时 JSON，不写文件或 Git")
     group = parser.add_mutually_exclusive_group()
     group.add_argument("--no-push", action="store_true", help="写文件但不 push")
@@ -226,11 +239,19 @@ def main() -> int:
     try:
         if args.dry_run and (args.no_push or args.push):
             raise ValueError("--dry-run 不可与 --push/--no-push 同时使用")
-        start, end = window(args.start_date, args.end_date)
-        rows = validate_rows(read_rows(args.db_path, start, end), start, end)
-        first = dt.datetime.strptime(start, "%Y%m%d").date()
-        last = dt.datetime.strptime(end, "%Y%m%d").date()
-        dates = [(first + dt.timedelta(days=i)).strftime("%Y%m%d") for i in range((last - first).days + 1)]
+        if args.all and (args.start_date or args.end_date):
+            raise ValueError("--all 不可与 --start-date/--end-date 同时使用")
+        if args.all:
+            rows = validate_rows(read_rows(args.db_path, None, None), "00010101", "99991231")
+            dates = available_dates(rows)
+            start = dates[-1] if dates else "无日期"
+            end = dates[0] if dates else "无日期"
+        else:
+            start, end = window(args.start_date, args.end_date)
+            rows = validate_rows(read_rows(args.db_path, start, end), start, end)
+            first = dt.datetime.strptime(start, "%Y%m%d").date()
+            last = dt.datetime.strptime(end, "%Y%m%d").date()
+            dates = [(first + dt.timedelta(days=i)).strftime("%Y%m%d") for i in range((last - first).days + 1)]
         repo = Path(__file__).resolve().parents[1]
         with tempfile.TemporaryDirectory(prefix="af-click-gap-") as temp:
             generated = Path(temp) / "data"

scripts/run_pipeline.sh

@@ -0,0 +1,79 @@
+#!/bin/sh
+# Run the SQLite pipeline with the repository's dedicated Gitea credential.
+set -eu
+
+SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
+PROJECT_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
+PYTHON_BIN=${PYTHON_BIN:-python3}
+CREDENTIALS_FILE=${HOME:?}/.config/gitea/credentials.env
+
+if [ ! -r "$CREDENTIALS_FILE" ]; then
+    printf '%s\n' "Credential file is not readable: $CREDENTIALS_FILE" >&2
+    exit 1
+fi
+if [ "$(stat -f '%Lp' "$CREDENTIALS_FILE")" != 600 ]; then
+    printf '%s\n' "Credential file must have mode 600: $CREDENTIALS_FILE" >&2
+    exit 1
+fi
+
+credential_value() {
+    awk -F= -v wanted="$1" '$1 ~ "^[[:space:]]*" wanted "[[:space:]]*$" { sub(/^[[:space:]]*/, "", $2); sub(/[[:space:]]*$/, "", $2); print $2; exit }' "$CREDENTIALS_FILE"
+}
+
+AGENTSTAGE_USER=$(credential_value GITEA_USER)
+AGENTSTAGE_TOKEN=$(credential_value GITEA_TOKEN)
+missing=''
+[ -n "$AGENTSTAGE_USER" ] || missing="$missing AGENTSTAGE_USER"
+[ -n "$AGENTSTAGE_TOKEN" ] || missing="$missing AGENTSTAGE_TOKEN"
+if [ -n "$missing" ]; then
+    printf 'Missing required .env variable(s):%s\n' "$missing" >&2
+    exit 1
+fi
+if [ "$AGENTSTAGE_USER" != "liuji" ]; then
+    printf '%s\n' "AGENTSTAGE_USER must be liuji" >&2
+    exit 1
+fi
+export AGENTSTAGE_USER AGENTSTAGE_TOKEN
+
+# Override Git only for this process tree: never read osxkeychain and never alter .git/config.
+REMOTE_URL=$(git -C "$PROJECT_ROOT" config --get remote.origin.url || true)
+case "$REMOTE_URL" in
+    https://*)
+        REMOTE_TARGET=${REMOTE_URL#https://}
+        REMOTE_TARGET=${REMOTE_TARGET#*@}
+        REMOTE_URL="https://liuji@$REMOTE_TARGET"
+        ;;
+    *)
+        printf '%s\n' "remote.origin.url must use HTTPS" >&2
+        exit 1
+        ;;
+esac
+export GIT_CONFIG_COUNT=3
+export GIT_CONFIG_KEY_0=credential.helper
+export GIT_CONFIG_VALUE_0=
+export GIT_CONFIG_KEY_1=remote.origin.url
+export GIT_CONFIG_VALUE_1="$REMOTE_URL"
+export GIT_CONFIG_KEY_2=remote.origin.pushurl
+export GIT_CONFIG_VALUE_2="$REMOTE_URL"
+
+umask 077
+ASKPASS=$(mktemp "${TMPDIR:-/tmp}/af-click-gap-askpass.XXXXXX")
+cleanup() {
+    rm -f -- "$ASKPASS"
+}
+trap cleanup EXIT HUP INT TERM
+
+cat >"$ASKPASS" <<'ASKPASS_SCRIPT'
+#!/bin/sh
+case "$1" in
+    *[Uu]sername*) printf '%s\n' "$AGENTSTAGE_USER" ;;
+    *[Pp]assword*) printf '%s\n' "$AGENTSTAGE_TOKEN" ;;
+    *) exit 1 ;;
+esac
+ASKPASS_SCRIPT
+chmod 700 "$ASKPASS"
+
+export GIT_ASKPASS="$ASKPASS"
+export GIT_TERMINAL_PROMPT=0
+"$PYTHON_BIN" "$PROJECT_ROOT/scripts/af_click_gap_pipeline.py" --push "$@"
+exit $?