新版可用

This commit is contained in:
2025-11-27 18:40:08 +08:00
parent 46de43ce72
commit 4fef65bd93
26 changed files with 39568 additions and 504 deletions

View File

@@ -1,73 +0,0 @@
# 微信公众号文章爬取工具Go版本
这是一个基于Go语言开发的微信公众号文章爬取工具可以自动获取指定公众号的所有文章列表和详细内容。
## 功能特性
- 获取公众号所有文章列表
- 获取每篇文章的详细内容
- 获取文章的阅读量、点赞数、转发数等统计信息
- 支持获取文章评论
- 自动保存文章列表和详细内容
## 环境要求
- Go 1.20 或更高版本
- Windows 操作系统脚本已针对Windows优化
## 安装使用
### 1. 配置Cookie
-`cookie.txt.example` 重命名为 `cookie.txt`
- 按照文件中的说明获取微信公众平台的Cookie
- 将Cookie信息粘贴到 `cookie.txt` 文件中
### 2. 运行程序
直接双击 `run.bat` 脚本文件,程序会自动:
- 下载所需依赖
- 编译Go程序
- 运行爬取工具
## 项目结构
```
backend/
├── cmd/
│ └── main.go # 主程序入口
├── configs/
│ └── config.go # 配置管理
├── pkg/
│ ├── utils/ # 工具函数
│ │ └── utils.go
│ └── wechat/ # 微信相关功能实现
│ └── access_articles.go
├── data/ # 数据存储目录
├── cookie.txt # Cookie文件需要手动创建
├── go.mod # Go模块定义
├── run.bat # Windows启动脚本
└── README.md # 使用说明
```
## 注意事项
1. 使用本工具前,请确保您已获得相关公众号的访问权限
2. 请遵守相关法律法规,合理使用本工具
3. 频繁请求可能会触发微信的反爬虫机制,请控制爬取频率
4. 由于微信接口可能会变化,工具可能需要相应调整
## 常见问题
### Q: 获取Cookie失败怎么办
A: 请确保您已登录微信公众平台并且在开发者工具中正确复制了完整的Cookie信息。
### Q: 爬取过程中出现网络错误怎么办?
A: 工具会自动处理简单的网络错误,请确保网络连接正常。如果持续失败,可能是微信接口发生了变化。
### Q: 如何修改爬取的公众号?
A: 工具会自动从Cookie中获取当前登录用户可访问的公众号信息。如果需要爬取不同的公众号请在微信公众平台中切换账号后重新获取Cookie。
## 许可证
本项目仅供学习和研究使用。

View File

@@ -0,0 +1,460 @@
# 📡 微信公众号文章爬虫 - API 接口文档
## 服务器信息
- **服务地址**: http://localhost:8080
- **协议**: HTTP/1.1
- **数据格式**: JSON
- **字符编码**: UTF-8
- **CORS**: 已启用(允许所有来源)
## 统一响应格式
所有API接口返回格式统一为
```json
{
"success": true, // 请求是否成功
"message": "操作成功", // 提示信息
"data": {} // 数据内容(可选)
}
```
## 接口列表
### 1. 提取公众号主页
**接口地址**: `/api/homepage/extract`
**请求方法**: POST
**功能说明**: 从文章链接中提取公众号主页链接
#### 请求参数
```json
{
"url": "https://mp.weixin.qq.com/s?__biz=xxx&mid=xxx"
}
```
| 参数 | 类型 | 必填 | 说明 |
|------|------|------|------|
| url | string | 是 | 公众号文章链接 |
#### 响应示例
**成功响应**:
```json
{
"success": true,
"message": "提取成功",
"data": {
"homepage": "https://mp.weixin.qq.com/mp/profile_ext?action=home&__biz=xxx&scene=124",
"output": "完整的命令行输出信息"
}
}
```
**失败响应**:
```json
{
"success": false,
"message": "未能提取到主页链接"
}
```
#### 调用示例
**jQuery**:
```javascript
$.ajax({
url: 'http://localhost:8080/api/homepage/extract',
method: 'POST',
contentType: 'application/json',
data: JSON.stringify({
url: 'https://mp.weixin.qq.com/s?__biz=xxx&mid=xxx'
}),
success: function(response) {
if (response.success) {
console.log('主页链接:', response.data.homepage);
}
}
});
```
**curl**:
```bash
curl -X POST http://localhost:8080/api/homepage/extract \
-H "Content-Type: application/json" \
-d '{"url":"https://mp.weixin.qq.com/s?__biz=xxx&mid=xxx"}'
```
---
### 2. 下载单篇文章
**接口地址**: `/api/article/download`
**请求方法**: POST
**功能说明**: 下载指定的单篇文章
#### 请求参数
```json
{
"url": "https://mp.weixin.qq.com/s?__biz=xxx",
"save_image": true,
"save_content": true
}
```
| 参数 | 类型 | 必填 | 说明 |
|------|------|------|------|
| url | string | 是 | 文章链接 |
| save_image | boolean | 否 | 是否保存图片默认false |
| save_content | boolean | 否 | 是否保存内容默认true |
#### 响应示例
```json
{
"success": true,
"message": "下载任务已启动",
"data": {
"url": "https://mp.weixin.qq.com/s?__biz=xxx"
}
}
```
---
### 3. 获取文章列表
**接口地址**: `/api/article/list`
**请求方法**: POST
**功能说明**: 批量获取公众号的文章列表
#### 请求参数
```json
{
"access_token": "https://mp.weixin.qq.com/mp/profile_ext?action=xxx&appmsg_token=xxx",
"pages": 0
}
```
| 参数 | 类型 | 必填 | 说明 |
|------|------|------|------|
| access_token | string | 是 | 包含appmsg_token的URL |
| pages | integer | 否 | 获取页数0表示全部默认0 |
#### 响应示例
```json
{
"success": true,
"message": "任务已启动"
}
```
---
### 4. 批量下载文章
**接口地址**: `/api/article/batch`
**请求方法**: POST
**功能说明**: 批量下载公众号的所有文章
#### 请求参数
```json
{
"official_account": "公众号名称或文章链接",
"save_image": true,
"save_content": true
}
```
| 参数 | 类型 | 必填 | 说明 |
|------|------|------|------|
| official_account | string | 是 | 公众号名称或任意文章链接 |
| save_image | boolean | 否 | 是否保存图片默认false |
| save_content | boolean | 否 | 是否保存内容默认true |
#### 响应示例
```json
{
"success": true,
"message": "任务已启动"
}
```
---
### 5. 获取数据列表
**接口地址**: `/api/data/list`
**请求方法**: GET
**功能说明**: 获取已下载的公众号数据列表
#### 请求参数
#### 响应示例
```json
{
"success": true,
"data": [
{
"name": "研招网资讯",
"article_count": 125,
"path": "D:\\workspace\\Access_wechat_article\\backend\\data\\研招网资讯",
"last_update": "2025-11-27"
}
]
}
```
| 字段 | 类型 | 说明 |
|------|------|------|
| name | string | 公众号名称 |
| article_count | integer | 文章数量 |
| path | string | 存储路径 |
| last_update | string | 最后更新时间 |
#### 调用示例
**jQuery**:
```javascript
$.get('http://localhost:8080/api/data/list', function(response) {
if (response.success) {
console.log('数据列表:', response.data);
}
});
```
**curl**:
```bash
curl http://localhost:8080/api/data/list
```
---
### 6. 获取任务状态
**接口地址**: `/api/task/status`
**请求方法**: GET
**功能说明**: 获取当前任务的执行状态
#### 请求参数
#### 响应示例
**任务运行中**:
```json
{
"success": true,
"data": {
"running": true,
"progress": 45,
"message": "正在下载第10篇文章..."
}
}
```
**无任务运行**:
```json
{
"success": true,
"data": {
"running": false,
"progress": 0,
"message": ""
}
}
```
| 字段 | 类型 | 说明 |
|------|------|------|
| running | boolean | 是否有任务运行中 |
| progress | integer | 任务进度0-100 |
| message | string | 任务状态描述 |
| error | string | 错误信息(可选) |
---
## 错误码说明
### HTTP状态码
| 状态码 | 说明 |
|--------|------|
| 200 | 请求成功 |
| 400 | 请求参数错误 |
| 500 | 服务器内部错误 |
### 业务错误码
所有业务错误通过响应中的 `success` 字段和 `message` 字段返回:
```json
{
"success": false,
"message": "具体的错误信息"
}
```
常见错误信息:
| 错误信息 | 说明 | 解决方法 |
|----------|------|----------|
| 请求参数错误 | JSON格式不正确或缺少必填参数 | 检查请求参数格式 |
| 执行失败 | 后端程序执行出错 | 查看详细错误信息 |
| 未能提取到主页链接 | 文章链接格式错误或解析失败 | 使用有效的文章链接 |
| 读取数据目录失败 | data目录不存在或无权限 | 检查目录权限 |
---
## 开发指南
### 本地测试
1. **启动API服务器**:
```bash
cd backend\api
start_api.bat
```
2. **测试接口**:
```bash
# 测试提取主页
curl -X POST http://localhost:8080/api/homepage/extract \
-H "Content-Type: application/json" \
-d "{\"url\":\"文章链接\"}"
# 测试获取数据列表
curl http://localhost:8080/api/data/list
```
### 跨域配置
API服务器已启用CORS允许所有来源访问
```go
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Header().Set("Access-Control-Allow-Methods", "GET, POST, OPTIONS")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
```
如需限制特定域名,修改 `server.go` 中的 `corsMiddleware` 函数。
### 超时设置
默认HTTP超时时间30秒
如需修改,在 `server.go` 中添加:
```go
server := &http.Server{
Addr: ":8080",
ReadTimeout: 30 * time.Second,
WriteTimeout: 30 * time.Second,
}
```
### 日志记录
API服务器使用标准输出记录日志
```go
log.Printf("[%s] %s - %s", r.Method, r.URL.Path, message)
```
---
## 接口更新计划
### v1.1.0(计划中)
- [ ] 添加用户认证机制
- [ ] 支持任务队列管理
- [ ] 增加下载进度推送WebSocket
- [ ] 提供文章搜索接口
### v1.2.0(计划中)
- [ ] 数据统计分析接口
- [ ] 导出功能PDF/Word
- [ ] 批量任务管理
- [ ] 定时任务支持
---
## 技术栈
- **语言**: Go 1.20+
- **Web框架**: net/http (标准库)
- **数据格式**: JSON
- **并发模型**: Goroutine
---
## 性能说明
### 并发能力
- 支持多客户端同时访问
- 但同一时间只能执行一个爬虫任务(`currentTask`
### 资源占用
- CPU: 低主要I/O操作
- 内存: <50MB
- 磁盘: 取决于下载的文章数量
### 性能优化建议
1. 使用连接池管理HTTP请求
2. 实现任务队列机制
3. 添加结果缓存
4. 启用gzip压缩
---
## 安全建议
### 1. 生产环境部署
- 添加HTTPS支持
- 实现API认证JWT/OAuth
- 限制跨域来源
- 添加请求频率限制
### 2. 数据安全
- 不要暴露敏感信息Cookie
- 定期清理临时文件
- 备份重要数据
### 3. 访问控制
- 添加IP白名单
- 实现用户权限管理
- 记录操作日志
---
## 常见问题
### Q1: 为什么任务启动后没有响应?
A: 检查后端 `wechat-crawler.exe` 是否存在并有执行权限
### Q2: 如何查看详细的错误信息?
A: 查看API服务器窗口的控制台输出
### Q3: 能同时执行多个下载任务吗?
A: 当前版本不支持同时只能执行一个任务
### Q4: 如何停止正在运行的任务?
A: 关闭API服务器窗口或重启服务器
---
**文档版本**: v1.0.0
**最后更新**: 2025-11-27
**维护者**: AI Assistant

26
backend/api/build.bat Normal file
View File

@@ -0,0 +1,26 @@
@echo off
chcp 65001 >nul
echo ===============================================
echo 📦 编译 API 服务器
echo ===============================================
echo.
echo 🔨 正在编译 api_server.exe...
go build -o api_server.exe server.go
if %errorlevel% neq 0 (
echo.
echo ❌ 编译失败!
echo.
pause
exit /b 1
)
echo.
echo ✅ 编译成功!
echo 📁 输出文件: api_server.exe
echo.
echo ===============================================
echo 编译完成
echo ===============================================
pause

543
backend/api/server.go Normal file
View File

@@ -0,0 +1,543 @@
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
)
// Response 统一响应结构
type Response struct {
Success bool `json:"success"`
Message string `json:"message"`
Data interface{} `json:"data,omitempty"`
}
// 任务状态
type TaskStatus struct {
Running bool `json:"running"`
Progress int `json:"progress"`
Message string `json:"message"`
Error string `json:"error,omitempty"`
}
var currentTask = &TaskStatus{Running: false}
func main() {
// 启用CORS
http.HandleFunc("/", corsMiddleware(handleRoot))
http.HandleFunc("/api/homepage/extract", corsMiddleware(extractHomepageHandler))
http.HandleFunc("/api/article/download", corsMiddleware(downloadArticleHandler))
http.HandleFunc("/api/article/list", corsMiddleware(getArticleListHandler))
http.HandleFunc("/api/article/batch", corsMiddleware(batchDownloadHandler))
http.HandleFunc("/api/data/list", corsMiddleware(getDataListHandler))
http.HandleFunc("/api/task/status", corsMiddleware(getTaskStatusHandler))
http.HandleFunc("/api/download/", corsMiddleware(downloadFileHandler))
port := ":8080"
fmt.Println("===============================================")
fmt.Println(" 🚀 微信公众号文章爬虫 API 服务器")
fmt.Println("===============================================")
fmt.Printf("🌐 服务地址: http://localhost%s\n", port)
fmt.Printf("⏰ 启动时间: %s\n", time.Now().Format("2006-01-02 15:04:05"))
fmt.Println("===============================================\n")
if err := http.ListenAndServe(port, nil); err != nil {
log.Fatal("服务器启动失败:", err)
}
}
// CORS中间件
func corsMiddleware(next http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Header().Set("Access-Control-Allow-Methods", "GET, POST, OPTIONS")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
if r.Method == "OPTIONS" {
w.WriteHeader(http.StatusOK)
return
}
next(w, r)
}
}
// 首页处理
func handleRoot(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
html := `
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>微信公众号文章爬虫 API</title>
<style>
body { font-family: Arial, sans-serif; max-width: 800px; margin: 50px auto; padding: 20px; }
h1 { color: #333; }
.endpoint { background: #f5f5f5; padding: 10px; margin: 10px 0; border-radius: 5px; }
.method { color: #4CAF50; font-weight: bold; }
</style>
</head>
<body>
<h1>🚀 微信公众号文章爬虫 API 服务器</h1>
<p>当前时间: ` + time.Now().Format("2006-01-02 15:04:05") + `</p>
<h2>可用接口:</h2>
<div class="endpoint">
<span class="method">POST</span> /api/homepage/extract - 提取公众号主页
</div>
<div class="endpoint">
<span class="method">POST</span> /api/article/download - 下载单篇文章
</div>
<div class="endpoint">
<span class="method">POST</span> /api/article/list - 获取文章列表
</div>
<div class="endpoint">
<span class="method">POST</span> /api/article/batch - 批量下载文章
</div>
<div class="endpoint">
<span class="method">GET</span> /api/data/list - 获取数据列表
</div>
<div class="endpoint">
<span class="method">GET</span> /api/task/status - 获取任务状态
</div>
</body>
</html>
`
w.Write([]byte(html))
}
// 提取公众号主页
func extractHomepageHandler(w http.ResponseWriter, r *http.Request) {
var req struct {
URL string `json:"url"`
}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeJSON(w, Response{Success: false, Message: "请求参数错误"})
return
}
// 执行命令(使用绝对路径)
exePath := filepath.Join("..", "wechat-crawler.exe")
absPath, _ := filepath.Abs(exePath)
log.Printf("尝试执行: %s", absPath)
cmd := exec.Command(absPath, req.URL)
workDir, _ := filepath.Abs("..")
cmd.Dir = workDir
output, err := cmd.CombinedOutput()
if err != nil {
log.Printf("执行失败: %v, 输出: %s", err, string(output))
writeJSON(w, Response{Success: false, Message: "执行失败: " + string(output)})
return
}
// 从输出中提取公众号主页链接
outputStr := string(output)
lines := strings.Split(outputStr, "\n")
var homepageURL string
for _, line := range lines {
if strings.Contains(line, "公众号主页链接") || strings.Contains(line, "https://mp.weixin.qq.com/mp/profile_ext") {
// 提取URL
if idx := strings.Index(line, "https://"); idx != -1 {
homepageURL = strings.TrimSpace(line[idx:])
break
}
}
}
if homepageURL == "" {
writeJSON(w, Response{Success: false, Message: "未能提取到主页链接"})
return
}
writeJSON(w, Response{
Success: true,
Message: "提取成功",
Data: map[string]string{
"homepage": homepageURL,
"output": outputStr,
},
})
}
// 下载单篇文章(这里需要实现具体逻辑)
func downloadArticleHandler(w http.ResponseWriter, r *http.Request) {
var req struct {
URL string `json:"url"`
SaveImage bool `json:"save_image"`
SaveContent bool `json:"save_content"`
}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeJSON(w, Response{Success: false, Message: "请求参数错误"})
return
}
currentTask.Running = true
currentTask.Progress = 0
currentTask.Message = "正在下载文章..."
// 注意:这里需要实际调用爬虫的下载功能
// 由于当前后端程序没有单独的下载单篇文章的命令行接口
// 需要后续实现或使用其他方式
writeJSON(w, Response{
Success: true,
Message: "下载任务已启动",
Data: map[string]interface{}{
"url": req.URL,
},
})
}
// 获取文章列表
func getArticleListHandler(w http.ResponseWriter, r *http.Request) {
var req struct {
AccessToken string `json:"access_token"`
Pages int `json:"pages"`
}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeJSON(w, Response{Success: false, Message: "请求参数错误"})
return
}
currentTask.Running = true
currentTask.Progress = 0
currentTask.Message = "正在获取文章列表..."
// 同步执行爬虫程序功能3
exePath := filepath.Join("..", "wechat-crawler.exe")
absPath, _ := filepath.Abs(exePath)
workDir, _ := filepath.Abs("..")
log.Printf("启动功能3: %s, 工作目录: %s", absPath, workDir)
cmd := exec.Command(absPath)
cmd.Dir = workDir
// 创建输入管道
stdin, err := cmd.StdinPipe()
if err != nil {
log.Printf("创建输入管道失败: %v", err)
currentTask.Running = false
writeJSON(w, Response{Success: false, Message: "创建输入管道失败: " + err.Error()})
return
}
// 启动命令
if err := cmd.Start(); err != nil {
log.Printf("启动命令失败: %v", err)
currentTask.Running = false
writeJSON(w, Response{Success: false, Message: "启动命令失败: " + err.Error()})
return
}
// 发送选项"3"功能3通过access_token获取文章列表
fmt.Fprintln(stdin, "3")
fmt.Fprintln(stdin, req.AccessToken)
if req.Pages > 0 {
fmt.Fprintf(stdin, "%d\n", req.Pages)
} else {
fmt.Fprintln(stdin, "0")
}
stdin.Close()
// 等待命令完成
if err := cmd.Wait(); err != nil {
log.Printf("命令执行失败: %v", err)
currentTask.Running = false
writeJSON(w, Response{Success: false, Message: "命令执行失败: " + err.Error()})
return
}
currentTask.Running = false
currentTask.Progress = 100
currentTask.Message = "文章列表获取完成"
// 查找生成的文件并返回下载链接
dataDir := "../data"
entries, err := os.ReadDir(dataDir)
if err != nil {
writeJSON(w, Response{Success: false, Message: "读取数据目录失败: " + err.Error()})
return
}
// 查找最新创建的公众号目录
var latestDir string
var latestTime time.Time
for _, entry := range entries {
if entry.IsDir() && entry.Name() != "." && entry.Name() != ".." {
info, _ := entry.Info()
if info.ModTime().After(latestTime) {
latestTime = info.ModTime()
latestDir = entry.Name()
}
}
}
if latestDir == "" {
writeJSON(w, Response{Success: false, Message: "未找到生成的数据目录"})
return
}
log.Printf("找到最新目录: %s", latestDir)
// 查找文章列表文件(优先查找直连链接文件)
accountPath := filepath.Join(dataDir, latestDir)
files, err := os.ReadDir(accountPath)
if err != nil {
writeJSON(w, Response{Success: false, Message: "读取公众号目录失败: " + err.Error()})
return
}
var excelFile string
// 优先查找直连链接文件(.xlsx或.txt
for _, file := range files {
if !file.IsDir() && strings.Contains(file.Name(), "直连链接") {
if strings.HasSuffix(file.Name(), ".xlsx") || strings.HasSuffix(file.Name(), ".txt") {
excelFile = file.Name()
log.Printf("找到直连链接文件: %s", excelFile)
break
}
}
}
// 如果没有直连链接文件,查找原始链接文件
if excelFile == "" {
for _, file := range files {
if !file.IsDir() && strings.Contains(file.Name(), "原始链接") {
if strings.HasSuffix(file.Name(), ".xlsx") || strings.HasSuffix(file.Name(), ".txt") {
excelFile = file.Name()
log.Printf("找到原始链接文件: %s", excelFile)
break
}
}
}
}
// 如果还是没有,查找任何文章列表文件
if excelFile == "" {
for _, file := range files {
if !file.IsDir() && strings.Contains(file.Name(), "文章列表") {
if strings.HasSuffix(file.Name(), ".xlsx") || strings.HasSuffix(file.Name(), ".txt") {
excelFile = file.Name()
log.Printf("找到文章列表文件: %s", excelFile)
break
}
}
}
}
if excelFile == "" {
// 列出所有文件用于调试
var fileList []string
for _, file := range files {
fileList = append(fileList, file.Name())
}
log.Printf("目录 %s 中的文件: %v", latestDir, fileList)
writeJSON(w, Response{Success: false, Message: "未找到Excel文件目录中的文件: " + strings.Join(fileList, ", ")})
return
}
writeJSON(w, Response{
Success: true,
Message: "文章列表获取成功",
Data: map[string]interface{}{
"account": latestDir,
"filename": excelFile,
"download": fmt.Sprintf("/download/%s/%s", latestDir, excelFile),
},
})
}
// 批量下载文章
func batchDownloadHandler(w http.ResponseWriter, r *http.Request) {
var req struct {
OfficialAccount string `json:"official_account"`
SaveImage bool `json:"save_image"`
SaveContent bool `json:"save_content"`
}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeJSON(w, Response{Success: false, Message: "请求参数错误"})
return
}
currentTask.Running = true
currentTask.Progress = 0
currentTask.Message = "正在批量下载文章..."
// 同步执行爬虫程序功能5
exePath := filepath.Join("..", "wechat-crawler.exe")
absPath, _ := filepath.Abs(exePath)
workDir, _ := filepath.Abs("..")
log.Printf("启动功能5: %s, 工作目录: %s", absPath, workDir)
cmd := exec.Command(absPath)
cmd.Dir = workDir
// 创建输入管道
stdin, err := cmd.StdinPipe()
if err != nil {
log.Printf("创建输入管道失败: %v", err)
currentTask.Running = false
writeJSON(w, Response{Success: false, Message: "创建输入管道失败: " + err.Error()})
return
}
// 启动命令
if err := cmd.Start(); err != nil {
log.Printf("启动命令失败: %v", err)
currentTask.Running = false
writeJSON(w, Response{Success: false, Message: "启动命令失败: " + err.Error()})
return
}
// 发送选项"5"功能5批量下载
fmt.Fprintln(stdin, "5")
fmt.Fprintln(stdin, req.OfficialAccount)
// 是否保存图片
if req.SaveImage {
fmt.Fprintln(stdin, "y")
} else {
fmt.Fprintln(stdin, "n")
}
stdin.Close()
// 等待命令完成
if err := cmd.Wait(); err != nil {
log.Printf("命令执行失败: %v", err)
currentTask.Running = false
writeJSON(w, Response{Success: false, Message: "命令执行失败: " + err.Error()})
return
}
currentTask.Running = false
currentTask.Progress = 100
currentTask.Message = "批量下载完成"
// 统计下载的文章数量
accountPath := filepath.Join("../data", req.OfficialAccount, "文章详细")
var articleCount int
if entries, err := os.ReadDir(accountPath); err == nil {
articleCount = len(entries)
}
writeJSON(w, Response{
Success: true,
Message: fmt.Sprintf("批量下载完成,共下载 %d 篇文章", articleCount),
Data: map[string]interface{}{
"account": req.OfficialAccount,
"articleCount": articleCount,
"path": accountPath,
},
})
}
// 获取数据列表
func getDataListHandler(w http.ResponseWriter, r *http.Request) {
dataDir := "../data"
var accounts []map[string]interface{}
entries, err := os.ReadDir(dataDir)
if err != nil {
// 如果目录不存在,返回空列表而不是错误
writeJSON(w, Response{
Success: true,
Data: accounts,
})
return
}
for _, entry := range entries {
if entry.IsDir() {
accountPath := filepath.Join(dataDir, entry.Name())
// 统计文章数量
detailPath := filepath.Join(accountPath, "文章详细")
var articleCount int
if detailEntries, err := os.ReadDir(detailPath); err == nil {
articleCount = len(detailEntries)
}
// 获取最后更新时间
info, _ := entry.Info()
lastUpdate := info.ModTime().Format("2006-01-02")
accounts = append(accounts, map[string]interface{}{
"name": entry.Name(),
"articleCount": articleCount,
"path": accountPath,
"lastUpdate": lastUpdate,
})
}
}
writeJSON(w, Response{
Success: true,
Data: accounts,
})
}
// 获取任务状态
func getTaskStatusHandler(w http.ResponseWriter, r *http.Request) {
writeJSON(w, Response{
Success: true,
Data: currentTask,
})
}
// 下载文件处理
func downloadFileHandler(w http.ResponseWriter, r *http.Request) {
// 从 URL 中提取路径 /api/download/公众号名称/文件名
path := strings.TrimPrefix(r.URL.Path, "/api/download/")
parts := strings.SplitN(path, "/", 2)
if len(parts) != 2 {
http.Error(w, "路径错误", http.StatusBadRequest)
return
}
accountName := parts[0]
filename := parts[1]
// 构建完整文件路径
filePath := filepath.Join("..", "data", accountName, filename)
absPath, _ := filepath.Abs(filePath)
// 检查文件是否存在
if _, err := os.Stat(absPath); os.IsNotExist(err) {
http.Error(w, "文件不存在", http.StatusNotFound)
return
}
log.Printf("下载文件: %s", absPath)
// 设置响应头
contentType := "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
if strings.HasSuffix(filename, ".txt") {
contentType = "text/plain; charset=utf-8"
}
w.Header().Set("Content-Type", contentType)
w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename*=UTF-8''%s", filename))
// 发送文件
http.ServeFile(w, r, absPath)
}
// 写入JSON响应
func writeJSON(w http.ResponseWriter, data interface{}) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
json.NewEncoder(w).Encode(data)
}

23
backend/api/start_api.bat Normal file
View File

@@ -0,0 +1,23 @@
@echo off
chcp 65001 >nul
title 微信公众号文章爬虫 - API服务器
:: 检查api_server.exe是否存在
if not exist "api_server.exe" (
echo ===============================================
echo ⚠️ API服务器未编译
echo ===============================================
echo.
echo 正在编译 API 服务器...
echo.
call build.bat
if %errorlevel% neq 0 (
echo 编译失败,无法启动服务器
pause
exit /b 1
)
)
:: 启动API服务器
cls
api_server.exe

View File

@@ -0,0 +1,11 @@
序号,创建时间,标题,链接
1,0,专家分析2026年考研报名人数,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500657&idx=1&sn=81eae7df4bfa2fdfc8bca69389489c52&chksm=ea981e22044aca6bbe5633849bfcd4903cb6f491646cd2ccf9321f4d9c852c64fe036f033c14&scene=27#wechat_redirect
2,0,教育部2026年全国硕士研究生报名人数为343万,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500650&idx=1&sn=9f230bbfefb24d98c18e42bd3651ad53&chksm=eac72972d56ff9b66f3658f0c3b1e6e363e56ddf879d56aba9c9c8f587b53ef00bcabe7992ff&scene=27#wechat_redirect
3,0,【小研来了】“务必再坚持坚持”,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500645&idx=1&sn=8e1d5921861dc4e3647f7bf8adaada81&chksm=ea26b17ce2f7255aacd9d1d6358c9aeb8d4e043c692efb8b4d8183cfc8363b3068be79d585c2&scene=27#wechat_redirect
4,0,学累了不点进来看看这4个“续航”方法,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500631&idx=1&sn=b640b0e43378e368166e50a7f46735f2&chksm=ea71f10a83b7811e1896cd9704eac5d064b763f3e020b5b37c72727c55bb1b0862a92e9c4cf0&scene=27#wechat_redirect
5,0,教育部:在“双一流”建设高校开展科技教育硕士培养,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500589&idx=1&sn=539d1229c9475ba5a2371698a362e9a7&chksm=ea4f97d3831139a276e50050f2f3307868b9c6ec7eb115bb9e288312f08572c47128a8016dce&scene=27#wechat_redirect
6,0,“研味儿”正浓,冲刺在即!请你一定别放弃,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500584&idx=1&sn=294b6ba8d12f0948913abf04af8cb188&chksm=ea4cfb5b16684bdd12634b6e46d8d8f3ab72ca9108be0d4d7f83dfded09c6ecb9f31b1531e31&scene=27#wechat_redirect
7,0,4个思维升级让我找回了读研的掌控感,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500579&idx=1&sn=fa00084c8711e3009ff7e31fe0b3bc51&chksm=eaff1ec212ddbb738d20542a965bbd1b79ae3a9d2e5af5704ddcf41de3a8b8d658e562771f0c&scene=27#wechat_redirect
8,0,研考网上确认成功后,需重点关注四件事,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500569&idx=1&sn=7707b698932ff6847de39d7351d3ac98&chksm=ea402eec6a96125a5bb02600aff24c3c1211eb5aaf5347080bbfe1f5861e9ca97fe9c400df21&scene=27#wechat_redirect
9,0,,
10,0,【小研来了】“小研,没有准考证照片怎么办?”,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500553&idx=1&sn=4fc6fd69684f02222e72d457c1004a81&chksm=eafc91ea346080790f9b641495fc3d9e31302ee5c2c9957eb4fa2bc9a139eda78163899b9219&scene=27#wechat_redirect

View File

@@ -4,6 +4,7 @@ import (
"fmt"
"io/ioutil"
"log"
"net/url"
"os"
"path/filepath"
"strings"
@@ -600,21 +601,48 @@ func parseAccessTokenParams(accessToken string) (string, string, string, string,
if err != nil {
return "", "", "", "", fmt.Errorf("未找到__biz参数")
}
// URL解码biz参数
biz, err = url.QueryUnescape(biz)
if err != nil {
fmt.Printf("警告: URL解码__biz失败: %v使用原始值\n", err)
}
uin, err := utils.ExtractFromRegex(accessToken, "uin=([^&]*)")
if err != nil {
return "", "", "", "", fmt.Errorf("未找到uin参数")
}
// URL解码uin参数
uin, err = url.QueryUnescape(uin)
if err != nil {
fmt.Printf("警告: URL解码uin失败: %v使用原始值\n", err)
}
key, err := utils.ExtractFromRegex(accessToken, "key=([^&]*)")
if err != nil {
return "", "", "", "", fmt.Errorf("未找到key参数")
}
// URL解码key参数
key, err = url.QueryUnescape(key)
if err != nil {
fmt.Printf("警告: URL解码key失败: %v使用原始值\n", err)
}
passTicket, err := utils.ExtractFromRegex(accessToken, "pass_ticket=([^&]*)")
if err != nil {
return "", "", "", "", fmt.Errorf("未找到pass_ticket参数")
}
// URL解码pass_ticket参数
passTicket, err = url.QueryUnescape(passTicket)
if err != nil {
fmt.Printf("警告: URL解码pass_ticket失败: %v使用原始值\n", err)
}
// 打印解码后的参数用于调试
fmt.Printf("\n提取到的参数已解码\n")
fmt.Printf(" __biz: %s\n", biz)
fmt.Printf(" uin: %s\n", uin)
fmt.Printf(" key长度: %d 字符\n", len(key))
fmt.Printf(" pass_ticket长度: %d 字符\n", len(passTicket))
return biz, uin, key, passTicket, nil
}

View File

@@ -1,27 +0,0 @@
package main
import (
"fmt"
"os"
"github.com/wechat-crawler/pkg/wechat"
)
func main() {
fmt.Println("开始测试文章内容提取功能...")
// 创建一个简单的爬虫实例
crawler := wechat.NewSimpleCrawler()
// 设置公众号名称(根据实际情况修改)
officialAccountName := "验证"
// 调用GetListArticleFromFile函数测试
err := crawler.GetListArticleFromFile(officialAccountName, false, true)
if err != nil {
fmt.Printf("测试失败: %v\n", err)
os.Exit(1)
}
fmt.Println("测试完成!请检查文章内容是否已正确提取。")
}

View File

@@ -1 +0,0 @@
__biz=MzUxMjA4MTI0MjA1; uin=MTIzNDU2Nzg5; key=abcdef1234567890abcdef1234567890; pass_ticket=abcdefghijklmnopqrstuvwxyz1234567890; version=63090b13; wxtype=1; pass_ticket=abcdefghijklmnopqrstuvwxyz1234567890;

View File

@@ -1,12 +0,0 @@
请将此文件重命名为cookie.txt并填入微信公众平台的cookie信息
如何获取cookie:
1. 打开浏览器,登录微信公众平台
2. 按F12打开开发者工具
3. 切换到Network标签
4. 刷新页面或访问任意页面
5. 选择一个请求查看Headers中的Cookie
6. 复制完整的Cookie到本文件中
Cookie格式示例:
__biz=MzUxMjA4MTI0MjA1; uin=MTIzNDU2Nzg5; key=abcdef1234567890abcdef1234567890; pass_ticket=abcdefghijklmnopqrstuvwxyz1234567890; version=63090b13; wxtype=1; pass_ticket=abcdefghijklmnopqrstuvwxyz1234567890;

36624
backend/debug_article_raw.html Normal file

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,246 @@
package main
import (
"encoding/json"
"fmt"
"log"
"github.com/wechat-crawler/pkg/database"
)
func main() {
fmt.Println("==============================================")
fmt.Println(" 微信公众号文章数据库管理系统示例")
fmt.Println("==============================================\n")
// 1. 初始化数据库
db, err := database.InitDB("../data/wechat_articles.db")
if err != nil {
log.Fatal("数据库初始化失败:", err)
}
defer db.Close()
// 2. 创建仓库实例
officialRepo := database.NewOfficialAccountRepository(db)
articleRepo := database.NewArticleRepository(db)
contentRepo := database.NewArticleContentRepository(db)
// 3. 示例:添加公众号
fmt.Println("📝 示例1: 添加公众号信息")
official := &database.OfficialAccount{
Biz: "MzI1NjEwMTM4OA==",
Nickname: "研招网资讯",
Homepage: "https://mp.weixin.qq.com/mp/profile_ext?action=home&__biz=MzI1NjEwMTM4OA==&scene=124",
Description: "中国研究生招生信息网官方公众号",
}
// 检查是否已存在
existing, err := officialRepo.GetByBiz(official.Biz)
if err != nil {
log.Fatal("查询公众号失败:", err)
}
var officialID int64
if existing == nil {
// 不存在,创建新记录
officialID, err = officialRepo.Create(official)
if err != nil {
log.Fatal("创建公众号失败:", err)
}
fmt.Printf("✅ 成功创建公众号: %s (ID: %d)\n\n", official.Nickname, officialID)
} else {
// 已存在
officialID = existing.ID
fmt.Printf(" 公众号已存在: %s (ID: %d)\n\n", existing.Nickname, officialID)
}
// 4. 示例:添加文章
fmt.Println("📝 示例2: 添加文章信息")
article := &database.Article{
OfficialID: officialID,
Title: "专家分析2026年考研报名人数",
Author: "研招网资讯",
Link: "https://mp.weixin.qq.com/s?__biz=MzI1NjEwMTM4OA==&mid=2651232405&idx=1",
PublishTime: "2024-11-27 10:00:00",
CreateTime: "2024-11-27 15:30:00",
CommentID: "2247491372",
ReadNum: 15234,
LikeNum: 456,
ShareNum: 123,
ContentPreview: "根据最新统计数据显示2026年全国硕士研究生报名人数预计将达到新高...",
ParagraphCount: 15,
}
// 检查文章是否已存在
existingArticle, err := articleRepo.GetByLink(article.Link)
if err != nil {
log.Fatal("查询文章失败:", err)
}
var articleID int64
if existingArticle == nil {
articleID, err = articleRepo.Create(article)
if err != nil {
log.Fatal("创建文章失败:", err)
}
fmt.Printf("✅ 成功创建文章: %s (ID: %d)\n\n", article.Title, articleID)
} else {
articleID = existingArticle.ID
fmt.Printf(" 文章已存在: %s (ID: %d)\n\n", existingArticle.Title, articleID)
}
// 5. 示例:添加文章内容
fmt.Println("📝 示例3: 添加文章详细内容")
paragraphs := []string{
"根据最新统计数据显示2026年全国硕士研究生报名人数预计将达到新高。",
"教育部相关负责人表示,随着社会对高层次人才需求的增加,考研热度持续上升。",
"专家建议考生理性选择,注重提升自身综合素质。",
}
images := []string{
"https://mmbiz.qpic.cn/mmbiz_jpg/xxx1.jpg",
"https://mmbiz.qpic.cn/mmbiz_jpg/xxx2.jpg",
}
content := &database.ArticleContent{
ArticleID: articleID,
HtmlContent: "<div>文章HTML内容</div>",
TextContent: "文章纯文本内容...",
Paragraphs: database.StringsToJSON(paragraphs),
Images: database.StringsToJSON(images),
}
// 检查内容是否已存在
existingContent, err := contentRepo.GetByArticleID(articleID)
if err != nil {
log.Fatal("查询文章内容失败:", err)
}
if existingContent == nil {
contentID, err := contentRepo.Create(content)
if err != nil {
log.Fatal("创建文章内容失败:", err)
}
fmt.Printf("✅ 成功添加文章内容 (ID: %d)\n\n", contentID)
} else {
fmt.Printf(" 文章内容已存在 (ID: %d)\n\n", existingContent.ID)
}
// 6. 示例:查询文章列表
fmt.Println("📋 示例4: 查询文章列表")
articles, total, err := articleRepo.List(officialID, 1, 10)
if err != nil {
log.Fatal("查询文章列表失败:", err)
}
fmt.Printf("共找到 %d 篇文章:\n", total)
for i, item := range articles {
fmt.Printf("%d. %s (👁️ %d | 👍 %d)\n", i+1, item.Title, item.ReadNum, item.LikeNum)
}
fmt.Println()
// 7. 示例:获取文章详情
fmt.Println("📖 示例5: 获取文章详情")
detail, err := contentRepo.GetArticleDetail(articleID)
if err != nil {
log.Fatal("获取文章详情失败:", err)
}
if detail != nil {
fmt.Printf("标题: %s\n", detail.Title)
fmt.Printf("作者: %s\n", detail.Author)
fmt.Printf("公众号: %s\n", detail.OfficialName)
fmt.Printf("发布时间: %s\n", detail.PublishTime)
fmt.Printf("阅读数: %d | 点赞数: %d\n", detail.ReadNum, detail.LikeNum)
fmt.Printf("段落数: %d\n", len(detail.Paragraphs))
fmt.Printf("图片数: %d\n", len(detail.Images))
if len(detail.Paragraphs) > 0 {
fmt.Printf("第一段: %s\n", detail.Paragraphs[0])
}
}
fmt.Println()
// 8. 示例:搜索文章
fmt.Println("🔍 示例6: 搜索文章")
searchResults, searchTotal, err := articleRepo.Search("考研", 1, 10)
if err != nil {
log.Fatal("搜索文章失败:", err)
}
fmt.Printf("搜索\"考研\"找到 %d 篇文章:\n", searchTotal)
for i, item := range searchResults {
fmt.Printf("%d. %s\n", i+1, item.Title)
}
fmt.Println()
// 9. 示例:获取统计信息
fmt.Println("📊 示例7: 获取统计信息")
stats, err := db.GetStatistics()
if err != nil {
log.Fatal("获取统计信息失败:", err)
}
fmt.Printf("公众号总数: %d\n", stats.TotalOfficials)
fmt.Printf("文章总数: %d\n", stats.TotalArticles)
fmt.Printf("总阅读数: %d\n", stats.TotalReadNum)
fmt.Printf("总点赞数: %d\n", stats.TotalLikeNum)
fmt.Println()
// 10. 示例:批量插入文章
fmt.Println("📦 示例8: 批量插入文章")
batchArticles := []*database.Article{
{
OfficialID: officialID,
Title: "教育部2026年全国硕士研究生报名人数为343万",
Author: "研招网资讯",
Link: "https://mp.weixin.qq.com/s?__biz=MzI1NjEwMTM4OA==&mid=2651232406",
PublishTime: "2024-11-26 09:00:00",
ReadNum: 8965,
LikeNum: 234,
ContentPreview: "教育部公布2026年研究生招生数据...",
ParagraphCount: 12,
},
{
OfficialID: officialID,
Title: "研考网上确认成功后,需重点关注四件事",
Author: "研招网资讯",
Link: "https://mp.weixin.qq.com/s?__biz=MzI1NjEwMTM4OA==&mid=2651232407",
PublishTime: "2024-11-25 15:30:00",
ReadNum: 6543,
LikeNum: 189,
ContentPreview: "网上确认通过后,考生还需要注意以下事项...",
ParagraphCount: 8,
},
}
err = articleRepo.BatchInsertArticles(batchArticles)
if err != nil {
log.Fatal("批量插入文章失败:", err)
}
fmt.Printf("✅ 成功批量插入 %d 篇文章\n\n", len(batchArticles))
// 11. 示例导出JSON数据
fmt.Println("💾 示例9: 导出文章列表为JSON")
allArticles, _, err := articleRepo.List(0, 1, 100)
if err != nil {
log.Fatal("查询文章列表失败:", err)
}
jsonData, err := json.MarshalIndent(allArticles, "", " ")
if err != nil {
log.Fatal("JSON序列化失败:", err)
}
fmt.Println("文章列表JSON (前200字符):")
if len(jsonData) > 200 {
fmt.Println(string(jsonData[:200]) + "...")
} else {
fmt.Println(string(jsonData))
}
fmt.Println()
fmt.Println("==============================================")
fmt.Println(" 数据库操作示例演示完成!")
fmt.Println("==============================================")
}

View File

@@ -1,7 +0,0 @@
module github.com/wechat-crawler
go 1.20
require github.com/go-resty/resty/v2 v2.10.0
require golang.org/x/net v0.17.0 // indirect

View File

@@ -1,44 +0,0 @@
github.com/go-resty/resty/v2 v2.10.0 h1:Qla4W/+TMmv0fOeeRqzEpXPLfTUnR5HZ1+lGs+CkiCo=
github.com/go-resty/resty/v2 v2.10.0/go.mod h1:iiP/OpA0CkcL3IGt1O0+/SIItFUbkkyw5BGXiVdTu+A=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf4=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM=
golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.13.0/go.mod h1:LTmsnFJwVN6bCy1rVCoS+qHT1HhALEFxKncY3WNNh4U=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4=
golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=

Binary file not shown.

Binary file not shown.

117
backend/pkg/database/db.go Normal file
View File

@@ -0,0 +1,117 @@
package database
import (
"database/sql"
"fmt"
"os"
"path/filepath"
_ "modernc.org/sqlite"
)
// DB 数据库实例
type DB struct {
*sql.DB
}
// InitDB 初始化数据库
func InitDB(dbPath string) (*DB, error) {
// 确保数据库目录存在
dbDir := filepath.Dir(dbPath)
if err := os.MkdirAll(dbDir, 0755); err != nil {
return nil, fmt.Errorf("创建数据库目录失败: %v", err)
}
// 打开数据库连接使用modernc.org/sqlite驱动
db, err := sql.Open("sqlite", dbPath)
if err != nil {
return nil, fmt.Errorf("打开数据库失败: %v", err)
}
// 测试连接
if err := db.Ping(); err != nil {
return nil, fmt.Errorf("数据库连接测试失败: %v", err)
}
// 创建表
if err := createTables(db); err != nil {
return nil, fmt.Errorf("创建数据表失败: %v", err)
}
fmt.Println("✅ 数据库初始化成功:", dbPath)
return &DB{db}, nil
}
// createTables 创建数据表
func createTables(db *sql.DB) error {
// 公众号表
officialAccountTable := `
CREATE TABLE IF NOT EXISTS official_accounts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
biz TEXT NOT NULL UNIQUE,
nickname TEXT NOT NULL,
homepage TEXT,
description TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_biz ON official_accounts(biz);
CREATE INDEX IF NOT EXISTS idx_nickname ON official_accounts(nickname);
`
// 文章表
articleTable := `
CREATE TABLE IF NOT EXISTS articles (
id INTEGER PRIMARY KEY AUTOINCREMENT,
official_id INTEGER NOT NULL,
title TEXT NOT NULL,
author TEXT,
link TEXT UNIQUE,
publish_time TEXT,
create_time TEXT,
comment_id TEXT,
read_num INTEGER DEFAULT 0,
like_num INTEGER DEFAULT 0,
share_num INTEGER DEFAULT 0,
content_preview TEXT,
paragraph_count INTEGER DEFAULT 0,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (official_id) REFERENCES official_accounts(id)
);
CREATE INDEX IF NOT EXISTS idx_official_id ON articles(official_id);
CREATE INDEX IF NOT EXISTS idx_title ON articles(title);
CREATE INDEX IF NOT EXISTS idx_publish_time ON articles(publish_time);
CREATE INDEX IF NOT EXISTS idx_link ON articles(link);
`
// 文章内容表
articleContentTable := `
CREATE TABLE IF NOT EXISTS article_contents (
id INTEGER PRIMARY KEY AUTOINCREMENT,
article_id INTEGER NOT NULL UNIQUE,
html_content TEXT,
text_content TEXT,
paragraphs TEXT,
images TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (article_id) REFERENCES articles(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_article_id ON article_contents(article_id);
`
// 执行创建表语句
tables := []string{officialAccountTable, articleTable, articleContentTable}
for _, table := range tables {
if _, err := db.Exec(table); err != nil {
return err
}
}
return nil
}
// Close 关闭数据库连接
func (db *DB) Close() error {
return db.DB.Close()
}

View File

@@ -0,0 +1,76 @@
package database
import (
"time"
)
// OfficialAccount 公众号信息
type OfficialAccount struct {
ID int64 `json:"id"`
Biz string `json:"biz"` // 公众号唯一标识
Nickname string `json:"nickname"` // 公众号名称
Homepage string `json:"homepage"` // 公众号主页链接
Description string `json:"description"` // 公众号描述
CreatedAt time.Time `json:"created_at"` // 创建时间
UpdatedAt time.Time `json:"updated_at"` // 更新时间
}
// Article 文章信息
type Article struct {
ID int64 `json:"id"`
OfficialID int64 `json:"official_id"` // 关联的公众号ID
Title string `json:"title"` // 文章标题
Author string `json:"author"` // 作者
Link string `json:"link"` // 文章链接
PublishTime string `json:"publish_time"` // 发布时间
CreateTime string `json:"create_time"` // 创建时间(抓取时间)
CommentID string `json:"comment_id"` // 评论ID
ReadNum int `json:"read_num"` // 阅读数
LikeNum int `json:"like_num"` // 点赞数
ShareNum int `json:"share_num"` // 分享数
ContentPreview string `json:"content_preview"` // 内容预览前200字
ParagraphCount int `json:"paragraph_count"` // 段落数
CreatedAt time.Time `json:"created_at"` // 数据库创建时间
UpdatedAt time.Time `json:"updated_at"` // 数据库更新时间
}
// ArticleContent 文章详细内容
type ArticleContent struct {
ID int64 `json:"id"`
ArticleID int64 `json:"article_id"` // 关联的文章ID
HtmlContent string `json:"html_content"` // HTML原始内容
TextContent string `json:"text_content"` // 纯文本内容
Paragraphs string `json:"paragraphs"` // 段落内容JSON数组
Images string `json:"images"` // 图片链接JSON数组
CreatedAt time.Time `json:"created_at"` // 创建时间
}
// ArticleListItem 文章列表项用于API返回
type ArticleListItem struct {
ID int64 `json:"id"`
Title string `json:"title"`
Author string `json:"author"`
PublishTime string `json:"publish_time"`
ReadNum int `json:"read_num"`
LikeNum int `json:"like_num"`
OfficialName string `json:"official_name"`
ContentPreview string `json:"content_preview"`
}
// ArticleDetail 文章详情用于API返回
type ArticleDetail struct {
Article
OfficialName string `json:"official_name"`
HtmlContent string `json:"html_content"`
TextContent string `json:"text_content"`
Paragraphs []string `json:"paragraphs"`
Images []string `json:"images"`
}
// Statistics 统计信息
type Statistics struct {
TotalOfficials int `json:"total_officials"` // 公众号总数
TotalArticles int `json:"total_articles"` // 文章总数
TotalReadNum int `json:"total_read_num"` // 总阅读数
TotalLikeNum int `json:"total_like_num"` // 总点赞数
}

View File

@@ -0,0 +1,455 @@
package database
import (
"database/sql"
"encoding/json"
"fmt"
"strings"
)
// OfficialAccountRepository 公众号数据仓库
type OfficialAccountRepository struct {
db *DB
}
// NewOfficialAccountRepository 创建公众号仓库
func NewOfficialAccountRepository(db *DB) *OfficialAccountRepository {
return &OfficialAccountRepository{db: db}
}
// Create 创建公众号
func (r *OfficialAccountRepository) Create(account *OfficialAccount) (int64, error) {
result, err := r.db.Exec(`
INSERT INTO official_accounts (biz, nickname, homepage, description)
VALUES (?, ?, ?, ?)
`, account.Biz, account.Nickname, account.Homepage, account.Description)
if err != nil {
return 0, err
}
return result.LastInsertId()
}
// GetByBiz 根据Biz获取公众号
func (r *OfficialAccountRepository) GetByBiz(biz string) (*OfficialAccount, error) {
account := &OfficialAccount{}
err := r.db.QueryRow(`
SELECT id, biz, nickname, homepage, description, created_at, updated_at
FROM official_accounts WHERE biz = ?
`, biz).Scan(&account.ID, &account.Biz, &account.Nickname, &account.Homepage,
&account.Description, &account.CreatedAt, &account.UpdatedAt)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
return account, nil
}
// GetByID 根据ID获取公众号
func (r *OfficialAccountRepository) GetByID(id int64) (*OfficialAccount, error) {
account := &OfficialAccount{}
err := r.db.QueryRow(`
SELECT id, biz, nickname, homepage, description, created_at, updated_at
FROM official_accounts WHERE id = ?
`, id).Scan(&account.ID, &account.Biz, &account.Nickname, &account.Homepage,
&account.Description, &account.CreatedAt, &account.UpdatedAt)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
return account, nil
}
// List 获取所有公众号列表
func (r *OfficialAccountRepository) List() ([]*OfficialAccount, error) {
rows, err := r.db.Query(`
SELECT id, biz, nickname, homepage, description, created_at, updated_at
FROM official_accounts ORDER BY created_at DESC
`)
if err != nil {
return nil, err
}
defer rows.Close()
var accounts []*OfficialAccount
for rows.Next() {
account := &OfficialAccount{}
err := rows.Scan(&account.ID, &account.Biz, &account.Nickname, &account.Homepage,
&account.Description, &account.CreatedAt, &account.UpdatedAt)
if err != nil {
return nil, err
}
accounts = append(accounts, account)
}
return accounts, nil
}
// Update 更新公众号信息
func (r *OfficialAccountRepository) Update(account *OfficialAccount) error {
_, err := r.db.Exec(`
UPDATE official_accounts
SET nickname = ?, homepage = ?, description = ?, updated_at = CURRENT_TIMESTAMP
WHERE id = ?
`, account.Nickname, account.Homepage, account.Description, account.ID)
return err
}
// ArticleRepository 文章数据仓库
type ArticleRepository struct {
db *DB
}
// NewArticleRepository 创建文章仓库
func NewArticleRepository(db *DB) *ArticleRepository {
return &ArticleRepository{db: db}
}
// Create 创建文章
func (r *ArticleRepository) Create(article *Article) (int64, error) {
result, err := r.db.Exec(`
INSERT INTO articles (
official_id, title, author, link, publish_time, create_time,
comment_id, read_num, like_num, share_num, content_preview, paragraph_count
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`, article.OfficialID, article.Title, article.Author, article.Link,
article.PublishTime, article.CreateTime, article.CommentID,
article.ReadNum, article.LikeNum, article.ShareNum,
article.ContentPreview, article.ParagraphCount)
if err != nil {
return 0, err
}
return result.LastInsertId()
}
// GetByID 根据ID获取文章
func (r *ArticleRepository) GetByID(id int64) (*Article, error) {
article := &Article{}
err := r.db.QueryRow(`
SELECT id, official_id, title, author, link, publish_time, create_time,
comment_id, read_num, like_num, share_num, content_preview,
paragraph_count, created_at, updated_at
FROM articles WHERE id = ?
`, id).Scan(&article.ID, &article.OfficialID, &article.Title, &article.Author,
&article.Link, &article.PublishTime, &article.CreateTime, &article.CommentID,
&article.ReadNum, &article.LikeNum, &article.ShareNum, &article.ContentPreview,
&article.ParagraphCount, &article.CreatedAt, &article.UpdatedAt)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
return article, nil
}
// GetByLink 根据链接获取文章
func (r *ArticleRepository) GetByLink(link string) (*Article, error) {
article := &Article{}
err := r.db.QueryRow(`
SELECT id, official_id, title, author, link, publish_time, create_time,
comment_id, read_num, like_num, share_num, content_preview,
paragraph_count, created_at, updated_at
FROM articles WHERE link = ?
`, link).Scan(&article.ID, &article.OfficialID, &article.Title, &article.Author,
&article.Link, &article.PublishTime, &article.CreateTime, &article.CommentID,
&article.ReadNum, &article.LikeNum, &article.ShareNum, &article.ContentPreview,
&article.ParagraphCount, &article.CreatedAt, &article.UpdatedAt)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
return article, nil
}
// List 获取文章列表(分页)
func (r *ArticleRepository) List(officialID int64, page, pageSize int) ([]*ArticleListItem, int, error) {
// 构建查询条件
whereClause := ""
args := []interface{}{}
if officialID > 0 {
whereClause = "WHERE a.official_id = ?"
args = append(args, officialID)
}
// 获取总数
countQuery := fmt.Sprintf("SELECT COUNT(*) FROM articles a %s", whereClause)
var total int
err := r.db.QueryRow(countQuery, args...).Scan(&total)
if err != nil {
return nil, 0, err
}
// 获取列表
offset := (page - 1) * pageSize
listQuery := fmt.Sprintf(`
SELECT a.id, a.title, a.author, a.publish_time, a.read_num, a.like_num,
a.content_preview, o.nickname
FROM articles a
LEFT JOIN official_accounts o ON a.official_id = o.id
%s
ORDER BY a.publish_time DESC
LIMIT ? OFFSET ?
`, whereClause)
args = append(args, pageSize, offset)
rows, err := r.db.Query(listQuery, args...)
if err != nil {
return nil, 0, err
}
defer rows.Close()
var items []*ArticleListItem
for rows.Next() {
item := &ArticleListItem{}
err := rows.Scan(&item.ID, &item.Title, &item.Author, &item.PublishTime,
&item.ReadNum, &item.LikeNum, &item.ContentPreview, &item.OfficialName)
if err != nil {
return nil, 0, err
}
items = append(items, item)
}
return items, total, nil
}
// Search 搜索文章
func (r *ArticleRepository) Search(keyword string, page, pageSize int) ([]*ArticleListItem, int, error) {
keyword = "%" + keyword + "%"
// 获取总数
var total int
err := r.db.QueryRow(`
SELECT COUNT(*) FROM articles WHERE title LIKE ? OR author LIKE ?
`, keyword, keyword).Scan(&total)
if err != nil {
return nil, 0, err
}
// 获取列表
offset := (page - 1) * pageSize
rows, err := r.db.Query(`
SELECT a.id, a.title, a.author, a.publish_time, a.read_num, a.like_num,
a.content_preview, o.nickname
FROM articles a
LEFT JOIN official_accounts o ON a.official_id = o.id
WHERE a.title LIKE ? OR a.author LIKE ?
ORDER BY a.publish_time DESC
LIMIT ? OFFSET ?
`, keyword, keyword, pageSize, offset)
if err != nil {
return nil, 0, err
}
defer rows.Close()
var items []*ArticleListItem
for rows.Next() {
item := &ArticleListItem{}
err := rows.Scan(&item.ID, &item.Title, &item.Author, &item.PublishTime,
&item.ReadNum, &item.LikeNum, &item.ContentPreview, &item.OfficialName)
if err != nil {
return nil, 0, err
}
items = append(items, item)
}
return items, total, nil
}
// Update 更新文章信息
func (r *ArticleRepository) Update(article *Article) error {
_, err := r.db.Exec(`
UPDATE articles
SET read_num = ?, like_num = ?, share_num = ?, updated_at = CURRENT_TIMESTAMP
WHERE id = ?
`, article.ReadNum, article.LikeNum, article.ShareNum, article.ID)
return err
}
// ArticleContentRepository 文章内容数据仓库
type ArticleContentRepository struct {
db *DB
}
// NewArticleContentRepository 创建文章内容仓库
func NewArticleContentRepository(db *DB) *ArticleContentRepository {
return &ArticleContentRepository{db: db}
}
// Create 创建文章内容
func (r *ArticleContentRepository) Create(content *ArticleContent) (int64, error) {
result, err := r.db.Exec(`
INSERT INTO article_contents (article_id, html_content, text_content, paragraphs, images)
VALUES (?, ?, ?, ?, ?)
`, content.ArticleID, content.HtmlContent, content.TextContent,
content.Paragraphs, content.Images)
if err != nil {
return 0, err
}
return result.LastInsertId()
}
// GetByArticleID 根据文章ID获取内容
func (r *ArticleContentRepository) GetByArticleID(articleID int64) (*ArticleContent, error) {
content := &ArticleContent{}
err := r.db.QueryRow(`
SELECT id, article_id, html_content, text_content, paragraphs, images, created_at
FROM article_contents WHERE article_id = ?
`, articleID).Scan(&content.ID, &content.ArticleID, &content.HtmlContent,
&content.TextContent, &content.Paragraphs, &content.Images, &content.CreatedAt)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
return content, nil
}
// GetArticleDetail 获取文章详情(包含内容)
func (r *ArticleContentRepository) GetArticleDetail(articleID int64) (*ArticleDetail, error) {
detail := &ArticleDetail{}
var paragraphsJSON, imagesJSON string
err := r.db.QueryRow(`
SELECT a.id, a.official_id, a.title, a.author, a.link, a.publish_time,
a.create_time, a.comment_id, a.read_num, a.like_num, a.share_num,
a.content_preview, a.paragraph_count, a.created_at, a.updated_at,
o.nickname, c.html_content, c.text_content, c.paragraphs, c.images
FROM articles a
LEFT JOIN official_accounts o ON a.official_id = o.id
LEFT JOIN article_contents c ON a.id = c.article_id
WHERE a.id = ?
`, articleID).Scan(
&detail.ID, &detail.OfficialID, &detail.Title, &detail.Author,
&detail.Link, &detail.PublishTime, &detail.CreateTime, &detail.CommentID,
&detail.ReadNum, &detail.LikeNum, &detail.ShareNum, &detail.ContentPreview,
&detail.ParagraphCount, &detail.CreatedAt, &detail.UpdatedAt,
&detail.OfficialName, &detail.HtmlContent, &detail.TextContent,
&paragraphsJSON, &imagesJSON,
)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
// 解析JSON数组
if paragraphsJSON != "" {
json.Unmarshal([]byte(paragraphsJSON), &detail.Paragraphs)
}
if imagesJSON != "" {
json.Unmarshal([]byte(imagesJSON), &detail.Images)
}
return detail, nil
}
// GetStatistics 获取统计信息
func (db *DB) GetStatistics() (*Statistics, error) {
stats := &Statistics{}
err := db.QueryRow(`
SELECT
(SELECT COUNT(*) FROM official_accounts) as total_officials,
(SELECT COUNT(*) FROM articles) as total_articles,
(SELECT COALESCE(SUM(read_num), 0) FROM articles) as total_read_num,
(SELECT COALESCE(SUM(like_num), 0) FROM articles) as total_like_num
`).Scan(&stats.TotalOfficials, &stats.TotalArticles, &stats.TotalReadNum, &stats.TotalLikeNum)
if err != nil {
return nil, err
}
return stats, nil
}
// BatchInsertArticles 批量插入文章
func (r *ArticleRepository) BatchInsertArticles(articles []*Article) error {
if len(articles) == 0 {
return nil
}
// 开始事务
tx, err := r.db.Begin()
if err != nil {
return err
}
defer tx.Rollback()
stmt, err := tx.Prepare(`
INSERT OR IGNORE INTO articles (
official_id, title, author, link, publish_time, create_time,
comment_id, read_num, like_num, share_num, content_preview, paragraph_count
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`)
if err != nil {
return err
}
defer stmt.Close()
for _, article := range articles {
_, err = stmt.Exec(
article.OfficialID, article.Title, article.Author, article.Link,
article.PublishTime, article.CreateTime, article.CommentID,
article.ReadNum, article.LikeNum, article.ShareNum,
article.ContentPreview, article.ParagraphCount,
)
if err != nil {
return err
}
}
return tx.Commit()
}
// Helper function: 将字符串数组转换为JSON字符串
func StringsToJSON(strs []string) string {
if len(strs) == 0 {
return "[]"
}
data, _ := json.Marshal(strs)
return string(data)
}
// Helper function: 生成内容预览
func GeneratePreview(content string, maxLen int) string {
if len(content) <= maxLen {
return content
}
// 移除换行符和多余空格
content = strings.ReplaceAll(content, "\n", " ")
content = strings.ReplaceAll(content, "\r", "")
content = strings.Join(strings.Fields(content), " ")
if len(content) <= maxLen {
return content
}
return content[:maxLen] + "..."
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,48 +0,0 @@
@echo off
echo WeChat Public Article Crawler Startup Script
echo =================================
REM Check if cookie.txt file exists
if not exist "cookie.txt" (
echo Error: cookie.txt file not found!
echo Please create cookie.txt file in backend directory and add WeChat public platform cookie information.
echo.
echo cookie.txt format example:
echo __biz=xxx; uin=xxx; key=xxx; pass_ticket=xxx;
echo.
pause
exit /b 1
)
REM Set Go environment variables (if needed)
REM set GOPATH=%USERPROFILE%\go
REM set GOROOT=C:\Go
REM set PATH=%PATH%;%GOROOT%\bin;%GOPATH%\bin
echo Downloading dependencies...
go mod tidy
if %errorlevel% neq 0 (
echo Failed to download dependencies!
pause
exit /b 1
)
echo Compiling program...
go build -o output\wechat-crawler.exe cmd\main.go
if %errorlevel% neq 0 (
echo Compilation failed!
pause
exit /b 1
)
echo Compilation successful! Starting program...
echo.
REM Ensure data directory exists
if not exist "data" mkdir data
REM Run the program
output\wechat-crawler.exe
pause

View File

@@ -1,57 +0,0 @@
@echo off
rem WeChat Official Account Article Crawler - Script for crawling via article link
setlocal enabledelayedexpansion
REM 检查是否有命令行参数传入
if "%1" neq "" (
REM 如果有参数,直接将其作为文章链接传入程序
echo.
echo Compiling and running...
go run "cmd/main.go" "%1"
if errorlevel 1 (
echo.
echo Failed to run, please check error messages above
pause
exit /b 1
)
echo.
echo Crawling completed successfully!
pause
exit /b 0
) else (
REM 如果没有参数,运行交互式模式
:input_loop
cls
echo ========================================
echo WeChat Official Account Article Crawler
echo ========================================
echo.
echo Please enter WeChat article link:
echo Example: https://mp.weixin.qq.com/s/4r_LKJu0mOeUc70ZZXK9LA
set /p ARTICLE_LINK=
if "%ARTICLE_LINK%"=="" (
echo.
echo Error: Article link cannot be empty!
pause
goto input_loop
)
echo.
echo Compiling and running...
go run "cmd/main.go" "%ARTICLE_LINK%"
if errorlevel 1 (
echo.
echo Failed to run, please check error messages above
pause
exit /b 1
)
echo.
echo Crawling completed successfully!
pause
)

21
backend/tools/view_db.bat Normal file
View File

@@ -0,0 +1,21 @@
@echo off
chcp 65001 >nul
cls
echo ===============================================
echo 📊 数据库内容查看工具
echo ===============================================
echo.
cd /d "%~dp0"
echo 正在查询数据库...
echo.
go run view_db.go
echo.
echo ===============================================
echo 查询完成!
echo ===============================================
pause

231
backend/tools/view_db.go Normal file
View File

@@ -0,0 +1,231 @@
package main
import (
"database/sql"
"encoding/json"
"fmt"
"log"
_ "modernc.org/sqlite"
)
func main() {
// 打开数据库
db, err := sql.Open("sqlite", "../../data/wechat_articles.db")
if err != nil {
log.Fatal("打开数据库失败:", err)
}
defer db.Close()
fmt.Println("=" + repeatStr("=", 80))
fmt.Println("📊 微信公众号文章数据库内容查看")
fmt.Println("=" + repeatStr("=", 80))
// 查询公众号
fmt.Println("\n📢 【公众号列表】")
fmt.Println(repeatStr("-", 80))
queryOfficialAccounts(db)
// 查询文章
fmt.Println("\n📝 【文章列表】")
fmt.Println(repeatStr("-", 80))
queryArticles(db)
// 查询文章内容
fmt.Println("\n📄 【文章详细内容】")
fmt.Println(repeatStr("-", 80))
queryArticleContents(db)
fmt.Println("\n" + repeatStr("=", 80))
}
func queryOfficialAccounts(db *sql.DB) {
rows, err := db.Query(`
SELECT id, biz, nickname, homepage, description, created_at, updated_at
FROM official_accounts
ORDER BY id
`)
if err != nil {
log.Printf("查询公众号失败: %v\n", err)
return
}
defer rows.Close()
count := 0
for rows.Next() {
var id int
var biz, nickname, homepage, description, createdAt, updatedAt string
err := rows.Scan(&id, &biz, &nickname, &homepage, &description, &createdAt, &updatedAt)
if err != nil {
log.Printf("读取数据失败: %v\n", err)
continue
}
count++
fmt.Printf("\n🔹 公众号 #%d\n", id)
fmt.Printf(" 名称: %s\n", nickname)
fmt.Printf(" BIZ: %s\n", biz)
fmt.Printf(" 主页: %s\n", homepage)
fmt.Printf(" 简介: %s\n", description)
fmt.Printf(" 创建时间: %s\n", createdAt)
fmt.Printf(" 更新时间: %s\n", updatedAt)
}
if count == 0 {
fmt.Println(" 暂无数据")
} else {
fmt.Printf("\n总计: %d 个公众号\n", count)
}
}
func queryArticles(db *sql.DB) {
rows, err := db.Query(`
SELECT a.id, a.official_id, a.title, a.author, a.link, a.publish_time,
a.read_num, a.like_num, a.share_num, a.paragraph_count,
a.content_preview, a.created_at, oa.nickname
FROM articles a
LEFT JOIN official_accounts oa ON a.official_id = oa.id
ORDER BY a.id
`)
if err != nil {
log.Printf("查询文章失败: %v\n", err)
return
}
defer rows.Close()
count := 0
for rows.Next() {
var id, officialID, readNum, likeNum, shareNum, paragraphCount int
var title, author, link, publishTime, contentPreview, createdAt, officialName sql.NullString
err := rows.Scan(&id, &officialID, &title, &author, &link, &publishTime,
&readNum, &likeNum, &shareNum, &paragraphCount, &contentPreview, &createdAt, &officialName)
if err != nil {
log.Printf("读取数据失败: %v\n", err)
continue
}
count++
fmt.Printf("\n🔹 文章 #%d\n", id)
fmt.Printf(" 标题: %s\n", getStringValue(title))
if officialName.Valid {
fmt.Printf(" 公众号: %s\n", officialName.String)
}
fmt.Printf(" 作者: %s\n", getStringValue(author))
fmt.Printf(" 链接: %s\n", getStringValue(link))
fmt.Printf(" 发布时间: %s\n", getStringValue(publishTime))
fmt.Printf(" 阅读数: %d | 点赞数: %d | 分享数: %d\n", readNum, likeNum, shareNum)
fmt.Printf(" 段落数: %d\n", paragraphCount)
if contentPreview.Valid && contentPreview.String != "" {
preview := contentPreview.String
if len(preview) > 100 {
preview = preview[:100] + "..."
}
fmt.Printf(" 内容预览: %s\n", preview)
}
fmt.Printf(" 抓取时间: %s\n", getStringValue(createdAt))
}
if count == 0 {
fmt.Println(" 暂无数据")
} else {
fmt.Printf("\n总计: %d 篇文章\n", count)
}
}
func queryArticleContents(db *sql.DB) {
rows, err := db.Query(`
SELECT ac.id, ac.article_id, ac.html_content, ac.text_content,
ac.paragraphs, ac.images, ac.created_at, a.title
FROM article_contents ac
LEFT JOIN articles a ON ac.article_id = a.id
ORDER BY ac.id
`)
if err != nil {
log.Printf("查询文章内容失败: %v\n", err)
return
}
defer rows.Close()
count := 0
for rows.Next() {
var id, articleID int
var htmlContent, textContent, paragraphs, images, createdAt, title sql.NullString
err := rows.Scan(&id, &articleID, &htmlContent, &textContent,
&paragraphs, &images, &createdAt, &title)
if err != nil {
log.Printf("读取数据失败: %v\n", err)
continue
}
count++
fmt.Printf("\n🔹 内容 #%d (文章ID: %d)\n", id, articleID)
if title.Valid {
fmt.Printf(" 文章标题: %s\n", title.String)
}
// HTML内容长度
htmlLen := 0
if htmlContent.Valid {
htmlLen = len(htmlContent.String)
}
fmt.Printf(" HTML内容长度: %d 字符\n", htmlLen)
// 文本内容
if textContent.Valid && textContent.String != "" {
text := textContent.String
if len(text) > 200 {
text = text[:200] + "..."
}
fmt.Printf(" 文本内容: %s\n", text)
}
// 段落信息
if paragraphs.Valid && paragraphs.String != "" {
var paragraphList []interface{}
if err := json.Unmarshal([]byte(paragraphs.String), &paragraphList); err == nil {
fmt.Printf(" 段落数量: %d\n", len(paragraphList))
}
}
// 图片信息
if images.Valid && images.String != "" {
var imageList []interface{}
if err := json.Unmarshal([]byte(images.String), &imageList); err == nil {
fmt.Printf(" 图片数量: %d\n", len(imageList))
if len(imageList) > 0 {
fmt.Printf(" 图片URL:\n")
for i, img := range imageList {
if i >= 3 {
fmt.Printf(" ... 还有 %d 张图片\n", len(imageList)-3)
break
}
fmt.Printf(" %d. %v\n", i+1, img)
}
}
}
}
fmt.Printf(" 存储时间: %s\n", getStringValue(createdAt))
}
if count == 0 {
fmt.Println(" 暂无数据")
} else {
fmt.Printf("\n总计: %d 条详细内容\n", count)
}
}
func getStringValue(s sql.NullString) string {
if s.Valid {
return s.String
}
return ""
}
func repeatStr(s string, n int) string {
result := ""
for i := 0; i < n; i++ {
result += s
}
return result
}