新版可用
This commit is contained in:
@@ -1,73 +0,0 @@
|
||||
# 微信公众号文章爬取工具(Go版本)
|
||||
|
||||
这是一个基于Go语言开发的微信公众号文章爬取工具,可以自动获取指定公众号的所有文章列表和详细内容。
|
||||
|
||||
## 功能特性
|
||||
|
||||
- 获取公众号所有文章列表
|
||||
- 获取每篇文章的详细内容
|
||||
- 获取文章的阅读量、点赞数、转发数等统计信息
|
||||
- 支持获取文章评论
|
||||
- 自动保存文章列表和详细内容
|
||||
|
||||
## 环境要求
|
||||
|
||||
- Go 1.20 或更高版本
|
||||
- Windows 操作系统(脚本已针对Windows优化)
|
||||
|
||||
## 安装使用
|
||||
|
||||
### 1. 配置Cookie
|
||||
|
||||
- 将 `cookie.txt.example` 重命名为 `cookie.txt`
|
||||
- 按照文件中的说明获取微信公众平台的Cookie
|
||||
- 将Cookie信息粘贴到 `cookie.txt` 文件中
|
||||
|
||||
### 2. 运行程序
|
||||
|
||||
直接双击 `run.bat` 脚本文件,程序会自动:
|
||||
- 下载所需依赖
|
||||
- 编译Go程序
|
||||
- 运行爬取工具
|
||||
|
||||
## 项目结构
|
||||
|
||||
```
|
||||
backend/
|
||||
├── cmd/
|
||||
│ └── main.go # 主程序入口
|
||||
├── configs/
|
||||
│ └── config.go # 配置管理
|
||||
├── pkg/
|
||||
│ ├── utils/ # 工具函数
|
||||
│ │ └── utils.go
|
||||
│ └── wechat/ # 微信相关功能实现
|
||||
│ └── access_articles.go
|
||||
├── data/ # 数据存储目录
|
||||
├── cookie.txt # Cookie文件(需要手动创建)
|
||||
├── go.mod # Go模块定义
|
||||
├── run.bat # Windows启动脚本
|
||||
└── README.md # 使用说明
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
1. 使用本工具前,请确保您已获得相关公众号的访问权限
|
||||
2. 请遵守相关法律法规,合理使用本工具
|
||||
3. 频繁请求可能会触发微信的反爬虫机制,请控制爬取频率
|
||||
4. 由于微信接口可能会变化,工具可能需要相应调整
|
||||
|
||||
## 常见问题
|
||||
|
||||
### Q: 获取Cookie失败怎么办?
|
||||
A: 请确保您已登录微信公众平台,并且在开发者工具中正确复制了完整的Cookie信息。
|
||||
|
||||
### Q: 爬取过程中出现网络错误怎么办?
|
||||
A: 工具会自动处理简单的网络错误,请确保网络连接正常。如果持续失败,可能是微信接口发生了变化。
|
||||
|
||||
### Q: 如何修改爬取的公众号?
|
||||
A: 工具会自动从Cookie中获取当前登录用户可访问的公众号信息。如果需要爬取不同的公众号,请在微信公众平台中切换账号后重新获取Cookie。
|
||||
|
||||
## 许可证
|
||||
|
||||
本项目仅供学习和研究使用。
|
||||
460
backend/api/API接口文档.md
Normal file
460
backend/api/API接口文档.md
Normal file
@@ -0,0 +1,460 @@
|
||||
# 📡 微信公众号文章爬虫 - API 接口文档
|
||||
|
||||
## 服务器信息
|
||||
|
||||
- **服务地址**: http://localhost:8080
|
||||
- **协议**: HTTP/1.1
|
||||
- **数据格式**: JSON
|
||||
- **字符编码**: UTF-8
|
||||
- **CORS**: 已启用(允许所有来源)
|
||||
|
||||
## 统一响应格式
|
||||
|
||||
所有API接口返回格式统一为:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true, // 请求是否成功
|
||||
"message": "操作成功", // 提示信息
|
||||
"data": {} // 数据内容(可选)
|
||||
}
|
||||
```
|
||||
|
||||
## 接口列表
|
||||
|
||||
### 1. 提取公众号主页
|
||||
|
||||
**接口地址**: `/api/homepage/extract`
|
||||
**请求方法**: POST
|
||||
**功能说明**: 从文章链接中提取公众号主页链接
|
||||
|
||||
#### 请求参数
|
||||
|
||||
```json
|
||||
{
|
||||
"url": "https://mp.weixin.qq.com/s?__biz=xxx&mid=xxx"
|
||||
}
|
||||
```
|
||||
|
||||
| 参数 | 类型 | 必填 | 说明 |
|
||||
|------|------|------|------|
|
||||
| url | string | 是 | 公众号文章链接 |
|
||||
|
||||
#### 响应示例
|
||||
|
||||
**成功响应**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "提取成功",
|
||||
"data": {
|
||||
"homepage": "https://mp.weixin.qq.com/mp/profile_ext?action=home&__biz=xxx&scene=124",
|
||||
"output": "完整的命令行输出信息"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**失败响应**:
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"message": "未能提取到主页链接"
|
||||
}
|
||||
```
|
||||
|
||||
#### 调用示例
|
||||
|
||||
**jQuery**:
|
||||
```javascript
|
||||
$.ajax({
|
||||
url: 'http://localhost:8080/api/homepage/extract',
|
||||
method: 'POST',
|
||||
contentType: 'application/json',
|
||||
data: JSON.stringify({
|
||||
url: 'https://mp.weixin.qq.com/s?__biz=xxx&mid=xxx'
|
||||
}),
|
||||
success: function(response) {
|
||||
if (response.success) {
|
||||
console.log('主页链接:', response.data.homepage);
|
||||
}
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**curl**:
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/api/homepage/extract \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"url":"https://mp.weixin.qq.com/s?__biz=xxx&mid=xxx"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. 下载单篇文章
|
||||
|
||||
**接口地址**: `/api/article/download`
|
||||
**请求方法**: POST
|
||||
**功能说明**: 下载指定的单篇文章
|
||||
|
||||
#### 请求参数
|
||||
|
||||
```json
|
||||
{
|
||||
"url": "https://mp.weixin.qq.com/s?__biz=xxx",
|
||||
"save_image": true,
|
||||
"save_content": true
|
||||
}
|
||||
```
|
||||
|
||||
| 参数 | 类型 | 必填 | 说明 |
|
||||
|------|------|------|------|
|
||||
| url | string | 是 | 文章链接 |
|
||||
| save_image | boolean | 否 | 是否保存图片(默认false) |
|
||||
| save_content | boolean | 否 | 是否保存内容(默认true) |
|
||||
|
||||
#### 响应示例
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "下载任务已启动",
|
||||
"data": {
|
||||
"url": "https://mp.weixin.qq.com/s?__biz=xxx"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. 获取文章列表
|
||||
|
||||
**接口地址**: `/api/article/list`
|
||||
**请求方法**: POST
|
||||
**功能说明**: 批量获取公众号的文章列表
|
||||
|
||||
#### 请求参数
|
||||
|
||||
```json
|
||||
{
|
||||
"access_token": "https://mp.weixin.qq.com/mp/profile_ext?action=xxx&appmsg_token=xxx",
|
||||
"pages": 0
|
||||
}
|
||||
```
|
||||
|
||||
| 参数 | 类型 | 必填 | 说明 |
|
||||
|------|------|------|------|
|
||||
| access_token | string | 是 | 包含appmsg_token的URL |
|
||||
| pages | integer | 否 | 获取页数,0表示全部(默认0) |
|
||||
|
||||
#### 响应示例
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "任务已启动"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. 批量下载文章
|
||||
|
||||
**接口地址**: `/api/article/batch`
|
||||
**请求方法**: POST
|
||||
**功能说明**: 批量下载公众号的所有文章
|
||||
|
||||
#### 请求参数
|
||||
|
||||
```json
|
||||
{
|
||||
"official_account": "公众号名称或文章链接",
|
||||
"save_image": true,
|
||||
"save_content": true
|
||||
}
|
||||
```
|
||||
|
||||
| 参数 | 类型 | 必填 | 说明 |
|
||||
|------|------|------|------|
|
||||
| official_account | string | 是 | 公众号名称或任意文章链接 |
|
||||
| save_image | boolean | 否 | 是否保存图片(默认false) |
|
||||
| save_content | boolean | 否 | 是否保存内容(默认true) |
|
||||
|
||||
#### 响应示例
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "任务已启动"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. 获取数据列表
|
||||
|
||||
**接口地址**: `/api/data/list`
|
||||
**请求方法**: GET
|
||||
**功能说明**: 获取已下载的公众号数据列表
|
||||
|
||||
#### 请求参数
|
||||
|
||||
无
|
||||
|
||||
#### 响应示例
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": [
|
||||
{
|
||||
"name": "研招网资讯",
|
||||
"article_count": 125,
|
||||
"path": "D:\\workspace\\Access_wechat_article\\backend\\data\\研招网资讯",
|
||||
"last_update": "2025-11-27"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| name | string | 公众号名称 |
|
||||
| article_count | integer | 文章数量 |
|
||||
| path | string | 存储路径 |
|
||||
| last_update | string | 最后更新时间 |
|
||||
|
||||
#### 调用示例
|
||||
|
||||
**jQuery**:
|
||||
```javascript
|
||||
$.get('http://localhost:8080/api/data/list', function(response) {
|
||||
if (response.success) {
|
||||
console.log('数据列表:', response.data);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**curl**:
|
||||
```bash
|
||||
curl http://localhost:8080/api/data/list
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. 获取任务状态
|
||||
|
||||
**接口地址**: `/api/task/status`
|
||||
**请求方法**: GET
|
||||
**功能说明**: 获取当前任务的执行状态
|
||||
|
||||
#### 请求参数
|
||||
|
||||
无
|
||||
|
||||
#### 响应示例
|
||||
|
||||
**任务运行中**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"running": true,
|
||||
"progress": 45,
|
||||
"message": "正在下载第10篇文章..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**无任务运行**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"running": false,
|
||||
"progress": 0,
|
||||
"message": ""
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| running | boolean | 是否有任务运行中 |
|
||||
| progress | integer | 任务进度(0-100) |
|
||||
| message | string | 任务状态描述 |
|
||||
| error | string | 错误信息(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 错误码说明
|
||||
|
||||
### HTTP状态码
|
||||
|
||||
| 状态码 | 说明 |
|
||||
|--------|------|
|
||||
| 200 | 请求成功 |
|
||||
| 400 | 请求参数错误 |
|
||||
| 500 | 服务器内部错误 |
|
||||
|
||||
### 业务错误码
|
||||
|
||||
所有业务错误通过响应中的 `success` 字段和 `message` 字段返回:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"message": "具体的错误信息"
|
||||
}
|
||||
```
|
||||
|
||||
常见错误信息:
|
||||
|
||||
| 错误信息 | 说明 | 解决方法 |
|
||||
|----------|------|----------|
|
||||
| 请求参数错误 | JSON格式不正确或缺少必填参数 | 检查请求参数格式 |
|
||||
| 执行失败 | 后端程序执行出错 | 查看详细错误信息 |
|
||||
| 未能提取到主页链接 | 文章链接格式错误或解析失败 | 使用有效的文章链接 |
|
||||
| 读取数据目录失败 | data目录不存在或无权限 | 检查目录权限 |
|
||||
|
||||
---
|
||||
|
||||
## 开发指南
|
||||
|
||||
### 本地测试
|
||||
|
||||
1. **启动API服务器**:
|
||||
```bash
|
||||
cd backend\api
|
||||
start_api.bat
|
||||
```
|
||||
|
||||
2. **测试接口**:
|
||||
```bash
|
||||
# 测试提取主页
|
||||
curl -X POST http://localhost:8080/api/homepage/extract \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"url\":\"文章链接\"}"
|
||||
|
||||
# 测试获取数据列表
|
||||
curl http://localhost:8080/api/data/list
|
||||
```
|
||||
|
||||
### 跨域配置
|
||||
|
||||
API服务器已启用CORS,允许所有来源访问:
|
||||
|
||||
```go
|
||||
w.Header().Set("Access-Control-Allow-Origin", "*")
|
||||
w.Header().Set("Access-Control-Allow-Methods", "GET, POST, OPTIONS")
|
||||
w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
|
||||
```
|
||||
|
||||
如需限制特定域名,修改 `server.go` 中的 `corsMiddleware` 函数。
|
||||
|
||||
### 超时设置
|
||||
|
||||
默认HTTP超时时间:30秒
|
||||
|
||||
如需修改,在 `server.go` 中添加:
|
||||
|
||||
```go
|
||||
server := &http.Server{
|
||||
Addr: ":8080",
|
||||
ReadTimeout: 30 * time.Second,
|
||||
WriteTimeout: 30 * time.Second,
|
||||
}
|
||||
```
|
||||
|
||||
### 日志记录
|
||||
|
||||
API服务器使用标准输出记录日志:
|
||||
|
||||
```go
|
||||
log.Printf("[%s] %s - %s", r.Method, r.URL.Path, message)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 接口更新计划
|
||||
|
||||
### v1.1.0(计划中)
|
||||
- [ ] 添加用户认证机制
|
||||
- [ ] 支持任务队列管理
|
||||
- [ ] 增加下载进度推送(WebSocket)
|
||||
- [ ] 提供文章搜索接口
|
||||
|
||||
### v1.2.0(计划中)
|
||||
- [ ] 数据统计分析接口
|
||||
- [ ] 导出功能(PDF/Word)
|
||||
- [ ] 批量任务管理
|
||||
- [ ] 定时任务支持
|
||||
|
||||
---
|
||||
|
||||
## 技术栈
|
||||
|
||||
- **语言**: Go 1.20+
|
||||
- **Web框架**: net/http (标准库)
|
||||
- **数据格式**: JSON
|
||||
- **并发模型**: Goroutine
|
||||
|
||||
---
|
||||
|
||||
## 性能说明
|
||||
|
||||
### 并发能力
|
||||
- 支持多客户端同时访问
|
||||
- 但同一时间只能执行一个爬虫任务(`currentTask`)
|
||||
|
||||
### 资源占用
|
||||
- CPU: 低(主要I/O操作)
|
||||
- 内存: <50MB
|
||||
- 磁盘: 取决于下载的文章数量
|
||||
|
||||
### 性能优化建议
|
||||
1. 使用连接池管理HTTP请求
|
||||
2. 实现任务队列机制
|
||||
3. 添加结果缓存
|
||||
4. 启用gzip压缩
|
||||
|
||||
---
|
||||
|
||||
## 安全建议
|
||||
|
||||
### 1. 生产环境部署
|
||||
- 添加HTTPS支持
|
||||
- 实现API认证(JWT/OAuth)
|
||||
- 限制跨域来源
|
||||
- 添加请求频率限制
|
||||
|
||||
### 2. 数据安全
|
||||
- 不要暴露敏感信息(Cookie)
|
||||
- 定期清理临时文件
|
||||
- 备份重要数据
|
||||
|
||||
### 3. 访问控制
|
||||
- 添加IP白名单
|
||||
- 实现用户权限管理
|
||||
- 记录操作日志
|
||||
|
||||
---
|
||||
|
||||
## 常见问题
|
||||
|
||||
### Q1: 为什么任务启动后没有响应?
|
||||
A: 检查后端 `wechat-crawler.exe` 是否存在并有执行权限。
|
||||
|
||||
### Q2: 如何查看详细的错误信息?
|
||||
A: 查看API服务器窗口的控制台输出。
|
||||
|
||||
### Q3: 能同时执行多个下载任务吗?
|
||||
A: 当前版本不支持,同时只能执行一个任务。
|
||||
|
||||
### Q4: 如何停止正在运行的任务?
|
||||
A: 关闭API服务器窗口或重启服务器。
|
||||
|
||||
---
|
||||
|
||||
**文档版本**: v1.0.0
|
||||
**最后更新**: 2025-11-27
|
||||
**维护者**: AI Assistant
|
||||
26
backend/api/build.bat
Normal file
26
backend/api/build.bat
Normal file
@@ -0,0 +1,26 @@
|
||||
@echo off
|
||||
chcp 65001 >nul
|
||||
echo ===============================================
|
||||
echo 📦 编译 API 服务器
|
||||
echo ===============================================
|
||||
echo.
|
||||
|
||||
echo 🔨 正在编译 api_server.exe...
|
||||
go build -o api_server.exe server.go
|
||||
|
||||
if %errorlevel% neq 0 (
|
||||
echo.
|
||||
echo ❌ 编译失败!
|
||||
echo.
|
||||
pause
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
echo.
|
||||
echo ✅ 编译成功!
|
||||
echo 📁 输出文件: api_server.exe
|
||||
echo.
|
||||
echo ===============================================
|
||||
echo 编译完成
|
||||
echo ===============================================
|
||||
pause
|
||||
543
backend/api/server.go
Normal file
543
backend/api/server.go
Normal file
@@ -0,0 +1,543 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Response 统一响应结构
|
||||
type Response struct {
|
||||
Success bool `json:"success"`
|
||||
Message string `json:"message"`
|
||||
Data interface{} `json:"data,omitempty"`
|
||||
}
|
||||
|
||||
// 任务状态
|
||||
type TaskStatus struct {
|
||||
Running bool `json:"running"`
|
||||
Progress int `json:"progress"`
|
||||
Message string `json:"message"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
var currentTask = &TaskStatus{Running: false}
|
||||
|
||||
func main() {
|
||||
// 启用CORS
|
||||
http.HandleFunc("/", corsMiddleware(handleRoot))
|
||||
http.HandleFunc("/api/homepage/extract", corsMiddleware(extractHomepageHandler))
|
||||
http.HandleFunc("/api/article/download", corsMiddleware(downloadArticleHandler))
|
||||
http.HandleFunc("/api/article/list", corsMiddleware(getArticleListHandler))
|
||||
http.HandleFunc("/api/article/batch", corsMiddleware(batchDownloadHandler))
|
||||
http.HandleFunc("/api/data/list", corsMiddleware(getDataListHandler))
|
||||
http.HandleFunc("/api/task/status", corsMiddleware(getTaskStatusHandler))
|
||||
http.HandleFunc("/api/download/", corsMiddleware(downloadFileHandler))
|
||||
|
||||
port := ":8080"
|
||||
fmt.Println("===============================================")
|
||||
fmt.Println(" 🚀 微信公众号文章爬虫 API 服务器")
|
||||
fmt.Println("===============================================")
|
||||
fmt.Printf("🌐 服务地址: http://localhost%s\n", port)
|
||||
fmt.Printf("⏰ 启动时间: %s\n", time.Now().Format("2006-01-02 15:04:05"))
|
||||
fmt.Println("===============================================\n")
|
||||
|
||||
if err := http.ListenAndServe(port, nil); err != nil {
|
||||
log.Fatal("服务器启动失败:", err)
|
||||
}
|
||||
}
|
||||
|
||||
// CORS中间件
|
||||
func corsMiddleware(next http.HandlerFunc) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Access-Control-Allow-Origin", "*")
|
||||
w.Header().Set("Access-Control-Allow-Methods", "GET, POST, OPTIONS")
|
||||
w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
|
||||
|
||||
if r.Method == "OPTIONS" {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
next(w, r)
|
||||
}
|
||||
}
|
||||
|
||||
// 首页处理
|
||||
func handleRoot(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
html := `
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<title>微信公众号文章爬虫 API</title>
|
||||
<style>
|
||||
body { font-family: Arial, sans-serif; max-width: 800px; margin: 50px auto; padding: 20px; }
|
||||
h1 { color: #333; }
|
||||
.endpoint { background: #f5f5f5; padding: 10px; margin: 10px 0; border-radius: 5px; }
|
||||
.method { color: #4CAF50; font-weight: bold; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>🚀 微信公众号文章爬虫 API 服务器</h1>
|
||||
<p>当前时间: ` + time.Now().Format("2006-01-02 15:04:05") + `</p>
|
||||
<h2>可用接口:</h2>
|
||||
<div class="endpoint">
|
||||
<span class="method">POST</span> /api/homepage/extract - 提取公众号主页
|
||||
</div>
|
||||
<div class="endpoint">
|
||||
<span class="method">POST</span> /api/article/download - 下载单篇文章
|
||||
</div>
|
||||
<div class="endpoint">
|
||||
<span class="method">POST</span> /api/article/list - 获取文章列表
|
||||
</div>
|
||||
<div class="endpoint">
|
||||
<span class="method">POST</span> /api/article/batch - 批量下载文章
|
||||
</div>
|
||||
<div class="endpoint">
|
||||
<span class="method">GET</span> /api/data/list - 获取数据列表
|
||||
</div>
|
||||
<div class="endpoint">
|
||||
<span class="method">GET</span> /api/task/status - 获取任务状态
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
`
|
||||
w.Write([]byte(html))
|
||||
}
|
||||
|
||||
// 提取公众号主页
|
||||
func extractHomepageHandler(w http.ResponseWriter, r *http.Request) {
|
||||
var req struct {
|
||||
URL string `json:"url"`
|
||||
}
|
||||
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeJSON(w, Response{Success: false, Message: "请求参数错误"})
|
||||
return
|
||||
}
|
||||
|
||||
// 执行命令(使用绝对路径)
|
||||
exePath := filepath.Join("..", "wechat-crawler.exe")
|
||||
absPath, _ := filepath.Abs(exePath)
|
||||
log.Printf("尝试执行: %s", absPath)
|
||||
|
||||
cmd := exec.Command(absPath, req.URL)
|
||||
workDir, _ := filepath.Abs("..")
|
||||
cmd.Dir = workDir
|
||||
output, err := cmd.CombinedOutput()
|
||||
|
||||
if err != nil {
|
||||
log.Printf("执行失败: %v, 输出: %s", err, string(output))
|
||||
writeJSON(w, Response{Success: false, Message: "执行失败: " + string(output)})
|
||||
return
|
||||
}
|
||||
|
||||
// 从输出中提取公众号主页链接
|
||||
outputStr := string(output)
|
||||
lines := strings.Split(outputStr, "\n")
|
||||
var homepageURL string
|
||||
|
||||
for _, line := range lines {
|
||||
if strings.Contains(line, "公众号主页链接") || strings.Contains(line, "https://mp.weixin.qq.com/mp/profile_ext") {
|
||||
// 提取URL
|
||||
if idx := strings.Index(line, "https://"); idx != -1 {
|
||||
homepageURL = strings.TrimSpace(line[idx:])
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if homepageURL == "" {
|
||||
writeJSON(w, Response{Success: false, Message: "未能提取到主页链接"})
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, Response{
|
||||
Success: true,
|
||||
Message: "提取成功",
|
||||
Data: map[string]string{
|
||||
"homepage": homepageURL,
|
||||
"output": outputStr,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// 下载单篇文章(这里需要实现具体逻辑)
|
||||
func downloadArticleHandler(w http.ResponseWriter, r *http.Request) {
|
||||
var req struct {
|
||||
URL string `json:"url"`
|
||||
SaveImage bool `json:"save_image"`
|
||||
SaveContent bool `json:"save_content"`
|
||||
}
|
||||
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeJSON(w, Response{Success: false, Message: "请求参数错误"})
|
||||
return
|
||||
}
|
||||
|
||||
currentTask.Running = true
|
||||
currentTask.Progress = 0
|
||||
currentTask.Message = "正在下载文章..."
|
||||
|
||||
// 注意:这里需要实际调用爬虫的下载功能
|
||||
// 由于当前后端程序没有单独的下载单篇文章的命令行接口
|
||||
// 需要后续实现或使用其他方式
|
||||
|
||||
writeJSON(w, Response{
|
||||
Success: true,
|
||||
Message: "下载任务已启动",
|
||||
Data: map[string]interface{}{
|
||||
"url": req.URL,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// 获取文章列表
|
||||
func getArticleListHandler(w http.ResponseWriter, r *http.Request) {
|
||||
var req struct {
|
||||
AccessToken string `json:"access_token"`
|
||||
Pages int `json:"pages"`
|
||||
}
|
||||
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeJSON(w, Response{Success: false, Message: "请求参数错误"})
|
||||
return
|
||||
}
|
||||
|
||||
currentTask.Running = true
|
||||
currentTask.Progress = 0
|
||||
currentTask.Message = "正在获取文章列表..."
|
||||
|
||||
// 同步执行爬虫程序(功能3)
|
||||
exePath := filepath.Join("..", "wechat-crawler.exe")
|
||||
absPath, _ := filepath.Abs(exePath)
|
||||
workDir, _ := filepath.Abs("..")
|
||||
|
||||
log.Printf("启动功能3: %s, 工作目录: %s", absPath, workDir)
|
||||
cmd := exec.Command(absPath)
|
||||
cmd.Dir = workDir
|
||||
|
||||
// 创建输入管道
|
||||
stdin, err := cmd.StdinPipe()
|
||||
if err != nil {
|
||||
log.Printf("创建输入管道失败: %v", err)
|
||||
currentTask.Running = false
|
||||
writeJSON(w, Response{Success: false, Message: "创建输入管道失败: " + err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
// 启动命令
|
||||
if err := cmd.Start(); err != nil {
|
||||
log.Printf("启动命令失败: %v", err)
|
||||
currentTask.Running = false
|
||||
writeJSON(w, Response{Success: false, Message: "启动命令失败: " + err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
// 发送选项"3"(功能3:通过access_token获取文章列表)
|
||||
fmt.Fprintln(stdin, "3")
|
||||
fmt.Fprintln(stdin, req.AccessToken)
|
||||
if req.Pages > 0 {
|
||||
fmt.Fprintf(stdin, "%d\n", req.Pages)
|
||||
} else {
|
||||
fmt.Fprintln(stdin, "0")
|
||||
}
|
||||
stdin.Close()
|
||||
|
||||
// 等待命令完成
|
||||
if err := cmd.Wait(); err != nil {
|
||||
log.Printf("命令执行失败: %v", err)
|
||||
currentTask.Running = false
|
||||
writeJSON(w, Response{Success: false, Message: "命令执行失败: " + err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
currentTask.Running = false
|
||||
currentTask.Progress = 100
|
||||
currentTask.Message = "文章列表获取完成"
|
||||
|
||||
// 查找生成的文件并返回下载链接
|
||||
dataDir := "../data"
|
||||
entries, err := os.ReadDir(dataDir)
|
||||
if err != nil {
|
||||
writeJSON(w, Response{Success: false, Message: "读取数据目录失败: " + err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
// 查找最新创建的公众号目录
|
||||
var latestDir string
|
||||
var latestTime time.Time
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() && entry.Name() != "." && entry.Name() != ".." {
|
||||
info, _ := entry.Info()
|
||||
if info.ModTime().After(latestTime) {
|
||||
latestTime = info.ModTime()
|
||||
latestDir = entry.Name()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if latestDir == "" {
|
||||
writeJSON(w, Response{Success: false, Message: "未找到生成的数据目录"})
|
||||
return
|
||||
}
|
||||
|
||||
log.Printf("找到最新目录: %s", latestDir)
|
||||
|
||||
// 查找文章列表文件(优先查找直连链接文件)
|
||||
accountPath := filepath.Join(dataDir, latestDir)
|
||||
files, err := os.ReadDir(accountPath)
|
||||
if err != nil {
|
||||
writeJSON(w, Response{Success: false, Message: "读取公众号目录失败: " + err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
var excelFile string
|
||||
// 优先查找直连链接文件(.xlsx或.txt)
|
||||
for _, file := range files {
|
||||
if !file.IsDir() && strings.Contains(file.Name(), "直连链接") {
|
||||
if strings.HasSuffix(file.Name(), ".xlsx") || strings.HasSuffix(file.Name(), ".txt") {
|
||||
excelFile = file.Name()
|
||||
log.Printf("找到直连链接文件: %s", excelFile)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 如果没有直连链接文件,查找原始链接文件
|
||||
if excelFile == "" {
|
||||
for _, file := range files {
|
||||
if !file.IsDir() && strings.Contains(file.Name(), "原始链接") {
|
||||
if strings.HasSuffix(file.Name(), ".xlsx") || strings.HasSuffix(file.Name(), ".txt") {
|
||||
excelFile = file.Name()
|
||||
log.Printf("找到原始链接文件: %s", excelFile)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 如果还是没有,查找任何文章列表文件
|
||||
if excelFile == "" {
|
||||
for _, file := range files {
|
||||
if !file.IsDir() && strings.Contains(file.Name(), "文章列表") {
|
||||
if strings.HasSuffix(file.Name(), ".xlsx") || strings.HasSuffix(file.Name(), ".txt") {
|
||||
excelFile = file.Name()
|
||||
log.Printf("找到文章列表文件: %s", excelFile)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if excelFile == "" {
|
||||
// 列出所有文件用于调试
|
||||
var fileList []string
|
||||
for _, file := range files {
|
||||
fileList = append(fileList, file.Name())
|
||||
}
|
||||
log.Printf("目录 %s 中的文件: %v", latestDir, fileList)
|
||||
writeJSON(w, Response{Success: false, Message: "未找到Excel文件,目录中的文件: " + strings.Join(fileList, ", ")})
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, Response{
|
||||
Success: true,
|
||||
Message: "文章列表获取成功",
|
||||
Data: map[string]interface{}{
|
||||
"account": latestDir,
|
||||
"filename": excelFile,
|
||||
"download": fmt.Sprintf("/download/%s/%s", latestDir, excelFile),
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// 批量下载文章
|
||||
func batchDownloadHandler(w http.ResponseWriter, r *http.Request) {
|
||||
var req struct {
|
||||
OfficialAccount string `json:"official_account"`
|
||||
SaveImage bool `json:"save_image"`
|
||||
SaveContent bool `json:"save_content"`
|
||||
}
|
||||
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeJSON(w, Response{Success: false, Message: "请求参数错误"})
|
||||
return
|
||||
}
|
||||
|
||||
currentTask.Running = true
|
||||
currentTask.Progress = 0
|
||||
currentTask.Message = "正在批量下载文章..."
|
||||
|
||||
// 同步执行爬虫程序(功能5)
|
||||
exePath := filepath.Join("..", "wechat-crawler.exe")
|
||||
absPath, _ := filepath.Abs(exePath)
|
||||
workDir, _ := filepath.Abs("..")
|
||||
|
||||
log.Printf("启动功能5: %s, 工作目录: %s", absPath, workDir)
|
||||
cmd := exec.Command(absPath)
|
||||
cmd.Dir = workDir
|
||||
|
||||
// 创建输入管道
|
||||
stdin, err := cmd.StdinPipe()
|
||||
if err != nil {
|
||||
log.Printf("创建输入管道失败: %v", err)
|
||||
currentTask.Running = false
|
||||
writeJSON(w, Response{Success: false, Message: "创建输入管道失败: " + err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
// 启动命令
|
||||
if err := cmd.Start(); err != nil {
|
||||
log.Printf("启动命令失败: %v", err)
|
||||
currentTask.Running = false
|
||||
writeJSON(w, Response{Success: false, Message: "启动命令失败: " + err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
// 发送选项"5"(功能5:批量下载)
|
||||
fmt.Fprintln(stdin, "5")
|
||||
fmt.Fprintln(stdin, req.OfficialAccount)
|
||||
|
||||
// 是否保存图片
|
||||
if req.SaveImage {
|
||||
fmt.Fprintln(stdin, "y")
|
||||
} else {
|
||||
fmt.Fprintln(stdin, "n")
|
||||
}
|
||||
stdin.Close()
|
||||
|
||||
// 等待命令完成
|
||||
if err := cmd.Wait(); err != nil {
|
||||
log.Printf("命令执行失败: %v", err)
|
||||
currentTask.Running = false
|
||||
writeJSON(w, Response{Success: false, Message: "命令执行失败: " + err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
currentTask.Running = false
|
||||
currentTask.Progress = 100
|
||||
currentTask.Message = "批量下载完成"
|
||||
|
||||
// 统计下载的文章数量
|
||||
accountPath := filepath.Join("../data", req.OfficialAccount, "文章详细")
|
||||
var articleCount int
|
||||
if entries, err := os.ReadDir(accountPath); err == nil {
|
||||
articleCount = len(entries)
|
||||
}
|
||||
|
||||
writeJSON(w, Response{
|
||||
Success: true,
|
||||
Message: fmt.Sprintf("批量下载完成,共下载 %d 篇文章", articleCount),
|
||||
Data: map[string]interface{}{
|
||||
"account": req.OfficialAccount,
|
||||
"articleCount": articleCount,
|
||||
"path": accountPath,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// 获取数据列表
|
||||
func getDataListHandler(w http.ResponseWriter, r *http.Request) {
|
||||
dataDir := "../data"
|
||||
var accounts []map[string]interface{}
|
||||
|
||||
entries, err := os.ReadDir(dataDir)
|
||||
if err != nil {
|
||||
// 如果目录不存在,返回空列表而不是错误
|
||||
writeJSON(w, Response{
|
||||
Success: true,
|
||||
Data: accounts,
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() {
|
||||
accountPath := filepath.Join(dataDir, entry.Name())
|
||||
|
||||
// 统计文章数量
|
||||
detailPath := filepath.Join(accountPath, "文章详细")
|
||||
var articleCount int
|
||||
if detailEntries, err := os.ReadDir(detailPath); err == nil {
|
||||
articleCount = len(detailEntries)
|
||||
}
|
||||
|
||||
// 获取最后更新时间
|
||||
info, _ := entry.Info()
|
||||
lastUpdate := info.ModTime().Format("2006-01-02")
|
||||
|
||||
accounts = append(accounts, map[string]interface{}{
|
||||
"name": entry.Name(),
|
||||
"articleCount": articleCount,
|
||||
"path": accountPath,
|
||||
"lastUpdate": lastUpdate,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
writeJSON(w, Response{
|
||||
Success: true,
|
||||
Data: accounts,
|
||||
})
|
||||
}
|
||||
|
||||
// 获取任务状态
|
||||
func getTaskStatusHandler(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, Response{
|
||||
Success: true,
|
||||
Data: currentTask,
|
||||
})
|
||||
}
|
||||
|
||||
// 下载文件处理
|
||||
func downloadFileHandler(w http.ResponseWriter, r *http.Request) {
|
||||
// 从 URL 中提取路径 /api/download/公众号名称/文件名
|
||||
path := strings.TrimPrefix(r.URL.Path, "/api/download/")
|
||||
parts := strings.SplitN(path, "/", 2)
|
||||
|
||||
if len(parts) != 2 {
|
||||
http.Error(w, "路径错误", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
accountName := parts[0]
|
||||
filename := parts[1]
|
||||
|
||||
// 构建完整文件路径
|
||||
filePath := filepath.Join("..", "data", accountName, filename)
|
||||
absPath, _ := filepath.Abs(filePath)
|
||||
|
||||
// 检查文件是否存在
|
||||
if _, err := os.Stat(absPath); os.IsNotExist(err) {
|
||||
http.Error(w, "文件不存在", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
log.Printf("下载文件: %s", absPath)
|
||||
|
||||
// 设置响应头
|
||||
contentType := "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
|
||||
if strings.HasSuffix(filename, ".txt") {
|
||||
contentType = "text/plain; charset=utf-8"
|
||||
}
|
||||
w.Header().Set("Content-Type", contentType)
|
||||
w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename*=UTF-8''%s", filename))
|
||||
|
||||
// 发送文件
|
||||
http.ServeFile(w, r, absPath)
|
||||
}
|
||||
|
||||
// 写入JSON响应
|
||||
func writeJSON(w http.ResponseWriter, data interface{}) {
|
||||
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
||||
json.NewEncoder(w).Encode(data)
|
||||
}
|
||||
23
backend/api/start_api.bat
Normal file
23
backend/api/start_api.bat
Normal file
@@ -0,0 +1,23 @@
|
||||
@echo off
|
||||
chcp 65001 >nul
|
||||
title 微信公众号文章爬虫 - API服务器
|
||||
|
||||
:: 检查api_server.exe是否存在
|
||||
if not exist "api_server.exe" (
|
||||
echo ===============================================
|
||||
echo ⚠️ API服务器未编译
|
||||
echo ===============================================
|
||||
echo.
|
||||
echo 正在编译 API 服务器...
|
||||
echo.
|
||||
call build.bat
|
||||
if %errorlevel% neq 0 (
|
||||
echo 编译失败,无法启动服务器
|
||||
pause
|
||||
exit /b 1
|
||||
)
|
||||
)
|
||||
|
||||
:: 启动API服务器
|
||||
cls
|
||||
api_server.exe
|
||||
11
backend/cmd/data/研招网资讯/文章列表(article_list)_直连链接.txt
Normal file
11
backend/cmd/data/研招网资讯/文章列表(article_list)_直连链接.txt
Normal file
@@ -0,0 +1,11 @@
|
||||
序号,创建时间,标题,链接
|
||||
1,0,专家分析2026年考研报名人数,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500657&idx=1&sn=81eae7df4bfa2fdfc8bca69389489c52&chksm=ea981e22044aca6bbe5633849bfcd4903cb6f491646cd2ccf9321f4d9c852c64fe036f033c14&scene=27#wechat_redirect
|
||||
2,0,教育部:2026年全国硕士研究生报名人数为343万,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500650&idx=1&sn=9f230bbfefb24d98c18e42bd3651ad53&chksm=eac72972d56ff9b66f3658f0c3b1e6e363e56ddf879d56aba9c9c8f587b53ef00bcabe7992ff&scene=27#wechat_redirect
|
||||
3,0,【小研来了】“务必再坚持坚持”,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500645&idx=1&sn=8e1d5921861dc4e3647f7bf8adaada81&chksm=ea26b17ce2f7255aacd9d1d6358c9aeb8d4e043c692efb8b4d8183cfc8363b3068be79d585c2&scene=27#wechat_redirect
|
||||
4,0,学累了不?点进来看看这4个“续航”方法,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500631&idx=1&sn=b640b0e43378e368166e50a7f46735f2&chksm=ea71f10a83b7811e1896cd9704eac5d064b763f3e020b5b37c72727c55bb1b0862a92e9c4cf0&scene=27#wechat_redirect
|
||||
5,0,教育部:在“双一流”建设高校开展科技教育硕士培养,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500589&idx=1&sn=539d1229c9475ba5a2371698a362e9a7&chksm=ea4f97d3831139a276e50050f2f3307868b9c6ec7eb115bb9e288312f08572c47128a8016dce&scene=27#wechat_redirect
|
||||
6,0,“研味儿”正浓,冲刺在即!请你一定别放弃,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500584&idx=1&sn=294b6ba8d12f0948913abf04af8cb188&chksm=ea4cfb5b16684bdd12634b6e46d8d8f3ab72ca9108be0d4d7f83dfded09c6ecb9f31b1531e31&scene=27#wechat_redirect
|
||||
7,0,4个思维升级,让我找回了读研的掌控感,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500579&idx=1&sn=fa00084c8711e3009ff7e31fe0b3bc51&chksm=eaff1ec212ddbb738d20542a965bbd1b79ae3a9d2e5af5704ddcf41de3a8b8d658e562771f0c&scene=27#wechat_redirect
|
||||
8,0,研考网上确认成功后,需重点关注四件事,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500569&idx=1&sn=7707b698932ff6847de39d7351d3ac98&chksm=ea402eec6a96125a5bb02600aff24c3c1211eb5aaf5347080bbfe1f5861e9ca97fe9c400df21&scene=27#wechat_redirect
|
||||
9,0,,
|
||||
10,0,【小研来了】“小研,没有准考证照片怎么办?”,http://mp.weixin.qq.com/s?__biz=MzI3NzQzODQ5OA==&mid=2247500553&idx=1&sn=4fc6fd69684f02222e72d457c1004a81&chksm=eafc91ea346080790f9b641495fc3d9e31302ee5c2c9957eb4fa2bc9a139eda78163899b9219&scene=27#wechat_redirect
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"fmt"
|
||||
"io/ioutil"
|
||||
"log"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -600,21 +601,48 @@ func parseAccessTokenParams(accessToken string) (string, string, string, string,
|
||||
if err != nil {
|
||||
return "", "", "", "", fmt.Errorf("未找到__biz参数")
|
||||
}
|
||||
// URL解码biz参数
|
||||
biz, err = url.QueryUnescape(biz)
|
||||
if err != nil {
|
||||
fmt.Printf("警告: URL解码__biz失败: %v,使用原始值\n", err)
|
||||
}
|
||||
|
||||
uin, err := utils.ExtractFromRegex(accessToken, "uin=([^&]*)")
|
||||
if err != nil {
|
||||
return "", "", "", "", fmt.Errorf("未找到uin参数")
|
||||
}
|
||||
// URL解码uin参数
|
||||
uin, err = url.QueryUnescape(uin)
|
||||
if err != nil {
|
||||
fmt.Printf("警告: URL解码uin失败: %v,使用原始值\n", err)
|
||||
}
|
||||
|
||||
key, err := utils.ExtractFromRegex(accessToken, "key=([^&]*)")
|
||||
if err != nil {
|
||||
return "", "", "", "", fmt.Errorf("未找到key参数")
|
||||
}
|
||||
// URL解码key参数
|
||||
key, err = url.QueryUnescape(key)
|
||||
if err != nil {
|
||||
fmt.Printf("警告: URL解码key失败: %v,使用原始值\n", err)
|
||||
}
|
||||
|
||||
passTicket, err := utils.ExtractFromRegex(accessToken, "pass_ticket=([^&]*)")
|
||||
if err != nil {
|
||||
return "", "", "", "", fmt.Errorf("未找到pass_ticket参数")
|
||||
}
|
||||
// URL解码pass_ticket参数
|
||||
passTicket, err = url.QueryUnescape(passTicket)
|
||||
if err != nil {
|
||||
fmt.Printf("警告: URL解码pass_ticket失败: %v,使用原始值\n", err)
|
||||
}
|
||||
|
||||
// 打印解码后的参数用于调试
|
||||
fmt.Printf("\n提取到的参数(已解码):\n")
|
||||
fmt.Printf(" __biz: %s\n", biz)
|
||||
fmt.Printf(" uin: %s\n", uin)
|
||||
fmt.Printf(" key长度: %d 字符\n", len(key))
|
||||
fmt.Printf(" pass_ticket长度: %d 字符\n", len(passTicket))
|
||||
|
||||
return biz, uin, key, passTicket, nil
|
||||
}
|
||||
|
||||
@@ -1,27 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/wechat-crawler/pkg/wechat"
|
||||
)
|
||||
|
||||
func main() {
|
||||
fmt.Println("开始测试文章内容提取功能...")
|
||||
|
||||
// 创建一个简单的爬虫实例
|
||||
crawler := wechat.NewSimpleCrawler()
|
||||
|
||||
// 设置公众号名称(根据实际情况修改)
|
||||
officialAccountName := "验证"
|
||||
|
||||
// 调用GetListArticleFromFile函数测试
|
||||
err := crawler.GetListArticleFromFile(officialAccountName, false, true)
|
||||
if err != nil {
|
||||
fmt.Printf("测试失败: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
fmt.Println("测试完成!请检查文章内容是否已正确提取。")
|
||||
}
|
||||
@@ -1 +0,0 @@
|
||||
__biz=MzUxMjA4MTI0MjA1; uin=MTIzNDU2Nzg5; key=abcdef1234567890abcdef1234567890; pass_ticket=abcdefghijklmnopqrstuvwxyz1234567890; version=63090b13; wxtype=1; pass_ticket=abcdefghijklmnopqrstuvwxyz1234567890;
|
||||
@@ -1,12 +0,0 @@
|
||||
请将此文件重命名为cookie.txt,并填入微信公众平台的cookie信息
|
||||
|
||||
如何获取cookie:
|
||||
1. 打开浏览器,登录微信公众平台
|
||||
2. 按F12打开开发者工具
|
||||
3. 切换到Network标签
|
||||
4. 刷新页面或访问任意页面
|
||||
5. 选择一个请求,查看Headers中的Cookie
|
||||
6. 复制完整的Cookie到本文件中
|
||||
|
||||
Cookie格式示例:
|
||||
__biz=MzUxMjA4MTI0MjA1; uin=MTIzNDU2Nzg5; key=abcdef1234567890abcdef1234567890; pass_ticket=abcdefghijklmnopqrstuvwxyz1234567890; version=63090b13; wxtype=1; pass_ticket=abcdefghijklmnopqrstuvwxyz1234567890;
|
||||
36624
backend/debug_article_raw.html
Normal file
36624
backend/debug_article_raw.html
Normal file
File diff suppressed because one or more lines are too long
246
backend/examples/database_example.go
Normal file
246
backend/examples/database_example.go
Normal file
@@ -0,0 +1,246 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
"github.com/wechat-crawler/pkg/database"
|
||||
)
|
||||
|
||||
func main() {
|
||||
fmt.Println("==============================================")
|
||||
fmt.Println(" 微信公众号文章数据库管理系统示例")
|
||||
fmt.Println("==============================================\n")
|
||||
|
||||
// 1. 初始化数据库
|
||||
db, err := database.InitDB("../data/wechat_articles.db")
|
||||
if err != nil {
|
||||
log.Fatal("数据库初始化失败:", err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
// 2. 创建仓库实例
|
||||
officialRepo := database.NewOfficialAccountRepository(db)
|
||||
articleRepo := database.NewArticleRepository(db)
|
||||
contentRepo := database.NewArticleContentRepository(db)
|
||||
|
||||
// 3. 示例:添加公众号
|
||||
fmt.Println("📝 示例1: 添加公众号信息")
|
||||
official := &database.OfficialAccount{
|
||||
Biz: "MzI1NjEwMTM4OA==",
|
||||
Nickname: "研招网资讯",
|
||||
Homepage: "https://mp.weixin.qq.com/mp/profile_ext?action=home&__biz=MzI1NjEwMTM4OA==&scene=124",
|
||||
Description: "中国研究生招生信息网官方公众号",
|
||||
}
|
||||
|
||||
// 检查是否已存在
|
||||
existing, err := officialRepo.GetByBiz(official.Biz)
|
||||
if err != nil {
|
||||
log.Fatal("查询公众号失败:", err)
|
||||
}
|
||||
|
||||
var officialID int64
|
||||
if existing == nil {
|
||||
// 不存在,创建新记录
|
||||
officialID, err = officialRepo.Create(official)
|
||||
if err != nil {
|
||||
log.Fatal("创建公众号失败:", err)
|
||||
}
|
||||
fmt.Printf("✅ 成功创建公众号: %s (ID: %d)\n\n", official.Nickname, officialID)
|
||||
} else {
|
||||
// 已存在
|
||||
officialID = existing.ID
|
||||
fmt.Printf("ℹ️ 公众号已存在: %s (ID: %d)\n\n", existing.Nickname, officialID)
|
||||
}
|
||||
|
||||
// 4. 示例:添加文章
|
||||
fmt.Println("📝 示例2: 添加文章信息")
|
||||
article := &database.Article{
|
||||
OfficialID: officialID,
|
||||
Title: "专家分析2026年考研报名人数",
|
||||
Author: "研招网资讯",
|
||||
Link: "https://mp.weixin.qq.com/s?__biz=MzI1NjEwMTM4OA==&mid=2651232405&idx=1",
|
||||
PublishTime: "2024-11-27 10:00:00",
|
||||
CreateTime: "2024-11-27 15:30:00",
|
||||
CommentID: "2247491372",
|
||||
ReadNum: 15234,
|
||||
LikeNum: 456,
|
||||
ShareNum: 123,
|
||||
ContentPreview: "根据最新统计数据显示,2026年全国硕士研究生报名人数预计将达到新高...",
|
||||
ParagraphCount: 15,
|
||||
}
|
||||
|
||||
// 检查文章是否已存在
|
||||
existingArticle, err := articleRepo.GetByLink(article.Link)
|
||||
if err != nil {
|
||||
log.Fatal("查询文章失败:", err)
|
||||
}
|
||||
|
||||
var articleID int64
|
||||
if existingArticle == nil {
|
||||
articleID, err = articleRepo.Create(article)
|
||||
if err != nil {
|
||||
log.Fatal("创建文章失败:", err)
|
||||
}
|
||||
fmt.Printf("✅ 成功创建文章: %s (ID: %d)\n\n", article.Title, articleID)
|
||||
} else {
|
||||
articleID = existingArticle.ID
|
||||
fmt.Printf("ℹ️ 文章已存在: %s (ID: %d)\n\n", existingArticle.Title, articleID)
|
||||
}
|
||||
|
||||
// 5. 示例:添加文章内容
|
||||
fmt.Println("📝 示例3: 添加文章详细内容")
|
||||
|
||||
paragraphs := []string{
|
||||
"根据最新统计数据显示,2026年全国硕士研究生报名人数预计将达到新高。",
|
||||
"教育部相关负责人表示,随着社会对高层次人才需求的增加,考研热度持续上升。",
|
||||
"专家建议考生理性选择,注重提升自身综合素质。",
|
||||
}
|
||||
|
||||
images := []string{
|
||||
"https://mmbiz.qpic.cn/mmbiz_jpg/xxx1.jpg",
|
||||
"https://mmbiz.qpic.cn/mmbiz_jpg/xxx2.jpg",
|
||||
}
|
||||
|
||||
content := &database.ArticleContent{
|
||||
ArticleID: articleID,
|
||||
HtmlContent: "<div>文章HTML内容</div>",
|
||||
TextContent: "文章纯文本内容...",
|
||||
Paragraphs: database.StringsToJSON(paragraphs),
|
||||
Images: database.StringsToJSON(images),
|
||||
}
|
||||
|
||||
// 检查内容是否已存在
|
||||
existingContent, err := contentRepo.GetByArticleID(articleID)
|
||||
if err != nil {
|
||||
log.Fatal("查询文章内容失败:", err)
|
||||
}
|
||||
|
||||
if existingContent == nil {
|
||||
contentID, err := contentRepo.Create(content)
|
||||
if err != nil {
|
||||
log.Fatal("创建文章内容失败:", err)
|
||||
}
|
||||
fmt.Printf("✅ 成功添加文章内容 (ID: %d)\n\n", contentID)
|
||||
} else {
|
||||
fmt.Printf("ℹ️ 文章内容已存在 (ID: %d)\n\n", existingContent.ID)
|
||||
}
|
||||
|
||||
// 6. 示例:查询文章列表
|
||||
fmt.Println("📋 示例4: 查询文章列表")
|
||||
articles, total, err := articleRepo.List(officialID, 1, 10)
|
||||
if err != nil {
|
||||
log.Fatal("查询文章列表失败:", err)
|
||||
}
|
||||
|
||||
fmt.Printf("共找到 %d 篇文章:\n", total)
|
||||
for i, item := range articles {
|
||||
fmt.Printf("%d. %s (👁️ %d | 👍 %d)\n", i+1, item.Title, item.ReadNum, item.LikeNum)
|
||||
}
|
||||
fmt.Println()
|
||||
|
||||
// 7. 示例:获取文章详情
|
||||
fmt.Println("📖 示例5: 获取文章详情")
|
||||
detail, err := contentRepo.GetArticleDetail(articleID)
|
||||
if err != nil {
|
||||
log.Fatal("获取文章详情失败:", err)
|
||||
}
|
||||
|
||||
if detail != nil {
|
||||
fmt.Printf("标题: %s\n", detail.Title)
|
||||
fmt.Printf("作者: %s\n", detail.Author)
|
||||
fmt.Printf("公众号: %s\n", detail.OfficialName)
|
||||
fmt.Printf("发布时间: %s\n", detail.PublishTime)
|
||||
fmt.Printf("阅读数: %d | 点赞数: %d\n", detail.ReadNum, detail.LikeNum)
|
||||
fmt.Printf("段落数: %d\n", len(detail.Paragraphs))
|
||||
fmt.Printf("图片数: %d\n", len(detail.Images))
|
||||
if len(detail.Paragraphs) > 0 {
|
||||
fmt.Printf("第一段: %s\n", detail.Paragraphs[0])
|
||||
}
|
||||
}
|
||||
fmt.Println()
|
||||
|
||||
// 8. 示例:搜索文章
|
||||
fmt.Println("🔍 示例6: 搜索文章")
|
||||
searchResults, searchTotal, err := articleRepo.Search("考研", 1, 10)
|
||||
if err != nil {
|
||||
log.Fatal("搜索文章失败:", err)
|
||||
}
|
||||
|
||||
fmt.Printf("搜索\"考研\"找到 %d 篇文章:\n", searchTotal)
|
||||
for i, item := range searchResults {
|
||||
fmt.Printf("%d. %s\n", i+1, item.Title)
|
||||
}
|
||||
fmt.Println()
|
||||
|
||||
// 9. 示例:获取统计信息
|
||||
fmt.Println("📊 示例7: 获取统计信息")
|
||||
stats, err := db.GetStatistics()
|
||||
if err != nil {
|
||||
log.Fatal("获取统计信息失败:", err)
|
||||
}
|
||||
|
||||
fmt.Printf("公众号总数: %d\n", stats.TotalOfficials)
|
||||
fmt.Printf("文章总数: %d\n", stats.TotalArticles)
|
||||
fmt.Printf("总阅读数: %d\n", stats.TotalReadNum)
|
||||
fmt.Printf("总点赞数: %d\n", stats.TotalLikeNum)
|
||||
fmt.Println()
|
||||
|
||||
// 10. 示例:批量插入文章
|
||||
fmt.Println("📦 示例8: 批量插入文章")
|
||||
batchArticles := []*database.Article{
|
||||
{
|
||||
OfficialID: officialID,
|
||||
Title: "教育部:2026年全国硕士研究生报名人数为343万",
|
||||
Author: "研招网资讯",
|
||||
Link: "https://mp.weixin.qq.com/s?__biz=MzI1NjEwMTM4OA==&mid=2651232406",
|
||||
PublishTime: "2024-11-26 09:00:00",
|
||||
ReadNum: 8965,
|
||||
LikeNum: 234,
|
||||
ContentPreview: "教育部公布2026年研究生招生数据...",
|
||||
ParagraphCount: 12,
|
||||
},
|
||||
{
|
||||
OfficialID: officialID,
|
||||
Title: "研考网上确认成功后,需重点关注四件事",
|
||||
Author: "研招网资讯",
|
||||
Link: "https://mp.weixin.qq.com/s?__biz=MzI1NjEwMTM4OA==&mid=2651232407",
|
||||
PublishTime: "2024-11-25 15:30:00",
|
||||
ReadNum: 6543,
|
||||
LikeNum: 189,
|
||||
ContentPreview: "网上确认通过后,考生还需要注意以下事项...",
|
||||
ParagraphCount: 8,
|
||||
},
|
||||
}
|
||||
|
||||
err = articleRepo.BatchInsertArticles(batchArticles)
|
||||
if err != nil {
|
||||
log.Fatal("批量插入文章失败:", err)
|
||||
}
|
||||
fmt.Printf("✅ 成功批量插入 %d 篇文章\n\n", len(batchArticles))
|
||||
|
||||
// 11. 示例:导出JSON数据
|
||||
fmt.Println("💾 示例9: 导出文章列表为JSON")
|
||||
allArticles, _, err := articleRepo.List(0, 1, 100)
|
||||
if err != nil {
|
||||
log.Fatal("查询文章列表失败:", err)
|
||||
}
|
||||
|
||||
jsonData, err := json.MarshalIndent(allArticles, "", " ")
|
||||
if err != nil {
|
||||
log.Fatal("JSON序列化失败:", err)
|
||||
}
|
||||
|
||||
fmt.Println("文章列表JSON (前200字符):")
|
||||
if len(jsonData) > 200 {
|
||||
fmt.Println(string(jsonData[:200]) + "...")
|
||||
} else {
|
||||
fmt.Println(string(jsonData))
|
||||
}
|
||||
fmt.Println()
|
||||
|
||||
fmt.Println("==============================================")
|
||||
fmt.Println(" 数据库操作示例演示完成!")
|
||||
fmt.Println("==============================================")
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
module github.com/wechat-crawler
|
||||
|
||||
go 1.20
|
||||
|
||||
require github.com/go-resty/resty/v2 v2.10.0
|
||||
|
||||
require golang.org/x/net v0.17.0 // indirect
|
||||
@@ -1,44 +0,0 @@
|
||||
github.com/go-resty/resty/v2 v2.10.0 h1:Qla4W/+TMmv0fOeeRqzEpXPLfTUnR5HZ1+lGs+CkiCo=
|
||||
github.com/go-resty/resty/v2 v2.10.0/go.mod h1:iiP/OpA0CkcL3IGt1O0+/SIItFUbkkyw5BGXiVdTu+A=
|
||||
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
|
||||
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
|
||||
golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf4=
|
||||
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
|
||||
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
|
||||
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
|
||||
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
|
||||
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
|
||||
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
|
||||
golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM=
|
||||
golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
|
||||
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
||||
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
|
||||
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
|
||||
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
|
||||
golang.org/x/term v0.13.0/go.mod h1:LTmsnFJwVN6bCy1rVCoS+qHT1HhALEFxKncY3WNNh4U=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
|
||||
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
|
||||
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
|
||||
golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
|
||||
golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4=
|
||||
golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
|
||||
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
|
||||
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
|
||||
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
BIN
backend/main.exe
BIN
backend/main.exe
Binary file not shown.
Binary file not shown.
117
backend/pkg/database/db.go
Normal file
117
backend/pkg/database/db.go
Normal file
@@ -0,0 +1,117 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
// DB 数据库实例
|
||||
type DB struct {
|
||||
*sql.DB
|
||||
}
|
||||
|
||||
// InitDB 初始化数据库
|
||||
func InitDB(dbPath string) (*DB, error) {
|
||||
// 确保数据库目录存在
|
||||
dbDir := filepath.Dir(dbPath)
|
||||
if err := os.MkdirAll(dbDir, 0755); err != nil {
|
||||
return nil, fmt.Errorf("创建数据库目录失败: %v", err)
|
||||
}
|
||||
|
||||
// 打开数据库连接(使用modernc.org/sqlite驱动)
|
||||
db, err := sql.Open("sqlite", dbPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("打开数据库失败: %v", err)
|
||||
}
|
||||
|
||||
// 测试连接
|
||||
if err := db.Ping(); err != nil {
|
||||
return nil, fmt.Errorf("数据库连接测试失败: %v", err)
|
||||
}
|
||||
|
||||
// 创建表
|
||||
if err := createTables(db); err != nil {
|
||||
return nil, fmt.Errorf("创建数据表失败: %v", err)
|
||||
}
|
||||
|
||||
fmt.Println("✅ 数据库初始化成功:", dbPath)
|
||||
return &DB{db}, nil
|
||||
}
|
||||
|
||||
// createTables 创建数据表
|
||||
func createTables(db *sql.DB) error {
|
||||
// 公众号表
|
||||
officialAccountTable := `
|
||||
CREATE TABLE IF NOT EXISTS official_accounts (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
biz TEXT NOT NULL UNIQUE,
|
||||
nickname TEXT NOT NULL,
|
||||
homepage TEXT,
|
||||
description TEXT,
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_biz ON official_accounts(biz);
|
||||
CREATE INDEX IF NOT EXISTS idx_nickname ON official_accounts(nickname);
|
||||
`
|
||||
|
||||
// 文章表
|
||||
articleTable := `
|
||||
CREATE TABLE IF NOT EXISTS articles (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
official_id INTEGER NOT NULL,
|
||||
title TEXT NOT NULL,
|
||||
author TEXT,
|
||||
link TEXT UNIQUE,
|
||||
publish_time TEXT,
|
||||
create_time TEXT,
|
||||
comment_id TEXT,
|
||||
read_num INTEGER DEFAULT 0,
|
||||
like_num INTEGER DEFAULT 0,
|
||||
share_num INTEGER DEFAULT 0,
|
||||
content_preview TEXT,
|
||||
paragraph_count INTEGER DEFAULT 0,
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
FOREIGN KEY (official_id) REFERENCES official_accounts(id)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_official_id ON articles(official_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_title ON articles(title);
|
||||
CREATE INDEX IF NOT EXISTS idx_publish_time ON articles(publish_time);
|
||||
CREATE INDEX IF NOT EXISTS idx_link ON articles(link);
|
||||
`
|
||||
|
||||
// 文章内容表
|
||||
articleContentTable := `
|
||||
CREATE TABLE IF NOT EXISTS article_contents (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
article_id INTEGER NOT NULL UNIQUE,
|
||||
html_content TEXT,
|
||||
text_content TEXT,
|
||||
paragraphs TEXT,
|
||||
images TEXT,
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
FOREIGN KEY (article_id) REFERENCES articles(id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_article_id ON article_contents(article_id);
|
||||
`
|
||||
|
||||
// 执行创建表语句
|
||||
tables := []string{officialAccountTable, articleTable, articleContentTable}
|
||||
for _, table := range tables {
|
||||
if _, err := db.Exec(table); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Close 关闭数据库连接
|
||||
func (db *DB) Close() error {
|
||||
return db.DB.Close()
|
||||
}
|
||||
76
backend/pkg/database/models.go
Normal file
76
backend/pkg/database/models.go
Normal file
@@ -0,0 +1,76 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// OfficialAccount 公众号信息
|
||||
type OfficialAccount struct {
|
||||
ID int64 `json:"id"`
|
||||
Biz string `json:"biz"` // 公众号唯一标识
|
||||
Nickname string `json:"nickname"` // 公众号名称
|
||||
Homepage string `json:"homepage"` // 公众号主页链接
|
||||
Description string `json:"description"` // 公众号描述
|
||||
CreatedAt time.Time `json:"created_at"` // 创建时间
|
||||
UpdatedAt time.Time `json:"updated_at"` // 更新时间
|
||||
}
|
||||
|
||||
// Article 文章信息
|
||||
type Article struct {
|
||||
ID int64 `json:"id"`
|
||||
OfficialID int64 `json:"official_id"` // 关联的公众号ID
|
||||
Title string `json:"title"` // 文章标题
|
||||
Author string `json:"author"` // 作者
|
||||
Link string `json:"link"` // 文章链接
|
||||
PublishTime string `json:"publish_time"` // 发布时间
|
||||
CreateTime string `json:"create_time"` // 创建时间(抓取时间)
|
||||
CommentID string `json:"comment_id"` // 评论ID
|
||||
ReadNum int `json:"read_num"` // 阅读数
|
||||
LikeNum int `json:"like_num"` // 点赞数
|
||||
ShareNum int `json:"share_num"` // 分享数
|
||||
ContentPreview string `json:"content_preview"` // 内容预览(前200字)
|
||||
ParagraphCount int `json:"paragraph_count"` // 段落数
|
||||
CreatedAt time.Time `json:"created_at"` // 数据库创建时间
|
||||
UpdatedAt time.Time `json:"updated_at"` // 数据库更新时间
|
||||
}
|
||||
|
||||
// ArticleContent 文章详细内容
|
||||
type ArticleContent struct {
|
||||
ID int64 `json:"id"`
|
||||
ArticleID int64 `json:"article_id"` // 关联的文章ID
|
||||
HtmlContent string `json:"html_content"` // HTML原始内容
|
||||
TextContent string `json:"text_content"` // 纯文本内容
|
||||
Paragraphs string `json:"paragraphs"` // 段落内容(JSON数组)
|
||||
Images string `json:"images"` // 图片链接(JSON数组)
|
||||
CreatedAt time.Time `json:"created_at"` // 创建时间
|
||||
}
|
||||
|
||||
// ArticleListItem 文章列表项(用于API返回)
|
||||
type ArticleListItem struct {
|
||||
ID int64 `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Author string `json:"author"`
|
||||
PublishTime string `json:"publish_time"`
|
||||
ReadNum int `json:"read_num"`
|
||||
LikeNum int `json:"like_num"`
|
||||
OfficialName string `json:"official_name"`
|
||||
ContentPreview string `json:"content_preview"`
|
||||
}
|
||||
|
||||
// ArticleDetail 文章详情(用于API返回)
|
||||
type ArticleDetail struct {
|
||||
Article
|
||||
OfficialName string `json:"official_name"`
|
||||
HtmlContent string `json:"html_content"`
|
||||
TextContent string `json:"text_content"`
|
||||
Paragraphs []string `json:"paragraphs"`
|
||||
Images []string `json:"images"`
|
||||
}
|
||||
|
||||
// Statistics 统计信息
|
||||
type Statistics struct {
|
||||
TotalOfficials int `json:"total_officials"` // 公众号总数
|
||||
TotalArticles int `json:"total_articles"` // 文章总数
|
||||
TotalReadNum int `json:"total_read_num"` // 总阅读数
|
||||
TotalLikeNum int `json:"total_like_num"` // 总点赞数
|
||||
}
|
||||
455
backend/pkg/database/repository.go
Normal file
455
backend/pkg/database/repository.go
Normal file
@@ -0,0 +1,455 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// OfficialAccountRepository 公众号数据仓库
|
||||
type OfficialAccountRepository struct {
|
||||
db *DB
|
||||
}
|
||||
|
||||
// NewOfficialAccountRepository 创建公众号仓库
|
||||
func NewOfficialAccountRepository(db *DB) *OfficialAccountRepository {
|
||||
return &OfficialAccountRepository{db: db}
|
||||
}
|
||||
|
||||
// Create 创建公众号
|
||||
func (r *OfficialAccountRepository) Create(account *OfficialAccount) (int64, error) {
|
||||
result, err := r.db.Exec(`
|
||||
INSERT INTO official_accounts (biz, nickname, homepage, description)
|
||||
VALUES (?, ?, ?, ?)
|
||||
`, account.Biz, account.Nickname, account.Homepage, account.Description)
|
||||
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return result.LastInsertId()
|
||||
}
|
||||
|
||||
// GetByBiz 根据Biz获取公众号
|
||||
func (r *OfficialAccountRepository) GetByBiz(biz string) (*OfficialAccount, error) {
|
||||
account := &OfficialAccount{}
|
||||
err := r.db.QueryRow(`
|
||||
SELECT id, biz, nickname, homepage, description, created_at, updated_at
|
||||
FROM official_accounts WHERE biz = ?
|
||||
`, biz).Scan(&account.ID, &account.Biz, &account.Nickname, &account.Homepage,
|
||||
&account.Description, &account.CreatedAt, &account.UpdatedAt)
|
||||
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return account, nil
|
||||
}
|
||||
|
||||
// GetByID 根据ID获取公众号
|
||||
func (r *OfficialAccountRepository) GetByID(id int64) (*OfficialAccount, error) {
|
||||
account := &OfficialAccount{}
|
||||
err := r.db.QueryRow(`
|
||||
SELECT id, biz, nickname, homepage, description, created_at, updated_at
|
||||
FROM official_accounts WHERE id = ?
|
||||
`, id).Scan(&account.ID, &account.Biz, &account.Nickname, &account.Homepage,
|
||||
&account.Description, &account.CreatedAt, &account.UpdatedAt)
|
||||
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return account, nil
|
||||
}
|
||||
|
||||
// List 获取所有公众号列表
|
||||
func (r *OfficialAccountRepository) List() ([]*OfficialAccount, error) {
|
||||
rows, err := r.db.Query(`
|
||||
SELECT id, biz, nickname, homepage, description, created_at, updated_at
|
||||
FROM official_accounts ORDER BY created_at DESC
|
||||
`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var accounts []*OfficialAccount
|
||||
for rows.Next() {
|
||||
account := &OfficialAccount{}
|
||||
err := rows.Scan(&account.ID, &account.Biz, &account.Nickname, &account.Homepage,
|
||||
&account.Description, &account.CreatedAt, &account.UpdatedAt)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
accounts = append(accounts, account)
|
||||
}
|
||||
|
||||
return accounts, nil
|
||||
}
|
||||
|
||||
// Update 更新公众号信息
|
||||
func (r *OfficialAccountRepository) Update(account *OfficialAccount) error {
|
||||
_, err := r.db.Exec(`
|
||||
UPDATE official_accounts
|
||||
SET nickname = ?, homepage = ?, description = ?, updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = ?
|
||||
`, account.Nickname, account.Homepage, account.Description, account.ID)
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
// ArticleRepository 文章数据仓库
|
||||
type ArticleRepository struct {
|
||||
db *DB
|
||||
}
|
||||
|
||||
// NewArticleRepository 创建文章仓库
|
||||
func NewArticleRepository(db *DB) *ArticleRepository {
|
||||
return &ArticleRepository{db: db}
|
||||
}
|
||||
|
||||
// Create 创建文章
|
||||
func (r *ArticleRepository) Create(article *Article) (int64, error) {
|
||||
result, err := r.db.Exec(`
|
||||
INSERT INTO articles (
|
||||
official_id, title, author, link, publish_time, create_time,
|
||||
comment_id, read_num, like_num, share_num, content_preview, paragraph_count
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`, article.OfficialID, article.Title, article.Author, article.Link,
|
||||
article.PublishTime, article.CreateTime, article.CommentID,
|
||||
article.ReadNum, article.LikeNum, article.ShareNum,
|
||||
article.ContentPreview, article.ParagraphCount)
|
||||
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return result.LastInsertId()
|
||||
}
|
||||
|
||||
// GetByID 根据ID获取文章
|
||||
func (r *ArticleRepository) GetByID(id int64) (*Article, error) {
|
||||
article := &Article{}
|
||||
err := r.db.QueryRow(`
|
||||
SELECT id, official_id, title, author, link, publish_time, create_time,
|
||||
comment_id, read_num, like_num, share_num, content_preview,
|
||||
paragraph_count, created_at, updated_at
|
||||
FROM articles WHERE id = ?
|
||||
`, id).Scan(&article.ID, &article.OfficialID, &article.Title, &article.Author,
|
||||
&article.Link, &article.PublishTime, &article.CreateTime, &article.CommentID,
|
||||
&article.ReadNum, &article.LikeNum, &article.ShareNum, &article.ContentPreview,
|
||||
&article.ParagraphCount, &article.CreatedAt, &article.UpdatedAt)
|
||||
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return article, nil
|
||||
}
|
||||
|
||||
// GetByLink 根据链接获取文章
|
||||
func (r *ArticleRepository) GetByLink(link string) (*Article, error) {
|
||||
article := &Article{}
|
||||
err := r.db.QueryRow(`
|
||||
SELECT id, official_id, title, author, link, publish_time, create_time,
|
||||
comment_id, read_num, like_num, share_num, content_preview,
|
||||
paragraph_count, created_at, updated_at
|
||||
FROM articles WHERE link = ?
|
||||
`, link).Scan(&article.ID, &article.OfficialID, &article.Title, &article.Author,
|
||||
&article.Link, &article.PublishTime, &article.CreateTime, &article.CommentID,
|
||||
&article.ReadNum, &article.LikeNum, &article.ShareNum, &article.ContentPreview,
|
||||
&article.ParagraphCount, &article.CreatedAt, &article.UpdatedAt)
|
||||
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return article, nil
|
||||
}
|
||||
|
||||
// List 获取文章列表(分页)
|
||||
func (r *ArticleRepository) List(officialID int64, page, pageSize int) ([]*ArticleListItem, int, error) {
|
||||
// 构建查询条件
|
||||
whereClause := ""
|
||||
args := []interface{}{}
|
||||
|
||||
if officialID > 0 {
|
||||
whereClause = "WHERE a.official_id = ?"
|
||||
args = append(args, officialID)
|
||||
}
|
||||
|
||||
// 获取总数
|
||||
countQuery := fmt.Sprintf("SELECT COUNT(*) FROM articles a %s", whereClause)
|
||||
var total int
|
||||
err := r.db.QueryRow(countQuery, args...).Scan(&total)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
|
||||
// 获取列表
|
||||
offset := (page - 1) * pageSize
|
||||
listQuery := fmt.Sprintf(`
|
||||
SELECT a.id, a.title, a.author, a.publish_time, a.read_num, a.like_num,
|
||||
a.content_preview, o.nickname
|
||||
FROM articles a
|
||||
LEFT JOIN official_accounts o ON a.official_id = o.id
|
||||
%s
|
||||
ORDER BY a.publish_time DESC
|
||||
LIMIT ? OFFSET ?
|
||||
`, whereClause)
|
||||
|
||||
args = append(args, pageSize, offset)
|
||||
rows, err := r.db.Query(listQuery, args...)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var items []*ArticleListItem
|
||||
for rows.Next() {
|
||||
item := &ArticleListItem{}
|
||||
err := rows.Scan(&item.ID, &item.Title, &item.Author, &item.PublishTime,
|
||||
&item.ReadNum, &item.LikeNum, &item.ContentPreview, &item.OfficialName)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
items = append(items, item)
|
||||
}
|
||||
|
||||
return items, total, nil
|
||||
}
|
||||
|
||||
// Search 搜索文章
|
||||
func (r *ArticleRepository) Search(keyword string, page, pageSize int) ([]*ArticleListItem, int, error) {
|
||||
keyword = "%" + keyword + "%"
|
||||
|
||||
// 获取总数
|
||||
var total int
|
||||
err := r.db.QueryRow(`
|
||||
SELECT COUNT(*) FROM articles WHERE title LIKE ? OR author LIKE ?
|
||||
`, keyword, keyword).Scan(&total)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
|
||||
// 获取列表
|
||||
offset := (page - 1) * pageSize
|
||||
rows, err := r.db.Query(`
|
||||
SELECT a.id, a.title, a.author, a.publish_time, a.read_num, a.like_num,
|
||||
a.content_preview, o.nickname
|
||||
FROM articles a
|
||||
LEFT JOIN official_accounts o ON a.official_id = o.id
|
||||
WHERE a.title LIKE ? OR a.author LIKE ?
|
||||
ORDER BY a.publish_time DESC
|
||||
LIMIT ? OFFSET ?
|
||||
`, keyword, keyword, pageSize, offset)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var items []*ArticleListItem
|
||||
for rows.Next() {
|
||||
item := &ArticleListItem{}
|
||||
err := rows.Scan(&item.ID, &item.Title, &item.Author, &item.PublishTime,
|
||||
&item.ReadNum, &item.LikeNum, &item.ContentPreview, &item.OfficialName)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
items = append(items, item)
|
||||
}
|
||||
|
||||
return items, total, nil
|
||||
}
|
||||
|
||||
// Update 更新文章信息
|
||||
func (r *ArticleRepository) Update(article *Article) error {
|
||||
_, err := r.db.Exec(`
|
||||
UPDATE articles
|
||||
SET read_num = ?, like_num = ?, share_num = ?, updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = ?
|
||||
`, article.ReadNum, article.LikeNum, article.ShareNum, article.ID)
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
// ArticleContentRepository 文章内容数据仓库
|
||||
type ArticleContentRepository struct {
|
||||
db *DB
|
||||
}
|
||||
|
||||
// NewArticleContentRepository 创建文章内容仓库
|
||||
func NewArticleContentRepository(db *DB) *ArticleContentRepository {
|
||||
return &ArticleContentRepository{db: db}
|
||||
}
|
||||
|
||||
// Create 创建文章内容
|
||||
func (r *ArticleContentRepository) Create(content *ArticleContent) (int64, error) {
|
||||
result, err := r.db.Exec(`
|
||||
INSERT INTO article_contents (article_id, html_content, text_content, paragraphs, images)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
`, content.ArticleID, content.HtmlContent, content.TextContent,
|
||||
content.Paragraphs, content.Images)
|
||||
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return result.LastInsertId()
|
||||
}
|
||||
|
||||
// GetByArticleID 根据文章ID获取内容
|
||||
func (r *ArticleContentRepository) GetByArticleID(articleID int64) (*ArticleContent, error) {
|
||||
content := &ArticleContent{}
|
||||
err := r.db.QueryRow(`
|
||||
SELECT id, article_id, html_content, text_content, paragraphs, images, created_at
|
||||
FROM article_contents WHERE article_id = ?
|
||||
`, articleID).Scan(&content.ID, &content.ArticleID, &content.HtmlContent,
|
||||
&content.TextContent, &content.Paragraphs, &content.Images, &content.CreatedAt)
|
||||
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return content, nil
|
||||
}
|
||||
|
||||
// GetArticleDetail 获取文章详情(包含内容)
|
||||
func (r *ArticleContentRepository) GetArticleDetail(articleID int64) (*ArticleDetail, error) {
|
||||
detail := &ArticleDetail{}
|
||||
var paragraphsJSON, imagesJSON string
|
||||
|
||||
err := r.db.QueryRow(`
|
||||
SELECT a.id, a.official_id, a.title, a.author, a.link, a.publish_time,
|
||||
a.create_time, a.comment_id, a.read_num, a.like_num, a.share_num,
|
||||
a.content_preview, a.paragraph_count, a.created_at, a.updated_at,
|
||||
o.nickname, c.html_content, c.text_content, c.paragraphs, c.images
|
||||
FROM articles a
|
||||
LEFT JOIN official_accounts o ON a.official_id = o.id
|
||||
LEFT JOIN article_contents c ON a.id = c.article_id
|
||||
WHERE a.id = ?
|
||||
`, articleID).Scan(
|
||||
&detail.ID, &detail.OfficialID, &detail.Title, &detail.Author,
|
||||
&detail.Link, &detail.PublishTime, &detail.CreateTime, &detail.CommentID,
|
||||
&detail.ReadNum, &detail.LikeNum, &detail.ShareNum, &detail.ContentPreview,
|
||||
&detail.ParagraphCount, &detail.CreatedAt, &detail.UpdatedAt,
|
||||
&detail.OfficialName, &detail.HtmlContent, &detail.TextContent,
|
||||
¶graphsJSON, &imagesJSON,
|
||||
)
|
||||
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 解析JSON数组
|
||||
if paragraphsJSON != "" {
|
||||
json.Unmarshal([]byte(paragraphsJSON), &detail.Paragraphs)
|
||||
}
|
||||
if imagesJSON != "" {
|
||||
json.Unmarshal([]byte(imagesJSON), &detail.Images)
|
||||
}
|
||||
|
||||
return detail, nil
|
||||
}
|
||||
|
||||
// GetStatistics 获取统计信息
|
||||
func (db *DB) GetStatistics() (*Statistics, error) {
|
||||
stats := &Statistics{}
|
||||
|
||||
err := db.QueryRow(`
|
||||
SELECT
|
||||
(SELECT COUNT(*) FROM official_accounts) as total_officials,
|
||||
(SELECT COUNT(*) FROM articles) as total_articles,
|
||||
(SELECT COALESCE(SUM(read_num), 0) FROM articles) as total_read_num,
|
||||
(SELECT COALESCE(SUM(like_num), 0) FROM articles) as total_like_num
|
||||
`).Scan(&stats.TotalOfficials, &stats.TotalArticles, &stats.TotalReadNum, &stats.TotalLikeNum)
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return stats, nil
|
||||
}
|
||||
|
||||
// BatchInsertArticles 批量插入文章
|
||||
func (r *ArticleRepository) BatchInsertArticles(articles []*Article) error {
|
||||
if len(articles) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// 开始事务
|
||||
tx, err := r.db.Begin()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
|
||||
stmt, err := tx.Prepare(`
|
||||
INSERT OR IGNORE INTO articles (
|
||||
official_id, title, author, link, publish_time, create_time,
|
||||
comment_id, read_num, like_num, share_num, content_preview, paragraph_count
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer stmt.Close()
|
||||
|
||||
for _, article := range articles {
|
||||
_, err = stmt.Exec(
|
||||
article.OfficialID, article.Title, article.Author, article.Link,
|
||||
article.PublishTime, article.CreateTime, article.CommentID,
|
||||
article.ReadNum, article.LikeNum, article.ShareNum,
|
||||
article.ContentPreview, article.ParagraphCount,
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// Helper function: 将字符串数组转换为JSON字符串
|
||||
func StringsToJSON(strs []string) string {
|
||||
if len(strs) == 0 {
|
||||
return "[]"
|
||||
}
|
||||
data, _ := json.Marshal(strs)
|
||||
return string(data)
|
||||
}
|
||||
|
||||
// Helper function: 生成内容预览
|
||||
func GeneratePreview(content string, maxLen int) string {
|
||||
if len(content) <= maxLen {
|
||||
return content
|
||||
}
|
||||
// 移除换行符和多余空格
|
||||
content = strings.ReplaceAll(content, "\n", " ")
|
||||
content = strings.ReplaceAll(content, "\r", "")
|
||||
content = strings.Join(strings.Fields(content), " ")
|
||||
|
||||
if len(content) <= maxLen {
|
||||
return content
|
||||
}
|
||||
return content[:maxLen] + "..."
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,48 +0,0 @@
|
||||
@echo off
|
||||
|
||||
echo WeChat Public Article Crawler Startup Script
|
||||
echo =================================
|
||||
|
||||
REM Check if cookie.txt file exists
|
||||
if not exist "cookie.txt" (
|
||||
echo Error: cookie.txt file not found!
|
||||
echo Please create cookie.txt file in backend directory and add WeChat public platform cookie information.
|
||||
echo.
|
||||
echo cookie.txt format example:
|
||||
echo __biz=xxx; uin=xxx; key=xxx; pass_ticket=xxx;
|
||||
echo.
|
||||
pause
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
REM Set Go environment variables (if needed)
|
||||
REM set GOPATH=%USERPROFILE%\go
|
||||
REM set GOROOT=C:\Go
|
||||
REM set PATH=%PATH%;%GOROOT%\bin;%GOPATH%\bin
|
||||
|
||||
echo Downloading dependencies...
|
||||
go mod tidy
|
||||
if %errorlevel% neq 0 (
|
||||
echo Failed to download dependencies!
|
||||
pause
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
echo Compiling program...
|
||||
go build -o output\wechat-crawler.exe cmd\main.go
|
||||
if %errorlevel% neq 0 (
|
||||
echo Compilation failed!
|
||||
pause
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
echo Compilation successful! Starting program...
|
||||
echo.
|
||||
|
||||
REM Ensure data directory exists
|
||||
if not exist "data" mkdir data
|
||||
|
||||
REM Run the program
|
||||
output\wechat-crawler.exe
|
||||
|
||||
pause
|
||||
@@ -1,57 +0,0 @@
|
||||
@echo off
|
||||
|
||||
rem WeChat Official Account Article Crawler - Script for crawling via article link
|
||||
setlocal enabledelayedexpansion
|
||||
|
||||
REM 检查是否有命令行参数传入
|
||||
if "%1" neq "" (
|
||||
REM 如果有参数,直接将其作为文章链接传入程序
|
||||
echo.
|
||||
echo Compiling and running...
|
||||
go run "cmd/main.go" "%1"
|
||||
|
||||
if errorlevel 1 (
|
||||
echo.
|
||||
echo Failed to run, please check error messages above
|
||||
pause
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
echo.
|
||||
echo Crawling completed successfully!
|
||||
pause
|
||||
exit /b 0
|
||||
) else (
|
||||
REM 如果没有参数,运行交互式模式
|
||||
:input_loop
|
||||
cls
|
||||
echo ========================================
|
||||
echo WeChat Official Account Article Crawler
|
||||
echo ========================================
|
||||
echo.
|
||||
echo Please enter WeChat article link:
|
||||
echo Example: https://mp.weixin.qq.com/s/4r_LKJu0mOeUc70ZZXK9LA
|
||||
set /p ARTICLE_LINK=
|
||||
|
||||
if "%ARTICLE_LINK%"=="" (
|
||||
echo.
|
||||
echo Error: Article link cannot be empty!
|
||||
pause
|
||||
goto input_loop
|
||||
)
|
||||
|
||||
echo.
|
||||
echo Compiling and running...
|
||||
go run "cmd/main.go" "%ARTICLE_LINK%"
|
||||
|
||||
if errorlevel 1 (
|
||||
echo.
|
||||
echo Failed to run, please check error messages above
|
||||
pause
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
echo.
|
||||
echo Crawling completed successfully!
|
||||
pause
|
||||
)
|
||||
21
backend/tools/view_db.bat
Normal file
21
backend/tools/view_db.bat
Normal file
@@ -0,0 +1,21 @@
|
||||
@echo off
|
||||
chcp 65001 >nul
|
||||
cls
|
||||
|
||||
echo ===============================================
|
||||
echo 📊 数据库内容查看工具
|
||||
echo ===============================================
|
||||
echo.
|
||||
|
||||
cd /d "%~dp0"
|
||||
|
||||
echo 正在查询数据库...
|
||||
echo.
|
||||
|
||||
go run view_db.go
|
||||
|
||||
echo.
|
||||
echo ===============================================
|
||||
echo 查询完成!
|
||||
echo ===============================================
|
||||
pause
|
||||
231
backend/tools/view_db.go
Normal file
231
backend/tools/view_db.go
Normal file
@@ -0,0 +1,231 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// 打开数据库
|
||||
db, err := sql.Open("sqlite", "../../data/wechat_articles.db")
|
||||
if err != nil {
|
||||
log.Fatal("打开数据库失败:", err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
fmt.Println("=" + repeatStr("=", 80))
|
||||
fmt.Println("📊 微信公众号文章数据库内容查看")
|
||||
fmt.Println("=" + repeatStr("=", 80))
|
||||
|
||||
// 查询公众号
|
||||
fmt.Println("\n📢 【公众号列表】")
|
||||
fmt.Println(repeatStr("-", 80))
|
||||
queryOfficialAccounts(db)
|
||||
|
||||
// 查询文章
|
||||
fmt.Println("\n📝 【文章列表】")
|
||||
fmt.Println(repeatStr("-", 80))
|
||||
queryArticles(db)
|
||||
|
||||
// 查询文章内容
|
||||
fmt.Println("\n📄 【文章详细内容】")
|
||||
fmt.Println(repeatStr("-", 80))
|
||||
queryArticleContents(db)
|
||||
|
||||
fmt.Println("\n" + repeatStr("=", 80))
|
||||
}
|
||||
|
||||
func queryOfficialAccounts(db *sql.DB) {
|
||||
rows, err := db.Query(`
|
||||
SELECT id, biz, nickname, homepage, description, created_at, updated_at
|
||||
FROM official_accounts
|
||||
ORDER BY id
|
||||
`)
|
||||
if err != nil {
|
||||
log.Printf("查询公众号失败: %v\n", err)
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
count := 0
|
||||
for rows.Next() {
|
||||
var id int
|
||||
var biz, nickname, homepage, description, createdAt, updatedAt string
|
||||
err := rows.Scan(&id, &biz, &nickname, &homepage, &description, &createdAt, &updatedAt)
|
||||
if err != nil {
|
||||
log.Printf("读取数据失败: %v\n", err)
|
||||
continue
|
||||
}
|
||||
count++
|
||||
|
||||
fmt.Printf("\n🔹 公众号 #%d\n", id)
|
||||
fmt.Printf(" 名称: %s\n", nickname)
|
||||
fmt.Printf(" BIZ: %s\n", biz)
|
||||
fmt.Printf(" 主页: %s\n", homepage)
|
||||
fmt.Printf(" 简介: %s\n", description)
|
||||
fmt.Printf(" 创建时间: %s\n", createdAt)
|
||||
fmt.Printf(" 更新时间: %s\n", updatedAt)
|
||||
}
|
||||
|
||||
if count == 0 {
|
||||
fmt.Println(" 暂无数据")
|
||||
} else {
|
||||
fmt.Printf("\n总计: %d 个公众号\n", count)
|
||||
}
|
||||
}
|
||||
|
||||
func queryArticles(db *sql.DB) {
|
||||
rows, err := db.Query(`
|
||||
SELECT a.id, a.official_id, a.title, a.author, a.link, a.publish_time,
|
||||
a.read_num, a.like_num, a.share_num, a.paragraph_count,
|
||||
a.content_preview, a.created_at, oa.nickname
|
||||
FROM articles a
|
||||
LEFT JOIN official_accounts oa ON a.official_id = oa.id
|
||||
ORDER BY a.id
|
||||
`)
|
||||
if err != nil {
|
||||
log.Printf("查询文章失败: %v\n", err)
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
count := 0
|
||||
for rows.Next() {
|
||||
var id, officialID, readNum, likeNum, shareNum, paragraphCount int
|
||||
var title, author, link, publishTime, contentPreview, createdAt, officialName sql.NullString
|
||||
err := rows.Scan(&id, &officialID, &title, &author, &link, &publishTime,
|
||||
&readNum, &likeNum, &shareNum, ¶graphCount, &contentPreview, &createdAt, &officialName)
|
||||
if err != nil {
|
||||
log.Printf("读取数据失败: %v\n", err)
|
||||
continue
|
||||
}
|
||||
count++
|
||||
|
||||
fmt.Printf("\n🔹 文章 #%d\n", id)
|
||||
fmt.Printf(" 标题: %s\n", getStringValue(title))
|
||||
if officialName.Valid {
|
||||
fmt.Printf(" 公众号: %s\n", officialName.String)
|
||||
}
|
||||
fmt.Printf(" 作者: %s\n", getStringValue(author))
|
||||
fmt.Printf(" 链接: %s\n", getStringValue(link))
|
||||
fmt.Printf(" 发布时间: %s\n", getStringValue(publishTime))
|
||||
fmt.Printf(" 阅读数: %d | 点赞数: %d | 分享数: %d\n", readNum, likeNum, shareNum)
|
||||
fmt.Printf(" 段落数: %d\n", paragraphCount)
|
||||
if contentPreview.Valid && contentPreview.String != "" {
|
||||
preview := contentPreview.String
|
||||
if len(preview) > 100 {
|
||||
preview = preview[:100] + "..."
|
||||
}
|
||||
fmt.Printf(" 内容预览: %s\n", preview)
|
||||
}
|
||||
fmt.Printf(" 抓取时间: %s\n", getStringValue(createdAt))
|
||||
}
|
||||
|
||||
if count == 0 {
|
||||
fmt.Println(" 暂无数据")
|
||||
} else {
|
||||
fmt.Printf("\n总计: %d 篇文章\n", count)
|
||||
}
|
||||
}
|
||||
|
||||
func queryArticleContents(db *sql.DB) {
|
||||
rows, err := db.Query(`
|
||||
SELECT ac.id, ac.article_id, ac.html_content, ac.text_content,
|
||||
ac.paragraphs, ac.images, ac.created_at, a.title
|
||||
FROM article_contents ac
|
||||
LEFT JOIN articles a ON ac.article_id = a.id
|
||||
ORDER BY ac.id
|
||||
`)
|
||||
if err != nil {
|
||||
log.Printf("查询文章内容失败: %v\n", err)
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
count := 0
|
||||
for rows.Next() {
|
||||
var id, articleID int
|
||||
var htmlContent, textContent, paragraphs, images, createdAt, title sql.NullString
|
||||
err := rows.Scan(&id, &articleID, &htmlContent, &textContent,
|
||||
¶graphs, &images, &createdAt, &title)
|
||||
if err != nil {
|
||||
log.Printf("读取数据失败: %v\n", err)
|
||||
continue
|
||||
}
|
||||
count++
|
||||
|
||||
fmt.Printf("\n🔹 内容 #%d (文章ID: %d)\n", id, articleID)
|
||||
if title.Valid {
|
||||
fmt.Printf(" 文章标题: %s\n", title.String)
|
||||
}
|
||||
|
||||
// HTML内容长度
|
||||
htmlLen := 0
|
||||
if htmlContent.Valid {
|
||||
htmlLen = len(htmlContent.String)
|
||||
}
|
||||
fmt.Printf(" HTML内容长度: %d 字符\n", htmlLen)
|
||||
|
||||
// 文本内容
|
||||
if textContent.Valid && textContent.String != "" {
|
||||
text := textContent.String
|
||||
if len(text) > 200 {
|
||||
text = text[:200] + "..."
|
||||
}
|
||||
fmt.Printf(" 文本内容: %s\n", text)
|
||||
}
|
||||
|
||||
// 段落信息
|
||||
if paragraphs.Valid && paragraphs.String != "" {
|
||||
var paragraphList []interface{}
|
||||
if err := json.Unmarshal([]byte(paragraphs.String), ¶graphList); err == nil {
|
||||
fmt.Printf(" 段落数量: %d\n", len(paragraphList))
|
||||
}
|
||||
}
|
||||
|
||||
// 图片信息
|
||||
if images.Valid && images.String != "" {
|
||||
var imageList []interface{}
|
||||
if err := json.Unmarshal([]byte(images.String), &imageList); err == nil {
|
||||
fmt.Printf(" 图片数量: %d\n", len(imageList))
|
||||
if len(imageList) > 0 {
|
||||
fmt.Printf(" 图片URL:\n")
|
||||
for i, img := range imageList {
|
||||
if i >= 3 {
|
||||
fmt.Printf(" ... 还有 %d 张图片\n", len(imageList)-3)
|
||||
break
|
||||
}
|
||||
fmt.Printf(" %d. %v\n", i+1, img)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Printf(" 存储时间: %s\n", getStringValue(createdAt))
|
||||
}
|
||||
|
||||
if count == 0 {
|
||||
fmt.Println(" 暂无数据")
|
||||
} else {
|
||||
fmt.Printf("\n总计: %d 条详细内容\n", count)
|
||||
}
|
||||
}
|
||||
|
||||
func getStringValue(s sql.NullString) string {
|
||||
if s.Valid {
|
||||
return s.String
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func repeatStr(s string, n int) string {
|
||||
result := ""
|
||||
for i := 0; i < n; i++ {
|
||||
result += s
|
||||
}
|
||||
return result
|
||||
}
|
||||
Reference in New Issue
Block a user