使⽤mitmproxy+appium+python全⾃动抓取⽂章
1. 先使⽤mitmproxy代理抓历史⽂章列表
实现思路：
1.1 安装mitmproxy
需要先安装Python 3.6 (或更⾼版本)和 pip3.6 ，之后执⾏以下命令安装
sudo pip3.6 install -U pip
sudo pip3.6 install mitmproxy
1.2 准备代理脚本wechat_proxy.py
更多的例⼦可以参考
# -*- coding: utf-8 -*-
# @Time : 2019/7/11
# @Author : zl
from mitmproxy import http
from mitmproxy import ctx
from urllib import parse
import json
def response(flow: http.HTTPFlow):
try:
info = ctx.log.info
req = quest
resp = sponse
url = req.url
purl = parse.urlparse(url)
param_dict = parse.parse_qs(purl.query)
host = purl.hostname
path = purl.path
if host == 'mp.weixin.qq' and path == '/mp/profile_ext':
action = param_dict['action'][0]
if action in ['home', 'getmsg']:
msg = {}
msg['url'] = url
msg['action'] = action
msg['req_header'] = dict(req.headers)
msg['resp_header'] = dict(resp.headers)
if action == 'getmsg':
msg['resp_body'] = json.t)
info('msg : %s----' % msg)
except Exception as e:
('wechatproxy error, %s' % e)
执⾏下⾯的脚本启动：
mitmdump -p 8888 -s ${script_home}/wechat_proxy.py
mitmproxy其他的⽤法可以参考
2. 使⽤appium实现⾃动化
2.1 前提
2.1.1 安装JDK 并设置环境变量
1. 到Java官⽹下载相应的JDK并安装
2. 设置环境变量
- 添加JAVA_HOME
- 在PATH变量末尾追加 ;%JAVA_HOME%/bin;
- 添加CLASSPATH，设置值为%JAVA_HOME%\lib;%JAVA_HOME%\lib\tools.jar
2.1.2 安装Android SDK 并设置环境变量
注意：安装Android SDK需要并更新你的SDK repository
1. 到[Android 官⽹](developer.android/sdk/index.html#Other)下载并安装SDK
8.0怎么更新2. 设置环境变量
-
添加`ANDROID_HOME`
- 在`path`环境变量值末尾追加：`;%ANDROID_HOME%\tools;%ANDROID_HOME%\platform-tools;` 2.1.3 安装Nodejs
#先安装⼀个node版本管理软件
brew install n
#或者
curl -L git.io/n-install | bash
#安装node8.6
sudo n 8.6.0
#后⾯安装的时候有可能没权限
sudo chown -R $(whoami):admin /usr/local/**
注：不要安装太⾼版本，否则后⾯安装某些依赖会不兼容。
2.2 安装appium
安装appium有两种⽅式：
使⽤ npm (Node js 的管理与分发⼯具) 安装Appium
使⽤Appium官⽅安装包安装
下⾯只介绍npm安装⽅式，如果要装desktop，直接去最新版安装即可。
将npm的下载地址更改为淘宝的NPM镜像源
npm config set registry=registry.
npm config get registry
确认镜像源替换成功之后执⾏下⾯命令安装Appium
npm install -g appium
2.3 安装其他依赖
2.3.1 安装opencv4nodejs
如果已经安装openCV，则需要在安装opencv4nodejs之前设置
export OPENCV4NODEJS_DISABLE_AUTOBUILD=1
export OPENCV_INCLUDE_DIR=/usr/local/include/opencv4
export OPENCV_LIB_DIR=/usr/local/lib
export OPENCV_BIN_DIR=/usr/local/bin
npm安装
npm install -g opencv4nodejs
2.3.2 安装idevicelocation
brew install libzip libplist openssl libimobiledevice
git clone github/JonGabilondoAngulo/idevicelocation.git
cd idevicelocation
make
sudo make install
遇到问题
checking for libimobiledevice-1.0 >= 1. no
configure: error: Package requirements (libimobiledevice-1.0 >= 1.2.1) were not met:
Package 'openssl', required by 'libimobiledevice-1.0', not found
解决⽅法
ln -s /usr/local/Cellar/openssl/${YOUR_OPENSSL_VERSION}/lib/pkgconfig/* /usr/local/lib/pkgconfig/
2.3.3 安装idb 及 idb_companion
brew tap facebook/fb
brew install idb-companion
brew tap cartr/qt4
brew tap-pin cartr/qt4
brew install cartr/qt4/qt@4
pip3.6 install fb-idb
brew install cmake usbmuxd libimobiledevice
执⾏
$ idb list-targets
...
iPhone X | 569C0F94-5D53-40D2-AF8F-F4AA5BAA7D5E | Shutdown | simulator | iOS 12.2 | x86_64 | No Companion Connected iPhone Xs | 2A1C6A5A-0C67-46FD-B3F5-3CB42FFB38B5 | Shutdown | simulator | iOS 12.2 | x86_64 | No Companion Connected iPhone Xs Max | D3CF178F-EF61-4CD3-
BB3B-F5ECAD246310 | Shutdown | simulator | iOS 12.2 | x86_64 | No Companion Connected iPhone Xʀ | 74064851-4B98-473A-8110-225202BB86F6 | Shutdown | simulator | iOS 12.2 | x86_64 | No Companion Connected ...
遇到问题1
Error: An unexpected error occurred during the `brew link` step
The formula built, but is not symlinked into /usr/local
Permission denied @ dir_s_mkdir - /usr/local/Frameworks
Error: Permission denied @ dir_s_mkdir - /usr/local/Frameworks
解决
sudo chown -R $(whoami):admin /usr/local/**
sudo mkdir /usr/local/Frameworks
sudo chown $(whoami):admin /usr/local/Frameworks
遇到问题2
执⾏idb命令报错
TypeError: __init__() got an unexpected keyword argument 'serialized_options'
解决
pip3.6 install -U protobuf
2.3.4 安undletool
cd ${ANDROID_HOME}
mkdir bundle-tools
wget github/google/bundletool/releases/download/0.10.1/bundletool-all-0.10.1.jar
mv bundletool-all-0.10.1.jar bundletool.jar
sudo chmod 755 bundletool.jar
sudo chown $(whoami):admin bundletool.jar
2.3.5 安装剩余依赖
npm install -g ffmpeg
npm install -g mjpeg-consumer
brew tap wix/brew
brew install applesimutils
brew install ios-deploy
brew install ios-webkit-debug-proxy
brew cask install osxfuse
brew install ifuse --HEAD
brew install carthage
brew install fbsimctl --HEAD
gem install xcpretty
2.4 验证安装
通过appium-doctor的命令来检查当前appium安装是否完善，当前的JDK、SDK等环境是否配置正确。
npm install -g appium-doctor
appium-doctor
2.5 运⾏测试demo准备
2.5.1 下载。
git clone github/appium/appium.git
2.5.2 启动appium 服务
appium
2.5.3 安装客户端驱动程序
cd到${code_dir}/appium/sample-code/python⽬录下，运⾏下⾯命令就可以运⾏⽰例了，appium本⾝⽀持多种语⾔的客户端，我这⾥语⾔选择的是python
npm install -g appium-uiautomator2-driver
npm install -g webdriverio
npm install -g app-inspector
pip3.6 install -
pip3.6 install -U facebook-wda
2.6 在模拟器和真机上运⾏测试脚本
2.6.1 ⽤Android 模拟器进⾏测试
报错
E WebDriverException: Message: An unknown server-side error occurred while processing the command. Original error: Unable to find an active device or emulator with OS 8.0. The following are available: emulator-5554 (9)
这⾥已经提⽰的很清楚了，就是没有到Android 8.0版本的设备或者模拟器，发现了可⽤的设备，Android平台版本是9，设备版本是emulator-5554，根据提⽰修改helpers.py的默认版本或者设置环境变量，
ANDROID_BASE_CAPS = {
'app': os.path.abspath('../apps/ApiDemos-debug.apk'),
'automationName': 'UIAutomator2',
'platformName': 'Android',
'platformVersion': os.getenv('ANDROID_PLATFORM_VERSION') or '9',
'deviceName': os.getenv('ANDROID_DEVICE_VERSION') or 'emulator-5554',
}
或者
export ANDROID_PLATFORM_VERSION=9
export ANDROID_DEVICE_VERSION=emulator-5554
注释掉helpers.py中上报到saucelabs的调⽤
def fin():
# report_to_sauce(driver.session_id)
export SAUCE_LABS=
export SAUCE_USERNAME=
export SAUCE_ACCESS_KEY=
然后再执⾏测试脚本
2.6.2 ⽤Android真机测试
2.6.2.1 连接⼿机
使⽤数据线通过USB接⼝将⼿机与电脑连接，在终端中执⾏如下命令:
adb devices
发布评论