# python获取网页内容

By [KeepLearning](https://paragraph.com/@keeplearning-2) · 2022-12-25

---

20221225

分三步：

response = requests.get(url, headers=headers)

res=json.loads(response.text)////res = response.json()

#下面这一步还没弄清楚什么时候能成功，什么时候会失败。但是需要的数据在第二步肯定可以了。字典可以按照下面方法变成df,如果出现

if using all scalar values you must pass an index

告警,则加上index=\[0\]这个参数

df = pd.DataFrame(response.json()\['data'\]\['list'\]\[0\], index=\[0\])

df=pd.DataFrame(res)

import json import requests import pandas as pd

#headers的值按F15,然后F5刷新后可以看到，url也在里面找，参考下图，有些网页内容不用

headers也能抓下来，但是有些就不可以。至于原因参考以下帖子，说是为了反扒。

[https://zhuanlan.zhihu.com/p/147175546](https://zhuanlan.zhihu.com/p/147175546)

![](https://storage.googleapis.com/papyrus_images/5d3b7071ab593e32dbaf2ef27cd373e9d3972a19bc009c481ab16606a55b3aa9.png)

headers = { "authority": "web3alerts.app", "accept": "application/json", "user-agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36", "sec-fetch-site": "same-origin", "sec-fetch-mode": "cors", "sec-fetch-dest": "empty", "referer": "[https://web3alerts.app/](https://web3alerts.app/)", "accept-language": "zh-CN,zh;q=0.9" } url = "[https://web3alerts.app/api/new\_projects](https://web3alerts.app/api/new_projects)" response = requests.get(url, headers=headers)

res=json.loads(response.text) df=pd.DataFrame(res)

---

*Originally published on [KeepLearning](https://paragraph.com/@keeplearning-2/python-2)*