标签归档:python-requests

使用Python中的请求库发送“用户代理”

问题:使用Python中的请求库发送“用户代理”

我想在"User-agent"使用Python请求请求网页时发送的值。我不确定是否可以将其作为标头的一部分发送,如以下代码所示:

debug = {'verbose': sys.stderr}
user_agent = {'User-agent': 'Mozilla/5.0'}
response  = requests.get(url, headers = user_agent, config=debug)

调试信息未显示请求期间发送的标头。

在标头中发送此信息是否可以接受?如果没有,我该如何发送?

I want to send a value for "User-agent" while requesting a webpage using Python Requests. I am not sure is if it is okay to send this as a part of the header, as in the code below:

debug = {'verbose': sys.stderr}
user_agent = {'User-agent': 'Mozilla/5.0'}
response  = requests.get(url, headers = user_agent, config=debug)

The debug information isn’t showing the headers being sent during the request.

Is it acceptable to send this information in the header? If not, how can I send it?


回答 0

user-agent应指定为在报头中的字段。

这是HTTP标头字段列表,您可能会对特定请求的字段感兴趣,其中包括User-Agent

如果您使用的是v2.13及更高版本的请求

执行所需操作的最简单方法是创建字典并直接指定标题,例如:

import requests

url = 'SOME URL'

headers = {
    'User-Agent': 'My User Agent 1.0',
    'From': 'youremail@domain.com'  # This is another valid field
}

response = requests.get(url, headers=headers)

如果您使用的是v2.12.x及更高版本的请求

旧版本的requests默认标头已损坏,因此您需要执行以下操作来保留默认标头,然后向其添加自己的标头。

import requests

url = 'SOME URL'

# Get a copy of the default headers that requests would use
headers = requests.utils.default_headers()

# Update the headers with your custom ones
# You don't have to worry about case-sensitivity with
# the dictionary keys, because default_headers uses a custom
# CaseInsensitiveDict implementation within requests' source code.
headers.update(
    {
        'User-Agent': 'My User Agent 1.0',
    }
)

response = requests.get(url, headers=headers)

The user-agent should be specified as a field in the header.

Here is a list of HTTP header fields, and you’d probably be interested in request-specific fields, which includes User-Agent.

If you’re using requests v2.13 and newer

The simplest way to do what you want is to create a dictionary and specify your headers directly, like so:

import requests

url = 'SOME URL'

headers = {
    'User-Agent': 'My User Agent 1.0',
    'From': 'youremail@domain.com'  # This is another valid field
}

response = requests.get(url, headers=headers)

If you’re using requests v2.12.x and older

Older versions of requests clobbered default headers, so you’d want to do the following to preserve default headers and then add your own to them.

import requests

url = 'SOME URL'

# Get a copy of the default headers that requests would use
headers = requests.utils.default_headers()

# Update the headers with your custom ones
# You don't have to worry about case-sensitivity with
# the dictionary keys, because default_headers uses a custom
# CaseInsensitiveDict implementation within requests' source code.
headers.update(
    {
        'User-Agent': 'My User Agent 1.0',
    }
)

response = requests.get(url, headers=headers)

回答 1

使用session更方便,这样您就不必每次都设置标题了:

session = requests.Session()
session.headers.update({'User-Agent': 'Custom user agent'})

session.get('https://httpbin.org/headers')

默认情况下,会话也会为您管理cookie。如果您要禁用该功能,请参阅此问题

It’s more convenient to use a session, this way you don’t have to remember to set headers each time:

session = requests.Session()
session.headers.update({'User-Agent': 'Custom user agent'})

session.get('https://httpbin.org/headers')

By default, session also manages cookies for you. In case you want to disable that, see this question.


如何在python中发送带有请求的“ multipart / form-data”?

问题:如何在python中发送带有请求的“ multipart / form-data”?

如何multipart/form-data在python中发送带有请求的请求?我了解如何发送文件,但是如何使用这种方法发送表单数据尚不清楚。

How to send a multipart/form-data with requests in python? How to send a file, I understand, but how to send the form data by this method can not understand.


回答 0

基本上,如果您指定files参数(字典),requests则将发送multipart/form-dataPOST而不是application/x-www-form-urlencodedPOST。您不限于在该词典中使用实际文件,但是:

>>> import requests
>>> response = requests.post('http://httpbin.org/post', files=dict(foo='bar'))
>>> response.status_code
200

httpbin.org可以让您知道您发布了哪些标题;在response.json()我们有:

>>> from pprint import pprint
>>> pprint(response.json()['headers'])
{'Accept': '*/*',
 'Accept-Encoding': 'gzip, deflate',
 'Connection': 'close',
 'Content-Length': '141',
 'Content-Type': 'multipart/form-data; '
                 'boundary=c7cbfdd911b4e720f1dd8f479c50bc7f',
 'Host': 'httpbin.org',
 'User-Agent': 'python-requests/2.21.0'}

更好的是,您可以通过使用元组而不是单个字符串或字节对象来进一步控制每个部分的文件名,内容类型和其他标题。元组应包含2到4个元素;文件名,内容,可选的内容类型以及其他标头的可选字典。

我将使用元组形式None作为文件名,以便filename="..."从那些部分的请求中删除参数:

>>> files = {'foo': 'bar'}
>>> print(requests.Request('POST', 'http://httpbin.org/post', files=files).prepare().body.decode('utf8'))
--bb3f05a247b43eede27a124ef8b968c5
Content-Disposition: form-data; name="foo"; filename="foo"

bar
--bb3f05a247b43eede27a124ef8b968c5--
>>> files = {'foo': (None, 'bar')}
>>> print(requests.Request('POST', 'http://httpbin.org/post', files=files).prepare().body.decode('utf8'))
--d5ca8c90a869c5ae31f70fa3ddb23c76
Content-Disposition: form-data; name="foo"

bar
--d5ca8c90a869c5ae31f70fa3ddb23c76--

files 如果需要排序和/或具有相同名称的多个字段,也可以是二值元组的列表:

requests.post(
    'http://requestb.in/xucj9exu',
    files=(
        ('foo', (None, 'bar')),
        ('foo', (None, 'baz')),
        ('spam', (None, 'eggs')),
    )
)

如果同时指定filesdata,然后它取决于价值data东西将被用于创建POST体。如果data是字符串,则仅使用它将;否则datafiles都使用和,data首先列出元素。

还有一个出色的requests-toolbelt项目,其中包括高级的Multipart支持。它采用与files参数格式相同的字段定义,但是与不同requests,它默认不设置文件名参数。另外,它可以从打开的文件对象流式传输请求,requests首先将在内存中构造请求主体:

from requests_toolbelt.multipart.encoder import MultipartEncoder

mp_encoder = MultipartEncoder(
    fields={
        'foo': 'bar',
        # plain file object, no filename or mime type produces a
        # Content-Disposition header with just the part name
        'spam': ('spam.txt', open('spam.txt', 'rb'), 'text/plain'),
    }
)
r = requests.post(
    'http://httpbin.org/post',
    data=mp_encoder,  # The MultipartEncoder is posted as data, don't use files=...!
    # The MultipartEncoder provides the content-type header with the boundary:
    headers={'Content-Type': mp_encoder.content_type}
)

字段遵循相同的约定;使用包含2到4个元素的元组来添加文件名,部分mime类型或额外的标头。不像files参数,没有试图找到一个默认filename值,如果你不使用的元组。

Basically, if you specify a files parameter (a dictionary), then requests will send a multipart/form-data POST instead of a application/x-www-form-urlencoded POST. You are not limited to using actual files in that dictionary, however:

>>> import requests
>>> response = requests.post('http://httpbin.org/post', files=dict(foo='bar'))
>>> response.status_code
200

and httpbin.org lets you know what headers you posted with; in response.json() we have:

>>> from pprint import pprint
>>> pprint(response.json()['headers'])
{'Accept': '*/*',
 'Accept-Encoding': 'gzip, deflate',
 'Connection': 'close',
 'Content-Length': '141',
 'Content-Type': 'multipart/form-data; '
                 'boundary=c7cbfdd911b4e720f1dd8f479c50bc7f',
 'Host': 'httpbin.org',
 'User-Agent': 'python-requests/2.21.0'}

Better still, you can further control the filename, content type and additional headers for each part by using a tuple instead of a single string or bytes object. The tuple is expected to contain between 2 and 4 elements; the filename, the content, optionally a content type, and an optional dictionary of further headers.

I’d use the tuple form with None as the filename, so that the filename="..." parameter is dropped from the request for those parts:

>>> files = {'foo': 'bar'}
>>> print(requests.Request('POST', 'http://httpbin.org/post', files=files).prepare().body.decode('utf8'))
--bb3f05a247b43eede27a124ef8b968c5
Content-Disposition: form-data; name="foo"; filename="foo"

bar
--bb3f05a247b43eede27a124ef8b968c5--
>>> files = {'foo': (None, 'bar')}
>>> print(requests.Request('POST', 'http://httpbin.org/post', files=files).prepare().body.decode('utf8'))
--d5ca8c90a869c5ae31f70fa3ddb23c76
Content-Disposition: form-data; name="foo"

bar
--d5ca8c90a869c5ae31f70fa3ddb23c76--

files can also be a list of two-value tuples, if you need ordering and/or multiple fields with the same name:

requests.post(
    'http://requestb.in/xucj9exu',
    files=(
        ('foo', (None, 'bar')),
        ('foo', (None, 'baz')),
        ('spam', (None, 'eggs')),
    )
)

If you specify both files and data, then it depends on the value of data what will be used to create the POST body. If data is a string, only it willl be used; otherwise both data and files are used, with the elements in data listed first.

There is also the excellent requests-toolbelt project, which includes advanced Multipart support. It takes field definitions in the same format as the files parameter, but unlike requests, it defaults to not setting a filename parameter. In addition, it can stream the request from open file objects, where requests will first construct the request body in memory:

from requests_toolbelt.multipart.encoder import MultipartEncoder

mp_encoder = MultipartEncoder(
    fields={
        'foo': 'bar',
        # plain file object, no filename or mime type produces a
        # Content-Disposition header with just the part name
        'spam': ('spam.txt', open('spam.txt', 'rb'), 'text/plain'),
    }
)
r = requests.post(
    'http://httpbin.org/post',
    data=mp_encoder,  # The MultipartEncoder is posted as data, don't use files=...!
    # The MultipartEncoder provides the content-type header with the boundary:
    headers={'Content-Type': mp_encoder.content_type}
)

Fields follow the same conventions; use a tuple with between 2 and 4 elements to add a filename, part mime-type or extra headers. Unlike the files parameter, no attempt is made to find a default filename value if you don’t use a tuple.


回答 1

自从编写了先前的答案以来,请求已更改。请查看Github上的bug线程以获取更多详细信息,并以示例的形式发表评论

简而言之,files参数使用a dict作为参数,键是表单字段的名称,值是字符串或2、3或4个长度的元组,如在请求中发布多部分编码文件一节中所述快速开始:

>>> url = 'http://httpbin.org/post'
>>> files = {'file': ('report.xls', open('report.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}

在上面,元组的组成如下:

(filename, data, content_type, headers)

如果该值只是一个字符串,则文件名将与键相同,如下所示:

>>> files = {'obvius_session_id': '72c2b6f406cdabd578c5fd7598557c52'}

Content-Disposition: form-data; name="obvius_session_id"; filename="obvius_session_id"
Content-Type: application/octet-stream

72c2b6f406cdabd578c5fd7598557c52

如果值是一个元组,并且第一个条目是Nonefilename属性,将不包括在内:

>>> files = {'obvius_session_id': (None, '72c2b6f406cdabd578c5fd7598557c52')}

Content-Disposition: form-data; name="obvius_session_id"
Content-Type: application/octet-stream

72c2b6f406cdabd578c5fd7598557c52

Since the previous answers were written, requests have changed. Have a look at the bug thread at Github for more detail and this comment for an example.

In short, the files parameter takes a dict with the key being the name of the form field and the value being either a string or a 2, 3 or 4-length tuple, as described in the section POST a Multipart-Encoded File in the requests quickstart:

>>> url = 'http://httpbin.org/post'
>>> files = {'file': ('report.xls', open('report.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}

In the above, the tuple is composed as follows:

(filename, data, content_type, headers)

If the value is just a string, the filename will be the same as the key, as in the following:

>>> files = {'obvius_session_id': '72c2b6f406cdabd578c5fd7598557c52'}

Content-Disposition: form-data; name="obvius_session_id"; filename="obvius_session_id"
Content-Type: application/octet-stream

72c2b6f406cdabd578c5fd7598557c52

If the value is a tuple and the first entry is None the filename property will not be included:

>>> files = {'obvius_session_id': (None, '72c2b6f406cdabd578c5fd7598557c52')}

Content-Disposition: form-data; name="obvius_session_id"
Content-Type: application/octet-stream

72c2b6f406cdabd578c5fd7598557c52

回答 2

即使不需要上传任何文件,也需要使用该files参数发送多部分表单POST请求。

从原始请求来源:

def request(method, url, **kwargs):
    """Constructs and sends a :class:`Request <Request>`.

    ...
    :param files: (optional) Dictionary of ``'name': file-like-objects``
        (or ``{'name': file-tuple}``) for multipart encoding upload.
        ``file-tuple`` can be a 2-tuple ``('filename', fileobj)``,
        3-tuple ``('filename', fileobj, 'content_type')``
        or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``,
        where ``'content-type'`` is a string
        defining the content type of the given file
        and ``custom_headers`` a dict-like object 
        containing additional headers to add for the file.

相关部分是: file-tuple can be a2-tuple,。3-tupleor a4-tuple

基于上述内容,最简单的多部分表单请求包括要上传的文件和表单字段,如下所示:

multipart_form_data = {
    'file2': ('custom_file_name.zip', open('myfile.zip', 'rb')),
    'action': (None, 'store'),
    'path': (None, '/path1')
}

response = requests.post('https://httpbin.org/post', files=multipart_form_data)

print(response.content)

请注意,None作为纯文本字段的元组中的第一个参数-这是文件名字段的占位符,仅用于文件上传,但对于文本字段,传递None第一个参数是必需的,以便提交数据。

具有相同名称的多个字段

如果您需要发布多个具有相同名称的字段,那么您可以将有效负载定义为元组列表(或元组),而不是字典:

multipart_form_data = (
    ('file2', ('custom_file_name.zip', open('myfile.zip', 'rb'))),
    ('action', (None, 'store')),
    ('path', (None, '/path1')),
    ('path', (None, '/path2')),
    ('path', (None, '/path3')),
)

流请求API

如果上述API对您来说还不够Python,那么请考虑使用request工具带pip install requests_toolbelt),它是核心请求模块的扩展,该模块提供对文件上传流的支持以及MultipartEncoder(可以代替来使用)files,并且还可以您可以将有效负载定义为字典,元组或列表。

MultipartEncoder可以用于有或没有实际上传字段的多部分请求。必须将其分配给data参数。

import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder

multipart_data = MultipartEncoder(
    fields={
            # a file upload field
            'file': ('file.zip', open('file.zip', 'rb'), 'text/plain')
            # plain text fields
            'field0': 'value0', 
            'field1': 'value1',
           }
    )

response = requests.post('http://httpbin.org/post', data=multipart_data,
                  headers={'Content-Type': multipart_data.content_type})

如果您需要发送多个具有相同名称的字段,或者表单字段的顺序很重要,则可以使用元组或列表代替字典:

multipart_data = MultipartEncoder(
    fields=(
            ('action', 'ingest'), 
            ('item', 'spam'),
            ('item', 'sausage'),
            ('item', 'eggs'),
           )
    )

You need to use the files parameter to send a multipart form POST request even when you do not need to upload any files.

From the original requests source:

def request(method, url, **kwargs):
    """Constructs and sends a :class:`Request <Request>`.

    ...
    :param files: (optional) Dictionary of ``'name': file-like-objects``
        (or ``{'name': file-tuple}``) for multipart encoding upload.
        ``file-tuple`` can be a 2-tuple ``('filename', fileobj)``,
        3-tuple ``('filename', fileobj, 'content_type')``
        or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``,
        where ``'content-type'`` is a string
        defining the content type of the given file
        and ``custom_headers`` a dict-like object 
        containing additional headers to add for the file.

The relevant part is: file-tuple can be a2-tuple, 3-tupleor a4-tuple.

Based on the above, the simplest multipart form request that includes both files to upload and form fields will look like this:

multipart_form_data = {
    'file2': ('custom_file_name.zip', open('myfile.zip', 'rb')),
    'action': (None, 'store'),
    'path': (None, '/path1')
}

response = requests.post('https://httpbin.org/post', files=multipart_form_data)

print(response.content)

Note the None as the first argument in the tuple for plain text fields — this is a placeholder for the filename field which is only used for file uploads, but for text fields passing None as the first parameter is required in order for the data to be submitted.

Multiple fields with the same name

If you need to post multiple fields with the same name then instead of a dictionary you can define your payload as a list (or a tuple) of tuples:

multipart_form_data = (
    ('file2', ('custom_file_name.zip', open('myfile.zip', 'rb'))),
    ('action', (None, 'store')),
    ('path', (None, '/path1')),
    ('path', (None, '/path2')),
    ('path', (None, '/path3')),
)

Streaming requests API

If the above API is not pythonic enough for you, then consider using requests toolbelt (pip install requests_toolbelt) which is an extension of the core requests module that provides support for file upload streaming as well as the MultipartEncoder which can be used instead of files, and which also lets you define the payload as a dictionary, tuple or list.

MultipartEncoder can be used both for multipart requests with or without actual upload fields. It must be assigned to the data parameter.

import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder

multipart_data = MultipartEncoder(
    fields={
            # a file upload field
            'file': ('file.zip', open('file.zip', 'rb'), 'text/plain')
            # plain text fields
            'field0': 'value0', 
            'field1': 'value1',
           }
    )

response = requests.post('http://httpbin.org/post', data=multipart_data,
                  headers={'Content-Type': multipart_data.content_type})

If you need to send multiple fields with the same name, or if the order of form fields is important, then a tuple or a list can be used instead of a dictionary:

multipart_data = MultipartEncoder(
    fields=(
            ('action', 'ingest'), 
            ('item', 'spam'),
            ('item', 'sausage'),
            ('item', 'eggs'),
           )
    )

回答 3

以下是使用请求上传带有其他参数的单个文件的简单代码段:

url = 'https://<file_upload_url>'
fp = '/Users/jainik/Desktop/data.csv'

files = {'file': open(fp, 'rb')}
payload = {'file_id': '1234'}

response = requests.put(url, files=files, data=payload, verify=False)

请注意,您不需要显式指定任何内容类型。

注意:想对以上答案之一发表评论,但由于声誉不佳而无法发表评论,因此在此处起草了一个新的答复。

Here is the simple code snippet to upload a single file with additional parameters using requests:

url = 'https://<file_upload_url>'
fp = '/Users/jainik/Desktop/data.csv'

files = {'file': open(fp, 'rb')}
payload = {'file_id': '1234'}

response = requests.put(url, files=files, data=payload, verify=False)

Please note that you don’t need to explicitly specify any content type.

NOTE: Wanted to comment on one of the above answers but could not because of low reputation so drafted a new response here.


回答 4

您需要使用name网站HTML中的上传文件的属性。例:

autocomplete="off" name="image">

看到了 name="image">吗 您可以在用于上传文件的网站的HTML中找到它。您需要使用它来上传文件Multipart/form-data

脚本:

import requests

site = 'https://prnt.sc/upload.php' # the site where you upload the file
filename = 'image.jpg'  # name example

在这里,在图片的位置,以HTML添加上传文件的名称

up = {'image':(filename, open(filename, 'rb'), "multipart/form-data")}

如果上传需要单击上传按钮,则可以这样使用:

data = {
     "Button" : "Submit",
}

然后开始请求

request = requests.post(site, files=up, data=data)

完成,文件成功上传

You need to use the name attribute of the upload file that is in the HTML of the site. Example:

autocomplete="off" name="image">

You see name="image">? You can find it in the HTML of a site for uploading the file. You need to use it to upload the file with Multipart/form-data

script:

import requests

site = 'https://prnt.sc/upload.php' # the site where you upload the file
filename = 'image.jpg'  # name example

Here, in the place of image, add the name of the upload file in HTML

up = {'image':(filename, open(filename, 'rb'), "multipart/form-data")}

If the upload requires to click the button for upload, you can use like that:

data = {
     "Button" : "Submit",
}

Then start the request

request = requests.post(site, files=up, data=data)

And done, file uploaded succesfully


回答 5

发送多部分/表单数据键和值

curl命令:

curl -X PUT http://127.0.0.1:8080/api/xxx ...
-H 'content-type: multipart/form-data; boundary=----xxx' \
-F taskStatus=1

python 请求-更复杂的POST请求

    updateTaskUrl = "http://127.0.0.1:8080/api/xxx"
    updateInfoDict = {
        "taskStatus": 1,
    }
    resp = requests.put(updateTaskUrl, data=updateInfoDict)

发送多部分/表单数据文件

curl命令:

curl -X POST http://127.0.0.1:8080/api/xxx ...
-H 'content-type: multipart/form-data; boundary=----xxx' \
-F file=@/Users/xxx.txt

python 请求-发布多部分编码的文件

    filePath = "/Users/xxx.txt"
    fileFp = open(filePath, 'rb')
    fileInfoDict = {
        "file": fileFp,
    }
    resp = requests.post(uploadResultUrl, files=fileInfoDict)

就这样。

Send multipart/form-data key and value

curl command:

curl -X PUT http://127.0.0.1:8080/api/xxx ...
-H 'content-type: multipart/form-data; boundary=----xxx' \
-F taskStatus=1

python requests – More complicated POST requests:

    updateTaskUrl = "http://127.0.0.1:8080/api/xxx"
    updateInfoDict = {
        "taskStatus": 1,
    }
    resp = requests.put(updateTaskUrl, data=updateInfoDict)

Send multipart/form-data file

curl command:

curl -X POST http://127.0.0.1:8080/api/xxx ...
-H 'content-type: multipart/form-data; boundary=----xxx' \
-F file=@/Users/xxx.txt

python requests – POST a Multipart-Encoded File:

    filePath = "/Users/xxx.txt"
    fileFp = open(filePath, 'rb')
    fileInfoDict = {
        "file": fileFp,
    }
    resp = requests.post(uploadResultUrl, files=fileInfoDict)

that’s all.


回答 6

这是您需要将一个大的单个文件作为多部分表单数据上传的python代码段。使用NodeJs Multer中间件在服务器端运行。

import requests
latest_file = 'path/to/file'
url = "http://httpbin.org/apiToUpload"
files = {'fieldName': open(latest_file, 'rb')}
r = requests.put(url, files=files)

对于服务器端,请在以下位置查看multer文档:https : //github.com/expressjs/multer, 此处字段single(’fieldName’)用于接受一个文件,如下所示:

var upload = multer().single('fieldName');

Here is the python snippet you need to upload one large single file as multipart formdata. With NodeJs Multer middleware running on the server side.

import requests
latest_file = 'path/to/file'
url = "http://httpbin.org/apiToUpload"
files = {'fieldName': open(latest_file, 'rb')}
r = requests.put(url, files=files)

For the server side please check the multer documentation at: https://github.com/expressjs/multer here the field single(‘fieldName’) is used to accept one single file, as in:

var upload = multer().single('fieldName');

Python请求-无连接适配器

问题:Python请求-无连接适配器

我正在使用Requests:HTTP for Humans库,但出现了这个奇怪的错误,我不知道这是什么意思。

No connection adapters were found for '192.168.1.61:8080/api/call'

有人有主意吗?

I’m using the Requests: HTTP for Humans library and I got this weird error and I don’t know what is mean.

No connection adapters were found for '192.168.1.61:8080/api/call'

Anybody has an idea?


回答 0

您需要包括协议方案:

'http://192.168.1.61:8080/api/call'

没有这个http://部分,requests就不知道如何连接到远程服务器。

请注意,协议方案必须全部为小写。如果您的URL以HTTP://例如开头,则也找不到http://连接适配器。

You need to include the protocol scheme:

'http://192.168.1.61:8080/api/call'

Without the http:// part, requests has no idea how to connect to the remote server.

Note that the protocol scheme must be all lowercase; if your URL starts with HTTP:// for example, it won’t find the http:// connection adapter either.


回答 1

另一个原因,也许您的URL包含一些隐藏的字符,例如’\ n’。

如果您按如下方式定义网址,则会引发此异常:

url = '''
http://google.com
'''

因为字符串中有’\ n’隐藏。该网址实际上变为:

\nhttp://google.com\n

One more reason, maybe your url include some hiden characters, such as ‘\n’.

If you define your url like below, this exception will raise:

url = '''
http://google.com
'''

because there are ‘\n’ hide in the string. The url in fact become:

\nhttp://google.com\n

Python请求-打印整个HTTP请求(原始)?

问题:Python请求-打印整个HTTP请求(原始)?

在使用requests模块时,有什么方法可以打印原始HTTP请求?

我不仅想要标题,还想要请求行,标题和内容打印输出。是否有可能看到HTTP请求最终构造了什么?

While using the requests module, is there any way to print the raw HTTP request?

I don’t want just the headers, I want the request line, headers, and content printout. Is it possible to see what ultimately is constructed from HTTP request?


回答 0

从v1.2.3开始,请求添加了PreparedRequest对象。按照文档“它包含将发送到服务器的确切字节”。

可以使用它来漂亮地打印请求,如下所示:

import requests

req = requests.Request('POST','http://stackoverflow.com',headers={'X-Custom':'Test'},data='a=1&b=2')
prepared = req.prepare()

def pretty_print_POST(req):
    """
    At this point it is completely built and ready
    to be fired; it is "prepared".

    However pay attention at the formatting used in 
    this function because it is programmed to be pretty 
    printed and may differ from the actual request.
    """
    print('{}\n{}\r\n{}\r\n\r\n{}'.format(
        '-----------START-----------',
        req.method + ' ' + req.url,
        '\r\n'.join('{}: {}'.format(k, v) for k, v in req.headers.items()),
        req.body,
    ))

pretty_print_POST(prepared)

生成:

-----------START-----------
POST http://stackoverflow.com/
Content-Length: 7
X-Custom: Test

a=1&b=2

然后,您可以使用以下命令发送实际请求:

s = requests.Session()
s.send(prepared)

这些链接是可用的最新文档,因此它们的内容可能会发生变化: 高级-准备的请求API-较低级别的类

Since v1.2.3 Requests added the PreparedRequest object. As per the documentation “it contains the exact bytes that will be sent to the server”.

One can use this to pretty print a request, like so:

import requests

req = requests.Request('POST','http://stackoverflow.com',headers={'X-Custom':'Test'},data='a=1&b=2')
prepared = req.prepare()

def pretty_print_POST(req):
    """
    At this point it is completely built and ready
    to be fired; it is "prepared".

    However pay attention at the formatting used in 
    this function because it is programmed to be pretty 
    printed and may differ from the actual request.
    """
    print('{}\n{}\r\n{}\r\n\r\n{}'.format(
        '-----------START-----------',
        req.method + ' ' + req.url,
        '\r\n'.join('{}: {}'.format(k, v) for k, v in req.headers.items()),
        req.body,
    ))

pretty_print_POST(prepared)

which produces:

-----------START-----------
POST http://stackoverflow.com/
Content-Length: 7
X-Custom: Test

a=1&b=2

Then you can send the actual request with this:

s = requests.Session()
s.send(prepared)

These links are to the latest documentation available, so they might change in content: Advanced – Prepared requests and API – Lower level classes


回答 1

import requests
response = requests.post('http://httpbin.org/post', data={'key1':'value1'})
print(response.request.body)
print(response.request.headers)

我正在使用请求版本2.18.4和Python 3

import requests
response = requests.post('http://httpbin.org/post', data={'key1':'value1'})
print(response.request.body)
print(response.request.headers)

I am using requests version 2.18.4 and Python 3


回答 2

注意:此答案已过时。requests 作为AntonioHerraizS的答复文件,较新的支持版本直接获取请求内容

无法获得请求的真实原始内容requests,因为它仅处理更高级别的对象,例如标头方法类型requests利用urllib3发送请求,但urllib3 不能与原始数据处理-它使用httplib。这是请求的代表性堆栈跟踪:

-> r= requests.get("http://google.com")
  /usr/local/lib/python2.7/dist-packages/requests/api.py(55)get()
-> return request('get', url, **kwargs)
  /usr/local/lib/python2.7/dist-packages/requests/api.py(44)request()
-> return session.request(method=method, url=url, **kwargs)
  /usr/local/lib/python2.7/dist-packages/requests/sessions.py(382)request()
-> resp = self.send(prep, **send_kwargs)
  /usr/local/lib/python2.7/dist-packages/requests/sessions.py(485)send()
-> r = adapter.send(request, **kwargs)
  /usr/local/lib/python2.7/dist-packages/requests/adapters.py(324)send()
-> timeout=timeout
  /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py(478)urlopen()
-> body=body, headers=headers)
  /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py(285)_make_request()
-> conn.request(method, url, **httplib_request_kw)
  /usr/lib/python2.7/httplib.py(958)request()
-> self._send_request(method, url, body, headers)

httplib机器内部,我们可以看到HTTPConnection._send_request间接使用HTTPConnection._send_output,它最终创建了原始请求主体(如果存在),并使用HTTPConnection.send了分别发送它们。send终于到达插座。

由于没有钩子可以做您想做的事情,因此,万不得已时,您可以Monkey补丁httplib来获取内容。这是一个脆弱的解决方案,如果httplib进行了更改,您可能需要对其进行调整。如果您打算使用此解决方案分发软件,则可能要考虑打包httplib而不是使用系统包,这很容易,因为它是纯python模块。

las,不费吹灰之力,解决方案:

import requests
import httplib

def patch_send():
    old_send= httplib.HTTPConnection.send
    def new_send( self, data ):
        print data
        return old_send(self, data) #return is not necessary, but never hurts, in case the library is changed
    httplib.HTTPConnection.send= new_send

patch_send()
requests.get("http://www.python.org")

产生输出:

GET / HTTP/1.1
Host: www.python.org
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/2.1.0 CPython/2.7.3 Linux/3.2.0-23-generic-pae

Note: this answer is outdated. Newer versions of requests support getting the request content directly, as AntonioHerraizS’s answer documents.

It’s not possible to get the true raw content of the request out of requests, since it only deals with higher level objects, such as headers and method type. requests uses urllib3 to send requests, but urllib3 also doesn’t deal with raw data – it uses httplib. Here’s a representative stack trace of a request:

-> r= requests.get("http://google.com")
  /usr/local/lib/python2.7/dist-packages/requests/api.py(55)get()
-> return request('get', url, **kwargs)
  /usr/local/lib/python2.7/dist-packages/requests/api.py(44)request()
-> return session.request(method=method, url=url, **kwargs)
  /usr/local/lib/python2.7/dist-packages/requests/sessions.py(382)request()
-> resp = self.send(prep, **send_kwargs)
  /usr/local/lib/python2.7/dist-packages/requests/sessions.py(485)send()
-> r = adapter.send(request, **kwargs)
  /usr/local/lib/python2.7/dist-packages/requests/adapters.py(324)send()
-> timeout=timeout
  /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py(478)urlopen()
-> body=body, headers=headers)
  /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py(285)_make_request()
-> conn.request(method, url, **httplib_request_kw)
  /usr/lib/python2.7/httplib.py(958)request()
-> self._send_request(method, url, body, headers)

Inside the httplib machinery, we can see HTTPConnection._send_request indirectly uses HTTPConnection._send_output, which finally creates the raw request and body (if it exists), and uses HTTPConnection.send to send them separately. send finally reaches the socket.

Since there’s no hooks for doing what you want, as a last resort you can monkey patch httplib to get the content. It’s a fragile solution, and you may need to adapt it if httplib is changed. If you intend to distribute software using this solution, you may want to consider packaging httplib instead of using the system’s, which is easy, since it’s a pure python module.

Alas, without further ado, the solution:

import requests
import httplib

def patch_send():
    old_send= httplib.HTTPConnection.send
    def new_send( self, data ):
        print data
        return old_send(self, data) #return is not necessary, but never hurts, in case the library is changed
    httplib.HTTPConnection.send= new_send

patch_send()
requests.get("http://www.python.org")

which yields the output:

GET / HTTP/1.1
Host: www.python.org
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/2.1.0 CPython/2.7.3 Linux/3.2.0-23-generic-pae

回答 3

更好的主意是使用requests_toolbelt库,该库可以将请求和响应都作为字符串转储出去,以供您打印到控制台。它使用上述解决方案无法很好处理的文件和编码来处理所有棘手的情况。

就这么简单:

import requests
from requests_toolbelt.utils import dump

resp = requests.get('https://httpbin.org/redirect/5')
data = dump.dump_all(resp)
print(data.decode('utf-8'))

资料来源:https : //toolbelt.readthedocs.org/en/latest/dumputils.html

您只需输入以下内容即可安装:

pip install requests_toolbelt

An even better idea is to use the requests_toolbelt library, which can dump out both requests and responses as strings for you to print to the console. It handles all the tricky cases with files and encodings which the above solution does not handle well.

It’s as easy as this:

import requests
from requests_toolbelt.utils import dump

resp = requests.get('https://httpbin.org/redirect/5')
data = dump.dump_all(resp)
print(data.decode('utf-8'))

Source: https://toolbelt.readthedocs.org/en/latest/dumputils.html

You can simply install it by typing:

pip install requests_toolbelt

回答 4

这是一个代码,与上面相同,但是带有响应头:

import socket
def patch_requests():
    old_readline = socket._fileobject.readline
    if not hasattr(old_readline, 'patched'):
        def new_readline(self, size=-1):
            res = old_readline(self, size)
            print res,
            return res
        new_readline.patched = True
        socket._fileobject.readline = new_readline
patch_requests()

我花了很多时间寻找这个,所以如果有人需要,我就把它留在这里。

Here is a code, which makes the same, but with response headers:

import socket
def patch_requests():
    old_readline = socket._fileobject.readline
    if not hasattr(old_readline, 'patched'):
        def new_readline(self, size=-1):
            res = old_readline(self, size)
            print res,
            return res
        new_readline.patched = True
        socket._fileobject.readline = new_readline
patch_requests()

I spent a lot of time searching for this, so I’m leaving it here, if someone needs.


回答 5

我使用以下函数来格式化请求。就像@AntonioHerraizS一样,除了它还会在主体中漂亮地打印JSON对象,并标记请求的所有部分。

format_json = functools.partial(json.dumps, indent=2, sort_keys=True)
indent = functools.partial(textwrap.indent, prefix='  ')

def format_prepared_request(req):
    """Pretty-format 'requests.PreparedRequest'

    Example:
        res = requests.post(...)
        print(format_prepared_request(res.request))

        req = requests.Request(...)
        req = req.prepare()
        print(format_prepared_request(res.request))
    """
    headers = '\n'.join(f'{k}: {v}' for k, v in req.headers.items())
    content_type = req.headers.get('Content-Type', '')
    if 'application/json' in content_type:
        try:
            body = format_json(json.loads(req.body))
        except json.JSONDecodeError:
            body = req.body
    else:
        body = req.body
    s = textwrap.dedent("""
    REQUEST
    =======
    endpoint: {method} {url}
    headers:
    {headers}
    body:
    {body}
    =======
    """).strip()
    s = s.format(
        method=req.method,
        url=req.url,
        headers=indent(headers),
        body=indent(body),
    )
    return s

我有一个类似的函数来格式化响应:

def format_response(resp):
    """Pretty-format 'requests.Response'"""
    headers = '\n'.join(f'{k}: {v}' for k, v in resp.headers.items())
    content_type = resp.headers.get('Content-Type', '')
    if 'application/json' in content_type:
        try:
            body = format_json(resp.json())
        except json.JSONDecodeError:
            body = resp.text
    else:
        body = resp.text
    s = textwrap.dedent("""
    RESPONSE
    ========
    status_code: {status_code}
    headers:
    {headers}
    body:
    {body}
    ========
    """).strip()

    s = s.format(
        status_code=resp.status_code,
        headers=indent(headers),
        body=indent(body),
    )
    return s

I use the following function to format requests. It’s like @AntonioHerraizS except it will pretty-print JSON objects in the body as well, and it labels all parts of the request.

format_json = functools.partial(json.dumps, indent=2, sort_keys=True)
indent = functools.partial(textwrap.indent, prefix='  ')

def format_prepared_request(req):
    """Pretty-format 'requests.PreparedRequest'

    Example:
        res = requests.post(...)
        print(format_prepared_request(res.request))

        req = requests.Request(...)
        req = req.prepare()
        print(format_prepared_request(res.request))
    """
    headers = '\n'.join(f'{k}: {v}' for k, v in req.headers.items())
    content_type = req.headers.get('Content-Type', '')
    if 'application/json' in content_type:
        try:
            body = format_json(json.loads(req.body))
        except json.JSONDecodeError:
            body = req.body
    else:
        body = req.body
    s = textwrap.dedent("""
    REQUEST
    =======
    endpoint: {method} {url}
    headers:
    {headers}
    body:
    {body}
    =======
    """).strip()
    s = s.format(
        method=req.method,
        url=req.url,
        headers=indent(headers),
        body=indent(body),
    )
    return s

And I have a similar function to format the response:

def format_response(resp):
    """Pretty-format 'requests.Response'"""
    headers = '\n'.join(f'{k}: {v}' for k, v in resp.headers.items())
    content_type = resp.headers.get('Content-Type', '')
    if 'application/json' in content_type:
        try:
            body = format_json(resp.json())
        except json.JSONDecodeError:
            body = resp.text
    else:
        body = resp.text
    s = textwrap.dedent("""
    RESPONSE
    ========
    status_code: {status_code}
    headers:
    {headers}
    body:
    {body}
    ========
    """).strip()

    s = s.format(
        status_code=resp.status_code,
        headers=indent(headers),
        body=indent(body),
    )
    return s

回答 6

requests支持所谓的事件挂钩(从2.23开始,实际上只有response挂钩)。该钩子可用于请求以打印完整的请求-响应对数据,包括有效的URL,标头和正文,例如:

import textwrap
import requests

def print_roundtrip(response, *args, **kwargs):
    format_headers = lambda d: '\n'.join(f'{k}: {v}' for k, v in d.items())
    print(textwrap.dedent('''
        ---------------- request ----------------
        {req.method} {req.url}
        {reqhdrs}

        {req.body}
        ---------------- response ----------------
        {res.status_code} {res.reason} {res.url}
        {reshdrs}

        {res.text}
    ''').format(
        req=response.request, 
        res=response, 
        reqhdrs=format_headers(response.request.headers), 
        reshdrs=format_headers(response.headers), 
    ))

requests.get('https://httpbin.org/', hooks={'response': print_roundtrip})

运行它会打印:

---------------- request ----------------
GET https://httpbin.org/
User-Agent: python-requests/2.23.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

None
---------------- response ----------------
200 OK https://httpbin.org/
Date: Thu, 14 May 2020 17:16:13 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 9593
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

<!DOCTYPE html>
<html lang="en">
...
</html>

如果响应为二进制res.textres.content则可能需要更改为。

requests supports so called event hooks (as of 2.23 there’s actually only response hook). The hook can be used on a request to print full request-response pair’s data, including effective URL, headers and bodies, like:

import textwrap
import requests

def print_roundtrip(response, *args, **kwargs):
    format_headers = lambda d: '\n'.join(f'{k}: {v}' for k, v in d.items())
    print(textwrap.dedent('''
        ---------------- request ----------------
        {req.method} {req.url}
        {reqhdrs}

        {req.body}
        ---------------- response ----------------
        {res.status_code} {res.reason} {res.url}
        {reshdrs}

        {res.text}
    ''').format(
        req=response.request, 
        res=response, 
        reqhdrs=format_headers(response.request.headers), 
        reshdrs=format_headers(response.headers), 
    ))

requests.get('https://httpbin.org/', hooks={'response': print_roundtrip})

Running it prints:

---------------- request ----------------
GET https://httpbin.org/
User-Agent: python-requests/2.23.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

None
---------------- response ----------------
200 OK https://httpbin.org/
Date: Thu, 14 May 2020 17:16:13 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 9593
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

<!DOCTYPE html>
<html lang="en">
...
</html>

You may want to change res.text to res.content if the response is binary.


我可以为request.request设置max_retries吗?

问题:我可以为request.request设置max_retries吗?

Python的请求模块既简单又优雅,但有一件事困扰着我。有可能得到一个 requests.exception.ConnectionError有这样的消息:

Max retries exceeded with url: ...

这意味着请求可以尝试多次访问数据。但是在文档的任何地方都没有提及这种可能性。在源代码中,我没有找到可以更改默认值(大概为0)的地方。

那么是否有可能以某种方式设置请求的最大重试次数?

The Python requests module is simple and elegant but one thing bugs me. It is possible to get a requests.exception.ConnectionError with a message like:

Max retries exceeded with url: ...

This implies that requests can attempt to access the data several times. But there is not a single mention of this possibility anywhere in the docs. Looking at the source code I didn’t find any place where I could alter the default (presumably 0) value.

So is it possible to somehow set the maximum number of retries for requests?


回答 0

urllib3重试是底层库。要设置其他最大重试计数,请使用备用传输适配器

from requests.adapters import HTTPAdapter

s = requests.Session()
s.mount('http://stackoverflow.com', HTTPAdapter(max_retries=5))

max_retries参数接受一个整数或一个Retry()对象 ; 后者使您可以对重试哪种类型的故障进行细粒度的控制(将整数值转换为Retry()仅处理连接故障的实例;默认情况下,不处理连接后的错误,因为这些错误可能会导致副作用) 。


旧答案,早于请求1.2.1的发布

requests库实际上并没有使它可配置,也没有打算(请参阅此拉取请求))。当前(请求1.1),重试次数设置为0。如果您确实想将其设置为更高的值,则必须全局设置此值:

import requests

requests.adapters.DEFAULT_RETRIES = 5

此常量未记录;使用它的后果自负,因为将来的发行版可能会更改其处理方式。

更新:这确实改变了;在1.2.1版中,添加了在设置max_retries参数的选项,因此现在您必须使用替代的传输适配器,请参见上文。除非您也修补默认值(不建议这样做),否则Monkey修补方法将不再起作用。HTTPAdapter()HTTPAdapter.__init__()

It is the underlying urllib3 library that does the retrying. To set a different maximum retry count, use alternative transport adapters:

from requests.adapters import HTTPAdapter

s = requests.Session()
s.mount('http://stackoverflow.com', HTTPAdapter(max_retries=5))

The max_retries argument takes an integer or a Retry() object; the latter gives you fine-grained control over what kinds of failures are retried (an integer value is turned into a Retry() instance which only handles connection failures; errors after a connection is made are by default not handled as these could lead to side-effects).


Old answer, predating the release of requests 1.2.1:

The requests library doesn’t really make this configurable, nor does it intend to (see this pull request). Currently (requests 1.1), the retries count is set to 0. If you really want to set it to a higher value, you’ll have to set this globally:

import requests

requests.adapters.DEFAULT_RETRIES = 5

This constant is not documented; use it at your own peril as future releases could change how this is handled.

Update: and this did change; in version 1.2.1 the option to set the max_retries parameter on the HTTPAdapter() class was added, so that now you have to use alternative transport adapters, see above. The monkey-patch approach no longer works, unless you also patch the HTTPAdapter.__init__() defaults (very much not recommended).


回答 1

这不仅会更改max_retries,还会启用退避策略,该策略会使对所有http://地址的请求在重试之前休眠一段时间(总计5次):

import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

s = requests.Session()

retries = Retry(total=5,
                backoff_factor=0.1,
                status_forcelist=[ 500, 502, 503, 504 ])

s.mount('http://', HTTPAdapter(max_retries=retries))

s.get('http://httpstat.us/500')

根据文档说明Retry:如果backoff_factor为0.1,则sleep()将在重试之间睡眠[0.1s,0.2s,0.4s,…]。这也将迫使重试,如果返回的状态代码是500502503504

各种其他选择 Retry可以进行更精细的控制:

  • total –允许的重试总数。
  • 连接 –重试多少个与连接有关的错误。
  • 读取 -重试几次读取错误。
  • 重定向 -要执行多少重定向。
  • method_whitelist –我们应重试的大写HTTP方法动词集。
  • status_forcelist –我们应强制重试的一组HTTP状态代码。
  • backoff_factor –在两次尝试之间应用的退避因子。
  • raise_on_redirect –如果重定向次数已用尽,则引发MaxRetryError,还是返回响应代码在3xx范围内的响应。
  • raise_on_status -类似含义raise_on_redirect:我们是否应该抛出一个异常,或返回响应,如果状态落在status_forcelist范围和重试次数已经用尽。

注意raise_on_status相对较新,尚未将其发布到urllib3或请求中。 raise_on_status在python 3.6版中关键字自变量似乎最多已进入标准库。

要使请求重试特定的HTTP状态代码,请使用status_forcelist。例如,status_forcelist = [503]将重试状态码503(服务不可用)。

默认情况下,重试仅针对以下情况触发:

  • 无法从池获得连接。
  • TimeoutError
  • HTTPException(从Python 3中的http.client或其他httplib)引发。这似乎是低级HTTP异常,例如URL或协议格式不正确。
  • SocketError
  • ProtocolError

请注意,所有这些都是阻止接收常规HTTP响应的异常。如果生成任何常规响应,则不会重试。不使用status_forcelist,即使状态为500的响应也不会重试。

以使其以这样的方式,其是用于与远程API或web服务器工作的更直观的行为,我会用上面的代码段,其在状态力的重试500502503504,所有这些都并不少见上网络和(可能)在足够大的退避期后可以恢复。

编辑Retry直接从urllib3导入类。

This will not only change the max_retries but also enable a backoff strategy which makes requests to all http:// addresses sleep for a period of time before retrying (to a total of 5 times):

import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

s = requests.Session()

retries = Retry(total=5,
                backoff_factor=0.1,
                status_forcelist=[ 500, 502, 503, 504 ])

s.mount('http://', HTTPAdapter(max_retries=retries))

s.get('http://httpstat.us/500')

As per documentation for Retry: if the backoff_factor is 0.1, then sleep() will sleep for [0.1s, 0.2s, 0.4s, …] between retries. It will also force a retry if the status code returned is 500, 502, 503 or 504.

Various other options to Retry allow for more granular control:

  • total – Total number of retries to allow.
  • connect – How many connection-related errors to retry on.
  • read – How many times to retry on read errors.
  • redirect – How many redirects to perform.
  • method_whitelist – Set of uppercased HTTP method verbs that we should retry on.
  • status_forcelist – A set of HTTP status codes that we should force a retry on.
  • backoff_factor – A backoff factor to apply between attempts.
  • raise_on_redirect – Whether, if the number of redirects is exhausted, to raise a MaxRetryError, or to return a response with a response code in the 3xx range.
  • raise_on_status – Similar meaning to raise_on_redirect: whether we should raise an exception, or return a response, if status falls in status_forcelist range and retries have been exhausted.

NB: raise_on_status is relatively new, and has not made it into a release of urllib3 or requests yet. The raise_on_status keyword argument appears to have made it into the standard library at most in python version 3.6.

To make requests retry on specific HTTP status codes, use status_forcelist. For example, status_forcelist=[503] will retry on status code 503 (service unavailable).

By default, the retry only fires for these conditions:

  • Could not get a connection from the pool.
  • TimeoutError
  • HTTPException raised (from http.client in Python 3 else httplib). This seems to be low-level HTTP exceptions, like URL or protocol not formed correctly.
  • SocketError
  • ProtocolError

Notice that these are all exceptions that prevent a regular HTTP response from being received. If any regular response is generated, no retry is done. Without using the status_forcelist, even a response with status 500 will not be retried.

To make it behave in a manner which is more intuitive for working with a remote API or web server, I would use the above code snippet, which forces retries on statuses 500, 502, 503 and 504, all of which are not uncommon on the web and (possibly) recoverable given a big enough backoff period.

EDITED: Import Retry class directly from urllib3.


回答 2

请注意,Martijn Pieters的答案不适用于1.2.1+版本。如果不修补库,则无法全局设置。

您可以改为:

import requests
from requests.adapters import HTTPAdapter

s = requests.Session()
s.mount('http://www.github.com', HTTPAdapter(max_retries=5))
s.mount('https://www.github.com', HTTPAdapter(max_retries=5))

Be careful, Martijn Pieters’s answer isn’t suitable for version 1.2.1+. You can’t set it globally without patching the library.

You can do this instead:

import requests
from requests.adapters import HTTPAdapter

s = requests.Session()
s.mount('http://www.github.com', HTTPAdapter(max_retries=5))
s.mount('https://www.github.com', HTTPAdapter(max_retries=5))

回答 3

在为这里的一些答案苦苦挣扎之后,我找到了一个名为backoff的库,该库对我的情况更好。一个基本的例子:

import backoff

@backoff.on_exception(
    backoff.expo,
    requests.exceptions.RequestException,
    max_tries=5,
    giveup=lambda e: e.response is not None and e.response.status_code < 500
)
def publish(self, data):
    r = requests.post(url, timeout=10, json=data)
    r.raise_for_status()

我仍然建议您尝试一下该库的本机功能,但是如果遇到任何问题或需要更广泛的控制,可以选择退避。

After struggling a bit with some of the answers here, I found a library called backoff that worked better for my situation. A basic example:

import backoff

@backoff.on_exception(
    backoff.expo,
    requests.exceptions.RequestException,
    max_tries=5,
    giveup=lambda e: e.response is not None and e.response.status_code < 500
)
def publish(self, data):
    r = requests.post(url, timeout=10, json=data)
    r.raise_for_status()

I’d still recommend giving the library’s native functionality a shot, but if you run into any problems or need broader control, backoff is an option.


回答 4

获得更高控制权的一种更干净的方法可能是将重试内容打包到一个函数中,并使用装饰器将该函数重试,并将异常列入白名单。

我在这里创建了相同的文件:http : //www.praddy.in/retry-decorator-whitelisted-exceptions/

复制该链接中的代码:

def retry(exceptions, delay=0, times=2):
"""
A decorator for retrying a function call with a specified delay in case of a set of exceptions

Parameter List
-------------
:param exceptions:  A tuple of all exceptions that need to be caught for retry
                                    e.g. retry(exception_list = (Timeout, Readtimeout))
:param delay: Amount of delay (seconds) needed between successive retries.
:param times: no of times the function should be retried


"""
def outer_wrapper(function):
    @functools.wraps(function)
    def inner_wrapper(*args, **kwargs):
        final_excep = None  
        for counter in xrange(times):
            if counter > 0:
                time.sleep(delay)
            final_excep = None
            try:
                value = function(*args, **kwargs)
                return value
            except (exceptions) as e:
                final_excep = e
                pass #or log it

        if final_excep is not None:
            raise final_excep
    return inner_wrapper

return outer_wrapper

@retry(exceptions=(TimeoutError, ConnectTimeoutError), delay=0, times=3)
def call_api():

A cleaner way to gain higher control might be to package the retry stuff into a function and make that function retriable using a decorator and whitelist the exceptions.

I have created the same here: http://www.praddy.in/retry-decorator-whitelisted-exceptions/

Reproducing the code in that link :

def retry(exceptions, delay=0, times=2):
"""
A decorator for retrying a function call with a specified delay in case of a set of exceptions

Parameter List
-------------
:param exceptions:  A tuple of all exceptions that need to be caught for retry
                                    e.g. retry(exception_list = (Timeout, Readtimeout))
:param delay: Amount of delay (seconds) needed between successive retries.
:param times: no of times the function should be retried


"""
def outer_wrapper(function):
    @functools.wraps(function)
    def inner_wrapper(*args, **kwargs):
        final_excep = None  
        for counter in xrange(times):
            if counter > 0:
                time.sleep(delay)
            final_excep = None
            try:
                value = function(*args, **kwargs)
                return value
            except (exceptions) as e:
                final_excep = e
                pass #or log it

        if final_excep is not None:
            raise final_excep
    return inner_wrapper

return outer_wrapper

@retry(exceptions=(TimeoutError, ConnectTimeoutError), delay=0, times=3)
def call_api():

使用请求包时发生SSL InsecurePlatform错误

问题:使用请求包时发生SSL InsecurePlatform错误

我正在使用Python 2.7.3和请求。我通过pip安装了Requests。我相信这是最新版本。我正在Debian Wheezy上运行。

过去,我已经使用Requests很多次了,但是从未遇到过这个问题,但是当Requests我发出https请求时,似乎出现了InsecurePlatform异常。

错误提到urllib3,但我没有安装。我确实安装了它以检查它是否解决了错误,但是没有成功。

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3
/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not
available. This prevents urllib3 from configuring SSL appropriately and 
may cause certain SSL connections to fail. For more information, see 
https://urllib3.readthedocs.org/en/latest  
/security.html#insecureplatformwarning.

关于我为什么要得到这个的任何想法?我已经按照错误消息中的说明检查了文档,但是文档说要导入urllib3并禁用警告或提供证书。

Im using Python 2.7.3 and Requests. I installed Requests via pip. I believe it’s the latest version. I’m running on Debian Wheezy.

I’ve used Requests lots of times in the past and never faced this issue, but it seems that when making https requests with Requests I get an InsecurePlatform exception.

The error mentions urllib3, but I don’t have that installed. I did install it to check if it resolved the error, but it didn’t.

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3
/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not
available. This prevents urllib3 from configuring SSL appropriately and 
may cause certain SSL connections to fail. For more information, see 
https://urllib3.readthedocs.org/en/latest  
/security.html#insecureplatformwarning.

Any ideas as to why I’m getting this? I’ve checked the docs, as specified in the error message, but the docs are saying to import urllib3 and either disable the warning, or provide a certificate.


回答 0

使用有些隐藏的安全功能:

pip install requests[security] 要么 pip install pyOpenSSL ndg-httpsclient pyasn1

这两个命令都安装以下额外的软件包:

  • pyOpenSSL
  • 密码学
  • 艾德娜

请注意,这对于python-2.7.9 +不是必需的。

如果pip install失败并显示错误,请检查您是否具有必需的开发包libffilibssl使用发行版的包管理器将其python安装在系统中:

  • Debian的 / Ubuntu的python-dev libffi-dev libssl-dev包。

  • Fedora的openssl-devel python-devel libffi-devel包。

上面的发行列表不完整。

解决方法请参阅@TomDotTom的原始答案

万一您无法安装某些必需的开发包,还有一个选项可以禁用该警告:

import requests.packages.urllib3
requests.packages.urllib3.disable_warnings()

如果您pip自己受到InsecurePlatformWarningPyPI的影响并且无法从PyPI安装任何东西,则可以通过此分步指南进行修复,以手动部署其他python软件包。

Use the somewhat hidden security feature:

pip install requests[security] or pip install pyOpenSSL ndg-httpsclient pyasn1

Both commands install following extra packages:

  • pyOpenSSL
  • cryptography
  • idna

Please note that this is not required for python-2.7.9+.

If pip install fails with errors, check whether you have required development packages for libffi, libssl and python installed in your system using distribution’s package manager:

  • Debian/Ubuntupython-dev libffi-dev libssl-dev packages.

  • Fedoraopenssl-devel python-devel libffi-devel packages.

Distro list above is incomplete.

Workaround (see the original answer by @TomDotTom):

In case you cannot install some of the required development packages, there’s also an option to disable that warning:

import requests.packages.urllib3
requests.packages.urllib3.disable_warnings()

If your pip itself is affected by InsecurePlatformWarning and cannot install anything from PyPI, it can be fixed with this step-by-step guide to deploy extra python packages manually.


回答 1

在2.6版之前,Requests 2.6针对python的用户引入了此警告,仅提供了可用的SSL模块。

假设您无法升级到新版本的python,这将安装更多最新的python SSL库:

pip install --upgrade ndg-httpsclient 

但是,在某些没有pyOpenSSL的构建依赖性的系统上,这可能会失败。在debian系统上,在上面的pip命令之前运行此命令足以使pyOpenSSL构建:

apt-get install python-dev libffi-dev libssl-dev

Requests 2.6 introduced this warning for users of python prior to 2.7.9 with only stock SSL modules available.

Assuming you can’t upgrade to a newer version of python, this will install more up-to-date python SSL libraries:

pip install --upgrade ndg-httpsclient 

HOWEVER, this may fail on some systems without the build-dependencies for pyOpenSSL. On debian systems, running this before the pip command above should be enough for pyOpenSSL to build:

apt-get install python-dev libffi-dev libssl-dev

回答 2

我不会在生产中使用它,只是一些测试跑步者。并重申urllib3文档

如果您知道自己在做什么,并想禁用此​​警告和其他警告

import requests.packages.urllib3
requests.packages.urllib3.disable_warnings()

编辑/更新:

以下内容也应该起作用:

import logging
import requests

# turn down requests log verbosity
logging.getLogger('requests').setLevel(logging.CRITICAL)

I don’t use this in production, just some test runners. And to reiterate the urllib3 documentation

If you know what you are doing and would like to disable this and other warnings

import requests.packages.urllib3
requests.packages.urllib3.disable_warnings()

Edit / Update:

The following should also work:

import logging
import requests

# turn down requests log verbosity
logging.getLogger('requests').setLevel(logging.CRITICAL)

回答 3

如果您无法将Python版本升级到2.7.9,并希望禁止显示警告,

您可以将“请求”版本降级为2.5.3:

sudo pip install requests==2.5.3

关于版本:http : //fossies.org/diffs/requests/2.5.3_vs_2.6.0/requests/packages/urllib3/util/ssl_.py-diff.html

If you are not able to upgrade your Python version to 2.7.9, and want to suppress warnings,

you can downgrade your ‘requests’ version to 2.5.3:

sudo pip install requests==2.5.3

About version: http://fossies.org/diffs/requests/2.5.3_vs_2.6.0/requests/packages/urllib3/util/ssl_.py-diff.html


回答 4

实际上,您可以尝试一下。

requests.post("https://www.google.com", verify=False)

您可以阅读请求代码。

"C:\Python27\Lib\site-packages\requests\sessions.py"

class Session(SessionRedirectMixin):
......
 def request(self, method, url,
    params=None,
    data=None,
    headers=None,
    cookies=None,
    files=None,
    auth=None,
    timeout=None,
    allow_redirects=True,
    proxies=None,
    hooks=None,
    stream=None,
    verify=None,  # <========
    cert=None):
    """
    ...
    :param verify: (optional) if True, the SSL cert will be verified.
         A CA_BUNDLE path can also be provided.
    ...
    """

In fact, you can try this.

requests.post("https://www.google.com", verify=False)

you can read the code for requests.

"C:\Python27\Lib\site-packages\requests\sessions.py"

class Session(SessionRedirectMixin):
......
 def request(self, method, url,
    params=None,
    data=None,
    headers=None,
    cookies=None,
    files=None,
    auth=None,
    timeout=None,
    allow_redirects=True,
    proxies=None,
    hooks=None,
    stream=None,
    verify=None,  # <========
    cert=None):
    """
    ...
    :param verify: (optional) if True, the SSL cert will be verified.
         A CA_BUNDLE path can also be provided.
    ...
    """

回答 5

这里给出的所有解决方案都没有帮助(我仅限于python 2.6.6)。我在一个简单的开关中找到了要传递给pip的答案:

$ sudo pip install --trusted-host pypi.python.org <module_name>

这告诉pip,可以从pypi.python.org抓取模块。

对我来说,问题是防火墙后的我公司的代理服务器,使它看起来像某些服务器的恶意客户端。万岁安全。


更新:有关PyPi域中的更改以及可以添加的其他选项,请参见@Alex 的 答案--trusted-host。(我将在此处复制/粘贴,但是他的回答是,所以+1)

All of the solutions given here haven’t helped (I’m constrained to python 2.6.6). I’ve found the answer in a simple switch to pass to pip:

$ sudo pip install --trusted-host pypi.python.org <module_name>

This tells pip that it’s OK to grab the module from pypi.python.org.

For me, the issue is my company’s proxy behind it’s firewall that makes it look like a malicious client to some servers. Hooray security.


Update: See @Alex ‘s answer for changes in the PyPi domains, and additional --trusted-host options that can be added. (I’d copy/paste here, but his answer, so +1 him)


回答 6

这个答案无关紧要,但是如果您想摆脱警告并从请求中获得以下警告:

InsecurePlatformWarning /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.

您可以通过将以下行添加到python代码中来禁用它:

requests.packages.urllib3.disable_warnings()

This answer is unrelated, but if you wanted to get rid of warning and get following warning from requests:

InsecurePlatformWarning /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.

You can disable it by adding the following line to your python code:

requests.packages.urllib3.disable_warnings()


回答 7

我必须先去bash(从ZSH)。然后

sudo -H pip install 'requests[security]' --upgrade

解决了问题。

I had to go to bash (from ZSH) first. Then

sudo -H pip install 'requests[security]' --upgrade

fixed the problem.


回答 8

过去的一周来了,我在Ubuntu 14.04(与Python 2.7.6),我做了之后apt-get dist-upgrade,包括libssl1.1:amd64deb.sury.org

由于我是certbot-auto renew来自Cron作业,因此我也使用--no-self-upgrade来减少计划外的维护。这似乎是麻烦的根源。

要解决该错误,我所需要做的就是成为root用户(使用su--login开关),然后certbot-auto进行自我升级。即:

sudo su --login
/usr/local/bin/certbot-auto renew 
# ... Upgrading certbot-auto 0.8.1 to 0.18.2... blah blah blah ...

而不是通常从root的crontab运行的内容:

5 7 * * * /usr/local/bin/certbot-auto renew --quiet --no-self-upgrade

之后,letsencrypt renwals再次正常运行。

This came up for me on Ubuntu 14.04 (with Python 2.7.6) last week after i did a apt-get dist-upgrade that included libssl1.1:amd64 from deb.sury.org.

Since I run certbot-auto renew from a cron job, I also use the --no-self-upgrade to cut down on unscheduled maintenance. This seems to have been the source of the trouble.

To fix the error, all I needed to do was become root (with su‘s --login switch) and let certbot-auto upgrade itself. I.e:

sudo su --login
/usr/local/bin/certbot-auto renew 
# ... Upgrading certbot-auto 0.8.1 to 0.18.2... blah blah blah ...

instead of what normally runs from root’s crontab:

5 7 * * * /usr/local/bin/certbot-auto renew --quiet --no-self-upgrade

After that, letsencrypt renwals ran normally once again.


回答 9

对我来说没有工作,我需要升级点…。

Debian / Ubuntu

安装依赖

sudo apt-get install libpython-dev libssl-dev libffi-dev

升级pip并安装软件包

sudo pip install -U pip
sudo pip install -U pyopenssl ndg-httpsclient pyasn1

如果要删除依赖项

sudo apt-get remove --purge libpython-dev libssl-dev libffi-dev
sudo apt-get autoremove

For me no work i need upgrade pip….

Debian/Ubuntu

install dependencies

sudo apt-get install libpython-dev libssl-dev libffi-dev

upgrade pip and install packages

sudo pip install -U pip
sudo pip install -U pyopenssl ndg-httpsclient pyasn1

If you want remove dependencies

sudo apt-get remove --purge libpython-dev libssl-dev libffi-dev
sudo apt-get autoremove

回答 10

我在CentOS 5服务器上遇到了类似的问题,在较旧版本的python2.7之上的/ usr / local中安装了python 2.7.12。目前尚无法在此服务器上升级到CentOS 6或7。

某些python 2.7模块仍旧存在于较早版本的python中,但是pip升级失败,因为CentOS 5软件包不支持较新的加密软件包。

具体来说,“ pip安装请求[安全]”失败了,因为CentOS 5上的openssl版本是0.9.8e,而加密> 1.4.0不再支持。

为了解决OP的原始问题,我做到了:

1) pip install 'cryptography<1.3.5,>1.3.0'.  

此安装的加密技术1.3.4可与openssl-0.9.8e一起使用。cryptograpy 1.3.4也足以满足以下命令的要求。

2) pip install 'requests[security]'

现在安装此命令,因为它不会尝试安装> 1.4.0的密码。

请注意,在Centos 5上,我还需要:

yum install openssl-devel

允许建立密码

I just had a similar issue on a CentOS 5 server where I installed python 2.7.12 in /usr/local on top of a much older version of python2.7. Upgrading to CentOS 6 or 7 isn’t an option on this server right now.

Some of the python 2.7 modules were still existing from the older version of python, but pip was failing to upgrade because the newer cryptography package is not supported by the CentOS 5 packages.

Specifically, ‘pip install requests[security]’ was failing because the openssl version on the CentOS 5 was 0.9.8e which is no longer supported by cryptography > 1.4.0.

To solve the OPs original issue I did:

1) pip install 'cryptography<1.3.5,>1.3.0'.  

This installed cryptography 1.3.4 which works with openssl-0.9.8e. cryptograpy 1.3.4 is also sufficient to satisfy the requirement for the following command.

2) pip install 'requests[security]'

This command now installs because it doesn’t try to install cryptography > 1.4.0.

Note that on Centos 5 I also needed to:

yum install openssl-devel

To allow cryptography to build


回答 11

下面是它在Python 3.6上对我的工作方式:

import requests
import urllib3

# Suppress InsecureRequestWarning: Unverified HTTPS
urllib3.disable_warnings()

Below is how it’s working for me on Python 3.6:

import requests
import urllib3

# Suppress InsecureRequestWarning: Unverified HTTPS
urllib3.disable_warnings()

回答 12

不要安装pyOpenSSL,因为它将很快被弃用。目前最好的方法是-

import requests
requests.packages.urllib3.disable_warnings()

Dont install pyOpenSSL as it shall soon be deprecated. Current best approach is-

import requests
requests.packages.urllib3.disable_warnings()

回答 13

如果您只想停止不安全的警告,例如:

/usr/lib/python3/dist-packages/urllib3/connectionpool.py:794:InsecureRequestWarning:发出未经验证的HTTPS请求。强烈建议添加证书验证。请参阅: https: //urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning)

做:

requests.METHOD("https://www.google.com", verify=False)

验证=假

是关键,以下方面并不擅长:

requests.packages.urllib3.disable_warnings()

要么

urllib3.disable_warnings()

但是,您必须知道,这可能会导致潜在的安全风险

if you just want to stopping insecure warning like:

/usr/lib/python3/dist-packages/urllib3/connectionpool.py:794: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning)

do:

requests.METHOD("https://www.google.com", verify=False)

verify=False

is the key, followings are not good at it:

requests.packages.urllib3.disable_warnings()

or

urllib3.disable_warnings()

but, you HAVE TO know, that might cause potential security risks.


回答 14


Mac
Pycharm社区版2019.3
Python解释器3.6 遇到相同的问题。
用20.0.2升级点对我有用。
Pycharm --> Preferences --> Project Interpreter --> click on pip --> specify version 20.0.2 --> Install package

I had same problem with
Mac
Pycharm community edition 2019.3
Python interpreter 3.6.
Upgrading pip with 20.0.2 worked for me.
Pycharm --> Preferences --> Project Interpreter --> click on pip --> specify version 20.0.2 --> Install package


如何在Python请求中禁用安全证书检查

问题:如何在Python请求中禁用安全证书检查

我在用

import requests
requests.post(url='https://foo.com', data={'bar':'baz'})

但我收到了request.exceptions.SSLError。该网站的证书已过期,但是我没有发送敏感数据,因此对我来说无关紧要。我可以想象有一个像’verifiy = False’这样的参数可以使用,但是我似乎找不到。

I am using

import requests
requests.post(url='https://foo.com', data={'bar':'baz'})

but I get a request.exceptions.SSLError. The website has an expired certficate, but I am not sending sensitive data, so it doesn’t matter to me. I would imagine there is an argument like ‘verifiy=False’ that I could use, but I can’t seem to find it.


回答 0

文档中

requests如果设置verify为False,也可以忽略验证SSL证书 。

>>> requests.get('https://kennethreitz.com', verify=False)
<Response [200]>

如果您使用的是第三方模块,并且想禁用检查功能,则可以使用上下文管理器来修补requests和更改它,从而使它verify=False成为默认设置并禁止显示警告。

import warnings
import contextlib

import requests
from urllib3.exceptions import InsecureRequestWarning


old_merge_environment_settings = requests.Session.merge_environment_settings

@contextlib.contextmanager
def no_ssl_verification():
    opened_adapters = set()

    def merge_environment_settings(self, url, proxies, stream, verify, cert):
        # Verification happens only once per connection so we need to close
        # all the opened adapters once we're done. Otherwise, the effects of
        # verify=False persist beyond the end of this context manager.
        opened_adapters.add(self.get_adapter(url))

        settings = old_merge_environment_settings(self, url, proxies, stream, verify, cert)
        settings['verify'] = False

        return settings

    requests.Session.merge_environment_settings = merge_environment_settings

    try:
        with warnings.catch_warnings():
            warnings.simplefilter('ignore', InsecureRequestWarning)
            yield
    finally:
        requests.Session.merge_environment_settings = old_merge_environment_settings

        for adapter in opened_adapters:
            try:
                adapter.close()
            except:
                pass

使用方法如下:

with no_ssl_verification():
    requests.get('https://wrong.host.badssl.com/')
    print('It works')

    requests.get('https://wrong.host.badssl.com/', verify=True)
    print('Even if you try to force it to')

requests.get('https://wrong.host.badssl.com/', verify=False)
print('It resets back')

session = requests.Session()
session.verify = True

with no_ssl_verification():
    session.get('https://wrong.host.badssl.com/', verify=True)
    print('Works even here')

try:
    requests.get('https://wrong.host.badssl.com/')
except requests.exceptions.SSLError:
    print('It breaks')

try:
    session.get('https://wrong.host.badssl.com/')
except requests.exceptions.SSLError:
    print('It breaks here again')

请注意,一旦您离开上下文管理器,此代码将关闭处理已打补丁请求的所有打开的适配器。这是因为请求维护每个会话的连接池,并且证书验证每个连接仅进行一次,因此将发生以下意外情况:

>>> import requests
>>> session = requests.Session()
>>> session.get('https://wrong.host.badssl.com/', verify=False)
/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
<Response [200]>
>>> session.get('https://wrong.host.badssl.com/', verify=True)
/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
<Response [200]>

From the documentation:

requests can also ignore verifying the SSL certificate if you set verify to False.

>>> requests.get('https://kennethreitz.com', verify=False)
<Response [200]>

If you’re using a third-party module and want to disable the checks, here’s a context manager that monkey patches requests and changes it so that verify=False is the default and suppresses the warning.

import warnings
import contextlib

import requests
from urllib3.exceptions import InsecureRequestWarning


old_merge_environment_settings = requests.Session.merge_environment_settings

@contextlib.contextmanager
def no_ssl_verification():
    opened_adapters = set()

    def merge_environment_settings(self, url, proxies, stream, verify, cert):
        # Verification happens only once per connection so we need to close
        # all the opened adapters once we're done. Otherwise, the effects of
        # verify=False persist beyond the end of this context manager.
        opened_adapters.add(self.get_adapter(url))

        settings = old_merge_environment_settings(self, url, proxies, stream, verify, cert)
        settings['verify'] = False

        return settings

    requests.Session.merge_environment_settings = merge_environment_settings

    try:
        with warnings.catch_warnings():
            warnings.simplefilter('ignore', InsecureRequestWarning)
            yield
    finally:
        requests.Session.merge_environment_settings = old_merge_environment_settings

        for adapter in opened_adapters:
            try:
                adapter.close()
            except:
                pass

Here’s how you use it:

with no_ssl_verification():
    requests.get('https://wrong.host.badssl.com/')
    print('It works')

    requests.get('https://wrong.host.badssl.com/', verify=True)
    print('Even if you try to force it to')

requests.get('https://wrong.host.badssl.com/', verify=False)
print('It resets back')

session = requests.Session()
session.verify = True

with no_ssl_verification():
    session.get('https://wrong.host.badssl.com/', verify=True)
    print('Works even here')

try:
    requests.get('https://wrong.host.badssl.com/')
except requests.exceptions.SSLError:
    print('It breaks')

try:
    session.get('https://wrong.host.badssl.com/')
except requests.exceptions.SSLError:
    print('It breaks here again')

Note that this code closes all open adapters that handled a patched request once you leave the context manager. This is because requests maintains a per-session connection pool and certificate validation happens only once per connection so unexpected things like this will happen:

>>> import requests
>>> session = requests.Session()
>>> session.get('https://wrong.host.badssl.com/', verify=False)
/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
<Response [200]>
>>> session.get('https://wrong.host.badssl.com/', verify=True)
/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
<Response [200]>

回答 1

使用requests.packages.urllib3.disable_warnings()verify=Falseon requests方法。

import requests
from urllib3.exceptions import InsecureRequestWarning

# Suppress only the single warning from urllib3 needed.
requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)

# Set `verify=False` on `requests.post`.
requests.post(url='https://example.com', data={'bar':'baz'}, verify=False)

Use requests.packages.urllib3.disable_warnings() and verify=False on requests methods.

import requests
from urllib3.exceptions import InsecureRequestWarning

# Suppress only the single warning from urllib3 needed.
requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)

# Set `verify=False` on `requests.post`.
requests.post(url='https://example.com', data={'bar':'baz'}, verify=False)

回答 2

要添加到Blender的答案中,您可以使用禁用所有请求的SSLSession.verify = False

import requests

session = requests.Session()
session.verify = False
session.post(url='https://foo.com', data={'bar':'baz'})

请注意urllib3,(请求使用),强烈建议您不要发送未经验证的HTTPS请求,并会引发InsecureRequestWarning

To add to Blender’s answer, you can disable SSL certificate validation for all requests using Session.verify = False

import requests

session = requests.Session()
session.verify = False
session.post(url='https://example.com', data={'bar':'baz'})

Note that urllib3, (which Requests uses), strongly discourages making unverified HTTPS requests and will raise an InsecureRequestWarning.


回答 3

也可以从环境变量中完成:

export CURL_CA_BUNDLE=""

Also can be done from the environment variable:

export CURL_CA_BUNDLE=""

回答 4

如果您想使用verify = False选项发送准确的发布请求,最快的方法是使用以下代码:

import requests

requests.api.request('post', url, data={'bar':'baz'}, json=None, verify=False)

If you want to send exactly post request with verify=False option, fastest way is to use this code:

import requests

requests.api.request('post', url, data={'bar':'baz'}, json=None, verify=False)

从请求库解析JSON响应的最佳方法是什么?

问题:从请求库解析JSON响应的最佳方法是什么?

我正在使用python requests模块将RESTful GET发送到服务器,对此我得到了JSON响应。JSON响应基本上只是列表的列表。

强制对本地Python对象进行响应的最佳方法是什么,以便我可以使用进行迭代或打印出来pprint

I’m using the python requests module to send a RESTful GET to a server, for which I get a response in JSON. The JSON response is basically just a list of lists.

What’s the best way to coerce the response to a native Python object so I can either iterate or print it out using pprint?


回答 0

您可以使用json.loads

import json
import requests

response = requests.get(...)
json_data = json.loads(response.text)

这会将给定的字符串转换为字典,从而使您可以轻松地在代码中访问JSON数据。

或者,您可以使用@Martijn的有用建议以及投票较高的答案response.json()

You can use json.loads:

import json
import requests

response = requests.get(...)
json_data = json.loads(response.text)

This converts a given string into a dictionary which allows you to access your JSON data easily within your code.

Or you can use @Martijn’s helpful suggestion, and the higher voted answer, response.json().


回答 1

由于您正在使用requests,因此您应该使用响应的json方法。

import requests

response = requests.get(...)
data = response.json()

自动检测要使用的解码器

Since you’re using requests, you should use the response’s json method.

import requests

response = requests.get(...)
data = response.json()

It autodetects which decoder to use.


如何查看Python应用程序发送的整个HTTP请求?

问题:如何查看Python应用程序发送的整个HTTP请求?

就我而言,我正在使用该requests库通过HTTPS调用PayPal的API。不幸的是,我从贝宝(PayPal)收到错误消息,贝宝(PayPal)支持人员无法弄清错误是什么或由什么引起的。他们要我“请提供整个请求,包括标头”。

我怎样才能做到这一点?

In my case, I’m using the requests library to call PayPal’s API over HTTPS. Unfortunately, I’m getting an error from PayPal, and PayPal support cannot figure out what the error is or what’s causing it. They want me to “Please provide the entire request, headers included”.

How can I do that?


回答 0

一种简单的方法:启用登录请求的最新版本(1.x及更高版本)。

请求用途http.clientlogging模块配置到控制日志记录级别,如所描述这里

示范

从链接文档中摘录的代码:

import requests
import logging

# These two lines enable debugging at httplib level (requests->urllib3->http.client)
# You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
# The only thing missing will be the response.body which is not logged.
try:
    import http.client as http_client
except ImportError:
    # Python 2
    import httplib as http_client
http_client.HTTPConnection.debuglevel = 1

# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

requests.get('https://httpbin.org/headers')

示例输出

$ python requests-logging.py 
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): httpbin.org
send: 'GET /headers HTTP/1.1\r\nHost: httpbin.org\r\nAccept-Encoding: gzip, deflate, compress\r\nAccept: */*\r\nUser-Agent: python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-48-generic\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Date: Sat, 29 Jun 2013 11:19:34 GMT
header: Server: gunicorn/0.17.4
header: Content-Length: 226
header: Connection: keep-alive
DEBUG:requests.packages.urllib3.connectionpool:"GET /headers HTTP/1.1" 200 226

A simple method: enable logging in recent versions of Requests (1.x and higher.)

Requests uses the http.client and logging module configuration to control logging verbosity, as described here.

Demonstration

Code excerpted from the linked documentation:

import requests
import logging

# These two lines enable debugging at httplib level (requests->urllib3->http.client)
# You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
# The only thing missing will be the response.body which is not logged.
try:
    import http.client as http_client
except ImportError:
    # Python 2
    import httplib as http_client
http_client.HTTPConnection.debuglevel = 1

# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

requests.get('https://httpbin.org/headers')

Example Output

$ python requests-logging.py 
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): httpbin.org
send: 'GET /headers HTTP/1.1\r\nHost: httpbin.org\r\nAccept-Encoding: gzip, deflate, compress\r\nAccept: */*\r\nUser-Agent: python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-48-generic\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Date: Sat, 29 Jun 2013 11:19:34 GMT
header: Server: gunicorn/0.17.4
header: Content-Length: 226
header: Connection: keep-alive
DEBUG:requests.packages.urllib3.connectionpool:"GET /headers HTTP/1.1" 200 226

回答 1

r = requests.get('https://api.github.com', auth=('user', 'pass'))

r是回应。它具有一个request属性,其中包含您需要的信息。

r.request.allow_redirects  r.request.headers          r.request.register_hook
r.request.auth             r.request.hooks            r.request.response
r.request.cert             r.request.method           r.request.send
r.request.config           r.request.params           r.request.sent
r.request.cookies          r.request.path_url         r.request.session
r.request.data             r.request.prefetch         r.request.timeout
r.request.deregister_hook  r.request.proxies          r.request.url
r.request.files            r.request.redirect         r.request.verify

r.request.headers 给出标题:

{'Accept': '*/*',
 'Accept-Encoding': 'identity, deflate, compress, gzip',
 'Authorization': u'Basic dXNlcjpwYXNz',
 'User-Agent': 'python-requests/0.12.1'}

然后r.request.data将主体作为映射。urllib.urlencode如果他们愿意,可以将其转换为:

import urllib
b = r.request.data
encoded_body = urllib.urlencode(b)

根据响应的类型,.data-attribute可能会丢失,而.body-attribute会在那里。

r = requests.get('https://api.github.com', auth=('user', 'pass'))

r is a response. It has a request attribute which has the information you need.

r.request.allow_redirects  r.request.headers          r.request.register_hook
r.request.auth             r.request.hooks            r.request.response
r.request.cert             r.request.method           r.request.send
r.request.config           r.request.params           r.request.sent
r.request.cookies          r.request.path_url         r.request.session
r.request.data             r.request.prefetch         r.request.timeout
r.request.deregister_hook  r.request.proxies          r.request.url
r.request.files            r.request.redirect         r.request.verify

r.request.headers gives the headers:

{'Accept': '*/*',
 'Accept-Encoding': 'identity, deflate, compress, gzip',
 'Authorization': u'Basic dXNlcjpwYXNz',
 'User-Agent': 'python-requests/0.12.1'}

Then r.request.data has the body as a mapping. You can convert this with urllib.urlencode if they prefer:

import urllib
b = r.request.data
encoded_body = urllib.urlencode(b)

depending on the type of the response the .data-attribute may be missing and a .body-attribute be there instead.


回答 2

您可以使用HTTP Toolkit来做到这一点。

如果您需要在不更改代码的情况下快速执行此操作,则特别有用:您可以从HTTP Toolkit打开终端,从那里正常运行任何Python代码,并且可以查看每个HTTP / HTTPS的全部内容。立即提出要求。

有一个免费版本可以满足您的所有需求,并且它是100%开放源代码。

我是HTTP Toolkit的创建者;我实际上是自己构建的,以便不久前为我解决完全相同的问题!我也曾尝试调试付款集成,但他们的SDK无法正常工作,我无法告知原因,我需要知道实际情况如何对其进行正确修复。这非常令人沮丧,但是能够看到原始流量确实有所帮助。

You can use HTTP Toolkit to do exactly this.

It’s especially useful if you need to do this quickly, with no code changes: you can open a terminal from HTTP Toolkit, run any Python code from there as normal, and you’ll be able to see the full content of every HTTP/HTTPS request immediately.

There’s a free version that can do everything you need, and it’s 100% open source.

I’m the creator of HTTP Toolkit; I actually built it myself to solve the exact same problem for me a while back! I too was trying to debug a payment integration, but their SDK didn’t work, I couldn’t tell why, and I needed to know what was actually going on to properly fix it. It’s very frustrating, but being able to see the raw traffic really helps.


回答 3

如果您使用的是Python 2.x,请尝试安装urllib2打开器。尽管您可能必须将其与您用来击中HTTPS的其他开启器结合使用,但这应该可以打印出标题。

import urllib2
urllib2.install_opener(urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1)))
urllib2.urlopen(url)

If you’re using Python 2.x, try installing a urllib2 opener. That should print out your headers, although you may have to combine that with other openers you’re using to hit the HTTPS.

import urllib2
urllib2.install_opener(urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1)))
urllib2.urlopen(url)

回答 4

verbose配置选项可以让你看到你想要的东西。文档中一个示例

注意:请阅读以下注释:详细的配置选项似乎不再可用。

The verbose configuration option might allow you to see what you want. There is an example in the documentation.

NOTE: Read the comments below: The verbose config options doesn’t seem to be available anymore.


如何禁用请求库中的日志消息?

问题:如何禁用请求库中的日志消息?

默认情况下,Requests python库按照以下方式将日志消息写入控制台:

Starting new HTTP connection (1): example.com
http://example.com:80 "GET / HTTP/1.1" 200 606

我通常对这些消息不感兴趣,因此想禁用它们。使这些消息静音或降低请求的详细程度的最佳方法是什么?

By default, the Requests python library writes log messages to the console, along the lines of:

Starting new HTTP connection (1): example.com
http://example.com:80 "GET / HTTP/1.1" 200 606

I’m usually not interested in these messages, and would like to disable them. What would be the best way to silence those messages or decrease Requests’ verbosity?


回答 0

我发现了如何配置请求的日志记录级别,这是通过标准的日志记录模块完成的。我决定将其配置为不记录消息,除非它们至少是警告:

import logging

logging.getLogger("requests").setLevel(logging.WARNING)

如果您也希望将此设置应用于urllib3库(通常由请求使用),请添加以下内容:

logging.getLogger("urllib3").setLevel(logging.WARNING)

I found out how to configure requests‘s logging level, it’s done via the standard logging module. I decided to configure it to not log messages unless they are at least warnings:

import logging

logging.getLogger("requests").setLevel(logging.WARNING)

If you wish to apply this setting for the urllib3 library (typically used by requests) too, add the following:

logging.getLogger("urllib3").setLevel(logging.WARNING)

回答 1

如果您是来这里寻找修改任何(可能是深度嵌套的)模块的日志记录的方法,请使用它logging.Logger.manager.loggerDict来获取所有记录器对象的字典。然后,返回的名称可用作以下参数logging.getLogger

import requests
import logging
for key in logging.Logger.manager.loggerDict:
    print(key)
# requests.packages.urllib3.connectionpool
# requests.packages.urllib3.util
# requests.packages
# requests.packages.urllib3
# requests.packages.urllib3.util.retry
# PYREADLINE
# requests
# requests.packages.urllib3.poolmanager

logging.getLogger('requests').setLevel(logging.CRITICAL)
# Could also use the dictionary directly:
# logging.Logger.manager.loggerDict['requests'].setLevel(logging.CRITICAL)

请注意每个用户136036中的注释,此方法仅显示运行上述代码片段时存在的记录器。例如,如果某个模块在实例化一个类时创建了一个新的记录器,则必须创建该类之后放置此代码段以打印其名称。

In case you came here looking for a way to modify logging of any (possibly deeply nested) module, use logging.Logger.manager.loggerDict to get a dictionary of all of the logger objects. The returned names can then be used as the argument to logging.getLogger:

import requests
import logging
for key in logging.Logger.manager.loggerDict:
    print(key)
# requests.packages.urllib3.connectionpool
# requests.packages.urllib3.util
# requests.packages
# requests.packages.urllib3
# requests.packages.urllib3.util.retry
# PYREADLINE
# requests
# requests.packages.urllib3.poolmanager

logging.getLogger('requests').setLevel(logging.CRITICAL)
# Could also use the dictionary directly:
# logging.Logger.manager.loggerDict['requests'].setLevel(logging.CRITICAL)

Per user136036 in a comment, be aware that this method only shows you the loggers that exist at the time you run the above snippet. If, for example, a module creates a new logger when you instantiate a class, then you must put this snippet after creating the class in order to print its name.


回答 2

import logging
urllib3_logger = logging.getLogger('urllib3')
urllib3_logger.setLevel(logging.CRITICAL)

这样,来自urllib3的所有level = INFO消息都不会出现在日志文件中。

因此,您可以继续将level = INFO用作日志消息…只需为您正在使用的库修改它即可。

import logging
urllib3_logger = logging.getLogger('urllib3')
urllib3_logger.setLevel(logging.CRITICAL)

In this way all the messages of level=INFO from urllib3 won’t be present in the logfile.

So you can continue to use the level=INFO for your log messages…just modify this for the library you are using.


回答 3

在遇到类似于您的问题之后,让我复制/粘贴一周或两周前写的文档部分:

import requests
import logging

# these two lines enable debugging at httplib level (requests->urllib3->httplib)
# you will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
# the only thing missing will be the response.body which is not logged.
import httplib
httplib.HTTPConnection.debuglevel = 1

logging.basicConfig() # you need to initialize logging, otherwise you will not see anything from requests
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

requests.get('http://httpbin.org/headers')

Let me copy/paste the documentation section which it I wrote about week or two ago, after having a problem similar to yours:

import requests
import logging

# these two lines enable debugging at httplib level (requests->urllib3->httplib)
# you will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
# the only thing missing will be the response.body which is not logged.
import httplib
httplib.HTTPConnection.debuglevel = 1

logging.basicConfig() # you need to initialize logging, otherwise you will not see anything from requests
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

requests.get('http://httpbin.org/headers')

回答 4

对于使用的任何人,logging.config.dictConfig您都可以像这样更改字典中的请求库日志级别:

'loggers': {
    '': {
        'handlers': ['file'],
        'level': level,
        'propagate': False
    },
    'requests.packages.urllib3': {
        'handlers': ['file'],
        'level': logging.WARNING
    }
}

For anybody using logging.config.dictConfig you can alter the requests library log level in the dictionary like this:

'loggers': {
    '': {
        'handlers': ['file'],
        'level': level,
        'propagate': False
    },
    'requests.packages.urllib3': {
        'handlers': ['file'],
        'level': logging.WARNING
    }
}

回答 5

将记录器名称设置为requestsrequests.urllib3对我不起作用。我必须指定确切的记录器名称才能更改记录级别。

首先查看您已定义的记录器,以查看要删除的记录器

print(logging.Logger.manager.loggerDict)

您将看到类似以下内容:

{...'urllib3.poolmanager': <logging.Logger object at 0x1070a6e10>, 'django.request': <logging.Logger object at 0x106d61290>, 'django.template': <logging.Logger object at 0x10630dcd0>, 'django.server': <logging.Logger object at 0x106dd6a50>, 'urllib3.connection': <logging.Logger object at 0x10710a350>,'urllib3.connectionpool': <logging.Logger object at 0x106e09690> ...}

然后为确切的记录器配置级别:

   'loggers': {
    '': {
        'handlers': ['default'],
        'level': 'DEBUG',
        'propagate': True
    },
    'urllib3.connectionpool': {
        'handlers': ['default'],
        'level': 'WARNING',
        'propagate' : False
    },

Setting the logger name as requests or requests.urllib3 did not work for me. I had to specify the exact logger name to change the logging level.

First See which loggers you have defined, to see which ones you want to remove

print(logging.Logger.manager.loggerDict)

And you will see something like this:

{...'urllib3.poolmanager': <logging.Logger object at 0x1070a6e10>, 'django.request': <logging.Logger object at 0x106d61290>, 'django.template': <logging.Logger object at 0x10630dcd0>, 'django.server': <logging.Logger object at 0x106dd6a50>, 'urllib3.connection': <logging.Logger object at 0x10710a350>,'urllib3.connectionpool': <logging.Logger object at 0x106e09690> ...}

Then configure the level for the exact logger:

   'loggers': {
    '': {
        'handlers': ['default'],
        'level': 'DEBUG',
        'propagate': True
    },
    'urllib3.connectionpool': {
        'handlers': ['default'],
        'level': 'WARNING',
        'propagate' : False
    },

回答 6

如果您有配置文件,则可以对其进行配置。

在loggers部分中添加urllib3:

[loggers]
keys = root, urllib3

添加logger_urllib3部分:

[logger_urllib3]
level = WARNING
handlers =
qualname = requests.packages.urllib3.connectionpool

If You have configuration file, You can configure it.

Add urllib3 in loggers section:

[loggers]
keys = root, urllib3

Add logger_urllib3 section:

[logger_urllib3]
level = WARNING
handlers =
qualname = requests.packages.urllib3.connectionpool

回答 7

答案在这里: Python:如何抑制来自第三方库的日志记录语句?

您可以保留basicConfig的默认日志记录级别,然后在获取模块的记录器时设置DEBUG级别。

logging.basicConfig(format='%(asctime)s %(module)s %(filename)s:%(lineno)s - %(message)s')
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

logger.debug("my debug message")

This answer is here: Python: how to suppress logging statements from third party libraries?

You can leave the default logging level for basicConfig, and then you set the DEBUG level when you get the logger for your module.

logging.basicConfig(format='%(asctime)s %(module)s %(filename)s:%(lineno)s - %(message)s')
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

logger.debug("my debug message")

回答 8

import logging

# Only show warnings
logging.getLogger("urllib3").setLevel(logging.WARNING)

# Disable all child loggers of urllib3, e.g. urllib3.connectionpool
logging.getLogger("urllib3").propagate = False
import logging

# Only show warnings
logging.getLogger("urllib3").setLevel(logging.WARNING)

# Disable all child loggers of urllib3, e.g. urllib3.connectionpool
logging.getLogger("urllib3").propagate = False

回答 9

Kbrose关于查找哪个记录器正在生成日志消息的指南非常有用。对于我的Django项目,我不得不对120种不同的记录器进行分类,直到发现是elasticsearchPython库对我造成了问题。根据大多数问题的指导,我通过将其添加到记录器中来禁用了它:

      ...
      'elasticsearch': {
          'handlers': ['console'],
          'level': logging.WARNING,
      },     
      ...

如果有人在运行Elasticsearch查询时看到其他无用的日志消息,请在此处发布。

Kbrose’s guidance on finding which logger was generating log messages was immensely useful. For my Django project, I had to sort through 120 different loggers until I found that it was the elasticsearch Python library that was causing issues for me. As per the guidance in most of the questions, I disabled it by adding this to my loggers:

      ...
      'elasticsearch': {
          'handlers': ['console'],
          'level': logging.WARNING,
      },     
      ...

Posting here in case someone else is seeing the unhelpful log messages come through whenever they run an Elasticsearch query.


回答 10

简单:只需在requests.packages.urllib3.disable_warnings()之后添加import requests

simple: just add requests.packages.urllib3.disable_warnings() after import requests


回答 11

我不确定以前的方法是否已停止工作,但是无论如何,这是消除警告的另一种方法:

PYTHONWARNINGS="ignore:Unverified HTTPS request" ./do-insecure-request.py

基本上,在脚本执行的上下文中添加环境变量。

从文档中:https : //urllib3.readthedocs.org/en/latest/security.html#disabling-warnings

I’m not sure if the previous approaches have stopped working, but in any case, here’s another way of removing the warnings:

PYTHONWARNINGS="ignore:Unverified HTTPS request" ./do-insecure-request.py

Basically, adding an environment variable in the context of the script execution.

From the documentation: https://urllib3.readthedocs.org/en/latest/security.html#disabling-warnings