主页 / 问题 / 823608
Accepted
babyborn
Asked:2020-05-05 21:05:00 +0000 UTC2020-05-05 21:05:00 +0000 UTC

如何加快发送 3 个 GET 请求

  • 772

大家好！有 3 个 GET 请求

r1 = requests.get('http://mysite.com/check.php?' + 'userid=666&userage=18')
r2 = requests.get('http://mysite.com/check.php?' + 'userid=777&userage=22')
r3 = requests.get('http://mysite.com/check.php?' + 'userid=888&userage=26')

发送它们并收到响应需要很长时间，如何减少这个时间？这三个请求减慢了整个程序。

python

2 个回答

  1. 请求可以通过线程并行化。

    遍历所有元素后，结果将在results：

    import requests
from multiprocessing.dummy import Pool as ThreadPool


urls = [
    'http://mysite.com/check.php?userid=666&userage=18',
    'http://mysite.com/check.php?userid=777&userage=22',
    'http://mysite.com/check.php?userid=888&userage=26',
]

pool = ThreadPool()
results = pool.map(requests.get, urls)
print(results)

    PS。虽然multiprocessing它指的是进程中的处理，但它的模块multiprocessing.dummy是通过线程工作的。

    附言 执行结果results会保存在 中requests.get(url)，你可以在之后进行处理，例如：

    for x in results:
    print(x, len(x.content), x.content)
    • 2
  2. Best Answer

    一种选择是使用异步。

    import asyncio
import requests

async def main():
    loop = asyncio.get_event_loop()
    future1 = loop.run_in_executor(None, requests.get, 'http://mysite.com/check.php?userid=666&userage=18')
    future2 = loop.run_in_executor(None, requests.get, 'http://mysite.com/check.php?userid=777&userage=22')
    future3 = loop.run_in_executor(None, requests.get, 'http://mysite.com/check.php?userid=888&userage=26')
    response1 = await future1
    response2 = await future2
    response3 = await future3
    print(response1.text)
    print(response2.text)
    print(response3.text)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

    这里有详细描述

    更新。如果需要设置标题，可以使用functools.partial：

    future1 = loop.run_in_executor(None, functools.partial(requests.get, 'http://httpbin.org/get', headers={'Test': '1'}))

    或者使用 aiohttp：

    import asyncio
from aiohttp import ClientSession


async def fetch(url, session):
    async with session.get(url, headers={'key': 'value'}) as response:
        return await response.read()

async def run():
    urls = (
        'http://mysite.com/check.php?userid=666&userage=18',
        'http://mysite.com/check.php?userid=777&userage=22',
        'http://mysite.com/check.php?userid=888&userage=26',
    )

    async with ClientSession() as session:
        tasks = [
            asyncio.ensure_future(fetch(url, session))
            for url in urls
        ]
        responses = await asyncio.gather(*tasks)
        print(responses)


loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run())
loop.run_until_complete(future)

    此外，您可以考虑缓存查询结果（如果任务上下文允许），例如使用 redis（https://github.com/vivekn/redis-simple-cache作为选项）。

    from redis_cache import cache_it, SimpleCache


my_cache = SimpleCache(expire=60)


@cache_it(cache=my_cache)
def get_data(url):
    ...
    • 1

相关问题