LONG delay in requests.get Python

Question

I’m having a problem that when I perform requests.get diverse sometimes python starts to take A LONG time to get the request.

I already thought it may be the API itself refusing my access, because it would be doing more than allowed, but according to this link I was able to say that this was not the problem. (But since I really have no idea what’s going on, I got in touch and they told me that this is really the limit of requests, that the site is correct.)

And this problem has happened to me before with another link, so I don’t know why this problem is happening.

What my system does would be something similar to:

import requests
import time

While True:
  
 requests.get("https://brasilbitcoin.com.br/API/orderbook/XRP")

 '''
  trato o requests.get
 '''

 time.sleep(1)

I’ve confirmed that the problem is requests and not any other part of the code. Previously there was the time.sleep(1) but I ended up adding it through doubts.

When I say I make a lot of requests, I’m talking about one per second for 24 hours a day. But after a while the requisition started to get EXTREMELY slow. When I restart the server to see if this is the problem, it returns to make the requests very well, but when it arrives around 200 requests it starts to take too long again.

(Right now it is taking ~44 seconds PER request on this link, and the trend is being this time increasing more and more).

Can anyone tell me what might be going on?

I’m using Windows and Python 3.8.2

Answer 1

There’s nothing wrong with the Python nor with the library requests!

Your repeated requests with this short interval of time between them are certainly making the "target" server understand that this is an "attack".

Response time is getting longer as the server is penalizing its source address due to the large volume of requests in a short amount of time.

To documentation of the REST API you are accessing says the following:

Authentication and Answers

The authentication key must be sent in the request header, and the answers will be returned in JSON.

Limits of Requests

The request limit is 4 requests per minute for creating orders. For other calls, the limit is 1 request per second. The parameter 'Wait', when any, refers to the time at seconds left for a new call.

There is even the need to make this volume of requests in such a short time?

Answer 2

Already tested explicitly send a keep_alive=False? Try modifying the configuration just before your iteration. Implementing this way I could observe 500+ requests in stdout.

import requests
import time

i = 0
s = requests.Session()
s.stream = False

while True:
    req = s.get("https://brasilbitcoin.com.br/API/orderbook/XRP")
    print("req:", i)
    i += 1

    # trata o req

    time.sleep(1)

I believe you may or may be reaching the limit of requests opened by your python process, or the request limit imposed by the API. I would not rule out the second option even if you are following the documentation, respecting the range of 1 req/sec.

I’ve had problems with other Xchanges Apis, and the solution I took was to migrate to websocket, but unfortunately I believe that in your case is not an option since they do not offer a wss API :(

Version of the requests I used (output from pip3 show requests):

Name: requests
Version: 2.24.0
Summary: Python HTTP for Humans.
Home-page: https://requests.readthedocs.io
Author: Kenneth Reitz
Author-email: [email protected]
License: Apache 2.0
Location: /home/lzgustavo/.local/lib/python3.6/site-packages
Requires: certifi, urllib3, chardet, idna