How is the pymongo connection established?

Question

I’m building a program that scans a database every minute, and if some conditions are met, the program makes several requests via Apis and talks to Mongodb to process and save the data. It is very important that I do not lose any data while the program is running continuously.

To (try) avoid this, I created a function tries multiple times the internet connection. The function is much larger, but to get an idea:

def try_connection(func, *args, **kwargs):
    for _ in range(10):
        try:
            r = func(*args, **kwargs)

        #Diversos outros excepts

        except Exception as e:
            logging.error(f'Erro inesperado: {e.__class__}, {e}', exc_info=True)
            sleep(30)

I was passing my queries to Mongodb within the function try_connection. Something like:

alarms = try_connection(collection.find)
alarms = try_connection(alarms.sort, 'alarmID', -1)

However, in the tests I performed (on a network without VPN closed to the bank, making my access impossible) via console, connection problems do not happen at the time of query, that is, when the cursor is formed, but when I will iterate on it.

>>> user_mongo = urllib.parse.quote_plus(config.user_mongo)
>>> pass_mongo = urllib.parse.quote_plus(config.pass_mongo)
>>> c = try_connection(pymongo.MongoClient, 
        f'mongodb://{user_mongo}:{pass_mongo}@10.0.0.0:27017/{db_name}')
>>> db = c[db_name]
>>> collection = db[collection_name]
>>> alarms = try_connection(collection.find)
>>> alarms = try_connection(alarms.sort, 'alarmID', -1)
>>> print(alarms)
<pymongo.cursor.Cursor object at 0x034EDCF0>
>>> for alarm in alarms:
...     print(alarm)
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Murilo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\cursor.py", line 611, in __getitem__
    for doc in clone:
  File "C:\Users\Murilo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\cursor.py", line 1225, in next
    if len(self.__data) or self._refresh():
  File "C:\Users\Murilo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\cursor.py", line 1117, in _refresh
    self.__session = self.__collection.database.client._ensure_session()
  File "C:\Users\Murilo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\mongo_client.py", line 1598, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "C:\Users\Murilo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\mongo_client.py", line 1551, in __start_session
    server_session = self._get_server_session()
  File "C:\Users\Murilo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\mongo_client.py", line 1584, in _get_server_session
    return self._topology.get_server_session()
  File "C:\Users\Murilo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\topology.py", line 429, in get_server_session
    None)
  File "C:\Users\Murilo\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\topology.py", line 200, in _select_servers_loop
    self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: 10.0.0.0:27017: timed out

It made me think that the cursor only points to the places where the data from the query are located (rather than storing the data, as I thought. I thought this supposed storage was the reason we had to use a rewind() to access the cursor after an iteration). I tried to look for information in the official documents, but found no satisfactory answer.

So, how the connection is established? To try to keep me from losing some data I must always call try_connection when iterating on the cursores instead of calling at the time of queries?

Answer 1

To documentation sets a cursor as:

A Pointer to the result set of a query. Clients can iterate through a cursor to Retrieve Results.

That is, the cursor only points to the result, rather than storing it. To get around my problem I should encapsulate my iterations in a function and call try_connection whenever you execute her.

However, I thought it best to create one more method within my class Mongo and call it before performing an iteration.

class Mongo:
    def __init__(self, db):
        self.db = db

...
...
...

    def check_connection(self):
        try_connection(self.db.client.admin.command, 'ismaster')

c.admin.command('ismaster') is the command indicated by documentation pymongo to check if a connection has been established. (c == db.client)

In my main code, I do.

mongo = Mongo(db)

...
...
...

ids = clients.find(
                    {
                    }, 
                    {
                        'serviceID': 1, 
                        'wrongSeverity': 1, 
                        '_id': 0
                    }
                )
mongo.check_connection()
ids = [id['serviceID'] for id in ids if not id['wrongSeverity']]