python – GET operation using Django rest_framework gets ConnectionRefusedError [Errno 111] after several hours of normal operation

I setup use Django to create an APP (let’s call it central APP) that aggregate operation data from multiple applications on remote server.

I then installed Django REST_Framework, to allow other applications, like Prometheus to get information from my Django central APP. The related Django APPs is shown below

INSTALLED_APPS = [

    ... #central APP I created

    'django_prometheus', #This is not used
    'rest_framework',
    'django_filters',
    'drf_multiple_model',
   
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

Other applications, like Prometheus, can successfully get info return. The API call frequency is 1 minutes per GET request.

However, after around 5 hour of normal operation, Django will stop working. It seems the Djagon rest_framework cause the exception:

  File "/home/centos/.local/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc

The full exception return is shown below:

Internal Server Error: /cpnr_monitoring/cpnr_grafana/
Traceback (most recent call last):
  File "/home/centos/.local/lib/python3.8/site-packages/pymysql/connections.py", line 613, in connect
    sock = socket.create_connection(
  File "/usr/local/lib/python3.8/socket.py", line 808, in create_connection
    raise err
  File "/usr/local/lib/python3.8/socket.py", line 796, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/home/centos/.local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/home/centos/.local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 234, in get_new_connection
    return Database.connect(**conn_params)
  File "/home/centos/.local/lib/python3.8/site-packages/pymysql/connections.py", line 353, in __init__
    self.connect()
  File "/home/centos/.local/lib/python3.8/site-packages/pymysql/connections.py", line 664, in connect
    raise exc
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '127.0.0.1' ([Errno 111] Connection refused)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/centos/.local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/centos/.local/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/centos/.local/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/centos/.local/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/centos/.local/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/home/centos/.local/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/home/centos/.local/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/home/centos/.local/lib/python3.8/site-packages/rest_framework/views.py", line 497, in dispatch
    self.initial(request, *args, **kwargs)
  File "/home/centos/.local/lib/python3.8/site-packages/rest_framework/views.py", line 414, in initial
    self.perform_authentication(request)
  File "/home/centos/.local/lib/python3.8/site-packages/rest_framework/views.py", line 324, in perform_authentication
    request.user
  File "/home/centos/.local/lib/python3.8/site-packages/rest_framework/request.py", line 227, in user
    self._authenticate()
  File "/home/centos/.local/lib/python3.8/site-packages/rest_framework/request.py", line 380, in _authenticate
    user_auth_tuple = authenticator.authenticate(self)
  File "/home/centos/.local/lib/python3.8/site-packages/rest_framework/authentication.py", line 87, in authenticate
    return self.authenticate_credentials(userid, password, request)
  File "/home/centos/.local/lib/python3.8/site-packages/rest_framework/authentication.py", line 98, in authenticate_credentials
    user = authenticate(request=request, **credentials)
  File "/home/centos/.local/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 73, in authenticate
    user = backend.authenticate(request, **credentials)
  File "/home/centos/.local/lib/python3.8/site-packages/django/contrib/auth/backends.py", line 42, in authenticate
    user = UserModel._default_manager.get_by_natural_key(username)
  File "/home/centos/.local/lib/python3.8/site-packages/django/contrib/auth/base_user.py", line 45, in get_by_natural_key
    return self.get(**{self.model.USERNAME_FIELD: username})
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/models/query.py", line 425, in get
    num = len(clone)
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/models/query.py", line 269, in __len__
    self._fetch_all()
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/models/query.py", line 1308, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1154, in execute_sql
    cursor = self.connection.cursor()
  File "/home/centos/.local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 259, in cursor
    return self._cursor()
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 235, in _cursor
    self.ensure_connection()
  File "/home/centos/.local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/home/centos/.local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/home/centos/.local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/centos/.local/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 234, in get_new_connection
    return Database.connect(**conn_params)
  File "/home/centos/.local/lib/python3.8/site-packages/pymysql/connections.py", line 353, in __init__
    self.connect()
  File "/home/centos/.local/lib/python3.8/site-packages/pymysql/connections.py", line 664, in connect
    raise exc
django.db.utils.OperationalError: (2003, "Can't connect to MySQL server on '127.0.0.1' ([Errno 111] Connection refused)")
"GET /cpnr_monitoring/cpnr_grafana/?pretty=True HTTP/1.1" 500 21729

In the past two weeks, I have disabled the REST_Framework API url, meaning other applications can not get info from the central APP through REST API. And the system is running fine. So I am pretty sure the problem comes from the API call.

I have also tried things like changing the allowed memory allocation for MySQL, change the localhost to ‘127.0.0.1’, but none of them help the system stop after 5 hour normal operation. Right now I ran out of ideas how to debug/improve.

Any suggestions would be appreciated.

Leave a Comment