INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) INFO: 172.21.0.1:51452 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33600 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33614 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52148 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35252 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:35258 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56246 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52174 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39700 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47302 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47306 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38284 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36858 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36866 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46682 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53150 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34180 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58972 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58974 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39316 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34918 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34932 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43182 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50672 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38750 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49068 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49072 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36414 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47934 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47940 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37738 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:10:28,056 - lava_callback - INFO - Received callback for node 6962258b15f9f48f76d0ea25 2026-01-12 10:10:28,065 - lava_callback - INFO - Starting async job submit for node 6962258b15f9f48f76d0ea25 INFO: 172.21.0.1:37742 - "POST /node/6962258b15f9f48f76d0ea25 HTTP/1.0" 202 Accepted 2026-01-12 10:10:28,067 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-01-12 10:10:28,100 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962258b15f9f48f76d0ea25 HTTP/1.1" 502 150 2026-01-12 10:10:28,100 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962258b15f9f48f76d0ea25'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2026-01-12 10:10:28,100 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962258b15f9f48f76d0ea25 2026-01-12 10:10:28,102 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962258b15f9f48f76d0ea25 HTTP/1.1" 502 150 2026-01-12 10:10:28,102 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962258b15f9f48f76d0ea25'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2026-01-12 10:10:30,102 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962258b15f9f48f76d0ea25 2026-01-12 10:10:30,103 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962258b15f9f48f76d0ea25 HTTP/1.1" 502 150 2026-01-12 10:10:30,103 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962258b15f9f48f76d0ea25'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2026-01-12 10:10:34,103 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962258b15f9f48f76d0ea25 2026-01-12 10:10:34,105 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962258b15f9f48f76d0ea25 HTTP/1.1" 502 150 2026-01-12 10:10:34,105 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962258b15f9f48f76d0ea25'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:54880 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:10:42,105 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962258b15f9f48f76d0ea25 2026-01-12 10:10:42,106 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962258b15f9f48f76d0ea25 HTTP/1.1" 502 150 2026-01-12 10:10:42,106 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962258b15f9f48f76d0ea25'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:45760 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51750 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51764 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:10:58,107 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962258b15f9f48f76d0ea25 2026-01-12 10:10:58,108 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962258b15f9f48f76d0ea25 HTTP/1.1" 502 150 2026-01-12 10:10:58,108 - lava_callback - ERROR - Error processing callback for node 6962258b15f9f48f76d0ea25: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6962258b15f9f48f76d0ea25 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6962258b15f9f48f76d0ea25 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 284, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 668, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6962258b15f9f48f76d0ea25 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:56560 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39982 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39992 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60724 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47462 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39726 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:11:48,098 - lava_callback - INFO - Received callback for node 696225de15f9f48f76d0ea2a INFO: 172.21.0.1:39736 - "POST /node/696225de15f9f48f76d0ea2a HTTP/1.0" 202 Accepted 2026-01-12 10:11:48,104 - lava_callback - INFO - Starting async job submit for node 696225de15f9f48f76d0ea2a 2026-01-12 10:11:48,105 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-01-12 10:11:48,161 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/696225de15f9f48f76d0ea2a HTTP/1.1" 502 150 2026-01-12 10:11:48,162 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/696225de15f9f48f76d0ea2a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2026-01-12 10:11:48,162 - urllib3.connectionpool - DEBUG - Retry: /latest/node/696225de15f9f48f76d0ea2a 2026-01-12 10:11:48,162 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/696225de15f9f48f76d0ea2a HTTP/1.1" 502 150 2026-01-12 10:11:48,162 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/696225de15f9f48f76d0ea2a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2026-01-12 10:11:50,163 - urllib3.connectionpool - DEBUG - Retry: /latest/node/696225de15f9f48f76d0ea2a 2026-01-12 10:11:50,164 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/696225de15f9f48f76d0ea2a HTTP/1.1" 502 150 2026-01-12 10:11:50,164 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/696225de15f9f48f76d0ea2a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53604 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:11:54,164 - urllib3.connectionpool - DEBUG - Retry: /latest/node/696225de15f9f48f76d0ea2a 2026-01-12 10:11:54,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/696225de15f9f48f76d0ea2a HTTP/1.1" 502 150 2026-01-12 10:11:54,165 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/696225de15f9f48f76d0ea2a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53608 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:12:02,166 - urllib3.connectionpool - DEBUG - Retry: /latest/node/696225de15f9f48f76d0ea2a 2026-01-12 10:12:02,167 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/696225de15f9f48f76d0ea2a HTTP/1.1" 502 150 2026-01-12 10:12:02,167 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/696225de15f9f48f76d0ea2a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39754 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56622 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56630 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:12:18,167 - urllib3.connectionpool - DEBUG - Retry: /latest/node/696225de15f9f48f76d0ea2a 2026-01-12 10:12:18,168 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/696225de15f9f48f76d0ea2a HTTP/1.1" 502 150 2026-01-12 10:12:18,169 - lava_callback - ERROR - Error processing callback for node 696225de15f9f48f76d0ea2a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/696225de15f9f48f76d0ea2a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/696225de15f9f48f76d0ea2a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 284, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 668, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/696225de15f9f48f76d0ea2a (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:54358 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50962 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45272 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53856 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53870 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34854 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60030 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60036 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49972 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:13:28,191 - lava_callback - INFO - Received callback for node 6962264115f9f48f76d0ea37 INFO: 172.21.0.1:49982 - "POST /node/6962264115f9f48f76d0ea37 HTTP/1.0" 202 Accepted 2026-01-12 10:13:28,201 - lava_callback - INFO - Starting async job submit for node 6962264115f9f48f76d0ea37 2026-01-12 10:13:28,202 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-01-12 10:13:28,215 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962264115f9f48f76d0ea37 HTTP/1.1" 502 150 2026-01-12 10:13:28,215 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962264115f9f48f76d0ea37'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2026-01-12 10:13:28,215 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962264115f9f48f76d0ea37 2026-01-12 10:13:28,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962264115f9f48f76d0ea37 HTTP/1.1" 502 150 2026-01-12 10:13:28,216 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962264115f9f48f76d0ea37'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2026-01-12 10:13:30,216 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962264115f9f48f76d0ea37 2026-01-12 10:13:30,217 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962264115f9f48f76d0ea37 HTTP/1.1" 502 150 2026-01-12 10:13:30,217 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962264115f9f48f76d0ea37'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2026-01-12 10:13:34,218 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962264115f9f48f76d0ea37 2026-01-12 10:13:34,219 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962264115f9f48f76d0ea37 HTTP/1.1" 502 150 2026-01-12 10:13:34,219 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962264115f9f48f76d0ea37'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40584 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:13:42,219 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962264115f9f48f76d0ea37 2026-01-12 10:13:42,220 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962264115f9f48f76d0ea37 HTTP/1.1" 502 150 2026-01-12 10:13:42,220 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962264115f9f48f76d0ea37'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:48692 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43058 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43066 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:13:58,221 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962264115f9f48f76d0ea37 2026-01-12 10:13:58,222 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962264115f9f48f76d0ea37 HTTP/1.1" 502 150 2026-01-12 10:13:58,222 - lava_callback - ERROR - Error processing callback for node 6962264115f9f48f76d0ea37: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6962264115f9f48f76d0ea37 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6962264115f9f48f76d0ea37 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 284, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 668, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6962264115f9f48f76d0ea37 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:46668 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54836 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54840 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55712 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33416 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40456 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:14:48,267 - lava_callback - INFO - Received callback for node 6962269315f9f48f76d0ea3c INFO: 172.21.0.1:40470 - "POST /node/6962269315f9f48f76d0ea3c HTTP/1.0" 202 Accepted 2026-01-12 10:14:48,276 - lava_callback - INFO - Starting async job submit for node 6962269315f9f48f76d0ea3c 2026-01-12 10:14:48,277 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-01-12 10:14:48,317 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962269315f9f48f76d0ea3c HTTP/1.1" 502 150 2026-01-12 10:14:48,318 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962269315f9f48f76d0ea3c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2026-01-12 10:14:48,318 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962269315f9f48f76d0ea3c 2026-01-12 10:14:48,319 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962269315f9f48f76d0ea3c HTTP/1.1" 502 150 2026-01-12 10:14:48,319 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962269315f9f48f76d0ea3c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2026-01-12 10:14:50,319 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962269315f9f48f76d0ea3c 2026-01-12 10:14:50,320 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962269315f9f48f76d0ea3c HTTP/1.1" 502 150 2026-01-12 10:14:50,320 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962269315f9f48f76d0ea3c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39648 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:14:54,320 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962269315f9f48f76d0ea3c 2026-01-12 10:14:54,322 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962269315f9f48f76d0ea3c HTTP/1.1" 502 150 2026-01-12 10:14:54,322 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962269315f9f48f76d0ea3c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39658 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:15:02,322 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962269315f9f48f76d0ea3c 2026-01-12 10:15:02,323 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962269315f9f48f76d0ea3c HTTP/1.1" 502 150 2026-01-12 10:15:02,323 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6962269315f9f48f76d0ea3c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:34134 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34828 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34832 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:15:18,324 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6962269315f9f48f76d0ea3c 2026-01-12 10:15:18,325 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6962269315f9f48f76d0ea3c HTTP/1.1" 502 150 2026-01-12 10:15:18,325 - lava_callback - ERROR - Error processing callback for node 6962269315f9f48f76d0ea3c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6962269315f9f48f76d0ea3c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6962269315f9f48f76d0ea3c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 284, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 668, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6962269315f9f48f76d0ea3c (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:39574 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39156 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45946 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59918 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59934 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40680 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:16:08,343 - lava_callback - INFO - Received callback for node 696226e415f9f48f76d0ea47 INFO: 172.21.0.1:40694 - "POST /node/696226e415f9f48f76d0ea47 HTTP/1.0" 202 Accepted 2026-01-12 10:16:08,353 - lava_callback - INFO - Starting async job submit for node 696226e415f9f48f76d0ea47 2026-01-12 10:16:08,356 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-01-12 10:16:08,417 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/696226e415f9f48f76d0ea47 HTTP/1.1" 502 150 2026-01-12 10:16:08,417 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/696226e415f9f48f76d0ea47'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2026-01-12 10:16:08,417 - urllib3.connectionpool - DEBUG - Retry: /latest/node/696226e415f9f48f76d0ea47 2026-01-12 10:16:08,418 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/696226e415f9f48f76d0ea47 HTTP/1.1" 502 150 2026-01-12 10:16:08,418 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/696226e415f9f48f76d0ea47'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2026-01-12 10:16:10,418 - urllib3.connectionpool - DEBUG - Retry: /latest/node/696226e415f9f48f76d0ea47 2026-01-12 10:16:10,419 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/696226e415f9f48f76d0ea47 HTTP/1.1" 502 150 2026-01-12 10:16:10,419 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/696226e415f9f48f76d0ea47'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2026-01-12 10:16:14,419 - urllib3.connectionpool - DEBUG - Retry: /latest/node/696226e415f9f48f76d0ea47 2026-01-12 10:16:14,420 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/696226e415f9f48f76d0ea47 HTTP/1.1" 502 150 2026-01-12 10:16:14,421 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/696226e415f9f48f76d0ea47'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41470 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41478 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:16:22,421 - urllib3.connectionpool - DEBUG - Retry: /latest/node/696226e415f9f48f76d0ea47 2026-01-12 10:16:22,424 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/696226e415f9f48f76d0ea47 HTTP/1.1" 502 150 2026-01-12 10:16:22,424 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/696226e415f9f48f76d0ea47'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38312 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53984 - "GET / HTTP/1.0" 200 OK 2026-01-12 10:16:38,425 - urllib3.connectionpool - DEBUG - Retry: /latest/node/696226e415f9f48f76d0ea47 2026-01-12 10:16:38,426 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/696226e415f9f48f76d0ea47 HTTP/1.1" 502 150 2026-01-12 10:16:38,426 - lava_callback - ERROR - Error processing callback for node 696226e415f9f48f76d0ea47: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/696226e415f9f48f76d0ea47 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/696226e415f9f48f76d0ea47 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 284, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 668, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/696226e415f9f48f76d0ea47 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:41868 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48060 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48070 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52296 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40162 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40166 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32904 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36838 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54724 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57812 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57816 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59582 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58014 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58028 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35444 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44922 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45456 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34340 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34356 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47030 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34396 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34406 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60276 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52670 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33888 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35268 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35270 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55876 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50940 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50950 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50320 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55048 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51694 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54988 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54992 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58956 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41202 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41208 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38934 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59320 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33676 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44124 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44138 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50384 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58154 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58158 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55128 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52116 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53138 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36032 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36044 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45274 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34134 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34142 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37790 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33706 - "GET / HTTP/1.0" 200 OK INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [1]